Stop Letting Every Service Handle Its Own Security
by Eric Hanson, Backend Developer at Clean Systems Consulting
The inconsistency problem at scale
With five services, inconsistent security implementations are a maintenance annoyance. With twenty services owned by eight teams, they become a genuine risk. Team 3 hardened their TLS configuration because someone on that team read a security advisory. Teams 1, 2, and 4 through 8 didn't. Team 6 rotates secrets quarterly. Teams 1 through 5 and 7 through 8 haven't rotated since initial deployment. Team 2 runs their containers as root because it was easier at the time.
This is not a people problem. It is a system design problem. When security is delegated to individual service teams, security posture varies with team knowledge, bandwidth, and priorities — all of which are inconsistent across teams and over time. The solution is to move security controls to the platform layer where they apply uniformly, automatically, and don't require every team to make correct independent decisions.
What the platform layer should enforce
TLS everywhere, enforced by infrastructure: a service mesh (Istio in STRICT mTLS mode, Linkerd) encrypts all inter-service traffic and enforces mutual authentication without any service needing to implement it. Teams write their service code; the mesh handles TLS. Teams cannot opt out, forget, or misconfigure it because the control is not in their hands.
Container security baselines via admission control: Kubernetes admission controllers (OPA Gatekeeper, Kyverno) enforce security policies at deploy time. A policy that prevents containers from running as root applies to every service without every team needing to remember to set securityContext.runAsNonRoot: true:
# Kyverno policy: enforce non-root containers cluster-wide
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-non-root-containers
spec:
validationFailureAction: Enforce
rules:
- name: check-runAsNonRoot
match:
resources:
kinds: [Pod]
validate:
message: "Containers must run as non-root"
pattern:
spec:
containers:
- =(securityContext):
runAsNonRoot: true
This policy applies at admission — a deployment that violates it is rejected before it ever reaches the cluster. No per-team action required.
Secrets management as infrastructure: every service gets secrets from Vault (or AWS Secrets Manager, GCP Secret Manager) via a platform-managed integration. The platform team configures the Vault sidecar injector; service teams annotate their deployments to request the secrets they need. Rotation is automatic. No service team owns a kubectl edit secret workflow.
# Pod annotation: request secrets from Vault, injected by platform
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "order-service"
vault.hashicorp.com/agent-inject-secret-db: "secret/data/order-service/database"
Network policies as default-deny: the platform provisions a default-deny NetworkPolicy for every namespace. Service teams request explicit ingress/egress rules for the connections they need. The default state is no connectivity, not full connectivity. This inverts the security model from "everything allowed unless explicitly blocked" to "nothing allowed unless explicitly permitted."
The platform team contract
Centralizing security enforcement only works if the platform team treats service teams as customers with a support contract:
- Policy changes must be communicated in advance with migration paths
- Emergency security patches (critical CVE, compromised credentials) must be applied quickly without requiring per-team action
- Service teams must be able to understand why a deployment was rejected and what they need to change
- Escape hatches (audit-mode policies, temporary exceptions) must exist for legitimate edge cases, with clear approval and expiry processes
Without this contract, service teams work around security enforcement. They find ways to deploy without the admission controller, they use alternative namespaces with looser policies, they run secrets in environment variables because getting Vault access takes three weeks. Platform security that's too restrictive to work with is worse than no platform security, because it creates an adversarial relationship between security and engineering.
What remains with service teams
Centralized platform security does not mean service teams have no security responsibility. The platform handles transport security, secrets injection, container policy, and network isolation. Service teams own:
- Authorization logic for their own resources (who can read or write what data)
- Input validation and sanitization (SQL injection, deserialization vulnerabilities)
- Dependency security (keeping libraries updated, responding to CVEs in direct dependencies)
- Secure coding practices (no hardcoded credentials, no logging of sensitive data, proper error handling that doesn't leak internal state)
These are domain-specific concerns that the platform cannot enforce for you. They require engineering discipline and code review, not admission controllers.
The division is clear: the platform enforces the infrastructure security baseline uniformly. Service teams own the application security concerns specific to their domain. Neither delegates to the other. Both are necessary.