Enforced Admission Control Policy Using Gatekeeper

Four violations from one kubectl run, and what a blocked debug pod taught me about OPA Gatekeeper on GKE Autopilot

I was troubleshooting a service in a GKE Autopilot cluster and did what everyone does: spin up a throwaway pod with curl in it and poke at things from inside the cluster.

kubectl run debug-pod \
  -n default \
  --image=curlimages/curl:8.10.1 \
  --restart=Never \
  -it --rm -- sh

Here’s what I got back:

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request:
[enforce-container-resources] Container 'debug-pod' is missing resources.limits.cpu
[enforce-container-resources] Container 'debug-pod' is missing resources.limits.memory
[deny-root-execution] Container 'debug-pod' must run as non-root. Set securityContext.runAsNonRoot=true or runAsUser > 0
[restrict-to-allowed-namespaces] Deployments are strictly forbidden in 'default'. Allowed namespaces are: ["dev"]

Four violations. One command. Zero pods created.

My first reaction was mild annoyance. My second reaction was that this is exactly what I built the policies for, and it worked on the person most likely to cut corners: me.

This post walks through installing Gatekeeper, the three policies doing the enforcing, how to read the denial message properly, one subtle thing the Autopilot warning tells you, and the gaps I found in my own Rego once I went back and looked at it honestly.

Installing Gatekeeper

Gatekeeper ships as a Helm chart. Add the repo and refresh:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update

"gatekeeper" has been added to your repositories

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gatekeeper" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Then install it, with the validating webhook scoped to the workload kinds I actually care about:

helm install gatekeeper gatekeeper/gatekeeper \
  --namespace gatekeeper-system \
  --create-namespace \
  '--set=validatingWebhookTimeoutSeconds=3' \
  '--set=validaterules[0].apiGroups[0]=*' \
  '--set=validaterules[0].apiVersions[0]=*' \
  '--set=validaterules[0].operations[0]=CREATE' \
  '--set=validaterules[0].operations[1]=UPDATE' \
  '--set=validaterules[0].resources[0]=pods' \
  '--set=validaterules[0].resources[1]=deployments' \
  '--set=validaterules[0].resources[2]=statefulsets' \
  '--set=validaterules[0].resources[3]=daemonsets' \
  '--set=validaterules[0].resources[4]=jobs'

A few notes on that command, because the flags are doing more work than they look like they are.

Add --create-namespace unless gatekeeper-system already exists. Helm's --namespace selects a namespace, it does not create one. Without the flag you get namespaces "gatekeeper-system" not found and no release.

validaterules is the reason this install is worth customising. The chart's default validating webhook configuration is broad, which means the API server calls Gatekeeper on a very wide range of admission requests. Every one of those calls is a network hop in the critical path of writing to your cluster. Narrowing the rule to pods, deployments, statefulsets, daemonsets and jobs on CREATE and UPDATE does two things: it cuts the request volume hitting the webhook, and it shrinks the blast radius if Gatekeeper ever becomes unhealthy. Nothing outside those five kinds can be affected by an outage in the policy controller.

Note that the values use plural, lowercase resource names (pods), while the constraints later use singular, capitalised kind names (Pod). That's not an inconsistency in the tooling. The webhook configuration speaks the API server's resource vocabulary and the constraint matcher speaks the kind vocabulary. Mixing them up is a common way to end up with a policy that never fires, because the webhook was never invited to the request in the first place.

Keep the webhook timeout short. Three seconds is a reasonable starting point. The webhook’s failure policy determines what happens when Gatekeeper doesn’t answer in time, and a long timeout on a struggling controller turns every pod creation in the cluster into a slow one.

--set values need quoting in most shells. The square brackets in validaterules[0].resources[0] will be eaten by glob expansion if you leave them bare, which is why each one is wrapped in single quotes above.

Confirm it came up before you write any policy:

kubectl get pods -n gatekeeper-system
kubectl get crd | grep gatekeeper

You should see the audit controller and the webhook replicas running, plus the constrainttemplates.templates.gatekeeper.sh CRD. Once the templates are applied, each one generates its own CRD too, which is how K8sRequireResources becomes a thing you can kubectl get.

One GKE Autopilot caveat: Autopilot manages its own namespaces and will not let a third-party webhook intercept objects in them. That’s a guardrail rather than a problem, but it does mean the exclusions in your constraints and the exclusions Autopilot enforces are two separate layers, and you should not assume one covers the other.

What’s doing the blocking

The cluster runs OPA Gatekeeper with three ConstraintTemplate + Constraint pairs. A ConstraintTemplate defines the logic in Rego and generates a new CRD. A Constraint is an instance of that CRD saying where the logic applies and with what parameters.

1. Namespace restriction

Workloads are only permitted in dev. Everything else is denied unless it's in the exclusion list.

violation[{"msg": msg}] {
  ns := input.review.object.metadata.namespace
  not namespace_is_allowed(ns, input.parameters.allowedNamespaces)
  msg := sprintf("Deployments are strictly forbidden in '%v'. Allowed namespaces are: %v",
                 [ns, input.parameters.allowedNamespaces])
}

namespace_is_allowed(ns, allowed_list) {
  ns == allowed_list[_]
}

The constraint matches Deployment, StatefulSet, DaemonSet, Pod, and Job, and excludes the GKE-managed namespaces (kube-system, gke-system, gke-gmp-system, the gke-managed-* family) plus the operations tooling: gatekeeper-system, istio-system, argocd, apisix, cert-manager, prometheus, grafana, kiali, jaeger, sonarqube.

Get Gayan Dissanayake’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That exclusion list is not optional housekeeping. If you forget gatekeeper-system, you can lock Gatekeeper itself out of scheduling its own pods and end up with a cluster that rejects the fix for the problem it has.

2. Required resource requests and limits

Four separate violation rules, one each for requests.cpu, requests.memory, limits.cpu, and limits.memory.

violation[{"msg": msg}] {
  containers[c]
  not c.resources.limits.cpu
  msg := sprintf("Container '%v' is missing resources.limits.cpu", [c.name])
}

containers[c] { c := input.review.object.spec.containers[_] }
containers[c] { c := input.review.object.spec.template.spec.containers[_] }

Those two containers rules at the bottom are the useful pattern. A bare Pod puts containers at spec.containers, while a Deployment or Job puts them at spec.template.spec.containers. Defining the same partial rule twice gives you one set that covers both, so the violation logic doesn't care what kind it's looking at.

3. No root, no privilege escalation

violation[{"msg": msg}] {
  containers[c]
  not is_non_root(c, input.review.object)
  msg := sprintf("Container '%v' must run as non-root. Set securityContext.runAsNonRoot=true or runAsUser > 0", [c.name])
}

is_non_root(c, review) { c.securityContext.runAsNonRoot == true }
is_non_root(c, review) { c.securityContext.runAsUser > 0 }
is_non_root(c, review) { review.spec.securityContext.runAsNonRoot == true }
is_non_root(c, review) { review.spec.securityContext.runAsUser > 0 }
is_non_root(c, review) { review.spec.template.spec.securityContext.runAsNonRoot == true }
is_non_root(c, review) { review.spec.template.spec.securityContext.runAsUser > 0 }

Six definitions of is_non_root, and in Rego any one of them succeeding is enough. That covers the container-level security context and the pod-level one, for both bare pods and templated workloads. It means a team can set runAsNonRoot once at the pod level instead of repeating it on every container.

Reading the denial properly

Two things about that error output are worth pointing out.

Gatekeeper returns every violation at once. It doesn’t stop at the first failure and make you fix problems one round trip at a time. All three constraints evaluated, all four violations came back in a single response. That’s a real developer-experience win over webhooks that fail fast, and it’s the difference between one fix and four rejected kubectl calls.

The message wording gave away a bug. Look at the last line:

[restrict-to-allowed-namespaces] Deployments are strictly forbidden in 'default'.

I didn’t create a Deployment. I created a Pod. The sprintf string is hardcoded to say "Deployments" while the constraint matches five different kinds. It's cosmetic, but policy messages are the only interface most developers ever have with your policies, and a message that describes the wrong object type sends people looking in the wrong place. The fix is one line:

msg := sprintf("%v is not allowed in namespace '%v'. Allowed namespaces are: %v",
               [input.review.kind.kind, ns, input.parameters.allowedNamespaces])

Write your violation messages as if the reader has never seen your Rego, because they haven’t.

Actually getting the debug pod

The fix is to stop arguing with the policy and satisfy it. Use the allowed namespace, and pass a --overrides blob that declares resources and a security context:

kubectl run debug-pod \
  -n dev \
  --image=curlimages/curl:8.10.1 \
  --restart=Never -it --rm \
  --overrides='{
    "spec": {
      "securityContext": { "runAsNonRoot": true, "runAsUser": 1000 },
      "containers": [{
        "name": "debug-pod",
        "image": "curlimages/curl:8.10.1",
        "stdin": true,
        "tty": true,
        "resources": {
          "requests": { "cpu": "250m", "memory": "256Mi" },
          "limits":   { "cpu": "250m", "memory": "256Mi" }
        }
      }]
    }
  }' -- sh

One caution on runAsNonRoot. The kubelet validates it against the image's USER directive, and if the image declares a username rather than a numeric UID, the kubelet can't verify the user is non-root and the pod fails to start with an error about a non-numeric user. Setting runAsUser explicitly, as above, avoids that class of problem regardless of how the image is built.

Honestly, though, if your team needs a debug pod more than twice a month, don’t make everyone hand-assemble JSON overrides on the command line. Commit a debug-pod.yaml that already complies and tell people to kubectl apply it. Policies that are tedious to satisfy get routed around, and the route around is usually someone quietly getting an exclusion added.

Gaps I found in my own policies

Going back through the Rego after this, three things stood out.

initContainers and ephemeralContainers aren't covered. The containers rule only collects spec.containers and spec.template.spec.containers. An init container can run as root with no limits and sail straight through. So can an ephemeral container, which is what kubectl debug attaches to a running pod. The fix is more partial rules:

containers[c] { c := input.review.object.spec.initContainers[_] }
containers[c] { c := input.review.object.spec.template.spec.initContainers[_] }
containers[c] { c := input.review.object.spec.ephemeralContainers[_] }

Privilege escalation is only blocked when explicitly requested. The rule fires on c.securityContext.allowPrivilegeEscalation == true. Omit the field entirely and there's no violation, even though omitting it is not the same as setting it to false in the kubelet's eyes. If the intent is "this must be off," require it to be off:

violation[{"msg": msg}] {
  containers[c]
  not c.securityContext.allowPrivilegeEscalation == false
  msg := sprintf("Container '%v' must set securityContext.allowPrivilegeEscalation: false", [c.name])
}

Absence-of-bad and presence-of-good are different policies. Pick the one you meant.

The resource policy has no parameter schema. K8sRequireResources has no validation.openAPIV3Schema block, so there's nowhere to put a minimum or maximum. Right now it enforces that limits exist, not that they're sane. A container asking for 64 CPUs passes cleanly. Adding parameters for maximum allowed values is the natural next step, and it's the same shape as the allowedNamespaces parameter that policy #1 already uses.

If you’re rolling this out on an existing cluster

Do not go straight to deny. Constraints support an enforcementAction field, and the sequence that avoids an incident is:

1. Deploy with enforcementAction: dryrun. Nothing is blocked.
2. Let the audit controller run. It periodically evaluates existing objects in the cluster against your constraints and writes results to the constraint’s status.violations. kubectl get k8srequireresources enforce-container-resources -o yaml will show you exactly what would have been rejected.
3. Fix the manifests that show up, or add legitimate namespaces to excludedNamespaces.
4. Optionally move to enforcementAction: warn so developers see the message without being blocked.
5. Switch to deny once the violation list is empty.

Worth remembering that Gatekeeper’s validating webhook only fires on admission. Flipping to deny does not evict anything already running. Your non-compliant workloads keep running happily until the next time something tries to recreate them, which is usually a node upgrade, at the least convenient hour.

And a note on where this sits in the stack: a namespace restriction enforced by admission policy is a backstop, not an access control. RBAC is what should stop someone from creating workloads in default in the first place. Gatekeeper catches the CI pipeline with a stale -n flag and the engineer who forgot to switch contexts. Those are worth catching. They're just not the same problem as an untrusted user.

The point

The best outcome from a policy engine is a boring one: you try to do something careless, it explains what’s wrong in plain language, and you do the right thing instead. Four violations in one response, each naming the exact field to set.

The part I didn’t expect was how much reviewing my own denial message taught me. One wrong word in a sprintf sent me back to the Rego, and I came out with three real gaps to close. If you've written Gatekeeper policies and never been blocked by them yourself, that's not evidence they work. It might just mean nobody's been running your fast path.

Go break your own cluster on purpose. Read the message carefully. It’s usually telling you more than you asked.