Cilium Part 2: Writing Network Policies Without Breaking Your Cluster

Here’s a request that should never work in production, and works perfectly in a fresh Kubernetes cluster:

$ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Panic: deathstar exploded

That’s from the Cilium docs’ Star Wars demo, where a ship calls a maintenance API it has no business touching. It’s a joke, but the problem is real: by default, every pod in a Kubernetes cluster can talk to every other pod, on any port, with any request.

In Part 1 ([https://pratikmahalle.medium.com/what-cilium-is-and-what-it-actually-does-with-your-packets-dc7adc264a90]), I covered what Cilium is and how it uses identities instead of IP addresses. This part puts that to work. Using the same demo app from the docs, we’ll lock it down step by step: rehearse a policy without breaking anything, allow only the traffic we need, restrict an HTTP API, control what pods can reach on the internet, and debug things when they get blocked.

Locking down the Death Star, one rule at a time

Everything is allowed until it isn’t

The one rule that explains most policy surprises is this: a pod allows all traffic until a policy selects it. The moment any policy selects a pod, that pod switches to default deny, and only traffic you’ve explicitly allowed gets through.

That switch happens separately for each direction. If the policy has an ingress section, incoming traffic becomes default deny. If it has an egress section, outgoing traffic does. A policy with only ingress rules leaves the pod's outgoing traffic completely open.

There’s also a cluster-wide setting, policyEnforcementMode, that changes this starting point. The default mode works as described above, always puts every pod in default deny even with no policy, and never turns enforcement off. Most clusters stay on the default, and that's what this post assumes.

One more thing worth knowing early: a blocked connection doesn’t fail with an error. By default Cilium silently drops the packets, so the application just waits until it times out. If you’ve ever stared at a hanging curl wondering whether the service is down, this is often why.

Setting up the playground

The demo has three apps. deathstar is an HTTP service on port 80, tiefighter is an Empire ship, and xwing is a Rebel ship. Their labels are what our policies will use: org=empire for the Empire's pods and org=alliance for the X-wing.

$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.20.3/examples/minikube/http-sw-app.yaml
service/deathstar created
deployment.apps/deathstar created
pod/tiefighter created
pod/xwing created

Right now both ships can land, because nothing selects any pod yet:

$ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

You can see the same thing from Cilium’s side. Every pod shows up as an endpoint, with its identity and with policy enforcement Disabled in both directions (output trimmed):

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])
           ENFORCEMENT        ENFORCEMENT
232        Disabled           Disabled          16530      k8s:class=deathstar
1673       Disabled           Disabled          31028      k8s:class=tiefighter
3184       Disabled           Disabled          22654      k8s:class=xwing

Keep an eye on those identity numbers: 16530 for the Death Star, 31028 for the TIE fighter and 22654 for the X-wing. They’ll show up again in every verdict below. By the end of this post, the traffic will look like this:

Solid arrows are traffic a rule allows. Dashed ones end at a bar where Cilium drops or rejects the request.

Rehearse first with audit mode

The scary part of the first policy is that it flips pods into default deny, and anything you forgot to allow stops working. Cilium has a safety net for this called policy audit mode. With it turned on, Cilium enforces nothing, but it logs every connection that would have been dropped. You get to see the consequences before they happen.

helm upgrade cilium cilium/cilium --version 1.20.3 \
   --namespace kube-system \
   --reuse-values \
   --set policyAuditMode=true

Now apply a default-deny policy for every Empire pod. The empty rule - {} under ingress allows nothing; its only job is to select the pods and switch their incoming traffic to default deny:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "empire-default-deny"
spec:
  description: "Default-deny ingress policy for the empire"
  endpointSelector:
    matchLabels:
      org: empire
  ingress:
  - {}

Send a landing request and look at the verdict with Hubble (see Part 1 for setting it up):

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

$ hubble observe --type policy-verdict --last 1
Feb  7 12:53:39.168: default/tiefighter:54134 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none AUDITED (TCP Flags: SYN)

The ship still landed, but Hubble says policy-verdict:none AUDITED: no rule matched, so with enforcement on, this request would have been dropped. That's your to-do list. Every AUDITED line is traffic you need to either allow or accept losing. Audit mode isn't meant for production, so treat it as a rehearsal and switch it off once your policies are in place.

Allow only what you need

The Death Star should only take landing requests from Empire ships, on port 80. Here’s that rule, written entirely in labels:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: "L3-L4 policy to restrict deathstar access to empire ships only"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP

Read it top to bottom: this applies to pods labelled org=empire, class=deathstar, and lets in traffic from any pod labelled org=empire on TCP port 80. You don't need a matching rule for the replies, because Cilium tracks connections and lets the response packets back automatically.

With the policy applied, the same request that was AUDITED before now matches a rule:

$ hubble observe --type policy-verdict --last 1
Feb  7 13:06:45.130: default/tiefighter:59824 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)

Once the verdicts look right, turn audit mode off by running the same helm upgrade with policyAuditMode=false. Now the policy is real. The TIE fighter still lands, and the X-wing's request just hangs until it times out:

$ kubectl exec xwing -- curl --connect-timeout 3 -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
command terminated with exit code 28

$ hubble observe --type policy-verdict --from-pod xwing --last 1
Feb  7 13:43:46.791: default/xwing:54842 (ID:22654) <> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none DENIED (TCP Flags: SYN)

If you run cilium-dbg endpoint list again, the Death Star pods now show ingress enforcement Enabled, while their egress is still Disabled, because our policy only has an ingress section.

Going up to HTTP

The X-wing is out, but we haven’t fixed the opening problem. The TIE fighter is an Empire ship, so it’s allowed in on port 80, and port 80 includes PUT /v1/exhaust-port. An L3/L4 rule can say who may connect and on which port, but not what they're allowed to ask for.

Get Pratik Mahalle’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That’s what L7 rules are for. Adding a rules.http block to the same policy limits Empire ships to the one call they actually need:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: "L7 policy to restrict access to specific HTTP call"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/v1/request-landing"

Now the landing request still works, and the dangerous one is refused:

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

$ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Access denied

Notice the difference from the X-wing. This time the request didn’t hang; it got an immediate Access denied, which is an HTTP 403. As covered in Part 1, HTTP rules can't be checked on a single packet, so Cilium sends this traffic through its Envoy proxy, which reads each request and answers rejected ones itself.

Two small things worth knowing: path is a regular expression that has to match the whole path, so to allow everything under /v1/ you'd write "/v1/.*". And once a port has HTTP rules, any request on that port that doesn't match one is denied, so list every call the client legitimately makes.

Egress, and the DNS trap

So far we’ve only controlled who can reach the Death Star. Controlling what a pod can reach is just as important, especially on the internet. The docs use a mediabot pod that needs GitHub's API and nothing else.

Here’s the trap. The moment you add an egress section, the pod goes into default deny for outgoing traffic, and that includes DNS. Allow api.github.com and nothing else, and the pod can't even look up api.github.com, so everything fails with errors that look nothing like a policy problem.

The fix is to always allow DNS to kube-dns alongside your egress rules. Cilium turns this into a feature: the rules.dns block sends lookups through Cilium's DNS proxy, which is how a toFQDNs rule learns which IP addresses belong to a domain name. You never write IP addresses yourself:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

The first rule allows api.github.com. The second allows DNS queries to kube-dns, and the k8s:io.kubernetes.pod.namespace label is needed because policies only look at the policy's own namespace unless you say otherwise. The result:

$ kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/2 200

$ kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
curl: (28) Connection timed out after 5000 milliseconds

If you need a whole family of domains, matchPattern: "*.github.com" matches any single subdomain, but not github.com itself. You can also add a toPorts section next to toFQDNs to allow only port 443.

When something gets blocked

Sooner or later something legitimate will get blocked, and because drops look like timeouts, it won’t announce itself. This is the order I’d check things in:

1. Ask Hubble first. hubble observe --pod <pod> --verdict DROPPED shows what was dropped and whether it was an L3/L4 or an HTTP decision. Adding --type policy-verdict --print-policy-names also tells you which policy made the call.
2. Check whether the pod is selected at all. In cilium-dbg endpoint list, the enforcement columns tell you whether ingress or egress is in default deny. If egress says Enabled and you didn't expect it, some policy with an egress section is selecting that pod.
3. Check both ends. A connection needs to be allowed by the sender’s egress rules and the receiver’s ingress rules, if each side is in default deny.
4. Check DNS. If a pod with egress rules can’t reach anything at all, it’s almost always a missing kube-dns rule.

One last tool worth knowing about: deny rules. Fields like ingressDeny and egressDeny block matching traffic even if another rule allows it, so they're useful for hard limits that should never be opened up by accident. Most of the time, though, the default-deny-plus-allow pattern above is all you need.

Rules of thumb I’d start with

If you’re writing your first Cilium policies, these five habits will save you most of the pain:

• Rehearse in audit mode, read the AUDITED verdicts, and only then enforce.
• Write rules against labels, never pod IPs. That’s the whole point of identities.
• Every policy with an egress section needs a DNS rule next to it.
• Use HTTP rules for any API with endpoints that only some callers should reach.
• When something hangs, assume a silent drop and ask Hubble before anything else.

The Death Star is a joke, but the pattern is exactly what you’d do with a real payments API or an internal admin endpoint: deny by default, allow by identity, and narrow down to the exact requests each caller needs.

Further reading

• Getting Started with the Star Wars Demo
• Policy Enforcement Modes
• Creating Policies from Verdicts (audit mode)
• Locking Down External Access with DNS-Based Policies
• Layer 3 Policies