Service Mesh Explained: Building a Proxy Injector in Rust (with code) 💻

Kubernetes service meshes rely on “sidecar” proxies to handle traffic routing transparently, security
policies, and observability for your microservices—but manually bolting those proxies onto every Pod
spec quickly becomes a maintenance nightmare.

What if you could have Kubernetes do the work for you, automatically injecting the proxy whenever a Pod is created?

In this tutorial, we’re going to build exactly that: a Mutating Admission Webhook in Rust that hooks
into the Kubernetes API server, inspects incoming Pod specs, and—if they meet your criteria—patches
them on the fly to include an init‑container (for iptables setup) and your proxy‑sidecar.

Along the way, you’ll learn how to:

• Define the AdmissionReview/AdmissionRequest and AdmissionResponse data structures
• Wire up an async handler in Axum, complete with #[instrument] tracing for per-request logging
• Craft a JSONPatch that adds init‑containers and sidecar containers via a base64-encoded payload
• Stand up a TLS‑secured HTTP server using Rustls so Kubernetes can trust your webhook

By the end, you’ll have a drop‑in proxy injector that can be deployed alongside your service mesh
control plane—no more manual injection, no more drift, just automatic, consistent proxy injection
across your cluster.

All the code we walk through here is available on our GitHub repository—feel free to clone and explore it!

Let’s dive in!🚀

Admission Webhooks

What Are Admission Webhooks?

Admission webhooks are a type of dynamic admission controller in Kubernetes. They allow you to validate or modify (mutate) Kubernetes objects as they are submitted to the cluster.

There are two types of admission webhooks:

• Validating Admission Webhooks – used to validate requests to the Kubernetes API server. They can accept or reject the request, but cannot modify the object.
• Mutating Admission Webhooks – used to modify (mutate) objects before they are persisted. They can change or enrich the resource definition, such as injecting sidecars into pods.

Admission webhooks are HTTP callbacks that are invoked during the admission phase of an API request. The Kubernetes API server sends an AdmissionReview request to the webhook service, which then evaluates the request and responds with an AdmissionReview response.

The admission phase takes place after authentication and authorization, but before the object is stored in etcd.

You can configure the Kubernetes API server to call specific webhook services when certain operations (like CREATE, UPDATE, or DELETE) are performed on specific resources (such as Pods, Deployments, etc.).

How It Works

When a request is made to the Kubernetes API:

1. The request is authenticated and authorized.
2. The object goes through the admission phase, where it is passed to:
   • Mutating webhooks (in sequence),
   • Followed by validating webhooks (in parallel).
3. Based on the webhook responses, the request is either allowed, denied, or modified.
4. If allowed, the object is persisted in etcd.

Controllers vs. Webhooks

It’s important to distinguish between admission controllers and webhooks:

• Admission controllers are built into the Kubernetes API server binary. They are enabled and configured by cluster administrators and cannot be extended at runtime.
• Webhooks, on the other hand, are external HTTP services configured through the Kubernetes API. They provide a more flexible and extensible way to implement custom admission logic, and can be written in any language or framework.

Admission controllers can validate, mutate, or perform both operations depending on their configuration. While validating controllers can only inspect and accept/reject objects, mutating controllers can modify them before they are stored.

Building a Proxy Injector: The Structures

To begin, we need to define the data structures that will be used within our injector code. We use the pub keyword to make these structures accessible from other files within the module.
The first structure we need is AdmissionRequest, which represents a request sent to the admission webhook:

#[derive(Debug, Serialize, Deserialize)]
pub struct AdmissionRequest {
    uid: String,
    object: serde_json::Value,
}

• uid: A unique identifier for this admission request, provided by the Kubernetes API server. It's used to correlate requests and responses.
• object: This field contains the Kubernetes object (usually a Pod) being submitted. It's stored as a raw JSON value so we can inspect or mutate it flexibly.

Next, we define the AdmissionReview structure. This wraps the admission request and is used to process it:

#[derive(Debug, Deserialize, Serialize)]
pub struct AdmissionReview {
    #[serde(rename = "apiVersion", default = "default_api_version")]
    pub api_version: String,
    #[serde(default = "default_kind")]
    pub kind: String,
    pub request: AdmissionRequest,
    #[serde(skip_deserializing)]
    pub response: Option<AdmissionResponse>,
}

• api_version: The version of the AdmissionReview API we're handling.
• kind: Always "AdmissionReview" for admission webhooks.
• request: Contains the actual AdmissionRequest sent by the API server.
• response: Optional at deserialization time (we don't receive it from the client), but we populate it before responding to the API server. The default values for apiVersion and kind are provided by the following functions:

fn default_api_version() -> String {
    "admission.k8s.io/v1".to_string()
}

fn default_kind() -> String {
    "AdmissionReview".to_string()
}

After that, we define the AdmissionResponse structure, which is used to send a response back from the admission webhook:

#[derive(Debug, Serialize)]
pub struct AdmissionResponse {
    uid: String,
    allowed: bool,
    patch: Option<String>,
    #[serde(rename = "patchType")]
    patch_type: Option<String>,
}

• uid: Must match the request's UID so Kubernetes knows which request this response is for.
• allowed: Indicates whether the request is approved or denied.
• patch: If set, this is a base64-encoded JSON patch to modify the original object before it's persisted in etcd.
• patch_type: Typically "JSONPatch" if you're modifying the object. Required when patch is provided.

Building a Proxy injector: The injection logic

After defining the main structures, we need to create the proper injection logic.

We want a modular logic that can adapt to future changes and users' needs, while maintaining a simple program structure.

First of all, we need to create a simple function called check_and_validate_pod.

This function ensures that the pod meets our requirements before injecting the sidecar proxy into a pod.

The function follows this logic:

• Checks if containers are present

  • Iterates over each container in the pod's spec.containers.
  • If a container's name contains "cortexflow-proxy":
  • Logs an error.
  • Returns an error: "The pod is not eligible for proxy injection. Sidecar proxy already present."`

• Validates namespace annotations

  • Retrieves the pod's namespace from metadata.namespace.
  • Checks metadata.annotations for the key "proxy-injection".
  • If it's set to "disabled":
    • Logs a warning.
    • Returns an error: "Automatic namespace injection is disabled."

• Validates pod-level annotations

  • Checks if the pod itself has "proxy-injection": "disabled" in metadata.annotations.
  • If so:
    • Logs a warning.
    • Returns an error: "Automatic pod injection is disabled."

• If all checks pass

  • Returns Ok(true) indicating the pod is eligible for injection.

For the sake of brevity, I am not including the code below, but you can find the check_and_validate_pod code here.

Going back to our inject function, after calling the validation function, we expect two behaviours:

1. The pod is ready and eligible for injection
2. The pod is not eligible for injection

In the first case, we can apply the patch, which we'll define in the next chapter, and return an allowed: true Admission Response.
In the second case, we are not injecting the patch, and we return an allowed: false Admission Response.

⚠️For an unexpected issue, I can't include the code directly. You can find the code here

Building a Proxy injector: The patch

Now the magic happens ⭐. The patch is one of the most crucial parts in the proxy injector and is where all the variables are defined.

We are using serde_json to create a JSON Patch and lazy_static to optimize the resources by initializing the variable when it is first accessed, in contrast to the regular static data, which is initialized at compile time.

The patch is divided into two parts:

1. Initialize Iptables
2. Initialize the proxy

In the first part, we are using iptables to redirect all the external traffic — in particular, TCP and UDP traffic — to specific ports.

We decided to bind the TCP traffic to port 5054 and the UDP traffic to port 5053.

Note:The init-iptables operation cannot be skipped. Otherwise, our system will not bind the traffic to the ports we chose, resulting in endless hours of debugging.

In the second part, we're doing another add operation to include the image of the proxy server.

We're also explicitly setting the TCP (5054) and UDP (5053) ports using the containerPort key.

⚠️For an unexpected issue, I can't include the patch code directly. You can find the code here

Building a Proxy injector: The server logic

In the last part we need to create a server to serve the API we made in the previous step. For this step we use the axum crate and we proceed creating a route. We decided to call the endpoint /mutate as a reminder for our Mutating Admission Webhook. As second step we proceed to associate the inject function as POST request and we bind the 9443, this ends the route configuration. The last step is to load the TLS certificate files tls.crt and tls.key.

Note:
Kubernetes requires TLS certificates to serve APIs over HTTPS. Failing to provide the certificates will result in a non-functional webhook service

How to generate a TLS certificate?

Working with TLS certificates may be something unfamiliar to the majority of people reading this article. Cert-manager is the easiest way to generate the tls.key and tls.crt keys. All you have to do is installing cert-manager using the kubernetes CLI

kubectl apply -f https://github.com/cert-manager/certmanager/releases/latest/download/cert-manager.yaml

The installation may take a while so you can take a small break to let your mind rest a little bit!

After cert-manager is installed you can get the secrets using the following commands

• Return the data.ca file


kubectl get secret proxy-injector-tls -n cortexflow -o jsonpath='{.data.ca\.crt}'


• Return the tls.key file


kubectl get secret proxy-injector-tls -n cortexflow -o jsonpath='{.data.tls\.key}'


• Return the tls.crt file


kubectl get secret proxy-injector-tls -n cortexflow -o jsonpath='{.data.tls\.crt}'


Note:
For security reasons, do not share these secrets with anyone. Leaking them may compromise your system’s security and get you in trouble.

We decided to automate this process in the install.sh script that you can find in the repository

⚠️For an unexpected issue, I can't include the code directly. You can find the code here

Building a Proxy injector: Deploying to Kubernetes

Now that all components are in place, the final step is to create a Kubernetes manifest to deploy the application into our cluster.
Below is an example YAML file we used to deploy the proxy injector within our namespace.

Pay special attention to the spec section: here, we define a custom selector that grants the necessary permissions for the injector to modify incoming Pod definitions—specifically, to add the sidecar proxy container automatically.

⚠️For an unexpected issue, I can't include the manifest code directly. You can find the code here

Conclusion

In the first part, we've covered the foundamentals of proxy injection,going through admission webhooks and admission controllers, while in the second part we have built all the logic from scratch using the Rust programming covering a lot of practical aspects such as defining the structures,building the patch, launching the axum server and interacting with the Kubernetes API.

In the next part of this series, we’ll create a sidecar proxy and all the basic functions such as service discovery, metrics, observability and messaging 🚀

Enjoying the content? Show us some love with a ⭐ on GitHub! And be sure to catch the first episode of the series, where we take a deep dive into the world of service meshes.
Stay tuned—and stay curious. 🌐🧩