[Kubernetes Lab] Understanding Gateway API Inference Extension

If you have an inference service running on Kubernetes, how could you determine which pods to send traffic to? This is where Inference Extension comes into play. This article would show you how to setup end-to-end North-South Networking for Inference on Kubernetes.

This is a practical walkthrough guide, and expected you understand basic concepts of Gateway API.
We’ll be using vllm-d-inference-sim image, which is a simulator for inference and can be run on CPU-only Kubernetes cluster.

Video walkthrough for this article: https://youtu.be/ozSN-_CWdUg

What is Gateway API Inference extension?

Gateway API Inference extension is an extension from Gateway API for Inferences workload (ohh wow…)

Basically it contains 1 new CRD, InferencePool (there’s also InferenceObjective and InferencePoolImport CRDs we’re not going there in this lab) and EndpointPicker (EPP), which is a set of deploy-service k8s objects, using EPP docker image with gRPC.

The Problem

Unlike normal deployment, inference has to be scheduled on less utilized pod based on specific metrics. For the best inference performance, we need to route traffic to least utilized LLM pod based on some metrics.

This borned the extension.

• EPP basically continuously scrape LLM pod metrics and tell you which pod is less utilized
• Inference Pool (infpool) connects 2 deployments together : LLM and EPP
• Then infpool is referenced by HTTPRoute.spec.rules.backendRef

In other words…

InferencePool is just another backendRefs for Gateway, i.e.

Gateway -> HTTPRoute -> InferencePool (instead of Service) -> EndpointPicker

Gateway API Inference Extension

The Setup

Stage 1 — The LLM

1. Deploy this https://github.com/llm-d/llm-d-inference-sim/blob/main/manifests/deployment.yaml, basically a deployment and service for simulated LLM
2. After deployed, run curl <svc IP>:<svc port>/v1/models , or simulated response with

curl http://<svc IP>:<svc port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-397B-A17B",
    "messages": [
      {"role": "user", "content": "Explain Kubernetes in one sentence."}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

The response is simulated

Tada! You got a simplest LLM inference running with 1 deploy and 1 svc.

Stage 2 — The Gateway API

So now, instead of using the service cluster IP, I want to access it through a Gateway, with a domain name and a path, such like

curl kyle-is-not-handsome.com/inference/v1/chat/completions

To achieve this, the concept is like:

We need a gateway that use domain: kyle-is-not-handsome.com , then a httproute adding /inference on that domain, mapping to the service.

The step-by-step is like:

1. Install Gateway API CRD
2. Install Gateway Class (eg. istio, nginx, etc…)
3. Setup Gateway refer to the gc in (1), mention the domain
4. Setup httproute, add /inference path, map to the service
5. (locally) add <gateway IP> <domain> into /etc/hosts, so the domain would be mapped into the gateway IP.

Let’s do it!

1. Install Gateway API CRD

Before and After: run k api-resources | grep gateway , you’ll see several gateway components like GatewayClass, Gateway, HTTPRoute after installation

kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.6.0" | kubectl apply -f -; }

2. Install GatewayClass

Get Kyle Law’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

I use istio Gateway Class here:

• 2.1) Install istioctl: https://istio.io/latest/docs/setup/install/istioctl/
• 2.2) istioctl install --set profile=minimal -y
• 2.3) Check GatewayClass

ubuntu@k8s-t7-cp1:~$ k get gc
NAME           CONTROLLER                    ACCEPTED   AGE
istio          istio.io/gateway-controller   True       68m
istio-remote   istio.io/unmanaged-gateway    True       68m

3. Setup Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inf-gtw
  namespace: inf
  annotations:
    networking.istio.io/service-type: NodePort
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    hostname: "kyle-is-not-handsome.com"

Note: For local testing and istio gc, you need to add annotations to NodePort. By default it’s using Loadbalancer.

After apply, run k get svc to observe the corresponding NodePort svc created from the gtw

4. Setup HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inf-route
  namespace: inf
spec:
  parentRefs:
  - name: inf-gtw
  hostnames:
  - "kyle-is-not-handsome.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /inference
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: vllm-sim-demo-svc
      port: 8000

5. add DNS mapping into /etc/hosts

<gtw svc IP> kyle-is-not-handsome.com

6. Check you result

ubuntu@k8s-t7-cp1:~/inf$ curl kyle-is-not-handsome.com/inference/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen3.5-397B-A17B",
    "messages": [
      {"role": "user", "content": "Explain Kubernetes in one sentence."}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }' | jq
{
  "id": "chatcmpl-81f105fe-8a03-511c-a500-96351edb247a",
  "model": "Qwen/Qwen3.5-397B-A17B",
  "object": "chat.completion",
  "created": 1790150848,
  "usage": {
    "prompt_tokens": 17,
    "completion_tokens": 39,
    "total_tokens": 56,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "To be or not to be that is the question. Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime Alas"
      }
    }
  ]
}
ubuntu@k8s-t7-cp1:~/inf$ curl kyle-is-not-handsome.com/inference/v1/models  | jq
{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3.5-397B-A17B",
      "object": "model",
      "created": 1790150850,
      "owned_by": "vllm",
      "root": "Qwen/Qwen3.5-397B-A17B",
      "parent": null,
      "max_model_len": 1024
    }
  ]
}

yeah, kyle-is-not-handsome.com/inference is now mapping to the LLM service

So far, we’re just utilizing the NORMAL Gateway API usage, next we’re adding the Inference Extension

Stage 3: The Idol of this article — — — — Inference Extension!

1. Install CRDs

kubectl apply -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd

1.1 Enable Istio extension (based on your gateway class)

You need to enable Inference Extension on your GatewayClass. For instance, I’m using Istio here

istioctl install --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true -y

For other gateway class, please google ‘<your gateway class name> enable inference extension’

Note: If you’re running on restricted Kubernetes cluster like on Killercoda, kindly restrict your resources allocation else your Istio deployment will be stuck:

istioctl install --set profile=minimal \
  --set values.pilot.env.SUPPORT_GATEWAY_API_INFERENCE_EXTENSION=true \
  --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true \
  --set values.pilot.resources.requests.memory=512Mi \
  --set values.pilot.resources.requests.cpu=100m \
  -y

2. Deploy the EPP and InferencePool with Helm (easiest)

git clone https://github.com/llm-d/llm-d-router.git && cd llm-d-router

helm dependency build ./config/charts/llm-d-router-gateway

helm -n inf install vllm-sim-demo ./config/charts/llm-d-router-gateway \
  --set router.modelServers.matchLabels.app=vllm-sim-demo \
  --set provider.name=istio \
  --set router.epp.resources.requests.cpu=500m \
  --set router.epp.resources.requests.memory=512Mi \
  --set router.epp.resources.limits.memory=1Gi

Check: k -n inf get infpool

3. Modify HTTPRoute to reference the infpool to the backendRef

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inf-route
  namespace: inf
spec:
  hostnames:
  - kyle-is-not-handsome.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inf-gtw
  rules:
  - backendRefs:
####### THIS PARTTTT
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: vllm-sim-demo
      port: 8000
      weight: 1
########
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: /
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /inference

4. Test again the gateway

curl -s kyle-is-not-handsome.com/inference/v1/models   | jq
{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3.5-397B-A17B",
      "object": "model",
      "created": 1790344406,
      "owned_by": "vllm",
      "root": "Qwen/Qwen3.5-397B-A17B",
      "parent": null,
      "max_model_len": 1024
    }
  ]
}

It worked! You’ve setup an North-South Inference system with Gateway API and Inference Pool on Kubernetes ⭐️️️️