agentgateway in the world of Agentic developement

17 min readAug 17, 2026

MCP is agent-to-tool: an agent calls a fixed function that does one discrete thing and returns a result — no autonomy, no judgment. We saw implementation of Agent gateway for MCPs in our previous article. We also have a registry setup for it.

Agent-to-agent communication is different. It’s for when the thing on the other end is itself capable of reasoning — potentially with its own LLM, its own tools, its own decision-making — and one agent needs to delegate a sub-task to another and get back a reasoned result, not just a lookup.

A concrete example that motivated everything that follows: a bank account-opening process genuinely involves several independent concerns — identity verification, customer history, background/criminal checks, financial capability, public records — each of which could reasonably be owned by a different team, built independently, and (eventually) reasoned about independently. That’s the shape of problem agent-to-agent architecture is built for.

The Full Account-Opening Team: Five Specialist Agents

We scaled the identity agent’s pattern to four more specialists, each a self-contained FastAPI service with its own port:

Single Identity Verification Agent, plain FastAPI, one endpoint.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Identity Verification Agent")


class VerifyRequest(BaseModel):
    customer_id: str
    document_type: str
    document_number: str


class VerifyResponse(BaseModel):
    customer_id: str
    status: str
    confidence: float
    notes: str


@app.get("/health")
async def health():
    return {"agent": "identity-verification", "status": "ok"}


@app.post("/invoke", response_model=VerifyResponse)
async def invoke(req: VerifyRequest):
    return VerifyResponse(
        customer_id=req.customer_id,
        status="VERIFIED",
        confidence=0.97,
        notes=f"{req.document_type} {req.document_number} matched on file.",
    )


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=9001)

This is the whole pattern every agent in the system follows: a Pydantic request model, a Pydantic response model, one /invoke endpoint, one /health endpoint. Verified with:

curl -s http://localhost:9001/health
curl -s -X POST http://localhost:9001/invoke \
  -H "Content-Type: application/json" \
  -d '{"customer_id":"CUST-5567","document_type":"passport","document_number":"X1234567"}'

We can build rest of the 8 services exactly the same way. Code for each of these services can be found here.

The First Orchestrator: Parallel Calls with `asyncio.gather`

The first version of the Orchestrator called all eight agents concurrently and synthesized a decision from the combined results:

async def call_agent(client, name, payload):
    try:
        resp = await client.post(f"{AGENTS[name]}/invoke", json=payload, timeout=10.0)
        resp.raise_for_status()
        return resp.json()
    except httpx.HTTPError as e:
        return {"status": "ERROR", "error": str(e)}

@app.post("/open-account")
async def open_account(app_data: AccountApplication):
    async with httpx.AsyncClient() as client:
        identity, history, background, financial, public_records = await asyncio.gather(
            call_agent(client, "identity", {...}),
            call_agent(client, "customer_history", {...}),
            call_agent(client, "background_check", {...}),
            call_agent(client, "financial_capability", {...}),
            call_agent(client, "public_records", {...}),
            ............
            ............
        )
    # decision synthesized from all five results

Limitation that motivated the next improvisation: All agents were always called, every time, even if an early check had already failed. For checks that cost real money (background checks and public-records lookups are often paid third-party services), that’s real waste. There was also no distinction between the internal reason for a decision and a customer-facing message — and error handling was a single generic try/except shared across all five calls rather than something auditable per step.

Let’s change this implementation to use a real LangGraph based orchestrator

Code for LangGraph implementation can be found here — orchestrator_error_handling.py

This is how the entire orchestration happens

async def call_agent(agent_name: str, payload: dict) -> dict:
    """Never raises -- every failure mode becomes status='ERROR' so the
    graph can route uniformly instead of crashing."""
    url = f"{AGENTS[agent_name]}/invoke"
    try:
        async with httpx.AsyncClient(timeout=TIMEOUT_SECONDS) as client:
            resp = await client.post(url, json=payload)
            resp.raise_for_status()
            return resp.json()
    except httpx.TimeoutException:
        return {"status": "ERROR", "error": f"{agent_name} agent timed out after {TIMEOUT_SECONDS}s"}
    except httpx.ConnectError:
        return {"status": "ERROR", "error": f"{agent_name} agent is unreachable"}
    except httpx.HTTPStatusError as e:
        return {"status": "ERROR", "error": f"{agent_name} agent returned {e.response.status_code}"}
    except Exception as e:
        return {"status": "ERROR", "error": f"{agent_name} agent call failed: {e}"}


def route_after_identity(state) -> str:
    result = state.get("identity_result") or {}
    if result.get("status") == "ERROR":
        state.setdefault("reasons", []).append(f"Identity check failed: {result.get('error')}")
        return "decline"
    if result.get("status") != "VERIFIED":
        state.setdefault("reasons", []).append("Identity could not be verified.")
        return "decline"
    return "continue"

# ... one routing function per check, same shape ...

builder = StateGraph(ApplicationState)
builder.add_node("identity_check", identity_check)
builder.add_node("customer_history_check", customer_history_check)
builder.add_node("background_check", background_check)
builder.add_node("financial_capability_check", financial_capability_check)
builder.add_node("public_records_check", public_records_check)
builder.add_node("decline", decline_node)
builder.add_node("approve", approve_node)

builder.add_edge(START, "identity_check")
builder.add_conditional_edges("identity_check", route_after_identity,
    {"continue": "customer_history_check", "decline": "decline"})
# ... same pattern chained through all five checks ...
builder.add_edge("decline", END)
builder.add_edge("approve", END)

graph = builder.compile()

This is a straight forward LangGraph solution with error handling interlaced at each step.

This still does not have the complexity we want to handle.

Multiple Products, One LLM Planner

The account-opening flow only covers one banking product. Real banks also issue credit cards, debit cards, and gift cards — each needing a genuinely different set of checks. Rather than hardcoding an if/else per product type, we introduced an LLM planner (GPT-5.1 via Azure OpenAI) that reads a plain-English enquiry and a catalog of available agents, and decides which agents apply and in what order — real reasoning, not branching logic.

Three new product-specific agents were added:

With these new agents in the mix, we now have 3 flows — Opening a bank account, Link a debit card and purchase a git card. To differentiate and handle the flow, we need an LLM in mix as well

The executor became a single node that loops through whatever the LLM decided, not a fixed chain of graph edges — verified directly (bypassing the real LLM call, since it needs live Azure credentials) by feeding it manually-constructed plans:

A gift-card plan with one step (gift_card_compliance) — confirmed it approved under the AML threshold and declined over it, calling nothing else.
A three-step credit-card plan (identity → financial_capability → credit_limit) — confirmed all three ran in the correct order with correct results.

We can use a simple local script to start all of our agents — start_agent.sh

# A simple starter for all agents
uv run agents/background_check_agent.py &
uv run agents/card_linking_agent.py &
uv run agents/identity_agent.py &
uv run agents/customer_history_agent.py &
uv run agents/public_records_agent.py &
uv run agents/financial_capability_agent.py &
uv run agents/gift_card_compliance_agent.py &
uv run orchestrator_langgraph_llm.py &

chmod a+x ./start_agent.sh
# Run it
./start_agents.sh 
(account-opening-agents) krishnansriram@Krishnans-MacBook-Pro account-opening-agents % INFO:     Started server process [22183]
INFO:     Started server process [22184]
INFO:     Started server process [22185]
INFO:     Started server process [22182]
INFO:     Started server process [22186]
INFO:     Started server process [22187]
INFO:     Started server process [22188]
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9007 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9001 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9005 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9008 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9003 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9002 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:9004 (Press CTRL+C to quit)
INFO:     Started server process [22190]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9000 (Press CTRL+C to quit)

Once the the agents are up and running, let’s give it a test

curl -s -X POST http://localhost:9000/process-enquiry \
  -H "Content-Type: application/json" \
  -d '{
    "enquiry_text": "I would like to open a new checking account",
    "customer_id": "CUST-9003",
    "full_name": "Jordan Alex Smith",
    "date_of_birth": "1990-04-12",
    "document_type": "passport",
    "document_number": "X1234567",
    "declared_annual_income": 95000,
    "address": "123 Main St, Columbus, OH"
  }' | python3 -m json.tool
Handling connection for 9000
{
    "customer_id": "CUST-9003",
    "product_type": "account_opening",
    "planned_steps": [
        "identity",
        "financial_capability",
        "background_check",
        "public_records"
    ],
    "planner_reasoning": "This is a new deposit account opening request. Identity should be verified first for the new customer, followed by financial capability checks needed for account opening. Background screening and public records checks are also required for opening a new deposit account.",
    "decision": "APPROVED",
    "customer_message": "Your account opening request has been approved.",
    "reasons": [
        "All required checks passed."
    ],
    "step_results": {
        "identity": {
            "customer_id": "CUST-9003",
            "status": "VERIFIED",
            "confidence": 0.97,
            "notes": "passport X1234567 matched on file."
        },
        "financial_capability": {
            "customer_id": "CUST-9003",
            "status": "PASS",
            "income_verified": true,
            "estimated_credit_score": 742,
            "affordability_status": "ADEQUATE"
        },
        "background_check": {
            "customer_id": "CUST-9003",
            "status": "PASS",
            "criminal_record_found": false,
            "sanctions_hit": false,
            "watchlist_hit": false,
            "notes": "No adverse findings for Jordan Alex Smith."
        },
        "public_records": {
            "customer_id": "CUST-9003",
            "status": "PASS",
            "address_verified": true,
            "bankruptcy_history": false,
            "litigation_history": false
        }
    }
}

Do another one

curl -s -X POST http://localhost:9000/process-enquiry \
  -H "Content-Type: application/json" \
  -d '{
    "enquiry_text": "I would like to link a debit card to my checking account",
    "customer_id": "CUST-9002",
    "full_name": "Priya Nair",
    "date_of_birth": "1994-03-08",
    "document_type": "passport",
    "document_number": "X5566778",
    "declared_annual_income": 65000,
    "address": "12 Maple Rd, Dublin, OH",
    "linked_account_number": "ACC-99887766"
  }' | python3 -m json.tool
Handling connection for 9000
{
    "customer_id": "CUST-9002",
    "product_type": "debit_card",
    "planned_steps": [
        "customer_history",
        "card_linking"
    ],
    "planner_reasoning": "This is a debit card request. First check customer history to confirm the customer relationship and whether identity/KYC is already established, then perform card linking to verify the checking account and link the debit card.",
    "decision": "APPROVED",
    "customer_message": "Your debit card request has been approved.",
    "reasons": [
        "All required checks passed."
    ],
    "step_results": {
        "customer_history": {
            "customer_id": "CUST-9002",
            "status": "PASS",
            "existing_customer": true,
            "relationship_years": 3.5,
            "prior_accounts": 1,
            "kyc_status": "CURRENT"
        },
        "card_linking": {
            "customer_id": "CUST-9002",
            "status": "PASS",
            "account_verified": true,
            "notes": "Account ACC-99887766 verified and eligible for debit card linking."
        }
    }
}

Let’s stop all of these agents with another friendly script — stop_agent.sh

# This all that we need to kill our agents - stop_agent.sh
pkill -f "uv run agents/"
pkill -f "orchestrator_langgraph_llm.py"

chmod a+x ./stop_agent.sh
# Execute stop agent
./stop_agents.sh 
INFO:     Shutting down
INFO:     Shutting down                                                                                                                  
INFO:     Shutting down
INFO:     Shutting down
INFO:     Shutting down
INFO:     Shutting down
INFO:     Shutting down
(account-opening-agents) krishnansriram@Krishnans-MacBook-Pro account-opening-agents % INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [22185]
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Waiting for application shutdown.
INFO:     Finished server process [22186]
INFO:     Application shutdown complete.
INFO:     Finished server process [22190]
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [22182]
INFO:     Waiting for application shutdown.
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Application shutdown complete.
INFO:     Finished server process [22184]
INFO:     Finished server process [22187]
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [22188]
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [22183]

We now have working agents, but we are far from done. Our next step is to ensure we are able to run it behind AgentGateway and add a much required Registry service as well.

Deploy in cluster

Create a namespace for all our agents. This will make our eventual discover process easierbanking-agents

kubectl get namespace banking-agents 2>/dev/null || kubectl create namespace banking-agents

One Generic Dockerfile for Every Agent

We have 2 different docker file. One for all these application agents and one for registry and orchestrator service. For now, let’s get all of the application agents up and running.

Get Krishnan Sriram’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Since every agent is a plain FastAPI app with the same shape, one Dockerfile serves all of them — Dockerfile.agent

FROM python:3.13-slim

COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

WORKDIR /app

COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev --no-install-project

COPY agents ./agents

ARG APP_MODULE
ENV APP_MODULE=${APP_MODULE}

CMD uv run python ${APP_MODULE}

Deploying Each Agent, One at a Time

Full walkthrough for agent #1 (Identity), then the same three-command pattern for the remaining seven:


docker build -t identity-agent:v1 \
  --build-arg APP_MODULE=agents/identity_agent.py \
  -f Dockerfile.agent .

kind load docker-image identity-agent:v1 --name agentgateway

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: identity-agent
  namespace: banking-agents
spec:
  replicas: 1
  selector:
    matchLabels: {app: identity-agent}
  template:
    metadata:
      labels: {app: identity-agent}
    spec:
      containers:
      - name: identity-agent
        image: identity-agent:v1
        imagePullPolicy: IfNotPresent
        ports: [{containerPort: 9001}]
---
apiVersion: v1
kind: Service
metadata:
  name: identity-agent
  namespace: banking-agents
spec:
  selector: {app: identity-agent}
  ports: [{port: 80, targetPort: 9001}]
  type: ClusterIP
EOF

kubectl rollout status deploy/identity-agent -n banking-agents --timeout=60s

Repeated for all eight agents (customer-history-agent :9002, background-check-agent :9003, financial-capability-agent :9004, public-records-agent :9005, credit-limit-agent :9006, card-linking-agent :9007, gift-card-compliance-agent :9008), substituting name/image/port each time.

Sanity check:

kubectl get pods -n banking-agents

NAME                                          READY   STATUS    RESTARTS   AGE
background-check-agent-566d595b77-h69zg       1/1     Running   0          19h
card-linking-agent-786f4899-96s6c             1/1     Running   0          19h
credit-limit-agent-7fb6db7b76-b8zzf           1/1     Running   0          19h
customer-history-agent-b765d95fd-2pzh5        1/1     Running   0          19h
financial-capability-agent-5c879c8879-98sgp   1/1     Running   0          19h
gift-card-compliance-agent-75c55c74d4-qzpx4   1/1     Running   0          19h
identity-agent-668c66fff8-stgxt               1/1     Running   0          19h
public-records-agent-66dfd78c8b-qrcvh         1/1     Running   0          19h

All eight should show 1/1 Running.

Deploying the Registry

Registry service is the one that references all other services operating in this spectrum for our application needs. Unless this service is up and running, rest of the service do not have a map to how agents can be connectes.

Note: Down the road, we’ll secure this service to be accessible only from application agents. This way, agents are secure and have a clear way to talk to each other.

docker build -t agent-registry:v1 --build-arg APP_MODULE=agent_registry_service.py -f Dockerfile .
kind load docker-image agent-registry:v1 --name agentgateway

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-registry
  namespace: banking-agents
spec:
  replicas: 1
  selector:
    matchLabels: {app: agent-registry}
  template:
    metadata:
      labels: {app: agent-registry}
    spec:
      containers:
      - name: agent-registry
        image: agent-registry:v1
        imagePullPolicy: IfNotPresent
        ports: [{containerPort: 9100}]
---
apiVersion: v1
kind: Service
metadata:
  name: agent-registry
  namespace: banking-agents
spec:
  selector: {app: agent-registry}
  ports: [{port: 80, targetPort: 9100}]
  type: ClusterIP
EOF

kubectl rollout status deploy/agent-registry -n banking-agents --timeout=60s

kubectl port-forward -n banking-agents svc/agent-registry 9100:80 &

Verified it can actually reach the other eight agents by DNS name: You should see a map of all application services and this acts as a registry map for them. It not only has location map, also carries a description of it…ideally paving way for skills

curl -s http://localhost:9100/agents | python3 -m json.tool
#Execution results
Handling connection for 9100
{
    "agents": [
        {
            "name": "identity",
            "url": "http://identity-agent.banking-agents.svc.cluster.local:80",
            "description": "Verifies a customer's identity documents (passport, license, etc). Needed any time a NEW customer's identity hasn't already been established.",
            "input_schema": {
                "customer_id": "customer_id",
                "document_type": "document_type",
                "document_number": "document_number"
            }
        },
        {
            "name": "customer_history",
            "url": "http://customer-history-agent.banking-agents.svc.cluster.local:80",
            "description": "Looks up existing relationship, prior accounts, and KYC status for a customer. Useful to check whether identity verification can be skipped for an existing customer.",
            "input_schema": {
                "customer_id": "customer_id"
            }
        },
        {
            "name": "background_check",
            "url": "http://background-check-agent.banking-agents.svc.cluster.local:80",
            "description": "Criminal record, sanctions, and watchlist screening. Required for opening a new deposit account; usually not required for issuing a card to an already-verified customer.",
            "input_schema": {
                "customer_id": "customer_id",
                "full_name": "full_name",
                "date_of_birth": "date_of_birth"
            }
        },
        {
            "name": "financial_capability",
            "url": "http://financial-capability-agent.banking-agents.svc.cluster.local:80",
            "description": "Verifies income and estimates a credit score. Required for account opening and for credit card applications; not required for debit or gift cards.",
            "input_schema": {
                "customer_id": "customer_id",
                "declared_annual_income": "declared_annual_income"
            }
        },
        {
            "name": "public_records",
            "url": "http://public-records-agent.banking-agents.svc.cluster.local:80",
            "description": "Checks address verification, litigation, and bankruptcy history. Required for opening a new deposit account.",
            "input_schema": {
                "customer_id": "customer_id",
                "address": "address"
            }
        },
        {
            "name": "credit_limit",
            "url": "http://credit-limit-agent.banking-agents.svc.cluster.local:80",
            "description": "Determines an approved credit limit based on income and the requested limit. Required ONLY for credit card applications.",
            "input_schema": {
                "customer_id": "customer_id",
                "declared_annual_income": "declared_annual_income",
                "requested_credit_limit": "requested_credit_limit"
            }
        },
        {
            "name": "card_linking",
            "url": "http://card-linking-agent.banking-agents.svc.cluster.local:80",
            "description": "Verifies a bank account to link a debit card to. Required ONLY for debit card applications.",
            "input_schema": {
                "customer_id": "customer_id",
                "linked_account_number": "linked_account_number"
            }
        },
        {
            "name": "gift_card_compliance",
            "url": "http://gift-card-compliance-agent.banking-agents.svc.cluster.local:80",
            "description": "Checks a gift card purchase amount against AML limits. Required ONLY for gift card purchases.",
            "input_schema": {
                "customer_id": "customer_id",
                "purchase_amount": "purchase_amount"
            }
        }
    ]
}

Deploy orchestrator

Azure OpenAI credentials, delivered as a Secret rather than baked into the image:

kubectl create secret generic azure-openai-secret \
  -n banking-agents \
  --from-literal=AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com" \
  --from-literal=AZURE_OPENAI_DEPLOYMENT="gpt-5-1-chat" \
  --from-literal=AZURE_OPENAI_API_KEY="your-key" \
  --from-literal=AZURE_OPENAI_API_VERSION="2025-04-14"

Build and deploy our image. This is the same pattern we followed for all other services as well.

docker build -t orchestrator:v1 --build-arg APP_MODULE=orchestrator_dynamic.py -f Dockerfile .

kind load docker-image orchestrator:v1 --name agentgateway

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orchestrator
  namespace: banking-agents
spec:
  replicas: 1
  selector:
    matchLabels: {app: orchestrator}
  template:
    metadata:
      labels: {app: orchestrator}
    spec:
      containers:
      - name: orchestrator
        image: orchestrator:v1
        imagePullPolicy: IfNotPresent
        ports: [{containerPort: 9000}]
        env:
        - name: AGENT_REGISTRY_URL
          value: "http://agent-registry.banking-agents.svc.cluster.local:80"
        envFrom:
        - secretRef:
            name: azure-openai-secret
---
apiVersion: v1
kind: Service
metadata:
  name: orchestrator
  namespace: banking-agents
spec:
  selector: {app: orchestrator}
  ports: [{port: 80, targetPort: 9000}]
  type: ClusterIP
EOF

kubectl rollout status deploy/orchestrator -n banking-agents --timeout=60s

kubectl port-forward -n banking-agents svc/orchestrator 9000:80 &

Time to test

curl -s -X POST http://localhost:9000/process-enquiry \
  -H "Content-Type: application/json" \
  -d '{
    "enquiry_text": "I would like to purchase a $500 gift card",
    "customer_id": "CUST-7001", "full_name": "Jordan Smith", "date_of_birth": "1990-04-12",
    "document_type": "passport", "document_number": "X1234567",
    "declared_annual_income": 95000, "address": "123 Main St, Columbus, OH",
    "purchase_amount": 500
  }' | python3 -m json.tool
Handling connection for 9000
{
    "customer_id": "CUST-7001",
    "product_type": "gift_card",
    "planned_steps": [
        "gift_card_compliance"
    ],
    "planner_reasoning": "This is a gift card purchase, so the only relevant specialist is gift_card_compliance to check the amount against AML limits. No identity, credit, account, or background screening steps are needed for this request.",
    "decision": "APPROVED",
    "customer_message": "Your gift card request has been approved.",
    "reasons": [
        "All required checks passed."
    ],
    "step_results": {
        "gift_card_compliance": {
            "customer_id": "CUST-7001",
            "status": "PASS",
            "within_aml_limit": true,
            "notes": "Purchase amount 500.0 vs AML threshold 2000.0."
        }
    }
}

There you go. We now have a working multi agent with orchestrator running in your cluster. We have come a long way

Here’s what we got here

Let’s frontend orchestrator with gateway. If you are wondering why?

One stable, real entry point — https://your-domain/orchestrator instead of a Kubernetes-admin-only tunnel. Any real client (a mobile app backend, another internal service, a partner integration) can reach it the normal way.
Auth enforced before your code ever runs — the JWT/RBAC policy from the security work means an unauthenticated or unauthorized caller gets rejected at the gateway, never even reaching the orchestrator’s application logic.
Rate limiting and abuse protection — without it, nothing stops a misbehaving client (or a bug in an upstream caller) from hammering the orchestrator, which in turn hammers GPT-5.4-mini and your Foundry spend.
One place to see all traffic — centralized logging/observability for every call, instead of needing to instrument the orchestrator itself.
A stable surface if the orchestrator itself changes — swap orchestrator versions, add a second replica, or move it to a different Service, and callers never notice, because they only ever talked to the gateway’s path, not the orchestrator directly.

The banking MCP servers taught this same lesson at the tool level; this is the identical argument one layer up, at the orchestrator/agent level — the thing that’s the actual “front door” of your whole multi-agent system deserves the same governance the individual agents got.

For plain HTTP backend like the orchestrator (no MCP translation, no AI provider config needed), you don’t need an AgentgatewayBackend at all. HTTPRoute can point directly at a Kubernetes Service

The one wrinkle: the orchestrator’s Service lives in banking-agents, but the HTTPRoute lives in agentgateway-system — a different namespace. Gateway API requires an explicit ReferenceGrant in the target namespace before it'll allow that cross-namespace reference.

Step 1 — Allow the cross-namespace reference

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-agentgateway-to-orchestrator
  namespace: banking-agents
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: agentgateway-system
  to:
  - group: ""
    kind: Service
    name: orchestrator
EOF

Step 2 — Route directly to the Service, no AgentgatewayBackend

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: banking-orchestrator-route
  namespace: agentgateway-system
spec:
  parentRefs:
  - name: agentgateway-proxy
  rules:
  - matches:
    - path: {type: PathPrefix, value: /orchestrator}
    backendRefs:
    - name: orchestrator
      namespace: banking-agents
      port: 80
EOF

kubectl get httproute banking-orchestrator-route -n agentgateway-system

Verify and test

curl -s -X POST http://localhost:8080/orchestrator/process-enquiry \
  -H "Content-Type: application/json" \
  -d '{
    "enquiry_text": "I would like to purchase a $500 gift card",
    "customer_id": "CUST-7001", "full_name": "Jordan Smith", "date_of_birth": "1990-04-12",
    "document_type": "passport", "document_number": "X1234567",
    "declared_annual_income": 95000, "address": "123 Main St, Columbus, OH",
    "purchase_amount": 500
  }' | python3 -m json.tool
# Execution Results
Handling connection for 8080
{
    "customer_id": "CUST-7001",
    "product_type": "gift_card",
    "planned_steps": [
        "gift_card_compliance"
    ],
    "planner_reasoning": "This enquiry is for a gift card purchase. The only required specialist agent is gift_card_compliance to check the $500 amount against AML limits; no identity, background, or account-linking checks are needed for a straightforward gift card purchase.",
    "decision": "APPROVED",
    "customer_message": "Your gift card request has been approved.",
    "reasons": [
        "All required checks passed."
    ],
    "step_results": {
        "gift_card_compliance": {
            "customer_id": "CUST-7001",
            "status": "PASS",
            "within_aml_limit": true,
            "notes": "Purchase amount 500.0 vs AML threshold 2000.0."
        }
    }
}

# 2nd Execution
curl -s http://localhost:8080/orchestrator/health
Handling connection for 8080
# Execution results
{"agent":"dynamic-banking-orchestrator","status":"ok","registry_url":"http://agent-registry.banking-agents.svc.cluster.local:80"}

What agentgateway offers out of the box for “agent filters”

Built-in filters are called guardrails / prompt guards in agentgateway’s docs, and they’re specifically about LLM traffic content — not generic REST payloads. That means they apply to /llm route (the Foundry connection), not directly to /orchestrator's plain JSON.

The obviously relevant one for a bank: PII masking on the LLM route

Orchestrator sends customer PII (names, DOBs, document numbers, income, addresses) to GPT-5.4-mini as part of the planning prompt.

Note: Agentic implementation is not using /llm for all LLM access. Agegateway provides OOB gaurdrails for sensitive data on /llm. In our case we have more of a REST api kind of implementation, so OOB gaurdrails won’t kick in.

Right way to fix this solution is to use /llm route for LLM needs. We will make this change in our next article.

Rate limiting

This one’s a genuine general traffic policy, applies directly to your orchestrator route:

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: banking-orchestrator-ratelimit
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: banking-orchestrator-route
  traffic:
    rateLimit:
      local:
      - requests: 60
        unit: Minutes
        burst: 10
EOF

burst: 10 is optional but worth including — it allows short spikes above the steady 60/minute rate (useful for a real client that might legitimately fire a few requests close together) without changing the sustained limit.

Verify apply

kubectl get agentgatewaypolicy banking-orchestrator-ratelimit -n agentgateway-system
NAME                             ACCEPTED   ATTACHED   AGE
banking-orchestrator-ratelimit   True       True       62s

Let’s test it

for i in $(seq 1 75); do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/orchestrator/health
done | sort | uniq -c
# Execution results
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
............................
............................
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
  70 200
   5 429

You should see the first ~70 (60 steady + burst allowance) return 200, with 429s appearing after that within the same minute window. If everything comes back 200, either the policy didn't attach correctly (check kubectl describe agentgatewaypolicy banking-orchestrator-ratelimit -n agentgateway-system for a targetRef resolution error) or 75 requests just didn't clear the burst allowance fast enough — worth bumping the loop to 100 if that's the case.

Here’s what’s been implmented

This brings us to the end of all agentic implementation. Next we will see how to handle security and access gaurdrails between agents to make them secure.

Until then stay safe and keep reading!!

Krishnan Sriram

agentgateway in the world of Agentic developement

The First Orchestrator: Parallel Calls with asyncio.gather

Multiple Products, One LLM Planner

Deploy in cluster

One Generic Dockerfile for Every Agent

Get Krishnan Sriram’s stories in your inbox

Deploying Each Agent, One at a Time

Deploying the Registry

Deploy orchestrator

Step 1 — Allow the cross-namespace reference

Step 2 — Route directly to the Service, no AgentgatewayBackend

What agentgateway offers out of the box for “agent filters”

The obviously relevant one for a bank: PII masking on the LLM route

Rate limiting

Written by Krishnan Sriram

No responses yet

More from Krishnan Sriram

MCP vs CLI vs Skills — Let’s get a better understanding

MCP is a universal adapter. Instead of writing a custom connector for every AI-to-tool combination, MCP gives every service a standardised…

AgentGateway — For MCP implementation

In our previous article we setup AgentGateway to help and govern our LLM needs. We continue from there to use AgentGateway for MCP…

AgentGateway — AI gateway with a difference

What is AgentGateway?

Building Document Extraction Pipeline with PII Redaction and PCI Blocking on Azure

Documents that arrive at your business includes — invoices, forms, contracts, statements — are full of exactly the data you don’t want…

Recommended from Medium

AI Agents (07): Agentic Harness (Frameworks)

From RAG, Orchestration, Observability and Tools to Skills — All You Need to Know About AI Agentic Harnesses & Frameworks

8 New AI Terms You Should Know to Be Ahead of 90% of People

A few more AI terms to know beyond tokens, prompts, and hallucinations.

10 AI Tools Everyone Will Be Using in Six Months, But Almost Nobody’s Found Yet

You’re going to see these in someone’s toolkit next year and wonder why you didn’t hear about them first. Here’s your head start.

Building a Governance Knowledge Graph for AI Agents

Signed human mandates, a fail-closed policy gate, and an RDF graph that explains every agent action but can never authorize one. Tested…

Beyond Chatbots: How to Architect an Enterprise Agentic AI Platform

Why the future of enterprise AI depends more on execution architecture, governance, and reliability than on increasingly sophisticated…

The Anatomy of an AI Agent Harness

The 7 harness concepts that give an AI agent memory, tools, context, safety, and a way to know when the job is actually done.