<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Claudio Acquaviva on Medium]]></title>
        <description><![CDATA[Stories by Claudio Acquaviva on Medium]]></description>
        <link>https://medium.com/@claudioacquaviva?source=rss-878574aaf46f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MS35QLpDNdMwc-V1MKEp6w.jpeg</url>
            <title>Stories by Claudio Acquaviva on Medium</title>
            <link>https://medium.com/@claudioacquaviva?source=rss-878574aaf46f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:14:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@claudioacquaviva/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Agent with Strands SDK, Kong AI/MCP Gateway and Amazon Bedrock]]></title>
            <link>https://medium.com/@claudioacquaviva/ai-agent-with-strands-sdk-kong-ai-mcp-gateway-and-amazon-bedrock-a947b9e04ad8?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a947b9e04ad8</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 13:37:00 GMT</pubDate>
            <atom:updated>2026-03-02T11:59:44.613Z</atom:updated>
            <content:encoded><![CDATA[<p>The “<a href="https://konghq.com/blog/engineering/ai-gateway-mcp-gateway-mcp-server-breakdown">Kong AI/MCP Gateway and Kong MCP Server technical breakdown</a>” blog post described the new capabilities added to Kong AI Gateway to support MCP (Model Context Protocol). The blog post focused exclusively on consuming MCP Server and MCP Tools through Kong MCP Gateway. Now, it’s time to check how an AI Agent can leverage the AI and MCP infrastructure exposed and protected by Kong AI/MCP Gateway.</p><p><a href="https://strandsagents.com/">Strands Agents</a>, the Python-based framework for building agents, was <a href="https://aws.amazon.com/blogs/opensource/introducing-strands-agents-an-open-source-ai-agents-sdk/">introduced</a> by AWS in May, 2025. Strands makes the integration with Tools, GenAI Models and Services straightforward by providing a consistent way for agents to interact with external systems. It simplifies how developers orchestrate tools, gather context, and orchestrate reasoning, turning complex multi-service workflows into maintainable, event-driven agent logic.</p><p>At the infrastructure layer, the Kong AI/MCP Gateway extends these capabilities by securely exposing GenAI model endpoints, AI tools and enterprise APIs through standardized GenAI and MCP interfaces. When combined with Amazon Bedrock, which offers scalable access to leading foundation models, the result is an architecture where agents can reliably call Bedrock models and MCP based tools through a unified Gateway.</p><p>The “Kong AI/MCP Gateway and Kong MCP Server technical breakdown” blog post discusses the fundamental MCP abstractions and how Kong AI/MCP Gateway addresses them. The blog post does not show how to leverage the Gateway to implement an AI Agent though. This next blog post explores how Strands SDK, Kong AI/MCP Gateway and Amazon Bedrock can work together to build robust, production-ready AI Agents.</p><h3>Kong MCP Gateway Introduction</h3><p>In October, 2025, Kong announced <a href="https://konghq.com/blog/product-releases/enterprise-mcp-gateway">Kong Gateway 3.12</a> with several new capabilities. In a nutshell, this version opens a new chapter in Kong’s AI story: the introduction of the Kong MCP Gateway. The following diagram illustrates the new component:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HoH0a6wsAFHEwDbA" /></figure><p><em>Similarly to what the Kong AI Gateway does with the GenAI models, the LLMs we consume, the Kong MCP Gateway protects and controls the MCP Servers consumption.</em></p><p>The new MCP Gateway allows you, just like you do with LLMs and other GenAI models with Kong AI Gateway, to host your MCP Servers sitting behind the Gateway and, therefore, leveraging the capabilities provided by it. That includes:</p><ul><li>Specific MCP based security mechanisms like <a href="https://modelcontextprotocol.io/docs/tutorials/security/authorization">OAuth 2.1</a>.</li><li>Combine historical Kong API Gateway plugins to implement policies like Transformations, Security, etc.</li><li>Take advantage of the Observability plugins provided by the Kong API Gateway and Konnect.</li></ul><h3>Kong AI Gateway and Amazon Bedrock</h3><p>We should get started discussing the benefits Kong AI Gateway brings to Amazon Bedrock for an LLM-based application. As we stated before, Kong AI Gateway leverages the existing Kong API Gateway extensibility model to provide specific AI-based plugins. Here are some values provided by Kong AI Gateway and its plugins:</p><ul><li>AI Proxy and AI Proxy Advanced plugins: the Multi-LLM capability allows the AI Gateway to abstract Amazon Bedrock (and other LLMs as well) load balancing models based on several policies including latency time, model usage, semantics etc. These plugins extract LLM observability metrics (like number of requests, latencies, and errors for each LLM provider) and the number of incoming prompt tokens and outgoing response tokens. All this is in addition to hundreds of metrics already provided by Kong Gateway on the underlying API requests and responses. Finally, Kong AI Gateway leverages the observability capabilities provided by Konnect to track Amazon Bedrock usage out of the box, as well as generate reports based on monitoring data.</li><li>Prompt Engineering:</li><li>AI Prompt Template plugin, responsible for pre-configuring AI prompts to users</li><li>AI Prompt Decorator plugin, which injects messages at the start or end of a caller’s chat history.</li><li>AI Prompt Compressor</li><li>AI Prompt Guard plugin lets you configure a series of PCRE-compatible regular expressions to allow and block specific prompts, words, phrases, or otherwise and have more control over a LLM service, controlled by Amazon Bedrock.</li><li>AI Semantic Prompt Guard plugin to self-configure semantic (or natural-language based pattern-matching) prompt protection.</li><li>AI RAG Injector</li><li>AI Semantic Response Guard</li><li>AI Semantic Cache plugin caches responses based on threshold, to improve performance (and therefore end-user experience) and cost.</li><li>AI Rate Limiting Advanced, you can tailor per-user or per-model policies based on the tokens returned by the LLM provider under Amazon Bedrock management or craft a custom function to count the tokens for requests.</li><li>AI Request Transformer and AI Response Transformer plugins seamlessly integrate with the LLM on Amazon Bedrock, enabling introspection and transformation of the request’s body before proxying it to the Upstream Service and prior to forwarding the response to the client.</li><li>AI AWS Guardrails</li><li>AI LLM as Judge</li><li>AI PII Sanitizer</li></ul><p>Besides, the Kong AI/MCP Gateway use cases can combine policies implemented by hundreds of Kong Gateway plugins, such as:</p><ul><li>Authentication and Authorization: OIDC, mTLS, API Key, LDAP, SAML, Open Policy Agent (OPA)</li><li>Traffic Control: Request Validator and Size Limiting, WebSocket support, Route by Header, etc.</li><li>Observability: OpenTelemetry (OTel), Prometheus, TCP-Log, etc.</li></ul><p>Also, from the architecture perspective, in a nutshell, the Konnect Control Plane and Data Plane nodes topology remains the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*5UpzL8ldQCwYxHOU" /></figure><p><em>By leveraging the same underlying core of Kong API Gateway, we’re reducing complexity in deploying the AI/MCP Gateway capabilities as well. And of course, it works on Konnect, Kubernetes, self-hosted, or across multiple clouds.</em></p><h3>Kong AI/MCP Gateway and Amazon Bedrock Integration and APIs</h3><p>Let’s take a look at the specific integration point between Kong AI/MCP Gateway and Amazon Bedrock. To get a better understanding, here’s the architecture cut isolating both:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*GqN6x7UN7Xde7Dh9" /></figure><p>The consumer can be any RESTful-based component, in our case will be a Strands Agent.</p><p>As you can see, there are two important topics here:</p><ul><li>OpenAI API specification</li><li>Amazon Bedrock Converse API and EKS Pod Identity</li></ul><p>Let’s discuss each one of them.</p><h3>OpenAI API support</h3><p>Kong AI Gateway supports the <a href="https://platform.openai.com/docs/overview">OpenAI API specification</a>. That means the consumer can send standard OpenAI requests to the Kong AI Gateway. As a basic example, consider this OpenAI request:</p><pre>curl https://api.openai.com/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;gpt-5&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><p>When we add Kong AI Gateway, sitting in front of Amazon Bedrock, we’re not just exposing it but also allowing the consumers to use the same mechanism — in this case, OpenAI APIs — to consume it. That leads to a very flexible and powerful capability when we come to development processes. In other words, Kong AI Gateway normalizes the consumption of any LLM infrastructure, including Amazon Bedrock, Mistral, OpenAI, Cohere, etc.</p><p>As an exercise, the new request should be something like this. The request has some minor differences:</p><ul><li>It sends a request to the Kong API Gateway Data Plane Node.</li><li>It replaces the OpenAI endpoint with a Kong API Gateway route.</li><li>The API Key is actually managed by the Kong API Gateway now.</li><li>We’re using an Amazon Bedrock Model, us.amazon.nova-micro-v1:0.</li></ul><pre>curl http://$DATA_PLANE_LB/bedrock-route \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;apikey: $KONG_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;us.amazon.nova-micro-v1:0&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><h3>Amazon EKS Pod Identity</h3><p>The Konnect Data Plane Node, where the Kong AI Gateway runs, has to send requests to Amazon Bedrock on behalf of the Gateway consumer. In order to do it, we need to grant permissions to the Data Plane deployment to access the Amazon Bedrock API, more precisely the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html">Converse API</a>, used by the AI Gateway to interact with it.</p><p>For example, here’s a request to Amazon Bedrock, using the AWS CLI. Use “aws configure” first to set your Access and Secret Key as well as the AWS region you want to use then run the command.</p><pre>aws bedrock-runtime converse \<br> --model-id us.amazon.nova-micro-v1:0 \<br> --messages &#39;[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:[{&quot;text&quot;:&quot;Tell me about John Coltrane&quot;}]}]&#39;</pre><p>It’s reasonable to consume Bedrock service with local CLI commands. However, for Amazon EKS deployments, the recommended approach is <a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html">EKS Pod Identity</a>, instead of simple long-term credentials like AWS Access and Secret Keys. In a nutshell, EKS Pod Identity allows the Data Plane Pod’s container to use the AWS SDK and send API requests to AWS services using AWS Identity and Access Management (IAM) permissions.</p><p>Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances.</p><p>As another best practice, we recommend the Private Key and Digital Certificate pair used by the Konnect Control Plane and the Data Plane connectivity to be stored in AWS Secrets Manager. In this sense, the Data Plane deployment refers to the secrets to get installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ETysRVOlh7BlG3jB" /></figure><h3>Amazon Elastic Kubernetes Service (EKS) installation and preparation</h3><p>Now, we’re ready to get started with our Kong AI Gateway deployment. As the installation architecture defines, it will be running on an EKS Cluster.</p><h3>Amazon EKS Cluster creation</h3><p>In order to create the EKS Cluster, you can use <a href="https://eksctl.io/">eksctl</a>, the official CLI for Amazon EKS, like this:</p><pre>eksctl create cluster --name kong-aws \<br> --version 1.34 \<br> --region us-east-2 \<br> --nodegroup-name kong-node \<br> --node-type g6.xlarge \<br> --nodes 1</pre><p>Note command creates an EKS Cluster, version 1.34, with a single node based on the g6.xlarge instance type, powered by NVIDIA GPUs. That is particularly interesting if you’re planning to deploy and run LLMs locally in the EKS Cluster.</p><h3>Cluster preparation</h3><p>After the installation, we should prepare the Cluster to receive the other components:</p><ul><li><a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/">AWS Load Balancer Controller</a> to expose both Kong AI Gateway with public <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load Balancers</a> (NLB).</li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html">EKS Pod Identity Agent</a> to be able to define Pod Identity Association and grant permissions to the Kong AI Gateway Data Plane Pod to access both Amazon Bedrock and AWS Secrets Manager.</li></ul><p>Please, refer to the official documentation to learn more about the components and their installation processes.</p><h3>Kubernetes Kong Operator and Konnect Control Plane/Data Plane installation</h3><p>The Kong AI/MCP Gateway deployment process can be divided into two steps:</p><ul><li>Pod Identity configuration</li><li>Kong Control Plane and Data Plane deployment using the Kubernetes Kong Operator</li></ul><h3>Pod Identity configuration</h3><p>In this first step, we configure EKS Pod Identity describing which AWS Services the Data Plane Pods should be allowed to access. In our case, we need to consume Amazon Bedrock and AWS Secrets Manager.</p><h4>IAM Policy</h4><p>Pod Identity relies on IAM policies to check which AWS Services can be consumed. Our policy should allow access to AWS Bedrock actions so the Data Plane will be able to send requests to Bedrock APIs, more precisely, <a href="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html">Converse</a> and <a href="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html">ConverseStream</a> APIs. The Converse API requires permission to the InvokeModel action as ConverseStream needs access to InvokeModelWithResponseStream.</p><p>Also, we’re going to use AWS Secrets Manager to store our Private Key and Digital Certificate pair, which the Konnect Control Plane and Data Plane used to communicate.</p><p>Considering all this, let’s create the IAM policy with the following request:</p><pre>aws iam create-policy \<br> --policy-name bedrock-policy \<br> --policy-document &#39;{<br>    &quot;Version&quot;: &quot;2012–10–17&quot;,<br>    &quot;Statement&quot;: [<br>      {<br>        &quot;Effect&quot;: &quot;Allow&quot;,<br>        &quot;Action&quot;: [<br>          &quot;bedrock:InvokeModel&quot;,<br>          &quot;bedrock:InvokeModelWithResponseStream&quot;,<br>          &quot;secretsmanager:ListSecrets&quot;,<br>          &quot;secretsmanager:GetSecretValue&quot;<br>        ],<br>        &quot;Resource&quot;: &quot;*&quot;<br>      }<br>    ]<br>}&#39;</pre><h4>Pod Identity Association</h4><p>Pod Identity takes a Kubernetes Service Account to manage the permissions. So create the Kubernetes namespace for the Kong Data Plane deployment and a simple Service Account inside of it.</p><pre>kubectl create namespace kong<br>kubectl create sa kaigateway-podid-sa -n kong</pre><p>Now we’re ready to create the Pod Identity Association. We use the same eksctl command to do it:</p><pre>eksctl create podidentityassociation \<br> --cluster kong-aws<br> --region us-east-2 \<br> --namespace kong \<br> --service-account-name kaigateway-podid-sa \<br> --role-name kaigateway-aws-podid-role \<br> --permission-policy-arns arn:aws:iam::&lt;your_aws_account&gt;:policy/bedrock-policy</pre><p>The command above is responsible for:</p><ul><li>IAM Role creation based on the IAM Policy we previously defined</li><li>Associating the IAM Role to the existing Kubernetes Service Account</li></ul><p>You can check the Pod Identity Association with:</p><pre>eksctl get podidentityassociation \<br> --cluster kong-aws \<br> --region us-east-2 \<br> --namespace kong</pre><p>Check the IAM Role and Policies attached with:</p><pre>aws iam get-role --role-name kaigateway-aws-podid-role<br>aws iam list-attached-role-policies --role-name kaigateway-aws-podid-role</pre><h3>Kong Operator and Control Plane/Data Plane deployment</h3><p>The Data Plane deployment comprises the following steps:</p><ul><li>Konnect subscription</li><li>Kong Operator installation</li><li>Konnect Control Plane creation</li><li>Konnect Data Plane deployment</li></ul><h4>Konnect subscription</h4><p>This fundamental step is required to get access to Konnect. Click on the <a href="https://konghq.com/products/kong-konnect/register">Registration</a> link and present your credentials. Or, if you already have a Konnect subscription, <a href="https://cloud.konghq.com/us">log in to it</a>.</p><p>Any Konnect subscription has a “default” Control Plane defined. You can proceed using it or optionally create a new one. The following instructions are based on a new Control Plane.</p><h4>Kong Operator installation</h4><p>The Konnect Control Plane and Data Plane creation and deployments are totally managed by the <a href="https://developer.konghq.com/operator/">Kong Operator</a> (KO) which is fully compliant with the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Kubernetes Operator</a> standards. First, we need to install it. Check the documentation to learn more.</p><pre>helm repo add kong https://charts.konghq.com<br>helm repo update kong<br>helm upgrade - install kong-operator kong/kong-operator \<br>-n kong-system \<br> --create-namespace \<br> --set image.tag=2.0.6 \<br> --set kubernetes-configuration-crds.enabled=true \<br> --set env.ENABLE_CONTROLLER_KONNECT=true</pre><p>You can check the Operator’s log with:</p><pre>kubectl logs -f $(kubectl get pod -n kong-system -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kong-operator&quot;))&#39; | jq -r &#39;.name&#39;) -n kong-system</pre><h4>Konnect Control Plane creation</h4><p>In order to start using the Kong Operator, you need to issue a <a href="https://developer.konghq.com/konnect-api/#konnect-api-authentication">Konnect Personal Access Token (PAT)</a> or a System Access Token (SAT). To generate your PAT, go to Konnect UI, click on your initials in the upper right corner of the Konnect home page, then select “Personal Access Tokens.” Click on “+ Generate Token,” name your PAT, set its expiration time, and be sure to copy and save it as an environment variable also named as PAT. Konnect won’t display your PAT again.</p><pre>export PAT=&lt;YOUR_PAT&gt;</pre><p>Now, you can create your <a href="https://developer.konghq.com/operator/konnect/crd/control-planes/hybrid/">Control Plane</a> with the first Kong Operator declaration. The first CRD tells the Operator which Konnect region you’re using, and what token (PAT or SAT) to use to authenticate.</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>kind: KonnectAPIAuthConfiguration<br>apiVersion: konnect.konghq.com/v1alpha1<br>metadata:<br>  name: konnect-api-auth-conf<br>  namespace: kong<br>spec:<br>  type: token<br>  token: $PAT<br>  serverURL: us.api.konghq.com<br>EOF</pre><p>The second CRD creates the new Control Plane:</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>kind: KonnectGatewayControlPlane<br>apiVersion: konnect.konghq.com/v1alpha2<br>metadata:<br>  name: kong-aws<br>  namespace: kong<br>spec:<br>  createControlPlaneRequest:<br>    name: kong-aws<br>  konnect:<br>    authRef:<br>      name: konnect-api-auth-conf<br>EOF</pre><h4>Konnect Data Plane deployment</h4><p>Finally, in the last step we will deploy the <a href="https://developer.konghq.com/operator/dataplanes/get-started/hybrid/deploy-dataplane/">Data Plane</a>. The following KonnectExtension CRD allows you to define your Konnect Control Plane details. The DataPlane CRD actually creates the Konnect Data Plane, attaching it to the Control Plane.</p><p>Note the DataPlane declaration:</p><ul><li>Adds the service annotation to request a public NLB for the Data Plane.</li><li>Uses the Kubernetes Service Account that has been used to create the Pod Identity Association, so the Data Plane can have access to both Amazon Bedrock and Secrets Manager.</li></ul><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>kind: KonnectExtension<br>apiVersion: konnect.konghq.com/v1alpha2<br>metadata:<br>  name: konnect-config-aws<br>  namespace: kong<br>spec:<br>  clientAuth:<br>    certificateSecret:<br>      provisioning: Automatic<br>  konnect:<br>    controlPlane:<br>      ref:<br>        type: konnectNamespacedRef<br>        konnectNamespacedRef:<br>          name: kong-aws<br>---<br>apiVersion: gateway-operator.konghq.com/v1beta1<br>kind: DataPlane<br>metadata:<br>  name: kong-aws-dp<br>  namespace: kong<br>spec:<br>  extensions:<br>  - kind: KonnectExtension<br>    name: konnect-config-aws<br>    group: konnect.konghq.com<br>  deployment:<br>    podTemplateSpec:<br>      spec:<br>        containers:<br>        - name: proxy<br>          image: kong/kong-gateway:3.12.0.2<br>        serviceAccountName: kaigateway-podid-sa<br>  network:<br>    services:<br>      ingress:<br>        name: proxy-kong-aws<br>        type: LoadBalancer<br>        annotations:<br>          &quot;service.beta.kubernetes.io/aws-load-balancer-scheme&quot;: &quot;internet-facing&quot;<br>          &quot;service.beta.kubernetes.io/aws-load-balancer-nlb-target-type&quot;: &quot;ip&quot;<br>EOF</pre><h3>Checking the Data Plane</h3><p>Use the Load Balancer created during the deployment:</p><pre>export DATA_PLANE_LB=$(kubectl get service -n kong proxy-kong-aws --output=jsonpath=&#39;{.status.loadBalancer.ingress[].hostname}&#39;)<br>export DATA_PLANE_PORT=$(kubectl get service -n kong proxy-kong-aws --output=jsonpath=&#39;{.spec.ports[].port}&#39;)</pre><p>You should get a response like this:</p><pre>% curl -i $DATA_PLANE_LB:$DATA_PLANE_PORT<br>HTTP/1.1 404 Not Found<br>Date: Tue, 09 Dec 2025 12:55:42 GMT<br>Content-Type: application/json; charset=utf-8<br>Connection: keep-alive<br>Content-Length: 103<br>X-Kong-Response-Latency: 0<br>Server: kong/3.12.0.1-enterprise-edition<br>X-Kong-Request-Id: 95b09b2a40dc745d59ea6d684f9c4a13<br><br>{<br>  &quot;message&quot;:&quot;no Route matched with those values&quot;,<br>  &quot;request_id&quot;:&quot;95b09b2a40dc745d59ea6d684f9c4a13&quot;<br>}</pre><p>Now we can define the Kong Objects necessary to expose and control Bedrock, including Kong Gateway Service, Routes, and Plugins.</p><h3>decK</h3><p>With <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong) you can manage Kong Konnect configuration and create Kong Objects in a declarative way. decK state files describe the configuration of Kong API Gateway. State files encapsulate the complete configuration of Kong in a declarative format, including services, routes, plugins, consumers, and other entities that define how requests are processed and routed through Kong. Please check the decK documentation to learn how to <a href="https://docs.konghq.com/deck/latest/installation/">install</a> it.</p><p>You can ping Konnect using your PAT with:</p><pre>% deck gateway ping --konnect-control-plane-name kong-aws --konnect-token $PAT<br>Successfully Konnected to the Kong organization!</pre><h3>Strands Agent SDK — AI Agent and MCP fundamentals</h3><p>With all the components we need for our AI Agent in place, it’s time for the most exciting part of this blog post: the Strands Agent itself. Let’s start with a very basic code:</p><pre>from strands import Agent<br>from strands.models.bedrock import BedrockModel<br><br>import boto3<br>import os<br><br><br>aws_access_key_id = os.getenv(&quot;AWS_ACCESS_KEY_ID&quot;);<br>aws_secret_access_key = os.getenv(&quot;AWS_SECRET_ACCESS_KEY&quot;);<br><br><br>session = boto3.Session(<br>  aws_access_key_id=f&quot;{aws_access_key_id}&quot;,<br>  aws_secret_access_key=f&quot;{aws_secret_access_key}&quot;<br>)<br><br>bedrock_model = BedrockModel(<br>  model_id=&quot;us.anthropic.claude-sonnet-4–20250514-v1:0&quot;,<br>  boto_session=session<br>)<br><br><br>agent = Agent(model=bedrock_model)<br>agent(&quot;Who is Aldous Huxley?&quot;)</pre><p>The code is straightforward. The diagram shows the two components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*kCZQuzC9QwOR1poD" /></figure><p>Strands has an extensive list of Model Providers where <a href="https://strandsagents.com/latest/documentation/docs/user-guide/concepts/model-providers/amazon-bedrock/">Amazon Bedrock</a> is one of them. The provider requires AWS credentials which, for this code, are set in the environment variables.</p><p>The code instantiates an <a href="https://strandsagents.com/latest/documentation/docs/api-reference/agent/#strands.agent.agent.Agent">Agent</a> which sends a simple prompt to the model. Nothing really special until you send prompts like:</p><pre>agent(&quot;What is the weather like in my location?&quot;)</pre><p>If you do, you’ll get a response like:</p><pre>&quot;I don&#39;t have access to your current location or real-time weather data, so I can&#39;t tell you what the weather is like where you are.<br><br>To get current weather information, you could:<br>- Check a weather app on your phone<br>- Visit weather websites like Weather.com, AccuWeather, or your local meteorological service<br>- Ask a voice assistant with location access<br>- Look outside your window for immediate conditions<br><br>If you&#39;d like to tell me your city or region, I could provide general information about the typical climate there, but I wouldn&#39;t be able to give you current conditions.&quot;</pre><h3>MCP principles</h3><p>That’s one of the main reasons we should add Tools to our Agent. In fact, LLMs are not able to process prompts like this without setting some context. By context, we mean artifacts like transcripts or documents, presentations or functions so the LLM can respond accordingly. That’s the purpose of MCPs: to provide a standardized mechanism for LLMs to access some context.</p><p>As defined in the documentation, there are <a href="https://modelcontextprotocol.io/docs/learn/architecture#primitives">three core primitives</a> that a MCP server can expose:</p><ul><li>Tools: Executable functions that AI applications can invoke to perform actions (e.g., file operations, API calls, database queries)</li><li>Resources: Data sources that provide contextual information to AI applications (e.g., file contents, database records, API responses)</li><li>Prompts: Reusable templates that help structure interactions with language models (e.g., system prompts, few-shot examples)</li></ul><p>Now, before adding Tools to our Agent, let’s create a Kong version for it.</p><h3>Kong Version</h3><h3>Kong declarations</h3><p>Kong needs to be configured to understand how to connect and consume Bedrock. Here’s the decK declaration saying so:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: kong-aws<br>services:<br>- name: agent-service<br>  url: http://httpbin.konghq.com<br>  routes:<br>  - name: agent-route-bedrock<br>    paths:<br>    - /bedrock-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-bedrock&quot;<br>      enabled: true<br>      config:<br>        max_request_body_size: 65536<br>        targets:<br>        - auth:<br>            allow_override: false<br>          route_type: &quot;llm/v1/chat&quot;<br>          model:<br>            provider: &quot;bedrock&quot;<br>            options:<br>              bedrock:<br>                aws_region: &quot;us-east-2&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true</pre><p>The declaration defines multiple Kong Objects:</p><ul><li><a href="https://docs.konghq.com/gateway/latest/get-started/services-and-routes/">Kong Gateway Service</a> named “agent-service”. The service doesn’t need to map to any real upstream URL. In fact, it can point somewhere empty, for example, <a href="http://localhost:32000.">http://localhost:32000.</a> This is because the AI Proxy plugin, also configured in the declaration, overwrites the upstream URL. This requirement will be removed in a later Kong revision.</li><li>Kong Route: the Gateway Service has a route defined with the “/bedrock-route” path. That’s the route we’re going to consume to reach out to Bedrock.</li><li>Kong Route Plugins: the Kong Route has some plugins configured. Note that only the AI Proxy and Key Auth Plugins are enabled. The other ones are configured but disabled.</li><li>Kong <a href="https://developer.konghq.com/plugins/ai-proxy-advanced/">AI Proxy Advanced Plugin</a>: that’s the Plugin that allows us to connect to the LLM infrastructure. For <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy/how-to/machine-learning-platform-integration-guides/bedrock/">Bedrock</a>, among other things, we need to configure which AWS region we should connect to.</li></ul><p>The declaration has been tagged as “agent” so you can manage its objects without impacting any other ones you might have created previously. Also, note the declaration is saying it should be applied to the “kong-aws” Konnect Control Plane.</p><p>You can submit the declaration to your Konnect Control Plane with the following decK command:</p><pre>deck gateway sync --konnect-token $PAT &lt;YOUR_DECLARATION_FILE&gt;</pre><h3>First Strands Kong AI Gateway Agent</h3><p>A very noticeable improvement in this updated Kong version is that, since we have Pod Identity configured in our EKS Cluster, we don’t need to set up AWS credentials. Besides, as Kong exposes itself as an OpenAI complaint server, the Agent uses the OpenAI Model Provider to talk to it.</p><pre>from strands import Agent<br>from strands.models.openai import OpenAIModel<br><br>import os<br><br><br>data_plane_lb = os.getenv(&quot;DATA_PLANE_LB&quot;);<br>data_plane_port = os.getenv(&quot;DATA_PLANE_PORT&quot;);<br><br>kong_dp = f&quot;http://{data_plane_lb}:{data_plane_port}&quot;<br>kong_dp_route = f&quot;http://{data_plane_lb}:{data_plane_port}/bedrock-route&quot;<br><br>print(&quot;kong_dp_route&quot;)<br>print(kong_dp_route)<br><br><br>openai_model = OpenAIModel(<br>  client_args={<br>    &quot;base_url&quot;: kong_dp_route,<br>    &quot;api_key&quot;: &quot;dummy&quot;<br>  },<br>  model_id=&quot;us.anthropic.claude-sonnet-4–20250514-v1:0&quot;<br>)<br><br><br>agent = Agent(model=openai_model)<br>agent(&quot;Who is Aldous Huxley?&quot;)</pre><p>Here’s the new topology:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*SpX4ck4KxBp1WP_r" /></figure><p>As all requests are reported back to the Konnect Control Plane, you can check them in the Analytics tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*43c9FuPziUl3zOxO" /></figure><p>However, we don’t have any Tools defined so we still get a similar response if we send some weather related prompt.</p><h3>Strands Tools Decorator</h3><p>Tools can be added to Agents using several techniques. The most basic one is <a href="https://strandsagents.com/latest/documentation/docs/api-reference/tools/#strands.tools.decorator">Tool Decorator</a>, which transforms a Python function into a Strands Tool. Our next Python code has some Tools defined with Decorators.</p><pre>from strands import Agent, tool<br>from strands.models.openai import OpenAIModel<br>from strands.types.tools import ToolResult, ToolUse<br><br>import os<br>from typing import Any<br>import httpx<br><br>data_plane_lb = os.getenv(&quot;DATA_PLANE_LB&quot;);<br>data_plane_port = os.getenv(&quot;DATA_PLANE_PORT&quot;);<br>kong_dp = f&quot;http://{data_plane_lb}:{data_plane_port}&quot;<br>kong_dp_route = f&quot;http://{data_plane_lb}:{data_plane_port}/bedrock-route&quot;<br>print(&quot;kong_dp_route&quot;)<br>print(kong_dp_route)<br><br><br>model_id=&quot;us.anthropic.claude-sonnet-4–20250514-v1:0&quot;<br>print(&quot;model_id&quot;)<br>print(model_id)<br><br><br>@tool<br>def get_user_geolocation() -&gt; str:<br>#def get_user_geolocation(tool: ToolUse, **kwargs: Any) -&gt; ToolResult:<br>  &quot;&quot;&quot;Get the user&#39;s geolocation<br>     description: Location query. Takes no parameter and returns lat and lng of current location.<br>  &quot;&quot;&quot;<br><br>  city = kong_dp + &quot;/geolocation&quot;<br>  result = httpx.post(city)<br>  lat = result.json()[&quot;location&quot;][&quot;lat&quot;]<br>  lng = result.json()[&quot;location&quot;][&quot;lng&quot;]<br><br>  return f&quot;{lat},{lng}&quot;<br><br><br>@tool<br>def get_user_geocode(location: str) -&gt; str:<br>#def get_user_location(tool: ToolUse, **kwargs: Any) -&gt; ToolResult:<br>  &quot;&quot;&quot;Get the user&#39;s geocode<br>     description: Geo Code query. Take a location defined as &quot;lat,lng&quot; and return the name of the city.<br>  &quot;&quot;&quot;<br><br>  city = kong_dp + &quot;/geocode&quot;<br>  result = httpx.get(city, params={&quot;latlng&quot;: location})<br>  result = result.json()[&quot;results&quot;][0][&quot;address_components&quot;][-3][&quot;long_name&quot;]<br>  print(f&quot;city: {result}&quot;)<br><br>  return result<br><br><br>@tool<br>def get_weather(geocode: str) -&gt; str:<br>#def get_weather(tool: ToolUse, **kwargs: Any) -&gt; ToolResult:<br>  &quot;&quot;&quot;Get the geocode&#39;s weather<br>     description: Weather query. Accepts US Zipcode, UK Postcode, Canada Postalcode, IP address, latitude/longitude, or city name.<br>  &quot;&quot;&quot;<br><br>  weatherapi_url = kong_dp + &quot;/weather&quot;<br>  result = httpx.get(weatherapi_url, params={&quot;q&quot;: geocode})<br>  weather_info = result.json()[&quot;current&quot;]<br><br>  return result.json()<br><br><br><br><br><br><br>openai_model = OpenAIModel(<br>  client_args={<br>    &quot;base_url&quot;: kong_dp_route,<br>    &quot;api_key&quot;: &quot;dummy&quot;<br>  },<br>  model_id=model_id<br>)<br><br><br>tools = [get_user_geolocation, get_user_geocode, get_weather]<br>agent = Agent(model=openai_model, tools=tools)<br><br># Use the agent with the custom tools<br>prompt = &quot;What is the weather like in Tokyo?&quot;<br><br>print(&quot;prompt&quot;)<br>print(prompt)<br>agent(prompt)</pre><p>Here’s the new diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*Gs2LtKEPPoJ86Ggt" /></figure><h3>decK</h3><p>All Tools are sitting behind the Kong Data Plane, so, before running the code, we need to define the new Kong Objects. Here’s the new decK declaration:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: kong-aws<br>services:<br>- name: agent-service<br>  url: http://httpbin.konghq.com<br>  routes:<br>  - name: agent-route-bedrock<br>    paths:<br>    - /bedrock-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-bedrock&quot;<br>      enabled: true<br>      config:<br>        max_request_body_size: 65536<br>        targets:<br>        - auth:<br>            allow_override: false<br>          route_type: &quot;llm/v1/chat&quot;<br>          model:<br>            provider: &quot;bedrock&quot;<br>            options:<br>              bedrock:<br>                aws_region: &quot;us-east-2&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true<br>- name: geolocation-service<br>  url: https://www.googleapis.com/geolocation/v1/geolocate<br>  routes:<br>  - name: geolocation-route<br>    paths:<br>    - /geolocation<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-geolocation<br>      enabled: true<br>      config:<br>        add:<br>          querystring:<br>          - key:${{ env &quot;DECK_GOOGLEAPI_API_KEY&quot; }}<br>- name: geocode-service<br>  url: https://maps.googleapis.com/maps/api/geocode/json<br>  routes:<br>  - name: geocode-route<br>    paths:<br>    - /geocode<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-geocode<br>      enabled: true<br>      config:<br>        add:<br>          querystring:<br>          - key:${{ env &quot;DECK_GOOGLEAPI_API_KEY&quot; }}<br>- name: weather-service<br>  url: https://api.weatherapi.com/v1/current.json<br>  routes:<br>  - name: weather-route<br>    paths:<br>    - /weather<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-apiweathermap<br>      enabled: true<br>      config:<br>        add:<br>          headers:<br>          - key:${{ env &quot;DECK_WEATHERAPI_API_KEY&quot; }}<br>plugins:<br>- name: post-function<br>  instance_name: post-function1<br>  enabled: false<br>  config:<br>    access:<br>    - kong.log.err(&quot;request.body&quot;, kong.request.get_raw_body())<br>    body_filter:<br>    - kong.log.err(&quot;response.body&quot;, kong.response.get_raw_body())</pre><p>You can submit the declaration with:</p><pre>deck gateway sync --konnect-token $PAT &lt;YOUR_DECLARATION_FILE&gt;</pre><p>The configuration defines regular Kong Gateway Services to each one of the external services:</p><ul><li>Geolocation: based on Google’s service. It can be consumed directly with requests like:</li></ul><pre>curl -X POST &quot;https://www.googleapis.com/geolocation/v1/geolocate?key=&lt;YOUR_GOOGLE_APIKEY?&quot;</pre><p>You should get a response similar to:</p><pre>{<br>  &quot;location&quot;: {<br>  &quot;lat&quot;: -23.5732992,<br>  &quot;lng&quot;: -46.6550784<br>  },<br>  &quot;accuracy&quot;: 2377.7249247988957<br>}</pre><ul><li>Geocode: also based on Google’s service. For example, if you submit the following request, with the latitude and longitude you got from the previous one, you should get a response like:</li></ul><pre>curl -sX POST &quot;https://maps.googleapis.com/maps/api/geocode/json?latlng=-23.5732992,-46.6550784&amp;key=&lt;YOUR_GOOGLE_APIKEY&gt;&quot; | jq -r &#39;.plus_code.compound_code&#39;<br>C8GV+MXM São Paulo, State of São Paulo, Brazil</pre><ul><li><a href="https://www.weatherapi.com/">WeatherAPI</a>: that’s the actual service responsible for returning the weather in a given city. Here’s an example:</li></ul><pre>curl -sX POST &quot;https://api.weatherapi.com/v1/current.json?q=S%C3%A3o%20Paulo&quot; -H key:&lt;YOUR_WEATHERAPI_APIKEY&gt; | jq -r &#39;.current.condition&#39;<br>{<br>  &quot;text&quot;: &quot;Patchy rain nearby&quot;,<br>  &quot;icon&quot;: &quot;//cdn.weatherapi.com/weather/64x64/day/176.png&quot;,<br>  &quot;code&quot;: 1063<br>}</pre><ul><li>Each Kong Route defined for the Kong Gateway Services is configured with the <a href="https://developer.konghq.com/plugins/request-transformer-advanced/">Request Transformer Advanced</a> Plugin. Each plugin instance injects the corresponding API Keys defined in the decK environment variables key:${{ env “DECK_GOOGLEAPI_API_KEY” }} and key:${{ env “DECK_WEATHERAPI_API_KEY” }}</li><li>Lastly, we have configured the <a href="https://developer.konghq.com/plugins/post-function/">Post Function</a> Plugin globally so we can check the raw bodies of all coming requests and responses.</li></ul><h3>Observability</h3><p>From the Observability perspective, it’s interesting to check the metrics and logs the Data Plane reports back to its Control Plane. For example, if you execute the code, invoking the Agent with the prompt “What is the weather like in Tokyo?”, you should see in the Konnect Analytics Explorer page the consumption of two Kong Gateway Services, “agent-service” and “weather-service”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hqYERkHgEPor5Csg" /></figure><p>Note that the “agent-service” was called twice. That’s where the Post Function plugin can be really helpful. If you check the Data Plane log you should see all requests that have been processed. Let’s check them now. You can do it with the following “kubectl” command:</p><pre>kubectl logs -f $(kubectl get pod -n kong -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kong-kong&quot;))&#39; | jq -r &#39;.name&#39;) -n kong</pre><h4>Request and Response #1</h4><p>In this first request sent through the Kong AI Gateway, the Agent asks the Bedrock model, in our case us.anthropic.claude-sonnet-4–20250514-v1:0, as specified in the code, to tell which Tool should be called to answer the prompt.</p><p>Note the Kong AI Gateway Proxy Advanced Plugin sends a request to Bedrock using the expected “/chat/completions”. Here’s the request body:</p><pre>2025/12/08 15:06:10 [error] 2680#0: *10917 [kong] [string &quot;kong.log.err(&quot;request.body&quot;, kong.request.get…&quot;]:1 [post-function] request.body<br>{<br>  &quot;messages&quot;:[<br>    {<br>      &quot;role&quot;:&quot;user&quot;,<br>      &quot;content&quot;:[<br>        {<br>          &quot;type&quot;:&quot;text&quot;,<br>          &quot;text&quot;:&quot;What is the weather like in Tokyo?&quot;<br>        }<br>      ]<br>    }<br>  ],<br>  &quot;inferenceConfig&quot;:{<br>  },<br>  &quot;anthropic_version&quot;:&quot;bedrock-2023–05–31&quot;,<br>  &quot;toolConfig&quot;:{<br>    &quot;tools&quot;:[<br>      {<br>        &quot;toolSpec&quot;:{<br>          &quot;description&quot;:&quot;Get the user&#39;s geolocation\ndescription: Location query. Takes no parameter and returns lat and lng of current location.&quot;,<br>          &quot;inputSchema&quot;:{<br>            &quot;json&quot;:{<br>              &quot;type&quot;:&quot;object&quot;,<br>              &quot;required&quot;:[],<br>              &quot;properties&quot;:{}<br>            }<br>          },<br>          &quot;name&quot;:&quot;get_user_geolocation&quot;<br>        }<br>      },<br>      {<br>        &quot;toolSpec&quot;:{<br>          &quot;description&quot;:&quot;Get the user&#39;s geocode\ndescription: Geo Code query. Take a location defined as \&quot;lat,lng\&quot; and return the name of the city.&quot;,<br>          &quot;inputSchema&quot;:{<br>            &quot;json&quot;:{<br>              &quot;type&quot;:&quot;object&quot;,<br>              &quot;properties&quot;:{<br>                &quot;location&quot;:{<br>                  &quot;description&quot;:&quot;Parameter location&quot;,<br>                  &quot;type&quot;:&quot;string&quot;<br>                }<br>              },<br>              &quot;required&quot;:[<br>                &quot;location&quot;<br>              ]<br>            }<br>          },<br>          &quot;name&quot;:&quot;get_user_geocode&quot;<br>        }<br>      },<br>      {<br>        &quot;toolSpec&quot;:{<br>          &quot;description&quot;:&quot;Get the geocode&#39;s weather\ndescription: Weather query. Accepts US Zipcode, UK Postcode, Canada Postalcode, IP address, latitude/longitude, or city name.&quot;,<br>          &quot;inputSchema&quot;:{<br>            &quot;json&quot;:{<br>              &quot;type&quot;:&quot;object&quot;,<br>              &quot;properties&quot;:{<br>                &quot;geocode&quot;:{<br>                  &quot;description&quot;:&quot;Parameter geocode&quot;,<br>                  &quot;type&quot;:&quot;string&quot;<br>                }<br>              },<br>              &quot;required&quot;:[<br>                &quot;geocode&quot;<br>              ]<br>            }<br>          },<br>          &quot;name&quot;:&quot;get_weather&quot;<br>        }<br>      }<br>    ]<br>  }<br>}<br>, client: 192.168.77.15, server: kong, request: &quot;POST /bedrock-route/chat/completions HTTP/1.1&quot;, host: &quot;k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com&quot;, request_id: &quot;21303c3016a136366e17ed07c40bfd2c&quot;</pre><p>Bedrock replies back with a stream based response. If you check it you’ll see it instructs the Agent to call the “get_weather” Tool passing “Tokyo” as the expected “geocode” parameter.</p><h4>Request and Response #2</h4><p>This request actually calls the second Kong Route “/weather”, which exposes the “weather-service” Gateway Service and, therefore, sends requests to the WeatherAPI external Service, injecting its API Key.</p><h4>Request and Response #3</h4><p>In the final request, the Agent invokes the Kong AI Gateway, which routes the request to Bedrock model, to get the final response, now considering the context for the city the prompt included:</p><pre>{<br>  &quot;role&quot;:&quot;user&quot;,<br>  &quot;content&quot;:[<br>  {<br>    &quot;toolResult&quot;:{<br>      &quot;content&quot;:[<br>        {<br>        &quot;json&quot;:{<br>          &quot;result&quot;:[<br>            {<br>              &quot;type&quot;:&quot;text&quot;,<br>              &quot;text&quot;:&quot;{&#39;location&#39;: {&#39;name&#39;: &#39;Tokyo&#39;, &#39;region&#39;: &#39;Tokyo&#39;, &#39;country&#39;: &#39;Japan&#39;, &#39;lat&#39;: 35.6895, &#39;lon&#39;: 139.6917, &#39;tz_id&#39;: &#39;Asia/Tokyo&#39;, &#39;localtime_epoch&#39;: 1765206397, &#39;localtime&#39;: &#39;2025–12–09 00:06&#39;}, &#39;current&#39;: {&#39;last_updated_epoch&#39;: 1765206000, &#39;last_updated&#39;: &#39;2025–12–09 00:00&#39;, &#39;temp_c&#39;: 12.0, &#39;temp_f&#39;: 53.6, &#39;is_day&#39;: 0, &#39;condition&#39;: {&#39;text&#39;: &#39;Partly Cloudy&#39;, &#39;icon&#39;: &#39;//cdn.weatherapi.com/weather/64x64/night/116.png&#39;, &#39;code&#39;: 1003}, &#39;wind_mph&#39;: 11.0, &#39;wind_kph&#39;: 17.6, &#39;wind_degree&#39;: 357, &#39;wind_dir&#39;: &#39;N&#39;, &#39;pressure_mb&#39;: 1016.0, &#39;pressure_in&#39;: 30.0, &#39;precip_mm&#39;: 0.0, &#39;precip_in&#39;: 0.0, &#39;humidity&#39;: 47, &#39;cloud&#39;: 0, &#39;feelslike_c&#39;: 10.1, &#39;feelslike_f&#39;: 50.2, &#39;windchill_c&#39;: 8.2, &#39;windchill_f&#39;: 46.8, &#39;heatindex_c&#39;: 10.5, &#39;heatindex_f&#39;: 50.8, &#39;dewpoint_c&#39;: -0.2, &#39;dewpoint_f&#39;: 31.7, &#39;vis_km&#39;: 10.0, &#39;vis_miles&#39;: 6.0, &#39;uv&#39;: 0.0, &#39;gust_mph&#39;: 15.4, &#39;gust_kph&#39;: 24.8}}&quot;<br>            }<br>          ]<br>        }<br>        }<br>      ],<br>      &quot;toolUseId&quot;:&quot;tooluse_PIgSZiFLTXSTsHTqMKUeIQ&quot;<br>    }<br>  }<br>  ]<br>}</pre><p>The response is also a stream and you can see it after executing the code.</p><pre>$ python3 &lt;YOUR_CODE&gt;.py<br>kong_dp_route<br>http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com:80/bedrock-route<br>model_id<br>us.anthropic.claude-sonnet-4–20250514-v1:0<br>prompt<br>What is the weather like in Tokyo?<br>I&#39;ll get the weather information for Tokyo for you.<br>Tool #1: get_weather<br>Based on the current weather data for Tokyo, here&#39;s what it&#39;s like right now:<br><br>**Current Weather in Tokyo:**<br>- **Temperature:** 12.2°C (54°F)<br>- **Condition:** Partly cloudy<br>- **Time:** 2:23 AM local time (December 9, 2025)<br>- **Feels like:** 10.3°C (50.6°F)<br>- **Wind:** 18 km/h (11.2 mph) from the North<br>- **Humidity:** 47%<br>- **Cloud coverage:** 75%<br>- **Visibility:** 10 km (6 miles)<br>- **Pressure:** 1017 mb<br><br>It&#39;s currently nighttime in Tokyo with partly cloudy skies and cool temperatures. The weather is quite pleasant for this time of year, with moderate humidity and good visibility. There&#39;s a gentle northerly wind making it feel a bit cooler than the actual temperature.</pre><p>Note that, considering the direct prompt, the Agent just called only the “get_weather” Tool. You may find interesting to run the same Agent code with different prompts like:</p><ul><li>prompt = “What is the weather like in my location?”</li><li>prompt = “What is the city I’m currently located in?”</li><li>prompt = “What is the city located at latitude 52.52000659999999 and longitude 13.404954?”</li></ul><p>Depending on the prompts the Agent will invoke other Tools to respond properly to it. For example, for the prompt:</p><pre>prompt = &quot;What is the weather in the city located at latitude 52.52000659999999 and longitude 13.404954?&quot;</pre><p>The execution of the Agent should look like this. This time the Agent had to call two Tools: “get_user_code” and “get_weather”</p><pre>kong_dp_route<br>http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com:80/bedrock-route<br>model_id<br>us.anthropic.claude-sonnet-4–20250514-v1:0<br>prompt<br>What is the weather in the city located at latitude 52.52000659999999 and longitude 13.404954<br>I&#39;ll help you get the weather for that location. Let me first find out what city is at those coordinates, and then get the weather information.<br>Tool #1: get_user_geocode<br>city: Berlin<br>Now let me get the weather for Berlin:<br>Tool #2: get_weather<br>The coordinates you provided (latitude 52.52000659999999, longitude 13.404954) are in **Berlin, Germany**.<br><br>Here&#39;s the current weather in Berlin:<br><br>**Current Conditions (as of 6:45 PM local time):**<br>- **Temperature:** 12.2°C (54°F)<br>- **Condition:** Partly cloudy<br>- **Feels like:** 11.0°C (51.8°F)<br>- **Humidity:** 94%<br>- **Wind:** 11.9 kph (7.4 mph) from the SSW<br>- **Pressure:** 1013.0 mb<br>- **Visibility:** 10 km (6 miles)<br>- **Cloud coverage:** 75%<br>- **No precipitation** currently<br><br>It&#39;s currently nighttime in Berlin with partly cloudy skies and relatively mild temperatures for December.</pre><h3>Strands MCPClient and Kong AI MCP Proxy Plugin</h3><p>The current Agent code is helpful but the Tools are defined using Decorators and are not MCP based. That is, the Agent integrates with Kong AI Gateway and Bedrock with the Strands OpenAI Model Provider but consumes the Tools by using the “<a href="https://www.python-httpx.org/">httpx</a>” Python package which sends regular REST/HTTP based requests to the Kong Data Plane.</p><p>What we really want is to take the existing Kong Gateway Services and convert them as MCP Tools. Moreover, the Agent should call them as regular Tools and not manage REST based calls. To illustrate the scenario, here’s a new diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*-YlPa8iv1waOHPDo" /></figure><p>On the Kong AI Gateway side, it’s time to configure the <a href="https://developer.konghq.com/plugins/ai-mcp-proxy/">AI MCP Proxy Plugin</a> to take the existing Kong Gateway Service and create MCP Tools based on it. In order to do it, check the new decK declaration:</p><h3>decK</h3><p>The main difference here is that we have enabled the AI MCP Proxy Plugin to each one of our Kong Gateway Services related to the external services “geolocation”, “geocode” and “weather”.</p><p>In summary, as the AI Proxy Advanced plugin takes care of the LLM model connection, the AI MCP Proxy plugin provides similar capabilities to the MCP Tools and external services.</p><p>Inside the <a href="https://developer.konghq.com/plugins/ai-mcp-proxy/reference/#schema--config-mode">“config”</a> section you can see the “mode: conversion-listener” configuration. That means the plugin will not just convert RESTful API paths into MCP tools but also accept incoming MCP requests on the Route path.</p><p>The “<a href="https://developer.konghq.com/plugins/ai-mcp-proxy/reference/#schema--config-tools">tools</a>” section of the AI MCP Proxy declaration is an OpenAPI snippet. It is used to instruct how the plugin should integrate with the external service. It has the following main configuration parameters:</p><ul><li>method: it’s related to the HTTP method the plugin should use to consume the external service.</li><li><a href="https://developer.konghq.com/plugins/ai-mcp-proxy/reference/#schema--config-tools-parameters">parameters</a>: it maps the API parameters the external service expects.</li></ul><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: kong-aws<br>vaults:<br>- name: aws<br>  prefix: aws-vault<br>  config:<br>    region: us-east-2<br>services:<br>- name: agent-service<br>  url: http://httpbin.konghq.com<br>  routes:<br>  - name: agent-route-bedrock<br>    paths:<br>    - /bedrock-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-bedrock&quot;<br>      enabled: true<br>      config:<br>        max_request_body_size: 65536<br>        targets:<br>        - auth:<br>            allow_override: false<br>          route_type: &quot;llm/v1/chat&quot;<br>          model:<br>            provider: &quot;bedrock&quot;<br>            options:<br>              bedrock:<br>                aws_region: &quot;us-east-2&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true<br>- name: geolocation-service<br>  url: https://www.googleapis.com/geolocation/v1/geolocate<br>  routes:<br>  - name: geolocation-route<br>    paths:<br>    - /geolocation<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-geolocation<br>      enabled: true<br>      config:<br>        add:<br>          querystring:<br>          - &quot;{vault://aws-vault/google-apikey/google-key}&quot;<br>    - name: ai-mcp-proxy<br>      instance_name: ai-mcp-proxy-geolocation<br>      enabled: true<br>      config:<br>        mode: conversion-listener<br>        tools:<br>        - description: Get current geo location<br>          method: POST<br>        server:<br>          timeout: 60000<br>- name: geocode-service<br>  url: https://maps.googleapis.com/maps/api/geocode/json<br>  routes:<br>  - name: geocode-route<br>  paths:<br>  - /geocode<br>  plugins:<br>  - name: request-transformer-advanced<br>    instance_name: request-transformer-advanced-geocode<br>    enabled: true<br>    config:<br>      add:<br>        querystring:<br>        - &quot;{vault://aws-vault/google-apikey/google-key}&quot;<br>  - name: ai-mcp-proxy<br>    instance_name: ai-mcp-proxy-geocode<br>    enabled: true<br>    config:<br>      mode: conversion-listener<br>      tools:<br>      - description: Get current geo code<br>        method: GET<br>        parameters:<br>        - name: latlng<br>          in: query<br>          required: true<br>          schema:<br>            type: string<br>          description: Get Code query. Accepts Lat and Lng.<br>        server:<br>          timeout: 60000<br>- name: weather-service<br>  url: https://api.weatherapi.com/v1/current.json<br>  routes:<br>  - name: weather-route<br>  paths:<br>  - /weather<br>  plugins:<br>  - name: request-transformer-advanced<br>    instance_name: request-transformer-advanced-apiweathermap<br>    enabled: true<br>    config:<br>      add:<br>        headers:<br>        - &quot;{vault://aws-vault/weather-apikey/weather-key}&quot;<br>  - name: ai-mcp-proxy<br>    instance_name: ai-mcp-proxy-apiweathermap<br>    enabled: true<br>    config:<br>    mode: conversion-listener<br>    tools:<br>    - description: Get current weather for a location<br>      method: GET<br>      parameters:<br>      - name: q<br>        in: query<br>        required: true<br>        schema:<br>          type: string<br>        description: Weather query. Accepts US Zipcode, UK Postcode, Canada Postalcode,<br>          IP address, latitude/longitude, or city name.<br>      server:<br>        timeout: 60000</pre><h3>Secrets</h3><p>A second update we did in code is to add a Vault to our Konnect environment to get the API Keys from AWS Secrets Manager. It provides a more secure and recommended environment to store secrets like the API Keys the Gateway Services use.</p><p>This is done in the “vault” section of the declaration. The secrets can be injected into your AWS Secrets Manager with:</p><pre>aws secretsmanager create-secret --name google-apikey --region us-east-2 --secret-string &quot;{\&quot;google-key\&quot;: \&quot;key:$DECK_GOOGLEAPI_API_KEY\&quot;}&quot;</pre><pre>aws secretsmanager create-secret --name weather-apikey --region us-east-2 --secret-string &quot;{\&quot;weather-key\&quot;: \&quot;key:$DECK_WEATHERAPI_API_KEY\&quot;}&quot;</pre><p>You can check the secrets with:</p><pre>kubectl exec -ti $(kubectl get pod -n kong -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kong-kong&quot;))&#39; | jq -r &#39;.name&#39;) -n kong -- kong vault get {vault://aws-vault/google-apikey/google-key}</pre><p>Or even consuming the WeatherAPI Service:</p><pre>curl &quot;$DATA_PLANE_LB/weather?q=Milan&quot;</pre><h3>The Code</h3><p>This time the code takes the Bedrock model to be consumed from an environment variable. The most critical update is the construction of Streamable HTTP based MCP Client with the <a href="https://strandsagents.com/latest/documentation/docs/api-reference/tools/#strands.tools.mcp.mcp_client.MCPClient">MCPClient</a> package. Note that we have one client for each MCP Tool defined by the AI MCP Proxy Plugin. To reach out to them we use the same route path, e.g. “/weather”.</p><p>The Agent object is created the same way we did before. The main difference is that the Tools are based on the Streamable HTTP based MCP Client and not on Tool Decorators.</p><pre>from strands import Agent<br>from strands.models.openai import OpenAIModel<br>from strands.tools.mcp import MCPClient<br>from mcp.client.streamable_http import streamablehttp_client<br>import os<br><br>data_plane_lb = os.getenv(&quot;DATA_PLANE_LB&quot;);<br>data_plane_port = os.getenv(&quot;DATA_PLANE_PORT&quot;);<br>model_id = os.getenv(&quot;MODEL_ID&quot;);<br><br>kong_dp = f&quot;http://{data_plane_lb}:{data_plane_port}&quot;<br>kong_dp_route = f&quot;http://{data_plane_lb}:{data_plane_port}/bedrock-route&quot;<br><br>openai_model = OpenAIModel(<br>  client_args={<br>    &quot;base_url&quot;: kong_dp_route,<br>    &quot;api_key&quot;: &quot;dummy&quot;<br>  },<br>  model_id=model_id<br>)<br><br><br># Define the Streamable HTTP MCP Clients<br>streamable_http_mcp_weather_client = MCPClient(lambda: streamablehttp_client(f&quot;{kong_dp}/weather&quot;))<br><br>streamable_http_mcp_geolocation_client = MCPClient(lambda: streamablehttp_client(f&quot;{kong_dp}/geolocation&quot;))<br><br>streamable_http_mcp_geocode_client = MCPClient(lambda: streamablehttp_client(f&quot;{kong_dp}/geocode&quot;))<br><br><br># Call the Agent<br>with streamable_http_mcp_weather_client, streamable_http_mcp_geolocation_client, streamable_http_mcp_geocode_client:<br>  tools = streamable_http_mcp_weather_client.list_tools_sync() + \<br>          streamable_http_mcp_geolocation_client.list_tools_sync() + \<br>          streamable_http_mcp_geocode_client.list_tools_sync()<br>  agent = Agent(model=openai_model, tools=tools)<br><br>  prompt = &quot;What is the weather like in my location?&quot;<br>  print(&quot;prompt&quot;)<br>  print(prompt)<br>  agent(prompt)</pre><p>If you run the code above you should see something like this. Note the Agent had to call two Tools this time: the first one to get the city name and the second one to get its actual weather.</p><pre>% python3 &lt;YOUR_CODE&gt;.py<br>prompt<br>What is the weather like in my location?<br>I&#39;ll help you get the weather for your current location. First, let me find out where you are, and then I&#39;ll get the weather information for that location.<br>Tool #1: geolocation-route-1<br>Now let me get the weather for your location using those coordinates:<br>Tool #2: weather-route-1<br>Based on your current location in Pickerington, Ohio, here&#39;s the current weather:<br><br>**Current Weather Conditions:**<br>- **Temperature:** 29°F (-2°C)<br>- **Condition:** Sunny ☀️<br>- **Feels like:** 20°F (-7°C)<br>- **Wind:** Northeast at 10 mph (16 km/h)<br>- **Humidity:** 39%<br>- **Visibility:** 9 miles (16 km)<br>- **Pressure:** 30.22 in (1023 mb)<br>- **UV Index:** 0.8 (Low)<br><br>It&#39;s a cold but sunny day in your area! The temperature is below freezing, so make sure to dress warmly if you&#39;re heading outside. The wind chill is making it feel even colder at around 20°F.</pre><h3>Aggregate MCP Tools</h3><p>One last enhancement should be done in our code: as you can see, there is one MCP Client for each MCP Tool enabled by the Kong AI MCP Proxy plugin. That’s not so optimized and does not abstract all Tools in a single MCP Server. That’s the purpose of the AI MCP Proxy plugin “conversion-only” and “listener” modes. Here’s the new abstraction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*X2kvxmqM9sAO0nLM" /></figure><h3>decK</h3><p>The decK basically sets the AI MCP Proxy plugin instances as “conversion-only”. That means the Tool will be created but not exposed. Moreover, each AI MCP Proxy instance has been tagged as “mcp-tools”.</p><p>At the same time, we create a new and serviceless Kong Route with the AI MCP Proxy plugin also enabled. The only difference is that this instance is configured as “mode: listener” and aggregates all AI MCP Proxy instances defined with the tag “mcp-tools”.</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: kong-aws<br>vaults:<br>- name: aws<br>  prefix: aws-vault<br>  config:<br>    region: us-east-2<br>services:<br>- name: agent-service<br>  url: http://httpbin.konghq.com<br>  routes:<br>  - name: agent-route-bedrock<br>  paths:<br>  - /bedrock-route<br>  plugins:<br>  - name: ai-proxy-advanced<br>    instance_name: &quot;ai-proxy-advanced-bedrock&quot;<br>    enabled: true<br>    config:<br>      max_request_body_size: 65536<br>      targets:<br>      - auth:<br>          allow_override: false<br>        route_type: &quot;llm/v1/chat&quot;<br>        model:<br>          provider: &quot;bedrock&quot;<br>          options:<br>            bedrock:<br>              aws_region: &quot;us-east-2&quot;<br>        logging:<br>          log_payloads: true<br>          log_statistics: true<br>- name: geolocation-service<br>  url: https://www.googleapis.com/geolocation/v1/geolocate<br>  routes:<br>  - name: geolocation-route<br>    paths:<br>    - /geolocation<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-geolocation<br>      enabled: true<br>      config:<br>        add:<br>          querystring:<br>          - &quot;{vault://aws-vault/google-apikey/google-key}&quot;<br>    - name: ai-mcp-proxy<br>      instance_name: ai-mcp-proxy-geolocation<br>      enabled: true<br>      tags:<br>      - mcp-tools<br>      config:<br>        mode: conversion-only<br>        tools:<br>        - description: Get current geo location<br>          method: POST<br>- name: geocode-service<br>  url: https://maps.googleapis.com/maps/api/geocode/json<br>  routes:<br>  - name: geocode-route<br>    paths:<br>    - /geocode<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-geocode<br>      enabled: true<br>      config:<br>        add:<br>          querystring:<br>          - &quot;{vault://aws-vault/google-apikey/google-key}&quot;<br>  - name: ai-mcp-proxy<br>    instance_name: ai-mcp-proxy-geocode<br>    enabled: true<br>    tags:<br>    - mcp-tools<br>    config:<br>      mode: conversion-only<br>      tools:<br>      - description: Get current geo code<br>        method: GET<br>        parameters:<br>        - name: latlng<br>          in: query<br>          required: true<br>          schema:<br>            type: string<br>          description: Get Code query. Accepts Lat and Lng.<br>- name: weather-service<br>  url: https://api.weatherapi.com/v1/current.json<br>  routes:<br>  - name: weather-route<br>    paths:<br>    - /weather<br>    plugins:<br>    - name: request-transformer-advanced<br>      instance_name: request-transformer-advanced-apiweathermap<br>      enabled: true<br>      config:<br>        add:<br>          headers:<br>          - &quot;{vault://aws-vault/weather-apikey/weather-key}&quot;<br>    - name: ai-mcp-proxy<br>      instance_name: ai-mcp-proxy-apiweathermap<br>      enabled: true<br>      tags:<br>      - mcp-tools<br>      config:<br>        mode: conversion-only<br>        tools:<br>        - description: Get current weather for a location<br>          method: GET<br>          parameters:<br>          - name: q<br>            in: query<br>            required: true<br>            schema:<br>              type: string<br>            description: Location query. Accepts US Zipcode, UK Postcode, Canada Postalcode,<br>              IP address, latitude/longitude, or city name.<br>routes:<br>  - name: mcp-listener-route<br>    paths:<br>    - /mcp-listener<br>    plugins:<br>    - name: ai-mcp-proxy<br>      config:<br>        mode: listener<br>        server:<br>          tag: mcp-tools<br>          timeout: 45000<br>        logging:<br>          log_statistics: true<br>          log_payloads: false<br>        max_request_body_size: 32768</pre><h3>The Code</h3><p>With the aggregation in place, the code is even simpler:</p><pre>from strands import Agent<br>from strands.models.openai import OpenAIModel<br>from strands.tools.mcp import MCPClient<br>from mcp.client.streamable_http import streamablehttp_client<br>import os<br><br>data_plane_LB = os.getenv(&quot;DATA_PLANE_LB&quot;);<br>data_plane_port = os.getenv(&quot;DATA_PLANE_PORT&quot;);<br>model_id = os.getenv(&quot;MODEL_ID&quot;);<br><br>kong_dp = f&quot;http://{data_plane_lb}:{data_plane_port}&quot;<br>kong_dp_route = f&quot;http://{data_plane_lb}:{data_plane_port}/bedrock-route&quot;<br><br><br>openai_model = OpenAIModel(<br>  client_args={<br>    &quot;base_url&quot;: kong_dp_route,<br>    &quot;api_key&quot;: &quot;dummy&quot;<br>  },<br>  model_id=model_id<br>)<br><br><br>streamable_http_mcp_listener_client = MCPClient(lambda: streamablehttp_client(f&quot;{kong_dp}/mcp-listener&quot;))<br><br>with streamable_http_mcp_listener_client:<br>    tools = streamable_http_mcp_listener_client.list_tools_sync()<br>    agent = Agent(model=openai_model, tools=tools)<br>    agent(&quot;What is the weather like in my location?&quot;)</pre><p>The result should be the same:</p><pre>% python3 &lt;YOUR_CODE&gt;.py<br>kong_dp_route<br>http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com:80/bedrock-route<br>model_id<br>us.anthropic.claude-sonnet-4–20250514-v1:0<br>I&#39;ll help you get the weather for your current location. Let me first find out where you are, and then get the weather information for that location.<br>Tool #1: geolocation-route-1<br>Now let me get the weather for your location using those coordinates:<br>Tool #2: weather-route-1<br>Based on your current location in Pickerington, Ohio, here&#39;s the current weather:<br><br>**Current Weather:**<br>- **Temperature:** 28.9°F (-1.7°C)<br>- **Condition:** Sunny ☀️<br>- **Feels like:** 20.6°F (-6.3°C)<br>- **Wind:** 8.7 mph from the northeast<br>- **Humidity:** 39%<br>- **Visibility:** 9 miles<br>- **Pressure:** 30.22 inches<br><br>It&#39;s a cold but sunny day! The wind chill is making it feel quite a bit colder than the actual temperature, so make sure to dress warmly if you&#39;re heading outside.</pre><p>You can also check the Konnect Analytics’ Explorer dashboard once again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*adtPnNrsIkip4Ri7" /></figure><h3>Conclusion</h3><p>This blog post has presented a basic AI Agent using Kong AI Gateway, Strands and Amazon Bedrock, including LLM models and MCP Servers. It’s totally feasible to implement advanced AI Agents with query transformation, multiple data sources, multiple retrieval stages, RAG, etc. Moreover, Kong AI Gateway provides other plugins to enrich the relationship with the LLM providers and MCP Servers, including Semantic Cache, Semantic Routing, Request and Response Transformation, etc.</p><p>Also, it’s important to keep in mind that we can continue combining other API Gateway plugins to your AI-based use cases like using the OIDC plugin to secure your Foundation Models with AWS Cognito, using the Prometheus plugin to monitor your AI Gateway with Amazon Managed Prometheus and Grafana, and so on.</p><p>Finally, the architectural flexibility provided natively by Konnect and Kong AI Gateway allows us to deploy the Data Planes in a variety of platforms including AWS EC2 VMs, Amazon ECS, and Kong Dedicated Cloud Gateway, Kong’s SaaS service for the Data Planes running in AWS.</p><p>You can discover all the features available on the <a href="https://konghq.com/products/kong-ai-gateway">Kong AI Gateway</a> product page, or you can check the Kong and AWS landing page at <a href="https://konghq.com/partners/aws">https://konghq.com/partners/aws</a> to learn more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a947b9e04ad8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kong AI/MCP Gateway and Kong MCP Server technical breakdown]]></title>
            <link>https://medium.com/@claudioacquaviva/kong-ai-mcp-gateway-and-kong-mcp-server-technical-breakdown-13420f610ee6?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/13420f610ee6</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 13:28:35 GMT</pubDate>
            <atom:updated>2026-01-16T12:55:25.395Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kong AI/MCP Gateway and Kong MCP Server technical breakdown</h3><p>“Too much information running through my brain.”</p><p>When The Police sang this opening line on their 1981 album <a href="https://en.wikipedia.org/wiki/Ghost_in_the_Machine_(album)">Ghost in the Machine</a>, they weren’t thinking about Artificial Intelligence but the sentiment perfectly captures the state of modern AI.</p><p>The song warns of overload of data which parallels how modern AI agents process extensive collections of messages and data. <a href="https://modelcontextprotocol.io/">Model Context Protocol (MCP)</a> is designed to manage this potential mess by creating channels for models to receive only the information they need. In a world where AI systems risk becoming “information junkies”, MCP acts as a framework that organizes data, allowing AI to operate in a standardized and much cleaner way.</p><h3>Kong MCP Gateway Introduction</h3><p>In the latest <a href="https://konghq.com/blog/product-releases/enterprise-mcp-gateway">Kong Gateway 3.12 release</a>, announced last October 2025, specific MCP capabilities have been released:</p><ul><li><a href="https://developer.konghq.com/plugins/ai-mcp-proxy/">AI MCP Proxy</a> plugin: it works as a protocol bridge, translating between MCP and HTTP so that MCP-compatible clients can either call existing APIs or interact with upstream MCP servers through Kong.</li><li><a href="https://developer.konghq.com/plugins/ai-mcp-oauth2/">AI MCP OAuth2</a> plugin: implements the <a href="https://modelcontextprotocol.io/specification/draft/basic/authorization">OAuth 2.0 specification</a> for MCP servers.</li><li>The <a href="https://developer.konghq.com/plugins/prometheus/#mcp-traffic-metrics">Prometheus</a> plugin has been extended to generate MCP-specific metrics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SkOYl6SUnQOY3SDj" /></figure><p><em>The Kong AI Gateway sits in between the AI applications we build and the MCP Servers and GenAI models we consume.</em></p><p>In fact, the Kong MCP Gateway sits side-by-side with the Kong LLM Gateway. Here is a diagram illustrating the components. As you can see the Kong AI Gateway implements the fundamental capabilities as the LLM Gateway and MCP Gateway are responsible for their specific flows: MCP servers and GenAI models.</p><p>This blog post explores how developers can leverage Kong MCP Gateway to implement better and smarter AI agents. We are going to focus on:</p><ul><li>Converting Kong Gateway Services into MCP Servers</li><li>Consuming and protecting existing MCP Servers through Kong MCP Gateway</li></ul><h3>Kong and MCP</h3><h3>MCP Introduction</h3><p>We’ve seen a lot of momentum around the <a href="https://modelcontextprotocol.io/">Model Context Protocol (MCP)</a>, the new standard proposed by <a href="https://www.anthropic.com/news/model-context-protocol">Anthropic</a> in November 2024. For a great introduction to MCP, check the <a href="https://konghq.com/blog/learning-center/what-is-mcp">blog</a> Michael Field, Principal Technical Product Marketing Manager here at Kong, wrote early this year.</p><p>As described in its official portal, MCP follows a <a href="https://modelcontextprotocol.io/docs/learn/architecture#participants">client-server architecture</a> where an AI application (tool or AI Agent) connects to MCP Servers. The MCP specification defines three participants as illustrated above in the diagram taken from the MCP portal:</p><ul><li>MCP Host: The AI application (tool, application, agent) that coordinates and manages one or multiple MCP clients</li><li>MCP Client: A component that maintains a connection to an MCP server and obtains context from an MCP server for the MCP host to use</li><li>MCP Server: A program that provides context to MCP clients</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uGf7goMZrRDcP67C" /></figure><p>Each MCP Server is responsible for exposing <a href="https://modelcontextprotocol.io/docs/learn/architecture#primitives">MCP primitives</a> to the Clients:</p><ul><li>Tools: functions that a MCP Client can invoke.</li><li>Resources: Data sources that provide contextual information to the Client.</li><li>Prompts: Reusable templates that help structure interactions with language models (e.g., system prompts, few-shot examples)</li></ul><p>Check the <a href="https://modelcontextprotocol.io/docs/learn/architecture">Architecture Overview</a> page in the MCP documentation portal to learn more about it.</p><h3>Kong AI/MCP Gateway</h3><p>Currently most organizations still don’t have a consistent way to standardize the MCP development process or enforce best practices. As you’d expect, this often results in risky MCP server deployments and, iIn the long run, it also means longer and more expensive evolving and troubleshooting processes.</p><p>With Kong MCP Gateway we expose and secure all MCP Servers in a single place. Also, the MCP Gateway controls the consumption of the MCP Servers with <a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization">OAuth 2.1</a>, as described in the MCP documentation portal.</p><h3>MCP Communication Fundamentals</h3><p>MCP introduces the notion of <a href="https://modelcontextprotocol.io/docs/learn/architecture#layers">Layers</a>:</p><ul><li>Data layer: defines the protocol for MCP Client and MCP Server communication, based on JSON-RPC 2.0.</li><li>Transport Layer: defines the mechanisms to enable the data exchange between the MCP Client and MCP Server. Basically it supports two mechanisms:</li><li><a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#stdio">Stdio</a>: based on standard input/output streams. It’s helpful for communication between tools like Cursor, which plays the MCP Host role, and local MCP Server running in Docker, for example.</li><li><a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http">Streamable HTTP</a>: the MCP Client and MCP Server is based on HTTP.</li></ul><p>For the purpose of this blog post, where the Kong AI Gateway mediates the communication between the MCP Client and MCP Server, we are going to focus on the Streamable HTTP mechanism.</p><h3>JSON-RPC 2.0</h3><p>To get started let’s exercise a little bit the JSON-RPC messages we typically have in a MCP communication. To put simply, <a href="https://www.jsonrpc.org/">JSON-RPC</a> is a RPC based mechanism where the parts communicate through JSON based messages.</p><p>For example, here’s a simple JSON-RPC server, written in Python, which exposes the well known <a href="https://www.weatherapi.com/">WeatherAPI</a> public service. To give it a try, you can call the service directly with the following request. It assumes you have an environment variable set with your WeatherAPI API Key.</p><pre>curl &quot;https://api.weatherapi.com/v1/current.json?q=Milan&quot; -H key:$WEATHERAPI_API_KEY</pre><p>The architecture is quite simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*gamt8JQDrXQDUE5e" /></figure><p>The code uses the <a href="https://pypi.org/project/json-rpc/">“json-rpc” Python package</a> which implements the JSON-RPC specification. Some comments about the code:</p><ul><li>The json-rpc library defines the “dispatcher” construct where we define the methods supported by the JSON-RPC server.</li><li>The server uses the “http.server” package so it can be called using HTTP protocol by tools like Curl.</li></ul><pre>from http.server import BaseHTTPRequestHandler, HTTPServer<br>from jsonrpc import JSONRPCResponseManager, dispatcher<br>import json, os, requests<br><br># Constants<br>BASE_URL = &quot;https://api.weatherapi.com/v1/current.json&quot;<br>WEATHERAPI_API_KEY = os.getenv(&quot;WEATHERAPI_API_KEY&quot;);<br><br># Register multiple JSON-RPC methods on the same dispatcher<br>@dispatcher.add_method<br>def ping():<br>  return &quot;pong&quot;<br><br>@dispatcher.add_method<br>def add(x: int, y: int):<br>  return x + y<br><br>@dispatcher.add_method<br>def get_current_weather(city: str):<br>  &quot;&quot;&quot;Fetch current weather data for a given city.&quot;&quot;&quot;<br>  params = {&quot;key&quot;: WEATHERAPI_API_KEY, &quot;q&quot;: city}<br><br>  response = requests.get(BASE_URL, params=params)<br>  response.raise_for_status()<br><br>  data = response.json()<br>  location = data[&quot;location&quot;][&quot;name&quot;]<br>  temp_c = data[&quot;current&quot;][&quot;temp_c&quot;]<br>  temp_f = data[&quot;current&quot;][&quot;temp_f&quot;]<br>  condition = data[&quot;current&quot;][&quot;condition&quot;][&quot;text&quot;]<br><br>return f&quot;Weather in {location}: {temp_c} Celsius, {temp_f} Fahrenheit, {condition}&quot;<br><br><br><br><br># Define simple JSON-RPC handler<br>class JSONRPCHandler(BaseHTTPRequestHandler):<br>  def do_POST(self):<br>    content_length = int(self.headers.get(&quot;Content-Length&quot;, 0))<br>    request_json = self.rfile.read(content_length).decode(&quot;utf-8&quot;)<br><br>    # Handle JSON-RPC request<br>    response = JSONRPCResponseManager.handle(request_json, dispatcher)<br>    response_json = json.dumps(response.data)<br><br>    # Return response<br>    self.send_response(200)<br>    self.send_header(&quot;Content-Type&quot;, &quot;application/json&quot;)<br>    self.end_headers()<br>    self.wfile.write(response_json.encode(&quot;utf-8&quot;))<br><br>if __name__ == &quot;__main__&quot;:<br>  server = HTTPServer((&quot;127.0.0.1&quot;, 4000), JSONRPCHandler)<br>  print(&quot;JSON-RPC server running at http://127.0.0.1:4000&quot;)<br>  server.serve_forever()</pre><p>After starting the server, you can send requests like these ones:</p><pre># curl -X POST http://127.0.0.1:4000 -H &quot;Content-Type: application/json&quot; -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;, &quot;method&quot;:&quot;ping&quot;, &quot;id&quot;:1}&#39;<br># curl -X POST http://127.0.0.1:4000 -H &quot;Content-Type: application/json&quot; -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;, &quot;method&quot;:&quot;add&quot;, &quot;params&quot;:{&quot;x&quot;:4,&quot;y&quot;:5}, &quot;id&quot;:1}&#39;<br># curl -X POST http://127.0.0.1:4000 -H &quot;Content-Type: application/json&quot; -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;, &quot;method&quot;:&quot;get_current_weather&quot;, &quot;params&quot;:{&quot;city&quot;:&quot;Milan&quot;}, &quot;id&quot;:1}&#39;</pre><p>Let’s take a look at the messages going back and forth. The latest request sends a message like</p><pre>{<br>  &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>  &quot;method&quot;:&quot;get_current_weather&quot;,<br>  &quot;params&quot;:{<br>    &quot;city&quot;:&quot;Milan&quot;<br>  },<br>  &quot;id&quot;:1<br>}</pre><p>The message is fully compliant with the <a href="https://www.jsonrpc.org/specification">JSON-RPC 2.0 specification</a> where it defines the Request Object with the following members:</p><ul><li>jsonrpc: specifies the version of the JSON-RPC protocol, in order case, “2.0”.</li><li>method: contains the name of the method we want to invoke. Note the methods are defined as “dispatchers” methods in our code.</li><li>params: contains the structured value that holds the parameters to be used for the invocation. They are defined in our Python functions.</li><li>id: it’s an identifier defined by the caller. It acts as a correlation key so the client knows which result belongs to which call. When omitted, the request is just a notification and the server must not send a response.</li></ul><p>The Server is expected to return a Response Object. That’s done by the JSONRPCResponseManager class imported from the same “jsonrpc” package. Here’s ours:</p><pre>{<br>  &quot;result&quot;: &quot;Weather in Milan: 9.1 Celsius, 48.4 Fahrenheit, Sunny&quot;,<br>  &quot;id&quot;: 1,<br>  &quot;jsonrpc&quot;: &quot;2.0&quot;<br>}</pre><p>The Object has the following members:</p><ul><li>jsonrpc: the same member we found in the Request Object.</li><li>result: the actual response.</li><li>error: required on errors, which is not our case.</li><li>id: this is required and must be the same as the value of the id member in the Request Object. In our case, “1”.</li></ul><p>This is a good exercise but does not really implement our goal. What we really want is to have the Kong Data Plane playing the MCP Server role. That means we need to configure the Data Plane with the new MCP Gateway capabilities. However, before doing so, we need to dive a bit deeper into the MCP Transport Layer.</p><h3>MCP Transport Layer — Streamable HTTP</h3><p>As stated before, MCP defines two transport mechanisms: Stdio and Streamable HTTP. Since we are going to add Kong AI/MCP Gateway in our cases, we are going to focus on Streamable HTTP.</p><p>We can build Streamable HTTP based MCP Servers using multiple programming languages, for example, TypeScript, Python or C#. Specifically for Python, <a href="https://gofastmcp.com/">FastMCP</a> packages are recommended. In fact, the official <a href="https://github.com/modelcontextprotocol/python-sdk">MCP Python SDK</a> has incorporated FastMCP to provide a simpler MCP Server development.</p><h3>MCP Server</h3><p>Here’s the MCP Server version, using FastMCP, of our JSON-RPC 2.0 Server. As you can see the topology and the messages going back and forth are quite similar to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*i9wYHO3FSvbXxnPY" /></figure><pre>import os<br><br>import requests<br>from mcp.server.fastmcp import FastMCP<br><br># Initialize FastMCP server<br>mcp = FastMCP(&quot;weather&quot;, stateless_http=True)<br><br># Constants<br>BASE_URL = &quot;https://api.weatherapi.com/v1/current.json&quot;<br>WEATHERAPI_API_KEY = os.getenv(&quot;WEATHERAPI_API_KEY&quot;);<br><br># Register multiple MCP tools<br>@mcp.tool()<br>def ping() -&gt; str:<br>  &quot;&quot;&quot;Return pong&quot;&quot;&quot;<br>  return &quot;pong&quot;<br><br>@mcp.tool()<br>def add(x: int, y: int) -&gt; int:<br>  &quot;&quot;&quot;Add two numbers&quot;&quot;&quot;<br>  return x + y<br><br>@mcp.tool()<br>def get_current_weather(city: str = &quot;Firenze&quot;) -&gt; str:<br>  &quot;&quot;&quot;Fetch current weather data for a given city.&quot;&quot;&quot;<br>  params = {&quot;key&quot;: WEATHERAPI_API_KEY, &quot;q&quot;: city}<br><br>  response = requests.get(BASE_URL, params=params)<br>  response.raise_for_status()<br><br>  data = response.json()<br>  location = data[&quot;location&quot;][&quot;name&quot;]<br>  temp_c = data[&quot;current&quot;][&quot;temp_c&quot;]<br>  temp_f = data[&quot;current&quot;][&quot;temp_f&quot;]<br>  condition = data[&quot;current&quot;][&quot;condition&quot;][&quot;text&quot;]<br><br>  return f&quot;Weather in {location}: {temp_c} Celsius, {temp_f} Fahrenheit, {condition}&quot;<br><br><br>if __name__ == &quot;__main__&quot;:<br>  mcp.run(transport=&#39;streamable-http&#39;)</pre><p>Some comments here:</p><ul><li>By default, a MCP Server is stateful, meaning it starts and keeps sessions with MCP Clients. The official MCP documentation describes the <a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#session-management">Session Management</a> implemented by a Server and supported by the Client. At the same time, the FastMCP constructs allow a Stateless implementation. Read the <a href="https://github.com/modelcontextprotocol/python-sdk?tab=readme-ov-file#streamable-http-transport">Python SDK documentation</a> to learn more. You can check that in the line mcp = FastMCP(“weather”, stateless_http=True).</li><li>The code defines correspondent MCP Tools to each dispatcher we have in our JSON-RPC 2.0 Server.</li><li>By default, the MCP Server listens to the port 8000.</li></ul><p>With the MCP Server running we can send similar JSON-RPC requests to it. For example, to list all tools available in the MCP Server run:</p><pre>curl -X POST http://127.0.0.1:8000/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>    &quot;id&quot;: 1,<br>    &quot;method&quot;: &quot;tools/list&quot;<br>  }&#39;</pre><p>Note that the Server provides a specific “/mcp” path where we should send requests to. Also, the client (in our case, Curl) must accept both “application/json” and “text/event-stream” media types.</p><p>The MCP Server provides several methods to expose its capabilities, including Prompts, Resources and Tools. For example, our MCP Server code defines three Tools: “ping”, “add”, and “get_current_weather”. With the request above we can send a request to the MCP Server and get the list of its Tools with all their specifications.</p><p>Here’s the output of the request:</p><pre>event: message<br>data: {&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;result&quot;:{&quot;tools&quot;:[{&quot;name&quot;:&quot;ping&quot;,&quot;description&quot;:&quot;Return pong&quot;,&quot;inputSchema&quot;:{&quot;properties&quot;:{},&quot;title&quot;:&quot;pingArguments&quot;,&quot;type&quot;:&quot;object&quot;},&quot;outputSchema&quot;:{&quot;properties&quot;:{&quot;result&quot;:{&quot;title&quot;:&quot;Result&quot;,&quot;type&quot;:&quot;string&quot;}},&quot;required&quot;:[&quot;result&quot;],&quot;title&quot;:&quot;pingOutput&quot;,&quot;type&quot;:&quot;object&quot;}},{&quot;name&quot;:&quot;add&quot;,&quot;description&quot;:&quot;Add two numbers&quot;,&quot;inputSchema&quot;:{&quot;properties&quot;:{&quot;x&quot;:{&quot;title&quot;:&quot;X&quot;,&quot;type&quot;:&quot;integer&quot;},&quot;y&quot;:{&quot;title&quot;:&quot;Y&quot;,&quot;type&quot;:&quot;integer&quot;}},&quot;required&quot;:[&quot;x&quot;,&quot;y&quot;],&quot;title&quot;:&quot;addArguments&quot;,&quot;type&quot;:&quot;object&quot;},&quot;outputSchema&quot;:{&quot;properties&quot;:{&quot;result&quot;:{&quot;title&quot;:&quot;Result&quot;,&quot;type&quot;:&quot;integer&quot;}},&quot;required&quot;:[&quot;result&quot;],&quot;title&quot;:&quot;addOutput&quot;,&quot;type&quot;:&quot;object&quot;}},{&quot;name&quot;:&quot;get_current_weather&quot;,&quot;description&quot;:&quot;Fetch current weather data for a given city.&quot;,&quot;inputSchema&quot;:{&quot;properties&quot;:{&quot;city&quot;:{&quot;default&quot;:&quot;Firenze&quot;,&quot;title&quot;:&quot;City&quot;,&quot;type&quot;:&quot;string&quot;}},&quot;title&quot;:&quot;get_current_weatherArguments&quot;,&quot;type&quot;:&quot;object&quot;},&quot;outputSchema&quot;:{&quot;properties&quot;:{&quot;result&quot;:{&quot;title&quot;:&quot;Result&quot;,&quot;type&quot;:&quot;string&quot;}},&quot;required&quot;:[&quot;result&quot;],&quot;title&quot;:&quot;get_current_weatherOutput&quot;,&quot;type&quot;:&quot;object&quot;}}]}}</pre><p>The request and the corresponding response follow the <a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#sending-messages-to-the-server">Transport specification</a> for sending messages to a server.</p><p>And this second request calls the “get_current_weather” Tool:</p><pre>curl -X POST http://127.0.0.1:8000/mcp \<br>-H &quot;Content-Type: application/json&quot; \<br>-H &quot;Accept: application/json, text/event-stream&quot; \<br>-d &#39;{<br>  &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>  &quot;method&quot;:&quot;tools/call&quot;,<br>  &quot;params&quot;:{<br>    &quot;name&quot;: &quot;get_current_weather&quot;,<br>    &quot;arguments&quot;: {&quot;city&quot;:&quot;Milan&quot;}<br>  },<br>  &quot;id&quot;: 1<br>}&#39;</pre><p>And the response should follow the <a href="https://modelcontextprotocol.io/specification/2025-06-18/server/tools">Tools specification</a> and be something like:</p><pre>event: message<br>data: {&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;result&quot;:{&quot;content&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;Weather in Milan: 4.1 Celsius, 39.4 Fahrenheit, Light rain&quot;}],&quot;structuredContent&quot;:{&quot;result&quot;:&quot;Weather in Milan: 4.1 Celsius, 39.4 Fahrenheit, Light rain&quot;},&quot;isError&quot;:false}}</pre><h3>Kong version</h3><p>Now, let’s add the Kong API Gateway to the picture. Our goal is to have the Konnect Data Plane to keep exposing regular RESTful based services as well as playing the role of the MCP Server. In order to do it, we need to introduce the new <a href="https://developer.konghq.com/plugins/ai-mcp-proxy/">AI MCP Proxy plugin available</a> in Kong AI Gateway 3.12.</p><p>The new plugin can be configured for the following main use cases:</p><ul><li>Convert regular RESTful APIs, defined as Kong Gateway Services, into MCP tools. In this use case, the plugin acts as a protocol bridge between MCP and HTTP protocols.</li><li>Expose grouped MCP Tools as an MCP Server.</li><li>Proxy MCP requests to existing MCP Servers.</li></ul><p>As you can see, for any of these use cases, the Kong, playing the MCP Gateway role, provides critical values to the Agent development processes, normalizing the access to any MCP Tool or MCP Server you might have in your environment.</p><h3>Convert Kong Gateway Service into MCP Tool</h3><p>Let’s get started converting an existing Kong Gateway Service into a MCP Tool. The following diagram illustrates the architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*EYhTpHFWonTGoggo" /></figure><p>Consider the following <a href="https://developer.konghq.com/deck/">decK</a> (Declarations for Kong) declaration defines a fundamental Kong Gateway Service and Kong Route to expose the same WeatherAPI service, this time, through the Gateway.</p><pre>_format_version: &quot;3.0&quot;<br>services:<br>  - name: weather-service<br>    url: https://api.weatherapi.com/v1/current.json<br>    routes:<br>      - name: weather-route<br>        paths:<br>        - &quot;/weather&quot;<br>        plugins:<br>          - name: request-transformer-advanced<br>            enabled: true<br>            config:<br>              add:<br>                headers:<br>                - key:${{ env &quot;DECK_WEATHERAPI_API_KEY&quot; }}</pre><p>Besides the Kong Service and Route, the declaration enables the <a href="https://developer.konghq.com/plugins/request-transformer-advanced/">Request Transformer Advanced Plugin</a> to the Route to inject the WeatherAPI API Key. Again, we have an environment variable with the API Key. The env variable is named with the DECK_ prefix so decK can take care of it.</p><p>After submitting the declaration you should be able to consume the Kong Route like this:</p><pre># curl -sX POST &quot;$DATA_PLANE_URL/weather?q=Milan&quot; | jq<br>{<br>  &quot;location&quot;: {<br>  &quot;name&quot;: &quot;Milan&quot;,<br>  &quot;region&quot;: &quot;Lombardia&quot;,<br>  &quot;country&quot;: &quot;Italy&quot;,<br>  &quot;lat&quot;: 45.4667,<br>  &quot;lon&quot;: 9.2,<br>  &quot;tz_id&quot;: &quot;Europe/Rome&quot;,<br>  &quot;localtime_epoch&quot;: 1764078087,<br>  &quot;localtime&quot;: &quot;2025–11–25 14:41&quot;<br>},<br>&quot;current&quot;: {<br>  &quot;last_updated_epoch&quot;: 1764077400,<br>  &quot;last_updated&quot;: &quot;2025–11–25 14:30&quot;,<br>  &quot;temp_c&quot;: 8.3,<br>  &quot;temp_f&quot;: 46.9,<br>………<br>}</pre><h3>The AI MCP Proxy Plugin</h3><p>Up to this point, the Kong Gateway Service is not configured to get consumed as a MCP Tool. In order to do it, we need to extend the decK declaration:</p><pre>_format_version: &quot;3.0&quot;<br>services:<br>  - name: weather-service<br>    url: https://api.weatherapi.com/v1/current.json<br>    routes:<br>      - name: weather-route<br>        paths:<br>        - &quot;/weather&quot;<br>        plugins:<br>          - name: request-transformer-advanced<br>            enabled: true<br>            config:<br>              add:<br>                headers:<br>                - key:${{ env &quot;DECK_WEATHERAPI_API_KEY&quot; }}<br>          - name: ai-mcp-proxy<br>            enabled: true<br>            config:<br>              mode: conversion-listener<br>              tools:<br>              - description: Get current weather for a location<br>                method: GET<br>                parameters:<br>                - name: q<br>                  in: query<br>                  required: true<br>                  schema:<br>                    type: string<br>                  description: Location query. Accepts US Zipcode, UK Postcode, Canada Postalcode,<br>                    IP address, latitude/longitude, or city name.<br>                server:<br>                  timeout: 60000</pre><p>This time the Gateway Service has its route configured with the AI MCP Proxy plugin. The “conversion-listener” mode says the Gateway Service can be consumed as a MCP Tool. Inside the “tools” section, the “parameters” configuration has an <a href="https://swagger.io/docs/specification/v3_0/describing-parameters/">OpenAPI</a> snippet describing the parameters the plugin should send to the Gateway Service.</p><p>Differently to what we did before with our basic MCP Server, the AI MCP Proxy plugin implements the full<a href="https://modelcontextprotocol.io/specification/2025-03-26/basic/lifecycle"> MCP Connection Lifecycle</a> as described in the MCP documentation. The lifecycle comprises the main phases:</p><ul><li>Initialization: Capability negotiation and protocol version agreement</li><li>Operation: Normal protocol communication</li><li>Shutdown: Graceful termination of the connection</li></ul><p>Let’s send specific requests to each one of them:</p><h4>Initialization</h4><pre># curl -isX POST $DATA_PLANE_URL/weather \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -d &#39;{<br>  &quot;method&quot;:&quot;initialize&quot;,<br>  &quot;params&quot;:{<br>    &quot;protocolVersion&quot;:&quot;2025–06–18&quot;,<br>    &quot;capabilities&quot;:{},<br>    &quot;clientInfo&quot;:{<br>      &quot;name&quot;:&quot;mcp&quot;,<br>      &quot;Version&quot;:&quot;0.1.0&quot;<br>    }<br>  },<br>  &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>  &quot;id&quot;:1<br>}&#39;</pre><p>You should get a response like this. Note that, following the specification, the plugin has generated a “mcp-session-id” which can be accessed in the output header. The next request should include the header.</p><pre>HTTP/1.1 200 OK<br>Date: Tue, 25 Nov 2025 15:20:55 GMT<br>Content-Type: text/event-stream<br>Transfer-Encoding: chunked<br>Connection: keep-alive<br>mcp-session-id: 1b9f596a-82a3–43bf-80cd-14c65b0f1968<br>Cache-Control: no-store, no-transform<br>X-Kong-Response-Latency: 1<br>Server: kong/3.13.0.0-enterprise-edition<br>X-Kong-Request-Id: 6783e52a3908692c39fcdf5f903e6405<br><br>event: message<br>data: {&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;result&quot;:{&quot;capabilities&quot;:{&quot;tools&quot;:{&quot;listChanged&quot;:false}},&quot;instructions&quot;:&quot;This is a MCP Server which exposes API to MCP clients.&quot;,&quot;serverInfo&quot;:{&quot;name&quot;:&quot;Kong&quot;,&quot;version&quot;:&quot;3.13.0.0&quot;},&quot;protocolVersion&quot;:&quot;2025–06–18&quot;},&quot;id&quot;:1}</pre><p>The second request should be an “initialized” notification:</p><pre># curl -isX POST $DATA_PLANE_URL/weather \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -H &quot;mcp-session-id: 1b9f596a-82a3–43bf-80cd-14c65b0f1968&quot; \<br>  -d &#39;{<br>  &quot;method&quot;:&quot;notifications/initialized&quot;,<br>  &quot;jsonrpc&quot;:&quot;2.0&quot;<br>}&#39;</pre><p>And here’s the response:</p><pre>HTTP/1.1 202 Accepted<br>Date: Tue, 25 Nov 2025 15:22:00 GMT<br>Transfer-Encoding: chunked<br>Connection: keep-alive<br>X-Kong-Response-Latency: 0<br>Server: kong/3.13.0.0-enterprise-edition<br>X-Kong-Request-Id: ef5ec06b73712d28f935a755a0421a44</pre><h4>Operation</h4><p>Now, with the MCP Session defined, let’s send some requests to the MCP Gateway. The first one asks the Gateway to list the Tools available:</p><pre># curl -X POST $DATA_PLANE_URL/weather \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -H &quot;mcp-session-id: 1b9f596a-82a3–43bf-80cd-14c65b0f1968&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>    &quot;id&quot;: 0,<br>    &quot;method&quot;: &quot;tools/list&quot;<br>  }&#39;</pre><p>The expected and formatted response is the following. Note that the plugin names the Tool following the format “weather-route-1”. The parameter is also defined with the format “query_q”.</p><pre>event: message<br>data: {<br>  &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>  &quot;result&quot;:{<br>    &quot;tools&quot;:[<br>      {<br>        &quot;description&quot;:&quot;Get current weather for a location&quot;,<br>        &quot;name&quot;:&quot;weather-route-1&quot;,<br>        &quot;id&quot;:&quot;35129163-b328–422b-9d3a-0ef17a037b80&quot;,<br>        &quot;inputSchema&quot;:{<br>          &quot;type&quot;:&quot;object&quot;,<br>          &quot;properties&quot;:{<br>            &quot;query_q&quot;:{<br>              &quot;type&quot;:&quot;string&quot;,<br>              &quot;description&quot;:&quot;Location query. Accepts US Zipcode, UK Postcode, Canada Postalcode, IP address, latitude/longitude, or city name.&quot;<br>            }<br>          },<br>          &quot;required&quot;:[<br>            &quot;query_q&quot;<br>          ],<br>          &quot;additionalProperties&quot;:false<br>        }<br>      }<br>    ]<br>  },<br>  &quot;id&quot;:0<br>}</pre><p>And here’s the actual Tool call. The request should refer the actual name defined by the plugin and parameter name.</p><pre># curl -X POST $DATA_PLANE_URL/weather \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -H &quot;mcp-session-id: 1b9f596a-82a3–43bf-80cd-14c65b0f1968&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;:&quot;2.0&quot;,<br>    &quot;method&quot;:&quot;tools/call&quot;,<br>    &quot;params&quot;:{<br>      &quot;name&quot;: &quot;weather-route-1&quot;,<br>      &quot;arguments&quot;: {&quot;query_q&quot;:&quot;Milan&quot;}<br>    },<br>    &quot;id&quot;: 1<br>  }&#39;</pre><p>The response should be like:</p><pre>event: message<br>data: {&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;result&quot;:{&quot;isError&quot;:false,&quot;content&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;{\&quot;location\&quot;:{\&quot;name\&quot;:\&quot;Milan\&quot;,\&quot;region\&quot;:\&quot;Lombardia\&quot;,\&quot;country\&quot;:\&quot;Italy\&quot;,\&quot;lat\&quot;:45.4667,\&quot;lon\&quot;:9.2,\&quot;tz_id\&quot;:\&quot;Europe/Rome\&quot;,\&quot;localtime_epoch\&quot;:1764084568,\&quot;localtime\&quot;:\&quot;2025–11–25 16:29\&quot;},\&quot;current\&quot;:{\&quot;last_updated_epoch\&quot;:1764083700,\&quot;last_updated\&quot;:\&quot;2025–11–25 16:15\&quot;,\&quot;temp_c\&quot;:8.2,\&quot;temp_f\&quot;:46.8,\&quot;is_day\&quot;:1,\&quot;condition\&quot;:{\&quot;text\&quot;:\&quot;Partly Cloudy\&quot;,\&quot;icon\&quot;:\&quot;//cdn.weatherapi.com/weather/64x64/day/116.png\&quot;,\&quot;code\&quot;:1003},\&quot;wind_mph\&quot;:3.1,\&quot;wind_kph\&quot;:5.0,\&quot;wind_degree\&quot;:247,\&quot;wind_dir\&quot;:\&quot;WSW\&quot;,\&quot;pressure_mb\&quot;:999.0,\&quot;pressure_in\&quot;:29.5,\&quot;precip_mm\&quot;:0.0,\&quot;precip_in\&quot;:0.0,\&quot;humidity\&quot;:71,\&quot;cloud\&quot;:0,\&quot;feelslike_c\&quot;:7.7,\&quot;feelslike_f\&quot;:45.9,\&quot;windchill_c\&quot;:6.8,\&quot;windchill_f\&quot;:44.2,\&quot;heatindex_c\&quot;:7.4,\&quot;heatindex_f\&quot;:45.3,\&quot;dewpoint_c\&quot;:2.4,\&quot;dewpoint_f\&quot;:36.3,\&quot;vis_km\&quot;:10.0,\&quot;vis_miles\&quot;:6.0,\&quot;uv\&quot;:0.2,\&quot;gust_mph\&quot;:4.4,\&quot;gust_kph\&quot;:7.1}}&quot;}]},&quot;id&quot;:1}</pre><h4>Shutdown</h4><p>The last thing a client should do is to send a Shutdown message to the MCP Tool to gracefully terminate the connection.</p><pre># curl -iX DELETE $DATA_PLANE_URL/weather \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json, text/event-stream&quot; \<br>  -H &quot;mcp-session-id: 1b9f596a-82a3–43bf-80cd-14c65b0f1968&quot;</pre><p>The response should like:</p><pre>HTTP/1.1 200 OK<br>Date: Tue, 25 Nov 2025 15:40:29 GMT<br>Transfer-Encoding: chunked<br>Connection: keep-alive<br>X-Kong-Response-Latency: 0<br>Server: kong/3.13.0.0-enterprise-edition<br>X-Kong-Request-Id: b2387c39eff31360ca3cd25349eaa957</pre><h3>MCP Client</h3><p>To get a more realistic client, here’s a very simple example of a MCP Client consuming the MCP Tool, exposed by the Kong AI Gateway. The MCP_DN and MCP_PORT define where the Konnect Data Plane is located.</p><pre>import asyncio<br>import os<br>import json<br><br>from mcp import ClientSession<br>from mcp.client.streamable_http import streamablehttp_client<br><br>MCP_DN = os.getenv(&quot;MCP_DN&quot;);<br>MCP_PORT = os.getenv(&quot;MCP_PORT&quot;);<br>mcp_url = f&quot;http://{MCP_DN}:{MCP_PORT}/weather&quot;<br><br>print (&quot;MCP: &quot; + mcp_url)<br><br><br>async def main():<br>  # Connect to a streamable HTTP server<br>  async with streamablehttp_client(mcp_url) as (read_stream, write_stream, _,):<br>    print(&quot;Create a session using the client streams&quot;)<br>    async with ClientSession(read_stream, write_stream) as session:<br>      print(&quot;Initialize the connection&quot;)<br>      await session.initialize()<br>      print(&quot;List available tools&quot;)<br>      tools = await session.list_tools()<br>      print(f&quot;Available tools: {[tool.name for tool in tools.tools]}&quot;)<br><br>      result = await session.call_tool(&quot;weather-route-1&quot;, {&quot;query_q&quot;: &quot;Milan&quot;});<br>      print(result)<br><br><br>if __name__ == &quot;__main__&quot;:<br>  asyncio.run(main())</pre><h3>MCP Authorization</h3><p>As you might have noticed, until now, there’s no security mechanism defined to protect the MCP Tool exposed by the Kong AI Gateway. As usual, you can take advantage of the historical plugins the Kong API Gateway provides to implement <a href="https://developer.konghq.com/plugins/?category=authentication">Authentication</a> and <a href="https://developer.konghq.com/plugins/?category=security">Security</a> processes to your MCP Tool, such as Key Auth, OpenID Connect, Open Policy Agent, etc.</p><p>On the other hand, the MCP community has defined <a href="https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization">Authorization specifications</a>, based on OAuth 2 standards. To be fully compliant with the specs, the Kong AI Gateway provides a second plugin, responsible for implementing those specs: the <a href="https://developer.konghq.com/plugins/ai-mcp-oauth2/">AI MCP OAuth2 Plugin</a>.</p><h3>The AI MCP Proxy Plugin</h3><p>To exercise the new plugin, let’s extend the decK declaration one more time:</p><pre>_format_version: &quot;3.0&quot;<br>services:<br>  - name: weather-service<br>    url: https://api.weatherapi.com/v1/current.json<br>    routes:<br>      - name: weather-route<br>        paths:<br>        - &quot;/weather&quot;<br>        plugins:<br>          - name: request-transformer-advanced<br>            enabled: true<br>            config:<br>              add:<br>                headers:<br>                  - key:${{ env &quot;DECK_WEATHERAPI_API_KEY&quot; }}<br>          - name: ai-mcp-proxy<br>            enabled: true<br>            config:<br>              mode: conversion-listener<br>              tools:<br>              - description: Get current weather for a location<br>                method: GET<br>                parameters:<br>                - name: q<br>                  in: query<br>                  required: true<br>                  schema:<br>                    type: string<br>                    description: Location query. Accepts US Zipcode, UK Postcode, Canada Postalcode,<br>                      IP address, latitude/longitude, or city name.<br>              server:<br>                timeout: 60000<br>          - name: ai-mcp-oauth2<br>            enabled: true<br>            config:<br>              resource: ${{ env &quot;DECK_MCP_AUTH_URL&quot; }}<br>              authorization_servers:<br>              - ${{ env &quot;DECK_KEYCLOAK_AUTHZ_URL&quot; }}<br>              introspection_endpoint: ${{ env &quot;DECK_KEYCLOAK_INTROSPECTION_URL&quot; }}<br>              client_id: ${{ env &quot;DECK_CLIENT_ID&quot; }}<br>              client_secret: ${{ env &quot;DECK_CLIENT_SECRET&quot; }}<br>              insecure_relaxed_audience_validation: true</pre><p>The exercise relies on <a href="https://www.keycloak.org/">Keycloak</a>, as the external Identity Provider. In this sense, Keycloak is responsible for issuing Access Tokens and validating them during the MCP Tool consuming time. The validation follows the <a href="https://oauth.net/2/token-introspection/">OAuth2 Token Introspection</a> specification.</p><p>The following diagram shows the relationship between the Kong AI Gateway and the Identity Provider (IdP) for the Introspection flow:</p><ol><li>The MCP Consumer presents its credentials (Client ID + Client Secret) to the IdP.</li><li>The IdP authenticates the Consumer, issues an Access Token and returns it to the Consumer.</li><li>The MCP Consumer sends a request to the Gateway with the Access Token injected.</li><li>The AI/MCP Gateway sends a request to the IdP’s Introspection Endpoint to get the Access Token validated.</li><li>If the Token is still valid the Gateway calls the Tool.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*h-oYFFUFzwZ1U-et" /></figure><p>To get Keycloak installed you can follow the <a href="https://github.com/keycloak/keycloak-quickstarts">Quickstarts</a> available.</p><p>The declaration refers to some Keycloak endpoints and secrets, besides the actual Authorization URL. Let’s go through them:</p><ul><li>MCP_AUTH_URL: that’s the MCP Tool URL, exposed by the Kong Data Plane. Considering our declaration should be like this: http://&lt;DATA_PLANE_DN&gt;:&lt;DATA_PLANE_PORT&gt;/weather.</li><li>KEYCLOAK_AUTHZ_URL: that’s the Keycloak Authorization endpoint. Assuming an existing Keycloak realm, called “kong”, in our case, it should be something like: http://&lt;KEYCLOAK_DN&gt;:&lt;KEYCLOAK_PORT&gt;/realms/kong/protocol/openid-connect/auth.</li><li>KEYCLOAK_INTROSPECTION_URL: that’s the Keycloak Introspection endpoint. In our case, http://&lt;KEYCLOAK_DN:KEYCLOAK_PORT/realms/kong/protocol/openid-connect/token/introspect.</li><li>CLIENT_ID and CLIENT_SECRET: these are the secrets of an existing Keycloak Client.</li></ul><p>Keycloak provides the standard endpoint http://&lt;KEYCLOAK_DN:KEYCLOAK_PORT/realms/kong/.well-known/openid-configuration where you can get these and several other Keycloak configuration parameters.</p><h3>Exploring the Introspection Endpoint</h3><p>To exercise the Introspection Endpoint, let’s send some requests to Keycloak, acting as the Consumer and the Gateway.</p><p>In the first request we play the Consumer role, using the Client Credential Grant to get our Access Token</p><pre>TOKEN=$(curl -s -X POST &quot;http://$KEYCLOAK_LB:$KEYCLOAK_PORT/realms/kong/protocol/openid-connect/token&quot; \<br> --header &quot;Content-Type: application/x-www-form-urlencoded&quot; \<br> --data-urlencode &quot;client_id=$CLIENT_ID&quot; \<br> --data-urlencode &quot;client_secret=$CLIENT_SECRET&quot; \<br> --data-urlencode &quot;grant_type=client_credentials&quot; | jq -r &#39;.access_token&#39;)</pre><p>You can decode the Access Token with:</p><pre>echo $TOKEN | jwt decode -</pre><p>You should get an output like this:</p><pre>Token header<br>------------<br>{<br>  &quot;typ&quot;: &quot;JWT&quot;,<br>  &quot;alg&quot;: &quot;RS256&quot;,<br>  &quot;kid&quot;: &quot;OpQdtUoRz_ZAv-3mq5S92ewVVFzU4-XsDd5raulokp4&quot;<br>}<br><br>Token claims<br>------------<br>{<br>  &quot;acr&quot;: &quot;1&quot;,<br>  &quot;allowed-origins&quot;: [<br>    &quot;http://k8s-kong-kongkong-ad510e7789-cf3de00caeb82cef.elb.us-east-2.amazonaws.com:80&quot;<br>  ],<br>  &quot;aud&quot;: &quot;account&quot;,<br>  &quot;azp&quot;: &quot;kong_id&quot;,<br>  &quot;clientAddress&quot;: &quot;192.168.90.156&quot;,<br>  &quot;clientHost&quot;: &quot;192.168.90.156&quot;,<br>  &quot;client_id&quot;: &quot;kong_id&quot;,<br>  &quot;email_verified&quot;: false,<br>  &quot;exp&quot;: 1764098690,<br>  &quot;iat&quot;: 1764098630,<br>  &quot;iss&quot;: &quot;http://k8s-keycloak-keycloak-7158af03e9-5a453106278d5b6b.elb.us-east-2.amazonaws.com:8080/realms/kong&quot;,<br>  &quot;jti&quot;: &quot;trrtcc:84644440–93ae-5733–28d0-c97c25205f61&quot;,<br>  &quot;preferred_username&quot;: &quot;service-account-kong_id&quot;,<br>  &quot;realm_access&quot;: {<br>    &quot;roles&quot;: [<br>    &quot;offline_access&quot;,<br>    &quot;uma_authorization&quot;,<br>    &quot;default-roles-kong&quot;<br>    ]<br>  },<br>  &quot;resource_access&quot;: {<br>    &quot;account&quot;: {<br>      &quot;roles&quot;: [<br>        &quot;manage-account&quot;,<br>        &quot;manage-account-links&quot;,<br>        &quot;view-profile&quot;<br>      ]<br>    }<br>  },<br>  &quot;scope&quot;: &quot;profile email&quot;,<br>  &quot;sub&quot;: &quot;0583c218–038b-446b-a9cc-22ce0d5f7b93&quot;,<br>  &quot;typ&quot;: &quot;Bearer&quot;</pre><p>Now, playing the Gateway role, we are going to consume the Introspection Endpoint asking the IdP to validate the Access Token:</p><pre>curl -s -X POST &quot;http://$KEYCLOAK_LB:$KEYCLOAK_PORT/realms/kong/protocol/openid-connect/token/introspect&quot; \<br>  -d &quot;token=$TOKEN&quot; \<br>  -d &quot;client_id=$CLIENT_ID&quot; \<br>  -d &quot;client_secret=$CLIENT_SECRET&quot; | jq</pre><p>Here’s a typical response. The “active” at the bottom says the plugin is still good.</p><pre>{<br>  &quot;exp&quot;: 1764098858,<br>  &quot;iat&quot;: 1764098798,<br>  &quot;jti&quot;: &quot;trrtcc:8be1136d-d8de-eb82–875a-15b35781aef5&quot;,<br>  &quot;iss&quot;: &quot;http://k8s-keycloak-keycloak-7158af03e9-5a453106278d5b6b.elb.us-east-2.amazonaws.com:8080/realms/kong&quot;,<br>  &quot;aud&quot;: &quot;account&quot;,<br>  &quot;sub&quot;: &quot;0583c218–038b-446b-a9cc-22ce0d5f7b93&quot;,<br>  &quot;typ&quot;: &quot;Bearer&quot;,<br>  &quot;azp&quot;: &quot;kong_id&quot;,<br>  &quot;acr&quot;: &quot;1&quot;,<br>  &quot;allowed-origins&quot;: [<br>    &quot;http://k8s-kong-kongkong-ad510e7789-cf3de00caeb82cef.elb.us-east-2.amazonaws.com:80&quot;<br>  ],<br>  &quot;realm_access&quot;: {<br>    &quot;roles&quot;: [<br>      &quot;offline_access&quot;,<br>      &quot;uma_authorization&quot;,<br>      &quot;default-roles-kong&quot;<br>    ]<br>  },<br>  &quot;resource_access&quot;: {<br>    &quot;account&quot;: {<br>      &quot;roles&quot;: [<br>        &quot;manage-account&quot;,<br>        &quot;manage-account-links&quot;,<br>        &quot;view-profile&quot;<br>      ]<br>    }<br>  },<br>  &quot;scope&quot;: &quot;profile email&quot;,<br>  &quot;email_verified&quot;: false,<br>  &quot;preferred_username&quot;: &quot;service-account-kong_id&quot;,<br>  &quot;client_id&quot;: &quot;kong_id&quot;,<br>  &quot;username&quot;: &quot;service-account-kong_id&quot;,<br>  &quot;token_type&quot;: &quot;Bearer&quot;,<br>  &quot;active&quot;: true<br>}</pre><p>However, if you wait for the Access Token timeout (in Keycloak is 10min, by default), the Endpoint returns a different output saying so:</p><pre>{<br>  &quot;active&quot;: false<br>}</pre><h3>MCP Client and OAuth2</h3><p>Now, let’s evolve our MCP Client and protect it with the AI MCP OAuth2 Plugin. The code has a new function called “get_access_token” which is responsible for hitting the Keycloak’s Authorization Endpoint to get the Access Token. The token is then used to start the Streamable HTTP Session.</p><p>The code checks if a specific environment variable is defined with an Access Token. If it is, the code uses it to start the Session. One nice way to test the code would be running the code, setting the environment variable with the token, waiting for the timeout and running the code again. You should see an error message related to it.</p><pre>import asyncio<br>import os<br>import json<br><br>from mcp import ClientSession<br>from mcp.client.streamable_http import streamablehttp_client<br><br>import aiohttp<br><br><br>MCP_DN = os.getenv(&quot;MCP_DN&quot;);<br>MCP_PORT = os.getenv(&quot;MCP_PORT&quot;);<br>mcp_url = f&quot;http://{MCP_DN}:{MCP_PORT}/weather&quot;<br><br>print (&quot;MCP: &quot; + mcp_url)<br><br><br>KEYCLOAK_LB = os.getenv(&quot;KEYCLOAK_LB&quot;);<br>KEYCLOAK_PORT = os.getenv(&quot;KEYCLOAK_PORT&quot;);<br>token_url = f&quot;http://{KEYCLOAK_LB}:{KEYCLOAK_PORT}/realms/kong/protocol/openid-connect/token&quot;<br>client_id = os.getenv(&quot;DECK_CLIENT_ID&quot;)<br>client_secret = os.getenv(&quot;DECK_CLIENT_SECRET&quot;)<br><br>print (&quot;KEYCLOAK: &quot; + token_url)<br>print (&quot;CLIENT_ID: &quot; + client_id)<br>print (&quot;CLIENT_SECRET: &quot; + client_secret)<br><br><br>async def get_access_token():<br>  &quot;&quot;&quot;<br>  Fetch OAuth2 token using Client Credentials grant.<br>  &quot;&quot;&quot;<br>  token_data = os.getenv(&quot;TOKEN&quot;)<br>  if token_data is not None:<br>    print(&quot;Env Var Token is set&quot;)<br>    return token_data<br><br>  async with aiohttp.ClientSession() as http:<br>    async with http.post(<br>      token_url,<br>      data={<br>        &quot;grant_type&quot;: &quot;client_credentials&quot;,<br>        &quot;client_id&quot;: client_id,<br>        &quot;client_secret&quot;: client_secret,<br>      },<br>      headers={&quot;Content-Type&quot;: &quot;application/x-www-form-urlencoded&quot;}<br>    ) as resp:<br>      if resp.status != 200:<br>        raise RuntimeError(f&quot;OAuth token error: {resp.status} {await resp.text()}&quot;)<br>      token_data = await resp.json()<br>      return token_data[&quot;access_token&quot;]<br><br><br>async def main():<br>  token = await get_access_token()<br>  print(&quot;Received OAuth Access Token&quot;)<br>  print(token)<br><br>  # Connect to a streamable HTTP server<br>  async with streamablehttp_client(mcp_url, headers={&quot;Authorization&quot;: f&quot;Bearer {token}&quot;}) as (read_stream, write_stream, _,):<br>    print(&quot;Create a session using the client streams&quot;)<br>    async with ClientSession(read_stream, write_stream) as session:<br>      print(&quot;Initialize the connection&quot;)<br>      await session.initialize()<br>      print(&quot;List available tools&quot;)<br>      tools = await session.list_tools()<br>      print(f&quot;Available tools: {[tool.name for tool in tools.tools]}&quot;)<br><br>      result = await session.call_tool(&quot;weather-route-1&quot;, {&quot;query_q&quot;: &quot;Milan&quot;});<br>      print(result)<br><br><br>if __name__ == &quot;__main__&quot;:<br>  asyncio.run(main())</pre><h3>Kong MCP Server</h3><p>Now we are going to explore the other main use case implemented by the AI MCP Proxy Plugin which is to expose and protect an existing MCP Server. For the use case we are going to use two of them: Kong MCP Server and AWS Knowledge MCP Server.</p><p>The Kong MCP Server was <a href="https://konghq.com/blog/product-releases/mcp-server">introduced</a> last April, 2025 and it’s available in <a href="https://github.com/Kong/mcp-konnect">GitHub</a>. The Kong MCP Server allows you to interact with Konnect and get current configuration about your Control Planes, Kong Objects (Service, Routes, Plugins, Consumers), etc.</p><h3>Installation</h3><p>For the purpose of this blog post, you can deploy the Kong MCP Server in Kubernetes using these declarations.</p><p>Create a namespace and two secrets: one for the Konnect region and another for the your Konnect Access Token (e.g. <a href="https://developer.konghq.com/konnect-api/#konnect-api-authentication">PAT</a>)</p><pre>kubectl create namespace kong<br>kubectl create secret generic konnect-region -n kong --from-literal=region=&#39;us&#39;<br>kubectl create secret generic konnect-api-access-token -n kong --from-literal=token=&#39;&lt;YOUR_PAT&gt;&#39;</pre><p>Now submit the declaration for the Kubernetes Deployment and Service. The Services annotations are specific for Amazon EKS and <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller">AWS Load Balancer Controller</a>. Ideally, the Service should be deployed with an internal Load Balancer however, to make it consumable even without the Gateway, we are deploying it with an internet facing NLB (Network Load Balancer).</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: mcp-konnect<br>  namespace: kong<br>  labels:<br>    app: mcp-konnect<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: mcp-konnect<br>  template:<br>    metadata:<br>      labels:<br>        app: mcp-konnect<br>    spec:<br>      containers:<br>      - name: mcp-konnect<br>        image: claudioacquaviva/mcp-konnect<br>        imagePullPolicy: Always<br>        ports:<br>        - containerPort: 3001<br>        env:<br>        - name: KONNECT_REGION<br>          valueFrom:<br>            secretKeyRef:<br>              name: konnect-region<br>              key: region<br>        env:<br>        - name: KONNECT_ACCESS_TOKEN<br>          valueFrom:<br>            secretKeyRef:<br>              name: konnect-api-access-token<br>              key: token<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mcp-konnect<br>  namespace: kong<br>  annotations:<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;nlb&quot;<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-scheme&quot;: &quot;internet-facing&quot;<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-nlb-target-type&quot;: &quot;ip&quot;<br>spec:<br>  selector:<br>    app: mcp-konnect<br>  ports:<br>  - protocol: TCP<br>    port: 3001<br>    targetPort: 3001<br>type: LoadBalancer</pre><h3>Proxy MCP requests with MCP Gateway</h3><p>Once you have the Kong MCP Server deployed, you can start consuming it just like you did with the MCP Server we crafted before. However, we want this MCP Server to sit behind the MCP Gateway and get controlled by it.</p><p>That’s where the second AI MCP Proxy use case comes in: to proxy MCP requests to existing MCP Server. The architecture is illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*frOWwOFq0SQx2RIx" /></figure><p>Here’s our new decK declaration:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - mcp<br>_konnect:<br>  control_plane_name: kong-aws<br>services:<br>- name: mcp-service<br>  url: http://mcp-konnect.kong:3001/mcp<br>  routes:<br>  - name: mcp-route<br>    paths:<br>    - /mcp<br>    plugins:<br>    - name: ai-mcp-proxy<br>      enabled: true<br>      config:<br>        mode: passthrough-listener</pre><p>The declaration creates a new Kong Gateway Service based on the Kong MCP Server URL. In this case, the Kong MCP Server was deployed in the same Kubernetes as the Kong Data Plane so we can use the Kong MCP Server’s Kubernetes FQDN to connect to it. As expected the Route is reachable with the “/mcp” path.</p><p>The AI MCP Proxy plugin is one more time configured, but this time with the config mode as “passthrough-listener”, meaning the MCP Gateway will listen for incoming MCP requests and proxies them to the upstream URL of the Gateway Service (http://mcp-konnect.kong:30001/mcp). The main benefit is the MCP observability metrics generation for traffic.</p><h3>MCP Inspector</h3><p>After submitting the declaration, we can check the MCP Server. This time we’ll use the <a href="https://modelcontextprotocol.io/docs/tools/inspector">MCP Inspector</a> tool provided by the MCP project. You can run it as a container in your <a href="https://github.com/modelcontextprotocol/inspector">Docker or Podman</a> environment. For example, for Podman, run the following command:</p><pre>podman run -d --name mcp-inspector \<br>  -e HOST=0.0.0.0 \<br>  -e &quot;ALLOW_ORIGINS=http://localhost:6274&quot; \<br>  -e &quot;PUBLIC_URL=http://localhost:6274&quot; \<br>  -p 6274:6274 \<br>  -p 6277:6277 \<br>  ghcr.io/modelcontextprotocol/inspector:latest</pre><p>If you want to stop it and delete it run:</p><pre>podman stop mcp-inspector<br>podman container rm -v mcp-inspector<br>podman image rm ghcr.io/modelcontextprotocol/inspector:latest</pre><p>Check the MCP Inspector logs and get the Session Token:</p><pre>podman logs mcp-inspector</pre><p>In MacOS, start the application with the following command. You should get redirected to the MCP Inspector home page</p><pre>open -a &quot;Google Chrome&quot; &quot;http://localhost:6274?MCP_PROXY_AUTH_TOKEN=&lt;YOUR_SESSION_TOKEN&gt;&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hvsT8Ab8XVrHRLm3" /></figure><p>For “Transport Type” choose “Streamable HTTP” and add your Kong Data Plane URL (e.g. <a href="http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com/mcp).">http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com/mcp).</a> Make sure you have set the Connection Type as “Via Proxy” so the requests go through the MCP Inspector’s MCP Proxy.</p><p>Click “Connect”. MCP Inspector should open the “Tools” box. Inside the box, click “List Tools” to see all MCP Tools exposed by the Kong MCP Server. Choose “list_control_planes” and you should see a new box with all parameters available to run the Tool. Click “Run Tool” and see the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S0kHrtikmnew87BB" /></figure><h3>Kong AI Gateway Analytics</h3><p>At the same time, Kong AI Gateway is generating observability data related to the MCP Server and Tool consumption. Here’s the Kong Gateway Service “mcp-service” Analytics landing page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mu2elpRWNNt3kQgL" /></figure><h3>AWS Knowledge MCP Server</h3><p>AWS provides a long list of <a href="https://awslabs.github.io/mcp/servers">MCP Servers</a>. A particularly interesting one is the <a href="https://awslabs.github.io/mcp/servers/aws-knowledge-mcp-server">AWS Knowledge MCP Server</a> which provides real-time access to AWS documentation.</p><p>A similar decK declaration can be used to consume it:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - aws-mcp<br>_konnect:<br>  control_plane_name: kong-aws<br>services:<br>- name: aws-mcp-service<br>  url: https://knowledge-mcp.global.api.aws<br>  routes:<br>  - name: aws-mcp-route<br>    paths:<br>    - /aws-mcp<br>    plugins:<br>    - name: ai-mcp-proxy<br>      enabled: true<br>      config:<br>        mode: passthrough-listener</pre><h3>Kong Insomnia</h3><p>This time, to consume the external MCP Server, we are going to use <a href="https://insomnia.rest/">Insomnia</a>, the AI-native API platform for developers to design and test any endpoint, including MCP client support. Read the <a href="https://insomnia.rest/features/mcp-client">documentation</a> to learn more about it.</p><p>Inside Insomnia, start a new or use an existing Project. Click on “+” for MCP Client.</p><ul><li>For the HTTP box, add the Kong AI Gateway Data Plane URL for the AWS MCP Server we just defined. E.g.: <a href="http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com/aws-mcp">http://k8s-kong-kongkong-fae959197b-6192eafba1fb8e62.elb.us-east-2.amazonaws.com/aws-mcp</a></li><li>Click “Connect”. You should see on the list of available Tools on your left. Choose “aws__read_documentation.</li><li>In the middle pane, inside the “url” box type: “https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html” or any AWS documentation link. Click “Call Tool”.</li></ul><p>You should be able to see the response inside the “Preview” tile:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*coqF0Ad8C1Xl5AaY" /></figure><h3>Kong AI Gateway Analytics Explorer</h3><p>Return to Kong AI Gateway Analytics options and choose “Explorer”. Change the time period to “Last 15 minutes”, and show “Request Count” per “10 seconds” by “Gateway service”. You should see a diagram showing the consumption of both MCP Tools</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oN5bPLOkLOtv2oDu" /></figure><h3>Conclusion</h3><p>In this blog post, we explored how Kong MCP Gateway can serve as a powerful bridge between AI agents and enterprise systems by exposing internal capabilities through the Model Context Protocol. With Kong’s flexible plugin ecosystem and first-class support for MCP, it becomes straightforward to integrate tools, data sources, and downstream APIs in a secure and scalable way. Beyond simple tool invocation, Kong MCP Gateway enables advanced agentic patterns such as multi-step workflows, contextual enrichment, and safe function calling, while maintaining strong governance over every interaction.</p><p>Just like we do with traditional API workloads, the MCP Gateway can be combined with the rich set of Kong plugins to strengthen and enhance AI-based use cases. For example, you can protect MCP tool endpoints using the MCP OAuth2 Plugin.</p><p>To explore the full set of capabilities available for building MCP-enabled architectures, visit the <a href="https://konghq.com/products/kong-ai-gateway">Kong AI Gateway</a> product page, or check out the Kong + AWS partnership hub at <a href="https://konghq.com/partners/aws">https://konghq.com/partners/aws</a> to learn more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13420f610ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Voice Agents with Kong AI Gateway and Cerebras]]></title>
            <link>https://medium.com/@claudioacquaviva/ai-voice-agents-with-kong-ai-gateway-and-cerebras-ab2185f4b498?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab2185f4b498</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 02:49:16 GMT</pubDate>
            <atom:updated>2025-12-08T02:49:16.813Z</atom:updated>
            <content:encoded><![CDATA[<p>The integration of next-generation AI workloads — from large language models (LLMs) to speech-to-text and text-to-speech — demands a powerful, secure, and scalable infrastructure. This is especially true for building advanced AI voice agents, which rely on seamless, natural, and highly efficient user interaction.</p><p>This blog post explores how the combined power of Kong AI Gateway and Cerebras AI infrastructure creates an unparalleled foundation for deploying and managing these AI agents at scale. We’ll detail a reference architecture that leverages Kong AI Gateway for secure traffic control, policy enforcement, and observability, while utilizing Cerebras’ high-performance compute and LLM optimization to orchestrate STT, LLM (like the Qwen-3–32B model), and TTS models in a cohesive, production-ready solution.</p><h3>Kong Konnect</h3><p>Kong Gateway is an <a href="https://konghq.com/blog/learning-center/what-is-an-api-gateway">API gateway</a> and a core component of the <a href="https://konghq.com/products/kong-konnect">Kong Konnect</a> platform. Built on a plugin-based extensibility model, it centralizes essential functions such as proxying, routing, load balancing, and health checking, efficiently managing both microservices and traditional API traffic.</p><p>One of Kong Gateway’s greatest strengths lies in its extensible plugin ecosystem, which allows seamless integration of diverse policies and functionalities — including Authentication and Authorization, Rate Limiting, Proxy Caching, Request and Response Transformation, and Traffic Control.</p><p>Kong AI Gateway extends Konnect capabilities to the world of generative AI, including LLM models and providing a unified way to connect applications with several other infrastructures, including video, images, sound, etc.</p><p>Besides abstracting the complexity of interacting with diverse GenAI infrastructure through a single and standardized interface, it provides features such as Prompt Engineering, Semantic Processing, RAG, and MCP support. This makes it a key component for organizations building AI agents, enabling developers to experiment with models, optimize costs, and ensure compliance across all AI traffic.</p><h3>Cerebras</h3><p>Cerebras provides a cutting-edge computing platform for AI workloads. At the heart of its innovation is the Wafer-Scale Engine (WSE), a powerful processor designed to accelerate deep learning training and inference by orders of magnitude compared to traditional GPU or CPU clusters.</p><p>Beyond the hardware, Cerebras offers a complete AI supercomputing solution powered by the Cerebras Software Platform (CSoft), a platform that integrates seamlessly with existing AI frameworks, like PyTorch and TensorFlow. Cerebras also offers the <a href="https://inference-docs.cerebras.ai/quickstart">Cerebras Cloud</a>, enabling users to access its powerful AI compute infrastructure as a service, including multiple <a href="https://inference-docs.cerebras.ai/models/overview">GenAI Models</a>.</p><h3>AI Voice Agents</h3><p>An AI voice agent offers a seamless, natural, and highly efficient way for users to interact with digital systems through conversation. Typically, AI voice agents rely on speech-to-text (STT) models to convert spoken language into text while text-to-speech (TTS) models transform the agent’s textual responses back into speech.</p><p>The integration of Cerebras LLM models, STT and TTS models with Kong AI Gateway enables orchestration of AI voice agents. Developers can route audio streams to STT models for transcription, pass the resulting text through a Cerebras language model for understanding or generation, and send the output to TTS for natural speech synthesis — all governed and monitored by <a href="https://konghq.com/products/kong-ai-gateway">Kong AI Gateway</a>.</p><p>The following diagram depicts a reference architecture of the AI voice agent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*FVA4RZAL6FkTTRhh" /></figure><p>As you can see in the diagram, Kong AI Gateway abstracts all GenAI models, including Cerebras LLM as well as the STT and TTS models. Combined with the extensive list of AI-based capabilities like Prompt Decorator, Semantic Caching, etc. Kong AI Gateway provides an easy to use and monitor infrastructure ideal for AI Agents development.</p><h3>Kong AI Gateway and Cerebras at work</h3><p>It’s easier to see an AI voice agent consuming Kong AI Gateway, Cerebras LLM, and STT/TTS models than reading about it. The following video demonstrates a simple AI voice agent following the reference architecture:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsoJK2onJwY8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsoJK2onJwY8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FsoJK2onJwY8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9666ac3407b90ca79a6ce462971616ac/href">https://medium.com/media/9666ac3407b90ca79a6ce462971616ac/href</a></iframe><p>The AI Agent was written with <a href="https://livekit.io/">LiveKit</a> which provides the infrastructure for capturing, transmitting, and managing bi-directional audio streams between users and the AI agent.</p><p>From the AI Agent perspective, all GenAI models are abstracted by Kong AI Gateway, Each model is exposed to the AI Agent through a specific Kong AI Gateway Route.</p><p>Here’s the snippet of the AI Voice Agent defining the Agent Session and referring the Kong AI Gateway Routes:</p><pre>session = AgentSession(<br>  stt=openai.STT(<br>    model=&quot;Systran/faster-distil-whisper-small.en&quot;,<br>    base_url = os.environ.get(&quot;DATA_PLANE_URL&quot;) + &quot;/speaches-route-stt&quot;,<br>    api_key=&quot;dummy&quot;<br>  ),<br>  llm=openai.LLM(<br>    model=&quot;qwen-3–32b&quot;,<br>    base_url = os.environ.get(&quot;DATA_PLANE_URL&quot;) + &quot;/cerebras-route-llm&quot;,<br>    api_key=&quot;dummy&quot;<br>  ),<br>  tts=openai.TTS(<br>    model=&quot;speaches-ai/Kokoro-82M-v1.0-ONNX-int8&quot;,<br>    base_url = os.environ.get(&quot;DATA_PLANE_URL&quot;) + &quot;/speaches-route-tts&quot;,<br>    voice=&quot;af_alloy&quot;,<br>    api_key=&quot;dummy&quot;<br>  ),<br>  vad=silero.VAD.load(),<br>  turn_detection=MultilingualModel(),<br>)</pre><p>The DATA_PLANE_URL variable is where the Kong AI Gateway Data Plane is located, The AI Agent sends requests to the Gateway using the Routes defined for each model.</p><p>At the same time, here’s the Kong AI Gateway declaration defining the Gateway Services exposed by the Kong Routes:</p><pre>services:<br>- name: cerebras-service<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: cerebras-route-llm<br>    paths:<br>    - /cerebras-route-llm<br>  plugins:<br>  - name: ai-proxy-advanced<br>    instance_name: ai-proxy-advanced-llm<br>    config:<br>      genai_category: text/generation<br>      targets:<br>      - model:<br>        provider: openai<br>        options:<br>          upstream_url: &quot;https://api.cerebras.ai/v1/chat/completions&quot;<br>        route_type: llm/v1/chat<br>        auth:<br>          header_name: Authorization<br>          header_value: Bearer ${{ env &quot;DECK_CEREBRAS_API_KEY&quot; }}<br>        logging:<br>          log_payloads: true<br>          log_statistics: true<br>- name: speaches-service-tts<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: speaches-route-tts<br>    paths:<br>    - /speaches-route-tts<br>  plugins:<br>  - name: ai-proxy-advanced<br>    instance_name: ai-proxy-advanced-speaches-tts<br>    config:<br>      genai_category: audio/speech<br>      targets:<br>      - model:<br>        provider: openai<br>        options:<br>          upstream_url: &quot;http://speaches.speaches:8000/v1/audio/speech&quot;<br>        route_type: audio/v1/audio/speech<br>        auth:<br>          header_name: Authorization<br>          header_value: Bearer dummy<br>      max_request_body_size: 2097152<br>- name: speaches-service-stt<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: speaches-route-stt<br>    paths:<br>    - /speaches-route-stt<br>  plugins:<br>  - name: ai-proxy-advanced<br>    instance_name: ai-proxy-advanced-speaches-stt<br>    config:<br>    genai_category: audio/transcription<br>    targets:<br>    - model:<br>      provider: openai<br>      options:<br>        upstream_url: &quot;http://speaches.speaches:8000/v1/audio/transcriptions&quot;<br>      route_type: audio/v1/audio/transcriptions<br>      auth:<br>        header_name: Authorization<br>        header_value: Bearer dummy<br>    max_request_body_size: 2097152</pre><p>Again, for each GenAI model, there’s a configuration describing how the AI Gateway can integrate with them. Note the Gateway is taking care of the Cerebras’ API Key, providing a more secure environment for Agent development.</p><p>The STT and TTS models, referred to by the AI Agent, are deployed in the <a href="http://speaches.ai/">Speaches.AI</a> engine and exposed through the specific URLs. The LLM model, as expected, is totally managed by Cerebras Cloud.</p><h3>Observability</h3><p>Both Cerebras and Konnect provide observability capabilities. For example, here’s a Cerebras screenshot with the Qwen-3–32B model consumption used by the AI Voice Agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rMkGVVfltpwmH00g" /></figure><p>Similarly, Konnect provides ready-to-use dashboards and explorer capabilities to monitor how the models are getting consumed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ij2SE5Ve_FY6vGeD" /></figure><h3>Conclusion</h3><p>The integration of Kong AI Gateway with Cerebras AI infrastructure creates a powerful foundation for deploying and managing next-generation AI workloads at scale. Kong AI Gateway provides a secure, high-performance entry point for managing APIs and AI model endpoints, ensuring efficient traffic control, observability, and policy enforcement across diverse environments. Combined with Cerebras’ large-scale compute capabilities and LLM optimization, this architecture enables seamless orchestration of advanced AI services such as speech-to-text (STT), text-to-speech (TTS), and language understanding models.</p><p>Contact sales@konghq.com and sales@cerebras.com if you have questions or need support.</p><h3>Call to Action</h3><ul><li>For more information on setting up the integration with Kong/Cerebras view technical blog post here.</li><li>Register to both <a href="https://konghq.com/products/kong-konnect/register">Kong Konnect</a> and <a href="https://cloud.cerebras.ai/">Cerebras</a> to get a trial and start experimenting with both technologies.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab2185f4b498" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Konnect, Kong Serverless Data Plane and Amazon Cognito]]></title>
            <link>https://medium.com/@claudioacquaviva/konnect-kong-serverless-data-plane-and-amazon-cognito-4b29e3d76b3f?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b29e3d76b3f</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-cognito]]></category>
            <category><![CDATA[kong-konnect]]></category>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 09:41:55 GMT</pubDate>
            <atom:updated>2025-09-29T10:27:01.186Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This blog post is an update for the original one,<a href="https://medium.com/@claudioacquaviva/openid-connect-with-kong-ingress-controller-and-amazon-cognito-252506e5bc94"> “OpenID Connect with Kong Ingress Controller and Amazon Cognito”</a>, I wrote two years ago.</p><p>Basically, the main motivations for this update are:</p><ol><li>Although the original blob post remains a good reference for Kong Ingress Controller (KIC) based deployments, Kong Konnect has become the main Kong offering for hybrid deployments, including:</li></ol><ul><li>Self-managed Data Planes</li><li>Kong Ingress Controller which allows you to run Kong Gateway as a Kubernetes Ingress.</li><li>Dedicated Cloud Gateways — Data Plane nodes that are fully managed by Kong</li><li>Serverless gateways — lightweight API gateways, also managed by Kong</li></ul><p>Here’s a screen shot showing the four options to run Kong in Konnect:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KAtfWFdRcubtFBL5FWYPAQ.png" /></figure><p>2. The second motivation is related to the new Amazon Cognito UI. Differently to the processes described in the first post, the UI has changed significantly which deserves an update on the specific instructions on how to create an Identity Provider for both grants: Authorization Code and Client Credentials.</p><p>Amazon Cognito also provides a new generation for <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-managed-login.html">User Pools managed login</a> which allows you to implement advanced authentication processes. Also the new managed login provides a branding editor to customize the logos, backgrounds and styles.</p><h3>Konnect Control Plane and decK</h3><p>Since this update is based on Konnect, we are going to configure the Control Plane using <a href="https://developer.konghq.com/deck/">decK</a> (declarations for Kong), a command line that allows you to manage Kong Objects in a declarative way.</p><h3>Konnect Serverless Data Planes</h3><p>Serverless gateways are lightweight API gateways. Their Control Plane is hosted by Konnect and Data Plane nodes are automatically provisioned.</p><p>When you create a serverless gateway, Konnect creates a Control Plane that is hosted by Konnect. Then, a hosted Data Plane is provisioned automatically and configured to connect to the Control Plane.</p><h3>Konnect Control Plane creation</h3><p>Currently, if you register to Konnect, you get not just a Serverless Control Plane but also an actual Data Plane provisioned. You don’t need to do anything else to start defining your APIs, policies and consuming them.</p><p>Click “Gateways” inside the “API Gateway” menu option to see the “Gateway Manager” page with your first Serverless Control Plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3W6uYM1oYpxYgIjTEMwzA.png" /></figure><p><em>Kong Konnect Gateway Manager</em></p><p>If you want to create another Control Plane, Serverless or any other kind, click the “+ New gateway” button. Since the Serverless Control Plane is available, we are going to use it, so click on the “serverless-default” link. You should see the “Overview” page of your Serverless Control Plane. Something similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O8_rn2WmXLrVc7R8r85Aaw.png" /></figure><p><em>Kong Konnect Control Plane</em></p><h4>Consume the Data Plane</h4><p>Create an environment variable with the value of your Proxy URL. That’s the URL your Data Plane is located. Send a request to your Serverless Data Plane</p><pre>export DATA_PLANE_URL=&lt;YOUR_DATA_PLANE_URL&gt;<br>curl $DATA_PLANE_URL</pre><p>Expected result</p><pre>{<br>  &quot;message&quot;:&quot;no Route matched with those values&quot;,<br>  &quot;request_id&quot;:&quot;84fac2649eb6ae01f4d920115a4df70d&quot;<br>}</pre><h3>Personal Access Token</h3><p><a href="https://developer.konghq.com/deck/">decK</a> requires a <a href="https://developer.konghq.com/konnect-api/#personal-access-tokens">Konnect Personal Access Token (PAT)</a> to manage your Control Plane. To generate your PAT, click on your initials in the upper right corner of the Konnect home page, then select “Personal Access Tokens”. Click on “+ Generate Token”, name your PAT, set its expiration time, and be sure to copy and save it in an environment variable, as Konnect won’t display it again.</p><p>Check your communication sending a basic request to Konnect:</p><pre>export PAT=&lt;YOUR_PAT&gt;<br>deck gateway ping --konnect-control-plane-name serverless-default --konnect-token $PAT</pre><h3>Client Credentials</h3><p>The main use cases for the Client Credentials grant is to address application authentication (and authorization) rather than user authentication. Is such a scenario, authentication processes based on userid and password are not feasible. Instead, applications should deal with Client IDs and Client Secrets to authenticate and get a token.</p><h4>Cognito instance — Amazon Console</h4><p>The following process describes how to create a minimal Amazon Cognito User Pool, using its new console, supporting the Client Credentials Grant. Check the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html">Amazon Cognito documentation</a> to learn more.</p><ol><li>Go to the Amazon Cognito console. If prompted, enter your AWS credentials.</li></ol><p>2. Choose “User pools” and click &quot;Create user pool&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0enG9R5PxFFgkGJ9mtshnA.png" /></figure><p>3. Make sure you have &quot;Application Type&quot; set as &quot;Traditional Web Application&quot;.</p><p>4. Name your application as &quot;app1&quot;.</p><p>5. Inside &quot;Configure options&quot; box, set &quot;Email&quot; for &quot;Options for sign-in identifiers&quot;.</p><p>6. Click &quot;Create user directory&quot;.</p><p>7. In the &quot;Set up resources for your application&quot; page click &quot;Go to overview&quot;.</p><p>8. In the &quot;Overview&quot; page click &quot;Rename&quot; and rename your User Pool as &quot;kongpool&quot;.</p><p>9. Take note of the “User pool ID” (e.g. us-east-2_81wYTJy0D) and save it.</p><p>10. Click &quot;Domain&quot; inside &quot;Branding&quot; menu option. Click &quot;Create resource server&quot;.</p><p>11. On the “Create resource server” page</p><ul><li>Type “kongresource” for both “Resource server name” and “Resource server identifier” fields.</li><li>Click on “Add custom scope”.</li><li>Type “scope1” for both “Scope name” and “Description” fields.</li><li>Click on “Create resource server”</li></ul><p>12. Click “App clients” inside the “Applications” menu option and choose you “app1” application client. Copy both “Client ID” and “Client secret” and save them.</p><p>13. Click on the “Login pages” tab. Inside “Managed login pages configuration” tile click “Edit”.</p><ul><li>For “OAuth 2.0 grant types” remove “Authorization code grant” and add “Client credentials”.</li><li>For “Custom scopes”, choose the &quot;kongresource/scope1&quot; scope previously created.</li><li>Click &quot;Save changes&quot;</li></ul><h4>Cognito instance — Amazon CLI</h4><p>You can create the Cognito User Pool using the <a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cognito-idp/index.html">CLI</a> instead.</p><p><strong>Create the User Pool first</strong></p><pre>aws cognito-idp create-user-pool --pool-name kongpool --username-attributes &quot;email&quot; --region us-east-2</pre><p>You can check it with:</p><pre>% aws cognito-idp list-user-pools --max-results 1 --region us-east-2<br>{<br>    &quot;UserPools&quot;: [<br>        {<br>            &quot;Id&quot;: &quot;us-east-2_yof8aO5Gj&quot;,<br>            &quot;Name&quot;: &quot;kongpool&quot;,<br>            &quot;LambdaConfig&quot;: {},<br>            &quot;LastModifiedDate&quot;: &quot;2025-09-26T15:52:37.842000-03:00&quot;,<br>            &quot;CreationDate&quot;: &quot;2025-09-26T15:52:37.842000-03:00&quot;<br>        }<br>    ]<br>}</pre><p>Get the User Pool id:</p><pre>export USER_POOL_ID=$(aws cognito-idp list-user-pools --max-results 1 --region us-east-2 | jq -r &#39;.UserPools[0].Id&#39;)</pre><p>If you want to delete it run:</p><pre>aws cognito-idp delete-user-pool --user-pool-id $USER_POOL_ID --region us-east-2</pre><p><strong>Create a Resource for the Scope</strong></p><pre>aws cognito-idp create-resource-server --user-pool-id $USER_POOL_ID --identifier &quot;kongresource&quot; --name &quot;kongresource&quot; --scopes &quot;ScopeName=scope1,ScopeDescription=scope1&quot; --region us-east-2</pre><p>If you want to delete it:</p><pre>aws cognito-idp delete-resource-server --user-pool-id $USER_POOL_ID --identifier &quot;kongresource&quot; --region us-east-2</pre><p><strong>Create the User Pool Domain</strong></p><pre>aws cognito-idp create-user-pool-domain --user-pool-id $USER_POOL_ID --domain &quot;kongdomain&quot; --region us-east-2</pre><p>The full domain name will be `https://kongdomain.auth.&lt;aws-region&gt;.amazoncognito.com`.</p><p><strong>Create the User Pool Client</strong></p><pre>aws cognito-idp create-user-pool-client \<br>--user-pool-id $USER_POOL_ID \<br>--client-name &quot;kongapp&quot; \<br>--generate-secret \<br>--allowed-o-auth-flows-user-pool-client \<br>--allowed-o-auth-flows &quot;client_credentials&quot; \<br>--allowed-o-auth-scopes &quot;kongresource/scope1&quot; \<br>--region us-east-2<br><br>{<br>    &quot;UserPoolClient&quot;: {<br>        &quot;UserPoolId&quot;: &quot;us-east-2_yof8aO5Gj&quot;,<br>        &quot;ClientName&quot;: &quot;kongapp&quot;,<br>        &quot;ClientId&quot;: &quot;4u3vgbkq2hvtdtehsvd0srdb3v&quot;,<br>        &quot;ClientSecret&quot;: &quot;sv6bpoltau107k7t2vi0q0j514edji3fs8fassb4m499lbnhq6i&quot;,<br>        &quot;LastModifiedDate&quot;: &quot;2025-09-26T15:58:08.279000-03:00&quot;,<br>        &quot;CreationDate&quot;: &quot;2025-09-26T15:58:08.279000-03:00&quot;,<br>        &quot;RefreshTokenValidity&quot;: 30,<br>        &quot;TokenValidityUnits&quot;: {},<br>        &quot;AllowedOAuthFlows&quot;: [<br>            &quot;client_credentials&quot;<br>        ],<br>        &quot;AllowedOAuthScopes&quot;: [<br>            &quot;kongresource/scope1&quot;<br>        ],<br>        &quot;AllowedOAuthFlowsUserPoolClient&quot;: true,<br>        &quot;EnableTokenRevocation&quot;: true,<br>        &quot;EnablePropagateAdditionalUserContextData&quot;: false,<br>        &quot;AuthSessionValidity&quot;: 3<br>    }<br>}</pre><p>Get the Client ID:</p><pre>export COGNITO_CLIENT_ID=$(aws cognito-idp list-user-pool-clients --user-pool-id $USER_POOL_ID --region us-east-2 | jq -r &#39;.UserPoolClients[0].ClientId&#39;)</pre><p>Get the Client Secret:</p><pre>export COGNITO_CLIENT_SECRET=$(aws cognito-idp describe-user-pool-client --user-pool-id $USER_POOL_ID -=client-id $COGNITO_CLIENT_ID --region us-east-2 | jq -r &#39;.UserPoolClient.ClientSecret&#39;)</pre><p>If you want to delete it:</p><pre>aws cognito-idp delete-user-pool-client --user-pool-id $USER_POOL_ID --client-id $COGNITO_CLIENT)ID --region us-east-2</pre><h4>Get the Issuer</h4><p>To configure the OpenID Connect plugin we need to know the User Pool Issuer endpoint. The Issuer endpoint follows the structure:</p><p>`https://cognito-idp.&lt;aws-region&gt;.amazonaws.com/&lt;user-pool-id&gt;/.well-known/openid-configuration`.</p><p>You can hit it if you will. Use the User Pool Id you copied before.</p><pre>http https://cognito-idp.us-east-2.amazonaws.com/$USER_POOL_ID/.well-known/openid-configuration<br><br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Length: 891<br>Content-Type: application/json<br>Date: Fri, 26 Sep 2025 19:05:53 GMT<br>x-amzn-RequestId: 6bd301b6-fa44-4242-86cd-9dbf9c938de1<br><br>{<br>    &quot;authorization_endpoint&quot;: &quot;https://kongdomain.auth.us-east-2.amazoncognito.com/oauth2/authorize&quot;,<br>    &quot;end_session_endpoint&quot;: &quot;https://kongdomain.auth.us-east-2.amazoncognito.com/logout&quot;,<br>    &quot;id_token_signing_alg_values_supported&quot;: [<br>        &quot;RS256&quot;<br>    ],<br>    &quot;issuer&quot;: &quot;https://cognito-idp.us-east-2.amazonaws.com/us-east-2_yof8aO5Gj&quot;,<br>    &quot;jwks_uri&quot;: &quot;https://cognito-idp.us-east-2.amazonaws.com/us-east-2_yof8aO5Gj/.well-known/jwks.json&quot;,<br>    &quot;response_types_supported&quot;: [<br>        &quot;code&quot;,<br>        &quot;token&quot;<br>    ],<br>    &quot;revocation_endpoint&quot;: &quot;https://kongdomain.auth.us-east-2.amazoncognito.com/oauth2/revoke&quot;,<br>    &quot;scopes_supported&quot;: [<br>        &quot;openid&quot;,<br>        &quot;email&quot;,<br>        &quot;phone&quot;,<br>        &quot;profile&quot;<br>    ],<br>    &quot;subject_types_supported&quot;: [<br>        &quot;public&quot;<br>    ],<br>    &quot;token_endpoint&quot;: &quot;https://kongdomain.auth.us-east-2.amazoncognito.com/oauth2/token&quot;,<br>    &quot;token_endpoint_auth_methods_supported&quot;: [<br>        &quot;client_secret_basic&quot;,<br>        &quot;client_secret_post&quot;<br>    ],<br>    &quot;userinfo_endpoint&quot;: &quot;https://kongdomain.auth.us-east-2.amazoncognito.com/oauth2/userInfo&quot;<br>}</pre><h4>decK declaration</h4><p>Before submitting the declaration, set the following environment variables:</p><ul><li>COGNITO_CLIENT_ID and COGNITO_CLIENT_SECRET with the values you copied from the “app1” application client page.</li><li>ISSUER with the issuer returned by the request you sent to your Cognito User Pool. The ISSUER env variable should follow the format:</li></ul><pre>export ISSUER=https://cognito-idp.&lt;aws-region&gt;.amazonaws.com/&lt;user-pool-id&gt;/.well-known/openid-configuration</pre><p>The following decK declaration protects the<a href="http://httpbin.konghq.com/"> http://httpbin.konghq.com:80</a> echo service with the OpenID Connect plugin.</p><pre>echo &quot;_format_version: \&quot;3.0\&quot;<br>_konnect:<br> control_plane_name: serverless-default<br>_info:<br> select_tags:<br> - httpbin-service-route<br>services:<br>- name: httpbin-service<br>  url: http://httpbin.konghq.com:80<br>  routes:<br>  - name: oidc-route<br>    paths:<br>    - /oidc-route<br>    plugins:<br>    - name: openid-connect<br>      instance_name: openid-connect1<br>      config:<br>        auth_methods:<br>        - client_credentials<br>        scopes: [kongresource/scope1]<br>        issuer: $ISSUER&quot; &gt; ./oidc.yaml</pre><p>Submit the declaration with:</p><pre>deck gateway sync --konnect-token $PAT oidc.yaml</pre><p><strong>Consume the Kong Route</strong></p><p>We need to provide a client id and secret to consume the route. If we provide wrong credentials we get a specific error:</p><pre>% http $DATA_PLANE_URL/oidc-route/get -a 1:1<br>HTTP/1.1 401 Unauthorized<br>Connection: keep-alive<br>Content-Length: 26<br>Content-Type: application/json; charset=utf-8<br>Date: Fri, 19 Sep 2025 12:30:41 GMT<br>Server: kong/3.11.0.1-enterprise-edition<br>WWW-Authenticate: Bearer realm=&quot;cognito-idp.us-east-2.amazonaws.com&quot;, error=&quot;invalid_token&quot;<br>X-Kong-Request-Id: d042dc3114ba8c5fe72ba35cfc593913<br>X-Kong-Response-Latency: 434<br><br>{<br>   &quot;message&quot;: &quot;Unauthorized&quot;<br>}</pre><p>Use the client id and secret, separated by a “:”, that Cognito issued previously to send another request. For example:</p><pre>% http $DATA_PLANE_URL/oidc-route/get -a $COGNITO_CLIENT_ID:$COGNITO_CLIENT_SECRET<br>HTTP/1.1 200 OK<br>Access-Control-Allow-Credentials: true<br>Access-Control-Allow-Origin: *<br>Connection: keep-alive<br>Content-Length: 1461<br>Content-Type: application/json<br>Date: Fri, 19 Sep 2025 12:34:02 GMT<br>Server: gunicorn/19.9.0<br>Via: 1.1 kong/3.11.0.1-enterprise-edition<br>X-Kong-Proxy-Latency: 1<br>X-Kong-Request-Id: 445c3178425c5a761c67f73c149f0d2a<br>X-Kong-Upstream-Latency: 618<br><br>{<br>   &quot;args&quot;: {},<br>   &quot;headers&quot;: {<br>       &quot;Accept&quot;: &quot;*/*&quot;,<br>       &quot;Accept-Encoding&quot;: &quot;gzip, deflate&quot;,<br>       &quot;Authorization&quot;: &quot;Bearer eyJraWQiOiJGQ2Q3YWlUSzF5UUphXC9RYzZPRlFNOHRtSUxocU9TeUQyejhUWUlMc1Z0QT0iLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIxNTA1aGJyZXN0ZGc5cjFqbThsa2NqNm80ayIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoia29uZ3Jlc291cmNlXC9zY29wZTEiLCJhdXRoX3RpbWUiOjE3NTgyODUwOTUsImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzgxd1lUSnkwRCIsImV4cCI6MTc1ODI4ODY5NSwiaWF0IjoxNzU4Mjg1MDk1LCJ2ZXJzaW9uIjoyLCJqdGkiOiJkNDJiYzAzZi02ZmQ1LTQ4OWItODBlMS02MmViN2YwODcxY2UiLCJjbGllbnRfaWQiOiIxNTA1aGJyZXN0ZGc5cjFqbThsa2NqNm80ayJ9.DYffEK2_kF6HV38Qr0aW79pebq3RNasxgUonOlNDPmMQI2N0i5tL-evsmDPuVbtw432V4MEbMsIPyWVsxHsPsAXuHS4O82vhHtGzAGLyXBvniTvyXbypoofsbUVykxMo0hyT1YxcDm41prh8qeaXkEzFAvCpOjqqESWe3GfVywNRjbRgrNpIHLUi9l6LoFIIfPTiJlUZFp8lmNIMt4MDOMEbjhsuzhqyFpD1yTjmMiOi91TP0oRySvvR9g-iNXMEBKPeYzzIT9xWkZ1Sibb6BPbh_7F9r6A2FwMC2JTQwx3_UIH5B1_6Cs3chNNhOWbCYJenXuTKckzAR9he3IQ2Dw&quot;,<br>       &quot;Host&quot;: &quot;httpbin.org&quot;,<br>       &quot;User-Agent&quot;: &quot;HTTPie/3.2.4&quot;,<br>       &quot;X-Amzn-Trace-Id&quot;: &quot;Root=1-68cd4dba-1c196f6045b8ceb3247f2521&quot;,<br>       &quot;X-Forwarded-Host&quot;: &quot;1d1615c5ad.serverless.gateways.konggateway.com&quot;,<br>       &quot;X-Forwarded-Path&quot;: &quot;/oidc-route/get&quot;,<br>       &quot;X-Forwarded-Prefix&quot;: &quot;/oidc-route&quot;,<br>       &quot;X-Kong-Request-Id&quot;: &quot;445c3178425c5a761c67f73c149f0d2a&quot;<br>   },<br>   &quot;origin&quot;: &quot;186.204.54.49, 3.214.24.226&quot;,<br>   &quot;url&quot;: &quot;http://1d1615c5ad.serverless.gateways.konggateway.com/get&quot;<br>}</pre><p>You can check the Access Token issued by Cognito with:</p><pre>http $DATA_PLANE_URL/oidc-route/get -a $COGNITO_CLIENT_ID:$COGNITO_CLIENT_SECRET | jq -r &quot;.headers.Authorization&quot; | cut -d &#39; &#39; -f 2 | jwt decode -j -<br>{<br> &quot;header&quot;: {<br>   &quot;alg&quot;: &quot;RS256&quot;,<br>   &quot;kid&quot;: &quot;FCd7aiTK1yQJa/Qc6OFQM8tmILhqOSyD2z8TYILsVtA=&quot;<br> },<br> &quot;payload&quot;: {<br>   &quot;auth_time&quot;: 1758285095,<br>   &quot;client_id&quot;: &quot;1505hbrestdg9r1jm8lkcj6o4k&quot;,<br>   &quot;exp&quot;: 1758288695,<br>   &quot;iat&quot;: 1758285095,<br>   &quot;iss&quot;: &quot;https://cognito-idp.us-east-2.amazonaws.com/us-east-2_81wYTJy0D&quot;,<br>   &quot;jti&quot;: &quot;d42bc03f-6fd5-489b-80e1-62eb7f0871ce&quot;,<br>   &quot;scope&quot;: &quot;kongresource/scope1&quot;,<br>   &quot;sub&quot;: &quot;1505hbrestdg9r1jm8lkcj6o4k&quot;,<br>   &quot;token_use&quot;: &quot;access&quot;,<br>   &quot;version&quot;: 2<br> }<br>}</pre><h3>Authorization Code</h3><p>The Authorization Code grant is used by web and native apps to authenticate user. In order to allow users to provide their credentials (userid/pwd, OTP, etc.), the grant should provide integration with the IdP UI.</p><h4>Cognito instance — Amazon Console</h4><p>Just like we did for the Client Credentials grant, the following process describes how to create a minimal Amazon Cognito User Pool for the Authorization Code Grant. Check the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html">Amazon Cognito documentation</a> to learn more.</p><ol><li>Go to the Amazon Cognito console. If prompted, enter your AWS credentials.</li><li>Delete the existing “kongpool” user pool.</li><li>Choose “Create user pool”.</li><li>Make sure you have “Application Type” set as “Traditional Web Application”.</li><li>Name your application as “app1”.</li><li>Inside “Configure options” box, set “Email” for “Options for sign-in identifiers”.</li></ol><p>7. Click “Create user directory”.</p><p>8 In the “Set up resources for your application” page click “Go to overview”.</p><p>9. In the “Overview” page click “Rename” and rename your User Pool as “kongpool”.</p><p>10. Take note of the “User pool ID” (e.g. us-east-2_htCeOrw2T) and save it.</p><p>11. Click “App clients” inside the “Applications” menu option and choose your “app1” application client. Copy both “Client ID” and “Client secret” (e.g. 6mcl15q9q823f655i5an6p2ggh and 19d46niik7ctpf3oenbl3mb7et4u3iek8usqsem70vhr191vj0sp) and save them.</p><p>12. Click on the “Login pages” tab. Inside “Managed login pages configuration” tile click “Edit”.</p><ul><li>The URL field in “Allowed callback URLs” section is used by Cognito to control the incoming URL where the Authentication process got started. After authentication, Cognito redirects the use to this URL. This should be set with the application’s URL where an authenticated user is required. In order case, since we’re going to protect the same ingress, it should be set as `https://&lt;_DataPlane_Proxy_URL_&gt;/oidc-route/get`. Note that Cognito requires a https based URL.</li><li>“OAuth 2.0 grant types”: it’s already set to support the Authorization Code Grant.</li><li>“OpenID Connect scopes”: it’s also already set with “OpenID” scope, required by the grant.</li><li>Accept all settings and click “Save changes”.</li></ul><h4>Cognito instance — Amazon CLI</h4><p>Again, you can use the CLI to create both User and Identity Pools:</p><p><strong>Cognito User Pool</strong></p><p>From a local terminal, start creating the User Pool with standard settings. Please check the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/getting-started-with-cognito-user-pools.html">Amazon Cognito User Pool</a> documentation to learn more.</p><pre>aws cognito-idp create-user-pool \<br>--pool-name kongpool \<br>--username-attributes &quot;email&quot; \<br>--email-configuration=EmailSendingAccount=COGNITO_DEFAULT \<br>--auto-verified-attributes &quot;email&quot; \<br>--user-attribute-update-settings=AttributesRequireVerificationBeforeUpdate=email \<br>--region us-east-2</pre><p>Retrieve the User Pool Id with:</p><pre>export USER_POOL_ID=$(aws cognito-idp list-user-pools --max-results 1 --region us-east-2 | jq -r &quot;.UserPools[].Id&quot;)</pre><p><strong>Create User Pool Domain</strong></p><p>Next, we have to define a domain for our User Pool. The “managed-login-version” parameter set as “2”, tells the Cognito we want to use the new managed login capability.</p><pre>aws cognito-idp create-user-pool-domain --user-pool-id $USER_POOL_ID --domain kongdomain --managed-login-version 2 --region us-east-2</pre><p>Note that the full domain name will be `https://kongdomain.auth.&lt;aws-region&gt;.amazoncognito.com`</p><p><strong>Create User Pool Client</strong></p><p>Last, we create our first User Pool Client with “kongapp” as client name. Cognito will then create a ClientId and ClientSecret for us.</p><p>When the Cognito User Pool is available we need to login to it and get the tokens. The Callback URL parameter refers to the URL our application is use to handle the OAuth Grant, “https://&lt;_DataPlane_Proxy_URL_&gt;/oidcroute/get”.</p><pre>aws cognito-idp create-user-pool-client \<br>--user-pool-id $USER_POOL_ID \<br>--client-name kongapp \<br>--generate-secret \<br>--allowed-o-auth-flows-user-pool-client \<br>--allowed-o-auth-flows &quot;code&quot; \<br>--allowed-o-auth-scopes &quot;openid&quot; \<br>--callback-urls $DATA_PLANE_URL/oidc-route/get \<br>--supported-identity-provider &quot;COGNITO&quot; \<br>--region us-east-2</pre><p>You can retrieve both “Client ID” and “Client Secret” with:</p><pre>export COGNITO_CLIENT_ID=$(aws cognito-idp list-user-pool-clients --user-pool-id $USER_POOL_ID --region us-east-2 | jq -r &quot;.UserPoolClients[].ClientId&quot;)</pre><pre>export COGNITO_CLIENT_SECRET=$(aws cognito-idp describe-user-pool-client --user-pool-id $USER_POOL_ID --client-id $COGNITO_CLIENT_ID --region us-east-2 | jq -r &quot;.UserPoolClient.ClientSecret&quot;)</pre><p>Set the default Login Branding.</p><pre>aws cognito-idp create-managed-login-branding \<br> --user-pool-id $USER_POOL_ID \<br> --client-id $COGNITO_CLIENT_ID \<br> --use-cognito-provided-values</pre><p><strong>Cognito Identity Pool creation</strong></p><p>Now we need to create the Cognito Identity Pool. The User Pool Id and ClientId parameters are used to connect both Pools. Please check the <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/getting-started-with-identity-pools.html">Amazon Cognito Identity Pool</a> documentation to learn more.</p><pre>aws cognito-identity create-identity-pool \<br>--identity-pool-name kongidpool \<br>--no-allow-unauthenticated-identities \<br>--allow-classic-flow \<br>--cognito-identity-providers=ProviderName=cognito-idp.us-east-2.amazonaws.com/$USER_POOL_ID,ClientId=$COGNITO_CLIENT_ID,ServerSideTokenCheck=false \<br>--region us-east-2</pre><p>Just in case, retrieve the Identity Pool Id with:</p><pre>export IDENTITY_POOL_ID=$(aws cognito-identity list-identity-pools --max-results 1 --region us-east-2 | jq -r &quot;.IdentityPools[].IdentityPoolId&quot;)</pre><h4>decK declaration</h4><p>Before submitting the declaration, set the following environment variables:</p><p>. ISSUER with the issuer returned by the request you sent to your Cognito User Pool.</p><p>. COGNITO_CLIENT_ID and COGNITO_CLIENT_SECRET with the values you copied from the “app1” application client page.</p><p>The ISSUER env variable should follow the format:</p><pre>export ISSUER=https://cognito-idp.&lt;aws-region&gt;.amazonaws.com/&lt;user-pool-id&gt;/.well-known/openid-configuration</pre><p>The following decK declaration protects the<a href="http://httpbin.konghq.com/"> http://httpbin.konghq.com:80</a> echo service with the OpenID Connect plugin.</p><pre>echo &quot;_format_version: \&quot;3.0\&quot;<br>_konnect:<br> control_plane_name: serverless-default<br>_info:<br> select_tags:<br> - httpbin-service-route<br>services:<br>- name: httpbin-service<br>  url: http://httpbin.konghq.com:80<br>  routes:<br>  - name: oidc-route<br>    paths:<br>    - /oidc-route<br>    plugins:<br>    - name: openid-connect<br>      instance_name: openid-connect1<br>      config:<br>        auth_methods:<br>        - authorization_code<br>        redirect_uri:<br>        - $DATA_PLANE_URL/oidc-route/get<br>        client_id:<br>        - $COGNITO_CLIENT_ID<br>        client_secret:<br>        - $COGNITO_CLIENT_SECRET<br>        issuer: $ISSUER&quot; &gt; ./oidc.yaml</pre><p>Submit the declaration with:</p><pre>deck gateway sync --konnect-token $PAT oidc.yaml</pre><p><strong>Consume the Kong Route</strong></p><p>Redirect your browser to the URL you defined in the ingress your protecting with OpenID Connect plugin. For example:</p><pre>open -a &quot;Google Chrome&quot; &quot;$DATA_PLANE_URL/oidc-route/get&quot;</pre><p>Since you don’t have any token in your request, Kong Data Plane redirects you to the Amazon Cognito UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4_T1KX2ZV9VxC-WTRliBg.png" /></figure><p>Click on `Create an account` to register as a Cognito user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6NAzg99456JXbH4b4VGZQ.png" /></figure><p>You will receive an email with confirmation code. Enter it in the next page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SiE6EHlrkeTFfcHJ4LwsXQ.png" /></figure><p>After registration and authentication, Cognito redirects you back to Kong Data Plane with the Authorization Code injected. Kong Data Plane connects to Cognito, using Client Credentials Grant to exchange the Authorization Code with the actual Access Token. The request is routed to the Upstream based on the `http://httpbin.org` public echo system which returns all data related to the request. You can see the Access Token’s been injected by the Data Plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RlwHK4rmK2_JjOgc62xuGQ.png" /></figure><h3>Conclusion</h3><p>One of the main principles recommended to any Konnect customer is to have the Data Planes as close as possible to your Upstreams Services and Application. To support maximum flexibility, Konnect provides several options to the Data Plane deployment processes, including Self-Managed and Serverless Data Planes.</p><p>In this blog post we have used the Konnect User Interface to manage Serverless Control Plane and Data Plane. In the next blog post, I’ll explore the multiple options Kong provides to automate all flavours of Control Planes including Kong REST Admin APIs and Kong Operator (KGO) for Kubernetes deployments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b29e3d76b3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-LLM ReAct AI Agent with Kong AI Gateway 3.10 and LangGraph]]></title>
            <link>https://medium.com/@claudioacquaviva/multi-llm-react-ai-agent-with-kong-ai-gateway-3-10-and-langgraph-7429c9c1f5ee?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7429c9c1f5ee</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 12:58:32 GMT</pubDate>
            <atom:updated>2025-08-08T12:16:22.255Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>To put it simply: ChatGPT is an AI agent.</p><p>As such, ChatGPT is a type of artificial intelligence system designed to perceive natural language input, process information, and respond accordingly. Anthropic’s blog post, <a href="https://www.anthropic.com/engineering/building-effective-agents">“Building Effective Agents”</a>, has a nice and concise definition of AI Agents: “Agents are systems that leverage LLMs to autonomously coordinate their operations and tool interactions, exercising control over task execution.”</p><p>On the other hand, no single model has all the answers. A multi-GenAI Agent brings the best of multiple LLM, Audio, Video and Image models together. Instead of relying on one brain, this agent intelligently selects or coordinates between different language models like GPT-4, Claude, Mistral, and others, to deliver more accurate responses.</p><p>Ideally, we should isolate the AI Agent from the policies that drive the consumption of the models. Or, if you will, the AI Agent should rely on an external component, intelligent enough to understand what and how many models are available and, most importantly, decide which model should be called to address the incoming requests. For example, let’s say you have multiple available models, each one of them trained for specific topics like Mathematics, Classical Music, etc. Another scenario would consider digesting a variety of inputs, including texts, images and audios. Moreover, models can be added or removed from your environment along the way, making AI agent development even more challenging.</p><p>This blog post explores how developers can leverage Kong AI Gateway to implement better and smarter AI agents using <a href="https://www.langchain.com/langgraph">LangGraph</a>. We are going to focus on LLM models but keep in mind, Kong AI Gateway will support other kinds of models along the way, including audio, images, etc. Moreover, it’s important to note that LangGraph is an extensive AI Agent framework with lots of other features and capabilities. Refer to its <a href="https://langchain-ai.github.io/langgraph/">documentation portal</a> to learn more about it.</p><p>Before we get started, it’s important to make an important comment: the agents we are going to create and run are introductory ones and are not meant to be applied for production environments. The main purpose is to demonstrate how to leverage the LangGraph and Kong AI Gateway capabilities we have available when creating an AI Agent.</p><h3>MCP (Model Context Protocol)</h3><p>We have seen a lot of traction around the new protocol proposed last <a href="https://www.anthropic.com/news/model-context-protocol">November, 2024, by Anthropic</a>, called <a href="https://modelcontextprotocol.io/">MCP</a> (Model Context Protocol). At the same time Kong launched last April, 2025 the new <a href="https://konghq.com/blog/product-releases/mcp-server">Kong MCP Server</a>. MCP contributes tremendously to AI Agent developments, however, since this is an introductory blog post, we are going to exercise the inclusion of MCP in this architecture, integrating with Kong and LangGraph, in the next one.</p><h3>Blog structure</h3><p>The post is structured in following parts:</p><ul><li>Kong AI Gateway Introduction, Implementation Architecture and AI Agent fundamentals</li><li>Kong Version of a simple ReAct based AI Agent with no frameworks</li><li>LangGraph fundamentals and basic AI Agent</li><li>Tools and Function Calling with Kong AI Gateway and Observability Layer</li><li>Single LLM AI Agent with Kong AI Gateway and LangGraph</li><li>Multi LLM AI Agent</li><li>LangGraph Server</li></ul><h3>Kong AI Gateway Introduction</h3><p>In April, 2025, Kong announced Kong Gateway 3.10 with the <a href="https://konghq.com/blog/product-releases/ai-gateway-3-10">5th version</a> of Kong AI Gateway capabilities to address AI-based use cases including automated RAG, PII (Personally Identifiable Information) sanitization and load balance based on tokens and costs.</p><p>The diagram below represents the Kong AI Gateway capabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ukquJnuRl7Mj1e6z" /></figure><p><em>Kong AI Gateway functional capabilities</em></p><p>Also, from the architecture perspective, in a nutshell, the Konnect Control Plane and Data Plane nodes topology remains the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ASCcJWraybON0LFC" /></figure><p><em>The Kong AI Gateway sits in between the GenAI applications we build and the LLMs we consume. By leveraging the same underlying core of Kong Gateway, we’re reducing complexity in deploying the AI Gateway capabilities as well. And of course, it works on Konnect, Kubernetes, self-hosted, or across multiple clouds.</em></p><h3>Implementation Architecture</h3><p>The Agent implementation architecture should include components representing and responsible for the functional scope described above. The architecture comprises:</p><ul><li>Kong AI Gateway to abstract and protect:</li><li>LLM models</li><li>external functions used by the AI Agent</li><li>Mistral, Anthropic and OpenAI as LLM model infrastructures</li><li>Redis as the Vector Database</li><li>Ollama as the Embedding Model infrastructure</li><li>Observability layer with Grafana, Loki and Prometheus</li></ul><p>Kong AI Gateway, implemented as a regular Konnect Data Plane Node, Redis, Ollama and the Observability layer run on a Minikube Kubernetes Cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*kV_GK_9GuIKeDavV" /></figure><p><em>Multi-LLM ReAct AI Agent Reference Architecture</em></p><p>The artifacts used to implement the architecture are available at the following <a href="https://github.com/CAcquaviva/kong-ai-gateway-langgraph">GitHub repo</a>.</p><h3>AI Agent Architecture, Reasoning Frameworks and Prompt Engineering</h3><p>In September, 2024, Google launched a white paper, called <a href="https://www.kaggle.com/whitepaper-agents">“Agents”</a>, exploring the basics of AI Agents, including their architectures and components. The fundamental diagram, included in the paper, is a nice starting point to understand the main moving parts someone working with AI Agents should master.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kpmmp_JZyIkIJU6S" /></figure><p><em>AI Agent components</em></p><p>The diagram presents three main components of an agent:</p><ul><li>Model: the LLM model that will act as the centralized component responsible for guiding the agent’s decision-making processes.</li><li>Tools: allow Foundation Models to interact with external data and services</li><li>Orchestration: defines a recurring loop in which the agent receives input, plans, reasons and makes decisions to the Agent’s next action. Moreover, the Orchestration layer defines “Memories”, used for saving conversations.</li></ul><p>An interesting perspective is to think of agents like LLM enhanced with components and capabilities like tools, memory, reasoning, etc.</p><p>LangGraph <a href="https://langchain-ai.github.io/langgraph/agents/agents/">documentation</a> also provides a concise introduction of AI Agents and its main components.</p><h3>Reasoning Frameworks</h3><p>As you can see in the diagram, one of the main responsibilities of the Orchestration layer is controlling the reasoning process implemented by the Model. In fact, at the core of the layer there’s a loop responsible for building prompts, making decisions, calling external systems, tracking the steps that have been processed, etc. as it interacts with the Model. The Model, in turn, implements the reasoning process itself supporting a framework.</p><p>There are some Reasoning Frameworks defined the LLM models usually support:</p><ul><li><a href="https://arxiv.org/abs/2201.11903">Chain of Thought</a> (CoT): based on step-by-step reasoning, defines a linear thought to arrive at an answer. It does not implement tools nor interactions with external components.</li><li><a href="https://arxiv.org/abs/2305.10601">Tree of Thought</a> (ToT): extends CoTs with multiple possible solutions creating a tree-like structure, evaluating intermediate states to choose the branch to continue.</li><li><a href="https://arxiv.org/abs/2210.03629">ReAct</a> (Reasoning and Action): also based on CoT adding tools, external interaction and observation. Besides, it’s supported by Agent Frameworks like LangGraph. Due to all these factors, ReAct has been considered the best Reasoning Framework to implement AI Agent.</li></ul><p>In the <a href="https://research.google/blog/react-synergizing-reasoning-and-acting-in-language-models/">blog post</a> the ReAct white paper authors wrote, there’s a nice diagram comparing the ReAct Framework with others:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*MIjd-KVDEZ3_2gv_" /></figure><p><em>ReAct Framework comparison</em></p><h3>Prompt Engineering</h3><p>Prompt engineering is the practice of crafting inputs (prompts) to get the most useful, accurate, or creative responses from LLM models.</p><p>It’s critical for AI Agents since you shape how the agents should communicate, think, act, interact with tools, reason and respond. So, the better Prompt Engineering you have the better AI Agent you are going to get.</p><p>The same blog post also includes a nice comparison of four prompting methods to respond to a <a href="https://arxiv.org/abs/1809.09600">HotpotQA</a>. HotpotQA is a question answering dataset designed for multi-hop reasoning. You can learn more about it in the <a href="https://hotpotqa.github.io/">HotpotQA GitHub repository</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/0*QphrJ3DyrCekYVjx" /></figure><p><em>Prompt Engineering comparison</em></p><h3>Simple ReAct based AI Agent</h3><p>With all these new elements in place, it’s time to exercise them. The following code implements a simple ReAct based AI Agent written in Python, including a reasoning loop only, without an Agent Framework or tools interacting with external environments. The AI Agent uses <a href="https://platform.openai.com/docs/api-reference/chat?lang=python">OpenAI’s Chat Completions API</a> to consume the GPT-4 LLM model.</p><p>As we have said, the prompt is critical to drive how the Agent should behave. As you can see, ours instructs the Agent with a ReAct pattern including the format it should follow as well as requesting to present the logical steps, with thoughts and observations, used to achieve the final answer.</p><pre>import os<br>from openai import OpenAI<br><br># Set your OpenAI key<br>client = OpenAI(<br>  api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br>)<br><br># Initial ReAct-style prompt<br>def build_prompt(task: str) -&gt; str:<br>  return f&quot;&quot;&quot;<br>You are an intelligent AI agent solving problems using the ReAct pattern.<br><br>Follow this format:<br>Thought: What are you thinking?<br>Action: The action you are taking<br>Observation: (result from imaginary action)<br>Thought: …<br>FINAL_ANSWER: &lt;your final answer&gt;<br><br>Task: {task}<br><br>Each time you respond, you should only give **one logical step** in your reasoning process.<br>Do not jump to the final answer immediately.<br>Each step should be presented using the format above.<br><br>Begin.<br>Thought:<br>&quot;&quot;&quot;<br><br># Reasoning loop with no tools<br>def run_simple_react_agent(task: str, max_steps: int = 10):<br>  prompt = build_prompt(task)<br><br>  for step in range(max_steps):<br>    print(f&quot;\n --&gt; Step {step + 1}&quot;)<br><br>    response = client.chat.completions.create(<br>      model=&quot;gpt-4&quot;,<br>      messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>      temperature=0.7<br>    )<br><br>    output = response.choices[0].message.content.strip()<br>    print(output)<br><br>    prompt += output + &quot;\n&quot;<br><br>    if &quot;FINAL_ANSWER:&quot; in output:<br>      print(&quot;\n Agent finished.&quot;)<br>      break<br><br>      # Continue loop by cueing next Thought<br>      prompt += &quot;Thought:\n&quot;<br>    else:<br>      print(&quot;\n Max steps reached without a final answer.&quot;)<br><br># Run example<br>run_simple_react_agent(&quot;Taking the &#39;The Grapes of Wrath&#39; novel by John Steinbeck, what are the main differences when comparing it to the movie?&quot;)</pre><p>Before executing it, make sure you have installed the openai Python package. If you run the script you should get a response like this:</p><pre>--&gt; Step 1<br>In order to compare the differences between &#39;The Grapes of Wrath&#39; novel and its movie adaptation, I need to understand the plot, characters, themes, and narrative structure of both the novel and the movie.<br><br>Action:<br>I will look up summaries, reviews, and analyses of both the novel and the movie to identify key points of similarity and divergence.<br><br>Observation:<br>I have gathered information about the plot, characters, themes, and narrative structure of both &#39;The Grapes of Wrath&#39; novel and its movie adaptation.<br><br>--&gt; Step 2<br>Now that I have the necessary information, I need to compare the two mediums to identify differences. I&#39;ll start by comparing the plot of the novel and the movie.<br><br>Action:<br>I will compare the plot of the novel &#39;The Grapes of Wrath&#39; to the plot of its movie adaptation.<br><br>--&gt; Step 3<br>After comparing the plot of the novel and the movie, I noticed that the movie omits some parts of the novel&#39;s plot and simplifies others.<br><br>Action:<br>Next, I will compare the characters in the novel and the movie to identify any differences.<br><br>Observation:<br>The characters in the movie are generally faithful to the novel, but some have been downplayed or omitted entirely, potentially reducing the complexity and depth of the character relationships and development seen in the novel.<br><br>Thought:<br>Now, I&#39;ll compare the themes presented in the novel and the movie.<br><br>Action:<br>I will compare the themes of the novel &#39;The Grapes of Wrath&#39; to the themes presented in its movie adaptation.<br><br>--&gt; Step 4<br>The themes in the movie are generally consistent with those in the novel, but the film doesn&#39;t delve as deeply into some of the more complex and nuanced themes that the novel does.<br><br>Action:<br>Finally, I will compare the narrative structure of the novel and the movie to identify any differences.<br><br>--&gt; Step 5<br>The narrative structure in the movie is more linear and simplified, whereas the novel uses intercalary chapters to add depth and context to the main story, showing a broader picture of the societal issues of the time.<br><br>Action:<br>Now, I will compile the differences I have identified into a coherent comparison.<br><br>FINAL_ANSWER:<br>The main differences between &#39;The Grapes of Wrath&#39; novel and its movie adaptation are as follows:<br>1. Plot: The movie omits and simplifies parts of the novel&#39;s plot.<br>2. Characters: Some characters in the movie are downplayed or omitted, reducing the complexity and depth of the character relationships and development seen in the novel.<br>3. Themes: The movie doesn&#39;t delve as deeply into some of the more complex and nuanced themes that the novel does.<br>4. Narrative Structure: The movie has a more linear narrative, whereas the novel uses intercalary chapters to add depth and context to the main story.<br><br>Agent finished.</pre><h3>Kong Version</h3><p>As our first exercise, let’s inject Kong Data Plane to our simple scenario. Here’s the architecture illustrating it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*QdY8k9rZhyIqH56u" /></figure><p><em>AI Agent with Kong Konnect Data Plane</em></p><h3>OpenAI API support</h3><p>Kong AI Gateway supports the <a href="https://platform.openai.com/docs/overview">OpenAI API specification</a> as well as Bedrock and Gemini as incoming formats. The consumer can then send standard OpenAI requests to the Kong AI Gateway. As a basic example, consider this OpenAI request:</p><pre>curl https://api.openai.com/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;gpt-4o&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><p>When we add Kong AI Gateway, sitting in front of the LLM model, we’re not just exposing it but also allowing the consumers to use the same mechanism — in this case, OpenAI APIs — to consume it. That leads to a very flexible and powerful capability when we come to development processes. In other words, Kong AI Gateway normalizes the consumption of any LLM infrastructure, including Amazon Bedrock, Mistral, OpenAI, Cohere, etc.</p><p>As an exercise, the new request should be something like this. The request has some minor differences:</p><ul><li>It sends a request to the Kong API Gateway Data Plane Node.</li><li>It replaces the OpenAI endpoint with a Kong API Gateway route.</li><li>The API Key is actually managed by the Kong API Gateway now.</li><li>It does not refer to any model, since it’s being resolved by the AI Gateway.</li></ul><pre>curl http://$DATA_PLANE_LB/bedrock-route \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;apikey: $KONG_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><h3>Minikube installation</h3><p>We are going to deploy our Data Plane in a <a href="https://minikube.sigs.k8s.io/">Minikube</a> Cluster over Docker or Podman. For example, you can start Podman with:</p><pre>podman machine set --memory 8196<br>podman machine start</pre><p>If you want to stop it run:</p><pre>podman machine stop</pre><p>Then you can install Minikube with:</p><pre>minikube start --driver=podman --memory=&#39;no-limit&#39;</pre><p>To be able to consume the Kubernetes Load Balancer Services, in another terminal run:</p><pre>minikube tunnel</pre><h3>Konnect subscription</h3><p>Now you have to get access to Konnect. Click on the <a href="https://konghq.com/products/kong-konnect/register">Registration</a> link and present your credentials. Or, if you already have a Konnect subscription, <a href="https://cloud.konghq.com/us">log in to it</a>.</p><h3>Konnect PAT</h3><p>In order to interact with Konnect, using command lines, you need a <a href="https://docs.konghq.com/konnect/gateway-manager/declarative-config/#generate-a-personal-access-token">Konnect Personal Access Token (PAT)</a>. To generate your PAT, go to Konnect UI, click on your initials in the upper right corner of the Konnect home page, then select “Personal Access Tokens.” Click on “+ Generate Token,” name your PAT, set its expiration time, and be sure to copy and save it as an environment variable also named as PAT. Konnect won’t display your PAT again.</p><h3>Kong Gateway Operator (KGO), Konnect Control Plane creation and Data Plane deployment</h3><p>For Kubernetes deployments, Kong provides <a href="https://docs.konghq.com/gateway-operator">Kong Gateway Operator (KGO)</a>, an Operator capable of managing all flavours of Kong installations, including Kong Ingress Controller, Kong Gateway Data Planes, for self-managed or Konnect based deployments.</p><p>Our topology comprises a hybrid deployment where a Data Plane, running on Minikube, is connected to the Konnect Control Plane.</p><p>Start adding the <a href="https://github.com/Kong/charts/tree/main/charts/gateway-operator">KGO Helm Charts</a> to your environment:</p><pre>helm repo add kong https://charts.konghq.com<br>helm repo update kong</pre><p>Install the Operator with:</p><pre>helm upgrade --install kgo kong/gateway-operator \<br>-n kong-system \<br>--create-namespace \<br>--set image.tag=1.6 \<br>--set kubernetes-configuration-crds.enabled=true \<br>--set env.ENABLE_CONTROLLER_KONNECT=true</pre><p>You can check the Operator’s logs with:</p><pre>kubectl logs -f $(kubectl get pod -n kong-system -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kgo-gateway&quot;))&#39; | jq -r &#39;.name&#39;) -n kong-system</pre><p>And if you want to uninstall it run:</p><pre>helm uninstall kgo -n kong-system<br>kubectl delete namespace kong-system</pre><h4>Konnect Control Plane creation</h4><p>The first thing to do, in order to get your Konnect Control Plane defined, you have to create a Kubernetes Secret with your PAT. KGO requires your secret to be labeled. The commands should be like these ones:</p><pre>kubectl create namespace kong<br>kubectl delete secret konnect-pat -n kong<br>kubectl create secret generic konnect-pat -n kong --from-literal=token=&#39;kpat_K6Cgbx…..&#39;<br>kubectl label secret konnect-pat -n kong &quot;konghq.com/credential=konnect&quot;</pre><p>Then, the following declaration defines an <a href="https://docs.konghq.com/gateway-operator/latest/reference/custom-resources/#konnectapiauthconfiguration">Authentication Configuration</a>, based on the Kubernetes Secret and referring to a Konnect API URL, and the actual <a href="https://docs.konghq.com/gateway-operator/1.5.x/reference/custom-resources/#konnectgatewaycontrolplane">Konnect Control Plane</a>. Check the <a href="https://docs.konghq.com/gateway-operator/latest/get-started/konnect/create-konnectextension/#create-an-access-token-in-konnect">documentation</a> to learn more about it.</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>kind: KonnectAPIAuthConfiguration<br>apiVersion: konnect.konghq.com/v1alpha1<br>metadata:<br>  name: konnect-api-auth-conf<br>  namespace: kong<br>spec:<br>  type: secretRef  <br>  secretRef:<br>    name: konnect-pat<br>    namespace: kong<br>  serverURL: us.api.konghq.com<br>---<br>kind: KonnectGatewayControlPlane<br>apiVersion: konnect.konghq.com/v1alpha1<br>metadata:<br>  name: ai-gateway<br>  namespace: kong<br>spec:<br>  name: ai-gateway<br>  konnect:<br>    authRef:<br>      name: konnect-api-auth-conf<br>EOF</pre><p>You should see your Control Plane listed in your Konnect Organization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pm3AQSyCYgVI9Pt-" /></figure><p><em>Kong Konnect Control Plane</em></p><h4>Konnect Data Plane deployment</h4><p>The next declaration instantiates a Data Plane connected to your Control Plane. It creates a <a href="https://docs.konghq.com/gateway-operator/1.5.x/reference/custom-resources/#konnectextension-1">KonnectExtension</a>, asking KGO to manage the certificate and private key provisioning automatically, and the actual Data Plane. The <a href="https://docs.konghq.com/gateway-operator/latest/reference/custom-resources/#dataplane">Data Plane</a> declaration specifies the Docker image, in our case 3.10, as well as how the Kubernetes Service, related to the Data Plane, should be created.</p><p>The Data Plane declaration also defines the “<a href="https://docs.konghq.com/gateway/latest/reference/configuration/#untrusted_lua_sandbox_requires">unstrusted_lua_sandbox_requires</a>” environment variables with functions used by the Observability layer in order to allow them to be executed inside the Data Plane’s Lua sandbox.</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>kind: KonnectExtension<br>apiVersion: konnect.konghq.com/v1alpha1<br>metadata:<br>  name: konnect-config1<br>  namespace: kong<br>spec:<br>  clientAuth:<br>    certificateSecret:<br>      provisioning: Automatic<br>  konnect:<br>    controlPlane:<br>      ref:<br>        type: konnectNamespacedRef<br>        konnectNamespacedRef:<br>          name: ai-gateway<br>---<br>apiVersion: gateway-operator.konghq.com/v1beta1<br>kind: DataPlane<br>metadata:<br>  name: dataplane1<br>  namespace: kong<br>spec:<br>  extensions:<br>  - kind: KonnectExtension<br>    name: konnect-config1<br>    group: konnect.konghq.com<br>  deployment:<br>    podTemplateSpec:<br>      spec:<br>        containers:<br>        - name: proxy<br>          image: kong/kong-gateway:3.10<br>          env:<br>          - name: KONG_UNTRUSTED_LUA_SANDBOX_REQUIRES<br>            value: pl.stringio, ffi-zlib, cjson.safe<br>  network:<br>    services:<br>      ingress:<br>        name: proxy1<br>        type: LoadBalancer<br>EOF</pre><p>You can check the Data Plane logs with</p><pre>kubectl logs -f $(kubectl get pod -n kong -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;dataplane-&quot;))&#39; | jq -r &#39;.name&#39;) -n kong</pre><p>Also, you should see your first Data Plane listed and connected to the previously created Konnect Control Plane:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0ND_DZmXU3tDzwHi" /></figure><p><em>Kong Konnect Data Plane</em></p><h4>Consume the Data Plane</h4><p>If you check the new Kubernetes Service, since it’s been created as “Load Balancer”, and we are running Minikube, you’ll see that its external ip is defined as “127.0.0.1”.</p><pre>% kubectl get service proxy1 -n kong<br>NAME     TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                     AGE<br>proxy1   LoadBalancer   10.103.170.14   127.0.0.1     80:30350/TCP,443:30510/TCP  7m34s</pre><p>You should also see that, in the terminal running Minikube’s tunnel, the “proxy1” service has been exposed. It might need to type your admin’s password.</p><pre>❗ The service/ingress proxy1 requires privileged ports to be exposed: [80 443]<br>🔑 sudo permission will be asked for it.<br>🏃 Starting tunnel for service proxy1.</pre><p>If you send a request to it you should get a response from the Data Plane</p><pre>% curl -i http://localhost<br>HTTP/1.1 404 Not Found<br>Date: Thu, 24 Apr 2025 14:07:26 GMT<br>Content-Type: application/json; charset=utf-8<br>Connection: keep-alive<br>Content-Length: 103<br>X-Kong-Response-Latency: 1<br>Server: kong/3.10.0.1-enterprise-edition<br>X-Kong-Request-Id: 2b83f976af69655067bb5a4320a4677f<br><br>{<br>  &quot;message&quot;:&quot;no Route matched with those values&quot;,<br>  &quot;request_id&quot;:&quot;2b83f976af69655067bb5a4320a4677f&quot;<br>}</pre><p>Finally, if you want to delete all components run:</p><pre>kubectl delete dataplane dataplane1 -n kong<br>kubectl delete konnectextensions.konnect.konghq.com konnect-config1 -n kong<br>kubectl delete konnectgatewaycontrolplane ai-gateway -n kong<br>kubectl delete konnectapiauthconfiguration konnect-api-auth-conf -n kong<br>kubectl delete secret konnect-pat -n kong<br>kubectl delete namespace kong</pre><h3>Creating Kong Objects using decK</h3><p>In this next step, you’ll create the Kong Objects required to consume the OpenAI LLM model: Kong Service, Kong Route and Kong Plugin. You can continue using KGO and Kubernetes style declarations. However, we are going to exercise <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong). With decK you can manage Kong Konnect configuration and create Kong Objects in a declarative way. Check the decK documentation to learn how to <a href="https://docs.konghq.com/deck/latest/installation/">install</a> it.</p><p>Once you have decK installed, you should ping Konnect to check if the connection is up using the same PAT you created previously. You can use the following command to do it:</p><pre>% deck gateway ping --konnect-token $PAT<br>Successfully Konnected to the Kong organization!</pre><h3>decK declaration</h3><p>Now, create a file named “kong_agent_simple.yaml” with the following decK declaration:</p><pre>cat &gt; kong_agent_simple.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: ai-gateway<br>services:<br>- name: agent-service<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: agent-route1<br>    paths:<br>    - /agent-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-openai-agent&quot;<br>      enabled: true<br>      config:<br>        targets:<br>        - auth:<br>            header_name: &quot;Authorization&quot;<br>            header_value: &quot;Bearer &lt;your_OPENAI_API_KEY&gt;&quot;<br>          route_type: &quot;llm/v1/chat&quot;<br>          model:<br>            provider: &quot;openai&quot;<br>            name: &quot;gpt-4&quot;<br>EOF</pre><p>The declaration defines multiple Kong Objects:</p><ul><li><a href="https://docs.konghq.com/gateway/latest/get-started/services-and-routes/">Kong Gateway Service</a> named “agent-service”. The service doesn’t need to map to any real upstream URL. In fact, it can point somewhere empty, for example, <a href="http://localhost:32000.">http://localhost:32000.</a> This is because the AI Proxy plugin, also configured in the declaration, overwrites the upstream URL. This requirement will be removed in a later Kong revision.</li><li>Kong Route: the Gateway Service has a route defined with the “/agent-route” path. That’s the route we’re going to consume to reach OpenAI’s GPT-4 LLM.</li><li>AI Proxy Advanced Plugin. It’s configured to consume OpenAI’s “gpt-4” model. The “route_type” parameter, set as “llm/v1/chat”, refers to OpenAI’s “https://api.openai.com/v1/chat/completions” endpoint. Kong recommends storing the API Keys as secrets in a <a href="https://docs.konghq.com/gateway/latest/kong-enterprise/secrets-management/">Secret Manager</a> like AWS Secrets Manager or HashiCorp Vault. The current configuration, including the OpenAI API Key in the declaration, is for lab environments only, not recommended for production. Please refer to the official <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy-advanced/">AI Proxy Advanced Plugin documentation page</a> to learn more about its configuration.</li></ul><p>The declaration has been tagged as “agent” so you can manage its objects without impacting any other ones you might have created previously. Also, note the declaration is saying it should be applied to your “ai-gateway” Konnect Control Plane.</p><p>You can submit the declaration with the following decK command:</p><pre>deck gateway sync --konnect-token $PAT kong_agent_simple.yaml</pre><p>If you want to destroy all objects run:</p><pre>deck gateway reset --konnect-control-plane-name &quot;ai-gateway&quot; --select-tag &quot;agent&quot; --konnect-token $PAT -f</pre><h3>AI Agent</h3><p>Now, we’re ready to get back to our Agent. There are some changes you have to apply to your code to make Kong Version out of it:</p><ul><li>To redirect the requests to Kong AI Data Plane, you have to change your OpenAI constructor to refer to Kong Route.</li><li>Note that the OpenAI constructor still requires the OpenAI API Key. However, since it’s managed by Kong, you can use any value for it.</li><li>Since the AI Proxy Advanced Plugin has been configured to consume the GPT-4 model, you can set the “model” parameter you have in the “chat.completions.create” call as an empty string.</li><li>And, of course, you can delete the line where you import the “os” package.</li></ul><p>Here’s the new constructor:</p><pre>…<br>client = OpenAI(<br>  base_url=&quot;http://localhost:80/agent-route&quot;,<br>  api_key=&quot;dummy&quot;<br>)<br>…</pre><p>If you run your new code, you are supposed to get similar results, as Kong AI Gateway is not applying any policy to your requests. So far, if you check your Data Plane, you’ll see all requests that have been processed by the Data Plane and routed to OpenAI’s GPT-4 model.</p><p>Here’s the Kong Service Analytics tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mTqEP_QdjvJuy-y5" /></figure><p><em>Kong Konnect Analytics</em></p><p>As you can see, the Agent sent 5 requests to Kong Data Plane (and therefore, GPT-4), one for each step it has processed to respond to our question.</p><p>Moreover, the Konnect Control Plane provides extensive Analytics capabilities where you can check all requests processed by the Data Planes. Click on “Analytics” menu option and “Requests”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hn-QTfuRc0HyURXl" /></figure><p><em>Kong Konnect processed requests</em></p><p>Note you can introspect all processed requests including all major data like HTTP method, Kong Route and Service, Upstream URI, etc. For performance and security reasons, the Data Plane does not report the Control Plane with the body of the requests, which might be interesting to check, especially for situations like AI Agents. In the next section, we are going to configure the Data Plane to externalize both requests and responses bodies</p><h3>Kong AI Gateway Plugins</h3><p>Before we jump to the next section, it’s important to keep in mind that Kong AI Gateway provides an extensive list of AI-related plugins to implement specific policies related to:</p><ul><li>Prompt Engineering</li><li>Semantic Processing</li><li>Rate Limiting based on tokens</li><li>Request and Response transformations based on LLM queries</li><li>PII (Personally Identifiable Information) sanitization</li></ul><p>As we stated before, this blog post will leverage Semantic Routing capabilities to manage multiple LLMs sitting behind and getting protected by the Kong AI Gateway.</p><h3>LangGraph fundamentals</h3><p>As you can see, our Agent was written using the OpenAI package only. It’s a nice approach if you want to see a “raw” Agent in action. However, for several reasons, you should consider writing Agents taking advantage of some framework. For example: AI Agent complexity, security, integration with external systems and environments, not to mention development time, will rapidly become a necessity. To help you out, currently, the marketplace offers several AI Agent framework options including LlamaIndex, AutoGen, CrewAI, <a href="https://www.langchain.com/langgraph">LangGraph</a>, etc.</p><p>We’ve seen a lot of traction around LangGraph so it’d be, at least, interesting to take a look at it, and how Kong AI Gateway can play with it.</p><h3>LangGraph and LangChain</h3><p><a href="https://python.langchain.com/docs/introduction/">LangChain</a> provides a standard interface to interact with LLMs and other components, useful for straight-forward chains and retrieval flows. LangGraph is a framework for AI Agents and it is more low-level and controllable than LangChain constructs. It’s totally licit working with LangGraph and not using LangChain classes.</p><h3>OpenAI RESTful request</h3><p>So, let’s get started with a fundamental and basic application exploring the main LangGraph concepts as described in this <a href="https://langchain-ai.github.io/langgraph/concepts/low_level/">LangGraph Glossary</a> page. Note this is not a full Agent implementation but a code where we start using the LangGraph constructs to build a basic graph.</p><p>The first LangGraph script is based on this very simple OpenAI RESTful request. Make sure you’ve set the OPENAI_API_KEY environment variable with your OpenAI API Key.</p><pre>curl https://api.openai.com/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;gpt-4&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;What&#39;&#39;s the longitude and latitude of Seattle&quot;<br>      }<br>    ]<br>}&#39;</pre><p>You should get a result similar to this:</p><pre>{<br>  &quot;id&quot;: &quot;chatcmpl-BO3x47lZx2TLcgGsgLx7rGchcIB9m&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;created&quot;: 1745074762,<br>  &quot;model&quot;: &quot;gpt-4–0613&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: &quot;The latitude and longitude of Seattle, Washington, USA is approximately 47.6062° N, 122.3321° W.&quot;,<br>        &quot;refusal&quot;: null,<br>        &quot;annotations&quot;: []<br>      },<br>      &quot;logprobs&quot;: null,<br>      &quot;finish_reason&quot;: &quot;stop&quot;<br>    }<br>  ],<br>  &quot;usage&quot;: {<br>    &quot;prompt_tokens&quot;: 14,<br>    &quot;completion_tokens&quot;: 29,<br>    &quot;total_tokens&quot;: 43,<br>    &quot;prompt_tokens_details&quot;: {<br>      &quot;cached_tokens&quot;: 0,<br>      &quot;audio_tokens&quot;: 0<br>    },<br>    &quot;completion_tokens_details&quot;: {<br>      &quot;reasoning_tokens&quot;: 0,<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;accepted_prediction_tokens&quot;: 0,<br>      &quot;rejected_prediction_tokens&quot;: 0<br>    }<br>  },<br>  &quot;service_tier&quot;: &quot;default&quot;,<br>  &quot;system_fingerprint&quot;: null<br>}</pre><h3>LangGraph Python Script</h3><p>The LangGraph Python Script equivalent to the request would look like the following:</p><pre>from typing import TypedDict<br>from langgraph.graph import StateGraph, START, END<br><br>class State(TypedDict):<br>  messages: str<br>  response: str<br><br>def node_1(state: State) -&gt; State:<br>  completions = client.chat.completions.create(<br>    model=&quot;gpt-4&quot;,<br>    messages=state[&quot;messages&quot;]<br>  )<br>  return {&quot;response&quot;: completions.choices[0].message.content}<br><br><br>builder = StateGraph(State)<br>builder.add_node(&quot;node_1&quot;, node_1)<br>builder.add_edge(START, &quot;node_1&quot;)<br>builder.add_edge(&quot;node_1&quot;, END)<br><br>graph = builder.compile()<br>print(graph.get_graph().draw_ascii())<br><br><br>from openai import OpenAI<br>import os<br>client = OpenAI(<br>  api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br>)<br><br><br>text = &quot;What&#39;s the longitude and latitude of Seattle&quot;<br>state = {&quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: text}], &quot;response&quot;: &quot;&quot;}<br><br>print(&quot;State before invoking the graph&quot;)<br>print(state)<br>print(&quot;------\n&quot;)<br><br>print(&quot;Invoking the graph&quot;)<br>state = graph.invoke(state)<br>print(&quot;------\n&quot;)<br><br>print(&quot;State after invoking the graph&quot;)<br>print(state)<br>print(&quot;------\n&quot;)</pre><p>Before executing it, make sure you have installed the following Python packages:</p><ul><li>langgraph</li><li>openai</li></ul><p>Now, you should see as a result:</p><pre>+-----------+<br>| __start__ |<br>+-----------+<br>      *<br>      *<br>      *<br>  +--------+<br>  | node_1 |<br>  +--------+<br>      *<br>      *<br>      *<br> +---------+<br> | __end__ |<br> +---------+<br>State before invoking the graph<br>{&#39;messages&#39;: [{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: &quot;What&#39;s the longitude and latitude of Seattle&quot;}], &#39;response&#39;: &#39;&#39;}<br>------<br><br>Invoking the graph<br>------<br><br>State after invoking the graph<br>{&#39;messages&#39;: [{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: &quot;What&#39;s the longitude and latitude of Seattle&quot;}], &#39;response&#39;: &#39;The latitude and longitude of Seattle, Washington, USA is approximately 47.6062° N, 122.3321° W.&#39;}<br>------</pre><h3>LangGraph main constructs</h3><p>Richard J. Trudeau, in his seminal book <a href="https://archive.org/details/introductiontogr0000trud">“Introduction to Graph Theory”</a>, has a succinct graph definition: “A graph is an object consisting of two sets called its ‘vertex set’ and its ‘edge set’. The vertex set is a finite nonempty set. The edge may be empty, but otherwise its elements are two-element subsets of the vertex set.”. In other words, a graph is a data structure consisting of a set of vertices (also called nodes) connected by edges. Here’s a graph with 5 vertices (nodes) and 6 edges:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/0*E3e8DXR6ABj4qZf2" /></figure><p><em>Basic graph with 5 vertices and 6 edges</em></p><p>LangGraph fundamentally represents agents as graphs. The graphs are built using three main components:</p><ul><li>Nodes: fundamental unit of processing or decision-making within a graph.</li><li>Edges: it determines which node should run next after a given node finishes executing.</li><li>State: it is a structured data object that represents the current context of a graph.</li></ul><p>Combining Nodes and Edges, you can construct agents that update the State as they run. As stated in the <a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#graphs">LangGraph glossary page</a>: Nodes do the work. Edges tell what to do next.</p><p>Taking a look at our code, we have the following main constructs:</p><ul><li><a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#start-node">START</a> and END nodes and constants: they represent the nodes that send user input to the graph and the terminal node.</li><li><a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#state">State</a> class: it represents the state of the graph and consists of a schema (in our case, “messages” and “response”) and <a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#reducers">reducer functions</a>, responsible for updating the graph status. In our case, we don’t have any reducer function defined so LangGraph assumes that all updates to a key should override it. Optionally you can have reducer functions to add new values and not override the key, for example. The schema can be defined using Python’s <a href="https://mypy.readthedocs.io/en/stable/typed_dict.html">TypedDict</a> or <a href="https://docs.pydantic.dev/latest/">Pydantic</a> models.</li><li><a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.state.StateGraph">StateGraph</a> class: that’s where you define and compile the graph with all nodes and edges. It returns a <a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.state.CompiledStateGraph">CompiledStateGraph</a> object. The basic graph code does not have multiple nodes so it flows straightforwardly. In the next section, we are going to add “conditional edges” so it will control which node the next step of its processing should go to.</li><li>The “<a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.state.CompiledStateGraph.get_graph">get_graph()</a>” method returns <a href="https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.graph.Graph.html#graph">Graph</a> object and its “draw_ascii()” method provides a convenient visual representation of the graph.</li><li><a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.state.CompiledStateGraph.invoke">CompiledStateGraph.invoke()</a> method: since our graph does not have a loop, returning single output, the “invoke” method suffices. For ReAct AI Agent, where we have a reasoning loop, we should handle the graph execution differently. In the next section, implementing such a graph, will describe it.</li></ul><h3>Tools and Function Calling</h3><p>As you might have noticed that our simple AI Agent has a logic totally confined and self-contained into its code. Of course, the only external component called by the Agent is the GPT-4 model.</p><p>That, usually, becomes a huge restriction when we think about a bit more advanced agents. A mechanism to allow AI agents to invoke external functions, systems and API is necessary. That’s the main purpose of Tools.</p><p>OpenAI, for instance, provides 4 classes of built-in <a href="https://platform.openai.com/docs/guides/tools">Tools</a>: Web Search, File Search, Computer Use and Function Calling. The latest is commonly used by agents to integrate and call external functionalities. As other examples, similar capabilities are provided by <a href="https://docs.mistral.ai/capabilities/function_calling/">Mistral</a> and <a href="https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview">Anthropic</a>.</p><h3>OpenAI Built-In Tools</h3><p>To illustrate how Tools and LLM models work together, let’s send some requests to GPT-4. Let’s say your AI Agent code has two functions defined:</p><ul><li>get_weather: returns weather information for a given city.</li><li>get_composer: returns the composer of a specific piece.</li></ul><p>The issue is that, depending on the prompt the AI Agent receives, it doesn’t know which function it should call. So, the AI Agent sends a request to the LLM model and the response coming from it will drive the agent’s behaviour.</p><p>Here are some calls exemplifying the situation. The first request has in the “input” parameter the prompt received. In the “tools” parameter there’s an array with the two functions with their respective descriptions. Let’s see what happens when we send it:</p><pre>curl -s https://api.openai.com/v1/responses \<br>-H &quot;Content-Type: application/json&quot; \<br>-H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>-d &#39;{<br>  &quot;model&quot;: &quot;gpt-4&quot;,<br>  &quot;input&quot;: &quot;Who wrote the Hungarian Rapsodies?&quot;,<br>  &quot;tools&quot;: [<br>    {<br>      &quot;type&quot;: &quot;function&quot;,<br>      &quot;name&quot;: &quot;get_weather&quot;,<br>      &quot;description&quot;: &quot;Call to get the weather from a specific location.&quot;,<br>      &quot;parameters&quot;: {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>          &quot;location&quot;: {<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;description&quot;: &quot;City and country e.g. Bogotá, Colombia&quot;<br>          }<br>        },<br>        &quot;required&quot;: [<br>          &quot;location&quot;<br>        ],<br>        &quot;additionalProperties&quot;: false<br>      }<br>    },<br>    {<br>      &quot;type&quot;: &quot;function&quot;,<br>      &quot;name&quot;: &quot;get_composer&quot;,<br>      &quot;description&quot;: &quot;Call to get the composer of a specific piece.&quot;,<br>      &quot;parameters&quot;: {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>          &quot;piece&quot;: {<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;description&quot;: &quot;Name of the composer of a classical piece e.g. Beethoven&#39;&#39;s 5th Symphony&quot;<br>          }<br>        },<br>        &quot;required&quot;: [<br>          &quot;piece&quot;<br>        ],<br>        &quot;additionalProperties&quot;: false<br>      }<br>    }<br>  ]<br>}&#39; | jq &#39;.output&#39;</pre><p>The “output” field has the following content. It instructs the AI Agent to call the function, with the specified arguments, to get a better response to the caller.</p><pre>[<br>  {<br>    &quot;id&quot;: &quot;fc_680a9768943c81919427d5b9272bf31a0ad325331b90322b&quot;,<br>    &quot;type&quot;: &quot;function_call&quot;,<br>    &quot;status&quot;: &quot;completed&quot;,<br>    &quot;arguments&quot;: &quot;{\n \&quot;piece\&quot;: \&quot;Hungarian Rhapsodies\&quot;\n}&quot;,<br>    &quot;call_id&quot;: &quot;call_aotUalsZrtfqCxle0qaLgwFX&quot;,<br>    &quot;name&quot;: &quot;get_composer&quot;<br>  }<br>]</pre><h3>Tools and Function Calling, Kong Version with LangGraph and LangChain</h3><p>With OpenAI’s model still sitting behind the AI Gateway, let’s exercise a Python script sending Tools requests using LangGraph. You’ll see the code also uses LangChain classes.</p><p>At the same time, from the Observability standpoint, it’d be interesting to check not just the requests and responses sent to and coming from the LLM model but also to explore the bodies that are going to be processed back and forth.</p><p>Here’s the diagram showing the new components of our architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*y_1Ti3eTzFQUkhwB" /></figure><p><em>The basic architecture with a new Observability layer and LangGraph Tools</em></p><h3>Observability Layer</h3><p>Before working with the Kong Version for our Tools and Function Calling use cases, we are going to build an Observability layer to our environment. It will be comprised by:</p><ul><li><a href="https://grafana.com/oss/loki/">Loki</a>: that’s the most critical component, playing the log processing role and receiving all requests and responses processed by the Kong AI Gateway Data Plane.</li><li><a href="https://prometheus.io/">Prometheus</a>: responsible for scraping and storing the metrics the Kong AI Gateway will generate.</li><li><a href="https://grafana.com/oss/grafana/">Grafana</a>: used to query and analyze logs and metrics.</li></ul><h4>Installing Loki</h4><p>First, all the Helm Charts</p><pre>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update</pre><p>Install Loki with the following Helm command. Since we are exposing it with a Load Balancer, Minikube will start a new tunnel for the port 3100.</p><pre>helm install loki -n loki grafana/loki-stack \<br>--create-namespace \<br>--set loki.image.tag=2.9.13 \<br>--set loki.service.type=LoadBalancer \<br>--set loki.service.port=3100 \<br>--set promtail.enabled=false</pre><p>Hit the port to make sure Loki is ready to accept requests:</p><pre>% http :3100/ready<br>HTTP/1.1 200 OK<br>Content-Length: 6<br>Content-Type: text/plain; charset=utf-8<br>Date: Fri, 25 Apr 2025 17:21:19 GMT<br>X-Content-Type-Options: nosniff<br><br>ready</pre><p>If you want to uninstall it run:</p><pre>helm uninstall loki -n loki<br>kubectl delete namespace loki</pre><h4>Installing Prometheus and Grafana</h4><p>The Prometheus community distribution embeds Grafana so we can install both with a single command. Add the Helm Charts first:</p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts</pre><p>Again, after the installation, we should have two new Minikube tunnels defined:</p><pre>helm install prometheus -n prometheus prometheus-community/kube-prometheus-stack \<br>--create-namespace \<br>--set alertmanager.enabled=false \<br>--set prometheus.service.type=LoadBalancer \<br>--set prometheus.service.port=9090 \<br>--set grafana.service.type=LoadBalancer \<br>--set grafana.service.port=3000 \<br>--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false</pre><p>If you want to uninstall them run:</p><pre>helm uninstall prometheus -n prometheus<br>kubectl delete namespace prometheus</pre><p>By default, Grafana administrador’s credentials are: admin/prom-operator</p><p>If you will, Kong provides Grafana dashboards for both API (id: 7424) and AI Gateways (ai: 21162)</p><p>In MacOS, you can open Grafana with:</p><pre>open -a &quot;Google Chrome&quot; &quot;http://localhost:3000&quot;</pre><h4>Prometheus ServiceMonitor</h4><p>The following declaration creates a <a href="https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.ServiceMonitor">Prometheus ServiceMonitor</a> to be used by Prometheus to get Kong metrics.</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: kong<br>  namespace: kong<br>  labels:<br>    app.kubernetes.io/name: kong<br>spec:<br>  selector:<br>    matchLabels:<br>      gateway-operator.konghq.com/dataplane-service-type: ingress<br>  endpoints:<br>  - targetPort: metrics<br>    scheme: http<br>    jobLabel: kong<br>    namespaceSelector:<br>      matchNames:<br>        - kong<br>EOF</pre><p>If you want to delete it run:</p><pre>kubectl delete servicemonitor kong -n kong</pre><h4>Kong Observability Plugins</h4><p>Now, we need to configure the Kong Plugins that are going to generate the Observability signals the Data Plane will export to Prometheus and Loki.</p><p>The following deck declaration configures three Plugins. Check the documentation to learn more about them.</p><ul><li><a href="https://docs.konghq.com/hub/kong-inc/prometheus/">Prometheus</a>: to get the Data Plane generating API and AI Gateway related metrics.</li><li><a href="https://docs.konghq.com/hub/kong-inc/http-log/">HTTP Log</a>: to push, as log entries, all requests that have been processed by the Data Plane. Loki requires a specific JSON format as you can see here (<a href="https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs">https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs</a>) so we need to customize the HTTP Log plugin config to support that. That’s what the Lua code, defined in the “custom_fields_by_lua” configuration parameter, does. Note that the plugin sends the requests to Loki using the Kubernetes Service FQDN.</li><li><a href="https://docs.konghq.com/hub/kong-inc/pre-function/">Pre-Function</a>: the plugin will inject the request and response bodies to the log entries before HTTPLog plugin sends them to Loki.</li></ul><p>Download the <a href="https://raw.githubusercontent.com/CAcquaviva/kong-ai-gateway-langgraph/refs/heads/main/konnect/kong_agent_observability.yaml">kong-agent-observability.yaml</a> file with the declaration to configure the plugins. Some Lua functions used in the code configured in the HTTP Log and Pre-Function plugins require specific Data Plane configuration. As you may recall the Data Plane was deployed with the “untrusted_lua_sandbox_requires” parameter with those functions listed. You can apply the declaration with:</p><pre>deck gateway sync --konnect-token $PAT ./kong-agent-observability.yaml</pre><h3>Kong Version with LangGraph and LangChain</h3><p>Now, let’s explore the new Python Script where we use LangGraph and LangChain to exercise Tools and Function Calling, protected by Kong AI Gateway. Note that this is a code to test Tools and Function Calling. We are not actually implement an Agent:</p><pre>from langchain_openai import ChatOpenAI<br>from langchain_core.tools import tool<br>from langgraph.prebuilt import ToolNode<br><br>kong_dp = &quot;http://localhost:80&quot;<br>agent_url = kong_dp + &quot;/agent-route&quot;<br><br>client = ChatOpenAI(base_url=agent_url, model=&quot;&quot;, api_key=&quot;dummy&quot;)<br><br>@tool<br>def get_weather(location: str):<br>  &quot;&quot;&quot;Call to get the weather from a specific location.&quot;&quot;&quot;<br>  if any([city in location.lower() for city in [&quot;sf&quot;, &quot;san francisco&quot;]]):<br>    return &quot;It&#39;s sunny in San Francisco, but you better look out if you&#39;re a Gemini 😈.&quot;<br>  else:<br>    return f&quot;I am not sure what the weather is in {location}&quot;<br><br>@tool<br>def get_coolest_cities():<br>  &quot;&quot;&quot;Get a list of coolest cities&quot;&quot;&quot;<br>  return &quot;nyc, sf&quot;<br><br><br>tools = [get_weather, get_coolest_cities]<br>client = client.bind_tools(tools)<br><br>client_response = client.invoke(&quot;What&#39;s the weather in San Francisco?&quot;).tool_calls<br>print(client_response)</pre><p>There are new and important abstractions in the code:</p><ul><li>We have changed the basic OpenAI class with LangChain’s <a href="https://python.langchain.com/docs/integrations/chat/openai/">ChatOpenAI</a>. That’s critical because this new class allows us to bind a collection of tools we have in our code. Note the constructor still refers to the same Kong Route.</li><li>The code defines two functions as Tools: “get_weather” and “get_coolest_cities” using the “<a href="https://python.langchain.com/api_reference/core/tools/langchain_core.tools.convert.tool.html">@tool</a>” decorator.</li><li>The next three line are responsible for call the model as well as the available tools depending on the responses coming from the model</li><li>List of functions available</li></ul><pre>tools = [get_weather, get_coolest_cities]</pre><ul><li>Bind the list to the client variable which has a reference to the LLM model</li></ul><pre>client = client.bind_tools(tools)</pre><ul><li>Invoke the model with a prompt</li></ul><pre>client_response = client.invoke(&quot;What&#39;s the weather in San Francisco?&quot;).tool_calls</pre><p>After executing the code we can check two things:</p><ul><li>The actual response the Agent provides</li><li>Use the Kong AI Gateway Observability layer to see the request sent to the LLM model and its respective response.</li></ul><p>Here’s the AI Agent response:</p><pre>[{&#39;name&#39;: &#39;get_weather&#39;, &#39;args&#39;: {&#39;location&#39;: &#39;San Francisco&#39;}, &#39;id&#39;: &#39;call_sK8reJDyzJ6JBeD8QqDpW9gK&#39;, &#39;type&#39;: &#39;tool_call&#39;}]</pre><p>That means the LLM Model took the prompt and returned saying the Agent should call the function “get_weather”, with the parameter ‘location’: ‘San Francisco’ to provide a better response to the caller.</p><p>Let’s check Loki’s log entries now. Here’s the request sent by the Agent to the LLM model. As you can see it is quite similar to the one we sent previously using Curl. It has a “tools” section with the two functions defined in our code.</p><pre>% curl -G -s http://localhost:3100/loki/api/v1/query_range \<br>--data-urlencode &#39;query={gateway=&quot;kong-gateway&quot;}&#39; | jq &#39;.data.result[].values[0][1]&#39; | jq fromjson | jq &#39;.request.body&#39;<br>{<br>  &quot;stream&quot;: false,<br>  &quot;model&quot;: &quot;&quot;,<br>  &quot;messages&quot;: [<br>    {<br>      &quot;role&quot;: &quot;user&quot;,<br>      &quot;content&quot;: &quot;What&#39;s the weather in San Francisco?&quot;<br>    }<br>  ],<br>  &quot;tools&quot;: [<br>    {<br>      &quot;function&quot;: {<br>        &quot;name&quot;: &quot;get_weather&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;properties&quot;: {<br>            &quot;location&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            }<br>          },<br>          &quot;type&quot;: &quot;object&quot;,<br>          &quot;required&quot;: [<br>            &quot;location&quot;<br>          ]<br>        },<br>        &quot;description&quot;: &quot;Call to get the weather from a specific location.&quot;<br>      },<br>      &quot;type&quot;: &quot;function&quot;<br>    },<br>    {<br>      &quot;function&quot;: {<br>        &quot;name&quot;: &quot;get_coolest_cities&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;properties&quot;: {},<br>          &quot;type&quot;: &quot;object&quot;<br>        },<br>        &quot;description&quot;: &quot;Get a list of coolest cities&quot;<br>      },<br>      &quot;type&quot;: &quot;function&quot;<br>    }<br>  ]<br>}</pre><p>Here’s the response the Agent got from the LLM model. As expected, it is instructing the Agent to call the “get_weather” function. The response coming from the Agent follows the instructions.</p><pre>% curl -G -s http://localhost:3100/loki/api/v1/query_range \<br>--data-urlencode &#39;query={gateway=&quot;kong-gateway&quot;}&#39; | jq &#39;.data.result[].values[0][1]&#39; | jq fromjson | jq &#39;.response.body&#39;<br>{<br>  &quot;id&quot;: &quot;chatcmpl-BQIPtyAVegfM3LoikCESdUDckDC6M&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;content&quot;: null,<br>        &quot;tool_calls&quot;: [<br>          {<br>            &quot;id&quot;: &quot;call_GYBQsc8M4PGetPEQK3pvfgqz&quot;,<br>            &quot;type&quot;: &quot;function&quot;,<br>            &quot;function&quot;: {<br>              &quot;name&quot;: &quot;get_weather&quot;,<br>              &quot;arguments&quot;: &quot;{\n \&quot;location\&quot;: \&quot;San Francisco\&quot;\n}&quot;<br>            }<br>          }<br>        ],<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;annotations&quot;: {},<br>        &quot;refusal&quot;: null<br>      },<br>      &quot;finish_reason&quot;: &quot;tool_calls&quot;,<br>      &quot;logprobs&quot;: null<br>    }<br>  ],<br>  &quot;system_fingerprint&quot;: null,<br>  &quot;created&quot;: 1745607021,<br>  &quot;service_tier&quot;: &quot;default&quot;,<br>  &quot;model&quot;: &quot;gpt-4–0613&quot;,<br>  &quot;usage&quot;: {<br>    &quot;prompt_tokens_details&quot;: {<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;cached_tokens&quot;: 0<br>    },<br>    &quot;total_tokens&quot;: 93,<br>    &quot;completion_tokens_details&quot;: {<br>      &quot;reasoning_tokens&quot;: 0,<br>      &quot;accepted_prediction_tokens&quot;: 0,<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;rejected_prediction_tokens&quot;: 0<br>    },<br>    &quot;prompt_tokens&quot;: 76,<br>    &quot;completion_tokens&quot;: 17<br>  },<br>  &quot;object&quot;: &quot;chat.completion&quot;<br>}</pre><p>You can check Kong Analytics dashboard, one more time, to see that only one request has been sent and processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I1VjzlLsDVB3yQO_" /></figure><p><em>Kong Konnect Analytics</em></p><h4>Calling the actual Function</h4><p>Let’s replace the line of the Agent code where it sends the Tools request to the LLM model so it can accept the instructions from the LLM and call the actual “get_weather” function.</p><p>Change the following line</p><pre>client_response = client.invoke(&quot;What&#39;s the weather in San Francisco?&quot;).tool_calls</pre><p>With these new ones. It’s important to keep the line where we bind the functions to the client reference of the model.</p><pre>client_response = client.invoke(&quot;What&#39;s the weather in San Francisco?&quot;)<br>print(client_response)<br>print(&quot;\n&quot;)<br>tool = ToolNode(tools)<br>response = tool.invoke({&quot;messages&quot;: [client_response]})</pre><p>Here’s the Agent’s response. The function is actually called by the “tool.invoke” line. The “client.invoke” line returns the right parameters to do so.</p><p>“client.invoke” returns:</p><pre>content=&#39;&#39; additional_kwargs={&#39;tool_calls&#39;: [{&#39;id&#39;: &#39;call_fP7aborYv4d08I86KtgpGGEI&#39;, &#39;function&#39;: {&#39;arguments&#39;: &#39;{\n &quot;location&quot;: &quot;San Francisco&quot;\n}&#39;, &#39;name&#39;: &#39;get_weather&#39;}, &#39;type&#39;: &#39;function&#39;}], &#39;refusal&#39;: None} response_metadata={&#39;token_usage&#39;: {&#39;completion_tokens&#39;: 17, &#39;prompt_tokens&#39;: 76, &#39;total_tokens&#39;: 93, &#39;completion_tokens_details&#39;: {&#39;accepted_prediction_tokens&#39;: 0, &#39;audio_tokens&#39;: 0, &#39;reasoning_tokens&#39;: 0, &#39;rejected_prediction_tokens&#39;: 0}, &#39;prompt_tokens_details&#39;: {&#39;audio_tokens&#39;: 0, &#39;cached_tokens&#39;: 0}}, &#39;model_name&#39;: &#39;gpt-4–0613&#39;, &#39;system_fingerprint&#39;: None, &#39;id&#39;: &#39;chatcmpl-BQImmdurzZOcBqq9FWHwUhw92UVoP&#39;, &#39;finish_reason&#39;: &#39;tool_calls&#39;, &#39;logprobs&#39;: None} id=&#39;run-0b6e1ced-c3b7–488b-80e8–5ad4a9b1e3f0–0&#39; tool_calls=[{&#39;name&#39;: &#39;get_weather&#39;, &#39;args&#39;: {&#39;location&#39;: &#39;San Francisco&#39;}, &#39;id&#39;: &#39;call_fP7aborYv4d08I86KtgpGGEI&#39;, &#39;type&#39;: &#39;tool_call&#39;}] usage_metadata={&#39;input_tokens&#39;: 76, &#39;output_tokens&#39;: 17, &#39;total_tokens&#39;: 93, &#39;input_token_details&#39;: {&#39;audio&#39;: 0, &#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;audio&#39;: 0, &#39;reasoning&#39;: 0}}</pre><p>“tool.invoke” returns:</p><pre>{&#39;messages&#39;: [ToolMessage(content=&quot;It&#39;s sunny in San Francisco, but you better look out if you&#39;re a Gemini 😈.&quot;, name=&#39;get_weather&#39;, tool_call_id=&#39;call_fP7aborYv4d08I86KtgpGGEI&#39;)]}</pre><h4>Sending a prompt with no function related</h4><p>If we send a request with a prompt not related to any function description, the LLM responds as a regular “message” request:</p><p>For example, change your “client.invoke” line to:</p><pre>client_response = client.invoke(&quot;Tell me about Fermat&#39;s last theorem&quot;)</pre><p>Here’s the AI Agent response. Since there’s no appropriate function to satisfy the request, the LLM processes the request.</p><pre>content=&quot;Fermat&#39;s Last Theorem is a statement in number theory that was first proposed by Pierre de Fermat in 1637. The theorem states that no three positive integers a, b, and c can satisfy the equation a^n + b^n = c^n for any integer value of n greater than two.\n\nWhile the cases n=1 and n=2 have been known to have infinitely many solutions since antiquity, Fermat claimed that he had a proof for all n, but he never revealed it. For hundreds of years, the theorem puzzled mathematicians, and it became one of the most famous unsolved problems in mathematics.\n\nThe theorem was finally proven by British mathematician Andrew Wiles in 1994. The proof is highly complex and uses many techniques from different areas of mathematics, some of which were not even invented until centuries after Fermat&#39;s death. Fermat&#39;s Last Theorem has few direct implications in everyday life, but the journey to its proof has led to substantial developments in number theory and algebraic geometry.&quot; additional_kwargs={&#39;refusal&#39;: None} response_metadata={&#39;token_usage&#39;: {&#39;completion_tokens&#39;: 210, &#39;prompt_tokens&#39;: 76, &#39;total_tokens&#39;: 286, &#39;completion_tokens_details&#39;: {&#39;accepted_prediction_tokens&#39;: 0, &#39;audio_tokens&#39;: 0, &#39;reasoning_tokens&#39;: 0, &#39;rejected_prediction_tokens&#39;: 0}, &#39;prompt_tokens_details&#39;: {&#39;audio_tokens&#39;: 0, &#39;cached_tokens&#39;: 0}}, &#39;model_name&#39;: &#39;gpt-4–0613&#39;, &#39;system_fingerprint&#39;: None, &#39;id&#39;: &#39;chatcmpl-BQJ1oJWYwOdFDnS0X2i6E0JAcyAIv&#39;, &#39;finish_reason&#39;: &#39;stop&#39;, &#39;logprobs&#39;: None} id=&#39;run-48a213c1–92e3–44f2-b071–48a76ca1177e-0&#39; usage_metadata={&#39;input_tokens&#39;: 76, &#39;output_tokens&#39;: 210, &#39;total_tokens&#39;: 286, &#39;input_token_details&#39;: {&#39;audio&#39;: 0, &#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;audio&#39;: 0, &#39;reasoning&#39;: 0}}<br>{&#39;messages&#39;: []}</pre><p>And here are the request and response bodies:</p><pre>% curl -G -s http://localhost:3100/loki/api/v1/query_range \<br>--data-urlencode &#39;query={gateway=&quot;kong-gateway&quot;}&#39; | jq &#39;.data.result[].values[0][1]&#39; | jq fromjson | jq &#39;.request.body&#39;<br>{<br>  &quot;stream&quot;: false,<br>  &quot;model&quot;: &quot;&quot;,<br>  &quot;messages&quot;: [<br>    {<br>      &quot;role&quot;: &quot;user&quot;,<br>      &quot;content&quot;: &quot;Tell me about Fermat&#39;s last theorem&quot;<br>    }<br>  ],<br>  &quot;tools&quot;: [<br>    {<br>      &quot;function&quot;: {<br>        &quot;name&quot;: &quot;get_weather&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;properties&quot;: {<br>            &quot;location&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            }<br>          },<br>          &quot;type&quot;: &quot;object&quot;,<br>          &quot;required&quot;: [<br>            &quot;location&quot;<br>          ]<br>        },<br>        &quot;description&quot;: &quot;Call to get the weather from a specific location.&quot;<br>      },<br>      &quot;type&quot;: &quot;function&quot;<br>    },<br>    {<br>      &quot;function&quot;: {<br>        &quot;name&quot;: &quot;get_coolest_cities&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;properties&quot;: {},<br>          &quot;type&quot;: &quot;object&quot;<br>        },<br>        &quot;description&quot;: &quot;Get a list of coolest cities&quot;<br>      },<br>      &quot;type&quot;: &quot;function&quot;<br>    }<br>  ]<br>}<br><br><br>% curl -G -s http://localhost:3100/loki/api/v1/query_range \<br>--data-urlencode &#39;query={gateway=&quot;kong-gateway&quot;}&#39; | jq &#39;.data.result[].values[0][1]&#39; | jq fromjson | jq &#39;.response.body&#39;<br>{<br>  &quot;id&quot;: &quot;chatcmpl-BQJ1oJWYwOdFDnS0X2i6E0JAcyAIv&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;annotations&quot;: {},<br>        &quot;content&quot;: &quot;Fermat&#39;s Last Theorem is a statement in number theory that was first proposed by Pierre de Fermat in 1637. The theorem states that no three positive integers a, b, and c can satisfy the equation a^n + b^n = c^n for any integer value of n greater than two.\n\nWhile the cases n=1 and n=2 have been known to have infinitely many solutions since antiquity, Fermat claimed that he had a proof for all n, but he never revealed it. For hundreds of years, the theorem puzzled mathematicians, and it became one of the most famous unsolved problems in mathematics.\n\nThe theorem was finally proven by British mathematician Andrew Wiles in 1994. The proof is highly complex and uses many techniques from different areas of mathematics, some of which were not even invented until centuries after Fermat&#39;s death. Fermat&#39;s Last Theorem has few direct implications in everyday life, but the journey to its proof has led to substantial developments in number theory and algebraic geometry.&quot;,<br>        &quot;refusal&quot;: null,<br>        &quot;role&quot;: &quot;assistant&quot;<br>      },<br>      &quot;finish_reason&quot;: &quot;stop&quot;,<br>      &quot;logprobs&quot;: null<br>    }<br>  ],<br>  &quot;system_fingerprint&quot;: null,<br>  &quot;created&quot;: 1745609372,<br>  &quot;service_tier&quot;: &quot;default&quot;,<br>  &quot;model&quot;: &quot;gpt-4–0613&quot;,<br>  &quot;usage&quot;: {<br>    &quot;total_tokens&quot;: 286,<br>    &quot;prompt_tokens_details&quot;: {<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;cached_tokens&quot;: 0<br>    },<br>    &quot;prompt_tokens&quot;: 76,<br>    &quot;completion_tokens_details&quot;: {<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;rejected_prediction_tokens&quot;: 0,<br>      &quot;reasoning_tokens&quot;: 0,<br>      &quot;accepted_prediction_tokens&quot;: 0<br>    },<br>    &quot;completion_tokens&quot;: 210<br>  },<br>  &quot;object&quot;: &quot;chat.completion&quot;<br>}</pre><h3>Kong AI Gateway plugins</h3><p>As we’d mentioned previously, one of main responsibilities of the Kong AI Gateway is to implement and enforce policies to make the upstream services, in our case the LLM models, are being consumed properly.</p><p>As an example, the next declaration enables two plugins:</p><ul><li><a href="https://docs.konghq.com/hub/kong-inc/key-auth/">Key Auth</a>: to implement an API Key security mechanism.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-rate-limiting-advanced/">AI Rate Limiting Advanced</a>: to implement a rate limiting policies based on tokens sent to the LLM model.</li></ul><h4>decK declaration</h4><p>The “kong_agent_simple_plugins.yaml” has the new decK declaration:</p><pre>cat &gt; kong_agent_simple_plugins.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: ai-gateway<br>services:<br>- name: agent-service<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: agent-route1<br>    paths:<br>    - /agent-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-openai-agent&quot;<br>      enabled: true<br>      config:<br>        targets:<br>        - auth:<br>            header_name: &quot;Authorization&quot;<br>            header_value: &quot;Bearer &lt;your_OPENAI_API_KEY&gt;&quot;<br>          route_type: &quot;llm/v1/chat&quot;<br>          model:<br>            provider: &quot;openai&quot;<br>            name: &quot;gpt-4&quot;<br>    - name: key-auth<br>      instance_name: key-auth1<br>      enabled: true<br>consumers:<br>- keyauth_credentials:<br>  - key: &quot;123456&quot;<br>  username: user1<br>  plugins:<br>  - name: ai-rate-limiting-advanced<br>    instance_name: ai-rate-limiting-advanced-consumer1<br>    enabled: true<br>    config:<br>      llm_providers:<br>      - name: openai<br>        window_size:<br>        - 60<br>        limit:<br>        - 800<br>EOF</pre><p>The declaration applies the two new plugins, AI Rate Limiting Advanced and Key Auth, and creates a <a href="https://docs.konghq.com/gateway/latest/key-concepts/consumers/">Kong Consumer</a> with an API Key. You can submit the declaration the same way you did before:</p><pre>deck gateway sync --konnect-token $PAT ./kong_agent_simple_plugins.yaml</pre><p>You have to change the ChatOpenAI constructor line adding the API Key defined to protect the Kong Route. The new constructor uses the “default_headers” parameter to do so as described in the <a href="https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.openai.ChatOpenAI.html">ChatOpenAI documentation</a>:</p><pre>client = ChatOpenAI(base_url=agent_url, model=&quot;&quot;, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})</pre><p>If you don’t change this line the code will get a 401 error code, saying that, since the request doesn’t have an API Key, you are unauthorized to consume the Kong Route. Moreover, if you execute the code several times, eventually you are going to get a 429 error code, meaning your Kong Consumer has reached the rate limiting policy threshold of 800 tokens per 60 seconds.</p><h3>LangGraph ReAct AI Agent with Tools</h3><p>Now that we are a bit more familiar with Tools and Function Calling, we are going to enhance our LangGraph AI Agent and add the same Tools. Here’s the new topology:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*9cZdBLd5YYHGchb9" /></figure><p><em>Reference Architecture with external functions getting protected by the Kong AI Gateway</em></p><p>Although using the same LangGraph framework, the original version of the Agent does not implement a reasoning loop. That is, when invoked, simply executes the Node which sends a request to the LLM and gets the results.This version implements a full ReAct Agent with a reasoning loop. Moreover, it reaches out to the LLM models through Kong AI Gateway.</p><p>Here are the main steps of the Agent:</p><ol><li>Think about the input.</li><li>Choose a tool (action).</li><li>Act by invoking the tool.</li><li>Observe the result.</li><li>Repeat until it reaches a final answer.</li></ol><p>The LangGraph documentation has a <a href="https://langchain-ai.github.io/langgraph/agents/agents/">page</a> where it describes an Agent with the main components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6NHyeb951Tewe-O2" /></figure><h3>The Code</h3><p>Here’s our new Agent:</p><pre>from typing import Annotated, TypedDict<br>from langgraph.graph import StateGraph, START, END<br>from langchain_core.messages import BaseMessage<br>from langchain_core.messages import SystemMessage<br>from langgraph.graph.message import add_messages<br>from langchain_openai import ChatOpenAI<br>from langchain_core.tools import tool<br>from langgraph.prebuilt import ToolNode<br><br>from langchain_community.utilities.openweathermap import OpenWeatherMapAPIWrapper<br>import httpx<br><br><br>class State(TypedDict):<br>  messages: Annotated[list[BaseMessage], add_messages]<br><br><br>@tool<br>def get_weather(location: str):<br>  &quot;&quot;&quot;Call to get the weather from a specific location.&quot;&quot;&quot;<br>  openweathermap_url = kong_dp + &quot;/openweathermap-route&quot;<br>  result = httpx.get(openweathermap_url, params={&quot;q&quot;: location})<br>  return result.json()<br><br><br>@tool<br>def get_composer(piece: str):<br>  &quot;&quot;&quot;Call to get the composer of a specific piece.&quot;&quot;&quot;<br>  deezer_url = kong_dp + &quot;/deezer-route&quot;<br>  result = httpx.get(deezer_url, params={&quot;q&quot;: piece})<br>  return result.json()[&quot;data&quot;][0][&quot;artist&quot;][&quot;name&quot;]<br><br><br>@tool<br>def get_mathematician(theorem: str):<br>  &quot;&quot;&quot;Call to get the mathematician of a specific theorem.&quot;&quot;&quot;<br>  wikipedia_url = kong_dp + &quot;/wikipedia-route&quot;<br>  result = httpx.get(wikipedia_url, params={&quot;srsearch&quot;: theorem})<br>  return result.json()[&quot;query&quot;][&quot;search&quot;][0]<br><br><br>def call_model(state: State):<br>  system_prompt = SystemMessage(<br>    &quot;You are a helpful AI assistant, please convert temperatures to Celsius.&quot;<br>  )<br>  response = client.invoke([system_prompt] + state[&quot;messages&quot;])<br>  return {&quot;messages&quot;: [response]}<br><br><br># Define the conditional edge that determines whether to continue or not<br>def should_continue(state: State):<br>  messages = state[&quot;messages&quot;]<br>  last_message = messages[-1]<br>  # If there is no function call, then we finish<br>  if not last_message.tool_calls:<br>    return &quot;end&quot;<br>  # Otherwise if there is, we continue<br>  else:<br>    return &quot;continue&quot;<br><br><br>def print_stream(stream):<br>  for s in stream:<br>    message = s[&quot;messages&quot;][-1]<br>    if isinstance(message, tuple):<br>      print(message)<br>    else:<br>      message.pretty_print()<br><br><br>tools = [get_weather, get_composer, get_mathematician]<br>tool_node = ToolNode(tools)<br><br><br>builder = StateGraph(State)<br>builder.add_node(&quot;agent_node&quot;, call_model)<br>builder.add_edge(START, &quot;agent_node&quot;)<br>builder.add_node(&quot;tool_node&quot;, tool_node)<br>builder.add_conditional_edges(&#39;agent_node&#39;, should_continue, {&quot;continue&quot;: &quot;tool_node&quot;, &quot;end&quot;: END})<br>builder.add_edge(&quot;tool_node&quot;, &quot;agent_node&quot;)<br><br><br>graph = builder.compile()<br>print(graph.get_graph().draw_ascii())<br><br>kong_dp = &quot;http://localhost:80&quot;<br>agent_url = kong_dp + &quot;/agent-route&quot;<br><br>client = ChatOpenAI(base_url=agent_url, model=&quot;&quot;, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})<br>client = client.bind_tools(tools)<br><br>text = &quot;What&#39;s the weather in the city where the composer of &#39;Like a Rolling Stone&#39; was born?&quot;<br>print(&quot;start streaming the graph&quot;)<br>inputs = {&quot;messages&quot;: [(&quot;user&quot;, text)]}<br>print_stream(graph.stream(inputs, stream_mode=&quot;values&quot;))<br>print(&quot;stop streaming the graph&quot;)</pre><h3>The Graph</h3><p>The main difference we’ve added to this new Agent version is the graph itself. First, if you check the “print(graph.get_graph().draw_ascii())” code line result you’ll see a different visual representation like this.</p><pre>            +-----------+<br>            | __start__ |<br>            +-----------+<br>                  *<br>                  *<br>                  *<br>           +------------+<br>           | agent_node |<br>           +------------+<br>           ...         ...<br>          .               .<br>        ..                 ..<br>+-----------+           +---------+<br>| tool_node |           | __end__ |<br>+-----------+           +---------+</pre><p>To better understand what’s going on, maybe we should dissect the graph definition snippet:</p><ul><li>Fundamentally, the graph definition starts with the same three lines. Since it’s associated with the “call_model” function, the “agent_node”, just like our first Agent version, is responsible for sending requests to the LLM model. It also defines a “<a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#normal-edges">normal edge</a>”, linking the START and “agent_node” nodes.</li></ul><pre>builder = StateGraph(State)<br>builder.add_node(&quot;agent_node&quot;, call_model)<br>builder.add_edge(START, &quot;agent_node&quot;)</pre><ul><li>Then it creates another node, named “tool_node”, associated with the “tool_nodes” variable, which is an array of all Tools’ functions (“get_weather”, “get_composer” and “get_mathematician”)</li></ul><pre>builder.add_node(&quot;tool_node&quot;, tool_node)</pre><ul><li>In the next line, we add a new LangGraph construct, a “<a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#conditional-edges">conditional edge</a>”. That’s where the agent makes decisions to determine which node should be processed next. The parameters required are:</li><li>The starting node, in this case, “agent_node”</li><li>The function that will determine which node should be called next. In our case, the “should_continue” function.</li><li>A mapping relating the function’s outputs to actual graph nodes. If the function returns “continue” the next node will be “toll_node”, otherwise the graph will be routed to the END node and will finish.</li></ul><pre>builder.add_conditional_edges(&#39;agent_node&#39;, should_continue, {&quot;continue&quot;: &quot;tool_node&quot;, &quot;end&quot;: END})</pre><ul><li>The following and last line adds the second “edge”, saying the “tool_node” is associated with the “agent” node. That is necessary to get the graph loop in place.</li></ul><pre>builder.add_edge(&quot;tool_node&quot;, &quot;agent_node&quot;)</pre><p>The “should_continue” function is straightforward. It checks if the last response coming from the LLM was related to a Tools request. If it was, that means the graph should continue, returning to the “tool_node”, and call another Tool, as instructed in the LLM response. Otherwise, the loop will end.</p><p>As a last comment, the “call_model” function adds a SystemMessage to the request asking the LLM to convert any temperature it gets to Celsius.</p><h3>Graph Stream</h3><p>Our first graph was very simple, with a single node and no reasoning loop. As you can check, the line “state = graph.invoke(state)” invokes the graph and gets the new State with the response returned by the graph.</p><p>This time, our graph does implement a loop so we need to handle the graph execution in a different way. You can still invoke the graph the same we did before. However, a more interesting way is to use the <a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.state.CompiledStateGraph.stream">CompiledStateGraph.stream</a> method instead. What this new method does is stream the graph steps with the responses coming from the graph loop. The code is using the “stream_mode” as “values” which emits all values in the state after each step.</p><p>Streaming the graph should generate a very helpful output like this. Note it describes the different kinds of message the graph deals with: “Human Message”, “Ai Message” (coming from the LLM) and “Tool Message” coming from the Tools.</p><pre>start streaming the graph<br>================================ Human Message =================================<br>What&#39;s the weather in the city where the composer of &#39;Like a Rolling Stone&#39; was born?<br>================================== Ai Message ==================================<br>Tool Calls:<br>  get_composer (call_EhAHCD9CqwLbKNiRWM7go0BS)<br>Call ID: call_EhAHCD9CqwLbKNiRWM7go0BS<br>  Args:<br>  piece: Like a Rolling Stone<br>================================= Tool Message =================================<br>Name: get_composer<br><br>Bob Dylan<br>================================== Ai Message ==================================<br>Bob Dylan was born in Duluth, Minnesota. Let&#39;s find out the weather.<br>Tool Calls:<br>    get_weather (call_L6jl2wlo6Ruf0hEK8h3ddaI1)<br>  Call ID: call_L6jl2wlo6Ruf0hEK8h3ddaI1<br>    Args:<br>      location: Duluth, Minnesota<br>================================= Tool Message =================================<br>Name: get_weather<br><br>{&quot;coord&quot;: {&quot;lon&quot;: -92.1066, &quot;lat&quot;: 46.7833}, &quot;weather&quot;: [{&quot;id&quot;: 804, &quot;main&quot;: &quot;Clouds&quot;, &quot;description&quot;: &quot;overcast clouds&quot;, &quot;icon&quot;: &quot;04d&quot;}], &quot;base&quot;: &quot;stations&quot;, &quot;main&quot;: {&quot;temp&quot;: 278.12, &quot;feels_like&quot;: 275.96, &quot;temp_min&quot;: 276.86, &quot;temp_max&quot;: 280.56, &quot;pressure&quot;: 1016, &quot;humidity&quot;: 91, &quot;sea_level&quot;: 1016, &quot;grnd_level&quot;: 982}, &quot;visibility&quot;: 10000, &quot;wind&quot;: {&quot;speed&quot;: 2.57, &quot;deg&quot;: 100}, &quot;clouds&quot;: {&quot;all&quot;: 100}, &quot;dt&quot;: 1746213924, &quot;sys&quot;: {&quot;type&quot;: 2, &quot;id&quot;: 2086417, &quot;country&quot;: &quot;US&quot;, &quot;sunrise&quot;: 1746183078, &quot;sunset&quot;: 1746235157}, &quot;timezone&quot;: -18000, &quot;id&quot;: 5024719, &quot;name&quot;: &quot;Duluth&quot;, &quot;cod&quot;: 200}<br>================================== Ai Message ==================================<br><br>The temperature in Duluth, Minnesota, where Bob Dylan was born, is currently 278.12 Kelvin. Let&#39;s convert that to Celsius.<br><br>The formula to convert from Kelvin to Celsius is: C = K - 273.15<br><br>So, 278.12 Kelvin is approximately 5 degrees Celsius.<br>stop streaming the graph</pre><p>You can check the requests processed by the Gateway using the Observability layer or the Analytics capabilities provided by Konnect UI. For example, here are the requests related to the AI Agent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y3P-lSot3D20PBPh" /></figure><h3>The State</h3><p>Another relevant change of the code is the AI Agent State. Differently from the first agent, the current State is defined as</p><pre>class State(TypedDict):<br>  messages: Annotated[list[BaseMessage], add_messages]</pre><p>Fundamentally, the State has only one key, “message”, which is based on the “Annotated” type. The “Annotated” type specifies:</p><ul><li>A list of “<a href="https://python.langchain.com/api_reference/core/messages/langchain_core.messages.base.BaseMessage.html">BaseMessage</a>” messages.</li><li>The “<a href="https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.message.add_messages">add_messages</a>” reducer function to the State. Used in a StateGraph, “add_messages” function appends the messages received from LLM models and Tools.</li></ul><p>Creating a State as a list of messages is <a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#working-with-messages-in-graph-state">justifiable and helpful</a> since we are managing multiple responses coming from LLM models and Tools. Moreover, the “Annotated” type could specify a list of regular strings. However, “<a href="https://python.langchain.com/api_reference/core/messages.html">BaseMessage</a>” class seems more appropriate because it is the base class for any other message classes we are going to deal with such as “HumanMessage” for user input or “AIMessage” for LLM response.</p><h3>The Tools</h3><p>Each tool is calling an external service, in our case:</p><ul><li>“get_weather”: it calls the public <a href="https://openweathermap.org/city/2643743">OpenWeather</a> service to get information about weather in a given location.</li><li>“get_composer”: it relies on public <a href="https://rapidapi.com/deezerdevs/api/deezer-1">RapidAPI’s Deezer</a> service to manage composers of a given musical piece.</li><li>“get_mathematician”: it sends requests to Wikipedia to get information about theorems.</li></ul><p>The main point is that, as illustrated in the diagram, the endpoints related to the functions are also protected by Kong AI Gateway. That means we can leverage the capabilities the Gateway provides to implement policies.</p><h3>decK declaration</h3><p>External service might require specific API keys to get called. As an example of a Gateway policy, and since it is sitting in front of the services it’s easy to let it manage the API Keys. Here’s the decK declaration for the three external service’s endpoints:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: ai-gateway<br>services:<br>- name: openweathermap-service<br>  url: https://api.openweathermap.org/data/2.5/weather<br>  routes:<br>  - name: openweathermap-route<br>    paths:<br>    - /openweathermap-route<br>    plugins:<br>    - name: request-transformer<br>      instance_name: request-transformer-openweathermap<br>      config:<br>        add:<br>          querystring:<br>          - &quot;appid:&lt;your_openweathermap_api_key&gt;&quot;<br>  - name: deezer-service<br>    url: https://deezerdevs-deezer.p.rapidapi.com/search<br>    routes:<br>    - name: deezer-route<br>      paths:<br>      - /deezer-route<br>      plugins:<br>      - name: request-transformer<br>        instance_name: request-transformer-deezer<br>        config:<br>          add:<br>            headers:<br>            - &quot;x-rapidapi-key:&lt;your_rapidapi_deezer_api_key&gt;&quot;<br>  - name: wikipedia-service<br>    url: https://en.wikipedia.org/w/api.php<br>    routes:<br>    - name: wikipedia-route<br>      paths:<br>      - /wikipedia-route<br>      plugins:<br>      - name: request-transformer<br>        instance_name: request-transformer-wikipedia<br>        config:<br>          add:<br>            querystring:<br>            - &quot;action:query&quot;<br>            - &quot;list:search&quot;<br>            - &quot;format:json&quot;<br>  - name: agent-service<br>    host: localhost<br>    port: 32000<br>    routes:<br>    - name: agent-route1<br>      paths:<br>      - /agent-route<br>      plugins:<br>      - name: ai-proxy-advanced<br>        instance_name: &quot;ai-proxy-advanced-agent&quot;<br>        enabled: true<br>        config:<br>          targets:<br>          - model:<br>              provider: &quot;openai&quot;<br>              name: &quot;gpt-4&quot;<br>            auth:<br>              header_name: &quot;Authorization&quot;<br>              header_value: &quot;Bearer &lt;your_openai_api_key&gt;&quot;<br>            route_type: &quot;llm/v1/chat&quot;<br>            logging:<br>              log_payloads: true<br>              log_statistics: true<br>      - name: key-auth<br>        instance_name: key-auth1<br>        enabled: true<br>consumers:<br>- keyauth_credentials:<br>  - key: &quot;123456&quot;<br>  username: user1<br>  plugins:<br>  - name: ai-rate-limiting-advanced<br>    instance_name: ai-rate-limiting-advanced-consumer1<br>    enabled: true<br>    config:<br>      llm_providers:<br>      - name: openai<br>        window_size:<br>        - 60<br>        limit:<br>        - 800</pre><p>The declaration creates Kong Services and Kong Routes for each one of the external services. With the <a href="https://docs.konghq.com/hub/kong-inc/request-transformer/">Request Transformer</a> plugin we inject the necessary API Key required by the external services. That provides a secure solution to manage API Keys. As we stated before, for production environments, we recommend leveraging Secrets Managers to store the keys.</p><p>Moreover, the Gateway abstracts, from the Agent perspective the endpoint addresses each external service exposes. That flexibility provides a transparency where any changes you might want to make to the external service doesn’t impact the Agent.</p><h3>LangGraph reasoning</h3><p>Taking a look at new Agent code, we can see that, In LangGraph, reasoning might happen in three different moments:</p><ul><li>Inside Nodes: it processes inputs and generates outputs based on prompts and intermediate state. This is what happened, for example, in our very first Agent where we used Prompt Engineering to state it should implement the ReAct pattern.</li><li>Conditional Edges can model logical flows and decisions.</li><li>States, acting as a memory, evolve as the graph progresses. Reasoning can involve reading from or writing to this state to keep track of thoughts, decisions, or partial results.</li></ul><h3>Multi-LLM ReAct AI Agent</h3><p>In this section of the blog post we are going to evolve the architecture one more time to add two new LLMs infrastructures sitting behind the Gateway, besides OpenAI: Mistral and Anthropic.</p><h3>Multi-LLM scenarios and use cases</h3><p>In the main scenario, the Agent needs to communicate to multiple LLMs selectively, depending on its needs. Having the Kong AI Gateway intermediating the communication provides several benefits:</p><ul><li>Decide which LLM to use based on the cost, latency times, reliability, and mainly on semantics (some LLMs are better at a specific topic, others at coding, etc).</li><li>Route queries to the appropriate LLM(s).</li><li>Act based on the results.</li><li>Fallback and redundancy: If one LLM fails or is slow, use another.</li></ul><h3>Semantic Routing Architecture</h3><p>Kong AI Gateway offers a range of semantic capabilities including Caching and Prompt Guard. To implement the Multi-LLM Agent infrastructure we are going to use the Semantic Routing capability provided by the AI Proxy Advanced plugin we’ve been using for the entire post.</p><p>The AI Proxy Advanced Plugin has the ability to implement various load balancing policies, including distributing requests based on semantics or similarity between the prompts and description of each model. For example, consider that you have three models: the first one has been trained in sports, the second in music and the third one in science. What we want to do is route the requests accordingly based on the topic each prompt has presented.</p><p>What happens is that, during configuration time, done by, for example, submitting decK declarations to Konnect Control Plane, the plugin hits the embeddings model for each description and stores the embeddings into the Vector database.</p><p>Then, for each incoming request, the plugin submits a VSS (or Virtual Similarity Search) to the Vector Database to decide to which LLM the request should be routed to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*P1zgwBGLhCBMOyT5" /></figure><p><em>Semantic Routing configuration and request processing times</em></p><h3>Redis</h3><p>To implement the Semantic Routing architecture we are going to use the <a href="https://github.com/redis-stack/helm-redis-stack">Redis-stack Helm Charts</a> to Redis as our Vector Database.</p><pre>helm repo add redis-stack https://redis-stack.github.io/helm-redis-stack<br>help repo update<br>helm install redis-stack redis-stack/redis-stack -n redis --create-namespace</pre><h3>Ollama</h3><p>As our Embedding model we are going to consumer the “mxbai-embed-large:latest” model handled locally by Ollama. Use the <a href="https://github.com/otwld/ollama-helm">Ollama Helm Charts</a> to install it.</p><pre>helm repo add ollama-helm https://otwld.github.io/ollama-helm/<br>helm repo update<br>helm install ollama ollama-helm/ollama \<br>-n ollama \<br>  --create-namespace \<br>  --set ollama.models.pull[0]=&quot;mxbai-embed-large:latest&quot; \<br>  --set service.type=LoadBalancer</pre><h3>Python Script</h3><p>In this final AI Agent Python script we have two main changes:</p><ul><li>We have replaced the Tools with new functions:</li><li>“get_music”: consumes the <a href="https://eventregistry.org/">Event Registry</a> service looking for music concerts.</li><li>“get_traffic”: it sends requests to <a href="https://tavily.com/">Tavily</a> service for traffic information.</li><li>“get_weather”: it remains the same, related to the <a href="https://openweathermap.org/">OpenWeather</a> public service.</li><li>Replaces the LangGraph calls to build the graph with another LangGraph pre-built function, “<a href="https://langchain-ai.github.io/langgraph/reference/agents/?h=create_react#langgraph.prebuilt.chat_agent_executor.create_react_agent">create_react_agent</a>”.</li></ul><p>The pre-built function “create_react_agent” is very helpful to implement the fundamental ReAct graph we created programmatically before. That is, the agent is composed by:</p><ul><li>A Node sending requests to the LLM</li><li>A “conditional_edge” associated with this Node and making decisions about how the Agent should proceed when getting a response from the LLM.</li><li>A Node to call Tools</li></ul><p>In fact, if you print the output of the graph with “graph.get_graph().draw_ascii())” function again, you will see the same graph structure we had in the previous version of the agent.</p><pre>from langgraph.prebuilt import create_react_agent<br>from langchain_openai import ChatOpenAI<br>from langchain_core.tools import tool<br>from langchain_community.utilities.openweathermap import OpenWeatherMapAPIWrapper<br>import httpx<br><br><br>@tool<br>def get_weather(location: str):<br>  &quot;&quot;&quot;Call to get the weather from a specific location.&quot;&quot;&quot;<br>  print(&quot;starting get_weather function&quot;)<br>  openweathermap_url = kong_dp + &quot;/openweathermap-route&quot;<br>  result = httpx.get(openweathermap_url, params={&quot;q&quot;: location})<br>  print(&quot;finishing get_weather function&quot;)<br>  return result.json()<br><br><br>@tool<br>def get_music_concert(location: str):<br>  &quot;&quot;&quot;Call to get the events in a given location.&quot;&quot;&quot;<br>  print(&quot;starting get_music_concerts function&quot;)<br>  searchevent_url = kong_dp + &quot;/searchevent-route&quot;<br>  location = location.replace(&quot; &quot;, &quot;_&quot;)<br>  data={<br>    &quot;query&quot;: {<br>      &quot;$query&quot;: {<br>        &quot;$and&quot;: [<br>          {<br>            &quot;categoryUri&quot;: &quot;dmoz/Arts/Music/Bands_and_Artists&quot;<br>          },<br>          {<br>            &quot;locationUri&quot;: f&quot;http://en.wikipedia.org/wiki/{location}&quot;<br>          }<br>        ]<br>      },<br>      &quot;$filter&quot;: {<br>        &quot;forceMaxDataTimeWindow&quot;: &quot;31&quot;<br>      }<br>    },<br>    &quot;resultType&quot;: &quot;events&quot;,<br>    &quot;eventsSortBy&quot;: &quot;date&quot;,<br>    &quot;eventImageCount&quot;: 1,<br>    &quot;storyImageCount&quot;: 1<br>  }<br>  result = httpx.post(searchevent_url, json=data)<br>  print(&quot;finishing get_music_concert function&quot;)<br>  return result.json()[&quot;events&quot;][&quot;results&quot;][0][&quot;concepts&quot;][0][&quot;label&quot;][&quot;eng&quot;]<br><br><br>@tool<br>def get_traffic(location: str):<br>  &quot;&quot;&quot;Call to get the traffic situation of a given location.&quot;&quot;&quot;<br>  print(&quot;starting get_traffic function&quot;)<br>  traffic_url = kong_dp + &quot;/tavily-traffic-route&quot;<br>  data={&quot;query&quot;: f&quot;Generally, what is the worst time of day for car traffic in {location}&quot;, &quot;search_depth&quot;: &quot;advanced&quot;}<br>  result = httpx.post(traffic_url, json=data)<br>  print(&quot;finishing get_traffic function&quot;)<br>  return result.json()[&quot;results&quot;][0][&quot;content&quot;]<br><br><br><br>tools = [get_weather, get_music_concert, get_traffic]<br><br><br>kong_dp = &quot;http://127.0.0.1&quot;<br>agent_url = kong_dp + &quot;/agent-route&quot;<br><br>client = ChatOpenAI(base_url=agent_url, model=&quot;&quot;, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})<br><br><br>graph = create_react_agent(client, tools)<br>print(graph.get_graph().draw_ascii())<br><br><br>def print_stream(stream):<br>  for s in stream:<br>    message = s[&quot;messages&quot;][-1]<br>    if isinstance(message, tuple):<br>      print(message)<br>    else:<br>      message.pretty_print()<br><br><br>inputs = {&quot;messages&quot;: [(&quot;user&quot;, &quot;In my next vacation, I&#39;m planning to visit the city where Jimi Hendrix was born? Is there any music concert to see? Also provide weather and traffic information about the city&quot;)]}<br>print_stream(graph.stream(inputs, stream_mode=&quot;values&quot;))</pre><p>For this execution, the AI Proxy Advanced Plugin will route the request to Mistral, since it’s related to music.</p><h3>decK Declaration</h3><p>Below you can check the new decK declaration for the Semantic Routing use case. The AI Proxy Advanced plugin has the following sections configured:</p><ul><li>embeddings: where the plugin should go to generate embeddings related to the LLM models.</li><li>vectordb: responsible for storing the embeddings and handling the VSS queries.</li><li>targets: an entry for each LLM model. The most important setting is the “description” where we define where the plugin should route the requests to.</li></ul><p>Besides, the declaration applies the <a href="https://docs.konghq.com/hub/kong-inc/ai-prompt-decorator/">AI Prompt Decorator</a> plugin so the Gateway asks the LLM to convert temperatures to Celsius.</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - agent<br>_konnect:<br>  control_plane_name: ai-gateway<br>services:<br>- name: openweathermap-service<br>  url: https://api.openweathermap.org/data/2.5/weather<br>  routes:<br>  - name: openweathermap-route<br>    paths:<br>    - /openweathermap-route<br>    plugins:<br>    - name: request-transformer<br>      instance_name: request-transformer-openweathermap<br>      config:<br>        add:<br>          querystring:<br>          - &quot;appid:&lt;your_openweathermap_api_key&gt;&quot;<br>- name: searchevent-service<br>  url: https://eventregistry.org/api/v1/event/getEvents<br>  routes:<br>  - name: searchevent-route<br>    paths:<br>    - /searchevent-route<br>    plugins:<br>    - name: request-transformer<br>      instance_name: request-transformer-events<br>      config:<br>        add:<br>          body:<br>          - &quot;apiKey:&lt;your_event_search_api_key&gt;&quot;<br>- name: tavily-trafficservice<br>  url: https://api.tavily.com/search<br>  routes:<br>  - name: tavily-traffic-route<br>    paths:<br>    - /tavily-traffic-route<br>    plugins:<br>    - name: request-transformer<br>      instance_name: request-transformer-tavily-traffic<br>      config:<br>        add:<br>          headers:<br>          - &quot;Authorization:Bearer &lt;your_tavely_api_key&gt;&quot;<br>- name: agent-service<br>  host: httpbin.default<br>  port: 8000<br>  routes:<br>  - name: agent-route1<br>    paths:<br>    - /agent-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: &quot;ai-proxy-advanced-agent&quot;<br>      enabled: true<br>      config:<br>        balancer:<br>          algorithm: semantic<br>        embeddings:<br>          model:<br>            provider: openai<br>            name: mxbai-embed-large<br>            options:<br>              upstream_url: &quot;http://ollama.ollama:11434/v1/embeddings&quot;<br>        vectordb:<br>          dimensions: 1024<br>          distance_metric: cosine<br>          strategy: redis<br>          threshold: 0.8<br>          redis:<br>            host: redis-stack.redis.svc.cluster.local<br>            port: 6379<br>        targets:<br>        - model:<br>            provider: &quot;openai&quot;<br>            name: &quot;o3-mini&quot;<br>          auth:<br>            header_name: &quot;Authorization&quot;<br>            header_value: &quot;Bearer &lt;your_openai_api_key&gt;&quot;<br>          route_type: &quot;llm/v1/chat&quot;<br>          description: &quot;sports, soccer, football, basketball, beisebol&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true<br>        - model:<br>            provider: &quot;mistral&quot;<br>            name: &quot;mistral-large-latest&quot;<br>            options:<br>              mistral_format: &quot;openai&quot;<br>              upstream_url: &quot;https://api.mistral.ai/v1/chat/completions&quot;<br>          auth:<br>            header_name: &quot;Authorization&quot;<br>            header_value: &quot;Bearer &lt;your_mistral_api_key&gt;&quot;<br>          route_type: &quot;llm/v1/chat&quot;<br>          description: &quot;piano, orchestra, liszt, classical music, rock, pop music&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true<br>        - model:<br>            provider: &quot;anthropic&quot;<br>            name: &quot;claude-3–7-sonnet-20250219&quot;<br>            options:<br>              anthropic_version: &quot;2023–06–01&quot;<br>              max_tokens: 200<br>          auth:<br>            header_name: &quot;x-api-key&quot;<br>            header_value: &quot;&lt;your_anthropic_api_key&gt;&quot;<br>          route_type: &quot;llm/v1/chat&quot;<br>          description: &quot;science, physics, relativity theory, mathematics, algebra, calculus, trigonometry&quot;<br>          logging:<br>            log_payloads: true<br>            log_statistics: true<br>    - name: ai-prompt-decorator<br>      instance_name: ai-prompt-decorator1<br>      enabled: true<br>      config:<br>        prompts:<br>          prepend:<br>          - role: system<br>            content: &quot;You are a helpful AI assistant, please convert temperatures to Celsius.&quot;<br>    - name: key-auth<br>      instance_name: key-auth1<br>      enabled: true<br>consumers:<br>- keyauth_credentials:<br>  - key: &quot;123456&quot;<br>  username: user1<br>  plugins:<br>  - name: ai-rate-limiting-advanced<br>    instance_name: ai-rate-limiting-advanced-consumer1<br>    enabled: true<br>    config:<br>      llm_providers:<br>      - name: openai<br>        window_size:<br>        - 60<br>        limit:<br>        - 800</pre><h3>Grafana Dashboards</h3><p>Download and install the <a href="https://github.com/CAcquaviva/kong-ai-gateway-langgraph/blob/main/grafana/Multi-LLM.json">Grafana Dashboard</a> available in the GitHub repository. It has two tiles:</p><ul><li>Counter of requests for each Kong Route.</li><li>Counter of requests for each LLM model.</li></ul><p>The dashboard is totally based on the metrics generated by the Prometheus plugin. The configuration is divided into two parts:</p><ul><li>AI Proxy Advanced plugin with the following parameters</li></ul><pre>logging:<br>  log_payloads: true<br>  log_statistics: true</pre><ul><li>Prometheus plugin with the parameter</li></ul><pre>ai_metrics: true</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bBFl02aqeLTnSrws" /></figure><p><em>Grafana Dashboard based on the metrics generated by the Prometheus plugin</em></p><h3>LangGraph Server</h3><p>Now that we have our final version of the AI Agent it is time to build a <a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_server/">LangGraph Server</a> based on it. You have multiple <a href="https://langchain-ai.github.io/langgraph/tutorials/deployment/#deployment-options">deployments</a> options to run your LangGraph Server but we are going to do in our own Minikube cluster in a deployment called “<a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_standalone_container/">Standalone Container</a>”</p><p><a href="https://langchain-ai.github.io/langgraph/cloud/reference/api/api_ref.html">https://langchain-ai.github.io/langgraph/cloud/reference/api/api_ref.html</a></p><p><a href="https://github.com/langchain-ai/helm/blob/main/charts/langgraph-cloud/README.md">https://github.com/langchain-ai/helm/blob/main/charts/langgraph-cloud/README.md</a></p><h3>Agent Docker Image</h3><p>The first step is to create the Docker image for the Server. The code below removes the lines where we execute the graph. Another change is for the Kong Data Plane address, referring to the Kubernetes FQDN Service.</p><pre>from langgraph.prebuilt import create_react_agent<br>from langchain_openai import ChatOpenAI<br>from langchain_core.tools import tool<br>from langchain_community.utilities.openweathermap import OpenWeatherMapAPIWrapper<br>import httpx<br><br><br>@tool<br>def get_weather(location: str):<br>  &quot;&quot;&quot;Call to get the weather from a specific location.&quot;&quot;&quot;<br>  print(&quot;calling get_weather function&quot;)<br>  openweathermap_url = kong_dp + &quot;/openweathermap-route&quot;<br>  result = httpx.get(openweathermap_url, params={&quot;q&quot;: location})<br>  return result.json()<br><br><br>@tool<br>def get_music_concerts(location: str):<br>  &quot;&quot;&quot;Call to get the events in a given location.&quot;&quot;&quot;<br>  print(&quot;calling get_music_concerts function&quot;)<br>  searchevent_url = kong_dp + &quot;/searchevent-route&quot;<br>  location = location.replace(&quot; &quot;, &quot;_&quot;)<br>  data={<br>    &quot;query&quot;: {<br>      &quot;$query&quot;: {<br>        &quot;$and&quot;: [<br>          {<br>            &quot;categoryUri&quot;: &quot;dmoz/Arts/Music/Bands_and_Artists&quot;<br>          },<br>          {<br>            &quot;locationUri&quot;: f&quot;http://en.wikipedia.org/wiki/{location}&quot;<br>          }<br>        ]<br>      },<br>      &quot;$filter&quot;: {<br>      &quot;forceMaxDataTimeWindow&quot;: &quot;31&quot;<br>    }<br>  },<br>  &quot;resultType&quot;: &quot;events&quot;,<br>  &quot;eventsSortBy&quot;: &quot;date&quot;,<br>  &quot;eventImageCount&quot;: 1,<br>  &quot;storyImageCount&quot;: 1<br>  }<br>  result = httpx.post(searchevent_url, json=data)<br>  return result.json()[&quot;events&quot;][&quot;results&quot;][0][&quot;concepts&quot;][0][&quot;label&quot;][&quot;eng&quot;]<br><br><br>@tool<br>def get_traffic(location: str):<br>  &quot;&quot;&quot;Call to get the traffic situation of a given location.&quot;&quot;&quot;<br>  print(&quot;calling get_traffic function&quot;)<br>  traffic_url = kong_dp + &quot;/tavily-traffic-route&quot;<br>  data={&quot;query&quot;: f&quot;Generally, what is the worst time of day for car traffic in {location}&quot;, &quot;search_depth&quot;: &quot;advanced&quot;}<br>  result = httpx.post(traffic_url, json=data)<br>  return result.json()[&quot;results&quot;][0][&quot;content&quot;]<br><br><br>tools = [get_weather, get_music_concerts, get_traffic]<br>#kong_dp = &quot;http://127.0.0.1&quot;<br>kong_dp = &quot;http://proxy1.kong&quot;<br>agent_url = kong_dp + &quot;/agent-route&quot;<br><br>client = ChatOpenAI(base_url=agent_url, model=&quot;&quot;, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})<br><br><br>graph = create_react_agent(client, tools)</pre><h4>langgraph.json</h4><p>The Docker image requires a “langgraph.json” file with the dependencies and the name of the graph variable inside the code, in our case “graph”.</p><pre>{<br>  &quot;dependencies&quot;: [<br>    &quot;.&quot;,<br>    &quot;langchain_openai&quot;,<br>    &quot;langchain_community&quot;<br>  ],<br>  &quot;graphs&quot;: {<br>    &quot;agent&quot;: &quot;./langgraph_react_kong.py:graph&quot;<br>  }<br>}</pre><h4>Docker image creation</h4><p>Create the image with the <a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_cli/?h=langgraph+cli">“langgraph” CLI</a> command. It requires Docker installed in your environment.</p><pre>langgraph build --platform linux/arm64 -t claudioacquaviva/langgraph-kong1</pre><p>or</p><pre>langgraph build --platform linux/amd64 -t claudioacquaviva/langgraph-kong1</pre><p>Push it to Docker Hub:</p><pre>docker push claudioacquaviva/langgraph-kong1</pre><h3>Agent Deployment</h3><p>Install your LangGraph Service using the Helm Chart available:</p><pre>helm repo add langchain https://langchain-ai.github.io/helm/</pre><p>The “values.yaml” defines the Service as “LoadBalancer” to make it available. Currently, only Postgres is supported as a database for LangGraph Server and Redis as the task queue. The file specifies Postgres resources for its Kubernetes deployment. Finally, LangGraph Server requires a <a href="https://docs.smith.langchain.com/">LangSmith</a> API Key. LangSmith is a platform used to monitor your server. Log to <a href="https://smith.langchain.com/">LangSmith</a> and create your API Key.</p><pre>cat &gt; values.yaml &lt;&lt; &#39;EOF&#39;<br>images:<br>  apiServerImage:<br>    pullPolicy: Always<br>    repository: claudioacquaviva/langgraph-kong1<br>    tag: latest<br><br>config:<br>  langGraphCloudLicenseKey: &quot;&quot;<br><br>studio:<br>  enabled: false<br><br>postgres:<br>  statefulSet:<br>    resources:<br>      limits:<br>        cpu: 500m<br>        memory: 1Gi<br>      requests:<br>        cpu: 500m<br>        memory: 1Gi<br><br>apiServer:<br>  service:<br>    type: LoadBalancer<br>    httpPort: 8090<br>  deployment:<br>    extraEnv:<br>      - name: LANGSMITH_API_KEY<br>        value: &quot;&lt;your_langsmith_api_key&gt;&quot;<br>EOF</pre><p>Deploy the LangGraph Server:</p><pre>helm install langgraph-cloud langchain/langgraph-cloud \<br>  -n langgraph \<br>  --create-namespace \<br>  --values values.yaml</pre><p>If you want to uninstall it run:</p><pre>helm uninstall langgraph-cloud -n langgraph<br>kubectl delete pvc data-langgraph-cloud-postgres-0 -n langgraph</pre><h3>LangGraph Server API</h3><p>If the LangGraph Server deployed you can use its <a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_server/#langgraph-server-api">API</a> to send requests to your graph:</p><p>Look for your assistants with:</p><pre>% curl -s -X POST http://0.0.0.0:8090/assistants/search \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>  &quot;metadata&quot;: {},<br>  &quot;graph_id&quot;: &quot;&quot;,<br>  &quot;limit&quot;: 10,<br>  &quot;offset&quot;: 0<br>}&#39; | jq</pre><p>The expected response is:</p><pre>[<br>  {<br>    &quot;assistant_id&quot;: &quot;fe096781–5601–53d2-b2f6–0d3403f7e9ca&quot;,<br>    &quot;graph_id&quot;: &quot;agent&quot;,<br>    &quot;created_at&quot;: &quot;2025–04–28T15:26:37.723363+00:00&quot;,<br>    &quot;updated_at&quot;: &quot;2025–04–28T15:26:37.723363+00:00&quot;,<br>    &quot;config&quot;: {},<br>    &quot;metadata&quot;: {<br>      &quot;created_by&quot;: &quot;system&quot;<br>    },<br>    &quot;version&quot;: 1,<br>    &quot;name&quot;: &quot;agent&quot;,<br>    &quot;description&quot;: null<br>  }<br>]</pre><p>Use the assistant’s name to invoke graph:</p><pre>curl -s http://localhost:8090/runs/wait \<br>--header &#39;Content-Type: application/json&#39; \<br>--data &#39;{<br>  &quot;assistant_id&quot;: &quot;agent&quot;,<br>  &quot;input&quot;: {<br>    &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;In my next vacation, I&#39;&#39;m planning to visit the city where Jimi Hendrix was born? Is there any music concert to see? Also provide weather and traffic information about the city.&quot;}]<br>  }<br>}&#39; | jq -r &#39;.messages[5].content&#39;</pre><p>Expected response</p><pre>In Seattle, it is currently overcast with a temperature of 69.8°F (20.4°C) and feels like 68.9°F (20.5°C). The city has a humidity of 80% and wind speed of 4.12 mph from the west. There is a music concert of Phish happening in the city, but be aware that the worst period of travel is generally Thursday afternoons, especially 4–6 pm.</pre><h3>Kong AI Gateway 3.11 and new GenAI models support</h3><p>With Kong AI Gateway 3.11 we will be able to support other GenAI infrastructures besides LLMs, including video, images, etc. The following diagram lists the new modes supported:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gjCAEiFrX_YQ3ZAa" /></figure><p>For example, here is an example of a Kong Route declaration with the AI Proxy Advanced plugin enabled to protect the text-to-image <a href="https://openai.com/index/dall-e-2/">OpenAI’s Dall-E 2 model</a>,</p><pre>- name: img-openai<br>  strip_path: true<br>  paths:<br>  - &quot;/img/openai&quot;<br>  plugins:<br>  - name: ai-proxy-advanced<br>    config:<br>      genai_category: image/generation<br>      targets:<br>      - route_type: image/v1/images/generations<br>        auth:<br>          header_name: &quot;Authorization&quot;<br>          header_value: &quot;&quot;<br>          allow_override: false<br>        logging:<br>          log_payloads: false<br>          log_statistics: true<br>        model:<br>          name: dall-e-2<br>          provider: openai</pre><p>In order to do it, Kong AI Gateway 3.11 defines new configuration parameters like:</p><ul><li>genai-category: used to configure the GenAI infrastructure the gateway protects. Besides image/generation, it supports, for example, text/generation and text/embeddings for regular LLMs and Embeddings models, audio/speech and audio/transcription for audio based models implementing speech recognition, audio-to-text, etc.</li><li>route_type: this existing parameter has been extended to support new types like:</li><li>LLM: llm/v1/responses, llm/v1/assistants, llm/v1/files and llm/v1/batches</li><li>Image: image/v1/images/generations, image/v1/images/edits</li><li>Audio: audio/v1/audio/speech, audio/v1/audio/transcriptions and audio/v1/audio/translations</li><li>Realtime: realtime/v1/realtime</li></ul><h3>Conclusion</h3><p>This blog post has presented a basic AI Agent using Kong AI Gateway and LangGraph. Redis was used as a Vector Database as a local Ollama was the infrastructure providing Embedding Model.</p><p>Behind the Gateway we have three LLM infrastructures (OpenAI, Mistral and Anthropic) and three external functions were used as Tools by the AI Agent.</p><p>The Gateway was responsible for abstracting the LLM infrastructures and protecting the external functions with specific policies including Rate Limiting and API Keys.</p><p>You can discover all the features available on the <a href="https://konghq.com/products/kong-ai-gateway">Kong AI Gateway</a> product page to learn more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7429c9c1f5ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OTel Demo App with Kong Konnect and Dynatrace]]></title>
            <link>https://medium.com/@claudioacquaviva/otel-demo-app-with-kong-konnect-and-dynatrace-cb1db304b34a?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb1db304b34a</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Mon, 12 May 2025 13:28:46 GMT</pubDate>
            <atom:updated>2025-05-13T12:13:25.024Z</atom:updated>
            <content:encoded><![CDATA[<p>The <a href="https://opentelemetry.io/docs/demo/">OpenTelemetry (OTel) Demo App</a> is an illustrative application designed to showcase how <a href="https://opentelemetry.io/">OpenTelemetry</a> can be used to observe, monitor, and troubleshoot distributed systems. Built around a microservices architecture, the app simulates a real-world cloud-native environment, making it an ideal learning tool for anyone exploring observability practices. The application is available in a specific <a href="https://github.com/open-telemetry/opentelemetry-demo">GitHub repository</a>.</p><p>The OTel Demo App implements the “Astronomy Shop”, a microservice-based e-commerce site. Here’s the application <a href="https://opentelemetry.io/docs/demo/architecture/">architecture diagram</a> taken from the official OpenTelemetry site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wgOxhPP7HlVOUTtv" /></figure><p>The most notable aspect of the diagram is the presence of the “Frontend” microservice. As expected, the microservice is responsible for calling all microservices responsible for the e-commerce business logic, such as “Product Catalog Service”, “Cart Service”, etc. The e-commerce UI is totally managed by the frontend microservices, including not just “Frontend” but also “Frontend Proxy”, “Image Provider”, etc.</p><p>In a more realistic scenario we should have an API Gateway mediating the requests coming from Frontend microservice. That would add several benefits:</p><ul><li>Request processing consolidation by the Gateway</li><li>Gateway should implement specific policies to control the backend microservices consumption such as AuthN/AuthZ, RateLimiting, Transformation, etc.</li><li>Multiple frontends could be more easily developed</li><li>The frontend and backend microservices could be deployed in different and hybrid environments.</li></ul><p>The new topology is a natural evolution of the original one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jURf8d65Eqz_oUtx" /></figure><p>That is possible because the App has been developed in a modular way and we can configure the Frontend microservice with new services addresses. Since Kong natively supports gRPC, the communication with Frontend and all backend services is straightforward and totally transparent from the Frontend code.</p><p>From the Observability perspective the App and Kong Gateway reach out to Dynatrace through the OpenTelemetry Collector and specific plugins Kong provides.</p><p>This blog post describes how to inject the Kong Gateway and get the Frontend and Backend layers running in totally independent environments. The exercise was built in an Amazon EKS Cluster with the layers deployed in different Kubernetes Namespaces. For an even more distributed environment, more runtimes could be used also, however, to make things a bit simpler, the layers are running in the same cluster.</p><h3>Reference Architecture</h3><p>To get started, here’s the Reference Architecture we are going to use for this exercise. The layers, running in their own Kubernetes Namespaces, are:</p><ul><li>Kong Konnect Data Plane (managed by Kong Konnect Control Plane)</li><li>OTel Demo App Frontend microservices</li><li>OTel Demo App Backend microservices</li><li>OTel Operator and Collector</li></ul><p>The Dynatrace tenant receives all OTel signals directly from the OTel Connector. There are no direct connections between Kong and Dynatrace. The same thing applies to the App itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*OCj5r6xoCVdvkECA" /></figure><h3>Kubernetes cluster creation</h3><p>Let’s create the Amazon EKS cluster using <a href="https://eksctl.io/">eksctl</a>, like this:</p><pre>eksctl create cluster --name kong-dynatrace-otel \<br> --version 1.32 \<br> --region us-east-2 \<br> --nodegroup-name kong-node \<br> --node-type m4.xlarge \<br> --nodes 1</pre><h3>OTel Demo Application Backend Services deployment</h3><p>To better understand our new deployment, we are going to deploy the Backend Services first and consume them directly, just like any Frontend Service should do. Taking advantage of the <a href="https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-demo">Open Telemetry Demo Helm Charts</a> provided by the Open Telemetry project, this should be a straightforward process. Add the new repo first:</p><pre>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts<br>helm repo update</pre><p>Install the Backend Service with the following command:</p><pre>helm install otel-demo-app open-telemetry/opentelemetry-demo \<br>-n open-telemetry \<br> --create-namespace \<br> --set opentelemetry-collector.enabled=false \<br> --set components.loadgenerator.enabled=false \<br> --set components.frontendProxy.enabled=false \<br> --set components.frontend.enabled=false \<br> --set components.imageprovider.enabled=false \<br> --set jaeger.enabled=false \<br> --set prometheus.enabled=false \<br> --set grafana.enabled=false</pre><p>The command disables all Frontend services and, since we are going to leverage Dynatrace technologies, all default Observability services as well. Check the deployment with the following command. Note all Kubernetes services are defined as type “ClusterIP”, which makes total sense since they are going to be exposed by the Kong API Gateway.</p><pre>kubectl get all -n open-telemetry</pre><h3>Consume the gRPC-based Backend Services</h3><p>As depicted in the architecture diagram, all Backend Services are consumable through their <a href="https://grpc.io/">gRPC</a> endpoints. As an exercise, let’s hit some of them playing a Frontend role. The following command starts a new Pod with Ubuntu Operating System with several CLIs, including <a href="https://github.com/fullstorydev/grpcurl">grpcurl</a>, which is a curl tool for gRPC servers. Again, we are going to add Kong API Gateway between the Frontend microservice and all Backend Services, so the Gateway will send similar requests to them.</p><pre>kubectl run --rm=true -i --tty ubuntu --image=claudioacquaviva/ubuntu-allclis:0.1 -- /bin/bash</pre><p>All <a href="https://grpc.io/docs/what-is-grpc/introduction/">gRPC services</a> are described in the demo.proto <a href="https://protobuf.dev/overview/">Protocol Buffer</a> file available in the OpenTelemetry Demo repo. You can download it with:</p><pre>wget https://raw.githubusercontent.com/open-telemetry/opentelemetry-demo/refs/heads/main/pb/demo.proto</pre><p>The demo.proto file defines an oteldemo package where all gRPC services reside:</p><pre>package oteldemo;</pre><p>And, as an example, here’s the snippet for the Product Catalog Service declaration:</p><pre>service ProductCatalogService {<br>  rpc ListProducts(Empty) returns (ListProductsResponse) {}<br>  rpc GetProduct(GetProductRequest) returns (Product) {}<br>  rpc SearchProducts(SearchProductsRequest) returns (SearchProductsResponse) {}<br>}</pre><p>Taking this service specification we can send requests to it with:</p><pre># grpcurl -plaintext otel-demo-app-productcatalogservice.open-telemetry:8080 oteldemo.ProductCatalogService.ListProducts | jq &#39;.products[].name&#39;</pre><p>Where:</p><ul><li>-plaintext option: use regular HTTP/2 connection with no TLS.</li><li>otel-demo-app-productcatalogservice.open-telemetry:8080: that’s the Kubernetes FQDN and port of the Product Catalog Service.</li><li>oteldemo.ProductCatalogService.ListProducts: the package, service and method we want to call.</li></ul><p>The command should return something like:</p><pre>&quot;National Park Foundation Explorascope&quot;<br>&quot;Starsense Explorer Refractor Telescope&quot;<br>&quot;Eclipsmart Travel Refractor Telescope&quot;<br>&quot;Lens Cleaning Kit&quot;<br>&quot;Roof Binoculars&quot;<br>&quot;Solar System Color Imager&quot;<br>&quot;Red Flashlight&quot;<br>&quot;Optical Tube Assembly&quot;<br>&quot;Solar Filter&quot;<br>&quot;The Comet Book&quot;</pre><h3>OTel Demo Backend Service exposure strategy</h3><p>With the Backend Services deployed it’s important to understand how they are going to be exposed by Kong Gateway and consumed by the Frontend microservice. Here’s a diagram illustrating the strategy. We’ve taken the Product Catalog as an example. All other Backend Services follow it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*NxCJRD4hhyIr0QvD" /></figure><p>In a “bottom-up” approach the strategy goes like this:</p><ul><li>The “Product Catalog” Backend Service has a Kong Service defined as a reference to it.</li><li>On top of the Kong Service, there is a host-based Kong Route exposing it.</li><li>The Kong Route has a peer Kubernetes Service defined.</li><li>The Frontend microservice sends requests using the Kubernetes Service.</li></ul><p>The Kubernetes Service is necessary, so Kong can identify which Kong Route it should process and, therefore, which Kong Service and the related Backend Service the request should be routed to.</p><p>The protocol used to get the components communicating remains as the original gRPC.</p><h3>Kong Konnect Data Plane deployment</h3><p>Now, it’s time to describe how to add the Kong Data Plane mediating the calls between the Frontend microservices and Backend Services. We can summarize it with the following steps:</p><ol><li>Pre-requisites: Kong Konnect registration</li><li>Kong Konnect Control Plane creation and Data Plane deployment</li><li>Kubernetes Services for OTel Demo Backend Services</li><li>Kong Objects creation</li></ol><h3>Pre-requisites: Kong Konnect registration</h3><p>Before deploying the Data Plane we should subscribe to Konnect. Click on the <a href="https://konghq.com/products/kong-konnect/register">Konnect Registration</a> link and present your credentials. Or, if you already have a Konnect subscription, <a href="https://cloud.konghq.com/us">log in to it</a>. You should get redirected to the Konnect landing page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7hdOyAYdLdN4bRLB" /></figure><h3>Kong Konnect Control Plane creating and Data Plane deployment</h3><p>Click on “Gateway Manager” and on the “+ Create a gateway” button. Make sure you have the “Self-Managed Hybrid” option selected, type “OTel” in the “Name” box and click on the “Next Step” button.</p><p>Inside the “Create a Data Plane Node” page, choose “Kubernetes” for the platform and click on “Generate certificate”. Copy and save the Digital Certificate and Private Key as tls.crt and tls.key files as described in the instructions. Also copy and save the configuration parameters in a values.yaml file. Change the values.yaml file as described below.</p><p>The main comments here are:</p><ul><li>Replace the cluster_* endpoints and server names with yours.</li><li>The “tracing_instrumentations: all” and “tracing_sampling_rate: 1.0” parameters are needed for the Kong Gateway OpenTelemetry plugin we are going to describe later.</li><li>The Kong Data Plane is going to be consumed exclusively by the Frontend microservice, therefore it can be deployed as “type: ClusterIP”.</li><li>The “status” section enables the “status” port, as default 8100. That’s the port Konnect Data Plane publishes the Prometheus-based metrics related to API requests and consumption.</li></ul><pre>cat &gt; values.yaml &lt;&lt; &#39;EOF&#39;<br>image:<br>  repository: kong/kong-gateway<br>  tag: &quot;3.9&quot;<br><br>secretVolumes:<br>- kong-cluster-cert<br><br>admin:<br>  enabled: false<br><br>manager:<br>  enabled: false<br><br>env:<br>  role: data_plane<br>  database: &quot;off&quot;<br>  cluster_mtls: pki<br>  cluster_control_plane: abcdefc352.us.cp0.konghq.com:443<br>  cluster_server_name: abcdefc352.us.cp0.konghq.com<br>  cluster_telemetry_endpoint: abcdefc352.us.tp0.konghq.com:443<br>  cluster_telemetry_server_name: abcdefc352.us.tp0.konghq.com<br>  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt<br>  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key<br>  lua_ssl_trusted_certificate: system<br>  konnect_mode: &quot;on&quot;<br>  vitals: &quot;off&quot;<br>  nginx_worker_processes: &quot;1&quot;<br>  upstream_keepalive_max_requests: &quot;100000&quot;<br>  nginx_http_keepalive_requests: &quot;100000&quot;<br>  proxy_access_log: &quot;off&quot;<br>  dns_stale_ttl: &quot;3600&quot;<br>  tracing_instrumentations: all<br>  tracing_sampling_rate: 1.0<br><br>ingressController:<br>  enabled: false<br>  installCRDs: false<br><br>resources:<br>  requests:<br>    cpu: 1<br>    memory: &quot;2Gi&quot;<br><br>proxy:<br>  type: ClusterIP<br><br>status:<br>  enabled: true<br>  http:<br>    enabled: true<br>    containerPort: 8100<br>    parameters: []<br>EOF</pre><p>Now, use the Helm command to deploy the Data Plane. First add Kong’s repo to your Helm environment.</p><pre>helm repo add kong https://charts.konghq.com<br>helm repo update</pre><p>Create a namespace, secrets for your Digital Certificate and Private Key pair and apply the values.yaml file.</p><pre>kubectl create namespace kong<br>kubectl create secret tls kong-cluster-cert -n kong - cert=./tls.crt - key=./tls.key<br>helm install kong kong/kong -n kong - values ./values.yaml</pre><p>You should see a “Data Plane Node has been found” message at the bottom of the Konnect page. Close the page and click on “Data Plane Nodes” to check your first Data Plane running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gedUjzrX8-pAG062" /></figure><p>You can check Kong Data Plane logs with:</p><pre>kubectl logs -f $(kubectl get pod -n kong -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kong-&quot;))&#39; | jq -r &#39;.name&#39;) -n kong</pre><p>Check the Kubernetes Service, related to the Kong Data Plane, with:</p><pre>% kubectl get service -n kong -o json | jq &#39;.items[].metadata.name&#39;<br>&quot;kong-kong-proxy&quot;</pre><h3>Kubernetes Services</h3><p>The next step is to define the Kubernetes Services which are going to be consumed by the Frontend microservice and, in turn, route the requests to Kong API Gateway. To better illustrate the configuration consider the specific “Product Catalog Service” declaration:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-productcatalogservice<br>  namespace: kong<br>spec:<br>  selector:<br>  app.kubernetes.io/name: kong<br>type: ClusterIP<br>ports:<br>- protocol: TCP<br>  port: 80<br>  targetPort: 8000</pre><p>The declaration defines a Kubernetes Service named “otel-demo-app-productcatalogservice” in the “kong” namespace, where the Kong Gateway is running. The Frontend microservice, as any Kubernetes Service consumer, should send requests to the otel-demo-app-productcatalogservice.kong:80 endpoint and the Kubernetes Service is configure to route the request to the targetPort 8000, where Kong Gateway is listening to. You can easily check that with the following command:</p><pre>% kubectl get service kong-kong-proxy -n kong -o json | jq &#39;.spec.ports[0]&#39;<br>{<br>  &quot;name&quot;: &quot;kong-proxy&quot;,<br>  &quot;nodePort&quot;: 30858,<br>  &quot;port&quot;: 80,<br>  &quot;protocol&quot;: &quot;TCP&quot;,<br>  &quot;targetPort&quot;: 8000<br>}</pre><p>To create all Kubernetes Service submit the following declaration:</p><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-adservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-cartservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-checkoutservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-currencyservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-productcatalogservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-recommendationservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-demo-app-shippingservice<br>  namespace: kong<br>spec:<br>  selector:<br>    app.kubernetes.io/name: kong<br>  type: ClusterIP<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>EOF</pre><h3>decK and Kong Objects creation</h3><p>With the Kong Data Plane and Kubernetes Services in place, we have to tell Kong about the Backend Services, creating new Kong Objects. The Objects, responsible for exposing the OTel Demo Backend microservices, will be configured using <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong), a command line that allows you to manage Kong Objects in a declarative way. In order to use it, we need a Konnect PAT (Personal Access Token). Please, refer to the Konnect documentation to learn how to generate a <a href="https://docs.konghq.com/konnect/org-management/access-tokens/">PAT</a>.</p><p>The following decK declaration creates and configures for each OTel Demo Backend microservices:</p><ul><li>A Kong Service referring to the microservice Kubernetes service and namespace.</li><li>A Kong Route to expose the Kong Service. The Kong Route is referring to a Kubernetes Service, previously created, related to each OTel Demo Backend service.</li></ul><p>Again, let’s take the “Product Catalog Service” as an example. The declaration snippet defines a Kong Service named otel-demo-app-productcatalogservice. Like any other Kong Service it’s based on an upstream endpoint, which is, in this case, otel-demo-app-productcatalogservice.open-telemetry:8080, through gRPC protocol, where the actual Product Catalog Service is deployed.</p><p>The declaration also creates a Kong Route to this Kong Service. It has important settings:</p><ul><li>As expected, it exposes the Kong Service (and therefore the Product Catalog microservice) though gRPC protocol also.</li><li>This is a Host based Route, meaning it considers the host that has been mentioned in the gRCP request. The host configuration matches the Kubernetes Service previously created.</li></ul><pre>services:<br>- name: otel-demo-app-productcatalogservice<br>  protocol: grpc<br>  host: otel-demo-app-productcatalogservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-productcatalogservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-productcatalogservice.kong</pre><p>Now, to make sure decK is working properly, you can ping Konnect with:</p><pre>deck gateway ping --konnect-token $PAT</pre><p>Create all Kong Services and Routes with the following command:</p><pre>deck gateway sync - konnect-token $PAT - &lt;&lt;EOF<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - otel-demo<br>_konnect:<br>  control_plane_name: OTel<br>services:<br>- name: otel-demo-app-cartservice<br>  protocol: grpc<br>  host: otel-demo-app-cartservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-cartservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-cartservice.kong<br>- name: otel-demo-app-productcatalogservice<br>  protocol: grpc<br>  host: otel-demo-app-productcatalogservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-productcatalogservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-productcatalogservice.kong<br>- name: otel-demo-app-adservice<br>  protocol: grpc<br>  host: otel-demo-app-adservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-adservice-route<br>  protocols:<br>  - grpc<br>  - grpcs<br>  hosts:<br>  - otel-demo-app-adservice.kong<br>- name: otel-demo-app-checkoutservice<br>  protocol: grpc<br>  host: otel-demo-app-checkoutservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-checkoutservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-checkoutservice.kong<br>- name: otel-demo-app-currencyservice<br>  protocol: grpc<br>  host: otel-demo-app-currencyservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-currencyservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-currencyservice.kong<br>- name: otel-demo-app-recommendationservice<br>  protocol: grpc<br>  host: otel-demo-app-recommendationservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-recommendationservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-recommendationservice.kong<br>- name: otel-demo-app-shippingservice<br>  protocol: grpc<br>  host: otel-demo-app-shippingservice.open-telemetry<br>  port: 8080<br>  routes:<br>  - name: otel-demo-app-shippingservice-route<br>    protocols:<br>    - grpc<br>    - grpcs<br>    hosts:<br>    - otel-demo-app-shippingservice.kong<br>EOF</pre><p>You can destroy all Kong Objects resetting the Control Plane:</p><pre>deck gateway reset --konnect-token $PAT --konnect-control-plane-name &quot;OTel&quot; -f</pre><p>Check the Konnect UI to see the new Kong Gateway Services and Kong Routes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RXEaI4k-r4RPDfuu" /></figure><p>Here’s the Kong Route created for the Product Catalog Kong Service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6Ru7Np074SYwGjb" /></figure><h3>Consume the Kong Routes</h3><p>If we get back to the Ubuntu Pod we should be able to consume the Kong Routes and the Backend services sitting behind the Gateway. Try to consume the Kong Data Plane’s Kubernetes Service first:</p><pre># curl -i http://kong-kong-proxy.kong<br>HTTP/1.1 404 Not Found<br>Date: Tue, 28 Jan 2025 17:24:50 GMT<br>Content-Type: application/json; charset=utf-8<br>Connection: keep-alive<br>Content-Length: 103<br>X-Kong-Response-Latency: 0<br>Server: kong/3.9.0.0-enterprise-edition<br>X-Kong-Request-Id: c15fa71b7b61801fd5013456b582c264<br>{<br>  &quot;message&quot;:&quot;no Route matched with those values&quot;,<br>  &quot;request_id&quot;:&quot;c15fa71b7b61801fd5013456b582c264&quot;<br>}</pre><p>Now send gRPC requests to Kong to consume the Backend microservices. You should get the same response as you did when consuming the Backend microservice directly.</p><pre># grpcurl -plaintext otel-demo-app-productcatalogservice.kong:80 oteldemo.ProductCatalogService.ListProducts | jq &#39;.products[].name&#39;</pre><h3>Dynatrace registration</h3><p>The next step is related to the OTel Collector configuration and deployment. However, since it’s going to send data to Dynatrace we need to get a Dynatrace Tenant. So to the <a href="https://www.dynatrace.com/signup/">Dynatrace signup link</a> to register and get a 15 day trial. You should get redirected to its landing page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nLRgA4t0fnY69wDN" /></figure><h3>OpenTelemetry Collector deployment</h3><p>Now, we are ready to deploy the OpenTelemetry Collector. To get a better control over it, we are going to do it through the <a href="https://github.com/open-telemetry/opentelemetry-operator">OpenTelemetry Kubernetes Operator</a>. In fact, the collector is also capable of auto-instrument applications and services using OpenTelemetry instrumentation libraries.</p><h3>Installing Cert-Manager</h3><p>The OpenTelemetry Operator requires <a href="https://cert-manager.io/">Cert-Manager</a> to be installed in your Kubernetes cluster. The Cert-Manager can then issue certificates to be used by the communication between the Kubernetes API Server and the existing webhook included in the operator.</p><p>Use the <a href="https://cert-manager.io/docs/installation/helm/">Cert-Manager Helm Charts</a> to get it installed. Add the repo first:</p><pre>helm repo add jetstack https://charts.jetstack.io</pre><p>Install Cert-Manager with:</p><pre>helm install \<br>  cert-manager jetstack/cert-manager \<br>  --namespace cert-manager \<br>  --create-namespace \<br>  --version v1.16.3 \<br>  --set crds.enabled=true</pre><h3>Installing OpenTelemetry Operator</h3><p>Now we are going to use the <a href="https://github.com/open-telemetry/opentelemetry-helm-charts">OpenTelemetry Helm Charts</a> to install it. Add its repo:</p><pre>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts</pre><p>Install the operator:</p><pre>helm install opentelemetry-operator open-telemetry/opentelemetry-operator \<br>  --namespace opentelemetry-operator-system \<br>  --create-namespace \<br>  --set &quot;manager.collectorImage.repository=otel/opentelemetry-collector-k8s&quot; \<br>  --set admissionWebhooks.certManager.enabled=true</pre><p>The “admissionWebhooks” parameter asks Cert-Manager to generate a self-signed certificate. The operator installs a new CRD used to create a new OpenTelemetry Collector. You can check it out with:</p><pre>kubectl describe crd opentelemetrycollectors.opentelemetry.io</pre><h3>Kong Gateway Plugins and OpenTelemetry Collector Receivers</h3><p>The OTelCollector configuration is structured in a way each OpenTelemetry signal type is addressed by a receiver and getting fed by a specific Kong Plugin:</p><ul><li>“<a href="https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver">otlp receiver</a>”: receives traces produced by <a href="https://docs.konghq.com/hub/kong-inc/opentelemetry/">Kong OpenTelemetry Plugin</a>.</li><li>“<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md">prometheus receiver</a>”: scrapes the Kong Gateway endpoint where the Prometheus based metrics are published by the <a href="https://docs.konghq.com/hub/kong-inc/prometheus/">Kong Prometheus Plugin</a>.</li><li>“<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver">tcp receiver</a>”: receives the access (or transaction) logs produced by the <a href="https://docs.konghq.com/hub/kong-inc/tcp-log/">Kong TCP Log Plugin</a>.</li></ul><p>Here’s a diagram to illustrate it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*g7spRwMPXMI4eZys" /></figure><p>With the operator in place, we can use the new CRD to deploy our OpenTelemetry Collector. Here’s the declaration:</p><pre>cat &gt; otelcollector-dynatrace-all-signals.yaml &lt;&lt; &#39;EOF&#39;<br>apiVersion: opentelemetry.io/v1beta1<br>kind: OpenTelemetryCollector<br>metadata:<br>  name: collector-kong<br>  namespace: opentelemetry-operator-system<br>spec:<br>  image: otel/opentelemetry-collector-contrib:0.112.0<br>  serviceAccount: collector<br>  mode: deployment<br>  env:<br>  - name: DT_ENDPOINT<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-endpoint<br>        name: dynatrace-endpoint<br>  - name: DT_API_TOKEN<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-access-token<br>        name: dynatrace-access-token<br>  config:<br>    receivers:<br>      otlp:<br>        protocols:<br>          http:<br>            endpoint: 0.0.0.0:4318<br>          grpc:<br>            endpoint: 0.0.0.0:4317<br>      prometheus:<br>        config:<br>          scrape_configs:<br>            - job_name: &#39;otel-collector&#39;<br>              scrape_interval: 5s<br>              kubernetes_sd_configs:<br>              - role: pod<br>              scheme: http<br>              tls_config:<br>                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt<br>              authorization:<br>                credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token<br>              metrics_path: /metrics<br>              relabel_configs:<br>              - source_labels: [__meta_kubernetes_namespace]<br>                action: keep<br>                regex: &quot;kong&quot;<br>              - source_labels: [__meta_kubernetes_pod_name]<br>                action: keep<br>                regex: &quot;kong-kong-(.+)&quot;<br>              - source_labels: [__meta_kubernetes_pod_container_name]<br>                action: keep<br>                regex: &quot;proxy&quot;<br>              - source_labels: [__meta_kubernetes_pod_container_port_number]<br>                action: keep<br>                regex: &quot;8100&quot;<br>      tcplog:<br>        listen_address: 0.0.0.0:54525<br>        operators:<br>          - type: json_parser<br><br>    exporters:<br>      otlphttp:<br>        endpoint: &quot;${env:DT_ENDPOINT}&quot;<br>        headers:<br>          Authorization: &quot;Api-Token ${env:DT_API_TOKEN}&quot;<br>      prometheus:<br>        endpoint: 0.0.0.0:8889<br>      debug:<br>        verbosity: detailed<br><br>    processors:<br>      cumulativetodelta:<br>        include:<br><br>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: []<br>          exporters: [otlphttp]<br>        metrics:<br>          receivers: [otlp, prometheus]<br>          processors: [cumulativetodelta]<br>          exporters: [otlphttp, prometheus]<br>        logs:<br>          receivers: [otlp, tcplog]<br>          processors: []<br>          exporters: [otlphttp]<br>EOF</pre><p>The most critical parameters defined are:</p><ul><li>image: it refers to the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib">“contrib” distribution</a> of the Collector.</li><li>mode: deployment. The collector can be deployed in <a href="https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file#deployment-modes">4 different modes</a>: “Deployment”, “DaemonSet”, “StatefulSet” and “Sidecar”. For a better control of the Controller we have chosen regular Kubernetes Deployment mode. Please, refer to the <a href="https://kubernetes.io/docs/concepts/workloads/">Kubernetes documentation</a> to learn more about them.</li><li>The collector requires the Dynatrace Endpoint and API Token to connect. They are stored as Kubernetes secrets and are referred to in the “env” section.</li><li>Inside the service configuration section, all three pipelines, “traces”, “metrics” and “logs” have the same “exporter”, “otlphttp”, which sends data to Dynatrace.</li><li>Inside the “metrics” pipeline we have:</li><li>The <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter">Prometheus exporter</a> configured, so we can access the metrics sending requests directly to the collector through port 8889 as described in the exporter section.</li><li>It also includes the “otlp” receiver, so it can grab metrics coming from the Backend microservices as well.</li><li>It has <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/cumulativetodeltaprocessor/README.md">“cumulativetodelta”</a> as a Processor. A <a href="https://opentelemetry.io/docs/collector/configuration/#processors">Processor</a> is another OTel Collector construct, responsible for taking the data collected by receivers and modifying it before sending it to the exporters. Basically, the “Cumulative to Delta” Processor converts the Histogram metrics with <a href="https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality">cumulative temporality</a>, produced by the Kong Prometheus plugin, to delta temporality, supported by Dynatrace.</li><li>For the “logs” pipeline, we have both <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver">TCP Receiver</a>, using the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/json_parser.md">“json_parser” operator</a> to send formatted data, coming from Kong Gateway TCP Log Plugin, to Dynatrace as well as the “otlp” receiver again to get log data coming from the Backend microservices.</li></ul><h3>Kubernetes Service Account for Prometheus Receiver</h3><p>The OTel Collector <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md">Prometheus Receiver</a> fully supports the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config">scraping configuration</a> defined by Prometheus. The receiver, more precisely, uses the “<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod">pod</a>” role of the Kubernetes Service Discovery configurations (“<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config">kubernetes_sd_config</a>”). Specific “<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config">relabel_config</a>” settings with “regex” expressions allow the receiver to discover Kubernetes Pods that belong to the Kong Data Plane deployment.</p><p>One of the relabeling configs is related to the port 8100. This port configuration is part of the Data Plane deployment we used to get it running. Here’s the snippet of the “values.yaml” file we used previously:</p><pre>status:<br>  enabled: true<br>  http:<br>    enabled: true<br>    containerPort: 8100<br>    parameters: []</pre><p>That’s the Kong Gateway’s Status API where the Prometheus plugin exposes the metrics produced. In fact, the endpoint the receiver scrapes is, as specified in the OTel Collector configuration, http://&lt;Data_Plane_Pod_IP&gt;:8100/metrics</p><p>On the other hand, the OTel Collector has to be allowed to scrape the endpoint. We can define such permission with a Kubernetes ClusterRole and apply it to a Kubernetes Service Account with a Kubernetes ClusterRoleBinding.</p><p>Here’s the ClusterRole declaration. It’s a quite open one but it’s good enough for this exercise.</p><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: pod-reader<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;pods&quot;]<br>  verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]<br>EOF</pre><p>Then we need to create a Kubernetes Service Account and bind the Role to it.</p><pre>kubectl create sa collector -n opentelemetry-operator-system<br><br>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: read-pods<br>roleRef:<br>  kind: ClusterRole<br>  name: pod-reader<br>subjects:<br>  - kind: ServiceAccount<br>  name: collector<br>  namespace: opentelemetry-operator-system<br>EOF</pre><p>Finally, note that the OTel Collector configuration is deployed using the Service Account with serviceAccount: collector and then it will be able to scrape the endpoint exposed by Kong Gateway.</p><h3>Dynatrace Secrets</h3><p>As a final step, before instantiating the collector, we need to create the Kubernetes secrets with the Dynatrace endpoint and API Token. The endpoint refers to your Dynatrace tenant:</p><p>kubectl create secret generic dynatrace-endpoint -n opentelemetry-operator-system — from-literal=dt-endpoint=’https://&lt;your_Dynatrace_tenant&gt;.live.dynatrace.com/api/v2/otlp’</p><p>And create a secret for the Dynatrace API Token. The Token needs to be created with the “Ingest metrics” scope. Since we are going to manage Logs and Traces, add the “Ingest logs” and “Ingest OpenTelemetry traces” scopes as well. Check the Dynatrace documentation to learn how to issue <a href="https://docs.dynatrace.com/docs/manage/identity-access-management/access-tokens-and-oauth-clients/access-tokens-legacy#api">API Tokens</a>.</p><pre>kubectl create secret generic dynatrace-access-token -n opentelemetry-operator-system --from-literal=dt-access-token=&#39;&lt;your_API_TOKEN&gt;&#39;</pre><p>In case you want to decode your secret, use the base64 command. For example:</p><pre>kubectl get secret dynatrace-endpoint -n opentelemetry-operator-system -o json | jq -r &#39;.data.&quot;dt-endpoint&quot;&#39; | base64 --decode</pre><h3>Deploy the collector</h3><p>You can instantiate the collector simply submitting the declaration:</p><pre>kubectl apply -f otelcollector-dynatrace-all-signals.yaml</pre><p>If you want to destroy it run:</p><pre>kubectl delete opentelemetrycollector collector-kong -n opentelemetry-operator-system</pre><p>Check the collector’s log with:</p><pre>kubectl logs -f $(kubectl get pod -n opentelemetry-operator-system -o json | jq &#39;.items[].metadata | select(.name | startswith(&quot;collector&quot;))&#39; | jq -r &#39;.name&#39;) -n opentelemetry-operator-system</pre><p>Based on the declaration, the deployment creates a Kubernetes service named “collector-kong-collector” listening to several ports:</p><pre>% kubectl get service collector-kong-collector -n opentelemetry-operator-system -o json | jq &#39;.spec.ports&#39;<br>[<br>  {<br>    &quot;appProtocol&quot;: &quot;grpc&quot;,<br>    &quot;name&quot;: &quot;otlp-grpc&quot;,<br>    &quot;port&quot;: 4317,<br>    &quot;protocol&quot;: &quot;TCP&quot;,<br>    &quot;targetPort&quot;: 4317<br>  },<br>  {<br>    &quot;appProtocol&quot;: &quot;http&quot;,<br>    &quot;name&quot;: &quot;otlp-http&quot;,<br>    &quot;port&quot;: 4318,<br>    &quot;protocol&quot;: &quot;TCP&quot;,<br>    &quot;targetPort&quot;: 4318<br>  },<br>  {<br>    &quot;name&quot;: &quot;prometheus&quot;,<br>    &quot;port&quot;: 8889,<br>    &quot;protocol&quot;: &quot;TCP&quot;,<br>    &quot;targetPort&quot;: 8889<br>  },<br>  {<br>    &quot;name&quot;: &quot;tcplog&quot;,<br>    &quot;port&quot;: 54525,<br>    &quot;protocol&quot;: &quot;TCP&quot;,<br>    &quot;targetPort&quot;: 54525<br>  }<br>]</pre><h3>Kong Gateway Plugins</h3><p>Now we are going to apply the Kong Gateway Plugins so the Kong Data Plane can integrate with the OTel Collector and send all signals required including traces, metrics and logs.</p><p>Here’s the second decK declaration</p><pre>deck gateway sync - konnect-token $PAT - &lt;&lt;EOF<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - otel-demo-plugins<br>_konnect:<br>  control_plane_name: OTel<br>plugins:<br>  - name: opentelemetry<br>    instance_name: opentelemetry1<br>    enabled: true<br>    config:<br>      traces_endpoint: http://collector-kong-collector.opentelemetry-operator-system:4318/v1/traces<br>      logs_endpoint: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318/v1/logs<br>      resource_attributes:<br>        service.name: &quot;kong-otel-demo&quot;<br>  - name: tcp-log<br>    instance_name: tcp-log1<br>    enabled: true<br>    config:<br>      host: collector-kong-collector.opentelemetry-operator-system.svc.cluster.local<br>      port: 54525<br>  - name: prometheus<br>    instance_name: prometheus1<br>    config:<br>      per_consumer: true<br>      status_code_metrics: true<br>      latency_metrics: true<br>      bandwidth_metrics: true<br>      upstream_health_metrics: true<br>      ai_metrics: true<br>  - name: rate-limiting<br>    instance_name: ratelimiting1<br>    enabled: false<br>    config:<br>      minute: 20<br>      policy: local<br>EOF</pre><p>The OpenTelemetry and TCP Log plugins refer to the OTel Collector’s Kubernetes FQDN (collector-kong-collector.opentelemetry-operator-system.svc.cluster.local). Each one of them connects to a specific collector port and path to send data to it. For example, the <a href="https://opentelemetry.io/docs/specs/otlp/#otlphttp-request">“/v1/traces”</a> path is the default the collector uses to handle requests with trace data.</p><p>The OpenTelemetry Plugin also pushes the Kong Data Plane server logs to the Collector using the “logs_endpoint” configuration parameter. The TCP Log Plugin is responsible for sending the access (or translation) logs to the Collector.</p><p>Check the Konnect UI one more time to see the Kong Gateway Plugins. Note the <a href="https://docs.konghq.com/hub/kong-inc/rate-limiting/">Kong Rate Limiting Plugin</a> is created but disabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Imw9fcfar7CAVIs" /></figure><h3>Upgrade the Backend Services to connect to OTel Collector</h3><p>So far, the Backend Services don’t know about the OTel Collector deployment. We can tell them about it upgrading their deployment and overriding the OTel Collector related parameters:</p><pre>helm upgrade otel-demo-app open-telemetry/opentelemetry-demo \<br>-n open-telemetry \<br>--create-namespace \<br>--set opentelemetry-collector.enabled=false \<br>--set components.loadgenerator.enabled=false \<br>--set components.frontendProxy.enabled=false \<br>--set components.frontend.enabled=false \<br>--set components.imageprovider.enabled=false \<br>--set jaeger.enabled=false \<br>--set prometheus.enabled=false \<br>--set grafana.enabled=false \<br>--set default.envOverrides[0].name=&quot;OTEL_COLLECTOR_NAME&quot; \<br>--set default.envOverrides[0].value=&quot;collector-kong-collector.opentelemetry-operator-system.svc.cluster.local&quot;</pre><h3>Consume the Kong Routes again</h3><p>With all moving parts deployed and properly configured, it’s time to consume the Kong Routes and Backend microservices again, so we can see all signals, traces, metrics and logs, going through the OTel collector and getting published in the Dynatrace tenant.</p><p>Inside the same Ubuntu Pod, send the same request to Kong:</p><pre># grpcurl -plaintext otel-demo-app-productcatalogservice.kong:80 oteldemo.ProductCatalogService.ListProducts | jq &#39;.products[].name&#39;</pre><h3>OTel Demo Application Frontend Services deployment</h3><p>So, with our Backend layer fully deployed with the Kong Data Plane, OTel Collector and OTel Demo Backend Services, we can proceed and deploy the Frontend layer.</p><p>Use the same <a href="https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-demo">Open Telemetry Demo Helm Charts</a>, provided by the Open Telemetry project, like this. Some comments regarding the Helm command:</p><ul><li>The Frontend Services are deployed in a different Kubernetes Namespace.</li><li>Since we are reproducing a scenario emulating two different and independent organizations, the OTel Collector for the Frontend Services is disabled.</li><li>It exposes the FrontendProxy Service, the entry point of the Frontend layer, with an AWS ELB.</li><li>All Backend and Observability components are disabled</li><li>The most important settings are overriding, for all Frontend Services, the Backend Services addresses with Kubernetes Services, which route the requests to Kong Gateway. As you recall, Kong Gateway, in turn, has host-based Kong Routes defined for each one of the Kubernetes Services.</li></ul><pre>helm install otel-demo-app open-telemetry/opentelemetry-demo \<br>-n open-telemetry-frontend \<br>--create-namespace \<br>--set opentelemetry-collector.enabled=false \<br>--set components.frontendProxy.service.type=LoadBalancer \<br>--set components.loadgenerator.enabled=false \<br>--set components.adService.enabled=false \<br>--set components.cartService.enabled=false \<br>--set components.checkoutService.enabled=false \<br>--set components.currencyService.enabled=false \<br>--set components.productCatalogService.enabled=false \<br>--set components.recommendationService.enabled=true \<br>--set components.shippingService.enabled=false \<br>--set components.accountingService.enabled=false \<br>--set components.kafka.enabled=false \<br>--set components.emailService.enabled=false \<br>--set components.frauddetectionService.enabled=false \<br>--set components.paymentService.enabled=false \<br>--set components.quoteService.enabled=false \<br>--set components.valkey.enabled=false \<br>--set opensearch.enabled=false \<br>--set jaeger.enabled=false \<br>--set prometheus.enabled=false \<br>--set grafana.enabled=false \<br>--set components.frontend.envOverrides[0].name=&quot;AD_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[0].value=&quot;otel-demo-app-adservice.kong:80&quot; \<br>--set components.frontend.envOverrides[1].name=&quot;CART_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[1].value=&quot;otel-demo-app-cartservice.kong:80&quot; \<br>--set components.frontend.envOverrides[2].name=&quot;CHECKOUT_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[2].value=&quot;otel-demo-app-checkoutservice.kong:80&quot; \<br>--set components.frontend.envOverrides[3].name=&quot;CURRENCY_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[3].value=&quot;otel-demo-app-currencyservice.kong:80&quot; \<br>--set components.frontend.envOverrides[4].name=&quot;PRODUCT_CATALOG_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[4].value=&quot;otel-demo-app-productcatalogservice.kong:80&quot; \<br>--set components.frontend.envOverrides[5].name=&quot;RECOMMENDATION_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[5].value=&quot;otel-demo-app-recommendationservice.kong:80&quot; \<br>--set components.frontend.envOverrides[6].name=&quot;SHIPPING_SERVICE_ADDR&quot; \<br>--set components.frontend.envOverrides[6].value=&quot;otel-demo-app-shippingservice.kong:80&quot;</pre><p>You can get the ELB address with:</p><pre>FRONTEND_LB=$(kubectl get service -n open-telemetry-frontend -o json | jq -r &#39;.items[] | select(.metadata.name) | select(.metadata.name | startswith(&quot;otel-demo-app-frontendproxy&quot;))&#39; | jq -r &#39;.status.loadBalancer.ingress[].hostname&#39;)</pre><p>For MacOS, you can open the Frontend Application with:</p><pre>open -a &quot;Google Chrome&quot; &quot;http://${FRONTEND_LB}:8080&quot;</pre><p>You should see the OpenTelemetry Demo App home page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iql9nS-O6PqlKzRs" /></figure><h3>Dynatrace Dashboards</h3><p>Start navigating the Frontend and check your incoming signals in Dynatrace. Here are some examples to the Dynatrace Dashboards:</p><h3>Distributed Tracing Dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bf0ukUR8vE7f7XNJ" /></figure><h3>Metrics Notebook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yJP1gaw8cb-j3Bmn" /></figure><p>The Notebook has three tiles defined:</p><ul><li>Histogram Metrics</li><li>Return Codes</li><li>Total of request for each Kong Route</li></ul><h3>Log DashboardMetrics Notebook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O92plkOQTej2GTEX" /></figure><p>It can be created with this DQL (“Dynatrace Query Language”) command</p><pre>fetch logs<br>| filter contains(content, &quot;otel-demo&quot;)<br>| sort timestamp desc</pre><h3>Kong API Gateway Policies</h3><p>As we previously mentioned, one of the main reasons to have Kong API Gateway exposing the Backend Services is the capability to define policies to protect them. For example, our Kong decK declaration, besides the Observability Plugins (OpenTelemetry, Prometheus and TCP-Log Plugins), creates the Kong Rate Limiting plugin.</p><p>The original declaration has the Plugin disabled. You can change it by updating and submitting the declaration again or using the Konnect UI.</p><p>As an example, if you enable the Kong Rate Limiting Plugin, Kong Gateway will prevent the Frontend microservice from sending more than 20 requests per minute. If it tries to do so, Kong Gateway will return 429 messages. Of course, you can change the policy accordingly. Check the <a href="https://docs.konghq.com/hub/kong-inc/rate-limiting/">Kong Rate Limiting Plugin</a> to learn more about it.</p><p>Keep in mind the Kong Rate Limiting Plugin is a basic example of policies you can define at the Gateway layer. Kong provides an extensive list of plugins to implement all sorts of policies, including Authentication/Authorization, Transformation, Traffic Control, etc. Visit the <a href="https://docs.konghq.com/hub/">Kong Plugin Hub</a> and check them all.</p><p>One last comment is that, ideally, the Frontend microservice should be enhanced with new exception handlings to properly process all these new return codes.</p><h3>Conclusion</h3><p>Kong Konnect technologies provide a powerful infrastructure to expose and protect applications and services. At the same time the integration of Dynatrace and Kong allow customers to have a strong observability infrastructure, grounded in OpenTelemetry standards.</p><p>The OTel demo application exercise is a nice starting point to explore how the Kong and Dynatrace technologies can be used to leverage existing applications to a more realistic scenario.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb1db304b34a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Processing and Vector Similarity Search with Kong AI Gateway and Redis]]></title>
            <link>https://medium.com/@claudioacquaviva/semantic-processing-and-vector-similarity-search-with-kong-ai-gateway-and-redis-cbf29b11eee5?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf29b11eee5</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Tue, 06 May 2025 12:33:39 GMT</pubDate>
            <atom:updated>2025-05-07T17:02:30.211Z</atom:updated>
            <content:encoded><![CDATA[<p>Kong has supported Redis since its early versions. In fact, the integration between Kong API Gateway and Redis is a powerful combination to enhance API management. We can summarize the integration points and use cases of Kong and Redis into three main groups:</p><ul><li>Kong API Gateway: Kong integrates with Redis via plugins that enable it to leverage Redis’ capabilities for enhanced API functionality including Caching, Rate Limiting and Session Management.</li><li>Kong AI Gateway: Starting with Kong Gateway 3.6, several new AI-based plugins leverage Redis Vector Databases to implement AI use cases, like Rate Limiting policies based on LLM tokens, Semantic Caching, Semantic Prompt Guards and Semantic Routing.</li><li>RAG and Agent Applications: Kong AI Gateway and Redis can collaborate for AI-based applications using frameworks like LangChain and LangGraph.</li></ul><p>For all use cases, Kong supports multiple flavors of Redis deployments, including <a href="https://redis.io/docs/latest/get-started/">Redis Community Edition</a> (including while using Redis Cluster for horizontal scalability or Redis Sentinel for high availability), <a href="https://redis.io/software/">Redis Software</a> (which provides enterprise capabilities often needed for production workloads) and <a href="https://redis.io/cloud/">Redis Cloud</a> (available on AWS, GCP and as Azure Managed Redis in Azure).</p><p>This blog post focuses on how Kong and Redis can be used to address Semantic Processing use cases like Similarity Search and Semantic Routing across multiple LLM environments.</p><h3>Kong AI Gateway Reference Architecture</h3><p>To get started let’s take a look at a high-level reference architecture of the Kong AI Gateway. As you can see, the Kong Gateway Data Plane, responsible for handling the incoming traffic, can be configured with two types of Kong Plugins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*yQd8063701sf0gZX" /></figure><h3>Kong API Gateway plugins</h3><p>One of the main capabilities provided by Kong Gateway is extensibility. An extensive list of plugins allows you to implement specific policies to protect and control the APIs deployed in the Gateway. The plugins offload critical and complex processing usually implemented by backend services and applications. With the Gateway and its plugins in place, the backend services can focus on business logic only, leading to a faster application development process. Each plugin is responsible for specific functionality, including:</p><ul><li>Authentication/authorization: to implement security mechanisms such as Basic Authentication, LDAP, Mutual TLS (mTLS), API Key, OPA (Open Policy Agent) based access control policies, etc.</li><li>Integration with Kafka-based Data/Event Streaming infrastructures.</li><li>Log processing: to externalize all requests processed by the Gateway to third-party infrastructures.</li><li>Analytics and monitoring: to provide metrics to external systems including OpenTelemetry- based systems and Prometheus.</li><li>Traffic control: to implement canary releases, mocking endpoints, routing policies based on request headers, etc.</li><li>Transformations: to transform requests before routing them to the upstreams and to transform responses before returning to the consumers.</li><li>For IoT projects, where MQTT over WebSockets connections are extensively used, Kong provides WebSockets Size Limit and WebSockets Validator plugins to control the events sent by the devices.</li></ul><p>Also, Kong API Gateway provides plugins which implement several integration points with Redis:</p><ul><li><a href="https://docs.konghq.com/hub/kong-inc/proxy-cache-advanced/">Proxy Caching Advanced</a> and <a href="https://docs.konghq.com/hub/kong-inc/graphql-proxy-cache-advanced/">GraphQL Proxy Caching Advanced</a> — cache and serve requested responses.</li><li><a href="https://docs.konghq.com/hub/kong-inc/rate-limiting/">Rate Limiting</a>, <a href="https://docs.konghq.com/hub/kong-inc/rate-limiting-advanced/">Rate Limiting Advanced</a>, <a href="https://docs.konghq.com/hub/kong-inc/service-protection/">Service Protection</a> and <a href="https://docs.konghq.com/hub/kong-inc/graphql-rate-limiting-advanced/">GraphQL Rate Limiting Advanced</a> — rate limit how many HTTP requests can be made in a given time frame.</li><li><a href="https://docs.konghq.com/hub/kong-inc/openid-connect/">OpenID Connect</a> and <a href="https://docs.konghq.com/hub/kong-inc/upstream-oauth/">Upstream OAuth</a> — for session storage and token caching.</li><li><a href="https://docs.konghq.com/hub/kong-inc/acme/">ACME</a> — stores digital certificates.</li></ul><h3>Kong AI Gateway plugins</h3><p>On the other hand Kong AI Gateway leverages the existing Kong API Gateway extensibility model to provide specific AI-based plugins, more precisely to protect LLM infrastructures:</p><ul><li><a href="https://docs.konghq.com/hub/kong-inc/ai-proxy/">AI Proxy</a> and <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy-advanced/">AI Proxy Advanced</a> plugins: the Multi-LLM capability allows the AI Gateway to abstract and load balancing multiple LLM models based on policies including latency time, model usage, semantics etc.</li><li>Prompt Engineering:</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-prompt-template/">AI Prompt Template</a> plugin, responsible for pre-configuring AI prompts to users</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-prompt-decorator/">AI Prompt Decorator</a> plugin, which injects messages at the start or end of a caller’s chat history.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-prompt-guard/">AI Prompt Guard</a> plugin lets you configure a series of PCRE-compatible regular expressions to allow and block specific prompts, words, phrases, or otherwise and have more control over a LLM service.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-semantic-prompt-guard/">AI Semantic Prompt Guard</a> plugin to self-configurable semantic (or pattern-matching) prompt protection.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-semantic-cache/">AI Semantic Cache</a> plugin caches responses based on threshold, to improve performance (and therefore end-user experience) and cost.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-rate-limiting-advanced/">AI Rate Limiting Advanced</a>, you can tailor per-user or per-model policies based on the tokens returned by the LLM provider or craft a custom function to count the tokens for requests.</li><li><a href="https://docs.konghq.com/hub/kong-inc/ai-request-transformer/">AI Request Transformer</a> and <a href="https://docs.konghq.com/hub/kong-inc/ai-response-transformer/">AI Response Transformer</a> plugins seamlessly integrate with the LLM, enabling introspection and transformation of the request’s body before proxying it to the Upstream Service and prior to forwarding the response to the client.</li></ul><p>By leveraging the same underlying core of Kong Gateway, and combining both categories of plugins, we can implement powerful policies and reduce complexity in deploying the AI Gateway capabilities as well.</p><p>The first use case we are going to focus on is Semantic Cache where the AI Gateway plugin integrates with Redis to perform Similarity Search. Then, we are going to explore how the AI Proxy Advanced Plugin can take advantage of Redis to implement Semantic Routing across multiple LLMs models.</p><p>As a remark, AI Rate Limiting Advanced and AI Semantic Prompt Guard Plugins are two other examples where AI Gateway and Redis work together.</p><p>Before diving into the first use case, let’s highlight and summarize the main concepts Kong AI Gateway and Redis rely on.</p><h3>Embeddings</h3><p>Embeddings (aka Vectors or even Vector Embeddings) are a representation of unstructured data like text, images, etc. In a LLM context, the dimensionality of the Embeddings refers to the number of characteristics captured in the vector representation of a given sentence: the more dimensions an Embedding has, the better and more effective it is.</p><p>There are multiple ML-based embedding methods used in NLP like:</p><ul><li><a href="https://en.wikipedia.org/wiki/One-hot">One-hot</a></li><li><a href="https://code.google.com/archive/p/word2vec/">word2vec</a></li><li>TF-IDF (Term frequency-inverse document frequency)</li><li><a href="https://nlp.stanford.edu/projects/glove/">GloVe</a> (Global Vectors for Word Representation)</li><li><a href="https://github.com/google-research/bert">BERT</a> (Bidirectional Encoder Representations from Transformers)</li></ul><p>Here’s an example of a Python script using <a href="https://sbert.net/">“Sentence Transformers”</a> module (aka SBERT, or Sentence-BERT, maintained by <a href="https://huggingface.co/">Hugging Face</a>), using the <a href="https://huggingface.co/sentence-transformers/all-mpnet-base-v2">“all-mpnet-base-v2”</a> Embedding Model to encode a simple sentence into an embedding:</p><pre>from sentence_transformers import SentenceTransformer<br>from sentence_transformers.util import truncate_embeddings<br><br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;, cache_folder=&quot;./&quot;)<br>embeddings = model.encode(&quot;Who is Joseph Conrad?&quot;)<br>embeddings = truncate_embeddings(embeddings, 3)<br>print(embeddings.size)<br>print(embeddings)</pre><p>The “all-mpnet-base-v2” Embedding Model encodes sentences to a 768 dimensional Vector. As an experiment, we have truncated the vector to 3 dimensions only.</p><p>The output should be like:</p><pre>3<br>[ 0.06030013 -0.00782523 0.01018228]</pre><h3>Vector Database</h3><p>A Vector Database stores and searches Vector Embeddings. They are essential for AI-based applications supporting images, texts, etc. providing Vector Stores, Vector Indexes and more importantly algorithms to implement Vector Similarity Searches.</p><p>Redis provides nice introductions about <a href="https://redis.io/glossary/vector-embeddings/">Vector Embeddings</a> and <a href="https://redis.io/blog/vector-databases-101/">Vector Databases</a>. Redis Query Engine is an in-built capability within Redis that provides vector search functionality (as well as other types of search such as full-text, numeric etc.) and is known for <a href="https://redis.io/blog/benchmarking-results-for-vector-databases/">industry-leading performance</a>. Redis is <strong>built for Speed</strong> and delivers unmatched performance with sub-millisecond latency, leveraging in-memory data structures and advanced optimizations to power real-time applications at scale. This is critical for gateway use-cases where deployment happens in the “hot-path” of LLM queries.</p><p>In addition, Redis can be deployed as as enterprise Software and/or as a Cloud service thereby adding several enterprise capabilities including:</p><ul><li><strong>Scalability</strong>: Redis can easily scale horizontally and effortlessly handle dynamic workloads and ability to manage massive datasets across distributed architectures.</li><li><strong>High availability and persistence</strong>: Redis supports high availability with built-in support for multi-AZ deployments, seamless failover, data persistence through backups and active-active architecture, enabling robust disaster recovery and consistent application performance.</li><li><strong>Flexibility</strong>: Redis natively supports multiple data-structures such as JSON, Hash, Strings, Streams and more to suit diverse application needs.</li><li><strong>Broad Ecosystem:</strong> As one of the world’s most popular databases, Redis has a rich ecosystem of client libraries (<a href="https://www.redisvl.com/index.html">redisvl</a> for GenAI use-cases), <a href="https://redis.io/docs/latest/develop/tools/">developer tools</a> and integrations.</li></ul><h3>Vector Similarity Search</h3><p>With similarity search we can find, in a typically unstructured dataset, items similar (or dissimilar) to a certain presented item. For example, given a picture of a cell phone, try to find similar ones considering its shape, color, etc. Or, given two pictures, check the similarity score between them.</p><p>In our NLP context, we are interested in similar responses returned by the LLM when applications send prompts to it. For example, these two following sentences “Who is Joseph Conrad?” and “Tell me more about Joseph Conrad”, semantically speaking, should have a high similarity score.</p><p>We can extend our Python script to try that out:</p><pre>from sentence_transformers import SentenceTransformer<br><br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;, cache_folder=&quot;./&quot;)<br><br>sentences = [<br>  &quot;Who is Joseph Conrad?&quot;,<br>  &quot;Tell me more about Joseph Conrad.&quot;,<br>  &quot;Living is easy with eyes closed.&quot;,<br>]<br><br>embeddings = model.encode(sentences)<br>print(embeddings.shape)<br><br>similarities = model.similarity(embeddings, embeddings)<br>print(similarities)</pre><p>The output should be like this. The embeddings are</p><pre>(3, 768)<br>tensor([[1.0000, 0.8600, 0.0628],<br>        [0.8600, 1.0000, 0.1377],<br>        [0.0628, 0.1377, 1.0000]])</pre><p>The “shape” is composed of 3 embeddings of 768 dimensions each. The code asks to cross-check the similarity of all embeddings. The more similar they are, the higher the score. Notice that the “1.0000” score is returned, as expected, when self-checking a given embedding.</p><p>The <a href="https://sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html">“similarity”</a> method returns a <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor">“Tensor”</a> object, which is implemented by <a href="https://pytorch.org/">PyTorch</a>, the ML library used by Sentence Transformer.</p><p>There are several techniques for similarity calculation including distance or angle between the vectors. The most common methods are:</p><ul><li><a href="https://redis.io/learn/howtos/solutions/vector/getting-started-vector#euclidean-distance-l2-norm">Euclidean Distance</a>: based on linear distance between two points</li><li><a href="https://redis.io/learn/howtos/solutions/vector/getting-started-vector#cosine-similarity">Cosine Similarity</a>: used by default by the “similarity” method: based on the angle between two vectors.</li><li><a href="https://redis.io/learn/howtos/solutions/vector/getting-started-vector#inner-product">Dot Product</a>: based on the product of vectors.</li></ul><p>In a Vector Database context, <a href="https://redis.io/learn/howtos/solutions/vector/getting-started-vector#what-is-vector-similarity">Vector Similarity Search (VSS)</a> is the process of finding vectors in the vector database that are similar to a given query vector.</p><h3>RediSearch and Redis VSS</h3><p>Back in 2022, Redis launched <a href="https://redis.io/docs/latest/develop/interact/search-and-query/administration/overview/">RediSearch</a> <a href="https://redis.io/docs/latest/operate/oss_and_stack/stack-with-enterprise/release-notes/redisearch/redisearch-2.4-release-notes/">2.4</a>, a text search engine built on top of Redis data store, with RedisVSS (Vector Similarity Search).</p><p>To get a better understanding of how RedisVSS works, consider this Python script implementing a basic similarity search. Make sure you have set the “OPENAI_API_TYPE” environment variable as “openai” before running the script.</p><pre>import redis<br>from redis.commands.search.field import TextField, VectorField<br>from redis.commands.search.indexDefinition import IndexDefinition, IndexType<br>from redis.commands.search.query import Query<br>import numpy as np<br>import openai<br>import os<br><br>### Get environment variables<br>openai.api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br>host = os.getenv(&quot;REDIS_LB&quot;)<br><br>### Create a Redis Index for the Vector Embeddings<br>client = redis.Redis(host=host, port=6379)<br><br>try:<br>    client.ft(&#39;index1&#39;).dropindex(delete_documents=True)<br>except:<br>    print(&quot;index does not exist&quot;)<br><br><br>schema = (<br>    TextField(&quot;name&quot;),<br>    TextField(&quot;description&quot;),<br>    VectorField(<br>        &quot;vector&quot;,<br>        &quot;FLAT&quot;,<br>        {<br>            &quot;TYPE&quot;: &quot;FLOAT32&quot;,<br>            &quot;DIM&quot;: 1536,<br>            &quot;DISTANCE_METRIC&quot;: &quot;COSINE&quot;,<br>        }<br>    ),<br>)<br><br>definition = IndexDefinition(prefix=[&quot;vectors:&quot;], index_type=IndexType.HASH)<br>res = client.ft(&quot;index1&quot;).create_index(fields=schema, definition=definition)<br><br><br>### Step 1: call OpenAI to generate Embeddings for the reference text and stores it in Redis<br><br>name = &quot;vector1&quot;<br>content = &quot;Who is Joseph Conrad?&quot;<br>redis_key = f&quot;vectors:{name}&quot;<br><br>res = openai.embeddings.create(input=content, model=&quot;text-embedding-3-small&quot;).data[0].embedding<br><br>embeddings = np.array(res, dtype=np.float32).tobytes()<br><br>pipe = client.pipeline()<br>pipe.hset(redis_key, mapping = {<br>  &quot;name&quot;: name,<br>  &quot;description&quot;: content,<br>  &quot;vector&quot;: embeddings<br>})<br>res = pipe.execute()<br><br><br>### Step 2: perform Vector Range queries with 2 new texts and get the distance (similarity) score<br><br>query = (<br>    Query(&quot;@vector:[VECTOR_RANGE $radius $vec]=&gt;{$yield_distance_as: distance_score}&quot;)<br>     .return_fields(&quot;id&quot;, &quot;distance_score&quot;)<br>     .dialect(2)<br>)<br><br># Text #1<br>content = &quot;Tell me more about Joseph Conrad&quot;<br>res = openai.embeddings.create(input=content, model=&quot;text-embedding-3-small&quot;).data[0].embedding<br>new_embeddings = np.array(res, dtype=np.float32).tobytes()<br><br>query_params = {<br>    &quot;radius&quot;: 1,<br>    &quot;vec&quot;: new_embeddings<br>}<br>res = client.ft(&quot;index1&quot;).search(query, query_params).docs<br>print(res)<br><br># Text #2<br>content = &quot;Living is easy with eyes closed&quot;<br>res = openai.embeddings.create(input=content, model=&quot;text-embedding-3-small&quot;).data[0].embedding<br>new_embeddings = np.array(res, dtype=np.float32).tobytes()<br><br>query_params = {<br>    &quot;radius&quot;: 1,<br>    &quot;vec&quot;: new_embeddings<br>}<br>res = client.ft(&quot;index1&quot;).search(query, query_params).docs<br>print(res)</pre><p>Initially, the script creates an index to receive the embeddings returned by OpenAI. We are using the “text-embedding-3-small” OpenAI model, which has 1536 dimensions, so the index has a VectorField defined to support those dimensions.</p><p>Next, the script has two steps:</p><ul><li>Stores the embeddings of a reference text, generated by OpenAI Embedding Model.</li><li>Performs Vector Range queries passing two new texts to check their similarity with the original one.</li></ul><p>Here’s a diagram representing the steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*gTkF95JM-Omvx_zP" /></figure><p>The code assumes you have a Redis environment available. Please check the <a href="https://redis.io/docs/latest/operate/">Redis Products documentation</a> to learn more about it. It also assumes you have two environment variables defined: OpenAI API Key and Load Balancer address where Redis is available.</p><p>The script was coded using two main libraries:</p><ul><li>“Python client for Redis” (<a href="https://redis.io/docs/latest/develop/clients/redis-py/">redis-py</a>) library to make Redis calls.</li><li><a href="https://github.com/openai/openai-python">“OpenAI Python API”</a> to interact with OpenAI.</li></ul><p>While executing the code, you can monitor Redis with, for example, <a href="https://redis.io/docs/latest/operate/rs/references/cli-utilities/redis-cli/">redis-cli</a> monitor. The code line res = client.ft(“index1”).search(query, query_params).docs should log a message like this one:</p><pre>&quot;FT.SEARCH&quot; &quot;index1&quot; &quot;@vector:[VECTOR_RANGE $radius $vec]=&gt;{$YIELD_DISTANCE_AS: score}&quot; &quot;RETURN&quot; &quot;2&quot; &quot;id&quot; &quot;score&quot; &quot;DIALECT&quot; &quot;2&quot; &quot;LIMIT&quot; &quot;0&quot; &quot;10&quot; &quot;params&quot; &quot;4&quot; &quot;radius&quot; &quot;1&quot; &quot;vec&quot; &quot;\xcb9\x9c&lt;\xf8T\x18=\xaa\xd4\xb5\xbcB\xc0.=\xb5……….&quot;</pre><p>Let’s examine the command. Implicitly, the <a href="https://redis-py.readthedocs.io/en/stable/commands.html#redis.commands.cluster.RedisClusterCommands.ft">.ft(“index1”)</a> method call gives us support to <a href="https://redis-py.readthedocs.io/en/stable/redismodules.html#redisearch-commands">Redis Search Commands</a>, as the <a href="https://redis-py.readthedocs.io/en/stable/redismodules.html#redis.commands.search.commands.SearchCommands.search">.search(query, query_params)</a> call sends an actual search query using the FT.SEARCH Redis command.</p><p>The FT.SEARCH command receives the parameters defined in both query and query_params objects. The query parameter, defined using the <a href="https://redis.io/docs/latest/develop/interact/search-and-query/query/">Query</a> object, specifies the actual command as well as the return fields and dialect.</p><pre>query = (<br>  Query(&quot;@vector:[VECTOR_RANGE $radius $vec]=&gt;{$yield_distance_as: distance_score}&quot;)<br>    .return_fields(&quot;id&quot;, &quot;distance_score&quot;)<br>    .dialect(2)<br>)</pre><p>We want to return the distance (similarity) score, so we must yield it via the $yield_distance_as attribute.</p><p>Query Dialects enable enhancements to the query API, allowing the introduction of new features while maintaining compatibility with existing applications. For Vector Queries like ours, Query Dialect should be set with a value equal or greater than 2. Please, check the specific <a href="https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/dialects/">Query Dialect documentation</a> page to learn more about it.</p><p>On the other hand, the query_params object defines extra parameters, including the radius and the embeddings it should be considered for the search.</p><pre>query_params = {<br>  &quot;radius&quot;: 1,<br>  &quot;vec&quot;: new_embeddings<br>}</pre><p>The final FT.SEARCH also includes parameters to define offset and number of results. Check the <a href="https://redis.io/docs/latest/commands/ft.search/">documentation</a> to learn more about it.</p><p>In fact, the FT.SEARCH command sent by the script is just an example of <a href="https://redis.io/docs/latest/develop/interact/search-and-query/query/vector-search/">Vector Search</a> supported by Redis. Basically, Redis supports two main types of searches:</p><ul><li><a href="https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/vectors/#knn-vector-search">KNN Vector Search</a>: this <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">algorithm</a> finds the top “k-nearest neighbors” to a query vector.</li><li><a href="https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/vectors/#vector-range-queries">Vector Range Query</a>: the query type the script uses filters the index based on a radius parameter. Radius defines the semantic distance between the two vectors: the input query vector and indexed vector.</li></ul><p>Our script’s intent is to examine the distance between the two vectors and not implement any filter. That’s the reason why it has set the Vector Range Query with “radius”: 1.</p><p>After running the script, its output should be like:</p><pre>[Document {&#39;id&#39;: &#39;vectors:vector1&#39;, &#39;payload&#39;: None, &#39;distance_score&#39;: &#39;0.123970687389&#39;}]<br>[Document {&#39;id&#39;: &#39;vectors:vector1&#39;, &#39;payload&#39;: None, &#39;distance_score&#39;: &#39;0.903066933155&#39;}]</pre><p>That means that, as expected, the stored embedding, related to the reference text “Who is Joseph Conrad?”, is closer to the first new text, “Tell me more about Joseph Conrad”, than to the second one “Living is easy with eyes closed”.</p><p>Now that we have an introductory perspective of how we can implement Vector Similarity Searches with Redis, let’s examine the Kong AI Gateway Semantic Cache Plugin which is responsible for implementing semantic caching. We’ll see it performs similar searches to what we have done with the Python script.</p><h3>Kong AI Semantic Cache Plugin</h3><p>To get started, logically speaking, we can analyze the Caching flow from two different perspectives:</p><ul><li>Request #1: We don’t have any data cached.</li><li>Request #2: Kong AI Gateway has stored some data in the Redis Vector Database.</li></ul><p>Here’s a diagram illustrating the scenarios:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*8juDM0uW5C4j2OTI" /></figure><h3>Konnect Data Plane deployment</h3><p>Before exploring how the Kong AI Gateway Semantic Cache Plugin and Redis work together we have to deploy a Konnect Data Plane (based on Kong Gateway). Please, refer to the <a href="https://docs.konghq.com/konnect/">Konnect documentation</a> to register and spin your first Data Plane up.</p><h3>Kong Gateway Objects creation</h3><p>Next, we need to create Kong Gateway objects (Kong Gateway Services, Kong Routes and Kong Plugins) to implement the use case. There are several ways to do that, including Konnect RESTfull APIs, Konnect GUI, etc. With <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong), we can manage Kong Konnect configuration and create Kong Objects in a declarative way. Please check the <a href="https://docs.konghq.com/deck/latest/guides/getting-started/">decK documentation</a> to learn how to <a href="https://docs.konghq.com/deck/latest/guides/konnect/">use it with Konnect</a>.</p><h3>AI Proxy and AI Semantic Cache Plugins</h3><p>Here’s the decK declaration we are going to submit to Konnect to implement the Semantic Cache use case:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - semantic-cache<br>_konnect:<br>  control_plane_name: default<br>services:<br>- name: service1<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: route1<br>    paths:<br>    - /openai-route<br>    plugins:<br>    - name: ai-proxy<br>      instance_name: ai-proxy-openai-route<br>      enabled: true<br>      config:<br>        auth:<br>          header_name: Authorization<br>          header_value: Bearer &lt;your_OPENAI_APIKEY&gt;<br>        route_type: llm/v1/chat<br>        model:<br>          provider: openai<br>          name: gpt-4<br>          options:<br>            max_tokens: 512<br>            temperature: 1.0<br>    - name: ai-semantic-cache<br>      instance_name: ai-semantic-cache-openai<br>      enabled: true<br>      config:<br>        embeddings:<br>          auth:<br>            header_name: Authorization<br>            header_value: Bearer &lt;your_OPENAI_APIKEY&gt;<br>          model:<br>            provider: openai<br>            name: text-embedding-3-small<br>            options:<br>              upstream_url: https://api.openai.com/v1/embeddings<br>        vectordb:<br>          dimensions: 1536<br>          distance_metric: cosine<br>          strategy: redis<br>          threshold: 0.2<br>          redis:<br>            host: redis-stack.redis.svc.cluster.local<br>            port: 6379</pre><p>The declaration creates the following Kong Objects in the “default” Konnect Control Plane::</p><ul><li>Kong Gateway Service “service1”. It’s a fake service. In fact, the Kong AI Proxy plugin created next will determine the actual upstream destination.</li><li>Kong Route “route1” with the path “/openai-route”. That’s the route which exposes the LLM model.</li><li>AI Proxy Plugin. It’s configured to consume OpenAI’s “gpt-4” model. The “route_type” parameter, set as “llm/v1/chat”, refers to OpenAI’s “https://api.openai.com/v1/chat/completions” endpoint. Kong recommends storing the API Keys as secrets in a Secret Manager like AWS Secrets Manager or HashiCorp Vault. The current configuration, including the OpenAI API Key in the declaration, is for lab environments only, not recommended for production. Please refer to the official <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy/">AI Proxy Plugin documentation page</a> to learn more about its configuration.</li><li>AI Semantic Cache Plugin. It has two settings. Please check the <a href="https://docs.konghq.com/hub/kong-inc/ai-semantic-cache/">documentation page</a> to learn more about these and other configuration parameters.</li><li>embeddings: consumed the “text-embedding-3-small” Embedding Model, the same model we used in our Python script.</li><li>vectordb: refers to an existing Redis infrastructure to store the embeddings and process the VSS requests. Again, it’s configured with the same dimensions and distance metric as we used before. The threshold is going to be translated as the “radius” parameter in the VSS queries sent by the Kong Gateway Data Plane.</li></ul><p>After submitting the decK declaration to Konnect, you should see the new Objects using the Konnect UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8gtFDcThVCf0rmDV" /></figure><h3>Request #1</h3><p>With the new Kong Objects in place, the Kong Data Plane is refreshed with them and we are ready to start sending requests to it. Here’s the first one with the same content we used in the Python script:</p><pre>curl -i -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>  &quot;messages&quot;: [<br>    {<br>      &quot;role&quot;: &quot;user&quot;,<br>      &quot;content&quot;: &quot;Who is Joseph Conrad?&quot;<br>    }<br>  ]<br>}&#39;</pre><p>You should get a response like this, meaning the Gateway successfully routed the request to OpenAI which returned an actual message to us. From Semantic Caching and Similarity perspective, the most important headers are:</p><ul><li>X-Cache-Status: Miss, telling us the Gateway wasn’t able to find any data in the cache to satisfy the request.</li><li>X-Kong-Upstream-Latency and X-Kong-Proxy-Latency, showing the latency times.</li></ul><pre>HTTP/1.1 200 OK<br>Content-Type: application/json<br>Connection: keep-alive<br>X-Cache-Status: Miss<br>x-ratelimit-limit-requests: 10000<br>CF-RAY: 8fce86cde915eae2-ORD<br>x-ratelimit-limit-tokens: 10000<br>x-ratelimit-remaining-requests: 9999<br>x-ratelimit-remaining-tokens: 9481<br>x-ratelimit-reset-requests: 8.64s<br>x-ratelimit-reset-tokens: 3.114s<br>access-control-expose-headers: X-Request-ID<br>x-request-id: req_29afd8838136a2f7793d6c129430b341<br>X-Content-Type-Options: nosniff<br>openai-organization: user-4qzstwunaw6d1dhwnga5bc5q<br>Date: Sat, 04 Jan 2025 22:05:00 GMT<br>alt-svc: h3=&quot;:443&quot;; ma=86400<br>openai-processing-ms: 10002<br>openai-version: 2020–10–01<br>CF-Cache-Status: DYNAMIC<br>strict-transport-security: max-age=31536000; includeSubDomains; preload<br>Server: cloudflare<br>Content-Length: 1456<br>X-Kong-LLM-Model: openai/gpt-4<br>X-Kong-Upstream-Latency: 10097<br>X-Kong-Proxy-Latency: 471<br>Via: 1.1 kong/3.9.0.0-enterprise-edition<br>X-Kong-Request-Id: 36f6b41df3b74f78f586ae327af27075<br>{<br>  &quot;id&quot;: &quot;chatcmpl-Am6YEtvUquPHdHdcI59eZC3UfOUVz&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;created&quot;: 1736028290,<br>  &quot;model&quot;: &quot;gpt-4–0613&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: &quot;Joseph Conrad was a Polish-British writer regarded as one of the greatest novelists to write in the English language. He was born on December 3, 1857, and died on August 3, 1924. Though he did not speak English fluently until his twenties, he was a master prose stylist who brought a non-English sensibility into English literature.\n\nConrad wrote stories and novels, many with a nautical setting, that depict trials of the human spirit in the midst of what he saw as an impassive, inscrutable universe. His notable works include \&quot;Heart of Darkness\&quot;, \&quot;Lord Jim\&quot;, and \&quot;Nostromo\&quot;. Conrad&#39;s writing often presents a deep, pessimistic view of the world and deals with the theme of the clash of cultures and moral ambiguity.&quot;,<br>        &quot;refusal&quot;: null<br>      },<br>      &quot;logprobs&quot;: null,<br>      &quot;finish_reason&quot;: &quot;stop&quot;<br>    }<br>  ],<br>  &quot;usage&quot;: {<br>    &quot;prompt_tokens&quot;: 12,<br>    &quot;completion_tokens&quot;: 163,<br>    &quot;total_tokens&quot;: 175,<br>    &quot;prompt_tokens_details&quot;: {<br>      &quot;cached_tokens&quot;: 0,<br>      &quot;audio_tokens&quot;: 0<br>    },<br>    &quot;completion_tokens_details&quot;: {<br>      &quot;reasoning_tokens&quot;: 0,<br>      &quot;audio_tokens&quot;: 0,<br>      &quot;accepted_prediction_tokens&quot;: 0,<br>      &quot;rejected_prediction_tokens&quot;: 0<br>    }<br>  },<br>  &quot;system_fingerprint&quot;: null<br>}</pre><h4>Redis Introspection</h4><p>Kong Gateway creates a new index. You can check it with redis-cli ft._list. The index should be named like: idx:vss_kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4</p><p>And redis-cli ft.search idx:vss_kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4 “*” return 1 — should return the id of OpenAI’s response. Something like:</p><pre>1<br>kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:fcdf7d8995a227392f839b4530f8d8c3055748b96275fa9558523619172fd2a8</pre><p>The following json.get command should return the actual response received from OpenAI</p><pre>redis-cli json.get kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:fcdf7d8995a227392f839b4530f8d8c3055748b96275fa9558523619172fd2a8 | jq &#39;.payload.choices[].message.content&#39;</pre><p>More importantly, redis-cli monitor tells us all commands the plugin sent to Redis to implement the cache. The main ones are:</p><ol><li>“FT.INFO” to check if the index exists.</li></ol><p>2. There’s no index, so create it. The actual command is the following. Notice the index is similar to the one we used in the Python script.</p><pre>&quot;FT.CREATE&quot; &quot;idx:vss_kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4&quot; &quot;ON&quot; &quot;JSON&quot; &quot;PREFIX&quot; &quot;1&quot; &quot;kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:&quot; &quot;SCORE&quot; &quot;1.0&quot; &quot;SCHEMA&quot; &quot;$.vector&quot; &quot;AS&quot; &quot;vector&quot; &quot;VECTOR&quot; &quot;FLAT&quot; &quot;6&quot; &quot;TYPE&quot; &quot;FLOAT32&quot; &quot;DIM&quot; &quot;1536&quot; &quot;DISTANCE_METRIC&quot; &quot;COSINE&quot;</pre><p>3. Run a VSS to check if there’s a key which satisfies the request. The command looks like this. Again, notice the command is the same we used in our Python script. The “range” parameter reflects the “threshold” configuration used in the decK declaration.</p><pre>&quot;FT.SEARCH&quot;&quot;idx:vss_kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4&quot; &quot;@vector:[VECTOR_RANGE $range $query_vector]=&gt;{$YIELD_DISTANCE_AS: vector_score}&quot; &quot;SORTBY&quot; &quot;vector_score&quot; &quot;DIALECT&quot; &quot;2&quot; &quot;LIMIT&quot; &quot;0&quot; &quot;4&quot; &quot;PARAMS&quot; &quot;4&quot; &quot;query_vector&quot; &quot;\x981\x87&lt;bE\xe4&lt;b\xa3\……….\xbc&quot; &quot;range&quot; &quot;0.2&quot;</pre><p>4. Since the index has been just created, the VSS couldn’t find any key so the plugin sends a “JSON.SET” command to add a new index key with the embeddings received from OpenAI.</p><p>5. “expire” command to set the expiration time of the key as, by default, 300 seconds.</p><p>You can check the new index key using the Redis dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KEW4PuJpvwBQzWt6" /></figure><h3>Request #2</h3><p>If we send another request with similar content, the Gateway should return the same response, since it’s going to take from the Cache, as noticed in the X-Cache-Status: Hit header. Besides, the response has specific header related to the cache: X-Cache-Key and X-Cache-Ttl.</p><p>The response should be returned faster, since the Gateway didn’t have to route the request to OpenAI.</p><pre>curl -i -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Tell me more about Joseph Conrad&quot;<br>      }<br>    ]<br>  }&#39;<br>HTTP/1.1 200 OK<br>Date: Sun, 05 Jan 2025 14:28:59 GMT<br>Content-Type: application/json; charset=utf-8<br>Connection: keep-alive<br>X-Cache-Status: Hit<br>Age: 0<br>X-Cache-Key: kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:fcdf7d8995a227392f839b4530f8d8c3055748b96275fa9558523619172fd2a8<br>X-Cache-Ttl: 288<br>Content-Length: 1020<br>X-Kong-Response-Latency: 221<br>Server: kong/3.9.0.0-enterprise-edition<br>X-Kong-Request-Id: eef1373a3a688a68f088a52f72318315<br><br>{&quot;object&quot;:&quot;chat.completion&quot;,&quot;system_fingerprint&quot;:null,&quot;id&quot;:&quot;fcdf7d8995a22…….</pre><p>If you send another request with a non-similar content the plugin creates a new index key. For example:</p><pre>curl -i -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Living is easy with eyes closed&quot;<br>      }<br>    ]<br>  }&#39;</pre><p>Check the index keys again with:</p><pre>redis-cli - scan<br>&quot;kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:fcdf7d8995a227392f839b4530f8d8c3055748b96275fa9558523619172fd2a8&quot;<br>&quot;kong_semantic_cache:511efd84–117b-4c89–87cb-f92f9b74a6c0:openai-gpt-4:22fbee1a1a45147167f29cc53183d0d2eef618c973e4284ad0179970209cf131&quot;</pre><h3>Kong AI Proxy Advanced Plugin and Semantic Routing</h3><p>Kong AI Gateway provides several semantic based capabilities besides caching. A powerful one is Semantic Routing. With such a feature, we can let the Gateway decide the best model to handle a given request. For example, you might have models trained in specific topics, like Mathematics or Classical Music, so it’d be interesting to route the requests depending on the presented content. By analyzing the content of the request, the plugin can match it to the most appropriate model that is known to perform better in similar contexts. This feature enhances the flexibility and efficiency of model selection, especially when dealing with a diverse range of AI providers and models.</p><p>In fact Semantic Routing is one of the load balancing algorithms supported by the <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy-advanced/">AI Proxy Advanced Plugin</a>. The other supported algorithms are:</p><ul><li>Round-robin</li><li>Weight-based</li><li>Lowest-usage</li><li>Lowest-latency</li><li><a href="https://docs.konghq.com/gateway/latest/how-kong-works/load-balancing/#consistent-hashing">Consistent-hashing</a> (sticky-session on given header value)</li></ul><p>For the purpose of this blog post we are going to explore the Semantic Routing algorithm.</p><p>The diagram below shows how the AI Proxy Advanced Plugin works:</p><ul><li>At the configuration time, the plugin sends requests to an Embeddings Model based on descriptions defined. The embeddings returned are stored in Redis Vector Database.</li><li>During request processing time, the plugin gets the request content and sends a VSS query to Redis Vector Database. Depending on the similarity score, the plugin routes the request to the best targeted LLM model sitting behind the Gateway.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*vg0CvlOpjitNacNV" /></figure><p>Here’s the new decK declaration:</p><pre>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - semantic-routing<br>_konnect:<br>  control_plane_name: default<br>services:<br>- name: service1<br>  host: localhost<br>  port: 32000<br>  routes:<br>  - name: route1<br>    paths:<br>    - /openai-route<br>    plugins:<br>    - name: ai-proxy-advanced<br>      instance_name: ai-proxy-openai-route<br>      enabled: true<br>      config:<br>        balancer:<br>          algorithm: semantic<br>        embeddings:<br>          auth:<br>            header_name: Authorization<br>            header_value: Bearer &lt;your_OPENAI_APIKEY&gt;<br>          model:<br>            provider: openai<br>            name: text-embedding-3-small<br>            options:<br>              upstream_url: &quot;https://api.openai.com/v1/embeddings&quot;<br>        vectordb:<br>          dimensions: 1536<br>          distance_metric: cosine<br>          strategy: redis<br>          threshold: 0.8<br>          redis:<br>            host: redis-stack.redis.svc.cluster.local<br>            port: 6379<br>        targets:<br>        - model:<br>            provider: openai<br>            name: gpt-4<br>          route_type: &quot;llm/v1/chat&quot;<br>          auth:<br>            header_name: Authorization<br>            header_value: Bearer &lt;your_OPENAI_APIKEY&gt;<br>          description: &quot;mathematics, algebra, calculus, trigonometry&quot;<br>        - model:<br>            provider: openai<br>            name: gpt-4o-mini<br>          route_type: &quot;llm/v1/chat&quot;<br>          auth:<br>            header_name: Authorization<br>            header_value: Bearer &lt;your_OPENAI_APIKEY&gt;<br>          description: &quot;piano, orchestra, liszt, classical music&quot;</pre><p>The main configuration sections are:</p><ul><li>balancer with algorithm: semantic, telling the load balancer will be based on Semantic Routing.</li><li>embeddings with the necessary setting to reach out the Embedding Model. The same observations made previously regarding the API Key remains here.</li><li>vectordb with the Redis host and Index configurations as well as the threshold to drive the VSS query.</li><li>targets: each one of them represents a LLM model. Note the description parameter is used to configure the load balancing algorithm according to the topic the model has been trained for.</li></ul><p>As you can see, for convenience’s sake, the configuration uses OpenAI’s model for embeddings and targets. Also, just for this exploration, we are also using the gpt-4 and gpt-4o-mini OpenAI’s models for the targets.</p><p>After submitting the decK declaration to Konnect Control Plane, the Redis Vector Database should have a new index defined and a key for each target created. We can then start sending requests to the Gateway. The first two requests have contents related to Classical Musical, so the response should come from the related model, gpt-4o-mini-2024–07–18.</p><pre>% curl -s -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Who wrote the Hungarian Rhapsodies piano pieces?&quot;<br>      }<br>    ]<br>  }&#39; | jq &#39;.model&#39;<br>&quot;gpt-4o-mini-2024–07–18&quot;<br><br>% curl -s -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Tell me a contemporary pianist of Chopin&quot;<br>      }<br>    ]<br>  }&#39; | jq &#39;.model&#39;<br>&quot;gpt-4o-mini-2024–07–18&quot;</pre><p>Now, the next request is related to Mathematics, therefore the response comes from the other model, “gpt-4–0613”.</p><pre>% curl -s -X POST \<br>  --url $DATA_PLANE_LB/openai-route \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Tell me about Fermat&#39;&#39;s last theorem&quot;<br>      }<br>    ]<br>  }&#39; | jq &#39;.model&#39;<br>&quot;gpt-4–0613&quot;</pre><h3>Conclusion</h3><p>Kong has historically supported Redis to implement a variety of critical policies and use cases. The most recent collaborations, implemented by the Kong AI Gateway, focus on Semantic Processing where Redis Vector Similarity Search capabilities play an important role.</p><p>This blog post explored two main semantic-based use cases: Semantic Caching and Semantic Routing. Check Kong’s and Redis’ documentation pages to learn more about the extensive list of API and AI Gateway use cases you can implement using both technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf29b11eee5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing OpenTelemetry Observability with Kong Konnect and Dynatrace]]></title>
            <link>https://medium.com/@claudioacquaviva/implementing-opentelemetry-observability-with-kong-konnect-and-dynatrace-b2cb86862ba2?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2cb86862ba2</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Thu, 10 Apr 2025 01:05:28 GMT</pubDate>
            <atom:updated>2025-04-23T01:36:03.541Z</atom:updated>
            <content:encoded><![CDATA[<p>Observability has become critical to ensuring the effective monitoring of application and system performance and health. It focuses on understanding a system’s internal state by analyzing the data it produces in the context of real-time events and actions across the infrastructure. Unlike traditional monitoring, which mainly notifies you when issues arise, observability offers the tools and insights needed to determine not only that a problem exists but also its root cause. This enables teams to take a proactive approach to optimizing and managing systems, rather than simply responding to failures.</p><p>To achieve this, observability focuses on three core pillars:</p><ul><li>Logs: Detailed, timestamped records of events and activities within a system, offering a granular view of operations</li><li>Metrics: Quantitative data points that capture various aspects of system performance, such as resource usage, response times, and throughput</li><li>Traces: Visual paths that requests follow as they traverse through different system components, enabling end-to-end analysis of transactions and interactions</li></ul><p>Implementing observability at the API gateway layer is crucial. As the API gateway serves as a central point for managing and routing traffic across distributed services, applying observability to it brings an enterprise-wide perspective on how applications are consumed. Here are some important benefits it can bring:</p><ul><li>Observability at the API gateway layer allows you to track request patterns, response times, and error rates across all applications.</li><li>It enables faster problem identification, such as high latencies, unexpected error rates, or traffic spikes, before they are routed to other services.</li><li>It provides a first line of defense for troubleshooting and mitigation monitoring and logging suspicious activity, failed authentication attempts, or potential attacks.</li><li>Since API gateways route requests to multiple applications, observability can help map dependencies between services, offering insights into how they interact, such as failures and bottlenecks.</li><li>The API gateway can start and maintain distributed tracing across multiple applications by injecting trace IDs into all requests.</li><li>Last but not least, observability at the API gateway layer provides important business metrics like:</li><li>Most accessed APIs</li><li>Usage patterns by different clients or regions</li><li>API performance trends over time</li><li>Monitor and report on service-level agreements (SLAs) and compliance metrics effectively</li></ul><p>This blog post starts with a short overview of OTel and a reference architecture showing how Kong and Dynatrace work together. Next, it describes a basic Kong Konnect deployment integrated with Dynatrace through the OpenTelemetry (OTel) plugin to implement observability processes.</p><h3>OpenTelemetry Introduction</h3><p>Here’s a concise definition of OpenTelemetry, available on its <a href="https://opentelemetry.io/docs/">website</a>:</p><p>“OpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs.”</p><p>Born as a consolidation of <a href="https://opentelemetry.io/blog/2023/sunsetting-opencensus/">OpenTracing and OpenCensus</a> initiatives, OpenTelemetry has become a de facto standard supported by several vendors, including <a href="https://www.dynatrace.com/integrations/opentelemetry/">Dynatrace</a>.</p><h3>OTel Collector</h3><p>The OTel specification comprises several <a href="https://opentelemetry.io/docs/what-is-opentelemetry/#main-opentelemetry-components">components</a>, including, for example, the <a href="https://opentelemetry.io/docs/specs/otlp/">OpenTelemetry Protocol (OTLP)</a>. From the architecture perspective, one of the main components is the OpenTelemetry Collector, which is responsible for receiving, processing, and exporting telemetry data. The following diagram is taken from the official <a href="https://opentelemetry.io/docs/collector/">OpenTelemetry Collector documentation page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*YpS4ubkj0DOmaWPR" /></figure><p>Although it’s totally valid to send telemetry signals directly from the application to the observability backends with <a href="https://opentelemetry.io/docs/collector/deployment/no-collector/">no collector</a> in place, it’s generally recommended to use the OTel Collector. The collector abstracts the backend observability infrastructure, so the services can normalize this kind of processing more quickly in a standardized manner as well as let the collector take care of error handling, encryption, data filtering, transformation, etc.</p><p>As you can see in the diagram, the collector defines multiple components such as:</p><ul><li><a href="https://opentelemetry.io/docs/collector/configuration/#receivers">Receivers</a>: Responsible for collecting telemetry data from the sources</li><li><a href="https://opentelemetry.io/docs/collector/configuration/#processors">Processors</a>: Apply transformation, filtering, and calculation to the received data</li><li><a href="https://opentelemetry.io/docs/collector/configuration/#exporters">Exporters</a>: Send data to the Observability backend</li></ul><p>OTel Collector offers other types of connectors and extensions. Please, refer to <a href="https://opentelemetry.io/docs/collector/configuration/">OTel Collector documentation</a> to learn more about these components.</p><p>The components are tied together in <a href="https://opentelemetry.io/docs/collector/configuration/#pipelines">Pipelines</a>, inside the <a href="https://opentelemetry.io/docs/collector/configuration/#service">Service</a> section of the collector configuration file.</p><p>From the deployment perspective, here’s the minimum recommended scenario called <a href="https://opentelemetry.io/docs/collector/deployment/agent/">Agent Pattern</a>. The application uses the OTel SDK to send telemetry data to the collector through OTLP. The collector, in turn, sends the data to the existing backends. The collector is also flexible enough to support a variety of topologies to address scalability, high availability, fan-out, etc. Check the <a href="https://opentelemetry.io/docs/collector/deployment/">OTel Collector deployment page</a> for more information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EB0SpR-xVSlfyfgM" /></figure><p>The OTel Collector comes from the community, but Dynatrace provides a distribution for the <a href="https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector">OpenTelemetry Collector</a>. It is a customized implementation tailored for typical use cases in a Dynatrace context. It ships with an optimized and verified set of collector components.</p><h3>Kong Konnect and Dynatrace reference architecture</h3><p>The Kong Konnect and Dynatrace topology is quite simple in this example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*iytdNBA4tQTfal6M" /></figure><p>The main components here are:</p><ul><li><strong>Konnect Control Plane: </strong>responsible for administration tasks including APIs and Policies definition</li><li><strong>Konnect Data Plane: </strong>handles the requests sent by the API consumers</li><li><strong>Kong Gateway Plugins:</strong> components running inside the Data Plane to produce OpenTelemetry signals</li><li><strong>Upstream Service:</strong> services or microservices protected by the Konnnect Data Plane</li><li><strong>OpenTelemetry Collector:</strong> handles and processes the signals sent by the OTel plugin and sends them to the Dynatrace tenant</li><li><strong>Dynatrace Platform:</strong> provides a single pane of glass with dashboards, reports, etc.</li></ul><h3>Simple e-commerce application</h3><p>To demonstrate Kong and Dynatrace working together in a more realistic scenario, we’re going to take a simple e-commerce application and get it protected by Kong and monitored by Dynatrace. The <a href="https://github.com/odigos-io/simple-demo">application</a> is available publicly and here’s a diagram with a high-level architecture of the application with its microservices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*SjkLC_wAm_O67qOE" /></figure><p>Logically speaking, we can separate the microservices into two main layers:</p><ul><li>Backend with Inventory, Pricing, Coupon, and Membership microservices.</li><li>Frontend, responsible for sending requests to the Backend Services</li></ul><p>As we mentioned earlier, one of the main goals of an API gateway is to abstract services and microservices from the frontend perspective so we can take advantage of all its capabilities, including policies, protocol abstraction, etc.</p><p>In this sense, all incoming requests coming from the frontend microservices are processed by Kong Gateway and routed accordingly to the microservices sitting behind it.</p><h3>Application Instrumentation</h3><p>One of the Kong Gateway’s responsibilities is to generate and provide all signals to Dynatrace. This is done by the Kong Gateway Plugins, which can be configured accordingly. For example, from the tracing perspective, the Kong OpenTelemetry Plugin starts all traces. The Kong Prometheus Plugin is responsible for producing Prometheus-based metrics as the Kong TCP Log Plugin transmits the Kong Gateway access logs to Dynatrace.</p><p>The e-commerce app microservices aren’t prepared to be part of an observability environment. That is: all components should be instrumented to be part of it. In other words, the microservices must add observability code to emit <a href="https://opentelemetry.io/docs/concepts/signals/traces/">traces</a>, <a href="https://opentelemetry.io/docs/concepts/signals/metrics/">metrics</a>, and <a href="https://opentelemetry.io/docs/concepts/signals/logs/">logs</a>.</p><p>There are two <a href="https://opentelemetry.io/docs/concepts/instrumentation/">instrumentation</a> options:</p><ul><li><a href="https://opentelemetry.io/docs/concepts/instrumentation/code-based/">Code-based</a>: Through the use of SDKs the application, or microservice, can be extended to process the OpenTelemetry signals.</li><li><a href="https://opentelemetry.io/docs/concepts/instrumentation/zero-code/">Zero-code</a>: As the name implies, it doesn’t require any code to get injected into the microservices.</li></ul><p>Specifically for Kubernetes deployments, the OpenTelemetry Operator provides a Zero-code option, called <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/">Auto-Instrumentation</a>, which injects necessary code to the application. That’s the mechanism we are going to use in this blog post.</p><p>So, let’s get started with the reference architecture implementation and application deployment.</p><h3>Kong Konnect Data Plane and Dynatrace observability deployment</h3><p>It’s time to describe the actual deployment of the reference architecture. We can summarize it with the following steps:</p><ol><li>Pre-requisites: Kong Konnect and Dynatrace registration</li><li>Kubernetes cluster creation</li><li>Kong Konnect Data Plane deployment</li><li>OpenTelemetry Operator installation</li><li>OpenTelemetry Collector instantiation</li><li>e-commerce application deployment and instrumentation</li><li>Kong Objects creation and Traces setup</li><li>Adding Metrics and Logs to the OpenTelemetry Connector configuration</li></ol><h3>1. Pre-requisites: Kong Konnect and Dynatrace registration</h3><p>Before deploying the Data Plane we should subscribe to Konnect. Click on the <a href="https://konghq.com/products/kong-konnect/register">Konnect Registration</a> link and present your credentials. Or, if you already have a Konnect subscription, <a href="https://cloud.konghq.com/us">log in to it</a>. You should get redirected to the Konnect landing page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*APWF383tZMjndle3" /></figure><p>Similarly, go to the <a href="https://www.dynatrace.com/signup/">Dynatrace signup link</a> to register and get a 15-day trial. You should get redirected to its landing page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PlgdZUTc67Z9JhFl" /></figure><h3>2. Kubernetes cluster creation</h3><p>Our Konnect Data Plane, with all Kong Gateway Plugins, alongside the OTel Collector will be running on an Amazon EKS cluster. So, let’s create it using <a href="https://eksctl.io/">eksctl</a>, the official CLI for Amazon EKS, like this:</p><pre>eksctl create cluster --name kong-dynatrace \<br>--version 1.32 \<br>--region us-east-2 \<br>--nodegroup-name kong-node \<br>--node-type m4.xlarge \<br>--nodes 1</pre><h3>3. Kong Konnect Data Plane deployment</h3><p>Any Konnect subscription has a “default” Control Plane defined. Click on it and, inside its landing page, click on “Create a New Data Plane Node”.</p><p>Choose Kubernetes as your platform. Click on “Generate certificate”, copy and save the Digital Certificate and Private Key as tls.crt and tls.key files as described in the instructions. Also copy and save the configuration parameters in a values.yaml file.</p><p>The main comments here are:</p><ul><li>Replace the cluster_* endpoints and server names with yours.</li><li>The “tracing_instrumentations: all” and “tracing_sampling_rate: 1.0” parameters are needed for the Kong Gateway OpenTelemetry plugin we are going to describe later.</li><li>The Kong Data Plane is going to be consumed exclusively by the Frontend microservice, therefore it can be deployed as “type: ClusterIP”.</li></ul><pre>cat &gt; values.yaml &lt;&lt; &#39;EOF&#39;<br>image:<br>  repository: kong/kong-gateway<br>  tag: &quot;3.9&quot;<br><br>secretVolumes:<br>- kong-cluster-cert<br><br>admin:<br>  enabled: false<br><br>manager:<br>  enabled: false<br><br>env:<br>  role: data_plane<br>  database: &quot;off&quot;<br>  cluster_mtls: pki<br>  cluster_control_plane: abcdefc352.us.cp0.konghq.com:443<br>  cluster_server_name: abcdefc352.us.cp0.konghq.com<br>  cluster_telemetry_endpoint: abcdefc352.us.tp0.konghq.com:443<br>  cluster_telemetry_server_name: abcdefc352.us.tp0.konghq.com<br>  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt<br>  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key<br>  lua_ssl_trusted_certificate: system<br>  konnect_mode: &quot;on&quot;<br>  vitals: &quot;off&quot;<br>  nginx_worker_processes: &quot;1&quot;<br>  upstream_keepalive_max_requests: &quot;100000&quot;<br>  nginx_http_keepalive_requests: &quot;100000&quot;<br>  proxy_access_log: &quot;off&quot;<br>  dns_stale_ttl: &quot;3600&quot;<br>  tracing_instrumentations: all<br>  tracing_sampling_rate: 1.0<br><br>ingressController:<br>  enabled: false<br>  installCRDs: false<br><br>resources:<br>  requests:<br>    cpu: 1<br>    memory: &quot;2Gi&quot;<br><br>proxy:<br>  type: ClusterIP<br><br>status:<br>  enabled: true<br>  http:<br>    enabled: true<br>    containerPort: 8100<br>    parameters: []<br>EOF</pre><p>Now, use the Helm command to deploy the Data Plane. First, add Kong’s repo to your Helm environment.</p><pre>helm repo add kong https://charts.konghq.com<br>helm repo update</pre><p>Create a namespace and a secret for your Digital Certificate and Private Key pair and apply the values.yaml file:</p><pre>kubectl create namespace kong<br>kubectl create secret tls kong-cluster-cert -n kong --cert=./tls.crt --key=./tls.key<br>helm install kong kong/kong -n kong --values ./values.yaml</pre><p>You can check Kong Data Plane logs with:</p><pre>kubectl logs -f $(kubectl get pod -n kong -o json | jq -r &#39;.items[].metadata | select(.name | startswith(&quot;kong-&quot;))&#39; | jq -r &#39;.name&#39;) -n kong</pre><p>Check the Kubernetes Service, related to the Kong Data Plane, with:</p><pre>% kubectl get service -n kong -o json | jq &#39;.items[].metadata.name&#39;<br>&quot;kong-kong-proxy&quot;</pre><h3>4. OpenTelemetry Operator installation</h3><p>The next step is to deploy the OpenTelemetry Collector. To get better control over it, we’re going to do it through the <a href="https://github.com/open-telemetry/opentelemetry-operator">OpenTelemetry Kubernetes Operator</a>. In fact, the collector is also capable of auto-instrument applications and services using OpenTelemetry instrumentation libraries.</p><h3>Installing Cert-Manager</h3><p>The OpenTelemetry Operator requires <a href="https://cert-manager.io/">Cert-Manager</a> to be installed in your Kubernetes cluster. The Cert-Manager can then issue certificates to be used by the communication between the Kubernetes API Server and the existing webhook included in the operator.</p><p>Use the <a href="https://cert-manager.io/docs/installation/helm/">Cert-Manager Helm Charts</a> to get it installed. Add the repo first:</p><pre>helm repo add jetstack https://charts.jetstack.io</pre><p>Install Cert-Manager with:</p><pre>helm install \<br>  cert-manager jetstack/cert-manager \<br>  --namespace cert-manager \<br>  --create-namespace \<br>  --version v1.16.2 \<br>  --set crds.enabled=true</pre><h3>Installing OpenTelemetry Operator</h3><p>Now we’re going to use the <a href="https://github.com/open-telemetry/opentelemetry-helm-charts">OpenTelemetry Helm Charts</a> to install it. Add its repo:</p><pre>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts</pre><p>Install the operator:</p><pre>helm install opentelemetry-operator open-telemetry/opentelemetry-operator \<br>--namespace opentelemetry-operator-system \<br>--create-namespace \<br>--set manager.collectorImage.repository=otel/opentelemetry-collector-k8s \<br>--set manager.image.tag=0.114.0 \<br>--set manager.extraArgs={&quot; - enable-go-instrumentation=true&quot;} \<br>--set admissionWebhooks.certManager.enabled=true</pre><p>As you may recall, one of the microservices of the e-Commerce application is written in <a href="https://go.dev/">Go Programming Language</a>. By default, the operator has the Go instrumentation disabled. The Helm command has the “extraArgs” parameter to enable it.</p><p>The “admissionWebhooks” parameter asks Cert-Manager to generate a self-signed certificate. The operator installs some new CRDs used to create a new OpenTelemetry Collector. You can check them out with:</p><pre>kubectl describe crd opentelemetrycollectors.opentelemetry.io<br>kubectl describe crd instrumentations.opentelemetry.io</pre><h3>5. OpenTelemetry Collector instantiation</h3><p>With the operator in place, we can use the new CRD to deploy our OpenTelemetry Collector. Dynatrace provides a first-class distribution of the <a href="https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector">collector</a>, including support and security patches independent of the OpenTelemetry Collector release.</p><p>The Dynatrace distribution of the OpenTelemetry Collector supports the following components, including Receivers, Processors, Exporters, and Extensions Connectors, described <a href="https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector#dt-collector-dist">here</a>.</p><h3>Create a collector declaration</h3><p>To get started we’re going to manage Traces first. Later on, we’ll enhance the collector to process both Metrics and Logs. Here’s the declaration:</p><pre>cat &gt; otelcollector-dynatrace-traces.yaml &lt;&lt; &#39;EOF&#39;<br>apiVersion: opentelemetry.io/v1beta1<br>kind: OpenTelemetryCollector<br>metadata:<br>  name: collector-kong<br>  namespace: opentelemetry-operator-system<br>spec:<br>  image: ghcr.io/dynatrace/dynatrace-otel-collector/dynatrace-otel-collector:latest<br>  mode: deployment<br>  env:<br>  - name: DT_ENDPOINT<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-endpoint<br>        name: dynatrace-endpoint<br>  - name: DT_API_TOKEN<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-access-token<br>        name: dynatrace-access-token<br>  - name: KONG_DEPLOYMENT_NAME<br>    value: kong-ecommerce-deployment<br>  config:<br>    receivers:<br>      otlp:<br>        protocols:<br>          grpc:<br>            endpoint: 0.0.0.0:4317<br>          http:<br>            endpoint: 0.0.0.0:4318<br><br>    exporters:<br>      otlphttp:<br>        endpoint: &quot;${env:DT_ENDPOINT}&quot;<br>        headers:<br>          Authorization: &quot;Api-Token ${env:DT_API_TOKEN}&quot;<br>      debug:<br>        verbosity: detailed<br><br>    processors:<br>      attributes:<br>        actions:<br>          - key: kong.deployment.name<br>            value: &quot;${KONG_DEPLOYMENT_NAME}&quot;<br>            action: insert<br><br>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: [attributes]<br>          exporters: [otlphttp]<br>EOF</pre><p>The declaration has critical parameters defined:</p><ul><li>image: it refers to the Dynatrace OTel Collector distribution.</li><li>mode: deployment. The collector can be deployed in <a href="https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file#deployment-modes">4 different modes</a>: “Deployment”, “DaemonSet”, “StatefulSet” and “Sidecar”. For better control of the Controller, we’ve chosen regular Kubernetes Deployment mode. Please, refer to the <a href="https://kubernetes.io/docs/concepts/workloads/">Kubernetes documentation</a> to learn more about them.</li><li>The collector requires the Dynatrace Endpoint and API Token to connect. They are stored as Kubernetes secrets and are referred to in the “env” section.</li><li>The config section has the collector components (receivers and exporters) as well as the “service” section defining the Pipeline.</li><li>The “receivers” section tells us the collector will be listening to ports 4317 and 4318 and will be receiving data over “grpc” and “http”.</li><li>The “exporters” section used the endpoint and the API Token to send data to Dynatrace.</li><li>The “service” section defines the Pipeline. Note we have a simple “attributes processor” defined setting the “kong.deployment.name” attribute as “kong-ecommerce-deployment” to be used by the Dynatrace dashboards.</li></ul><h3>Dynatrace Secrets</h3><p>Before instantiating the collector we need to create the Kubernetes secrets with the Dynatrace endpoint and API Token. The endpoint refers to your Dynatrace tenant:</p><pre>kubectl create secret generic dynatrace-endpoint -n opentelemetry-operator-system - from-literal=dt-endpoint=&#39;https://&lt;your_Dynatrace_tenant&gt;.live.dynatrace.com/api/v2/otlp&#39;</pre><p>And create another secret for the Dynatrace API Token. The Token needs to be created with the “Ingest metrics” scope. Since we’re going to manage Logs and Traces, add the “Ingest logs” and “Ingest OpenTelemetry traces” scopes as well. Check the Dynatrace documentation to learn how to issue <a href="https://docs.dynatrace.com/docs/manage/identity-access-management/access-tokens-and-oauth-clients/access-tokens-legacy#api">API Tokens</a>.</p><pre>kubectl create secret generic dynatrace-access-token -n opentelemetry-operator-system - from-literal=dt-access-token=&#39;&lt;your_API_TOKEN&gt;&#39;</pre><p>In case you want to decode your secret, use the base64 command. For example:</p><pre>kubectl get secret dynatrace-endpoint -n opentelemetry-operator-system -o json | jq -r &#39;.data.&quot;dt-endpoint&quot;&#39; | base64 - decode</pre><h3>Deploy the collector</h3><p>You can instantiate the collector by simply submitting the declaration:</p><pre>kubectl apply -f otelcollector-dynatrace-traces.yaml</pre><p>If you want to destroy it run:</p><pre>kubectl delete opentelemetrycollector collector-kong -n opentelemetry-operator-system</pre><p>Check the collector’s log with:</p><pre>kubectl logs -f $(kubectl get pod -n opentelemetry-operator-system -o json | jq &#39;.items[].metadata | select(.name | startswith(&quot;collector&quot;))&#39; | jq -r &#39;.name&#39;) -n opentelemetry-operator-system</pre><p>Based on the declaration, the deployment creates a Kubernetes service named “collector-kong-collector” listening to ports 4317 and 4318. That means that any application, including Kong Data Plane, should refer to the OTel Collector’s Kubernetes FQDN (e.g., <a href="http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318/v1/traces)">http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318/v1/traces)</a> to send data to the collector. The <a href="https://opentelemetry.io/docs/specs/otlp/#otlphttp-request">“/v1/traces”</a> path is the default the collector uses to handle requests with trace data.</p><pre>% kubectl get service collector-kong-collector -n opentelemetry-operator-system<br>NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE<br>collector-kong-collector   ClusterIP   10.100.23.139   &lt;none&gt;        4317/TCP,4318/TCP   29m</pre><h3>6. e-commerce application instrumentation and deployment</h3><p>One of the most powerful capabilities provided by the OpenTelemetry Operator is the ability to <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/">auto-instrument</a> your services. With such functionality, the operator injects and configures libraries for <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#net">.Net</a>, <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#java">Java</a>, <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#nodejs">Node.js</a>, <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#python">Python</a>, and <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#go">Go</a> services. Considering the multi-programming language scenario, it fits nicely to the e-Commerce application.</p><p>Here’s a diagram illustrating how Auto-Instrumentation works. The instrumentation depends on the Programming Language we used to develop the microservices. For some languages, the CRD uses the OTLP HTTP protocol, which is configured in the OTel Collector listening to port 4318. That’s the case of all languages, excluding Node.js, where the CRD uses gRPC and therefore should refer to port 4317.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*RzSGBV1pxkPWcdxf" /></figure><p>The instrumentation process is divided into two steps:</p><ul><li>Configuration using the Instrumentation Kubernetes CRD</li><li>Kubernetes deployment annotations to get the code instrumented.</li></ul><h3>Instrumentation Configuration</h3><p>First of all, we need to use the Instrumentation CRD installed to configure it. Here’s the declaration:</p><pre>cat &gt; otel-instrumentation.yaml &lt;&lt; &#39;EOF&#39;<br>apiVersion: opentelemetry.io/v1alpha1<br>kind: Instrumentation<br>metadata:<br>  name: instrumentation1<br>spec:<br>  propagators:<br>    - tracecontext<br>  sampler:<br>    type: parentbased_traceidratio<br>    argument: &quot;1&quot;<br>  nodejs:<br>    env:<br>      - name: OTEL_EXPORTER_OTLP_ENDPOINT<br>        value: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4317<br>  python:<br>    env:<br>      - name: OTEL_EXPORTER_OTLP_ENDPOINT<br>        value: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318<br>  dotnet:<br>    env:<br>      - name: OTEL_EXPORTER_OTLP_ENDPOINT<br>        value: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318<br>  go:<br>    env:<br>      - name: OTEL_EXPORTER_OTLP_ENDPOINT<br>        value: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318<br>      - name: OTEL_SERVICE_NAME<br>        value: membership<br>    image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.20.0<br>EOF</pre><p>Some comments about it:</p><ul><li>For tracing, with the “propagators” section, it supports the OpenTelemetry <a href="https://opentelemetry.io/docs/concepts/context-propagation/">Context Propagation</a>, which defines the <a href="https://www.w3.org/TR/trace-context/">W3C TraceContext</a> specification as the default propagator.</li><li>The “sampler” section controls the number of traces collected (<a href="https://opentelemetry.io/docs/concepts/sampling/">sampling</a>) and sent to the Observability system, in our case, Dynatrace.</li><li>As we mentioned earlier, the OTLP endpoint the Instrumentation should use depends on the Programming Language. The declaration has a specific configuration for each one of them.</li><li>Lastly, as an exercise, for the Go Lang section, we’ve configured the OTel service name as well as the Image the Auto-Instrumentation process should use.</li></ul><p>For all Instrumentation CRD declaration sections, check the <a href="https://opentelemetry.io/docs/languages/sdk-configuration/general/">General OpenTelemetry SDK configuration</a> page where you can find all related parameters. For example: the <a href="https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_traces_sampler">OTEL_TRACES_SAMPLER</a> SDK configuration maps the “sampler” CRD section declaration.</p><p>Configure the Auto-Instrumentation process by submitting the declaration:</p><pre>kubectl apply -f otel-instrumentation.yaml</pre><p>If you want to delete it run:</p><pre>kubectl delete instrumentation instrumentation1</pre><h3>Application deployment</h3><p>Now, with the Auto-Intrumentation process properly instructed, let’s deploy the Application. The original Kubernetes declaration can be downloaded from:</p><pre>wget https://raw.githubusercontent.com/odigos-io/simple-demo/refs/heads/main/kubernetes/deployment.yaml</pre><p>There are two main points we should change in order to get the Application properly deployed:</p><ul><li>Originally, the frontend microservice is configured to communicate directly with each one of the backend microservices (Inventory, Pricing and Coupon). We need to change that saying it should send requests to Kong API Gateway instead.</li><li>In order to get instrumented, each microservice has to have specific annotations so the Auto-Instrumentation process can take care of it and inject the necessary code into it.</li></ul><h4>Frontend deployment declaration</h4><p>Here’s the original Frontend microservice deployment declaration:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: frontend<br>  labels:<br>    app: frontend<br>spec:<br>  selector:<br>    matchLabels:<br>      app: frontend<br>  template:<br>    metadata:<br>      labels:<br>        app: frontend<br>    spec:<br>      containers:<br>      - name: frontend<br>        image: keyval/odigos-demo-frontend:v0.1.14<br>        imagePullPolicy: Always<br>        securityContext:<br>          runAsUser: 1000<br>        env:<br>        - name: INVENTORY_SERVICE_HOST<br>          value: inventory:8080<br>        - name: PRICING_SERVICE_HOST<br>          value: pricing:8080<br>        - name: COUPON_SERVICE_HOST<br>          value: coupon:8080<br>        ports:<br>        - containerPort: 8080</pre><p>You can manually update the declaration or use the <a href="https://github.com/mikefarah/yq">yq</a> tool. yq is a powerful YAML, JSON and XML text processor very useful if you want to automate the process.</p><p>The following commands update the “env” section, replacing the original endpoint with Kong references, including the Kong’s Kubernetes Service FQDN (“kong-kong-proxy.kong”) and the Kong Route (e.g. “/inventory”). We don’t have the Kong Routes defined yet as it is what the next section of the document describes. Please check the <a href="https://mikefarah.gitbook.io/yq">yq documentation</a> to learn more about it.</p><pre>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;frontend&quot;).spec.template.spec.containers[0].env[0].value |= &quot;kong-kong-proxy.kong/inventory&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;frontend&quot;).spec.template.spec.containers[0].env[1].value |= &quot;kong-kong-proxy.kong/pricing&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;frontend&quot;).spec.template.spec.containers[0].env[2].value |= &quot;kong-kong-proxy.kong/coupon&quot;&#39; deployment.yaml</pre><h4>Kubernetes deployment annotations</h4><p>The second update is going to add Kubernetes annotations so the Auto-Instrumentation process can do its jobs. Here are the yq commands. Note that for each microservice, we use a different <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#add-annotations-to-existing-deployments">annotation</a> to tell the Auto-Instrumentation process which Programming Language it should consider. For example, the “Inventory” microservice, written in Python, has the “instrumentation.opentelemetry.io/inject-python”: “true” annotation injected.</p><p>The Auto-Instrumentation process works slightly differently for <a href="https://opentelemetry.io/docs/kubernetes/operator/automatic/#opt-in-a-go-service">Go Lang</a>-based microservices, like “Membership”. It requires another annotation with the path of the target Container executable. If you check the “<a href="https://hub.docker.com/layers/keyval/odigos-demo-membership/v0.1.14/images/sha256-eb1b8cac40fec814f4d2d784200d08cc6a0a3349fdfc6e08fd37432b781984c9">keyval/odigos-demo-membership:v0.1.14</a>” image you’ll see the entry point is defined as “/membership”.</p><pre>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;coupon&quot;).spec.template.metadata.annotations.[&quot;instrumentation.opentelemetry.io/inject-nodejs&quot;] |= &quot;true&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;inventory&quot;).spec.template.metadata.annotations.[&quot;instrumentation.opentelemetry.io/inject-python&quot;] |= &quot;true&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;membership&quot;).spec.template.metadata.annotations.[&quot;instrumentation.opentelemetry.io/inject-go&quot;] |= &quot;true&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;membership&quot;).spec.template.metadata.annotations.[&quot;instrumentation.opentelemetry.io/otel-go-auto-target-exe&quot;] |= &quot;/membership&quot;&#39; deployment.yaml<br>yq -i e &#39;select(.kind == &quot;Deployment&quot; and .metadata.name == &quot;pricing&quot;).spec.template.metadata.annotations.[&quot;instrumentation.opentelemetry.io/inject-dotnet&quot;] |= &quot;true&quot;&#39; deployment.yaml</pre><p>Here’s a snippet showing the annotations injected for the “Pricing” microservice:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: pricing<br>  labels:<br>    app: pricing<br>spec:<br>  selector:<br>    matchLabels:<br>      app: pricing<br>  template:<br>    metadata:<br>      labels:<br>        app: pricing<br>      annotations:<br>        instrumentation.opentelemetry.io/inject-dotnet: &quot;true&quot;<br>    spec:<br>      containers:<br>        - name: pricing<br>          image: keyval/odigos-demo-pricing:v0.1.14<br>          imagePullPolicy: Always<br>          ports:<br>            - containerPort: 8080</pre><p>Now we’re finally ready to deploy the app:</p><pre>kubectl apply -f deployment.yaml</pre><p>If you want to delete it run:</p><pre>kubectl delete -f deployment.yaml</pre><h3>Check the deployment</h3><p>To get a better understanding of the manipulation the Auto-Instrumentation process does, let’s check, for example, the “Inventory” Pod, where the microservice is written in Python. If you run:</p><pre>kubectl get pod -o yaml $(kubectl get pod -o json | \<br>jq &#39;.items[].metadata | select(.name | startswith(&quot;inventory&quot;))&#39; | \<br>jq -r &#39;.name&#39;) | yq &#39;.spec&#39;</pre><p>You’ll see the Auto-Instrumentation process ran an Init Container to instrument the code and defined environment variables specifying how to connect to the OTel Collector. Check the <a href="https://github.com/open-telemetry/opentelemetry-python-contrib">Python auto-intrumentation repo</a> to learn more.</p><p>Another interesting check is the “Membership” Pod. Since the microservice is written in Go, the Auto-Instrumentation solves the problem in a different way. To check that out, if you run:</p><pre>kubectl get pod -o yaml $(kubectl get pod -o json | \<br>jq &#39;.items[].metadata | select(.name | startswith(&quot;membership&quot;))&#39; | \<br>jq -r &#39;.name&#39;) | yq &#39;.spec.containers[].name&#39;</pre><p>You should get:</p><pre>membership<br>opentelemetry-auto-instrumentation</pre><p>Meaning, the process has injected another container inside the Pod, playing the sidecar role. Again, it also defines environment variables just like it did for the first Pod we checked. Here’s the <a href="https://github.com/open-telemetry/opentelemetry-go-instrumentation">Go auto-instrumentation repo</a> to learn more.</p><h3>7. Kong Objects creation and Traces setup</h3><p>To complete our initial deployment we need to configure the Kong Objects to expose the e-commerce Backend microservices and <a href="https://docs.konghq.com/hub/kong-inc/opentelemetry/">Kong Gateway OpenTelemetry plugin</a>.</p><h3>decK and Konnect PAT</h3><p>The OpenTelemetry Plugin and the actual Kong Objects will be configured using <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong), a command line that allows you to manage Kong Objects in a declarative way. In order to use it, we need a Konnect PAT (Personal Access Token). Please, refer to the Konnect documentation to learn how to generate a <a href="https://docs.konghq.com/konnect/org-management/access-tokens/">PAT</a>.</p><h3>Kong Services and Routes decK declaration</h3><p>Below you can see two decK declarations. The first one defines Kong Services and Routes. The second manages the Kong Plugins.</p><p>For the first declaration here are the main comments:</p><ul><li>A Kong Service for each e-commerce Backend microservice (Coupon, Inventory, and Pricing). Note that, since the Membership microservice is not consumed directly by the Frontend microservice, we don’t need to define a Kong Service for it.</li><li>A Kong Route for each Kong Service to expose them with the specific paths “/coupon”, “/inventory” and “/pricing”. Each Kong Route matches the Kubernetes declaration update we did for the Frontend microservice.</li></ul><pre>cat &gt; kong-services-routes.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - kong-services-routes<br>_konnect:<br>  control_plane_name: default<br>services:<br>- name: ecommerce_coupon_service<br>  host: coupon.default<br>  port: 8080<br>  routes:<br>  - name: ecommerce_coupon_route<br>    paths:<br>    - /coupon<br>- name: ecommerce_inventory_service<br>  host: inventory.default<br>  port: 8080<br>  routes:<br>  - name: ecommerce_inventory_route<br>    paths:<br>    - /inventory<br>- name: ecommerce_pricing_service<br>  host: pricing.default<br>  port: 8080<br>  routes:<br>  - name: ecommerce_pricing_route<br>    paths:<br>    - /pricing<br>EOF</pre><h3>Kong Plugins decK declaration</h3><p>The second declaration defines</p><ul><li>A globally configured Kong OpenTelemetry Plugin, meaning it’s going to be applied to all Kong Services. The main configuration here is the “traces_endpoint” parameter. As you can see, it refers to the OpenTelemetry Collector instance we deployed previously.</li><li>The Plugin supports the OpenTelemetry Context Propagation, which defines the W3C TraceContext specification as the default propagator.</li><li>The Plugin sets “kong-otel” as the name of the Service getting monitored.</li></ul><pre>cat &gt; kong-plugins.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - kong-plugins<br>_konnect:<br>  control_plane_name: default<br>plugins:<br>- name: opentelemetry<br>  instance_name: opentelemetry1<br>  enabled: true<br>  config:<br>    traces_endpoint: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318/v1/traces<br>    propagation:<br>      default_format: &quot;w3c&quot;<br>      inject: [&quot;w3c&quot;]<br>    resource_attributes:<br>      service.name: &quot;kong-otel&quot;<br>EOF</pre><p>The Kong Gateway OpenTelemetry Plugin supports other propagators through the following headers: Zipkin, Jaeger, OpenTracing, Datadog, AWS X-Ray, and GCP X-Cloud-Trace-Context. The plugin also allows us to extract, inject, clear or preserve header to and out of the incoming requests.</p><p>All the instrumentations made by the plugin are configured with the “<a href="https://docs.konghq.com/gateway/latest/reference/configuration/#tracing_instrumentations">tracing_instrumentations</a>: all” and “<a href="https://docs.konghq.com/gateway/latest/reference/configuration/#tracing_sampling_rate">tracing_sampling_rate</a>: 1.0” parameters we used for the Kong Data Plane deployment. As you can imagine the sampling rate is critical and can impact the overall performance of the Data Plane. For the purpose of this blog post we have set it as “1.0”. For a production-ready environment you should configure it accordingly. The plugin also has a “sampling_rate” parameter that can be used to override the Data Plane configuration.</p><p>Now, before submitting the declaration to Konnect to create the Kong Objects, you can test the connection first. Please, define a PAT environment variable with your Konnect PAT.</p><pre>deck gateway ping --konnect-token $PAT</pre><p>Apply the declarations with:</p><pre>deck gateway sync --konnect-token $PAT kong-services-routes.yaml<br>deck gateway sync --konnect-token $PAT kong-plugins.yaml</pre><p>Now we’re free to use the application and get the Kong Routes consumed.</p><h3>Consume the e-Commerce Application</h3><p>The Frontend Service has been deployed as “type=ClusterIP” so we need to expose with, for example:</p><pre>kubectl port-forward service/frontend 8080</pre><p>For MacOS, you can open the application with:</p><pre>open -a &quot;Google Chrome&quot; &quot;http://localhost:8080&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*op0s65Bl5q87VBTz" /></figure><p>You may note that the Frontend application does not present any other page. In fact, you can see the microservice activities checking its logs like:</p><pre>kubectl logs -f $(kubectl get pod -o json | \<br>jq &#39;.items[].metadata | select(.name | startswith(&quot;frontend&quot;))&#39; | \<br>jq -r &#39;.name&#39;)</pre><h3>Dynatrace Distributed Tracing</h3><p>You should see new incoming <a href="https://docs.dynatrace.com/docs/analyze-explore-automate/distributed-tracing/app-overview#charts">traces in Dynatrace</a>. For example, the “/coupon” trace was started by Kong and has two spans added by the “Coupon” and “Membership” microservices, showing us the Auto-Instrumentation process is working properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Bch1OGFox4tAPUA" /></figure><h3>8. Adding Metrics and Logs to the OpenTelemetry Connector configuration</h3><p>Now, let’s add Metrics and Logs to our environment. Kong has supported Prometheus-based metrics for a long time through the <a href="https://docs.konghq.com/hub/kong-inc/prometheus/">Prometheus Plugin</a>. In an OpenTelemetry configuration scenario the plugin is an option, where we could add a specific <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md">“prometheusreceiver”</a> to the collector configuration. The receiver is responsible for scraping the Data Plane’s <a href="https://docs.konghq.com/gateway/latest/reference/configuration/#status_listen">Status API</a>, which, by default, is configured with the “:8100/metrics” endpoint.</p><p>To inject Kong Gateway’s Access Logs, we can use Log Processing plugin Kong Gateway provides, for example the <a href="https://docs.konghq.com/hub/kong-inc/tcp-log/">TCP Log Plugin</a>.</p><h3>New collector configuration</h3><pre>cat &gt; otelcollector-dynatrace-traces-metrics-logs.yaml &lt;&lt; &#39;EOF&#39;<br>apiVersion: opentelemetry.io/v1beta1<br>kind: OpenTelemetryCollector<br>metadata:<br>  name: collector-kong<br>  namespace: opentelemetry-operator-system<br>spec:<br>  image: otel/opentelemetry-collector-contrib:0.119.0<br>  serviceAccount: collector<br>  mode: deployment<br>  env:<br>  - name: DT_ENDPOINT<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-endpoint<br>        name: dynatrace-endpoint<br>  - name: DT_API_TOKEN<br>    valueFrom:<br>      secretKeyRef:<br>        key: dt-access-token<br>        name: dynatrace-access-token<br>  - name: KONG_DEPLOYMENT_NAME<br>    value: kong-ecommerce-deployment<br>  config:<br>    receivers:<br>      otlp:<br>        protocols:<br>          grpc:<br>            endpoint: 0.0.0.0:4317<br>          http:<br>            endpoint: 0.0.0.0:4318<br>      prometheus:<br>        config:<br>          scrape_configs:<br>            - job_name: &#39;otel-collector&#39;<br>              scrape_interval: 5s<br>              kubernetes_sd_configs:<br>              - role: pod<br>              scheme: http<br>              tls_config:<br>                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt<br>              authorization:<br>                credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token<br>              metrics_path: /metrics<br>              relabel_configs:<br>              - source_labels: [__meta_kubernetes_namespace]<br>                action: keep<br>                regex: &quot;kong&quot;<br>              - source_labels: [__meta_kubernetes_pod_name]<br>                action: keep<br>                regex: &quot;kong-kong-(.+)&quot;<br>              - source_labels: [__meta_kubernetes_pod_container_name]<br>                action: keep<br>                regex: &quot;proxy&quot;<br>              - source_labels: [__meta_kubernetes_pod_container_port_number]<br>                action: keep<br>                regex: &quot;8100&quot;<br>      tcplog:<br>        listen_address: 0.0.0.0:54525<br>        operators:<br>          - type: json_parser<br><br>    exporters:<br>      otlphttp:<br>        endpoint: &quot;${env:DT_ENDPOINT}&quot;<br>        headers:<br>          Authorization: &quot;Api-Token ${env:DT_API_TOKEN}&quot;<br>      prometheus:<br>        endpoint: 0.0.0.0:8889<br>      debug:<br>        verbosity: detailed<br><br>    processors:<br>      cumulativetodelta:<br>        include:<br>      attributes: {}<br>        actions:<br>          - key: kong.deployment.name<br>            value: &quot;${KONG_DEPLOYMENT_NAME}&quot;<br>            action: insert<br><br>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: []<br>          exporters: [otlphttp]<br>        metrics:<br>          receivers: [otlp, prometheus]<br>          processors: [cumulativetodelta, attributes]<br>          exporters: [otlphttp, prometheus]<br>        logs:<br>          receivers: [otlp, tcplog]<br>          processors: []<br>          exporters: [otlphttp]<br>EOF</pre><p>The declaration has critical parameters defined:</p><ul><li>image: it refers to the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib">“contrib” distribution</a> of the Collector.</li><li>A new <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver">TCP Receiver</a> has been added, listening to the port 54525, used by the Kong Gateway TCP Log Plugin. It uses the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/json_parser.md">“json_parser” operator</a> to send formatted data to Dynatrace.</li><li>Inside the “service” configuration section, a new “metrics” pipeline have been included:</li><li>The <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter">Prometheus exporter</a> configured, so we can access the metrics sending requests directly to the collector through port 8889 as described in the exporter section.</li><li>It also includes the “otlp” receiver, so it can grab metrics coming from the Backend microservices as well.</li><li>It has <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/cumulativetodeltaprocessor/README.md">“cumulativetodelta”</a> as a Processor. A <a href="https://opentelemetry.io/docs/collector/configuration/#processors">Processor</a> is another OTel Collector construct, responsible for taking the data collected by receivers and modifying it before sending it to the exporters. Basically, the “Cumulative to Delta” Processor converts the Histogram metrics with <a href="https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality">cumulative temporality</a>, produced by the Kong Prometheus plugin, to delta temporality, supported by Dynatrace.</li><li>Still inside the “service” section we have included the new “logs” pipeline. Its “receivers” are set to “otlp” and “tcplog” to get data from both Kong Gateway Plugin. Its “exporters” is set to the same “otlphttp” which sends data to Dynatrace.</li></ul><h3>Kubernetes Service Account for Prometheus Receiver</h3><p>The OTel Collector <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md">Prometheus Receiver</a> fully supports the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config">scraping configuration</a> defined by Prometheus. The receiver, more precisely, uses the “<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod">pod</a>” role of the Kubernetes Service Discovery configurations (“<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config">kubernetes_sd_config</a>”). Specific “<a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config">relabel_config</a>” settings with “regex” expressions allow the receiver to discover Kubernetes Pods that belong to the Kong Data Plane deployment.</p><p>One of the relabeling configs is related to the port 8100. This port configuration is part of the Data Plane deployment we used to get it running. Here’s the snippet of the “values.yaml” file we used previously:</p><pre>status:<br>  enabled: true<br>  http:<br>    enabled: true<br>    containerPort: 8100<br>    parameters: []</pre><p>That’s the Kong Gateway’s Status API where the Prometheus plugin exposes the metrics produced. In fact, the endpoint the receiver scrapes is, as specified in the OTel Collector configuration.</p><pre>http://&lt;Data_Plane_Pod_IP&gt;:8100/metrics</pre><p>On the other hand, the OTel Collector has to be allowed to scrape the endpoint. We can define such permission with a Kubernetes ClusterRole and apply it to a Kubernetes Service Account with a Kubernetes ClusterRoleBinding.</p><p>Here’s the ClusterRole declaration. It’s a quite open one but it’s good enough for this exercise.</p><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>    name: pod-reader<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;pods&quot;]<br>  verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]<br>EOF</pre><p>Then we need to create a Kubernetes Service Account and bind the Role to it.</p><pre>kubectl create sa collector -n opentelemetry-operator-system<br><br>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: read-pods<br>roleRef:<br>  kind: ClusterRole<br>  name: pod-reader<br>subjects:<br>- kind: ServiceAccount<br>  name: collector<br>  namespace: opentelemetry-operator-system<br>EOF</pre><p>Finally, note that the OTel Collector configuration is deployed using the Service Account with serviceAccount: collector and then it will be able to scrape the endpoint exposed by Kong Gateway.</p><h3>Deploy the collector</h3><p>Delete the current collector first and instantiate a new one simply submitting the declaration:</p><pre>kubectl delete opentelemetrycollector collector-kong -n opentelemetry-operator-system<br>kubectl apply -f otelcollector-dynatrace-traces-metrics-logs.yaml</pre><p>Interestingly enough, the collector service now listens to four ports:</p><pre>% kubectl get service collector-kong-collector -n opentelemetry-operator-system<br>NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                AGE<br>collector-kong-collector   ClusterIP   10.100.67.18   &lt;none&gt;        4317/TCP,4318/TCP,8889/TCP,54525/TCP   21h</pre><h3>Configure the Prometheus and TCP Log Plugins</h3><p>Add the Prometheus and TCP Log plugins to our decK declaration and submit it to Konnect:</p><pre>cat &gt; kong-plugins.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - kong-plugins<br>_konnect:<br>  control_plane_name: default<br>plugins:<br>- name: opentelemetry<br>  instance_name: opentelemetry1<br>  enabled: true<br>  config:<br>    traces_endpoint: http://collector-kong-collector.opentelemetry-operator-system.svc.cluster.local:4318/v1/traces<br>    propagation:<br>      default_format: &quot;w3c&quot;<br>      inject: [&quot;w3c&quot;]<br>    resource_attributes:<br>      service.name: &quot;kong-otel&quot;<br>- name: prometheus<br>  instance_name: prometheus1<br>  config:<br>    per_consumer: true<br>    status_code_metrics: true<br>    latency_metrics: true<br>    bandwidth_metrics: true<br>    upstream_health_metrics: true<br>    ai_metrics: true<br>- name: tcp-log<br>  instance_name: tcp-log1<br>  enabled: true<br>  config:<br>    host: collector-kong-collector.opentelemetry-operator-system.svc.cluster.local<br>    port: 54525<br>    custom_fields_by_lua:<br>      trace_id: local log_payload = kong.log.serialize() local trace_id = log_payload[&#39;trace_id&#39;][&#39;w3c&#39;] return trace_id<br>EOF</pre><p>Submit the new plugin declaration with:</p><pre>deck gateway sync --konnect-token $PAT kong-plugins.yaml</pre><p>Consume the Application and check collector’s Prometheus endpoint</p><p>Using “port-forward”, send a request to the collector’s Prometheus endpoint. In a terminal run:</p><pre>kubectl port-forward service/collector-kong-collector -n opentelemetry-operator-system 8889</pre><p>Continue navigating the Application to see some metrics getting generated. In another terminal send a request to Prometheus’ endpoint.</p><pre>% http :8889/metrics</pre><p>You should see several related Kong metrics including, for example, Histogram metrics like “kong_kong_latency_ms_bucket”, “kong_request_latency_ms_bucket” and “kong_upstream_latency_ms_bucket”. Maybe one of the most important is “kong_http_requests_total” where we can see consumption metrics. Here’s a snippet of the output:</p><pre># HELP kong_http_requests_total HTTP status codes per consumer/service/route in Kong<br># TYPE kong_http_requests_total counter<br>kong_http_requests_total{code=&quot;200&quot;,instance=&quot;192.168.76.233:8100&quot;,job=&quot;otel-collector&quot;,route=&quot;coupon_route&quot;,service=&quot;coupon_service&quot;,source=&quot;service&quot;,workspace=&quot;default&quot;} 1<br>kong_http_requests_total{code=&quot;200&quot;,instance=&quot;192.168.76.233:8100&quot;,job=&quot;otel-collector&quot;,route=&quot;inventory_route&quot;,service=&quot;inventory_service&quot;,source=&quot;service&quot;,workspace=&quot;default&quot;} 1<br>kong_http_requests_total{code=&quot;200&quot;,instance=&quot;192.168.76.233:8100&quot;,job=&quot;otel-collector&quot;,route=&quot;pricing_route&quot;,service=&quot;pricing_service&quot;,source=&quot;service&quot;,workspace=&quot;default&quot;} 1</pre><h3>Check Metrics and Logs in Dynatrace</h3><p>One of the main values provided by Dynatrace is Dashboard creation capabilities. You can create them visually and using <a href="https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-query-language">DQL (Dynatrace Query Language)</a>. As an example, Dynatrace provides a Kong Dashboard where we can manage the main metrics and the access log.</p><p>The Kong Dashboard should look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I_1sQBxwIRT1y-Dl" /></figure><h3>Connecting Log Data to Traces</h3><p>Dynatrace has the ability to connect <a href="https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-log-enrichment#connecting-log-data-to-traces">Log Events to Traces</a>. That allows us to navigate to the trace associated with a given log event.</p><p>In order to do it, the log event has to have a “trace_id” field with the actual trace id it is related to. By default, the OpenTelemetry Plugin injects such a field. However, it adds the format used, in order case “w3c”. For example:</p><pre>{<br>  &quot;trace_id&quot;:{<br>    &quot;w3c&quot;:&quot;3b7fb854f5442239c0e94edc69fd6886&quot;<br>  },<br>  &quot;route&quot;:{<br>    &quot;paths&quot;:[<br>      &quot;/coupon&quot;<br>    ],<br>    &quot;created_at&quot;:1738765016,<br>….<br>}</pre><p>As you can see <a href="https://docs.konghq.com/gateway/latest/plugin-development/custom-logic/#plugins-execution-order">here</a>, the TCP Log gets executed after the OpenTelemetry Plugin. So, to solve that, the TCP Log Plugin configuration has the “custom_fields_by_lua” set with a Lua code which removes the “w3c” part out of the field added by the OpenTelemetry Plugin. The new log event can then follow the format Dynatrace looks for:</p><pre>{<br>  &quot;trace_id&quot;:&quot;3b7fb854f5442239c0e94edc69fd6886&quot;,<br>  &quot;route&quot;:{<br>    &quot;paths&quot;:[<br>      &quot;/inventory&quot;<br>  ],<br>  &quot;created_at&quot;:1738765016,<br>….<br>}</pre><p>Here’s a Dynatrace Logs app with events generated by the TCP Log Plugin. Choose an event and you’ll see the right panel with the “Open trace” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V5Z2H0gZrZnZ9_-l" /></figure><p>If you click on it, you can choose to get redirected to the Dynatrace Trace apps. In the “Distributed Tracing” app you should see the trace with all spans related to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QtMNRCCzIW9Qjyd7" /></figure><h3>Conclusion</h3><p>The synergy of Dynatrace and Kong Konnect technologies provides a new era of observability architectures built on OpenTelemetry standards. By leveraging the combined capabilities of these technologies, organizations can strengthen their infrastructure with robust policies, laying a solid foundation for advanced observability platforms.</p><p>Try <a href="https://konghq.com/products/kong-konnect/register">Kong Konnect</a> and <a href="https://www.dynatrace.com/signup/">Dynatrace</a> for free today! Kong Konnect simplifies API management and improves security for all services infrastructure as Dynatrace provides end-to-end visibility and automated insights to optimize the performance, security, and user experiences of applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2cb86862ba2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Application with Kong AI Gateway, Amazon Bedrock, Redis and LangChain on Amazon EKS 1.31]]></title>
            <link>https://medium.com/@claudioacquaviva/rag-application-with-kong-ai-gateway-amazon-bedrock-redis-and-langchain-on-amazon-eks-1-31-580a244b7ee4?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/580a244b7ee4</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Thu, 17 Oct 2024 19:45:37 GMT</pubDate>
            <atom:updated>2025-02-27T13:33:11.159Z</atom:updated>
            <content:encoded><![CDATA[<p>For the last couple of years, Retrieval-Augmented Generation (RAG) architectures have become a rising trend for AI-based applications. Generally speaking, RAG offers a solution to some of the limitations in traditional generative AI models, such as accuracy and hallucinations, allowing companies to create more contextually relevant AI applications. The combination of retrieval-based systems like RAG and generative models such as Large Language Models (LLMs) will likely become more prevalent as companies seek to leverage their data for their new AI applications.</p><p>This blog post describes a RAG implementation using LangChain as the application orchestrator, driving Amazon Bedrock, protected by Kong AI Gateway, and Redis as the vector database.</p><h3>Kong AI Gateway Introduction</h3><p>Last February, Kong announced Kong Gateway 3.6 with the <a href="https://konghq.com/blog/product-releases/announcing-kong-ai-gateway">first version</a> of Kong AI Gateway with brand new capabilities to address AI-based use cases like multi-LLM (Large Language Models) integrations, Prompt Engineering, including templating, decoration and prompt guard, Request and Response Transformation based on LLM responses, AI analytics, and more. The next <a href="https://konghq.com/blog/product-releases/ai-gateway-goes-ga">Kong Gateway 3.7 version</a>, launched last May, included new features, such as support for OpenAI SDK, Rate Limiting policies based on tokens, streaming support, etc.</p><p>Now, with the latest <a href="https://konghq.com/blog/product-releases/ai-gateway-3-8">Kong Gateway 3.8 release</a>, announced during our <a href="https://konghq.com/events/conferences/api-summit">API Summit 2024</a>, new critical capabilities have been released:</p><ul><li>Semantic caching and prompt guard.</li><li>Six different Load Balancing algorithms, including semantic routing, to support multiple LLMs.</li><li>Official support to <a href="https://aws.amazon.com/bedrock">Amazon Bedrock</a> and GCP Vertex/Gemini (in addition to all the others).</li></ul><p>The Kong AI Gateway provides AI infrastructure for every team building new GenAI applications, and it combines the Kong API Gateway features with a set of AI-focused plugins that allow us to ensure data security and privacy, response quality and accuracy, and finally data and infra readiness for AI, while also providing out of the box observability for our LLM traffic and the ability to manage access tiers to the LLMs. This approach allows applications to adopt LLM infrastructures within their existing API traffic flow without having to rebuild common requirements that all AI applications need when they go into production.</p><p>The diagram below represents the architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PQOwnx74jIseyAya" /></figure><p><em>The Kong AI Gateway sits in between the GenAI applications we build and the LLMs we consume.</em></p><h3>RAG Applications and LLM Frameworks</h3><p>Concurrently, customers started to leverage their AI infrastructures to build LLM-based applications. One of the main processes used for such applications is Retrieval-Augmented Generation, or RAG, for short.</p><p>Fundamentally, LLMs are trained, for a long period of time, on massive public datasets. Although that turns into a powerful solution, it lacks context. That’s where RAG comes in: <a href="https://arxiv.org/abs/2005.11401">as described in its original paper</a>, it enhances the capabilities of LLMs by incorporating internal and domain-specific customer data optimizing the LLM output with a knowledge base outside the data sources used for the LLM training. That assimilation is solved during query time, meaning it’s done without retraining the model.</p><p>The picture describes the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*w17V_0RYX3txQnpt" /></figure><p>Basically, the RAG application comprises two main processes:</p><ul><li>Data Preparation: External data sources, including all sorts of documents, are chunked and submitted to an Embedding Model which converts them into embeddings. The embeddings are stored in a Vector Database.</li><li>Query time: A consumer wants to send a query to the actual Prompt/Chat LLM model. However, the query should be enhanced with relevant data, taken from the Vector Database, and not available in the LLM model. The following steps are then performed:</li></ul><ol><li>The consumer builds a Prompt.</li><li>The RAG application converts the Prompt into an embedding, calling the Embedding Model.</li><li>Leveraging Semantic Search, RAG matches the Prompt Embedding with the most relevant information and retrieves that Vector Database.</li><li>The Vector Database returns relevant data as a response to the search query.</li><li>The RAG application sends a query to the Prompt/Chat LLM Model combining the Prompt with the Relevant Data returned by the Vector Database.</li><li>The LLM Model returns a response.</li></ol><p>As you can see, there are multiple components in a typical RAG application implementation. You might see a need for a new one responsible for orchestrating the components. That’s the main responsibility of emerging LLM frameworks like <a href="https://www.langchain.com/langchain">LangChain</a>.</p><p>AWS provides nice introductions about <a href="https://aws.amazon.com/what-is/retrieval-augmented-generation/">RAG</a> and <a href="https://aws.amazon.com/what-is/langchain/">LangChain</a>.</p><h3>Kong AI Gateway and Amazon Bedrock</h3><p>We should get started discussing the benefits Kong AI Gateway brings to Amazon Bedrock for an LLM-based application. As we stated before, Kong AI Gateway leverages the existing Kong API Gateway extensibility model to provide specific AI-based plugins. Here are some values provided by Kong AI Gateway and its plugins:</p><ul><li>AI Proxy and AI Proxy Advanced plugins: the Multi-LLM capability allows the AI Gateway to abstract Amazon Bedrock (and other LLMs as well) load balancing models based on several policies including latency time, model usage, semantics etc. These plugins extract LLM observability metrics (like number of requests, latencies, and errors for each LLM provider) and the number of incoming prompt tokens and outgoing response tokens. All this is in addition to hundreds of metrics already provided by Kong Gateway on the underlying API requests and responses. Finally, Kong AI Gateway leverages the observability capabilities provided by Konnect to track Amazon Bedrock usage out of the box, as well as generate reports based on monitoring data.</li><li>Prompt Engineering:</li><li>AI Prompt Template plugin, responsible for pre-configuring AI prompts to users</li><li>AI Prompt Decorator plugin, which injects messages at the start or end of a caller’s chat history.</li><li>AI Prompt Guard plugin lets you configure a series of PCRE-compatible regular expressions to allow and block specific prompts, words, phrases, or otherwise and have more control over a LLM service, controlled by Amazon Bedrock.</li><li>AI Semantic Prompt Guard plugin to self-configurable semantic (or pattern-matching) prompt protection.</li><li>AI Semantic Cache plugin caches responses based on threshold, to improve performance (and therefore end-user experience) and cost.</li><li>AI Rate Limiting Advanced, you can tailor per-user or per-model policies based on the tokens returned by the LLM provider under Amazon Bedrock management or craft a custom function to count the tokens for requests.</li><li>AI Request Transformer and AI Response Transformer plugins seamlessly integrate with the LLM on Amazon Bedrock, enabling introspection and transformation of the request’s body before proxying it to the Upstream Service and prior to forwarding the response to the client.</li></ul><p>Besides, the Kong AI Gateway use cases can combine policies implemented by hundreds of Kong Gateway plugins, such as:</p><ul><li>Authentication and Authorization: OIDC, mTLS, API Key, LDAP, SAML, Open Policy Agent (OPA)</li><li>Traffic Control: Request Validator and Size Limiting, WebSocket support, Route by Header, etc.</li><li>Observability: OpenTelemetry (OTel), Prometheus, Zipkin, etc.</li></ul><p>Also, from the architecture perspective, in a nutshell, the Konnect Control Plane and Data Plane nodes topology remains the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ndC7eYv9TspVP0CN" /></figure><p><em>By leveraging the same underlying core of Kong Gateway, we’re reducing complexity in deploying the AI Gateway capabilities as well. And of course, it works on Konnect, Kubernetes, self-hosted, or across multiple clouds.</em></p><h3>Amazon Bedrock Foundation Models</h3><p>This blog post consumes some <a href="https://aws.amazon.com/what-is/foundation-models/">Foundation Models</a>, including Prompt/Chat and Embeddings Models. Please, make sure you have requested access to the following models:</p><ul><li>meta.llama3–70b-instruct-v1:0</li><li>amazon.titan-embed-text-v1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6bbxxCssNhLfsKQ4" /></figure><h3>Implementation Architecture</h3><p>The RAG implementation architecture should include components representing and responsible for the functional scope described above. The architecture comprises:</p><ul><li>Redis as the Vector Database</li><li>Amazon Bedrock supporting both Prompt/Chat and Embeddings Models</li><li>Kong AI Gateway to abstract and protect Amazon Bedrock as well as implement LLM-based use cases.</li></ul><p>Kong AI Gateway, implemented as a regular Konnect Data Plane Node, and Redis run on an Amazon EKS 3.1 Cluster.</p><p>The architecture can be analyzed from both perspectives: Data Preparation and actual Query time. Let’s take a look at the Data Preparation time.</p><h3>Data Preparation time</h3><p>The Data Preparation time is quite simple: the Document Loader component sends data chunks, based on the data taken, and calls Amazon Bedrock asking the Embedding Model to convert them into embeddings. The embeddings are then stored in the Redis Vector Database. Here’s a diagram describing the flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*PCurF5rrtSQqjzgq" /></figure><p>For the purpose of this blog post, we’re going to use the “amazon.titan-embed-text-v1” Embedding Model.</p><h3>Query time</h3><p>Logically speaking we can break the Query time into some main steps:</p><ol><li>The AI Application builds a Prompt and, using the same “amazon.titan-embed-text-v1” Embedding Model, generates embeddings to it.</li><li>The AI Application takes the embeddings and semantically searches the Vector Database. The database returns relevant data related to the context.</li><li>The AI Application sends the prompt, augmented with the retrieved data received from the Vector Database, to Kong AI Gateway.</li><li>Kong AI Gateway, using the AI Proxy (or AI Proxy Advanced) plugin, proxies the request to Amazon Bedrock and returns the response to the AI Application. Kong AI Gateway enforces eventual policies previously enabled.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*6YG9XzEqUEWDSE5i" /></figure><h3>Kong AI Gateway and Amazon Bedrock Integration and APIs</h3><p>Let’s take a look at the specific integration point between Kong AI Gateway and Amazon Bedrock. To get a better understanding, here’s the architecture cut isolating both:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*bCqLmyTzyYYwA6pT" /></figure><p>The consumer can be any RESTful-based component, in our case will be a LangChain application.</p><p>As you can see, there are two important topics here:</p><ul><li>OpenAI API specification</li><li>Amazon Bedrock Converse API and EKS Pod Identity</li></ul><p>Let’s discuss each one of them.</p><h3>OpenAI API support</h3><p>Kong AI Gateway supports the <a href="https://platform.openai.com/docs/overview">OpenAI API specification</a>. That means the consumer can send standard OpenAI requests to the Kong AI Gateway. As a basic example, consider this OpenAI request:</p><pre>curl https://api.openai.com/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer $OPENAI_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;gpt-4o&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><p>When we add Kong AI Gateway, sitting in front of Amazon Bedrock, we’re not just exposing it but also allowing the consumers to use the same mechanism — in this case, OpenAI APIs — to consume it. That leads to a very flexible and powerful capability when we come to development processes. In other words, Kong AI Gateway normalizes the consumption of any LLM infrastructure, including Amazon Bedrock, Mistral, OpenAI, Cohere, etc.</p><p>As an exercise, the new request should be something like this. The request has some minor differences:</p><ul><li>It sends a request to the Kong API Gateway Data Plane Node.</li><li>It replaces the OpenAI endpoint with a Kong API Gateway route.</li><li>The API Key is actually managed by the Kong API Gateway now.</li><li>We’re using an Amazon Bedrock Model, meta.llama3–70b-instruct-v1:0.</li></ul><pre>curl http://$DATA_PLANE_LB/bedrock-route \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;apikey: $KONG_API_KEY&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;meta.llama3–70b-instruct-v1:0&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Hello!&quot;<br>      }<br>    ]<br>  }&#39;</pre><h3>Amazon EKS Pod Identity</h3><p>The Konnect Data Plane Node, where the Kong AI Gateway runs, has to send requests to Amazon Bedrock on behalf of the Gateway consumer. In order to do it, we need to grant permissions to the Data Plane deployment to access the Amazon Bedrock API, more precisely the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html">Converse API</a>, used by the AI Gateway to interact with it.</p><p>For example, here’s a request to Amazon Bedrock, using the AWS CLI. Use “aws configure” first to set your Access and Secret Key as well as the AWS region you want to use then run the command.</p><pre>aws bedrock-runtime converse \<br>  --model-id amazon.titan-text-express-v1 \<br>  --messages &#39;[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:[{&quot;text&quot;:&quot;Tell me about John Coltrane&quot;}]}]&#39;</pre><p>It’s reasonable to consume Bedrock service with local CLI commands. However, for Amazon EKS deployments, the recommended approach is <a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html">EKS Pod Identity</a>, instead of simple long-term credentials like AWS Access and Secret Keys. In a nutshell, EKS Pod Identity allows the Data Plane Pod’s container to use the AWS SDK and send API requests to AWS services using AWS Identity and Access Management (IAM) permissions.</p><p>Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances.</p><p>As another best practice, we recommend the Private Key and Digital Certificate pair used by the Konnect Control Plane and the Data Plane connectivity to be stored in AWS Secrets Manager. In this sense, the Data Plane deployment refers to the secrets to get installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*gAfWwchNNK64Uq9V" /></figure><h3>Amazon Elastic Kubernetes Service (EKS) installation and preparation</h3><p>Now, we’re ready to get started with our Kong AI Gateway deployment. As the installation architecture defines, it will be running on an EKS Cluster.</p><h3>Amazon EKS Cluster creation</h3><p>In order to create the EKS Cluster, you can use <a href="https://eksctl.io/">eksctl</a>, the official CLI for Amazon EKS, like this:</p><pre>eksctl create cluster --name konnectaigateway-eks131 \<br>  --version 1.31 \<br>  --region us-east-2 \<br>  --nodegroup-name kong-node \<br>  --node-type g6.xlarge \<br>  --nodes 1</pre><p>Note command creates an EKS Cluster, version 1.31, with a single node based on the g6.xlarge instance type, powered by NVIDIA GPUs. That is particularly interesting if you’re planning to deploy and run LLMs locally in the EKS Cluster.</p><h3>Cluster preparation</h3><p>After the installation, we should prepare the Cluster to receive the other components:</p><ul><li><a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/">AWS Load Balancer Controller</a> to expose both Kong AI Gateway with public <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load Balancers</a> (NLB).</li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html">EKS Pod Identity Agent</a> to be able to define Pod Identity Association and grant permissions to the Kong AI Gateway Data Plane Pod to access both Amazon Bedrock and AWS Secrets Manager.</li></ul><p>Please, refer to the official documentation to learn more about the components and their installation processes.</p><h3>Redis installation</h3><p>As defined in the implementation architecture, Redis plays the Vector Database role. It’s going to be deployed and running in a specific EKS namespace and exposed with an NLB.</p><p>For its deployment, we’re going to be using its <a href="https://github.com/redis-stack">Helm Charts</a>. Get started adding its repo:</p><pre>helm repo add redis-stack https://redis-stack.github.io/helm-redis-stack<br>help repo update<br>helm install redis-stack redis-stack/redis-stack -n redis --create-namespace</pre><p>Expose the Redis Service with an ELB:</p><pre>kubectl patch service redis-stack -n redis -p &quot;{\&quot;spec\&quot;: { \&quot;type\&quot; : \&quot;LoadBalancer\&quot;}}&quot;</pre><p>Get the ELB hostname with:</p><pre>% kubectl get service redis-stack -n redis -o json | jq -r &#39;.status.loadBalancer.ingress[].hostname&#39;<br>a7c63f0c9a740483987398a6c0b6e444–967754848.us-east-2.elb.amazonaws.com</pre><p>You can check Redis Server with a simple command:</p><pre>% kubectl exec $(kubectl get pod -n redis -o json | jq -r &#39;.items[].metadata.name&#39;) -n redis - redis-server - version<br>Redis server v=7.4.1 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=5918102922f65a0d</pre><p>Redis’ dashboard is also available:</p><pre>http://a7c63f0c9a740483987398a6c0b6e444-967754848.us-east-2.elb.amazonaws.com:8001</pre><h3>Kong Data Plane installation</h3><p>The second component to be installed is the Kong AI Gateway. The process can be divided into two steps:</p><ul><li>Pod Identity configuration</li><li>Kong Data Plane deployment</li></ul><h3>Pod Identity configuration</h3><p>In this first step, we configure EKS Pod Identity describing which AWS Services the Data Plane Pods should be allowed to access. In our case, we need to consume Amazon Bedrock and AWS Secrets Manager.</p><h4>IAM Policy</h4><p>Pod Identity relies on IAM policies to check which AWS Services can be consumed. Our policy should allow access to AWS Bedrock actions so the Data Plane will be able to send requests to Bedrock APIs, more precisely, <a href="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html">Converse</a> and <a href="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html">ConverseStream</a> APIs. The Converse API requires permission to the InvokeModel action as ConverserStream needs access to InvokeModelWithResponseStream.</p><p>Also, we’re going to use AWS Secrets Manager to store our Private Key and Digital Certificate pair the Konnect Control Plane and Data Plane used to communicate.</p><p>Considering all this, let’s create the IAM policy with the following request:</p><pre>aws iam create-policy \<br>--policy-name bedrock-policy \<br>--policy-document &#39;{<br>  &quot;Version&quot;: &quot;2012–10–17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;bedrock:InvokeModel&quot;,<br>        &quot;bedrock:InvokeModelWithResponseStream&quot;,<br>        &quot;secretsmanager:ListSecrets&quot;,<br>        &quot;secretsmanager:GetSecretValue&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;<br>    }<br>  ]<br>}&#39;</pre><h4>Pod Identity Association</h4><p>Pod Identity takes a Kubernetes Service Account to manage the permissions. So create the Kubernetes namespace for the Kong Data Plane deployment and a simple Service Account inside of it.</p><pre>kubectl create namespace kong<br>kubectl create sa kaigateway-podid-sa -n kong</pre><p>Now we’re ready to create the Pod Identity Association. We use the same eksctl command to do it:</p><pre>eksctl create podidentityassociation \<br>--cluster konnectaigateway-eks131 \<br>--region us-east-2 \<br>--namespace kong \<br>--service-account-name kaigateway-podid-sa \<br>--role-name kaigateway-podid-role \<br>--permission-policy-arns arn:aws:iam::&lt;your_aws_account&gt;:policy/bedrock-policy</pre><p>The command above is responsible for:</p><ul><li>IAM Role creation based on the IAM Policy we previously defined</li><li>Associating the IAM Role to the existing Kubernetes Service Account</li></ul><p>You can check the Pod Identity Association with:</p><pre>eksctl get podidentityassociation \<br>--cluster konnectaigateway-eks131 \<br>--region us-east-2 \<br>--namespace kong</pre><p>Check the IAM Role and Policies attached with:</p><pre>aws iam get-role --role-name kaigateway-podid-role<br>aws iam list-attached-role-policies --role-name kaigateway-podid-role</pre><h3>Kong Data Plane deployment</h3><p>The Data Plane deployment comprises the following steps:</p><ul><li>Konnect subscription and Control Plane creation</li><li>Create a Konnect Vault for AWS Secrets Manager</li><li>Private Key and Digital Certification pair creation</li><li>Store the pair in AWS Secrets Manager</li><li>Data Plane deployment</li></ul><h4>Konnect subscription and Control Plane creation</h4><p>This fundamental step is required to get access to Konnect. Click on the <a href="https://konghq.com/products/kong-konnect/register">Registration</a> link and present your credentials. Or, if you already have a Konnect subscription, <a href="https://cloud.konghq.com/us">log in to it</a>.</p><p>Any Konnect subscription has a “default” Control Plane defined. You can proceed using it or optionally create a new one. The following instructions are based on a new Control Plane. You can create a new one by clicking on the “Gateway Manager” menu option. Then click on “New Control Plane” and choose the Kong Gateway option. Inside the “Kong Gateway” page type “AI Gateway” for the name and leave the “Self-Managed Hybrid Instances” option set.</p><h4>Create a Konnect Vault for AWS Secrets Manager</h4><p>Now, inside the new “AI Gateway” Control Plane, click on the “Vaults” menu option and create a Vault referring to AWS Secret Manager where the Private Key and Digital Certificate pair is going to be stored. Inside the “New Vault” page, define “Ohio (us-east-2)” for Region and “aws-secrets” for Prefix. Click on “Save.”</p><h4>Private Key and Digital Certificate pair creation</h4><p>Still inside the “AI Gateway” Control Plane, click on “Data Plane Nodes” menu option and “New Data Plane Node” button.</p><p>Choose Kubernetes as your platform. Click on “Generate certificate”, copy and save the Digital Certificate and Private Key as tls.crt and tls.key files as described in the instructions. Also copy and save the configuration parameters in a values.yaml file.</p><h4>Store the Private Key and Digital pair in AWS Secrets Manager</h4><p>Store the pair in AWS Secret Manager with the following commands:</p><pre>aws secretsmanager create-secret --name kongcp1-crt --region us-east-2 --secret-string &quot;{\&quot;cert\&quot;: \&quot;$(cat ./tls.crt)\&quot;}&quot;<br>aws secretsmanager create-secret --name kongcp1-key --region us-east-2 --secret-string &quot;{\&quot;key\&quot;: \&quot;$(cat ./tls.key)\&quot;}&quot;</pre><h4>Data Plane deployment</h4><p>Finally, the last step will deploy the Data Plane. Take the values.yaml file and change it adding the AWS Secret Manager references and the Kubernetes Service Account used by the Pod Identity Association. The updated version should be something like this. Replace the cluster_* endpoints and server names with yours:</p><pre>image:<br>  repository: kong/kong-gateway<br>  tag: &quot;3.8&quot;<br><br>admin:<br>  enabled: false<br><br>manager:<br>  enabled: false<br><br>env:<br>  role: data_plane<br>  database: &quot;off&quot;<br>  cluster_mtls: pki<br>  cluster_control_plane: abc71c0977.us.cp0.konghq.com:443<br>  cluster_server_name: abc71c0977.us.cp0.konghq.com<br>  cluster_telemetry_endpoint: abc71c0977.us.tp0.konghq.com:443<br>  cluster_telemetry_server_name: abc71c0977.us.tp0.konghq.com<br>  cluster_cert: &quot;{vault://aws/kongcp1-crt/cert}&quot;<br>  cluster_cert_key: &quot;{vault://aws/kongcp1-key/key}&quot;<br>  lua_ssl_trusted_certificate: system<br>  konnect_mode: &quot;on&quot;<br>  vitals: &quot;off&quot;<br>  nginx_worker_processes: &quot;1&quot;<br>  upstream_keepalive_max_requests: &quot;100000&quot;<br>  nginx_http_keepalive_requests: &quot;100000&quot;<br>  proxy_access_log: &quot;off&quot;<br>  dns_stale_ttl: &quot;3600&quot;<br><br>ingressController:<br>  enabled: false<br>  installCRDs: false<br><br>resources:<br>  requests:<br>    cpu: 1<br>    memory: &quot;2Gi&quot;<br><br>proxy:<br>  annotations:<br>    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing<br><br>deployment:<br>  serviceAccount:<br>    create: false<br>    name: kaigateway-podid-sa</pre><p>Basically, the changes are:</p><ul><li>Since the Private Key and Digital Certificate pair has been stored in AWS Secrets, remove the “secretVolumes” section.</li><li>Replace the “cluster_cert” and “cluster_cert_key” fields with Konnect Vault references and AWS Secret Manager secrets.</li><li>Add the proxy annotation to request a public NLB for the Data Plane.</li><li>Add the deployment section referring to the Kubernetes Service Account that has been used to create the Pod Identity Association.</li></ul><p>Use the Helm command to deploy the Data Plane:</p><pre>helm repo add kong https://charts.konghq.com<br>helm repo update<br>helm install kong kong/kong -n kong --values ./values.yaml</pre><h3>Checking the Data Plane</h3><p>Use the Load Balancer created during the deployment:</p><pre>DATA_PLANE_LB=$(kubectl get service -n kong kong-kong-proxy --output=jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;)</pre><p>You should get a response like this:</p><pre>% curl -i $DATA_PLANE_LB<br>HTTP/1.1 404 Not Found<br>Date: Thu, 03 Oct 2024 12:36:53 GMT<br>Content-Type: application/json; charset=utf-8<br>Connection: keep-alive<br>Content-Length: 103<br>X-Kong-Response-Latency: 0<br>Server: kong/3.8.0.0-enterprise-edition<br>X-Kong-Request-Id: c655f65a2d1fef7c7ed0550e3f7dc553<br><br>{<br>  &quot;message&quot;:&quot;no Route matched with those values&quot;,<br>  &quot;request_id&quot;:&quot;c655f65a2d1fef7c7ed0550e3f7dc553&quot;<br>}</pre><p>Now we can define the Kong Objects necessary to expose and control Bedrock, including Kong Gateway Service, Routes, and Plugins.</p><h3>decK</h3><p>With <a href="https://docs.konghq.com/deck/latest/">decK</a> (declarations for Kong) you can manage Kong Konnect configuration and create Kong Objects in a declarative way. decK state files describe the configuration of Kong API Gateway. State files encapsulate the complete configuration of Kong in a declarative format, including services, routes, plugins, consumers, and other entities that define how requests are processed and routed through Kong. Please check the decK documentation to learn how to <a href="https://docs.konghq.com/deck/latest/installation/">install</a> it.</p><h3>Konnect PAT</h3><p>Once you have decK installed, you should ping Konnect to check if the connection is up. In order to do it you need a <a href="https://docs.konghq.com/konnect/gateway-manager/declarative-config/#generate-a-personal-access-token">Konnect Personal Access Token (PAT)</a>. To generate your PAT, go to Konnect UI, click on your initials in the upper right corner of the Konnect home page, then select “Personal Access Tokens.” Click on “+ Generate Token,” name your PAT, set its expiration time, and be sure to copy and save it as an environment variable also named as PAT. Konnect won’t display your PAT again.</p><p>You can ping Konnect with:</p><pre>% deck gateway ping --konnect-token $PAT<br>Successfully Konnected to the Kong organization!</pre><h3>Kong declarations for the RAG Application</h3><p>Now, create a file named “kong_bedrock.yaml” with the following content:</p><pre>cat &gt; kong_bedrock.yaml &lt;&lt; &#39;EOF&#39;<br>_format_version: &quot;3.0&quot;<br>_info:<br>  select_tags:<br>  - rag-bedrock<br>_konnect:<br>  control_plane_name: AI Gateway<br>services:<br>- name: bedrock-rag-service<br>  host: httpbin.default<br>  port: 8000<br>  routes:<br>  - name: bedrock-rag-route1<br>    paths:<br>    - /bedrock-route<br>    plugins:<br>    - name: ai-proxy<br>      instance_name: &quot;ai-proxy-bedrock-rag&quot;<br>      enabled: true<br>      config:<br>        auth:<br>          param_name: &quot;allow_override&quot;<br>          param_value: &quot;false&quot;<br>          param_location: &quot;body&quot;<br>        route_type: &quot;llm/v1/chat&quot;<br>        model:<br>          provider: &quot;bedrock&quot;<br>          options:<br>            bedrock:<br>              aws_region: &quot;us-east-1&quot;<br>    - name: ai-prompt-decorator<br>      instance_name: ai-prompt-decorator-rag<br>      enabled: false<br>      config:<br>        prompts:<br>          prepend:<br>          - role: system<br>            content: &quot;You will always respond in the Portuguese (Brazil) language.&quot;<br>    - name: key-auth<br>      instance_name: key-auth-bedrock<br>      enabled: true<br>consumers:<br>- keyauth_credentials:<br>  - key: &quot;123456&quot;<br>  username: user1<br>  plugins:<br>  - name: ai-rate-limiting-advanced<br>    instance_name: ai-rate-limiting-advanced-consumer<br>    enabled: false<br>    config:<br>      llm_providers:<br>      - name: bedrock<br>        window_size: 60<br>        limit: 200<br>EOF</pre><p>The declaration defines multiple Kong Objects:</p><ul><li><a href="https://docs.konghq.com/gateway/latest/get-started/services-and-routes/">Kong Gateway Service</a> named “bedrock-rag-service”. The service doesn’t need to map to any real upstream URL. In fact, it can point somewhere empty, for example, <a href="http://localhost:32000.">http://localhost:32000.</a> This is because the AI Proxy plugin, also configured in the declaration, overwrites the upstream URL. This requirement will be removed in a later Kong revision.</li><li>Kong Route: the Gateway Service has a route defined with the “/bedrock-route” path. That’s the route we’re going to consume to reach out to Bedrock.</li><li>Kong Route Plugins: the Kong Route has some plugins configured. Note that only the AI Proxy and Key Auth Plugins are enabled. The other ones are configured but disabled.</li><li>Kong <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy/">AI Proxy Plugin</a>: that’s the Plugin that allows us to connect to the LLM infrastructure. For <a href="https://docs.konghq.com/hub/kong-inc/ai-proxy/how-to/machine-learning-platform-integration-guides/bedrock/">Bedrock</a>, among other things, we need to configure which AWS region we should connect to.</li><li>Kong <a href="https://docs.konghq.com/hub/kong-inc/ai-prompt-decorator/">AI Prompt Decorator Plugin</a>: the Plugin instructs Bedrock to manage the prompt and response in a particular way.</li><li>Kong <a href="https://docs.konghq.com/hub/kong-inc/key-auth/">Key Auth Plugin</a>: the Plugin requires the consumer to inject an API Key, named by default “apikey”, to the requests in order to consume the Kong Route.</li><li>Kong Consumer: the declaration also defines a Kong Consumer, named “user1” with an API Key based credential. The Kong Consumer will allow the Kong AI Gateway to map the incoming requests with the API Key “apikey” (required by the Key Auth Plugin) with the credential “123456” to the Kong Consumer “user1”. Once the Kong Consumer is mapped, the Kong AI Gateway can apply policies, defined with other Plugins, to this specific Kong Consumer.</li><li>Kong Consumer Plugins: the Kong Consumer “user1” has the <a href="https://docs.konghq.com/hub/kong-inc/ai-rate-limiting-advanced/">AI Rate Limiting Advanced Plugin</a> enabled. This Plugin, an extension of the standard Kong API Gateway <a href="https://docs.konghq.com/hub/kong-inc/rate-limiting-advanced/">Rate Limiting Advanced Plugin</a>, implements rate limiting policy based on the number of tokens returned by the LLM provider.</li></ul><p>The declaration has been tagged as “rag-bedrock” so you can manage its objects without impacting any other ones you might have created previously. Also, note the declaration is saying it should be applied to the “AI Gateway” Konnect Control Plane.</p><p>You can submit the declaration with the following decK command:</p><pre>$ deck gateway sync --konnect-token $PAT kong_bedrock.yaml<br>creating consumer user1<br>creating service bedrock-rag-service<br>creating key-auth 123456 for consumer user1<br>creating route bedrock-rag-route1<br>creating plugin key-auth for route bedrock-rag-route1<br>creating plugin ai-prompt-decorator for route bedrock-rag-route1<br>creating plugin ai-proxy for route bedrock-rag-route1<br>creating plugin ai-rate-limiting-advanced for consumer user1<br>Summary:<br>  Created: 8<br>  Updated: 0<br>  Deleted: 0</pre><h3>Consume the Kong AI Gateway Route</h3><p>You can now send a request consuming the Kong Route. Note we have injected the API Key with the expected name, “apikey”, and with a value that corresponds to the Kong Consumer, “123456”. If you change the value or the API Key name, or even not add it, you get a 401 (Unauthorized) error.</p><pre>curl -s -X POST http://$DATA_PLANE_LB/bedrock-route \<br>-H &quot;Content-Type: application/json&quot; \<br>-H &quot;apikey: 123456&quot; \<br>-d &#39;{<br>  &quot;messages&quot;: [<br>    {<br>      &quot;role&quot;: &quot;user&quot;,<br>      &quot;content&quot;: &quot;What is niilism?&quot;<br>    }<br>  ],<br>  &quot;model&quot;: &quot;meta.llama3–70b-instruct-v1:0&quot;<br>}&#39; | jq</pre><p>To enable the other Plugins you can change the declaration as “enabled: true” or use the Konnect UI.</p><h3>LangChain</h3><p>With all the components we need for our RAG application in place, it’s time for the most exciting part of this blog post: the RAG application.</p><p>The minimalist app we’re going to present focuses on the main steps of the RAG processes. It’s not intended to be used other than learning processes and maybe lab environments.</p><p>Basically, the application comprises two Python scripts, using <a href="https://www.langchain.com/langchain">LangChain</a>, one of the main frameworks available today to write AI applications including LLMs and RAG. The scripts are the following:</p><ul><li><strong>rag_redis_bedrock_langchain_embeddings.py</strong>: it implements the Data Preparation process with Redis and Bedrock’s “amazon.titan-embed-text-v1” Embedding Model.</li><li><strong>rag_redis_bedrock_langchain_kong_ai_gateway.py</strong>: responsible for the Query time implementation combining Kong AI Gateway, Redis, and Bedrock.</li></ul><p>From the RAG perspective, the application is based on an interview Marco Palladino, Kong’s co-founder and CTO, gave discussing Kong AI Gateway and our current technology environment.</p><h3>Data Preparation</h3><p>The first thing the code does is to download the transcript of the interview in a local file. Then it uses the LangChain Community’s <a href="https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.directory.DirectoryLoader.html">DirectoryLoader</a> and <a href="https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.text.TextLoader.html">TextLoader</a> as well as <a href="https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.CharacterTextSplitter.html">CharacterTextSplitter</a> Python packages to load and break the original text into chunks.</p><p>Next, the code uses the <a href="https://python.langchain.com/api_reference/community/embeddings/langchain_community.embeddings.bedrock.BedrockEmbeddings.html">BedrockEmbeddings</a> and <a href="https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.Redis.html">Redis</a> packages, also provided by LangChain, to generate the embeddings for each chunk and store all of them in Redis Vector Database’s collection named “docs-bedrock”. Note we hit Redis through the ELB we provisioned during its deployment. Note also the script forces the index recreation.</p><p>Before running the code make sure you have a Python3 environment set and the following packages installed:</p><ul><li>requests</li><li>langchain</li><li>langchain-community</li><li>langchain-aws</li><li>langchain-redis</li></ul><p>It’s important to note that this Python script assumes you have your AWS credentials set locally in order to connect to Bedrock. Please refer to the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html">Boto3 documentation</a> to learn how to manage them.</p><p>You can run the script with:</p><pre>python3 rag_redis_bedrock_langchain_embeddings.py</pre><p>After the execution, you can check the Redis database using its dashboard or with the following command. You should see 52 points keys:</p><pre>% kubectl exec $(kubectl get pod -n redis -o json | jq -r &#39;.items[].metadata.name&#39;) -n redis -- redis-cli info keyspace<br># Keyspace<br>db0:keys=52,expires=0,avg_ttl=0,subexpiry=0</pre><h4>Python script code</h4><p>Here’s the code of the first Python script:</p><pre>import requests<br><br>from langchain_community.vectorstores.redis import Redis<br>from langchain_aws import BedrockEmbeddings<br>from langchain.text_splitter import CharacterTextSplitter<br>from langchain_community.document_loaders import TextLoader<br>from langchain_community.document_loaders import DirectoryLoader<br><br># Download transcript text and split it into chunks<br># https://softwareengineeringdaily.com/2024/06/20/its-apis-all-the-way-down-with-marco-palladino<br>url = &quot;http://softwareengineeringdaily.com/wp-content/uploads/2024/06/SED1683-Kong.txt&quot;<br>headers = {&#39;User-Agent&#39;: &#39;LangChain&#39;}<br>res = requests.get(url, headers=headers)<br>with open(&quot;ragdoc.txt&quot;, &quot;w&quot;) as f:<br>f.write(res.text)<br><br>loader = DirectoryLoader(&quot;.&quot;, &quot;ragdoc.txt&quot;, loader_cls=TextLoader)<br>documents = loader.load()<br>text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=50)<br>chunks = text_splitter.split_documents(documents)<br><br><br># Generate embeddings based on the text chunks using Bedrock&#39;s Embedding Model and store them in Redis<br>redis_url = &quot;redis://a7c63f0c9a740483987398a6c0b6e444–967754848.us-east-2.elb.amazonaws.com:6379&quot;<br>embeddings = BedrockEmbeddings(region_name=&quot;us-east-1&quot;, model_id=&quot;amazon.titan-embed-text-v1&quot;)<br>index_name = &quot;docs-bedrock&quot;<br>vectorstore = Redis(<br>  redis_url=redis_url,<br>  embedding=embeddings,<br>  index_name=index_name,<br>)<br><br>vectorstore.drop_index(redis_url=redis_url, index_name=index_name, delete_documents=True)<br>vectorstore.add_documents(documents=chunks)</pre><h3>LangChain Chain</h3><p>Before exploring the second Python script, responsible for the actual RAG Query, it’s important to get a better understanding of what a LangChain Chain is. Here’s a simple example including Kong AI Gateway and Amazon Bedrock. Note that the code does not implement RAG, restricted to explore the Chain concept. The next section will evolve the example to include RAG.</p><p>The code requires the installation of the “langchain_openai” package.</p><pre>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>kong_dp = &quot;http://k8s-kong-kongkong-cf4df736b2-f8ba17426cc444cc.elb.us-east-2.amazonaws.com&quot;<br>bedrock_rag_url = kong_dp + &quot;/bedrock-route&quot;<br><br>prompt = ChatPromptTemplate.from_template(&quot;tell me what you think about {topic}&quot;)<br><br>llm = ChatOpenAI(base_url=bedrock_rag_url, model_name=&quot;meta.llama3–70b-instruct-v1:0&quot;, temperature=0, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})<br><br><br>chain = prompt | llm<br><br><br>print(&quot;\nanswer: &quot;, chain.invoke({&quot;topic&quot;: &quot;Ozempic&quot;}).content)<br>print(&quot;\nanswer: &quot;, chain.invoke({&quot;topic&quot;: &quot;Fantastic Negrito&quot;}).content)</pre><p>The most important line of the code is the creation of the LangChain Chain: chain = prompt | llm.</p><p>A Chain, the main LangChain component, can be created with <a href="https://python.langchain.com/docs/concepts/#langchain-expression-language-lcel">LCEL</a> (LangChain Expression Language), a declarative language. In fact, a Chain is responsible for executing a sequence of steps, “chained” together and separated by the <a href="https://python.langchain.com/docs/how_to/sequence/#the-pipe-operator-">Pipe (“|”) Operator</a>. All steps of a Chain implement the <a href="https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.base.Runnable.html">Runnable</a> LangChain Class.</p><p>In our case, the Chain takes the output of the Prompt, passes it to the LLM, which, as expected, sends the prompt to the LLM and gets the response.</p><p>Specifically, for the prompt object, it has been created with a {topic} template. So it needs to be informed by the chain.invoke Method.</p><p>On the other hand, the llm object actually refers to Kong AI Gateway. As we stated before, Kong AI Gateway supports the OpenAI API specification, meaning that the consumer, in our case the LangChain code, should interact with the Gateway through the standard LangChain <a href="https://python.langchain.com/api_reference/openai/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html">ChatOpenAI chat model class</a>.</p><p>Here’s the line:</p><p>llm = ChatOpenAI(base_url=bedrock_rag_url, model_name=”meta.llama3–70b-instruct-v1:0&quot;, temperature=0, api_key=”dummy”, default_headers={“apikey”: “123456”})</p><p>Note that the Class refers to the Kong AI Gateway Route. We’re asking the Gateway to consume the Bedrock model meta.llama3–70b-instruct-v1:0. Besides, since the Route has the Key Auth plugin enabled, we’ve added the API Key using the default_headers parameter. Finally, the standard and required api_key parameter should be passed but it’s totally ignored by the Gateway.</p><h3>RAG Query</h3><p>Now, let’s evolve the script to add RAG. This time, LangChain Chain will be updated to include the Vector Database.</p><p>First, remember that, in a RAG application, the Query should combine a Prompt and contextual and relevant data, related to the Prompt, coming from the Vector Database. Something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*T_ALLJqElpJTa-gD" /></figure><p>Considering this flow we should implement a parallel processing with both sources. That’s what the <a href="https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.RunnableParallel.html">LangChain’s RunnableParallel</a> class does. So, the new Chain would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*R_gDHLYPEDpISDr2" /></figure><p>And the code snippet with the new Chain should be like this. Fundamentally, the Chain has included two new steps: one before and another after the original prompt and llm steps.</p><pre>rag_chain = (<br>  retriever_prompt_setup<br>  | prompt<br>  | llm<br>  | StrOutputParser()<br>)</pre><p>So let’s review all of them. However, before jumping to the Chain, we have to describe a bit how it is going to hit the Redis Vector Database to get the Relevant Data coming from it.</p><h4>retriever</h4><p>The first line gets the reference to the Redis index we created in the first Python script.</p><pre>vectorstore = Redis(<br>  redis_url = &quot;redis://a7c63f0c9a740483987398a6c0b6e444–967754848.us-east-2.elb.amazonaws.com:6379&quot;,<br>  embedding = BedrockEmbeddings(region_name=&quot;us-east-1&quot;, model_id=&quot;amazon.titan-embed-text-v1&quot;),<br>  index_name = &quot;docs-bedrock&quot;,<br>)</pre><p>The second line calls the <a href="https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.Redis.html#langchain_community.vectorstores.redis.base.Redis.as_retriever">“as_retriever”</a> method of the Redis class so we can start sending queries to Redis with a <a href="https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.redis.base.RedisVectorStoreRetriever.html#langchain_community.vectorstores.redis.base.RedisVectorStoreRetriever">RedisVectorStoreRetriever</a> object. For example, here’s the “retriever” object construction set with “similarity distance threshold” search type:</p><pre>retriever = vectorstore.as_retriever(<br>  search_type=&quot;similarity_distance_threshold&quot;,<br>  search_kwargs={&quot;distance_threshold&quot;: 0.6}<br>)<br><br>print(&quot;Redis query: \n&quot; + str(retriever.invoke(input=&quot;what did Marco say about Ingress Controller?&quot;)[0]))</pre><p>Check this <a href="https://python.langchain.com/docs/how_to/vectorstore_retriever/">how-to guide</a> to learn more about Retrievers.</p><h3>New LangChain Chain</h3><p>Let’s check the steps of the new Chain now.</p><h4>retriever_prompt_setup</h4><p>retriever_prompt_setup = RunnableParallel({“context”: retriever, “question”: RunnablePassthrough()})</p><p>The line calls the <a href="https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.RunnableParallel.html">RunnableParallel</a> class which launches, in parallel, one task per parameter:</p><ul><li>“context”: refers to the retriever which is, in our case, the Redis Vector Database.</li><li>“question”: uses the <a href="https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.passthrough.RunnablePassthrough.html">RunnablePassthrough</a> class which takes the Chain’s input and passes it with no updates. The RunnablePassthrough() call basically says the “question” parameter is not defined yet but it will be provided eventually. Check the following <a href="https://python.langchain.com/docs/how_to/passthrough/">how-to guide</a> to learn more about the RunnablePassthrough class</li></ul><p>Both RunnableParallel and RunnablePassthrough classes are subclasses of the fundamental <a href="https://python.langchain.com/v0.2/api_reference/core/runnables/langchain_core.runnables.base.Runnable.html">Runnable</a> class.</p><p>The Chain’s flow will proceed only when the two tasks have been completed. The next step, “prompt”, will receive both data as follows:</p><p>{“context”: &lt;data_coming_from_Redis&gt;, “question”: &lt;question_from_the_chain’s_invoke_method&gt;}</p><h4>prompt</h4><p>Considering that it will receive the output of the “retriever_prompt_setup” step, it should be modified. Here it is:</p><pre>prompt = ChatPromptTemplate.from_template(template)<br>And the template needs to refer to both context and question:<br>template = &quot;&quot;&quot;<br>Answer the question based only on the following context:<br>{context}<br>Question: {question}<br>&quot;&quot;&quot;</pre><h4>llm</h4><p>The “llm” step remains exactly the same. It calls the Kong AI Gateway route which is responsible for routing the request to Amazon Bedrock:</p><pre>llm = ChatOpenAI(base_url=bedrock_rag_url, model_name=&quot;meta.llama3–70b-instruct-v1:0&quot;, temperature=0, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})</pre><h4>StrOutputParser()</h4><p>The <a href="https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.string.StrOutputParser.html">StrOutParser</a> class parses the output coming from the “llm” step in a readable way.</p><h3>Invoking the Chain</h3><p>We’re now ready to invoke the Chain. Here’s one example:</p><pre>query = &quot;What did Marco say about AI Gateway?&quot;<br>print(&quot;\nRAG query: &quot; + query)<br>print(&quot;\nRAG answer: &quot;, rag_chain.invoke(query))</pre><p>Of course, the response will be totally contextualized considering the data we have in our Vector Database.</p><p>LangChain provides a nice <a href="https://python.langchain.com/docs/tutorials/rag/">tutorial</a> on how to build a RAG application.</p><h3>Kong AI Gateway Plugins</h3><p>Once we have the RAG Application done, it’d be interesting to start enabling the AI Plugins we have created.</p><h3>Kong AI Prompt Decorator plugin</h3><p>For example, besides the Key Auth plugin, you can enable the AI Prompt Decorator, configured with decK, to see the response in Brazilian Portuguese.</p><p>You can use decK again, if you will or go to Konnect UI and look for the Kong Route’s Plugins page. Use the toggle button to enable the Plugin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FIXLP7l1oH48KdXV" /></figure><h3>Kong AI Rate Limiting Advanced plugin</h3><p>You can also enable the AI Rate Limiting Advanced plugin we have created for the user. Notice that, differently to the AI Prompt Decorator plugin, this plugin has been applied to the existing Kong Consumer created by the decK declaration. If you create another Kong Consumer (and a different Credential), the AI Rate Limiting plugin will not be enforced.</p><p>Also note that the AI Rate Limiting Advanced defines policies based on the number of tokens returned by the LLM provider, not the number of requests as we usually have for regular rate limiting plugins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f_iVnsbbNw-YL7T_" /></figure><h3>Python script code</h3><p>Here’s the code of the second Python script:</p><pre>from langchain_community.vectorstores.redis import Redis<br>from langchain_aws import BedrockEmbeddings<br><br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>from langchain_core.runnables import RunnablePassthrough, RunnableParallel<br>from langchain_core.output_parsers import StrOutputParser<br><br><br>kong_dp = &quot;http://k8s-kong-kongkong-af48daf85f-3e01263abaeeb50a.elb.us-east-2.amazonaws.com&quot;<br>bedrock_rag_url = kong_dp + &quot;/bedrock-route&quot;<br>redis_url = &quot;redis://a7c63f0c9a740483987398a6c0b6e444–967754848.us-east-2.elb.amazonaws.com:6379&quot;<br>embeddings = BedrockEmbeddings(region_name=&quot;us-east-1&quot;, model_id=&quot;amazon.titan-embed-text-v1&quot;)<br>index_name = &quot;docs-bedrock&quot;<br><br><br># Step 1: Get Redis Index<br>vectorstore = Redis(<br>  redis_url=redis_url,<br>  embedding=embeddings,<br>  index_name=index_name,<br>)<br><br><br># Step 2: Retrieve &amp; Augment<br># Once you have the vector database reference, you can define it as the retriever component,<br># which fetches the additional context based on the semantic similarity between the user query and the embedded chunks.<br>retriever = vectorstore.as_retriever(<br>  search_type=&quot;similarity_distance_threshold&quot;,<br>  search_kwargs={&quot;distance_threshold&quot;: 0.6}<br>)<br><br># Next, to augment the prompt with the additional context, you need to prepare a prompt template.<br># The prompt can be easily customized from a prompt template, as shown below.<br>template = &quot;&quot;&quot;<br>Answer the question based only on the following context:<br>{context}<br>Question: {question}<br>&quot;&quot;&quot;<br><br>prompt = ChatPromptTemplate.from_template(template)<br><br><br># Step 3: Generate<br># Finally, you can build a chain for the RAG pipeline, chaining together the retriever, the prompt template and the LLM.<br># Once the RAG chain is defined, you can invoke it.<br>llm = ChatOpenAI(base_url=bedrock_rag_url, model_name=&quot;meta.llama3–70b-instruct-v1:0&quot;, temperature=0, api_key=&quot;dummy&quot;, default_headers={&quot;apikey&quot;: &quot;123456&quot;})<br>retriever_prompt_setup = RunnableParallel({&quot;context&quot;: retriever, &quot;question&quot;: RunnablePassthrough()})<br><br>rag_chain = (<br>  retriever_prompt_setup<br>  | prompt<br>  | llm<br>  | StrOutputParser()<br>)<br><br><br><br>query = &quot;What did Marco say about AI Gateway?&quot;<br>print(&quot;\nRAG query: &quot; + query)<br>print(&quot;\nRAG answer: &quot;, rag_chain.invoke(query))<br><br><br>query = &quot;Is there any reference to Kubernetes in the context?&quot;<br>print(&quot;\nRAG query: &quot; + query)<br>print(&quot;\nRAG answer: &quot;, rag_chain.invoke(query))<br><br><br>query = &quot;What did MP say about Chopin?&quot;<br>print(&quot;\nRAG query: &quot; + query)<br>print(&quot;\nRAG answer: &quot;, rag_chain.invoke(query))</pre><h3>Conclusion</h3><p>This blog post has presented a basic RAG Application using Kong AI Gateway, LangChain, Amazon Bedrock, and Redis as the Vector Database. It’s totally feasible to implement advanced RAG apps with query transformation, multiple data sources, multiple retrieval stages, RAG Agents, etc. Moreover, Kong AI Gateway provides other plugins to enrich the relationship with the LLM providers, including Semantic Cache, Semantic Routing, Request and Response Transformation, etc.</p><p>Also, it’s important to keep in mind that, just like we did with the Key Auth plugins, we can continue combining other API Gateway plugins to your AI-based use cases like using the OIDC plugin to secure your Foundation Models with AWS Cognito, using the Prometheus plugin to monitor your AI Gateway with Amazon Managed Prometheus and Grafana, and so on.</p><p>Finally, the architect flexibility provided natively by Konnect and Kong API Gateway allows us to deploy the Data Planes in a variety of platforms including AWS EC2 VMs, Amazon ECS, and Kong Dedicated Cloud Gateway, Kong’s SaaS service for the Data Planes running in AWS.</p><p>You can discover all the features available on the <a href="https://konghq.com/products/kong-ai-gateway">Kong AI Gateway</a> product page, or you can check the Kong and AWS landing page at <a href="https://konghq.com/partners/aws">https://konghq.com/partners/aws</a> to learn more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=580a244b7ee4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi Amazon VPC Kong Konnect Data Plane 3.7 deployment with EKS 1.30 and AWS Transit Gateway]]></title>
            <link>https://medium.com/@claudioacquaviva/multi-amazon-vpc-kong-konnect-data-plane-3-7-deployment-with-eks-1-30-and-aws-transit-gateway-046176ef3ae3?source=rss-878574aaf46f------2</link>
            <guid isPermaLink="false">https://medium.com/p/046176ef3ae3</guid>
            <dc:creator><![CDATA[Claudio Acquaviva]]></dc:creator>
            <pubDate>Fri, 07 Jun 2024 11:50:30 GMT</pubDate>
            <atom:updated>2024-06-17T11:10:31.678Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The native hybrid deployment capabilities is one of the most critical benefits Kong brings to the table. Having the Control Plane totally managed by Kong as we manage how, when and where the Data Planes are deployed provide a powerful and highly flexible solution for the Kong Gateway Cluster implementation.</p><p>Considering a full AWS based deployment, one very compelling topology we usually face is the multi VPC scenario. In such a situation, the services and applications are running on a VPC as Kong, protecting these workloads, runs on a second VPC. Of course, in a deployment like this we should manage private IPs, since exposing the application with public ELBs is not recommended.</p><h3>Kong Konnect Data Plane and AWS Transit Gateway</h3><p>AWS provides several options to solve the <a href="https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/amazon-vpc-to-amazon-vpc-connectivity-options.html">VPC-to-VPC connectivity</a>, including VPC peering, Private Link, etc. For Kong Gateway Clusters implementations AWS Transit Gateway is the recommended solution.</p><p><a href="https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html">AWS Transit Gateway</a> implements a hub-and-spoke architecture where VPC can be attached to and start communicating. The following diagram depicts the architecture where Konnect Data Plane Nodes and the Upstream Services run on different VPCs and connect through AWS Transit Gateway. Note the Upstream Services are getting exposed with an Internal <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load Balancer</a> (NLB), meaning that only components running in the same VPC may consume it. Through AWS Transit Gateway, it is also consumable by components running on other VPCs attached to it. That is the case of the Konnect Data Plane Nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*giYbcXnZN6dmeTE5" /></figure><h3>Kong Konnect Dedicated Cloud Gateways</h3><p>Recently, Kong launched the Dedicated Cloud Gateways. With that you can have your Data Plane Nodes fully managed by Kong in Konnect. Dedicated Cloud Gateways provide some very critical and important benefits:</p><ul><li>Kong handles gateway upgrades automatically.</li><li>You can deploy your Dedicated Cloud Gateway in public or private modes.</li><li>Dedicated Cloud Gateways can automatically manage the Data Plane auto-scaling.</li></ul><p>This blog post describes how to deploy your self managed Data Plane Nodes and get them connected to your Upstream Services running on a different VPC by leveraging AWS Transit Gateway.</p><p>In the second part of the blog post we will examine the Dedicated Cloud Gateway capability provided by Konnect to deploy fully managed Data Planes.</p><h3>AWS Networking highlights</h3><p>An Amazon VPC is a logically isolated virtual network where we deploy resources like EC2s, EKS Clusters, etc. The following diagram was taken from the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html">AWS VPC</a> documentation page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/0*eWrmIzgfJeayeLUY" /></figure><p>Typically, a VPC is defined in an AWS region and has multiple Subnets. A Subnet is a range of IP addresses in the VPC and must reside in a single <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones">Availability Zone</a>.</p><p>An Availability Zone (AZ) is an isolated location defined in an AWS Region. A Region has multiple Availability Zones. For example, here are the AZs of the “us-east-2” (Ohio) AWS Region:</p><pre>% aws ec2 describe-availability-zones --region us-east-2 | jq &#39;.AvailabilityZones[].ZoneName&#39;<br>&quot;us-east-2a&quot;<br>&quot;us-east-2b&quot;<br>&quot;us-east-2c&quot;</pre><p>The Availability Zones are connected via low-latency links to provide replication and fault tolerance.</p><p>A <a href="https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html">Public Subnet</a> is connected to the Internet, otherwise it is called a Private Subnet. You connect a Subnet to the Internet with an <a href="https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html">Internet Gateway</a>. So, after creating the Internet Gateway and attaching it to the VPC, you manage the Route Table for the Subnet that directs internet-bound traffic to the Internet Gateway.</p><h3>Create the EKS Cluster for the Upstream Services</h3><p>Let’s create two EKS Clusters first: one for the Upstream Services and a second one where the Konnect Data Planes are going to be running. In order to create them, we are going to use <a href="https://eksctl.io/">eksctl</a>, the official CLI for Amazon EKS. By default, eksctl creates the EKS Cluster on a separate VPC. So, after getting the clusters running, we are going to configure AWS Transit Gateway to get both VPCs connected.</p><p>Here the eksctl command. To get a better control of the VPC it defines the <a href="https://aws.amazon.com/what-is/cidr/">CIDR (Classless Inter-Domain Routing)</a> the cluster should use for the IP addresses.</p><pre>eksctl create cluster -f - &lt;&lt;EOF<br>apiVersion: eksctl.io/v1alpha5<br>kind: ClusterConfig<br><br>metadata:<br>  name: upstream-eks130<br>  region: us-east-2<br>  version: &quot;1.30&quot;<br><br>vpc:<br>  cidr: 11.10.0.0/16<br><br>managedNodeGroups:<br>  - name: node-upstream<br>    instanceType: c5.2xlarge<br>    minSize: 1<br>    maxSize: 8<br>EOF</pre><p>You can delete the cluster with:</p><pre>eksctl delete cluster --name upstream-eks130 --region us-east-2</pre><h3>Checking the VPC</h3><p>By default, eksctl defines a new VPC. You can get its Id and explore it with the following commands.</p><pre>% aws eks describe-cluster --name upstream-eks130 --region us-east-2 | jq &#39;.cluster.resourcesVpcConfig.vpcId&#39;<br>&quot;vpc-05beced77af54aca4&quot;<br><br>aws ec2 describe-vpcs --region us-east-2 --filters \<br>--vpc-ids vpc-05beced77af54aca4</pre><h3>Subnets</h3><p>You can check the Subnets, created by eksctl, of your VPC. Note that:</p><ul><li>The Subnets are tagged following the EKS Cluster name.</li><li>A Tag Name has a suffix that describes if it’s Public or Private as well as the Availability Zone where it’s been created.</li></ul><pre>% aws ec2 describe-subnets --region us-east-2 \<br>  --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/*&quot; \<br>  | jq -r &#39;.Subnets[] |<br>             &quot;\(.SubnetId)&quot; +<br>             &quot; - subnet Name: &quot; +<br>             &quot;\(.Tags[] | select(.Key==&quot;Name&quot;) | .Value)&quot;&#39;<br>subnet-002c0622f2852512f - subnet Name: eksctl-upstream-eks130-cluster/SubnetPublicUSEAST2A<br>subnet-0befef218a0868335 - subnet Name: eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2A<br>subnet-07cf922431516ec1e - subnet Name: eksctl-upstream-eks130-cluster/SubnetPublicUSEAST2B<br>subnet-02fe464b49b111955 - subnet Name: eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2C<br>subnet-058ccefcb5417a67c - subnet Name: eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2B<br>subnet-05092380f6ede6b32 - subnet Name: eksctl-upstream-eks130-cluster/SubnetPublicUSEAST2C</pre><h3>Internet Gateway</h3><p>Similarly to the Subnets you can check the InternetGateway also created by eksctl and associated to the VPC:</p><pre>% aws ec2 describe-internet-gateways --region us-east-2 \<br>  --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/*&quot; \<br>  | jq -r &#39;.InternetGateways[] |<br>             &quot;\(.InternetGatewayId)&quot; +<br>             &quot; - &quot; +<br>             &quot;\(.Attachments[] | .VpcId)&quot;&#39;<br>igw-0ba77379ed9ce111b - vpc-05beced77af54aca4</pre><h3>Route Tables</h3><p>Also by default, eksctl creates one Route Table per Private Subnet and one Route Table for the Public Subnets:</p><pre>% aws ec2 describe-route-tables - region us-east-2 \<br>  --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/*&quot; \<br>  | jq -r &#39;.RouteTables[].Tags[] |<br>              select(.Key==&quot;Name&quot;) | .Value&#39;<br>eksctl-upstream-eks130-cluster/PrivateRouteTableUSEAST2A<br>eksctl-upstream-eks130-cluster/PrivateRouteTableUSEAST2B<br>eksctl-upstream-eks130-cluster/PublicRouteTable<br>eksctl-upstream-eks130-cluster/PrivateRouteTableUSEAST2C</pre><p>For example, here are the subnets associated with the PublicRouteTable:</p><pre>% aws ec2 describe-route-tables --region us-east-2 \<br>--filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/PublicRouteTable&quot; \<br>| jq -r &#39;.RouteTables[].Associations[].SubnetId&#39;<br>subnet-07cf922431516ec1e<br>subnet-002c0622f2852512f<br>subnet-05092380f6ede6b32</pre><p>Again, they are called PublicSubnets because the Route Table they are associated with has a Route to the InternetGateway:</p><pre>% aws ec2 describe-route-tables --region us-east-2 \<br>--filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/PublicRouteTable&quot; | jq -r &#39;.RouteTables[].Routes&#39;<br>[<br>  {<br>    &quot;DestinationCidrBlock&quot;: &quot;11.10.0.0/16&quot;,<br>    &quot;GatewayId&quot;: &quot;local&quot;,<br>    &quot;Origin&quot;: &quot;CreateRouteTable&quot;,<br>    &quot;State&quot;: &quot;active&quot;<br>  },<br>  {<br>    &quot;DestinationCidrBlock&quot;: &quot;0.0.0.0/0&quot;,<br>    &quot;GatewayId&quot;: &quot;igw-0ba77379ed9ce111b&quot;,<br>    &quot;Origin&quot;: &quot;CreateRoute&quot;,<br>    &quot;State&quot;: &quot;active&quot;<br>  }<br>]</pre><h3>AWS Load Balancer Controller</h3><p>As mentioned before, the Upstream Service will be exposed with Internal NLB. To get an easier deployment, let’s install the AWS Load Balancer Controller in the cluster. The AWS Load Balancer Controller supports several installation processes. The process described next is the recommended one, based on <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/deploy/installation/#option-a-recommended-iam-roles-for-service-accounts-irsa">IRSA</a>. Please, refer to the official <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/">documentation</a> to learn more about the Controller.</p><h3>IAM OIDC provider</h3><p>The Controller requires the <a href="https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html">OIDC provider</a> for the cluster. You can turn that on with another eksctl command:</p><pre>eksctl utils associate-iam-oidc-provider \<br>--cluster upstream-eks130 \<br>--region us-east-2 \<br>--approve</pre><h3>IAM Policy</h3><p>The Controller also requires a collection permissions to provision Load Balancers. The following iam_policy.json file has all needed permissions. Download the file and create the IAM policy with the following commands:</p><p>curl -o iam-policy.json <a href="https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.0/docs/install/iam_policy.json">https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.0/docs/install/iam_policy.json</a></p><pre>aws iam create-policy \<br>    --policy-name AWSLoadBalancerControllerIAMPolicy \<br>    --policy-document file://iam-policy.json</pre><h3>Kubernetes Service Account and IAM Role</h3><p>Now, we need to create and associate both Kubernetes Service Account and an IAM Role which refers to the IAM policy previously created. Again, eksctl does the job. The following command creates:</p><ul><li>The aws-load-balancer-controller Kubenetes Service Account inside the kube-system namespace.</li><li>The upstream-eks129-role IAM Role with the AWSLoadBalancerControllerIAMPolicy policy attached.</li></ul><pre>eksctl create iamserviceaccount \<br>--cluster=upstream-eks130 \<br>--namespace=kube-system \<br>--name=aws-load-balancer-controller \<br>--override-existing-serviceaccounts \<br>--role-name upstream-eks130-role \<br>--attach-policy-arn=arn:aws:iam::&lt;your_aws_account&gt;:policy/AWSLoadBalancerControllerIAMPolicy \<br>--region us-east-2 \<br>--approve</pre><h3>Install AWS Load Balancer Controller</h3><p>The last step uses the Helm Charts to install the Controller:</p><pre>helm repo add eks https://aws.github.io/eks-charts<br><br>helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \<br>--set clusterName=upstream-eks130 \<br>--set region=us-east-2 \<br>--set serviceAccount.create=false \<br>--set serviceAccount.name=aws-load-balancer-controller</pre><h3>Upstream</h3><p>The Upstream Service is based on the <a href="https://github.com/asoorm/go-bench-suite">Go Bench Suite</a> application, useful for benchmarking. Create the namespace first and then apply the declaration. Note the Service uses the AWS Load Balancer Controller annotation to request an Internal NLB.</p><p>Also, as an exercise, to get a better understanding of and control over the deployment, we are going to deploy the NLB in the Private Subnet of the same Availability Zone as our EKS Cluster’s Node has been created.</p><p>First, check the Availability Zone where our EKS Cluster Node is running:</p><pre>% kubectl get node -o json | jq -r &#39;.items[].metadata.labels.&quot;topology.kubernetes.io/zone&quot;&#39;<br>us-east-2b</pre><p>Although it’s not needed, If you will, get the Private Subnet Id associated to the Availability Zone:</p><pre>aws ec2 describe-subnets --region us-east-2 --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2B&quot; --query &quot;Subnets[0].SubnetId&quot;<br>&quot;subnet-058ccefcb5417a67c&quot;</pre><p>Now, deploy the application. The Load Balancer Controller, by default, deploys a NLB. Note the it has another annotation to specify it should be deployed in the same Availability Zone where the Kubernetes Node is:</p><pre>kubectl create namespace upstream<br><br>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: upstream<br>  namespace: upstream<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: upstream<br>  template:<br>    metadata:<br>      labels:<br>        app: upstream<br>        version: v1<br>    spec:<br>      containers:<br>      - name: upstream<br>        image: mangomm/go-bench-suite:latest<br>        command: [&quot;./go-bench-suite&quot;, &quot;upstream&quot;]<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: upstream<br>  namespace: upstream<br>  labels:<br>    run: upstream<br>  annotations:<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-internal&quot;: &quot;true&quot;<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-nlb-target-type&quot;: &quot;ip&quot;<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-subnets&quot;: &quot;eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2B&quot;<br>spec:<br>  type: LoadBalancer<br>  loadBalancerClass: &quot;service.k8s.aws/nlb&quot;<br>  ports:<br>    - port: 8000<br>      targetPort: 8000<br>      nodePort: 30080<br>  selector:<br>    app: upstream<br>EOF</pre><h3>Check the Deployment</h3><pre>% kubectl get pod -o wide -n upstream<br>NAME                       READY STATUS   RESTARTS  AGE   IP             NODE                                           NOMINATED NODE   READINESS GATES<br>upstream-bc448c694–7m8v5   1/1   Running  0         17h   11.10.72.131   ip-11–10–73–142.us-east-2.compute.internal     &lt;none&gt;           &lt;none&gt;</pre><h3>Check the NLB</h3><p>Get the NLB Domain Name with:</p><pre>% kubectl get service -n upstream<br>NAME       TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)          AGE<br>upstream   LoadBalancer   10.100.137.113   k8s-upstream-upstream-52709624f3–942476a572c8902a.elb.us-east-2.amazonaws.com   8000:30080/TCP   32s</pre><p>Check the Load Balancer Type and Scheme. Note the NLB name is used by its Domain Name:</p><pre>% aws elbv2 describe-load-balancers --region us-east-2 \<br>--name &quot;k8s-upstream-upstream-52709624f3&quot; | jq &quot;.LoadBalancers[].Type&quot;<br>&quot;network&quot;<br><br>$ aws elbv2 describe-load-balancers --region us-east-2 \<br>--name &quot;k8s-upstream-upstream-52709624f3&quot; | jq &quot;.LoadBalancers[].Scheme&quot;<br>&quot;internal&quot;</pre><p>The “service.beta.kubernetes.io/aws-load-balancer-nlb-target-type”: “ip” annotation specifies that the route traffic goes directly to the Pod IP, behind the Kubernetes service. Check the AWS Load Balancer documentation to get more familiar with the <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/guide/service/annotations/">Annotations</a>. You can check it out with the following commands:</p><p>Get the NLB Arn fist:</p><pre>% aws elbv2 describe-load-balancers --region us-east-2 \<br>--name &quot;k8s-upstream-upstream-52709624f3&quot; | jq &quot;.LoadBalancers[].LoadBalancerArn&quot;<br>&quot;arn:aws:elasticloadbalancing:us-east-2:&lt;your_aws_account&gt;:loadbalancer/net/k8s-upstream-upstream-52709624f3/942476a572c8902a&quot;</pre><p>Not get the Target Group Arn:</p><pre>% aws elbv2 describe-target-groups -region us-east-2 \<br>--load-balancer-arn &quot;arn:aws:elasticloadbalancing:us-east-2:&lt;your_aws_account&gt;:loadbalancer/net/k8s-upstream-upstream-52709624f3/942476a572c8902a&quot; | jq &#39;.TargetGroups[].TargetGroupArn&#39;<br>&quot;arn:aws:elasticloadbalancing:us-east-2:&lt;your_aws_account&gt;:targetgroup/k8s-upstream-upstream-903ec36ecd/8bbf2dddae509eef&quot;</pre><p>And now check the Target pointing the the Pod Address and Port:</p><pre>% aws elbv2 describe-target-health --region us-east-2 \<br>--target-group-arn &quot;arn:aws:elasticloadbalancing:us-east-2:&lt;your_aws_account&gt;:targetgroup/k8s-upstream-upstream-903ec36ecd/8bbf2dddae509eef&quot;<br>{<br>  &quot;TargetHealthDescriptions&quot;: [<br>    {<br>      &quot;Target&quot;: {<br>        &quot;Id&quot;: &quot;11.10.72.131&quot;,<br>        &quot;Port&quot;: 8000,<br>        &quot;AvailabilityZone&quot;: &quot;us-east-2b&quot;<br>      },<br>      &quot;HealthCheckPort&quot;: &quot;8000&quot;,<br>      &quot;TargetHealth&quot;: {<br>        &quot;State&quot;: &quot;healthy&quot;<br>      }<br>    }<br>  ]<br>}</pre><h3>Consume the Upstream Service</h3><p>Since it’s an Internal NLB, you won’t be able to consume externally, for example, from your laptop. Deploy a Pod to consume the NLB and Upstream Services:</p><pre>kubectl run --rm=true -i --tty ubuntu --image=claudioacquaviva/ubuntu-awscli:0.4 -- /bin/bash<br><br># http k8s-upstream-upstream-52709624f3–942476a572c8902a.elb.us-east-2.amazonaws.com:8000/json/valid<br>HTTP/1.1 200 OK<br>Content-Length: 67<br>Content-Type: text/plain; charset=utf-8<br>Date: Sat, 01 Jun 2024 22:26:31 GMT<br>Server: fasthttp<br>{<br>  &quot;time&quot;: &quot;2024–06–01 22:26:31.68602284 +0000 UTC m=+182.160841052&quot;<br>}</pre><p>Exit the Pod to delete it.</p><h3>Create the EKS Cluster for the Konnect Data Plane</h3><p>Similarly to what we’ve done, create another EKS Cluster, this time for the Konnect Data Planes. Again, we have explicitly defined the CIDR for the cluster.</p><pre>eksctl create cluster -f - &lt;&lt;EOF<br>apiVersion: eksctl.io/v1alpha5<br>kind: ClusterConfig<br><br>metadata:<br>  name: kong37-eks130<br>  region: us-east-2<br>  version: &quot;1.30&quot;<br><br>vpc:<br>  cidr: 12.10.0.0/16<br><br>managedNodeGroups:<br>  - name: node-kong<br>    instanceType: c5.2xlarge<br>    minSize: 1<br>    maxSize: 8<br>EOF</pre><p>You can delete the cluster with:</p><pre>eksctl delete cluster --name kong37-eks130 --region us-east-2</pre><h3>Try to consume the Upstream Service</h3><p>As expected, if you try to consume the same Upstream service, this time from the new EKS Cluster, the request will hang:</p><pre>kubectl create namespace kong<br><br>kubectl run -n kong --rm=true -i --tty ubuntu --image=claudioacquaviva/ubuntu-awscli:0.4 -- /bin/bash<br><br># http k8s-upstream-upstream-52709624f3–942476a572c8902a.elb.us-east-2.amazonaws.com:8000/json/valid<br>http: error: Request timed out (0s).</pre><h3>Create and Configure AWS Transit Gateway</h3><h3>Create TGW</h3><p>With the two EKS Clusters and respective VPCs in place, it’s time to create the Transit Gateway and attach the VPCs to it. The following command creates it:</p><pre>aws ec2 create-transit-gateway --region us-east-2 \<br>--tag-specification &#39;ResourceType=transit-gateway,Tags=[{Key=&quot;Name&quot;,Value=&quot;tgw1&quot;}]&#39;</pre><p>Check if it’s available with:</p><pre>aws ec2 describe-transit-gateways --region us-east-2 \<br>--filters &quot;Name=tag:Name,Values=tgw1&quot; --query &quot;TransitGateways[].State&quot;</pre><p>And get the Transit Gateway Id with:</p><pre>aws ec2 describe-transit-gateways --region us-east-2 \<br>--filters &quot;Name=tag:Name,Values=tgw1&quot; --query &quot;TransitGateways[].TransitGatewayId&quot;<br>[<br>  &quot;tgw-05300030c4f2b9fec&quot;<br>]</pre><h3>Attach the Upstream VPC to the Transit Gateway</h3><p>Make sure your Transit Gateway is available before starting to attach the VPCs to it. There are many options to do it, but, basically, we need to specify which Subnet should be used for the Transit Gateway. We are going to restrict it to the Subnet where the NLB is deployed. At least one Subnet must be selected and on Subnet per Availability Zone. Please read the <a href="https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpc-attachments.html">Transit Gateway Attachment</a> documentation to learn more about it.</p><p>Get the Upstream Cluster’s VPC Id with:</p><pre>aws eks describe-cluster --name upstream-eks130 --region us-east-2 --query &quot;cluster.resourcesVpcConfig.vpcId&quot;<br>&quot;vpc-05beced77af54aca4&quot;</pre><p>To attach the VPC to the Transit Gateway we need to specify the VPC’s Subnet Ids. Get the Private Subnet Id where the application and NLB are deployed. That’s the subnet Transit Gateway will use to route traffic.</p><pre>aws ec2 describe-subnets --region us-east-2 --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/SubnetPrivateUSEAST2B&quot; --query &quot;Subnets[].SubnetId&quot;<br>[<br>  &quot;subnet-058ccefcb5417a67c&quot;<br>]</pre><p>Attach the VPC using its Id, Subnets and the Transit Gateway Id:</p><pre>aws ec2 create-transit-gateway-vpc-attachment --region us-east-2 \<br>  --transit-gateway-id tgw-05300030c4f2b9fec \<br>  --vpc-id vpc-05beced77af54aca4 \<br>  --subnet-ids &quot;subnet-058ccefcb5417a67c&quot; \<br>  --tag-specification &#39;ResourceType=transit-gateway-attachment,Tags=[{Key=&quot;Name&quot;,Value=&quot;tgw1-upstream&quot;}]&#39;</pre><p>Check if the attachment is available:</p><pre>aws ec2 describe-transit-gateway-vpc-attachments --region us-east-2 --filters &quot;Name=tag:Name,Values=tgw1-upstream&quot; --query &quot;TransitGatewayVpcAttachments[].State&quot;</pre><h3>Attach the Konnect Data Plane VPC to the Transit Gateway</h3><p>Repeat the same process for the second VPC. However, we are going to use three existing Public Subnets, where the Deployment and Pods run by default.</p><p>Get the VPC Id:</p><pre>aws eks describe-cluster --name kong37-eks130 --region us-east-2 --query &quot;cluster.resourcesVpcConfig.vpcId&quot;<br>&quot;vpc-04e3b5a5f9de06d12&quot;</pre><p>Now get the three Private Subnet Ids:</p><pre>aws ec2 describe-subnets --region us-east-2 --filters &quot;Name=tag:Name,Values=eksctl-kong37-eks130-cluster/SubnetPublic*&quot; --query &quot;Subnets[].SubnetId&quot;<br>[<br>  &quot;subnet-056ff1b1c971313aa&quot;,<br>  &quot;subnet-0929867562e51469b&quot;,<br>  &quot;subnet-083b2f9e57ed6e273&quot;<br>]</pre><p>Attach the VPC similarly to what we did previously:</p><pre>aws ec2 create-transit-gateway-vpc-attachment --region us-east-2 \<br>  --transit-gateway-id tgw-05300030c4f2b9fec \<br>  --vpc-id vpc-04e3b5a5f9de06d12 \<br>  --subnet-ids &quot;subnet-056ff1b1c971313aa&quot; &quot;subnet-0929867562e51469b&quot; &quot;subnet-083b2f9e57ed6e273&quot; \<br>  --tag-specification &#39;ResourceType=transit-gateway-attachment,Tags=[{Key=&quot;Name&quot;,Value=&quot;tgw1-kong&quot;}]&#39;</pre><p>Check if you attachment is available:</p><pre>aws ec2 describe-transit-gateway-vpc-attachments --region us-east-2 --filters &quot;Name=tag:Name,Values=tgw1-kong&quot; --query &quot;TransitGatewayVpcAttachments[].State&quot;</pre><h3>VPC Route Tables</h3><p>There’s one last and very important step left: we have to tell the VPCs when and how to route the network traffic to the Transit Gateway. We do that by adding new routes to the VPC’s route tables.</p><h3>Konnect Data Plane Cluster Routes</h3><p>Let’s start with the Konnect Data Plane’s Route Table. By default, eksctl tags the Route Table with some predefined names. If you get the PublicRouteTable you see there are two Routes defined:</p><ul><li>Local Target to route VPC internal traffic</li><li>Route to Internet Gateway</li></ul><pre>% aws ec2 describe-route-tables --region us-east-2 --filters &quot;Name=tag:Name,Values=eksctl-kong37-eks130-cluster/PublicRouteTable&quot; --query &quot;RouteTables[].RouteTableId&quot;<br>[<br>  &quot;rtb-08efb2bda04c12687&quot;<br>]<br><br>% aws ec2 describe-route-tables --region us-east-2 --route-table-ids rtb-08efb2bda04c12687 --query &quot;RouteTables[].Routes[]&quot;<br>[<br>  {<br>    &quot;DestinationCidrBlock&quot;: &quot;12.10.0.0/16&quot;,<br>    &quot;GatewayId&quot;: &quot;local&quot;,<br>    &quot;Origin&quot;: &quot;CreateRouteTable&quot;,<br>    &quot;State&quot;: &quot;active&quot;<br>  },<br>  {<br>    &quot;DestinationCidrBlock&quot;: &quot;0.0.0.0/0&quot;,<br>    &quot;GatewayId&quot;: &quot;igw-0396932251e4d591e&quot;,<br>    &quot;Origin&quot;: &quot;CreateRoute&quot;,<br>    &quot;State&quot;: &quot;active&quot;<br>  }<br>]</pre><p>The new Route says the traffic should be redirected to the Transit Gateway for addresses defined in the CIDR we used when we created the Upstream EKS Cluster:</p><pre>aws ec2 create-route --region us-east-2 \<br>--route-table-id rtb-08efb2bda04c12687 \<br>--destination-cidr-block &quot;11.10.0.0/16&quot; \<br>--transit-gateway-id tgw-05300030c4f2b9fec</pre><h3>Upstream Cluster Routes</h3><p>Again, we are going to work with the Route Table defined for the same Private Subnet. Get the Route Table Id:</p><pre>aws ec2 describe-route-tables --region us-east-2 --filters &quot;Name=tag:Name,Values=eksctl-upstream-eks130-cluster/PrivateRouteTableUSEAST2B&quot; --query &quot;RouteTables[].RouteTableId&quot;<br>[<br>  &quot;rtb-07fbc75dbdd76d2c5&quot;<br>]</pre><p>Add the new Route referring the Transit Gateway the Kong EKS Cluster’s CIDR:</p><pre>aws ec2 create-route --region us-east-2 \<br>--route-table-id rtb-07fbc75dbdd76d2c5 \<br>--destination-cidr-block &quot;12.10.0.0/16&quot; \<br>--transit-gateway-id tgw-05300030c4f2b9fec</pre><h3>Try to consume the Upstream Service again</h3><p>You should be able to consume the Upstream Service now:</p><pre># http k8s-upstream-upstream-52709624f3–942476a572c8902a.elb.us-east-2.amazonaws.com:8000/json/valid<br>HTTP/1.1 200 OK<br>Content-Length: 70<br>Content-Type: text/plain; charset=utf-8<br>Date: Sun, 02 Jun 2024 14:49:59 GMT<br>Server: fasthttp<br>{<br>  &quot;time&quot;: &quot;2024–06–02 14:49:59.790928843 +0000 UTC m=+59190.265747067&quot;<br>}</pre><h3>Kong Konnect Data Plane deployment</h3><p>In this next step we are going to deploy the self managed Konnect Data Plane in its EKS Cluster.</p><h3>AWS Load Balancer Controller</h3><p>Repeat the process we did for the Upstream EKS Cluster to install the AWS Load Balancer Controller including:</p><ul><li>Turning the IAM OIDC Provider on.</li><li>Creating the Kubernetes Service Account and IAM Role with eksctl create iamserviceaccount command (since we’ve used the same AWS account to create both Clusters, use the same IAM Policy). Name them accordingly.</li><li>Installing the AWS Load Balancer Controller</li></ul><h3>Digital Certificate and Private Key pair issuing</h3><p>First of all, we need to create the Private Key and Digital Certificate both Konnect Control Plane and Data Plane use to build the mTLS connection.</p><p>For the purpose of this blog post, the secure communication will be based on the PKI Mode. You can use several tools to issue the pair including simple OpenSSL commands like this:</p><pre>openssl req -new -x509 -nodes -newkey ec:&lt;(openssl ecparam -name secp384r1) \<br>-keyout ./kongcp1.key \<br>-out ./kongcp1.crt \<br>-days 1095 \<br>-subj &quot;/CN=konnect_cp1&quot; \<br>-addext &quot;extendedKeyUsage=serverAuth,clientAuth&quot;</pre><p>The -addext “extendedKeyUsage=serverAuth,clientAuth” option tells that the certificate can be used for both server and client Authentication, meaning that, from the Konnect standpoint, it can be used for both Control Plane and Data Plane.</p><h3>Control Plane creation</h3><p>The first thing to do is create the new Konnect Control Plane (CP). You need to have a Konnect PAT (Personal Access Token) in order to send requests to Konnect. Read the <a href="https://docs.konghq.com/konnect/gateway-manager/declarative-config/#generate-a-personal-access-token">Konnect PAT</a> documentation page to learn how to generate one.</p><p>Create a Konnect Control Plane with the following command. It configures the <a href="https://docs.konghq.com/konnect/gateway-manager/data-plane-nodes/secure-communications">PKI Mode</a> for the CP and DP communication, meaning we are going to use the same Public Key to both CP and DP.</p><p>Create an environment variable with your PAT:</p><pre>PAT=kpat_eIYnOh7L8HSRM….</pre><p>Create the Control Plane with:</p><pre>curl -X POST \<br>https://us.api.konghq.com/v2/control-planes \<br>--header &quot;Authorization: Bearer $PAT&quot; \<br>--header &#39;Content-Type: application/json&#39; \<br>--header &#39;accept: application/json&#39; \<br>--data &#39;{<br>  &quot;name&quot;: &quot;cp1&quot;,<br>  &quot;description&quot;: &quot;Control Plane 1&quot;,<br>  &quot;cluster_type&quot;: &quot;CLUSTER_TYPE_HYBRID&quot;,<br>  &quot;labels&quot;:{},<br>  &quot;auth_type&quot;: &quot;pki_client_certs&quot;<br>}&#39;</pre><p>Get the CP Id with:</p><pre>CP_ID=$(curl -s https://us.api.konghq.com/v2/control-planes \<br>--header &quot;Authorization: Bearer $PAT&quot; | jq -r &#39;.data[] | select(.name==&quot;cp1&quot;) | .id&#39;)</pre><p>Get the CP’s Endpoints with:</p><pre>% curl -s https://us.api.konghq.com/v2/control-planes/$CP_ID \<br>--header &quot;Authorization: Bearer $PAT&quot; | jq -r &quot;.config&quot;<br>{<br>  &quot;control_plane_endpoint&quot;: &quot;https://9863135b6c.us.cp0.konghq.com&quot;,<br>  &quot;telemetry_endpoint&quot;: &quot;https://9863135b6c.us.tp0.konghq.com&quot;,<br>  &quot;cluster_type&quot;: &quot;CLUSTER_TYPE_CONTROL_PLANE&quot;,<br>  &quot;auth_type&quot;: &quot;pki_client_certs&quot;,<br>  &quot;cloud_gateway&quot;: false,<br>  &quot;proxy_urls&quot;: []<br>}</pre><p>Now we need to add the Digital Certificate. Use the CP Id in your request:</p><pre>cert=&quot;{\&quot;cert\&quot;: $(jq -sR . ./kongcp1.crt)}&quot;<br><br>curl -X POST https://us.api.konghq.com/v2/control-planes/$CP_ID/dp-client-certificates \<br>--header &quot;Authorization: Bearer $PAT&quot; \<br>--header &#39;Content-Type: application/json&#39; \<br>--header &#39;accept: application/json&#39; \<br>--data $cert</pre><h3>Konnect Data Plane</h3><p>Now, inject the Digital Certificate and Private Key pair in a Kubernetes secret.</p><pre>kubectl create secret tls kong-cluster-cert -n kong --cert=./kongcp1.crt --key=./kongcp1.key</pre><p>Here’s the declaration we’re going to be using. The main point here is we are asking AWS to deploy another NLB, this time for the Konnect Data Plane. Please check the <a href="https://docs.konghq.com/konnect/gateway-manager/data-plane-nodes/">documentation</a> to learn all options to deploy the Data Plane:</p><pre>cat &gt; values.yaml &lt;&lt; &#39;EOF&#39;<br>image:<br>  repository: kong/kong-gateway<br>  tag: &quot;3.7&quot;<br><br>secretVolumes:<br>- kong-cluster-cert<br><br>admin:<br>  enabled: false<br><br>manager:<br>  enabled: false<br><br>proxy:<br>  annotations:<br>    &quot;service.beta.kubernetes.io/aws-load-balancer-internal&quot;: &quot;false&quot;<br><br>env:<br>  role: data_plane<br>  database: &quot;off&quot;<br>  cluster_mtls: pki<br>  cluster_control_plane: 9863135b6c.us.cp0.konghq.com:443<br>  cluster_server_name: 9863135b6c.us.cp0.konghq.com<br>  cluster_telemetry_endpoint: 9863135b6c.us.tp0.konghq.com:443<br>  cluster_telemetry_server_name: 9863135b6c.us.tp0.konghq.com<br>  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt<br>  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key<br>  lua_ssl_trusted_certificate: system<br>  konnect_mode: &quot;on&quot;<br>  vitals: &quot;off&quot;<br><br>ingressController:<br>  enabled: false<br>  installCRDs: false<br>EOF</pre><p>Deploy the Konnect Data Plane with a Helm command:</p><pre>helm install kong kong/kong -n kong --values ./values.yaml</pre><p>You should see the Data Plane running:</p><pre>% kubectl get pod -n kong<br>NAME                         READY STATUS   RESTARTS  AGE<br>kong-kong-6755865687-t8hfk   1/1   Running  0         63s</pre><p>Check the Konnect GUI also:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jCCFvOsv2WNmzn0N" /></figure><h3>Kong Gateway Service and Route</h3><p>Now, let’s create a new Kong Service and Route. You can use the Konnect GUI if you like or, again, the Konnect RESTful API:</p><h3>Kong Gateway Service</h3><p>The Kong Service will refer to the Upstream EKS Cluster’s NLB. Due to the Transit Gateway attachments, the Kong Data should be able to consume it.</p><pre>http https://us.api.konghq.com/v2/control-planes/$CP_ID/core-entities/services name=service1 \<br>url=&#39;http://k8s-upstream-upstream-52709624f3-942476a572c8902a.elb.us-east-2.amazonaws.com:8000&#39; \<br>Authorization:&quot;Bearer $PAT&quot;</pre><p>Get your new Gateway Service Id with:</p><pre>SERVICE_ID=$(http https://us.api.konghq.com/v2/control-planes/$CP_ID/core-entities/services/service1 \<br>Authorization:&quot;Bearer $PAT&quot; | jq -r &quot;.id&quot;)</pre><h3>Kong Route</h3><p>Use the Service Id to define the Kong Route:</p><pre>http https://us.api.konghq.com/v2/control-planes/$CP_ID/core-entities/services/$SERVICE_ID/routes name=&#39;route1&#39; paths:=&#39;[&quot;/route1&quot;]&#39; Authorization:&quot;Bearer $PAT&quot;</pre><h3>Consume the Route</h3><p>Get the Load Balancer DNS name</p><pre>% kubectl get service kong-kong-proxy -n kong -o json | jq -r &quot;.status.loadBalancer.ingress[].hostname&quot;<br>k8s-kong-kongkong-29bdf46bcc-5a7c65d4d83660fe.elb.us-east-2.amazonaws.com</pre><p>Consume the Kong Route. Again, since both VPCs are attached to the Transit Gateway, the Data Plane should be able to consume the NLB.</p><pre>http k8s-kong-kongkong-29bdf46bcc-5a7c65d4d83660fe.elb.us-east-2.amazonaws.com/route1/json/valid<br>HTTP/1.1 200 OK<br>Connection: keep-alive<br>Content-Length: 70<br>Content-Type: text/plain; charset=utf-8<br>Date: Sun, 02 Jun 2024 17:35:52 GMT<br>Server: fasthttp<br>Via: kong/3.7.0.0-enterprise-edition<br>X-Kong-Proxy-Latency: 1<br>X-Kong-Request-Id: c5347455eae58a3b06952c05edfaa633<br>X-Kong-Upstream-Latency: 4<br>{<br>  &quot;time&quot;: &quot;2024–06–02 17:35:52.751754738 +0000 UTC m=+69143.226572949&quot;<br>}</pre><h3>Conclusion</h3><p>This blog post described Kong Konnect Data Plane deployment to connect to an Upstream Service running on a different VPC through AWS Transit Gateway.</p><p>In the second part of this series we are going to explore how to deploy a fully managed Dedicated Cloud Gateway, which also leverages AWS Transit Gateway in a multi-account scenario.</p><p><a href="https://konghq.com/konnect">Kong Konnect</a> simplifies API management and improves security for all services infrastructure. <a href="https://konghq.com/products/kong-konnect/register">Try it for free</a> today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=046176ef3ae3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>