<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Levi  Sharifi on Medium]]></title>
        <description><![CDATA[Stories by Levi  Sharifi on Medium]]></description>
        <link>https://medium.com/@levisharifi?source=rss-8996ff2d9ea2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DVc0SCvrMCtbcbuJWqiWYQ.jpeg</url>
            <title>Stories by Levi  Sharifi on Medium</title>
            <link>https://medium.com/@levisharifi?source=rss-8996ff2d9ea2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:08:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@levisharifi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Gateway Architecture with APIM]]></title>
            <link>https://medium.com/@levisharifi/ai-gateway-architecture-with-apim-916d87a8e179?source=rss-8996ff2d9ea2------2</link>
            <guid isPermaLink="false">https://medium.com/p/916d87a8e179</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[apim]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <dc:creator><![CDATA[Levi  Sharifi]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 21:43:18 GMT</pubDate>
            <atom:updated>2026-09-18T21:43:41.271Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0gKj3_CzFkblUVquqxZcA.png" /></figure><h3>Using APIM with Microsoft Foundry, LLMs, agents, and MCP servers without turning every AI call into a gateway hop</h3><p>Enterprise AI architectures are starting to look less like a single application calling a single model and more like a distributed system.</p><p>An application may call an AI agent. The agent may use one or more LLMs, invoke several MCP tools, retrieve enterprise data, and call existing APIs. Different teams may own each part of that path.</p><p>A simplified architecture might look like this:</p><pre>Application<br>    |<br>    v<br>AI Agent<br>    |<br>    +----&gt; LLM / Model<br>    |<br>    +----&gt; MCP Server ----&gt; Enterprise API<br>    |<br>    +----&gt; MCP Server ----&gt; Data Platform</pre><p>Once this architecture grows, the gateway question appears quickly:</p><p><strong>Should these interactions go through Azure API Management?</strong></p><p>APIM can absolutely operate as an AI gateway. Microsoft positions its AI gateway capabilities as an extension of API Management for securing, scaling, monitoring, and governing models, agents, remote MCP servers, and other AI endpoints. These capabilities exist across APIM, although availability varies by tier. Microsoft also has a dedicated <strong>AI Gateway tier</strong>, currently in preview, which should not be confused with the broader AI gateway capabilities available in API Management. (<a href="https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities">Microsoft Learn</a>)</p><p>But the fact that APIM <em>can</em> sit in the path does not mean it should sit in every path.</p><p>The better architecture question is:</p><blockquote><strong><em>Where do we need gateway capabilities, and is APIM the right implementation at that boundary?</em></strong></blockquote><h3>What does an AI gateway actually do?</h3><p>An AI gateway is not an agent orchestrator.</p><p>It does not decide what an agent should do next. It does not perform the agent’s reasoning, decide which business tool should be invoked, or automatically determine that a particular prompt is complex enough to require a larger model.</p><p>Those responsibilities belong elsewhere in the AI architecture.</p><p>A gateway has a more conventional architectural role: it mediates traffic between consumers and providers.</p><p>For AI workloads, APIM can provide authentication and authorization, backend authentication, routing, URL transformation, request and token limits, quotas, model endpoint abstraction, load balancing, circuit breaking, content-safety policies, observability, and centralized governance. APIM also supports semantic caching for LLM traffic and can emit token-consumption metrics into Azure monitoring systems. (<a href="https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities">Microsoft Learn</a>)</p><p>That makes APIM useful when an AI interaction crosses a boundary that the organization wants to control.</p><h3>APIM in front of LLMs</h3><p>The most obvious AI gateway scenario is model access.</p><p>Without a gateway:</p><pre>Agent A ----&gt; Model Endpoint A<br>Agent B ----&gt; Model Endpoint A<br>App C   ----&gt; Model Endpoint B<br>App D   ----&gt; Model Endpoint C</pre><p>Each application may need to understand the model endpoint, authentication mechanism, API contract, quota, and provider configuration.</p><p>With APIM:</p><pre>Agent A ----\<br>Agent B -----\<br>App C --------&gt; APIM ----&gt; Model A<br>App D -------/      \----&gt; Model B<br>                         \-&gt; Model C</pre><p>The consuming application now talks to a managed enterprise endpoint.</p><p>APIM can enforce token limits per consumer, subscription, IP address, or another policy-derived key. This is particularly useful because model capacity is often measured in tokens rather than simple request counts. APIM’s llm-token-limit policy supports both token-per-minute limits and longer-period token quotas. (<a href="https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities">Microsoft Learn</a>)</p><p>APIM can also distribute traffic across backend model endpoints. Backend pools support strategies such as round-robin, weighted, priority-based, and session-aware distribution, and backend circuit breakers can temporarily stop sending traffic to an unhealthy endpoint. (<a href="https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities">Microsoft Learn</a>)</p><p>This becomes useful when several applications share a finite amount of AI capacity. Instead of allowing one consumer to exhaust the underlying model quota, the gateway becomes the enforcement point.</p><h3>Routing is more than putting a URL behind APIM</h3><p>One practical benefit of APIM is separating the public interface from the physical backend.</p><p>Suppose consumers call:</p><pre>https://ai.company.com/models/chat</pre><p>but the actual implementation is:</p><pre>https://some-resource.openai.azure.com/...</pre><p>APIM does not require the external route to mirror the backend route.</p><p>The set-backend-service policy can redirect a request to a configured backend or another base URL, while rewrite-uri can transform the public request path into the structure expected by the backend. (<a href="https://learn.microsoft.com/en-us/azure/api-management/set-backend-service-policy?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>These are related but different concerns.</p><p>set-backend-service answers:</p><p><strong>Where should this request go?</strong></p><p>rewrite-uri answers:</p><p><strong>What should the backend request path look like?</strong></p><p>That means the implementation behind the gateway can evolve without forcing every AI consumer to change its endpoint configuration.</p><p>The same mechanism can support conditional routing. APIM policies can inspect request context and select different configured backends.</p><p>But there is an important distinction here.</p><p>APIM does not inherently reason about the semantic complexity of a prompt and decide that “this request deserves the expensive model.” If an organization wants routing such as:</p><pre>Workload type A -&gt; Model A<br>Workload type B -&gt; Model B</pre><p>the decision criteria must be explicitly represented in gateway policies or determined by another component.</p><p>The dedicated AI Gateway tier similarly routes models according to configured model names rather than independently reasoning about which model is best for a request. (<a href="https://learn.microsoft.com/en-us/azure/api-management/ai-gateway-overview?utm_source=chatgpt.com">Microsoft Learn</a>)</p><h3>Security: there are two authentication relationships</h3><p>One of the most important reasons to introduce APIM is credential separation.</p><p>Consider this architecture:</p><pre>Agent<br>  |<br>  | Identity A<br>  v<br>APIM<br>  |<br>  | Identity B<br>  v<br>Model</pre><p>There are two different trust relationships.</p><p>The first is:</p><pre>Consumer -&gt; APIM</pre><p>The second is:</p><pre>APIM -&gt; Backend</pre><p>They do not need to use the same authentication mechanism.</p><p>On the inbound side, APIM can validate tokens issued by Microsoft Entra ID or other identity providers. For example, APIM provides validate-azure-ad-token for Entra tokens and validate-jwt for more general JWT validation. (<a href="https://learn.microsoft.com/en-us/azure/api-management/validate-jwt-policy?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>On the backend side, APIM can use its own managed identity.</p><p>The authentication-managed-identity policy obtains an access token from Microsoft Entra ID and places that token in the backend Authorization header. Both system-assigned and user-assigned managed identities are supported. (<a href="https://learn.microsoft.com/en-us/azure/api-management/authentication-managed-identity-policy?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>The result can look like this:</p><pre>Foundry Agent<br>      |<br>      | Entra token / APIM key<br>      v<br>     APIM<br>      |<br>      | APIM Managed Identity<br>      v<br>Azure OpenAI / Foundry Model</pre><p>The agent no longer needs the backend model credential.</p><p>This is a useful architectural property. The consumer is authorized to use an enterprise capability, while the gateway owns the credential required to reach the physical provider.</p><p>If the provider changes, the consumer’s security contract does not necessarily need to change with it.</p><h3>Microsoft Foundry and APIM have different responsibilities</h3><p>APIM and Microsoft Foundry should not be treated as competing components.</p><p>They solve different problems.</p><p>Microsoft Foundry Agent Service provides the agent runtime and agent capabilities. APIM provides the managed network and policy boundary around the services those agents consume.</p><p>For example, Foundry Agent Service supports models hosted behind Azure API Management. A Foundry project can create a model connection whose base URL points to APIM, and the agent then consumes the model through that gateway. Foundry can authenticate to APIM using supported mechanisms including API keys and, for APIM configurations that support it, managed identity. (<a href="https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/ai-gateway?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>That gives us a topology such as:</p><pre>Microsoft Foundry Agent<br>          |<br>          v<br>         APIM<br>          |<br>          v<br>Enterprise Model</pre><p>A useful separation of responsibilities is:</p><pre>Microsoft Foundry<br>    Agent runtime<br>    Agent lifecycle<br>    Tool orchestration<br>    Model/tool integration</pre><pre>Azure API Management<br>    Gateway policies<br>    Security boundary<br>    Traffic control<br>    Backend identity<br>    Routing<br>    Quotas<br>    Observability</pre><p>Microsoft has also introduced direct AI-gateway integration with Foundry, currently documented as preview. Through this integration, organizations can govern models, agents, and tools and apply controls such as throttling, token limits, content safety, and telemetry through the connected APIM gateway. (<a href="https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities">Microsoft Learn</a>)</p><p>The important point is that the gateway does not replace the agent platform.</p><p>It governs traffic crossing the boundary around it.</p><h3>Where MCP changes the discussion</h3><p>MCP introduces another important gateway use case.</p><p>An agent might consume several tools:</p><pre>Agent<br>  |<br>  +----&gt; Product MCP<br>  |<br>  +----&gt; Inventory MCP<br>  |<br>  +----&gt; Order MCP</pre><p>If these are small, dedicated MCP servers owned by the same workload, direct communication may be sufficient.</p><p>But imagine those MCP servers become reusable enterprise capabilities consumed by many agents:</p><pre>Agent A ----\<br>Agent B -----\<br>Agent C ------&gt; APIM ----&gt; Product MCP<br>Agent D -----/      \----&gt; Inventory MCP<br>                        \-&gt; Order MCP</pre><p>Now APIM has a clearer responsibility.</p><p>It can provide consistent authentication, authorization, rate limiting, quotas, monitoring, IP controls, and other policies around access to those tools. APIM currently supports <strong>remote MCP servers</strong> using HTTP-based transports. Local stdio-based MCP servers are a different execution model and are not what APIM proxies. (<a href="https://learn.microsoft.com/en-us/azure/api-management/mcp-server-overview?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>APIM supports two particularly interesting MCP patterns.</p><p>The first is placing an existing remote MCP server behind APIM:</p><pre>Agent<br>  |<br>  v<br>APIM<br>  |<br>  v<br>Remote MCP Server</pre><p>In this case, the MCP implementation already exists. APIM provides the gateway boundary around it.</p><p>The second is exposing an existing REST API as an MCP server:</p><pre>Agent<br>  |<br>  | MCP<br>  v<br>APIM<br>  |<br>  v<br>Existing REST API</pre><p>APIM can expose operations from an HTTP-compatible REST API as MCP tools. This is useful for organizations that already have a large API estate and want to make selected capabilities available to agents without rebuilding every service as a separate MCP implementation. (<a href="https://learn.microsoft.com/en-us/azure/api-management/mcp-server-overview?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>There are current product limitations to account for. For REST APIs exposed as MCP servers, APIM currently supports MCP tools but not MCP resources or prompts. Existing remote MCP servers have somewhat different support, including tools and resources but currently not prompts. MCP capabilities are also currently not supported in APIM workspaces. (<a href="https://learn.microsoft.com/en-us/azure/api-management/mcp-server-overview?utm_source=chatgpt.com">Microsoft Learn</a>)</p><p>Those details matter when designing the platform because “APIM supports MCP” should not be interpreted as “every MCP capability is supported in every APIM topology.”</p><h3>Policies are the real value of the gateway</h3><p>A reverse proxy by itself does not provide much architectural value.</p><p>The interesting part of APIM is the policy pipeline.</p><p>A model request could conceptually move through the gateway like this:</p><pre>Request<br>   |<br>   v<br>Validate identity<br>   |<br>   v<br>Apply authorization rules<br>   |<br>   v<br>Apply request/token limits<br>   |<br>   v<br>Content safety policy<br>   |<br>   v<br>Select backend<br>   |<br>   v<br>Rewrite backend URL if required<br>   |<br>   v<br>Acquire backend identity<br>   |<br>   v<br>Call model<br>   |<br>   v<br>Record telemetry</pre><p>The same principle applies to MCP traffic, although the applicable policies and protocol behavior differ.</p><p>This is where centralized governance becomes concrete.</p><p>An architecture document can say that every team should respect a model quota. A gateway policy can actually enforce it.</p><p>An architecture document can say that agents should not hold backend credentials. APIM using managed identity can actually remove those credentials from the consumer.</p><p>An architecture document can say that only specific identities should reach a capability. An APIM token-validation policy can enforce that boundary.</p><p>That is a much stronger reason for introducing APIM than simply saying, “all AI traffic should use a gateway.”</p><h3>When APIM is a good choice</h3><p>APIM becomes particularly useful when several consumers share a capability, when access crosses team or security boundaries, when backend credentials should be hidden from consumers, when traffic or token consumption needs centralized control, when a stable enterprise endpoint should sit in front of changing model backends, or when reusable MCP tools need consistent security and observability.</p><p>The common characteristic is <strong>shared governance</strong>.</p><p>Consider:</p><pre>Enterprise AI Boundary</pre><pre>Applications<br>Agents          ----&gt; APIM ----&gt; Models<br>Copilots                    \--&gt; MCP Servers<br>Other platforms             \--&gt; AI APIs</pre><p>In this architecture, APIM represents something meaningful.</p><p>It is the boundary between independently developed consumers and centrally governed AI capabilities.</p><h3>When APIM may not be necessary</h3><p>Now consider a different topology:</p><pre>Foundry Agent<br>     |<br>     v<br>Dedicated MCP Server</pre><p>The MCP server exists only for that agent.</p><p>Both run inside the same controlled solution.</p><p>Identity is already handled.</p><p>There is no shared consumption.</p><p>There are no centralized quota requirements.</p><p>There is no need to hide a separate provider credential.</p><p>Observability is already sufficient.</p><p>Changing the architecture to:</p><pre>Foundry Agent<br>     |<br>     v<br>    APIM<br>     |<br>     v<br>Dedicated MCP Server</pre><p>may provide very little additional value.</p><p>It adds another runtime dependency, another network hop, another configuration surface, another component to monitor, and another potential point of failure.</p><p>The same argument applies to model access.</p><p>If one application calls one model and all required controls are already handled directly by the platform, putting APIM in the middle purely because “AI traffic should go through APIM” is difficult to justify.</p><p>A gateway should solve a problem.</p><p>It should not become an architecture convention looking for a problem.</p><h3>A practical decision test</h3><p>Before onboarding an AI interaction to APIM, I would ask one set of questions:</p><ol><li>Is this capability shared by multiple applications, agents, or teams?</li><li>Do we need centralized authentication or authorization?</li><li>Do we need to isolate backend credentials from consumers?</li><li>Do we need rate limits, token quotas, cost controls, content safety, or centralized observability?</li><li>Do we need routing or abstraction between consumers and the underlying model or MCP implementation?</li><li>Does this interaction cross an organizational, security, or platform boundary?</li><li>Are these requirements already adequately handled elsewhere?</li></ol><p>If several of the first six answers are yes, APIM becomes a strong candidate.</p><p>If the final answer is yes and another platform already provides everything required, APIM may simply duplicate controls.</p><p>That last question is important.</p><p>The decision is not:</p><p><strong>Can APIM do this?</strong></p><p>It often can.</p><p>The decision is:</p><p><strong>What capability do we gain by putting APIM here?</strong></p><h3>The architecture boundary matters more than the technology</h3><p>APIM’s AI gateway capabilities are a natural extension of what API gateways have always done: mediate communication across a boundary.</p><p>AI adds new concerns such as token consumption, model providers, agents, MCP tools, semantic caching, and AI-specific safety policies, but the architecture principle remains familiar.</p><p>Put the gateway where centralized control is valuable.</p><p>Use it to separate consumer identity from backend identity.</p><p>Use policies to turn governance requirements into enforceable runtime behavior.</p><p>Use routing and backend abstraction where consumers should not depend directly on physical model endpoints.</p><p>Use it around reusable MCP capabilities when those tools become part of an enterprise platform.</p><p>But do not assume that every LLM call, every agent interaction, or every MCP connection needs APIM.</p><p>The objective is not to route <strong>all AI traffic through APIM</strong>.</p><p>The objective is to identify the points where <strong>AI traffic needs a managed boundary</strong>, and then decide whether Azure API Management is the right technology to implement it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=916d87a8e179" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Pull-Based Messaging Gives Consumers More Freedom]]></title>
            <link>https://medium.com/@levisharifi/why-pull-based-messaging-gives-consumers-more-freedom-3cf912ceca88?source=rss-8996ff2d9ea2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cf912ceca88</guid>
            <category><![CDATA[message-broker]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[pull-based-system]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Levi  Sharifi]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 02:19:28 GMT</pubDate>
            <atom:updated>2026-07-18T02:24:14.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MmNGKa8krrCjTmWUGxIeeg.png" /></figure><h3>Why Pull-Based Messaging Gives Consumers More Freedom</h3><p>Modern distributed systems often need to process large volumes of messages without overwhelming downstream services.</p><p>One important architectural decision is whether the broker should push messages to consumers or whether consumers should pull messages from the broker.</p><p>In a pull-based messaging model, consumers decide when to request data and how much data to retrieve. Apache Kafka is a well-known example of a messaging platform that uses this approach.</p><p>This design gives consumers more control over their workloads and can improve backpressure, batching, scalability, and failure recovery.</p><h3>The Consumer Controls the Pace</h3><p>In a pull-based system, the consumer requests messages only when it is ready to process them.</p><p>A fast consumer can request data frequently and process large batches. A slower consumer can wait longer before requesting more messages.</p><pre>Consumer is ready<br>→ request messages<br>→ receive a batch<br>→ process the batch<br>→ request more messages</pre><p>The broker does not continuously send new messages to a consumer that is already busy.</p><p>This allows each consumer to operate according to its own CPU, memory, network capacity, and processing complexity.</p><p>For example, one consumer may perform a simple validation, while another may call an external API or write data to a database. Even though these consumers have different processing speeds, each can request messages at a rate that matches its own capacity.</p><p>Kafka, as an example of a pull-based messaging platform, allows consumers to fetch records through polling and decide when to perform the next poll.</p><h3>Natural Backpressure</h3><p>Backpressure occurs when messages are produced faster than consumers can process them.</p><p>In a push-based system, the broker may continue delivering messages until an in-flight or prefetch limit is reached. The broker then needs mechanisms such as acknowledgements, buffering, flow control, and delivery pauses to avoid overwhelming the consumer.</p><p>In a pull-based model, the consumer controls the pressure more naturally.</p><p>When the consumer is busy, it delays the next request. The unread messages remain stored in the broker until the consumer is ready.</p><pre>Consumer slows down<br>→ requests happen less frequently<br>→ messages remain in the broker<br>→ consumer continues when ready</pre><p>This does not prevent a backlog from growing. When producers publish messages faster than consumers can process them, lag will still increase.</p><p>However, the consumer is less likely to be overwhelmed by an uncontrolled flow of pushed messages.</p><p>Kafka, as an example of this model, keeps unread records in the topic while the consumer falls behind. The distance between the latest record and the consumer’s current position is commonly called consumer lag.</p><h3>Efficient Batch Processing</h3><p>Pull-based consumption also makes batching natural.</p><p>Instead of receiving one message at a time, a consumer can request many messages in a single operation.</p><pre>One request<br>→ hundreds of messages<br>→ batch processing<br>→ fewer network and database calls</pre><p>Batching can reduce network round trips, request overhead, database calls, serialization costs, and CPU usage.</p><p>The consumer can choose a batch size that fits its workload.</p><p>A low-latency service may prefer small, frequent batches. An analytics or data-processing service may prefer larger batches to maximize throughput.</p><p>Kafka, as an example of a pull-based platform, allows a consumer to receive multiple records in a single poll and process them together.</p><h3>Easier Horizontal Scaling</h3><p>Pull-based systems can scale by distributing work across multiple consumers.</p><p>The exact scaling model depends on the messaging platform.</p><p>Kafka, for example, divides topics into partitions. Consumers that belong to the same consumer group share those partitions.</p><pre>Partition 0 → Consumer A<br>Partition 1 → Consumer B<br>Partition 2 → Consumer C</pre><p>When more processing capacity is needed, additional consumers can be added to the group. Kafka can then redistribute the partitions across the available consumers.</p><pre>More partitions<br>+ more consumers<br>→ more parallel processing<br>→ higher throughput</pre><p>In Kafka’s implementation of the pull-based model, the number of partitions determines the maximum active parallelism within one consumer group.</p><p>For example, if a topic has six partitions, up to six consumers in the same group can actively process those partitions at the same time.</p><p>The broader pull-based benefit is that consumers can be scaled independently according to processing demand.</p><h3>Independent Consumer Workloads</h3><p>Pull-based messaging can also allow different applications to consume the same messages independently.</p><p>Each application can maintain its own progress and process messages at its own speed.</p><pre>Order events<br>├── Payment consumer<br>├── Fraud-detection consumer<br>├── Analytics consumer<br>└── Notification consumer</pre><p>The payment service may process events immediately, while the analytics service may process them in larger batches.</p><p>A slow analytics consumer does not necessarily block the payment consumer because each one manages its own progress.</p><p>Kafka is one example of this model. It uses consumer groups and offsets so that separate applications can read the same topic independently.</p><h3>Recovery and Replay</h3><p>Some pull-based messaging platforms keep messages available after they have been consumed.</p><p>This allows consumers to recover from failures or process older messages again.</p><p>Common use cases include restarting after a crash, reprocessing events after fixing a bug, rebuilding application state, processing historical data, and adding new consumers.</p><p>Kafka is a strong example of this approach.</p><p>Kafka stores events in an ordered log for a configured retention period. Consumers track their progress using offsets.</p><p>A consumer can continue from its latest committed offset or reset its position to an earlier offset and replay older records.</p><p>Replay is not automatically guaranteed by every pull-based system. It is a capability provided by platforms that retain messages and track consumer positions.</p><h3>Less Delivery Coordination for the Broker</h3><p>In a push-based system, the broker decides when messages should be delivered.</p><p>The broker may need to track whether the consumer is ready, how many messages are already in flight, which messages have been acknowledged, and when delivery should pause or resume.</p><p>This creates a relationship between consumer processing speed and broker delivery behavior.</p><pre>Consumer processing speed<br>→ acknowledgement speed<br>→ broker delivery rate</pre><p>In a pull-based system, more control moves to the consumer.</p><pre>Push-based model:<br>The broker adapts delivery to consumer capacity</pre><pre>Pull-based model:<br>The consumer adapts consumption to its own capacity</pre><p>Kafka is an example of the second model. Consumers request records instead of waiting for the broker to push each new record.</p><h3>When Push-Based Messaging Is a Better Choice</h3><p>Push-based messaging can be a better choice when immediate delivery and very low latency are more important than large-scale batch processing.</p><p>For example, real-time notifications, chat messages, user-interface updates, alerting systems, and lightweight event handlers may benefit from messages being delivered as soon as they are available.</p><p>Push-based brokers can also simplify consumer logic because the application does not need to poll continuously. Mature push-based platforms usually provide acknowledgements, prefetch limits, retry handling, and flow control to prevent consumers from being overwhelmed.</p><p>The decision should therefore depend on the workload. Pull-based messaging is often a strong fit for high-throughput processing, batching, replay, and independently scalable consumers. Push-based messaging can be more suitable when events must reach consumers immediately and processing is relatively fast and predictable.</p><h3>Freedom Comes with Responsibility</h3><p>Pull-based messaging gives consumers more freedom, but that freedom also creates responsibility.</p><p>Consumers must request messages regularly, track their processing position correctly, monitor lag, handle duplicate processing, recover safely from failures, and choose suitable batch sizes.</p><p>Kafka provides a practical example of these responsibilities.</p><p>A Kafka consumer must poll regularly. If it takes too long between polls, Kafka may consider it unavailable and reassign its partitions to another consumer.</p><p>Consumers must also commit offsets carefully. Committing too early can lead to missed processing after a failure, while committing too late can lead to duplicate processing.</p><p>The pull model provides control, but applications must use that control correctly.</p><h3>Final Thoughts</h3><p>Pull-based messaging gives consumers control over when they read messages, how many messages they retrieve, and how quickly they process them.</p><p>This can provide several important benefits:</p><ul><li>Natural backpressure</li><li>Consumer-controlled processing speed</li><li>Efficient batching</li><li>Independent consumer workloads</li><li>Horizontal scalability</li><li>Better failure recovery</li><li>Reduced broker-side delivery coordination</li></ul><p>Kafka is one prominent example of a pull-based messaging platform. Features such as polling, partitions, consumer groups, offsets, retention, and replay demonstrate how Kafka implements the model in practice.</p><p>However, the central idea belongs to pull-based messaging in general:</p><pre>The broker stores the messages.<br>The consumer decides when it is ready.</pre><p>That separation gives consumers more freedom and makes pull-based messaging a strong choice for scalable, high-throughput, event-driven systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cf912ceca88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Treat Context as a System: A Practical Habit for Working with AI Agents]]></title>
            <link>https://medium.com/@levisharifi/treat-context-as-a-system-a-practical-habit-for-working-with-ai-agents-d34bf5e52e07?source=rss-8996ff2d9ea2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d34bf5e52e07</guid>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Levi  Sharifi]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 03:31:27 GMT</pubDate>
            <atom:updated>2026-04-20T03:31:27.893Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkHeJ4SV3NPfRFViCWAeRw.png" /></figure><p>As AI agents become part of our daily workflow, most people focus on prompts, tools, or models. But one of the most overlooked factors is context.</p><p>AI agents are powerful, but they are fundamentally stateless. Every interaction starts from scratch unless you explicitly provide the relevant context again. This leads to repeated explanations, inconsistent outputs, and lost knowledge over time.</p><p>A simple but highly effective practice is to maintain a living Markdown (.md) file that acts as an external memory for your AI workflows, and to continuously update it with the help of AI itself.</p><h3>The Core Idea</h3><p>Instead of relying on long chat histories or trying to “teach” the model over time, you create a structured .md file that contains:</p><ul><li>Project context</li><li>Key decisions and trade-offs</li><li>Important constraints</li><li>Summaries of discussions</li><li>Reusable patterns and prompts</li></ul><p>This file is not static. It evolves continuously. After meaningful interactions, you ask the AI to summarize and update the document.</p><p>Over time, this becomes a high-signal, low-noise representation of your work.</p><h3>Why This Matters: AI Agents Are Stateless</h3><p>AI models do not remember your past sessions. Even within a session, they are limited by a context window. This creates several problems:</p><ul><li>You repeat the same explanations</li><li>The AI gives inconsistent answers across sessions</li><li>Important decisions get lost in chat history</li><li>Knowledge remains fragmented</li></ul><p>By maintaining a structured context file, you are effectively externalizing memory.</p><p>Instead of relying on:</p><blockquote><em>“What did we discuss last week?”</em></blockquote><p>You rely on:</p><blockquote><em>“Here is the current state of truth.”</em></blockquote><h3>Treat Context as a System</h3><p>The real shift is not just creating a Markdown file. It is treating context as a system.</p><p>Most people handle context in an ad-hoc way: scattered chats, random notes, and unstructured documents. This leads to noise, inconsistency, and poor AI outputs.</p><p>A system, on the other hand, has intent and design.</p><h3>1. Structure</h3><p>Context should be organized, not dumped into a single file. For example:</p><pre>/context<br>  ├── project.md<br>  ├── decisions.md<br>  ├── patterns.md<br>  └── current-task.md</pre><p>Each file has a clear purpose. This makes it easier for both humans and AI to understand and use.</p><h3>2. Rules</h3><p>Not everything belongs in your context.</p><p>Include:</p><ul><li>Key decisions and their rationale</li><li>Stable architectural knowledge</li><li>Important constraints</li></ul><p>Avoid:</p><ul><li>Raw chat logs</li><li>Temporary thoughts</li><li>Redundant information</li></ul><p>The goal is to maintain signal over noise.</p><h3>3. Lifecycle</h3><p>Context is not a one-time effort. It is continuously maintained.</p><ul><li>Update after important decisions</li><li>Refine summaries over time</li><li>Remove outdated or incorrect information</li></ul><p>Just like code, context needs refactoring.</p><h3>4. Integration with AI</h3><p>This is where many people fail.</p><p>A Markdown file sitting somewhere is useless unless it is actively used.</p><ul><li>Feed relevant sections into prompts</li><li>Use it as input for your AI agent</li><li>Connect it via retrieval mechanisms if needed</li></ul><p>Without integration, there is no system, only documentation.</p><h3>5. Ownership</h3><p>Context needs ownership.</p><p>Someone must be responsible for:</p><ul><li>Keeping it accurate</li><li>Keeping it clean</li><li>Ensuring it evolves with the project</li></ul><p>Otherwise, it quickly becomes outdated and loses value.</p><h3>Why Markdown?</h3><p>Markdown is simple, portable, and AI-friendly.</p><ul><li>Easy to read and write</li><li>Works in any environment</li><li>Version-controllable</li><li>Ideal for LLM input</li></ul><p>It fits naturally into engineering workflows without introducing unnecessary complexity.</p><h3>From Conversations to Knowledge</h3><p>Raw chat history is noisy and inefficient. It contains repetition, dead ends, and partial ideas.</p><p>A curated Markdown file transforms this into:</p><ul><li>Clear decisions</li><li>Structured knowledge</li><li>Reusable insights</li></ul><p>This shifts your workflow from:</p><blockquote><em>“Chatting with AI”</em></blockquote><p>to:</p><blockquote><em>“Building a knowledge system with AI”</em></blockquote><h3>Team-Level Impact</h3><p>At the team level, this practice becomes even more valuable.</p><p>A shared context system can act as:</p><ul><li>A lightweight knowledge base</li><li>A record of architectural decisions</li><li>A source of truth for constraints and patterns</li><li>A consistent input for all AI-assisted workflows</li></ul><p>This leads to:</p><ul><li>Faster onboarding</li><li>Better alignment across engineers</li><li>More consistent AI outputs</li><li>Reduced dependency on tribal knowledge</li></ul><p>Instead of knowledge being scattered across chats and individuals, it becomes shared, structured, and reusable.</p><h3>Final Thought</h3><p>AI agents are not limited by intelligence as much as they are limited by context.</p><p>If you improve the way you manage context, you improve the quality of everything the AI produces.</p><p>Treating context as a system is a small shift in practice, but it creates a significant difference in outcomes.</p><p>It turns AI from a tool you “chat with” into a system you build on top of.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d34bf5e52e07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using MCP Servers and LLMs to Enable Intelligent Operations on AKS]]></title>
            <link>https://medium.com/@levisharifi/using-mcp-servers-and-llms-to-enable-intelligent-operations-on-aks-efa449c1a6cb?source=rss-8996ff2d9ea2------2</link>
            <guid isPermaLink="false">https://medium.com/p/efa449c1a6cb</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mcps]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ak]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Levi  Sharifi]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 03:20:58 GMT</pubDate>
            <atom:updated>2026-04-20T03:21:46.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKYcSFqQLOme_rbA0w8ILw.png" /></figure><h3>Introduction</h3><p>There is increasing interest in using LLMs to assist with infrastructure operations, especially in Kubernetes environments such as Azure Kubernetes Service (AKS).</p><p>A common question arises early:</p><p>Why not allow an LLM to directly interact with the Kubernetes API?</p><p>While technically feasible, this approach introduces significant architectural risks. A more robust pattern is to introduce a controlled interaction layer using MCP servers and an orchestration backend.</p><p>This article outlines a practical architecture for enabling LLM-assisted operations on AKS while preserving control, security, and auditability.</p><h3>Why Not Let LLM Talk to AKS Directly?</h3><p>Before discussing the solution, it is important to understand why direct interaction is not appropriate.</p><h3>Unbounded control surface</h3><p>The Kubernetes API exposes powerful operations:</p><ul><li>scaling workloads</li><li>modifying deployments</li><li>accessing secrets</li><li>deleting resources</li></ul><p>Direct access effectively grants unrestricted control.</p><h3>Lack of deterministic policy enforcement</h3><p>Operational rules such as:</p><ul><li>restart cooldowns</li><li>production restrictions</li><li>approval requirements</li></ul><p>must be enforced in deterministic code, not prompts.</p><h3>Identity and compliance</h3><p>Enterprise systems require:</p><ul><li>strong identity (Entra ID)</li><li>RBAC enforcement</li><li>auditability</li></ul><p>An LLM is not a valid identity boundary.</p><h3>Non-deterministic behavior</h3><p>LLMs are probabilistic. They are useful for reasoning, but not safe for direct execution of infrastructure changes.</p><h3>Architectural Overview</h3><p>A controlled architecture introduces separation of concerns:</p><pre>User / UI<br>   ↓<br>Backend (MCP Client / Orchestration Layer)<br>   ↓<br>LLM (Azure OpenAI)<br>   ↓<br>MCP Server<br>   ↓<br>AKS</pre><h3>Responsibility Separation</h3><ul><li>LLM: reasoning and decision suggestion</li><li>MCP Server: controlled execution</li><li>Backend (MCP client): orchestration and policy</li><li>AKS: infrastructure</li></ul><p>The LLM never directly interacts with AKS.</p><h3>Physical Deployment (What Runs Where)</h3><p>A realistic production setup looks like this:</p><pre>Your Laptop / UI<br>   ↓<br>VM #1 (Backend / MCP Client)<br>   - REST API (/ops)<br>   - LLM integration<br>   - policy engine<br>   ↓<br>HTTPS<br>   ↓<br>Azure OpenAI (LLM - managed service)<br>   ↑<br>   ↓<br>VM #2 (MCP Server)<br>   - tool implementations<br>   - Kubernetes client<br>   ↓<br>HTTPS<br>   ↓<br>AKS API Server</pre><h3>Key Clarifications</h3><ul><li>The LLM is not running in your VM</li><li>It is a managed Azure service accessed over HTTPS</li><li>The MCP client is your backend application</li><li>The MCP server is another service exposing tools</li></ul><h3>How the Prompt Actually Flows (Real Network Chain)</h3><p>This is the most important part often misunderstood.</p><p>When you type:</p><blockquote><em>“check pods in prod”</em></blockquote><p>The flow is:</p><pre>1. UI → Backend (HTTP POST /ops)</pre><pre>2. Backend → Azure OpenAI (HTTPS REST call)</pre><pre>3. LLM → Backend (response with tool decision)</pre><pre>4. Backend → MCP Server (HTTP call)</pre><pre>5. MCP Server → AKS API (HTTPS)</pre><pre>6. Result → MCP Server → Backend → LLM → Backend → UI</pre><h3>Important Insight</h3><ul><li>The LLM never calls MCP</li><li>The LLM never calls AKS</li><li>All calls are initiated by the backend</li></ul><h3>Minimal Capability Surface</h3><p>Expose only safe operations:</p><pre>getPodStatus(namespace, deployment)<br>restartDeployment(namespace, deployment)</pre><p>Avoid:</p><pre>run_kubectl(command)</pre><h3>MCP Server (Tool Layer)</h3><pre>class K8sTools {</pre><pre>    PodStatus getPodStatus(String namespace, String deployment) {<br>        // Kubernetes API call<br>    }</pre><pre>    String restartDeployment(String namespace, String deployment) {<br>        // rollout restart<br>    }<br>}</pre><h3>Orchestration Layer (MCP Client)</h3><pre>tools = mcpClient.listTools()</pre><pre>response = llm.ask(request, tools)</pre><pre>while (response.hasToolCall()) {</pre><pre>    call = response.getToolCall()</pre><pre>    policyEngine.validate(call)</pre><pre>    result = mcpClient.callTool(call.name, call.args)</pre><pre>    response = llm.continueWith(result)<br>}</pre><pre>return response.finalAnswer()</pre><p>The backend is always in control.</p><h3>MCP Server Placement: Inside AKS vs External</h3><h3>Inside AKS</h3><pre>AKS<br> ├── Apps<br> ├── MCP Server<br> └── API</pre><p>Pros</p><ul><li>simple networking</li><li>low latency</li></ul><p>Cons</p><ul><li>shared failure domain</li><li>harder debugging during outages</li></ul><h3>External Control Plane (Recommended)</h3><pre>External VM<br>     |<br>     v<br>  MCP Server<br>     |<br>     v<br>    AKS</pre><p>Pros</p><ul><li>independent from cluster</li><li>better isolation</li><li>supports multi-cluster</li></ul><p>Cons</p><ul><li>more setup</li></ul><h3>Network and Authentication</h3><p>Authentication:</p><ul><li>Entra ID</li><li>kubelogin</li><li>RBAC</li></ul><p>Network:</p><ul><li>Public AKS: IP restrictions</li><li>Private AKS: VNet / VPN / peering</li></ul><h3>Evolution of Operations: From Runbooks to Code and Back to Reasoning</h3><p>Infrastructure operations have evolved through three phases.</p><h3>Phase 1: Human-Driven</h3><ul><li>engineers read logs</li><li>follow runbooks</li><li>make decisions</li></ul><h3>Phase 2: Code-Driven</h3><pre>if X happens → do Y</pre><ul><li>scripts</li><li>pipelines</li><li>rule-based automation</li></ul><h3>Phase 3: Reasoning-Assisted</h3><p>With LLMs:</p><pre>Code → exact control<br>LLM → contextual reasoning</pre><h3>Key Insight</h3><p>We are not moving to no-code.</p><p>We are moving to:</p><p>less rigid logic and more controlled reasoning.</p><h3>Extended Scenario: Log-Driven Investigation</h3><p>Alert:</p><p>“payment-service error rate increased”</p><p>Unclear cause:</p><ul><li>deployment</li><li>DB</li><li>config</li></ul><h3>Flow</h3><pre>1. LLM calls getPodStatus<br>2. LLM calls getEvents<br>3. LLM calls getLogs<br>4. MCP retrieves data<br>5. LLM analyzes<br>6. Backend enforces policy<br>7. restartDeployment if safe</pre><h3>Example Logs</h3><pre>ERROR: Unable to connect to database<br>Timeout after 30s</pre><h3>Key Value</h3><p>LLM helps with:</p><ul><li>log interpretation</li><li>hypothesis generation</li><li>investigation flow</li></ul><p>Execution remains controlled.</p><h3>Trade-offs</h3><p>Benefits</p><ul><li>faster troubleshooting</li><li>reduced manual work</li><li>flexible workflows</li></ul><p>Challenges</p><ul><li>system complexity</li><li>governance requirements</li><li>non-deterministic reasoning</li></ul><h3>Conclusion</h3><p>Direct LLM control of AKS is not a safe architecture.</p><p>A layered approach enables:</p><ul><li>controlled execution</li><li>policy enforcement</li><li>safe AI integration</li></ul><h3>Final Statement</h3><p>LLM provides reasoning. MCP provides controlled execution. The backend enforces policy.</p><p>Together, they enable intelligent and safe operations on AKS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efa449c1a6cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>