<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Santosh Pai on Medium]]></title>
        <description><![CDATA[Stories by Santosh Pai on Medium]]></description>
        <link>https://medium.com/@santoshpai?source=rss-7827c154fa0b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*TwsTeh1B495KUQ9W.jpeg</url>
            <title>Stories by Santosh Pai on Medium</title>
            <link>https://medium.com/@santoshpai?source=rss-7827c154fa0b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:50:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@santoshpai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an AI Native Developer Platform]]></title>
            <link>https://itnext.io/building-an-ai-native-developer-platform-b93a948f592c?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b93a948f592c</guid>
            <category><![CDATA[ai-control-plane]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Fri, 12 Jun 2026 12:04:41 GMT</pubDate>
            <atom:updated>2026-06-12T13:20:38.329Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Developers</strong> are the <strong>primary actors</strong> in a software engineering organization — and with that top of mind it becomes necessary to ensure that the developer experience in an organization is the best in class.</p><p><strong>Platform engineering</strong> emerged as an organisational response to the increase in complexity of software delivery. Internal developer platforms intend to reduce the operational burden and cognitive load placed on development teams by providing reusable workflows, self-service capabilities, standardised environments, operational guardrails, and golden paths.</p><p>Industry interest in platform engineering has grown rapidly. Gartner projected that by 2026, 80 percent of large software engineering organisations would establish platform engineering teams, compared with 45 percent in 2022. DORA describes platform engineering as a socio-technical discipline involving automation, self-service, repeatability, shared services, and golden paths. These definitions indicate that platform engineering involves more than the deployment of infrastructure or the introduction of a developer portal.</p><p>Maintaining the <strong>relationship between development and operations</strong> is core to <strong>good devops and platform engineering</strong> practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-u2ADBYWAfEz3XT.png" /><figcaption>Buildkit for devops engineering</figcaption></figure><h3>Declaring victory too early</h3><p>Platform teams often declare success after reaching visible technical milestones.</p><p>A service can be deployed. A cluster is operational. A continuous integration and delivery pipeline exists. Developers can request infrastructure through a portal, and dashboards indicate that workloads are healthy. From an infrastructure perspective, the platform may appear complete.</p><p>Yet adoption may remain limited.</p><p>Developers continue using locally maintained scripts. Teams bypass recommended workflows. Infrastructure engineers repeatedly respond to similar support requests. Instead of reducing complexity, the platform becomes another system that developers must learn and operate.</p><p>This outcome suggests that technical availability is not equivalent to platform effectiveness. A platform can contain sophisticated infrastructure while providing limited organisational leverage.</p><p>Low adoption is frequently attributed to developer resistance. However, developers often leave the supported path because it is slower, narrower, or more difficult than their existing workflow. A platform that exposes infrastructure complexity through a portal has changed the interface without necessarily reducing the cognitive burden. Similarly, a platform that supports only idealised application patterns may be abandoned when teams encounter production requirements that the golden path does not accommodate.</p><p>Poorly designed platforms can therefore increase the distance between developers and production rather than reduce it. They may centralise control without creating meaningful self-service, standardise deployment without standardising operations, or automate individual tasks without providing a coherent end-to-end experience.</p><p>This leads to a more useful question than whether an organisation has established a platform:</p><p><strong>Where does the organisation sit on a platform maturity scale?</strong></p><p>Answering this question requires examining both the experience of building software and the systems required to operate it.</p><h3>Three Interdependent Workframes</h3><p>Platform maturity should be considered through three related workframes.</p><h3>1. The Developer Experience Workframe</h3><p>The developer-experience workframe concerns how effectively engineers can transform an idea or requirement into a working software change. It includes:</p><ul><li>discovering available platform capabilities</li><li>creating repositories and services</li><li>provisioning development environments</li><li>obtaining approved dependencies</li><li>testing changes</li><li>deploying applications</li><li>diagnosing failures</li><li>accessing documentation and operational information.</li></ul><p>A strong developer experience reduces unnecessary cognitive load while preserving appropriate autonomy. Developers should not need to understand every internal implementation detail in order to complete routine delivery tasks. Yet, the platform should support the curious.</p><h3>2. The Production-Operations Workframe</h3><p>The production workframe concerns the organisation’s ability to operate software safely and economically. It includes:</p><ul><li>reliability and availability</li><li>security and policy enforcement</li><li>observability and incident response</li><li>scalability and capacity planning</li><li>governance and compliance</li><li>cost management</li><li>recovery and continuity</li><li>ownership and accountability.</li></ul><p>A platform optimised exclusively for developer speed may allow applications to reach production without sufficient operational controls. Conversely, a platform designed primarily for central governance may become too restrictive or cumbersome for development teams.</p><p>Mature platforms reconcile these workframes by providing developers with simple paths that automatically incorporate production requirements. The objective is not to remove production controls, but to embed them into reusable platform capabilities.</p><h3>3. The Agent Experience Workframe</h3><p>The agent-experience workframe should be coupled with the developer-experience and production-operations workframes because software delivery is increasingly performed not only by people, but also by AI agents acting on their behalf.</p><p>Agents need structured access to organisational context, platform capabilities, policies, repositories, environments, and operational signals. They also require stable interfaces, explicit permissions, bounded actions, approval points, and auditable execution paths.</p><p>A mature platform should therefore provide an agent-ready operating model in which developers express intent, agents plan and execute authorised tasks, and the control plane enforces identity, policy, state reconciliation, observability, and accountability. The agent-experience workframe does not replace developer experience; it extends it by ensuring that automation can participate safely and consistently in the same path from requirement to production.</p><h3>Five-Level Platform Maturity Model</h3><p>The model represents a progression from individual technical capability towards an integrated engineering system. Organisations may demonstrate characteristics from multiple levels, and maturity may vary between teams, applications, and business units. The model should therefore be used as an analytical framework rather than as a rigid classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bH9IV4aiM16hUWjc.png" /><figcaption>Platform maturity progresses from manual deployment to a continuously improving engineering system.</figcaption></figure><h3>Level 1: Deployment capability</h3><p>At Level 1, an application can be deployed and operated by an engineer or a small group of specialists.</p><p>Processes may rely on manual commands, undocumented conventions, copied configuration, direct infrastructure access, and individual expertise. The organisation has demonstrated technical feasibility, but successful delivery remains dependent on specific people.</p><p>This level is appropriate for <strong>prototypes, experiments, and personal systems</strong>. It should not be mistaken for an organisation-wide platform.</p><h3>Level 2: Repeatable automation</h3><p>At Level 2, recurring procedures are encoded through scripts, templates, build pipelines, and documented practices.</p><p>Deployment becomes more repeatable, although teams may continue maintaining their own versions of pipelines and configuration. The organisation has reduced some manual work, but developers still require substantial knowledge of the underlying infrastructure.</p><p>Automation at this level primarily reproduces existing processes. It does not necessarily provide a unified product experience.</p><h3>Level 3: Self-service platform</h3><p>At Level 3, developers can access common capabilities through <strong>stable APIs</strong>, command-line interfaces, declarative specifications, or developer portals.</p><p>The platform may provision repositories, environments, databases, messaging services, deployment pipelines, observability, and access controls. Organisational standards are incorporated into reusable workflows, reducing the need for tickets and direct intervention from infrastructure teams.</p><p>The transition from Level 2 to Level 3 is significant because the platform begins transferring operational capability without transferring all underlying complexity.</p><h3>Level 4: Governed and observable operations</h3><p>At Level 4, the platform extends beyond provisioning and deployment into continuous operational control.</p><p>It can answer questions such as:</p><ul><li>Who owns and operates a service?</li><li>Which version is currently deployed?</li><li>Which policy decisions affected the release?</li><li>What is the service’s cost and budget status?</li><li>Which dependency or deployment preceded an incident?</li><li>Is the service compliant with organisational requirements?</li><li>Can it be recovered using a tested process?</li></ul><p>Policies, telemetry, cost controls, audit evidence, and lifecycle management become integrated platform capabilities rather than external procedures applied after deployment.</p><h3>Level 5: Continuous engineering-system optimisation</h3><p>At Level 5, the platform is managed as a product whose users are internal development and operations teams.</p><p>The platform team evaluates adoption, task completion, developer satisfaction, lead time, deployment frequency, reliability, recovery time, platform support demand, and the frequency with which teams leave the recommended path.</p><p>The objective is no longer limited to automating infrastructure. The organisation continuously improves the overall engineering system using evidence gathered from developer behaviour and operational outcomes.</p><p>At this level, platform maturity is demonstrated by <strong>voluntary adoption</strong> and measurable organisational leverage.</p><h3>Kubernetes as the Underlying Platform</h3><p>Kubernetes is frequently associated with platform engineering because it provides a consistent, declarative, API-driven operating model. Its abstractions cover workloads, services, configuration, secrets, resource management, health checks, scheduling, scaling, and deployment strategies. Its extension mechanisms also allow organisations to define custom resource models and controllers.</p><p>These characteristics make Kubernetes suitable for platform automation and AI-assisted operations. <strong>Both humans and software agents can reason about desired state, observed state, and reconciliation through common APIs.</strong></p><p>However, Kubernetes is not a prerequisite for an internal developer platform.</p><p>The same platform principles can be implemented using:</p><ul><li>managed container services</li><li>serverless runtimes</li><li>virtual machines</li><li>platform-as-a-service products</li><li>infrastructure APIs</li><li>alternative schedulers</li><li>combinations of managed cloud services.</li></ul><p>Kubernetes becomes more compelling when the need for common definitions arise when traversing the SDLC with the intention to automate it. The library of terms used in the Kubernetes ecosystem is community approved, consistent and inline with verbal expectations — and one does not want to reinvent that.</p><p>The relevant architectural question is therefore not whether Kubernetes is intrinsically superior. It is whether its operational model provides sufficient value to justify its complexity in the organisation’s context.</p><h3>Behind the Scenes</h3><p>From the developer’s perspective, a mature platform should present a relatively simple interface: create a service, request an environment, deploy a change, inspect a failure, or grant access. Behind that interface, however, the platform coordinates a much larger set of systems and policies.</p><p>The <strong>control plane</strong> acts as the organisational layer that connects developer intent with infrastructure execution. It receives a request, determines which policies and platform standards apply, translates the request into changes across the underlying systems, and then observes whether the desired outcome has been achieved.</p><p>A typical platform workflow may involve several steps:</p><ol><li>A developer or AI agent submits an intent through a portal, command-line interface, API, or repository change.</li><li>The platform identifies the user, team, application, environment, and applicable policies.</li><li>It validates whether the requested action is permitted.</li><li>It generates or updates the required configuration.</li><li>It invokes the appropriate infrastructure, delivery, security, and observability systems.</li><li>It monitors execution and reconciles the actual state with the requested state.</li><li>It records the decision, change, ownership, cost, and operational outcome.</li></ol><p>The developer therefore interacts with a platform capability, while the control plane coordinates the implementation details.</p><h3>Example 1: Authentication, Authorisation, and Role-Based Access Control</h3><p>Consider a developer requesting access to production logs for a payment service.</p><p>Without a platform, this request may involve a support ticket, a manually updated cloud role, changes to Kubernetes permissions, and separate configuration in the logging system. Each step may be performed by a different team, and the resulting access may remain in place longer than intended.</p><p>Through a mature platform, the developer requests the capability rather than the individual permissions:</p><blockquote><em>Grant me read-only access to production logs for the payments service for the next four hours.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K73YNHwX8gwk5AYm.png" /><figcaption>Authentication Flow with RBAC and IAM</figcaption></figure><h3>Example 2: CI/CD and Production Delivery</h3><p>A similar pattern applies when a developer deploys a code change.</p><p>The visible action may be as simple as merging a pull request or selecting “Deploy to staging.” Behind the scenes, the platform may coordinate source control, build automation, security checks, artifact management, GitOps, progressive delivery, and observability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CVHnJY7fGD--MQMe.png" /><figcaption>Artefact delivery flows with Observability</figcaption></figure><h3>AI as an Amplifier of Platform Maturity</h3><p><strong>AI does not remove the requirement for a coherent platform. It depends on one.</strong></p><p>An AI assistant operating across inconsistent scripts, undocumented processes, fragmented permissions, and incomplete telemetry has no reliable model of the organisation’s engineering system. It may generate plausible changes without being able to determine whether those changes are safe, approved, or compatible with production requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ekWjK1iGv_MAAx5-.png" /><figcaption>Developers and AI agents interact with a shared control plane that connects software creation with governed production operations.</figcaption></figure><p>AI becomes more dependable when the platform provides:</p><ul><li>stable APIs</li><li>declarative resource models</li><li>approved templates</li><li>policy definitions</li><li>ownership metadata</li><li>environment state</li><li>change histories</li><li>observability data</li><li>testable workflows.</li></ul><p>Consequently, <strong>the contribution of AI changes with platform maturity.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3I6jHMNmsjarLPjx.png" /><figcaption>Tool choices for Platform Engineering</figcaption></figure><p>At Level 1, AI primarily assists individual engineers. At Level 2, it generates and maintains repeatable automation. At Level 3, it translates developer intent into approved self-service workflows. At Level 4, it interprets governance, reliability, cost, and production behaviour. At Level 5, it helps identify systemic friction and opportunities to improve the engineering organisation.</p><p>AI should therefore be understood as an amplifier. In a mature platform, it can increase access to well-defined capabilities. In an immature environment, it may automate inconsistency and accelerate the creation of operational debt.</p><h3>Conclusion</h3><p>Platform engineering should not be evaluated by the existence of Kubernetes, a developer portal, a service catalogue, or automated deployment pipelines. These are implementation components rather than indicators of organisational maturity.</p><p>A mature platform enables teams to deliver software through repeatable, self-service, governed, observable, and economically sustainable workflows. It aligns developer experience with production operations and evolves through continuous measurement and feedback.</p><p><strong>Artificial intelligence</strong> can extend this model across requirements, architecture, implementation, testing, deployment, operations, cost management, and governance. Its effectiveness <strong>depends on the quality of the platform context within which it operates. </strong>Without structured workflows and controls, AI can increase the speed at which inconsistency is produced. With a mature platform, it can make organisational capabilities easier to discover, understand, and use.</p><p><strong>Kubernetes provides one useful substrate for this model</strong>, particularly where scale, portability, extensibility, private deployment, or workload diversity justify it. It should nevertheless remain an implementation choice rather than the definition of the platform.</p><p>The final measure of platform maturity is whether engineering teams can repeatedly <strong>move from intent to production </strong>with appropriate autonomy, consistency, governance, reliability, and cost awareness — and whether they prefer the platform-supported path when doing so.</p><p>And that’s what we are building through at Agumbe.</p><h3>About Agumbe</h3><p><strong>Agumbe</strong> is an AI-native developer platform designed to connect developer experience, agent experience, and production operations through a common control plane. The platform helps engineering teams standardise how applications and AI workloads are created, governed, deployed, observed, and operated across environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b93a948f592c" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/building-an-ai-native-developer-platform-b93a948f592c">Building an AI Native Developer Platform</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Developer Abstractions that Accelerate (Part 7)]]></title>
            <link>https://itnext.io/from-prototype-to-production-developer-abstractions-that-accelerate-part-7-548fed473201?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/548fed473201</guid>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[control-plane]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-control-plane]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Wed, 13 May 2026 02:42:47 GMT</pubDate>
            <atom:updated>2026-05-15T07:24:38.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DgLUHwsMEBVL735axJnpaQ.png" /></figure><p>By this stage, the system has all the necessary layers in place. Requests are <strong>validated</strong> before they leave, <strong>routed intelligently,</strong> executed within defined <strong>cost</strong> boundaries, adapted across <strong>environments</strong>, and fully <strong>observable</strong>. Each of these capabilities addresses a specific aspect of running AI systems in production, but together they introduce a level of complexity that can be difficult for teams to work with directly.</p><h3>Why do developers need abstractions?</h3><p>The challenge shifts from building the system to making it usable. Without clear abstractions, developers are required to understand and manage multiple concerns simultaneously — policies, routing logic, cost constraints, and environment configurations. This often leads to <strong>duplication of effort</strong>, <strong>inconsistencies</strong> across services, and an <strong>increasing reliance on implicit knowledge</strong> rather than shared structure. It’s a <strong>cognitive overload.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qpCXw7oiqhRrrwn1.png" /></figure><p>Abstractions address this by introducing <strong>a unified interface</strong> through which <strong>developers interact</strong> with the system. Instead of handling each layer independently, requests flow through a single entry point that applies guardrails, selects models, enforces limits, and captures observability by default. This <strong>simplifies integration while preserving control</strong>, allowing teams to focus on building features rather than orchestrating infrastructure.</p><p>Over time, these abstractions extend beyond APIs into structured configurations, reusable templates, and shared workflows. Policies become explicit and versioned, routing strategies can be defined once and applied consistently, and common patterns are reused across teams. This reduces fragmentation and makes system behaviour predictable, even as complexity increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KggYcwB6tUYBn-LA.png" /></figure><p>What emerges is a set of tools, and a platform that balances flexibility with consistency. Developers can move quickly without bypassing safeguards, and teams can scale systems without introducing instability.</p><h3>The Need for an AI Control Plane</h3><p>The control plane enforces rules, enabling a way of building that remains reliable as systems evolve.</p><p>Looking across the layers — <strong>guardrails</strong>, <strong>routing</strong>, <strong>cost</strong> control, <strong>environment</strong> awareness, and <strong>observability</strong> — it becomes clear that the real value lies in how these capabilities are brought together and made accessible. This is what allows AI systems to transition from isolated prototypes to structured, production-ready systems that can be trusted to operate at scale.</p><h3>Shared Platform Capabilities</h3><p>Over time, the control plane becomes more than a gateway. It evolves into a shared platform capability used across teams and products.</p><p>New applications inherit observability automatically. Environment-aware policies are applied consistently. Routing strategies become reusable. Audit trails exist by default. Instead of each team solving operational concerns independently, the platform provides these capabilities as standardized building blocks.</p><p>This creates a significant shift in how AI systems are developed. Teams spend less time assembling infrastructure layers and more time focusing on product behaviour, user workflows, and business logic.</p><h3>Reducing Operational Fragmentation</h3><p>One of the less visible challenges in production AI systems is operational fragmentation. Different teams often introduce different SDKs, different routing strategies, different observability tools, and different governance models. Over time, this creates systems that behave inconsistently despite serving similar purposes.</p><p><strong>Developer abstractions reduce this fragmentation</strong> by creating a common operational language. Requests follow similar patterns regardless of the application. Policies are enforced consistently. Observability data becomes comparable across teams and environments.</p><p>Consistency at this layer is what allows organizations to scale AI adoption without losing operational control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f9GkSka5xtV-vKD6.png" /></figure><h3>The Shift from Integrations to Systems</h3><p>Early AI adoption focused on integrating models into applications. Production AI shifts the focus toward operating systems of behaviour — where governance, routing, observability, and cost management become shared infrastructure concerns rather than isolated implementation details.</p><h3>Series Summary</h3><p>This series explored the operational layers required to <strong>move AI systems from isolated prototypes to reliable production infrastructure</strong>.</p><p>We examined how <strong>guardrails</strong> define what is allowed, how <strong>routing</strong> determines where requests should go, how <strong>cost control</strong> keeps systems sustainable at scale, how <strong>environment-aware behaviour</strong> introduces operational context, and how <strong>observability and auditability</strong> make AI systems understandable over time.</p><p>The final layer focused on <strong>developer abstractions</strong> and the emergence of the <strong>AI Control Plane</strong> — a shared operational layer that brings governance, routing, cost management, environment policies, and observability together into a consistent system that teams can build on reliably.</p><p>Together, these layers represent a shift from simply integrating models into applications toward <strong>building structured, governable, and scalable AI systems</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=548fed473201" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-developer-abstractions-that-accelerate-part-7-548fed473201">From Prototype to Production — Developer Abstractions that Accelerate (Part 7)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Observability and Auditability (Part 6)]]></title>
            <link>https://itnext.io/from-prototype-to-production-observability-and-auditability-part-6-4e8a7b8f6e96?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e8a7b8f6e96</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-gateway]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Mon, 11 May 2026 05:40:58 GMT</pubDate>
            <atom:updated>2026-05-13T07:52:54.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbCjCQk2kMIIydPmiVBArw.png" /></figure><p>By this stage, requests are <strong>validated</strong> before they leave, <strong>routed intelligently</strong>, and executed within defined <strong>cost</strong> and <strong>environment</strong> boundaries. From the outside, it appears stable and predictable.</p><p>Can the <strong>system be understood</strong> though?</p><h3>Observability</h3><p>AI systems introduce a layer of opacity that is difficult to ignore in production. A single request passes through multiple stages — context assembly, policy evaluation, routing decisions, model execution — and each of these stages influences the final outcome. When something behaves unexpectedly, it is rarely obvious where the deviation occurred.</p><p>Traditional logging provides fragments of this journey, but not the full picture. What teams need is visibility at the level of the request itself — an <strong>end-to-end view</strong> that captures <strong>how a request moved through the system</strong>, what decisions were made along the way, and how those decisions affected the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDaor91qkjV7sqOr.png" /></figure><p>A complete trace makes it possible to answer practical questions. It shows <strong>which user triggered the request, what context was included, which policies were applied, which model was selected, how long the request took, and what it cos</strong>t. More importantly, it reveals how each step contributed to the final outcome.</p><p>This level of visibility changes how systems are debugged. Instead of reconstructing events from scattered logs, teams can directly observe system behaviour. Issues become easier to isolate, and fixes become more precise.</p><p>As systems scale, individual traces are not enough. Patterns across requests begin to matter more than isolated incidents. Teams need to understand how the system behaves over time — <strong>where latency is increasing, which models are driving cost</strong>, and <strong>which workflows are failing</strong> more often than expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TYpLAmpGFRgcQbkw.png" /></figure><p>Aggregated insights provide that perspective. They turn <strong>raw data into signals</strong> that can <strong>guide decisions</strong>. With this visibility, teams can identify inefficiencies early, refine routing strategies, and adjust policies based on real usage patterns rather than assumptions.</p><h3>OpenTelemetry</h3><p>Instead of building proprietary instrumentation, the system relies on open standards such as <strong>OpenTelemetry to generate traces, metrics, and logs.</strong> This ensures that every request can be observed in a consistent, vendor-neutral format, without requiring custom integrations. By <strong>subscribing to open standards</strong>, teams can plug into existing observability stacks, correlate AI activity with the rest of their infrastructure, and avoid being locked into a single tool or ecosystem. It also means that as the system evolves, the observability layer remains <strong>interoperable, portable,</strong> and aligned <strong>with the broader engineering ecosystem</strong>.</p><h3>Audit Trace</h3><p>Alongside observability, <strong>auditability</strong> becomes equally important. It provides a structured record of what happened and why. In many cases, teams need to look back and understand past decisions — whether for debugging, compliance, or internal review. A reliable audit trail ensures that these questions can be answered with clarity.</p><h3>Human Review &amp; Operational Feedback Loops</h3><p>Observability also creates the foundation for human oversight. In production systems, not every failure appears as an error. Responses may technically succeed while still being misleading, low quality, or misaligned with business expectations. This makes operational feedback loops essential.</p><p>Teams need mechanisms to <strong>review traces, inspect prompts and outputs, compare model behaviour</strong> across environments, and understand <strong>how policies or routing decisions influenced outcomes</strong> over time. In many cases, the most valuable signal is not whether a request failed, but whether the response quality gradually degraded without being immediately visible.</p><p>This feedback layer turns observability into something more actionable. Instead of only reacting to incidents, teams can continuously improve system behaviour through real-world usage patterns, human review, and operational analysis. Over time, this creates a system that not only becomes easier to debug, but also easier to optimize, govern, and trust.</p><h3>Conclusion</h3><p>Together, observability and auditability make the system understandable. They transform it from something that merely works into something that can be reasoned about, trusted, and improved over time.</p><h3>Next …</h3><p>In the final article, we will look at <strong>developer abstractions</strong>, and how simplifying access to these capabilities enables teams to move faster without losing control.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e8a7b8f6e96" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-observability-and-auditability-part-6-4e8a7b8f6e96">From Prototype to Production — Observability and Auditability (Part 6)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Environment Aware Control (Part 5)]]></title>
            <link>https://itnext.io/from-prototype-to-production-environment-aware-control-part-5-dc5f01133c08?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc5f01133c08</guid>
            <category><![CDATA[llm-gateway]]></category>
            <category><![CDATA[gateway]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Thu, 07 May 2026 05:23:28 GMT</pubDate>
            <atom:updated>2026-05-11T07:17:47.028Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YahrpIx0tYH_mccBlTLH3Q.png" /></figure><p>By this point, most teams have a working system.</p><p>Requests are safe. Routing decisions are in place. Costs are visible and somewhat controlled. On the surface, things look stable. But a different kind of problem begins to emerge — one that is less obvious, but equally important.</p><p><strong>The system behaves the same everywhere.</strong></p><h3>Environments</h3><p>In many setups, development, staging, and production environments are treated as variations of the same configuration. The same models are used, the same policies are applied, and the same limits are enforced. This simplifies implementation, but it introduces a subtle mismatch between intent and behaviour.</p><p>Because environments are not interchangeable.</p><p>They serve different purposes.</p><p><strong>Development environments</strong> are <strong>designed for exploration</strong>. They need flexibility, lower cost constraints, and the ability to experiment freely. Teams may want to test different models, iterate on prompts, or run workflows that would not be acceptable in production.</p><p><strong>Production environments</strong>, on the other hand, <strong>require</strong> <strong>stability</strong>. They need stricter guardrails, tighter cost controls, and predictable behaviour under load. The margin for error is smaller, and the consequences of failure are higher.</p><p><strong>Staging environments</strong> sit somewhere in between, acting as a bridge between experimentation and reliability.</p><p>When all of these environments behave identically, teams are forced into trade-offs.</p><p>If production-level restrictions are applied everywhere, development slows down.<br>If development-level flexibility is allowed everywhere, production becomes fragile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o9hiz4s9K92A_z2-.png" /></figure><p>The solution is not to create entirely separate systems. It is to create a single system that can <strong>adapt its behaviour based on context</strong>.</p><h3>Configs for Environments</h3><p><strong>Environment-aware control</strong> is about introducing that adaptability.</p><p>It allows the <strong>same underlying system to operate differently depending on where a request originates</strong>. Policies can be stricter in production, more relaxed in development, and selectively applied in staging. Routing decisions can favour lower-cost models during experimentation and higher-quality models in production. Rate limits, budgets, and guardrails can all be tuned to match the needs of each environment.</p><p>This approach preserves consistency while enabling flexibility.</p><p>In practice, this requires more than just configuration flags. It requires a structured way to define how behaviour changes across environments.</p><p><strong>Policies</strong> need to be <strong>aware of context</strong>.</p><p>A request that is acceptable in development may not be acceptable in production. A model that is suitable for experimentation may not meet the reliability requirements of a live system. Cost thresholds that are reasonable during testing may be too high at scale.</p><p>These <strong>distinctions need to be encoded</strong> explicitly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ReVsku0Z1E5DCirm.png" /></figure><p>As systems grow, environment-aware control also becomes essential for <strong>collaboration</strong>.</p><p>Different teams often operate at different stages of the lifecycle. One team may be experimenting with new features, while another is maintaining a production system. Without clear boundaries, changes in one environment can unintentionally affect another.</p><p>By isolating behaviour through environment-specific policies, teams can move independently without introducing instability.</p><h3>Progressive Delivery</h3><p>There is also <strong>an important operational benefit</strong>.</p><p>Environment-aware systems make it easier to <strong>promote</strong> changes <strong>safely</strong>.</p><p>A feature can be tested in development, validated in staging, and then promoted to production with confidence that it will behave as expected. The transition is a controlled progression, with policies and constraints evolving along the way.</p><p>This reduces the need for manual intervention and lowers the risk of unexpected outcomes.</p><h3>Conclusion</h3><p>At this point, the system begins to feel cohesive.</p><p><strong>Guardrails</strong> ensure that requests are safe.<br><strong>Routing</strong> ensures that they are handled correctly.<br><strong>Cost control</strong> ensures that they remain sustainable.<br><strong>Environment-aware control</strong> ensures that all of this behaves appropriately across contexts.</p><p>These layers reinforce each other.</p><p>A guardrail policy may vary by environment. Routing decisions may depend on cost constraints. Cost limits may be defined differently for staging and production.</p><p>The system becomes a set of interconnected controls rather than independent features.</p><p>This is where the idea of a <strong>control plane</strong> becomes concrete. As <strong>a unified layer that governs behaviour across the entire lifecycle of an AI system</strong>. Without this layer, teams rely on discipline and coordination.</p><p>With it, they rely on structure. And structure is what allows systems to scale.</p><h3>Next …</h3><p>There is one final piece that completes this picture. Even with all these controls in place, teams still need to understand what is happening inside the system.</p><p>At a high level, as well at the level of individual requests.</p><p>They need to see how decisions are made, how policies are applied, and how behaviour evolves over time. In the next article, we will explore <strong>observability and auditability</strong> — and why <strong>visibility is the foundation for trust</strong> in production AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc5f01133c08" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-environment-aware-control-part-5-dc5f01133c08">From Prototype to Production — Environment Aware Control (Part 5)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Cost Control That Scales (Part 4)]]></title>
            <link>https://itnext.io/from-prototype-to-production-cost-control-that-scales-part-4-33428eabf53b?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/33428eabf53b</guid>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[llm-gateway]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Wed, 06 May 2026 02:17:11 GMT</pubDate>
            <atom:updated>2026-05-06T07:44:01.557Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zia_z7tU0X6gyr0WIyoCw.png" /></figure><p>Cost is rarely the first concern when teams begin working with AI systems. In the <strong>early stages, the focus is on getting something to work</strong> — prototypes, experiments, and initial user flows. The system is small, usage is limited, and costs appear manageable. <strong>There is a natural assumption that cost will grow in proportion to adoption.</strong></p><p>As systems move into production, the structure of AI workloads begins to change.</p><p>A single user request often triggers multiple model calls. Context is added dynamically. Retrieval pipelines expand the amount of data processed. Workflows become layered, and a simple interaction evolves into a sequence of dependent operations.</p><p>Over time, these layers introduce compounding effects, where the cost of a single feature is no longer tied to a single request, but to a chain of interactions behind it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1sEWkDGLoWJPACX3.png" /></figure><h3>Agent (machine) identity</h3><p>This effect becomes more pronounced with the introduction of <strong>agents and automated workflows</strong>. Unlike traditional systems that respond once per user action, agents iterate. They retry, refine, and explore possible paths before returning a result.</p><p>Each of these steps may involve additional calls to models, often with increasing amounts of context. When multiplied across users and use cases, the resulting cost profile is difficult to anticipate using conventional assumptions</p><h3>Trends</h3><p>The challenge is not just the scale of usage, but its <strong>shape</strong>. Cost growth tends to be gradual at first, and then accelerates in ways that are not immediately visible. Small inefficiencies — an overly verbose prompt, an unnecessary retry, an unbounded loop — may seem insignificant in isolation, but they accumulate over time. By the time they are noticeable, they are already embedded in the system’s behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JrzKJ-46hIXjCAq7.png" /></figure><p>This is why cost control cannot be treated as a reporting function alone. <strong>Dashboards</strong> and billing summaries provide visibility into past usage, but they do not influence how the system behaves in real time. By the time a spike is observed, the underlying conditions that caused it have already taken effect.</p><p>What is needed instead is a mechanism for shaping behaviour as it happens.</p><h3>Budgets and Leaderboards</h3><p>Budgeting in this context is about a single top-level number, yet also about defining boundaries across different dimensions of the system. <strong>Usage can vary</strong> significantly <strong>across teams, features, and environments</strong>, and each of these needs to be <strong>governed differently</strong>. A development environment may require flexibility for experimentation, while production systems demand stricter controls. Similarly, internal tools and customer-facing features often have different performance and cost expectations.</p><p>By segmenting budgets along these lines — per user, per feature, per team, and per environment — systems can be designed to <strong>operate within predictable limits</strong>. These boundaries help ensure that no single component can disproportionately affect overall cost, even under unexpected conditions.</p><p>However, boundaries alone are not sufficient without visibility into how they are being used.</p><h3>Telemetry Traces with latency per hop</h3><p>Understanding cost in a meaningful way requires aggregated totals along with tracing usage back to its source — identifying which features generate the most activity, which workflows are inefficient, and where resources are being consumed without corresponding value. This level of insight allows teams to make informed decisions about optimization, rather than relying on assumptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8P_8LErY6WzG5i9I.png" /></figure><p>When visibility improves, behaviour tends to follow. Developers become more conscious of how prompts are structured. Teams begin to evaluate model choices more carefully. Inefficient workflows are identified and refined. Over time, cost management becomes less about restriction and more about feedback — an integral part of how the system evolves.</p><p>In this sense, cost control is closely tied to system design. It reflects how efficiently a system operates, how well it adapts to changing conditions, and how effectively it balances trade-offs between performance and resource usage. Without deliberate control, these trade-offs are left to chance.</p><h3>AI Control Plane for Costs</h3><p>This is where a centralized control layer becomes essential. Cost management cannot be distributed across individual services or left to manual oversight. It needs to be enforced consistently, with policies that apply across the entire system. By <strong>placing cost control within a shared layer</strong>, organizations can ensure that <strong>limits are respected, visibility is maintained</strong>, and usage remains aligned with expectations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RvpYeMjMs96VKlZ0.png" /></figure><p>At this stage, three foundational aspects of production AI systems begin to come together.</p><p><strong>Guardrails</strong> define what is allowed.</p><p><strong>Routing</strong> determines how requests are handled.</p><p><strong>Cost</strong> control ensures that the system remains sustainable as it scales.</p><p>Each of these <strong>addresses a different dimension of the same problem: how to operate reliably in a complex, evolving environment.</strong></p><p>The next layer builds on this foundation by introducing context. Within a request, but across the system as a whole. Because the way a system behaves in development should not mirror how it behaves in production. <strong>Differences in scale, risk, and intent require different configurations and policies.</strong> Managing these variations effectively is what allows teams to move quickly without compromising stability.</p><h3>Frequently Asked Questions</h3><p><strong>Why does cost behave differently in AI systems compared to traditional systems?</strong></p><p>In traditional systems, cost typically scales linearly with usage. In AI systems, a single request can trigger multiple model calls, retrieval steps, and iterative workflows. This creates compounding effects where cost grows based on how requests are structured, not just how many requests are made.</p><p><strong>What makes AI cost hard to predict in production?</strong></p><p>Cost is influenced by dynamic factors such as prompt size, number of model calls, retries, and workflow depth. As systems evolve, these factors interact in ways that are difficult to estimate upfront, especially when agents and automated pipelines are involved.</p><p><strong>What does real-time cost control mean in practice?</strong></p><p>Real-time cost control involves shaping how requests behave as they are executed. This includes enforcing limits, controlling prompt sizes, restricting retries, and applying policies that prevent inefficient workflows before they scale.</p><p><strong>How does observability help with cost control?</strong></p><p>Observability allows teams to trace cost back to its source. Instead of seeing total spend, teams can identify which features, workflows, or requests are driving usage. This makes optimization targeted rather than reactive.</p><p><strong>Why should cost control be centralized?</strong></p><p>Cost control cannot rely on individual services or developer discipline. A centralized control layer ensures that policies are applied consistently across all requests, preventing fragmentation and maintaining predictable system behavior.</p><h3>Next …</h3><p>In the next article, we will look at <strong>environment-aware control</strong>, and how aligning system behaviour with context enables <strong>safe and scalable iteration</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33428eabf53b" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-cost-control-that-scales-part-4-33428eabf53b">From Prototype to Production — Cost Control That Scales (Part 4)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Intelligent Routing for Right Responses (Part 3)]]></title>
            <link>https://itnext.io/from-prototype-to-production-intelligent-routing-for-right-responses-part-3-d20daa09f32e?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d20daa09f32e</guid>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[llm-gateway]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Tue, 05 May 2026 04:54:41 GMT</pubDate>
            <atom:updated>2026-05-05T07:29:27.938Z</atom:updated>
            <content:encoded><![CDATA[<h4>Not every request is the same. Routing layer decides how well the system performs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOlXJ77lfddOfKvj-sZPCg.png" /></figure><p>In the last article, we spoke about guardrails. About <strong>what should and should not leave your system</strong>. But let’s assume for a moment that you’ve solved that problem. Every request is now safe. Every prompt is compliant. Now what? Because safety is only the first layer. Once a request is allowed to leave your system, the next question becomes far more interesting:</p><p><strong>Where should it go?</strong></p><p>In early systems, this decision is straightforward. A model is selected, the request is sent, and the response is returned. As long as the system works, the choice of model rarely receives deeper attention.</p><p>This simplicity does not hold in production.</p><h3>Classifying Expectations</h3><p><strong>Modern AI systems operate with multiple models</strong>, each optimized for different trade-offs. Some models are better at reasoning, others are faster, and some are more cost-efficient under scale. Reliability also varies depending on load and provider conditions.</p><p>As usage grows, model selection begins to influence system behaviour across several dimensions. At scale, that decision is no longer trivial. It becomes a system problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OJGvpc4tciiVI-2b.png" /></figure><h3>Structured Signals</h3><p>A request is text — yes, but it also carries intent, context, and expectations. These signals determine how the request should be handled.</p><p>Even a simple instruction like <em>“Summarize this report”</em> requires interpretation. The system needs to decide whether the task is lightweight or complex, whether speed matters more than depth, and whether constraints such as cost or rate limits should influence execution.</p><p>And that’s before we even consider <strong>user-level policies</strong> or <strong>environment-specific constraints.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/0*UknKDu_cLNpQ4YiW.png" /></figure><p>Most systems begin with simple rules. Requests are mapped to models using fixed conditions, with basic fallback logic in place.</p><p>This approach works under controlled conditions. It breaks under real-world variability.</p><p>Traffic patterns change. Models evolve. Costs fluctuate. Latency varies across providers. <strong>Static rules cannot adapt</strong> to these changes without constant manual updates.</p><p>Over time, routing logic becomes brittle, and inconsistencies begin to appear across services.</p><h3>Adaptive Routing</h3><p>Routing needs to evolve from predefined mappings to real-time decision-making.</p><p>Adaptive routing evaluates each request in real time and <strong>selects the most appropriate model</strong> based on current conditions. The objective is not to identify a single “best” model, but to match each request to the model that fits its requirements — taking into account complexity, latency, cost, availability, and policy constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/0*zpkInnPUbvEb9eOx.png" /></figure><p>This introduces a different way of thinking about routing. Instead of relying on fixed rules, each decision is evaluated in real time by considering multiple factors simultaneously — such as the complexity of the request, latency expectations, cost constraints, model availability, historical performance, and applicable policies — so that <strong>every request is handled</strong> in a way that aligns <strong>with both system conditions and desired outcomes.</strong></p><h3>Intelligent Routing</h3><p>Without intelligent routing, teams tend to oscillate between two extremes. They either overpay for quality by defaulting to the most powerful model … Or they underperform by optimizing for cost too aggressively.</p><p>Neither scales.</p><p><strong>Intelligent Routing</strong> is what balances that trade-off. Continuously. And it has a few immediate effects.</p><p><strong>First</strong>, quality improves. Because each request is matched to the model most likely to succeed.</p><p><strong>Second</strong>, cost becomes controllable. Because not every request is sent to the most expensive model.</p><p><strong>Third</strong>, reliability increases. Because fallback and failover are built into the routing layer itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*Vsww_d_ZtnsIxscV.png" /></figure><h3>Handle Multi Model sprawl</h3><p>The introduction of multiple models increases capability, but also introduces <strong>operational complexity.</strong></p><p>Different teams may prefer different models. Environments may require different constraints. Providers evolve independently, with changing pricing, latency, and availability characteristics.</p><p>Over time, this creates fragmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SSEueDHsrxp-j3ho.png" /></figure><h3>AI Control Plane</h3><p>And this is where the <strong>control plane starts to show its real value</strong>. Because routing cannot live inside individual services. It cannot be duplicated across teams. It cannot be inconsistent.</p><p>It has to be <strong>centralized</strong>.</p><p>A control plane allows routing logic to exist as a shared system capability; as a consistent layer that every request passes through. That consistency is what allows systems to scale. With Traffic. At Complexity.</p><h3>Frequently Asked Questions</h3><p><strong>Why choosing a single model isn’t enough for production systems?</strong></p><p>Different models optimize for different outcomes — latency, cost, reasoning depth, and reliability. A single model cannot consistently satisfy all requirements as usage scales and workloads diversify. Production systems need flexibility to adapt these trade-offs per request.</p><p>Another important consideration is <strong>vendor-lock in</strong>. Enterprises wouldn’t want all their workflows tied to a single provider.</p><p><strong>What is the difference between routing and fallback logic?</strong></p><p><strong>Fallback logic is reactive</strong> — it handles failures after a request has already been sent to a model. <strong>Routing is proactive</strong> — it determines where the request should go before execution, based on constraints such as cost, latency, and availability.</p><p><strong>How does routing help control cost?</strong></p><p>Routing allows simpler requests to be handled by lower-cost models, while reserving more capable (and expensive) models for complex tasks. This prevents unnecessary use of high-cost models and improves overall efficiency.</p><p><strong>What challenges surface when using multiple models?</strong></p><p>Multi-model systems introduce differences in performance, cost, availability, and behavior. Without structured routing, these differences lead to inconsistent outcomes and operational complexity across services.</p><h3>Next …</h3><p>At this point, we’ve covered two fundamental layers:</p><p><strong>Guardrails</strong> — what is allowed</p><p><strong>Routing</strong> — where requests go</p><p>But there is another dimension that quietly grows alongside both. One that most teams only notice when it is too late.</p><p><strong>Cost.</strong></p><p>How much you spend. How that spend behaves over time. In the next article, we’ll look at <strong>cost control</strong> — and why it is one of the most underestimated challenges in production AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d20daa09f32e" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-intelligent-routing-for-right-responses-part-3-d20daa09f32e">From Prototype to Production — Intelligent Routing for Right Responses (Part 3)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Strong Guardrails for Safe AI Systems (Part 2)]]></title>
            <link>https://itnext.io/from-prototype-to-production-strong-guardrails-for-safe-ai-systems-part-2-a686e197f73b?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a686e197f73b</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-guardrails]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Sun, 03 May 2026 06:07:01 GMT</pubDate>
            <atom:updated>2026-05-03T11:55:37.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnIgisgAShotNGOv7Uru4w.png" /></figure><p>If you strip away all the complexity around AI systems, everything begins with a single unit of work.</p><p><strong>A request.</strong></p><p>A prompt leaves your system and goes to a model.</p><p>It sounds simple.</p><p>But in production, that “prompt” is rarely just a line of text typed by a user. It is a composite of multiple layers: <strong>user input, system context, memory, internal data, and business logic stitched together dynamically</strong>. By the time it is sent to a model, it often carries far more than what was originally intended.</p><p>Here we see the first real risk emerge</p><h3>Start of Build Cycles</h3><p>In early-stage systems, this risk is invisible.</p><p>Teams are focused on making things work. They optimize prompts, experiment with models, and iterate quickly. Data flows freely because speed matters more than structure. The assumption is simple: the model is a tool, and the prompt is just input.</p><p>But once the system starts interacting with real users and real data, that assumption breaks down.</p><p>Because prompts can carry sensitive information.</p><p>Personally <strong>identifiable information. Internal knowledge. API keys</strong>. Business logic that was never meant to leave the system boundary.</p><p>And unlike traditional APIs, LLMs are probabilistic. You don’t always know how the model will respond, what it might infer, or how that information might be transformed downstream.</p><p>The moment a request leaves your system, control is reduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vwHMJkoRax-n1Hyh.png" /></figure><h3>Reactive Fixes</h3><p>Most teams react to this problem <strong>after</strong> the fact.</p><p>A piece of sensitive data shows up where it shouldn’t. A prompt reveals more context than expected. A compliance concern is raised. At that point, fixes are introduced — filters, regex checks, maybe a validation layer added before the request is sent.</p><p>But these are patches.</p><p>They are not guardrails.</p><p><strong>Guardrails</strong> are about defining <strong>what is allowed and what is not allowed</strong> — systematically, consistently, and across every request.</p><p>This is a subtle but important distinction.</p><p>Blocking is reactive.</p><p><strong>Governance is proactive.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mL68zO2F3Xi6tzwR.png" /></figure><p>A well-designed guardrail system doesn’t just scan for PII or secrets. It understands the structure of requests. It enforces policies that define how data should flow. It ensures that prompts conform to expected patterns before they are ever sent out.</p><p>And most importantly, it does this uniformly across the entire system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G269ygAOLDs9X-lI.png" /></figure><h3>Reinventing over and …</h3><p>Without this layer, every team ends up solving the same problem in isolation.</p><p>One service introduces basic filtering. Another adds a different validation approach. A third ignores the problem entirely because it hasn’t surfaced yet. Over time, the system becomes inconsistent — because of fragmentation, even if the the intent was alright.</p><p>Fragmentation compounds risk. And the system is no longer predictable.</p><p>The challenge becomes even more pronounced when you consider <strong>how requests are constructed in modern AI systems.</strong></p><p>They are rarely static.</p><p>Prompts are <strong>assembled dynamically</strong> using templates, context injection, retrieval pipelines, and chained calls. What starts as a safe input can evolve into something unsafe through composition. A retrieval step might introduce sensitive data. A memory layer might add context that was never intended for that specific interaction.</p><p>Guardrails need to operate at that level of dynamism.</p><p>At the edges, as well as throughout the lifecycle of a request.</p><blockquote><em>“It’s taken care of in the codebase …” Developer</em></blockquote><p>This is why <strong>guardrails cannot live inside application code alone. </strong>They need to exist <strong>as a dedicated layer</strong> — a control point through which every request passes. A place where <strong>policies are enforced</strong> consistently. Where <strong>checks are not optional.</strong> Where <strong>behaviour is defined centrally</strong> rather than scattered across services.</p><p>This is where the control plane begins to take shape in a tangible way.</p><p>We now see it as an <strong>operational necessity.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*uBWIxg63jwpFgx-n.png" /></figure><h3>AI Control Plane with Guardrails</h3><p>A mature guardrail system typically evolves along a few dimensions.</p><p><strong>First, detection</strong>. The ability to identify sensitive patterns — PII, secrets, compliance risks — in real time.</p><p><strong>Second, enforcement</strong>. The ability to block, redact, or transform requests based on defined policies.</p><p><strong>Third, policy definition</strong>. The ability to express rules in a way that is both flexible and maintainable — often as code or structured configuration.</p><p><strong>And finally, observability</strong>. The ability to understand what is being blocked, why it is being blocked, and how often these patterns occur. Without observability, even the best guardrails operate in the dark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*3iPf2QGtn3H4t22z.png" /></figure><h3>Cultural Impact</h3><p>There is also an important cultural shift that comes with introducing guardrails. It forces teams to think explicitly about data boundaries.</p><ul><li><strong>What should leave the system?</strong></li><li><strong>What should remain internal?</strong></li><li><strong>What transformations are acceptable?</strong></li></ul><p>These questions are often implicit in early systems. Guardrails make them explicit. And in doing so, they create a foundation for trust.</p><h3>Frequently Asked Questions</h3><p><strong>Why are guardrails necessary if we already validate inputs in code?</strong></p><p>Validation in application code is usually fragmented and context-specific. As systems grow, different services implement different checks, often inconsistently. Guardrails introduce a centralized layer where policies are enforced uniformly across every request, ensuring that sensitive data handling is not left to individual implementations.</p><p><strong>What kind of data risks exist in LLM-based systems?</strong></p><p>Requests to LLMs often include more than just user input. They can contain system prompts, retrieved context, memory, and internal data. This combination can unintentionally expose PII, proprietary information, API keys, or business logic. Since LLMs are probabilistic, the way this data is used or inferred is not always predictable.</p><p><strong>Where should guardrails be implemented — in the application or as a separate layer?</strong></p><p>Guardrails are most effective when implemented as a dedicated layer outside individual services. Embedding them inside application code leads to duplication and inconsistency. A centralized control point ensures that all requests are evaluated against the same policies before reaching any model.</p><p><strong>How do guardrails impact developer velocity?</strong></p><p>Initially, guardrails introduce structure where there was flexibility. Over time, they reduce friction by standardizing how data is handled across the system. Developers no longer need to reimplement safety logic, and teams can move faster with a shared understanding of what is allowed.</p><h3>Next …</h3><p>As systems grow, guardrails become the first line of defence. But they are not the only layer that matters.</p><p>Even if every request is safe, the system still needs to <strong>behave correctly under load</strong>. It needs to <strong>handle failures gracefully</strong>. It needs to <strong>make decisions about where requests should go</strong> and <strong>how they should be executed.</strong></p><p>The next set of challenges begins here.</p><p>We covered safety, <strong>next we cover behaviour. </strong>Because in production, the system needs to be <strong>reliable and correct</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a686e197f73b" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-strong-guardrails-for-safe-ai-systems-part-2-a686e197f73b">From Prototype to Production — Strong Guardrails for Safe AI Systems (Part 2)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prototype to Production — Where AI Systems Actually Break (Part 1)]]></title>
            <link>https://itnext.io/from-prototype-to-production-where-ai-systems-actually-break-part-1-d07c1aa76fa2?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d07c1aa76fa2</guid>
            <category><![CDATA[llm-gateway]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <category><![CDATA[safety]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-production]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Fri, 01 May 2026 07:13:34 GMT</pubDate>
            <atom:updated>2026-05-01T08:44:50.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yPDOJ8q-SCR5m7kdk5byw.png" /></figure><h3>Navigating the quiet shift in how software is being built.</h3><p>The hardest part of starting a software project has always been starting up. Frontends took a lot of explaining and wire framing and we got used to wire-framing tools and long explanatory meetings. Backend scaffolding used to be slow because building a production-grade backend wasn’t just about writing APIs — it required a full team to handle resiliency, observability, database design, and adherence to core principles like statelessness and concurrency control. Infrastructure setup took even longer. Entire teams existed to make sure systems could run reliably once they were built.</p><p>That constraint is disappearing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/0*NpI-UTQcWxMgdNkQ.png" /></figure><p>Today, one can go from idea to working prototype in hours. Tools like <strong>Figma Make</strong>, <strong>Replit</strong>, and others have made it almost trivial to generate interfaces and wire up basic logic. Even complex workflows can be assembled quickly. The barrier to <em>building</em> has collapsed.</p><p>Yet — something interesting happens right after the prototype works.</p><p>We try to run it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/0*vhBmA43etaoDRcva.png" /></figure><p>The application now needs a stable backend, a database, and a place to live. It needs <strong>environments</strong> — development, staging, feature branches — so you can iterate safely without breaking production. It needs observability, logging, cost control, and a way to handle failure … amongst others.</p><p>In other words — it needs <strong>a system</strong>.</p><p>And that’s where teams <strong>realize</strong> that while <strong>building has become easy, running has not.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*xFatYjU0U0OsVxRS.png" /></figure><h3>Human and Machine (non-human) Identities</h3><p>At the same time, <strong>another shift</strong> is unfolding beneath the surface — one that is even more fundamental.</p><p>We are no longer building systems where humans are the only <strong>actors</strong>.</p><p>We are building systems where machines talk to machines.</p><p>Agents invoke models. Workflows trigger other workflows. Integrations continuously generate requests. A single user action can cascade into dozens of LLM calls across multiple services. What used to be a simple request-response pattern is now a <strong>network of autonomous interactions</strong>.</p><p>In many organizations, the ratio of human to machine identities is already skewed — sometimes 1:80, in more advanced systems even 1:500.</p><p>This changes how we build our systems.</p><p>Because when machines are generating traffic, the scale is continuous. Most decisions are automated, not supervised. And failures propagate quickly.</p><p>A bad request is no longer just a bad request. It becomes a pattern. A loop. Possibly A multiplier.</p><p>And soon, the system is no longer behaving the way you expect.</p><p>Most teams encounter this gradually.</p><p>At first, it looks like small issues. A prompt leaking more context than intended. A sudden spike in costs. A provider rate limit being hit. A fallback that doesn’t quite behave correctly. A missing log when something goes wrong.</p><p>Individually, these are manageable.</p><p>Collectively, they are not.</p><p>Because they are not independent problems.</p><p>They are symptoms of the same underlying gap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/0*v4-fSaOGuZIAUmSh.png" /></figure><h3>The Need for an AI Control Plane</h3><p>That gap is the absence of a <strong>control layer</strong>.</p><p>A layer that sits <strong>between your applications and the models they depend on</strong>. A layer that governs how requests are formed, where they go, what they are allowed to contain, how they behave under stress, and how they are observed once they leave your system.</p><p>Without that layer, every team ends up rebuilding the same pieces.</p><p>Guardrails get implemented in one service but not another. Cost controls are added reactively after a spike. Routing logic becomes tightly coupled to business code. Observability is fragmented across logs, dashboards, and guesswork.</p><p>And over time, the system becomes harder to reason about.</p><p>The team&#39;s efforts start as a collection of features that slowly reveal themselves — and find missing links.</p><p>It needs guardrails.</p><p>It needs routing.</p><p>It needs cost control.</p><p>So, we need <strong>a system of systems</strong> — one that needs to be <strong>governed consistently across teams, environments, and models.</strong></p><p>This is where <strong>the idea of a control plane emerges.</strong></p><p>It is a product category — but not that you decide to build upfront. The need surfaces when you follow the problems far enough.</p><p>A control plane makes requests <strong>predictable</strong>.</p><p>It ensures that <strong>every interaction with an LLM</strong> — whether triggered by a user, an agent, or another system — <strong>behaves in a way that is safe, reliable, and observable.</strong></p><p>It differentiates experimentation from production. In experimentation, failure is acceptable. In production, failure has consequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*E9vInsbA-Rf_D7db.png" /></figure><p>This series is an attempt to break down that system.</p><p>From a systems perspective.</p><h3>Next …</h3><p>Over the next few articles, we’ll look at the <strong>layers that actually matter when you move from prototype to production</strong>.</p><p>Like.</p><p><strong>Guardrails</strong> — what should and should not leave your system</p><p><strong>Runtime control</strong> — how systems behave under real conditions</p><p><strong>Cost</strong> — how usage scales, often invisibly</p><p><strong>Environments</strong> — how behavior changes across dev, staging, and prod</p><p><strong>Observability</strong> — how you understand what is happening</p><p><strong>Developer abstractions</strong> — how teams actually build on top of all this</p><p>Each of these appears simple in isolation.</p><p>Together, they define whether your AI system works — or quietly breaks over time.</p><p>We’ve made building AI applications easy. We haven’t yet made running them simple.</p><p>That is the gap.</p><p>And that is where <strong>the real work begins</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d07c1aa76fa2" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/from-prototype-to-production-where-ai-systems-actually-break-part-1-d07c1aa76fa2">From Prototype to Production — Where AI Systems Actually Break (Part 1)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reactive Manifesto — and why we commit to it]]></title>
            <link>https://itnext.io/reactive-manifesto-and-why-we-commit-to-it-3eabdf7a386a?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eabdf7a386a</guid>
            <category><![CDATA[message-driven]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[12-factor-app]]></category>
            <category><![CDATA[reactive-manifesto]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Tue, 21 May 2024 02:49:34 GMT</pubDate>
            <atom:updated>2024-05-21T07:17:22.644Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reactive Manifesto — and why we commit to it</h3><p>The technological landscape has witnessed a revolutionary shift over the past decade, with a significant emphasis on building responsive, resilient, elastic, and message-driven systems. These system properties form the core principles of the <a href="https://www.reactivemanifesto.org/"><strong>Reactive Manifesto</strong></a>, a strategic guideline for <strong>building</strong> <strong>systems that can cope with the modern world’s dynamic, <em>real-time</em>, and distributed nature</strong>.</p><p>In this blog post, we’ll dive into the core tenets of the <a href="https://www.reactivemanifesto.org/">Reactive Manifesto</a>, and explore why <strong>Event-Driven Architecture</strong> and the use of Message Queues/Message Buses are integral to building <strong>Reactive Systems</strong>.</p><h3>Understanding the Reactive Manifesto</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cNhy4M0oy9KuUaCHGQbHzA.png" /></figure><h4><strong>What is a Reactive System?</strong></h4><p>A reactive system is a distributed system with (to quote the manifesto)</p><blockquote>patterns for building software that look the same. These systems are more robust, more resilient, more flexible and better positioned to meet modern demands.</blockquote><p>The Reactive Manifesto, first published in 2013, has since become a cornerstone for developers and architects designing <strong>cloud-native, distributed systems</strong>. It lays out key principles:</p><h4><strong>Responsiveness</strong></h4><p>A reactive system is responsive. Reactive Systems need to consistently respond in a timely manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zy-F-l3df8SP53kJgJqgSg.png" /></figure><p>eg: Consider a ride-sharing app. When a user requests a ride, the system quickly shows available drivers and estimated pick-up times. The response is not just limited to ride availability — and could also include greetings, recommendations and offers among other responses. This responsiveness is crucial for a good user experience.</p><h4><strong>Resilience</strong></h4><p>The system must remain responsive, even in the face of failure.</p><p>Critical to note the distinction between a monolith and collection of microservices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XP0_vOQiUoWqCRT-_bCzRA.png" /></figure><p>Modern application derive nimbleness from fault-tolerant, stateless collections of microservices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LxojE82a78g78iDHwMUVmA.png" /></figure><p>eg: In an e-commerce platform, if the recommendation service fails, the system ensures that the core functionality of browsing items and making purchases remains unaffected. This isolation of services enhances resilience.</p><h4><strong>Elasticity</strong></h4><p>Reactive Systems should stay responsive under fluctuating workload, and resources within the system should scale up or down seamlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgW087zkTj561SwjoXvRqg.png" /></figure><p>eg: During Black Friday sales, online retail platforms experience a massive surge in traffic. An elastic system automatically scales up resources (like servers) to handle this increase and scales down as demand wanes.</p><h4><strong>Message Driven</strong></h4><p>The core idea that underpins everything else is that <strong>Reactive Systems are Message Driven</strong>. This is the <strong>foundation</strong> for all the other properties. Reactive Systems rely on <em>asynchronous</em> message-passing to establish a boundary between components, ensuring loose coupling, isolation, and location transparency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1p0oYEwRVnRdIzbEC86UQ.png" /></figure><p>eg: In a microservices architecture, different services communicate using messages. This approach decouples services (like user authentication, movie streaming, recommendations) and allows them to be developed, deployed, and scaled independently.</p><h4>Extensible</h4><p>A system is extensible if it can be easily modified to include additional features or to support more use cases.</p><h4>Maintainable</h4><p>Maintainability refers to how easily a software system can be understood, repaired, or enhanced.</p><h3>Why do we commit to the Reactive Manifesto?</h3><p>Committing to the Reactive Manifesto helps us to design and deploy systems that are robust, flexible, and responsive.</p><ol><li><strong>Enhanced system longevity</strong>: Adhering to reactive principles results in a system design that is resilient and able to adapt to changes and growth over time, ensuring longevity and reduced maintenance.</li><li><strong>Introduction to Microservices and Container Orchestration:</strong> The reactive approach encourages the use of microservices architecture, which enhances scalability and fault isolation. Leveraging Kubernetes for container orchestration further streamlines deployment, scaling, and management of containerized applications, promoting operational efficiency.</li><li><strong>Escape from inherently buggy systems:</strong> Reactive systems minimize stateful interactions, which often introduce bugs and complexity. By enabling stateless design wherever possible, reactive systems reduce dependencies and interaction complexities, resulting in cleaner and more maintainable code.</li><li><strong>Improved Responsiveness</strong>: Reactive systems are designed to respond in a timely manner. This responsiveness ensures that systems remain functional and user-friendly under varying load conditions, enhancing user satisfaction and engagement.</li><li><strong>Resource efficiency:</strong> By being responsive and elastic, reactive systems can manage resource utilization more effectively, dynamically scaling up or down based on demand. This not only optimizes resource use but also helps in reducing operational costs.</li><li><strong>Asynchronous Non-blocking Communication:</strong> The manifesto promotes non-blocking communication, which improves concurrency and utilizes system resources more effectively, thus boosting performance.</li><li><strong>Greater Fault Tolerance:</strong> The decentralized and isolated nature of reactive systems makes them less prone to failures. Even if one part of the system fails, the rest can continue to function, thereby ensuring high availability and reliability.</li></ol><h3>Conclusion</h3><blockquote>time to apply these design principles consciously from the start instead of rediscovering them each time.</blockquote><p>In essence, the Reactive Manifesto provides a roadmap to creating systems capable of dealing with today’s demanding technology requirements. The backbone of Reactive Systems is the Event-Driven Architecture supported by Message Queues and Message Buses.</p><p>If you are keen to explore how to get started building robust systems using <strong>Reactive Manifesto</strong> principles with <strong>12-factor</strong> guardrails — do give <a href="https://medium.com/itnext/12-factor-microservice-applications-on-kubernetes-db913008b018">this series</a> a read.</p><p>Cheers! and Happy Code Hunting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eabdf7a386a" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/reactive-manifesto-and-why-we-commit-to-it-3eabdf7a386a">Reactive Manifesto — and why we commit to it</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logging and Admin Processes — The Final Pillars of 12-factor Microservices]]></title>
            <link>https://itnext.io/logging-and-admin-processes-the-final-pillars-of-12-factor-microservices-376a6dfa268d?source=rss-7827c154fa0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/376a6dfa268d</guid>
            <category><![CDATA[12-factor]]></category>
            <category><![CDATA[modern-data-platform]]></category>
            <category><![CDATA[administration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <dc:creator><![CDATA[Santosh Pai]]></dc:creator>
            <pubDate>Wed, 15 May 2024 05:19:49 GMT</pubDate>
            <atom:updated>2024-05-15T07:07:45.521Z</atom:updated>
            <content:encoded><![CDATA[<h3>Logging and Admin Processes — The Final Pillars of 12-factor Microservices</h3><p>Welcome back to the finale of our deep-dive series into <a href="https://12factor.net/">12-factor</a> microservices! Having navigated the intricate terrains of <a href="https://medium.com/itnext/dev-prod-parity-for-12-factor-microservices-68c55e935b87">Dev/Prod Parity</a>, <a href="https://medium.com/itnext/concurrency-and-disposability-of-12-factor-microservices-65db2cf29c17">Concurrency, and Disposability</a>, we arrive at the last two factors: <strong>Logging</strong> and <strong>Admin Processes</strong>. As always, we’ll explore how our <strong>Cloud Native</strong> toolset seamlessly integrates these principles.</p><h3>The Eleventh Factor — Logging</h3><p>The eleventh factor asks us to …</p><blockquote>Treat logs as event streams.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I4vXMG8XnFI3RCS9" /></figure><h4>Why Logging Matters</h4><p>In a distributed system, logs are the critical for debugging and monitoring. Properly managed logs can provide invaluable insights into the system’s behavior, help diagnose issues, and even feed real-time analytics dashboards.</p><h4>Structured Logging: Beyond Text Files</h4><p>In a Kubernetes-based microservices architecture, logs are treated as structured data, collected and stored in a centralized logging system like ELK Stack or Grafana Loki.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: logging-config<br>data:<br>  logback.xml: |-<br>    &lt;configuration&gt;<br>      &lt;!-- Your logging config --&gt;<br>    &lt;/configuration&gt;</pre><h4>Observability: More Than Just Logs</h4><p>Logs alone aren’t enough. We use tools like Prometheus and Grafana to measure metrics and create visualizations. This observability stack is crucial for understanding system health and is a staple in our Kubernetes deployments.</p><h3>The Twelfth Factor — Admin Processes</h3><p>The final factor!</p><blockquote>Run admin/management tasks as one-off processes.</blockquote><h4>Why Admin Processes Matter</h4><p>Administrative tasks, like certificate renewals or compliance readiness, are integral but often overlooked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Avesuyac2IXoC4O3" /></figure><p>Some other useful admin processes are:</p><ol><li><strong>Database Maintenance:</strong> Running backups, integrity checks, index rebuilds, or cleanups.</li><li><strong>Cleanup Tasks:</strong> Removing old logs, temporary files, or unused resources.</li><li><strong>Batch Processing:</strong> Data processing tasks that can be run as a batch job.</li><li><strong>Data Migration: </strong>Transferring data from one storage system to another.</li><li><strong>Machine Learning Model Training</strong>: Scheduling regular training of ML models.</li><li><strong>Certificate Renewal:</strong> SSL/TLS certificate renewals for services.</li><li><strong>Compliance</strong>: Archiving logs for compliance and audit purposes.</li><li><strong>Seed Jobs:</strong> Seeding databases with initial or test data.</li></ol><p>These Jobs are defined and version-controlled within <strong>Git</strong>, enabling a high degree of customization (hello <strong>Kustomize</strong>) and traceability. This approach elevates administrative processes to the same level of importance and rigor as application deployments or Operator workflows, ensuring consistency, reliability, and repeatability across the board.</p><p>In the example below all certificates of the development can be acted on by a <strong>Kubernetes Job</strong>.</p><pre>~/manifests-index develop &gt;tree -L 4 infra/k8s/admin-tasks/certificate-renewal<br>infra/k8s/admin-tasks/certificate-renewal<br>├── base<br>│   ├── cert-job.yaml<br>│   └── kustomization.yaml<br>└── overlays<br>    └── development<br>        ├── kustomization.yaml<br>        └── schedule-patch.yaml</pre><pre>kustomize build infra/k8s/admin-tasks/certificate-renewal/development | kubectl apply -f -</pre><h4>Kubernetes Job</h4><p>A Kubernetes <strong>Job</strong> creates one or more Pods and ensures that a specified number of them successfully terminate. As soon as the specified number of Pods complete successfully, the Job is marked as complete. This makes Jobs ideal for batch processing, one-time tasks, or any operation that needs to run to completion.</p><pre>apiVersion: apps/v1<br>kind: Job<br>metadata:<br>  name: cleanup-job<br>spec:<br>  completions: 5<br>  parallelism: 2<br>  backoffLimit: 1<br>  selector:<br>    matchLabels:<br>      app: cleanup<br>  template:<br>    metadata:<br>      labels:<br>        app: cleanup<br>    spec:<br>        restartPolicy: Never<br>        containers:<br>        - name: cleanup<br>          image: xx-xxxx-docker.pkg.dev/xxx/cleanup/cleanup:latest<br>          env: <br>            - name: API_VERSION<br>              value: &#39;v1&#39;</pre><p>The properties completions, parallelism, and backoffLimit in a Kubernetes Job configuration play crucial roles in controlling how the job executes its tasks. Let&#39;s break down what each of these properties means:</p><ol><li><strong>completions</strong>: This field specifies the desired number of successfully finished Pods the Job should run. When you set completions to a specific number, Kubernetes will ensure that the Job runs that many Pods which exit successfully. For example, if completions is set to 5, the Job will run until five Pods have completed their tasks successfully.</li><li><strong>parallelism</strong>: This field defines the maximum number of Pods that should be running in parallel at any given time during the execution of the Job. In your example, parallelism is set to 2, which means that the Job will not have more than two Pods running simultaneously. This is useful for managing resource usage and ensuring that the system is not overloaded with too many tasks at once.</li><li><strong>backoffLimit</strong>: This property controls the number of times the Job will retry creating a Pod if the previous Pod fails. A Pod can fail for various reasons, like insufficient resources, application errors, etc. The backoffLimit helps in controlling how many times the Job controller will retry before marking the Job as failed. In this case, with backoffLimit set to 1, if a Pod fails, Kubernetes will attempt to run another Pod to complete the task, but if that second attempt also fails, the Job will be marked as failed.</li></ol><h4>Kubernetes CronJob</h4><p><strong>CronJob</strong>, on the other hand, is similar to Job but with scheduling capability. They allow you to run Jobs at regular intervals. This functionality is perfect for recurring tasks.</p><pre><br>apiVersion: apps/v1<br>kind: CronJob<br>metadata:<br>  name: cleanup-cronjob<br>spec:<br>  schedule: &quot;0 0 * * *&quot;<br>  jobTemplate:<br>    spec:<br>...</pre><h4>The Power of GitOps in Admin Processes</h4><p>We rely heavily on GitOps for our admin tasks. Each admin task is codified and version-controlled, ensuring that it goes through the same CI/CD pipeline as our regular application code. This ensures a single source of truth and enhances auditability.</p><h3>Sailing Beyond the Horizon: The Journey Continues</h3><p>Our commitment to the Reactive Manifesto and 12-factor principles is not a one-off endeavor but an ongoing journey. It shapes our architectural choices, influences our tooling decisions, and most importantly, sets the standard for operational excellence.</p><h3>Evaluating our lean toolkit</h3><p>Looking back we have picked up supporting tools for each pillar of a well formed data platform:</p><ul><li><strong>Underlying Platform</strong>: Kubernetes / OpenShift</li><li><strong>Continuous Delivery</strong>: ArgoCD / Flux</li><li><strong>Rollouts</strong>: Argo Rollouts</li><li><strong>Infrastructure Manifests</strong>: Kustomize, Helm</li><li><strong>Release Management</strong>: Purpose Built In-House Solution</li><li><strong>Data Flywheels</strong>: Purpose Built In-House Solution</li><li><strong>Data Ingestion and Storage</strong>: Confluent Kafka, Strimzi</li><li><strong>Data Pipelines</strong>: Purpose Built In-House Solution</li><li><strong>Development Scaffolding</strong>: Google Skaffold</li></ul><p>This is not an exhaustive list — but can get one started!</p><h3>A Glimpse into the Future</h3><p>As we wrap up this series, we’re already looking ahead. Our next series will focus on Platform Engineering, covering topics like “shift-left” for unit tests, “shift-down” for infrastructure abstraction, and much more.</p><p>Stay tuned, and as always, happy coding!</p><p>That’s a wrap on our 12-factor series! We hope you’ve found it actionable. Thank you for journeying with us through this exploration of best practices in modern software development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=376a6dfa268d" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/logging-and-admin-processes-the-final-pillars-of-12-factor-microservices-376a6dfa268d">Logging and Admin Processes — The Final Pillars of 12-factor Microservices</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>