<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ​Ahmed Adawy | AI Systems &amp; Python on Medium]]></title>
        <description><![CDATA[Stories by ​Ahmed Adawy | AI Systems &amp; Python on Medium]]></description>
        <link>https://medium.com/@anasahmedadawy?source=rss-1f846f48bb7d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5-ja9gs6SL_7CGp99FRXgg.jpeg</url>
            <title>Stories by ​Ahmed Adawy | AI Systems &amp;amp; Python on Medium</title>
            <link>https://medium.com/@anasahmedadawy?source=rss-1f846f48bb7d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:49:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anasahmedadawy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From AI Prototype to Production: The Complete Engineering Stack Every AI Developer Needs]]></title>
            <link>https://medium.com/@anasahmedadawy/from-ai-prototype-to-production-the-complete-engineering-stack-every-ai-developer-needs-64d10acd0bed?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d10acd0bed</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 17:07:48 GMT</pubDate>
            <atom:updated>2026-11-20T17:07:48.703Z</atom:updated>
            <content:encoded><![CDATA[<p>From AI Prototype to Production: The Complete Engineering Stack Every AI Developer Needs</p><p>Why building an AI application is only the beginning — and what it really takes to make it reliable, scalable, and production-ready.</p><p>There is a significant difference between building an AI project that works on your laptop and engineering a system that can survive in production.</p><p>A Python script that calls an API, a notebook that runs a machine learning experiment, or a prototype that generates impressive results can be a great starting point.</p><p>But what happens when the application receives thousands of requests?</p><p>What happens when dependencies break, a new feature introduces a regression, memory consumption increases, or a deployment fails?</p><p>And perhaps the most important question:</p><p>Can you confidently maintain, test, deploy, and scale the system after it leaves your development environment?</p><p>This is where AI engineering becomes much more than writing Python code.</p><p>It becomes a complete engineering discipline.</p><p>The missing link between AI development and production engineering</p><p>Many developers learn AI through isolated topics:</p><p>Python programming.</p><p>Machine learning fundamentals.</p><p>Large Language Models and Generative AI.</p><p>Data processing and vector operations.</p><p>Backend development.</p><p>Docker and deployment.</p><p>Each topic is valuable, but knowing them separately does not automatically mean you can combine them into a reliable production system.</p><p>Real-world engineering requires understanding how these components interact.</p><p>A high-performance Python implementation is not particularly useful if the application has no reliable tests.</p><p>A well-tested application is still difficult to maintain if deployment remains a manual process.</p><p>A successful Docker deployment is not the end of the journey when the system needs orchestration, monitoring, scaling, and operational reliability.</p><p>The engineering challenge is the entire lifecycle.</p><p>That is the idea behind my The Full-Stack AI Engineering &amp; MLOps Mastery Bundle.</p><p>Six practical books. One connected engineering journey.</p><p>I created this bundle to bring together six technical guides covering different layers of modern software and AI infrastructure.</p><p>Rather than focusing on one isolated tool, the collection follows the engineering journey from application architecture to production operations.</p><p>1. AI Systems Engineering: From Prototype to Production</p><p>AI systems need more than models and prompts.</p><p>This book explores the engineering foundations required to move AI and Generative AI applications from experimental environments into production.</p><p>Topics include:</p><p>LLM and Generative AI architecture.</p><p>Machine learning foundations.</p><p>Python performance optimization.</p><p>Scalable backend architectures.</p><p>Containerized microservices.</p><p>Production-oriented implementation practices.</p><p>The central idea is simple: an AI system should be designed as software infrastructure, not merely as an experiment.</p><p>2. High-Performance Python for AI &amp; Data Engineering</p><p>Performance problems often begin long before developers notice them.</p><p>Unnecessary loops, memory allocation, inefficient data structures, and execution overhead can become serious bottlenecks in data-heavy applications.</p><p>This guide explores:</p><p>CPython execution overhead.</p><p>Vectorization with NumPy and PyTorch.</p><p>Memory layout and cache locality.</p><p>Garbage collection and allocation behavior.</p><p>Production benchmarking.</p><p>Performance optimization strategies.</p><p>The objective is to understand why Python code becomes slow and how to engineer more efficient implementations.</p><p>3. Python Testing with pytest: A Practical Guide to Writing Reliable Tests</p><p>Reliable software requires more than manually checking whether a function produces the expected output.</p><p>Testing provides a repeatable way to verify behavior as applications evolve.</p><p>This book covers:</p><p>Assertions and test structure.</p><p>Arrange–Act–Assert.</p><p>Normal cases and edge cases.</p><p>Exception testing.</p><p>Fixtures and parametrization.</p><p>Test organization.</p><p>Code coverage.</p><p>Continuous integration.</p><p>Testing is not an optional finishing step. It is part of the architecture of maintainable software.</p><p>4. GitHub Actions for Python Projects: A Practical Guide to CI/CD Automation</p><p>Manual workflows become increasingly fragile as projects grow.</p><p>GitHub Actions provides a way to automate important development processes directly within a repository.</p><p>This guide explores:</p><p>Workflow structure.</p><p>Python environment configuration.</p><p>Dependency management.</p><p>Automated pytest execution.</p><p>Coverage reports.</p><p>Documentation builds.</p><p>Workflow artifacts.</p><p>Deployment-oriented automation.</p><p>The goal is to make routine engineering tasks repeatable instead of relying on manual execution.</p><p>5. The Production-Ready Docker &amp; Linux Pocket Guide</p><p>Moving an application from a local environment to a server introduces a different set of engineering concerns.</p><p>Developers need to understand containers, Linux environments, networking, permissions, and deployment configuration.</p><p>This practical guide covers:</p><p>Linux CLI and VPS essentials.</p><p>File permissions and system inspection.</p><p>Docker commands and container lifecycle.</p><p>Production-oriented Dockerfiles.</p><p>Docker Compose.</p><p>Multi-container application environments.</p><p>PostgreSQL integration.</p><p>Container security and system cleanup.</p><p>It focuses on practical configurations and workflows that developers can apply to real projects.</p><p>6. Kubernetes &amp; Cloud-Native Engineering</p><p>When applications grow beyond a single container, orchestration and infrastructure management become increasingly important.</p><p>Kubernetes provides a declarative approach to managing distributed workloads.</p><p>This book explores:</p><p>Kubernetes architecture and core abstractions.</p><p>Pods, services, and workload management.</p><p>Desired-state reconciliation.</p><p>Health checks and self-healing.</p><p>Resource management and autoscaling.</p><p>Observability and distributed tracing.</p><p>GitOps workflows.</p><p>Network policies and RBAC.</p><p>Kubernetes deployment patterns for AI and LLM workloads.</p><p>The focus is on understanding Kubernetes as an operational system rather than memorizing isolated commands.</p><p>The engineering workflow behind the bundle</p><p>The six books are connected by a practical progression:</p><p>Architect → Optimize → Test → Automate → Containerize → Orchestrate</p><p>Each stage solves a different engineering problem.</p><p>&lt;escape&gt;1.&lt;/escape&gt; Design an AI application with a production-oriented architecture.</p><p>&lt;escape&gt;2.&lt;/escape&gt; Optimize Python execution and resource consumption.</p><p>&lt;escape&gt;3.&lt;/escape&gt; Establish reliable automated testing.</p><p>&lt;escape&gt;4.&lt;/escape&gt; Automate development and delivery workflows.</p><p>&lt;escape&gt;5.&lt;/escape&gt; Package applications using Docker and Linux best practices.</p><p>&lt;escape&gt;6.&lt;/escape&gt; Deploy and operate distributed workloads with Kubernetes.</p><p>This progression is particularly useful because production engineering is not about mastering a collection of disconnected technologies.</p><p>It is about understanding how each engineering decision affects the next stage of the system lifecycle.</p><p>Who should explore this collection?</p><p>The bundle is designed for developers and technical practitioners who want to move beyond isolated tutorials.</p><p>It is particularly relevant to:</p><p>Python and backend developers.</p><p>AI and Machine Learning engineers.</p><p>Data engineers.</p><p>DevOps and Cloud engineers.</p><p>Site Reliability Engineers.</p><p>Platform engineers.</p><p>Software developers transitioning toward production AI infrastructure.</p><p>You do not need to become an expert in every technology overnight.</p><p>The value comes from developing a connected understanding of the tools, practices, and architectural decisions involved in building production-grade systems.</p><p>One bundle instead of six separate purchases</p><p>The collection includes six complete technical books.</p><p>According to the Leanpub listing:</p><p>Combined suggested price when purchased separately: $83.95</p><p>Bundle minimum price: $15.99</p><p>Bundle suggested price: $28.99</p><p>The bundle also includes free lifetime updates, so readers can receive future revisions and additions to the books.</p><p>You can choose your price within the available pricing options on the page.</p><p>Explore the Full-Stack AI Engineering &amp; MLOps Mastery Bundle</p><p>If you are serious about moving from writing AI prototypes to engineering complete production systems, this collection provides a practical place to start.</p><p>Explore all six books and choose the price that works for you:</p><p><a href="https://leanpub.com/b/thefull-stackaiengineeringmlopsmasterybundle">The Full-Stack AI Engineering &amp; MLOps Mastery Bundle – Ahmed Adawy</a></p><p>My goal with this collection is not simply to explain individual tools.</p><p>It is to help developers understand the engineering discipline that connects application architecture, performance, testing, automation, containers, and cloud-native infrastructure.</p><p>Because building an AI application that works is one achievement.</p><p>Building one that remains reliable, maintainable, and scalable in production is a different level of engineering.</p><p>Author: Ahmed Adawy<br>Focus: AI Engineering | Python | MLOps | Software Architecture | DevOps | Cloud-Native Systems</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d10acd0bed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ **Looking to take your AI engineering skills beyond prototypes and into real-world production?**]]></title>
            <link>https://medium.com/@anasahmedadawy/looking-to-take-your-ai-engineering-skills-beyond-prototypes-and-into-real-world-production-3d82ff215ec5?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d82ff215ec5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 20:55:27 GMT</pubDate>
            <atom:updated>2026-11-16T20:55:27.029Z</atom:updated>
            <content:encoded><![CDATA[<p>🚀 **Looking to take your AI engineering skills beyond prototypes and into real-world production?**</p><p>I highly recommend checking out **The AI Engineer Bootcamp in a Book – Volume II: Deploy, Scale &amp; Operate**.</p><p>This is a practical and valuable resource for developers and AI engineers who want to understand what it really takes to build, deploy, scale, and maintain production-ready AI systems.</p><p>📚 Inside, you&#39;ll explore important engineering concepts, production workflows, deployment strategies, scalability, infrastructure, and operational best practices.</p><p>Whether you&#39;re an aspiring AI engineer or an experienced developer looking to strengthen your production engineering skills, this book offers valuable knowledge and practical insights worth exploring.</p><p>🔥 **Build beyond the prototype. Engineer for production.**</p><p>📖 Check it out here:<br>https://leanpub.com/author/book/theaiengineervolumeiideployscaleoperate/home</p><p>#AIEngineering #MachineLearning #GenerativeAI #Python #DevOps #SoftwareEngineering #ProductionAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d82ff215ec5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Systems Engineering: From Prototype to Production — The Book That Bridges the Gap Between AI…]]></title>
            <link>https://medium.com/@anasahmedadawy/ai-systems-engineering-from-prototype-to-production-the-book-that-bridges-the-gap-between-ai-30696a203341?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/30696a203341</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 22:06:26 GMT</pubDate>
            <atom:updated>2026-11-12T22:06:26.182Z</atom:updated>
            <content:encoded><![CDATA[<p>AI Systems Engineering: From Prototype to Production — The Book That Bridges the Gap Between AI Demos and Real Systems.</p><p>Most AI projects do not fail because the model is weak. They fail because they never make it out of the notebook.</p><p>You build a prototype, run a few experiments, get impressive results in Jupyter, and then realize that production is a completely different world: APIs, scalability, reliability, security, monitoring, cost control, and real users.</p><p>AI Systems Engineering: From Prototype to Production by Ahmed Adawy addresses exactly that missing layer. It is a practical guide for engineers who want to build AI systems that can actually run in production, not just demos that look good in a presentation.</p><p>The book is currently available on Leanpub for $4.99, down from its suggested price of $23.99.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>Why This Book Stands Out</p><p>There is no shortage of tutorials explaining how to call an LLM API, fine-tune a model, or build a quick chatbot. What is much harder to find is practical guidance on turning those experiments into reliable software.</p><p>This book focuses on the engineering side of AI:</p><p>How to design systems around LLMs and Generative AI.</p><p>How to move from a prototype to a production-ready service.</p><p>How to improve performance and handle computational bottlenecks.</p><p>How to build scalable, secure backend architectures.</p><p>How to build AI pipelines using modern Python tooling.</p><p>Rather than treating models as magic black boxes, the book approaches them as components inside a larger software system. That is the mindset needed to build real AI products.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>What You Will Learn</p><p>The book covers four major areas:</p><p>AI systems fundamentals: A deeper understanding of LLMs, Generative AI, and core Machine Learning concepts.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>Performance engineering: Identifying bottlenecks, optimizing Python code, and using high-performance vectorized operations.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>Production deployment: Building secure and scalable microservices and backend architectures for AI workloads.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>Real-world applications: Creating AI pipelines with modern Python frameworks and production best practices.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>This makes the book particularly useful if you are trying to answer questions such as:</p><p>How do I deploy an AI model reliably?</p><p>How do I scale an LLM-based application?</p><p>How do I reduce latency and computational cost?</p><p>How do I structure a backend for AI services?</p><p>How do I turn a notebook experiment into a production system?</p><p>Who Should Read It?</p><p>This book is a strong fit for:</p><p>Software engineers moving into AI engineering.</p><p>Machine learning engineers who need to deploy models beyond experiments.</p><p>Backend developers building AI-powered products.</p><p>Solutions architects designing AI infrastructure.</p><p>Python developers who want to understand production AI systems.</p><p>Technical founders building AI-first products.</p><p>The book is explicitly aimed at software engineers, machine learning practitioners, and technical architects.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>The Price Is Exceptionally Good</p><p>At the time of writing, the book is priced at:</p><p>Current price: $4.99</p><p>Suggested price: $23.99</p><p>Minimum price: $4.99</p><p>That means you are getting it for less than a quarter of its suggested price.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>For a technical book focused on production AI systems, this is a very accessible entry point, especially if you are building your AI engineering skill set in 2026.</p><p>Bundles Worth Considering</p><p>If you want to go beyond a single book, the author also offers several Leanpub bundles:</p><p>BundleBooksPricePractical AI &amp; Semantic Search Engineering3$15.99 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]Generative AI &amp; LLM Engineering4$22.99 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]AI Infrastructure, DevOps &amp; Production Engineering5$16.99 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]Full-Stack AI Engineering &amp; MLOps Mastery6$10.00 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]Ultimate Agentic AI &amp; LLM Production Mastery4$7.99 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]LLM Foundations, Semantic Search &amp; Production Engineering4$7.99 [<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>The last bundle is especially interesting: four books for $7.99, while buying them separately would cost $85.97.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>About the Author</p><p>Ahmed Adawy focuses on connecting theoretical concepts with practical implementation. His work emphasizes applied mathematics, linear algebra, machine learning, and translating complex ideas into clear, executable Python code.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><p>That approach is visible throughout the book: it is not just theory about AI; it is about building systems that work.</p><p>My Take</p><p>The biggest strength of AI Systems Engineering is that it does not oversell the easy part of AI. Anyone can generate a response from an LLM. The hard part is building a system that remains reliable, secure, observable, and cost-efficient when real users depend on it.</p><p>If you are completely new to programming or AI, this may not be the first book you should read. But if you already have some engineering background and want to become a serious AI engineer, this is a highly relevant resource.</p><p>It is especially valuable for developers who want to move from:</p><p>“I built a model.”</p><p>to:</p><p>“I built a production system.”</p><p>Get the Book</p><p>You can find AI Systems Engineering: From Prototype to Production here:</p><p><a href="https://leanpub.com/aisystemsengineering">AI Systems Engineering on Leanpub</a></p><p>Current price: $4.99.[<a href="https://leanpub.com/aisystemsengineering">leanpub</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30696a203341" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Two Limited-Time Bundles for AI and Python Engineers]]></title>
            <link>https://medium.com/@anasahmedadawy/two-limited-time-bundles-for-ai-and-python-engineers-96fc5d7f2a71?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/96fc5d7f2a71</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 06:01:19 GMT</pubDate>
            <atom:updated>2026-10-23T06:01:19.770Z</atom:updated>
            <content:encoded><![CDATA[<p>semantic search, or Python-based engineering automation, I’m running two limited-time discounts on Leanpub.</p><h3>LLM Foundations, Semantic Search &amp; Production Engineering Bundle</h3><p>A practical bundle for engineers building AI-powered products and production systems. It covers LLM foundations, semantic search, and production engineering concepts.</p><p><strong>Price:</strong> $12.15<br><strong>Get it here:</strong><br><a href="https://leanpub.com/b/theproduction-readyllmautonomousaimasterybundle/c/LeanpubAuthorUpdate20261023">https://leanpub.com/b/theproduction-readyllmautonomousaimasterybundle/c/LeanpubAuthorUpdate20261023</a></p><h3>The Elite Python Optimization &amp; Engineering Automation Bundle</h3><p>A bundle for Python developers who want to improve performance, engineering quality, and automation in real projects.</p><p><strong>Price:</strong> $9.11<br><strong>Get it here:</strong><br><a href="https://leanpub.com/b/theelitepythonoptimizationengineeringautomationbundle/c/LeanpubAuthorUpdate20261023">https://leanpub.com/b/theelitepythonoptimizationengineeringautomationbundle/c/LeanpubAuthorUpdate20261023</a></p><p>The discounts are available until <strong>October 29, 2026</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96fc5d7f2a71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your LLM Is Slow. The GPU Might Not Be the Problem.]]></title>
            <link>https://medium.com/@anasahmedadawy/your-llm-is-slow-the-gpu-might-not-be-the-problem-77b5ca4506dc?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/77b5ca4506dc</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 20:01:16 GMT</pubDate>
            <atom:updated>2026-10-22T20:01:16.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your LLM Is Slow. The GPU Might Not Be the Problem.</h3><h3>A practical approach to tracing queueing, retrieval, prefill, and decoding in production AI systems.</h3><p>A user asks a question.</p><p>Your application authenticates the request, searches a knowledge base, builds a prompt, calls an inference endpoint, and streams an answer.</p><p>The user waits.</p><p>Your dashboard says:</p><pre>AI request duration: 11.4 seconds</pre><p>That number tells you the experience was slow.</p><p>It does not tell you what to fix.</p><p>Should you change the model? Add replicas? Tune retrieval? Reduce the prompt? Investigate queueing? Fix a buffering proxy?</p><p>Before changing infrastructure, I would ask a more basic question:</p><p>Where did the time go?</p><h3>One Request Contains Several Different Systems</h3><p>Consider a retrieval-augmented application:</p><pre>Client<br>  │<br>API gateway<br>  │<br>Authentication<br>  │<br>Query embedding<br>  │<br>Retrieval<br>  │<br>Reranking<br>  │<br>Prompt construction<br>  │<br>Inference endpoint<br>  │<br>Response streaming<br>  │<br>Client rendering</pre><p>This is an illustrative architecture, not a requirement that every application use all these components.</p><p>The important point is that the user experiences one operation while the application executes several.</p><p>A single request_duration measurement collapses those operations into one number.</p><p>That is useful for measuring the symptom. It is insufficient for choosing the intervention.</p><p>Google’s SRE guidance makes this distinction explicit: monitoring needs to answer both “what is broken?” and “why?” User-facing symptoms and internal causes are related, but they are not interchangeable. <a href="https://sre.google/sre-book/monitoring-distributed-systems/">sre</a></p><h3>Define the Measurement Boundary First</h3><p>“Time to first token” sounds precise until two teams measure it differently.</p><p>An inference service might start its timer when it receives a request.</p><p>Your backend might start when it calls that service.</p><p>The browser might start when the user submits the question.</p><p>Those are different boundaries.</p><p>For an application, I would record at least these timestamps:</p><pre>t0: Application receives request<br>t1: Retrieval starts<br>t2: Retrieval completes<br>t3: Model request is dispatched<br>t4: First content chunk arrives from model service<br>t5: First content chunk is forwarded to client<br>t6: Final response chunk is forwarded</pre><p>If client instrumentation is available, add:</p><pre>t7: First answer content is rendered</pre><p>Name the measurements according to what they actually observe.</p><p>For example:</p><pre>application_first_content_latency = t5 - t0<br>model_call_first_content_latency  = t4 - t3<br>retrieval_duration               = t2 - t1<br>stream_forwarding_delay         = t5 - t4</pre><p>A content chunk is not necessarily one model token. Avoid labeling a transport-level measurement as per-token latency unless the underlying data supports that interpretation.</p><p>Also, do not subtract timestamps from different machines as though their clocks were perfectly aligned. Prefer local elapsed-time measurements and a tracing system designed to represent distributed execution.</p><h3>Separate Waiting From Working</h3><p>Suppose the model call takes 2.3 seconds before returning its first content.</p><p>That observation alone does not reveal how much time was spent:</p><ul><li>Traveling to the service.</li><li>Waiting for admission.</li><li>Waiting for execution resources.</li><li>Processing uncached prompt tokens.</li><li>Producing and returning the initial output.</li></ul><p>If you operate the serving engine, expose those distinctions where possible.</p><p>If you use a hosted API and do not receive internal timing information, preserve that uncertainty.</p><p>Write:</p><pre>Model-call first-content latency increased.<br>Internal queueing and execution times are not available.</pre><p>Not:</p><pre>Prefill became slower.</pre><p>The second statement requires evidence that the first does not provide.</p><h3>Instrument Request Paths, Not Just Components</h3><p>A database dashboard can tell you the database is slow.</p><p>It cannot necessarily tell you whether the specific failed answer used that database path, a fallback path, or a cached result.</p><p>That is where distributed tracing becomes useful.</p><p>OpenTelemetry’s context-propagation model carries trace context across service boundaries. Logs can include trace and span IDs, making it possible to connect records to the execution that produced them. <a href="https://opentelemetry.io/docs/concepts/context-propagation/">opentelemetry</a></p><p>For a proposed AI application, I would start with a span tree like this:</p><pre>POST /answer<br>├── authenticate<br>├── embed_query<br>├── retrieve_documents<br>├── rerank_documents<br>├── construct_prompt<br>└── generate_answer<br>    ├── wait_for_first_content<br>    └── consume_response_stream</pre><p>The inference server may contribute additional internal spans if it is instrumented and propagates context appropriately.</p><p>Do not invent visibility that your deployment does not provide.</p><p>The goal is a navigable execution path, not an impressive-looking diagram.</p><h3>Attach the Context That Makes Comparisons Useful</h3><p>A trace becomes more informative when it records which version and workload it represents.</p><p>For the application design above, I would consider attributes such as:</p><pre>service.version<br>deployment.environment<br>model.identifier<br>prompt.template_version<br>retrieval.index_version<br>input.token_count<br>output.token_count<br>retrieval.document_count</pre><p>These are suggested fields; define naming and semantics consistently in your implementation.</p><p>The principle is more important than the exact names:</p><p>When performance changes, you need to identify what else changed.</p><p>Google’s SRE Workbook specifically recommends monitoring binary versions, feature flags, and dynamic configuration versions to help diagnose production regressions. <a href="https://sre.google/workbook/monitoring/">sre</a></p><p>For an AI application, I would extend that discipline to prompt templates, retrieval configuration, and model routing.</p><p>Do not put secrets, authorization headers, or raw private prompts into telemetry by default. Decide what is necessary to collect, who can access it, and how long it should be retained.</p><h3>An Illustrative Incident: Slow Answers After a Retrieval Change</h3><p>Imagine this before-and-after comparison:</p><p>Measurement</p><p>Before</p><p>After</p><p>Retrieval duration</p><p>180 ms</p><p>190 ms</p><p>Documents selected</p><p>5</p><p>18</p><p>Input tokens sent to model</p><p>2,400</p><p>10,800</p><p>Model-call first-content latency</p><p>650 ms</p><p>2,300 ms</p><p>These are invented numbers for an engineering example, not a benchmark.</p><p>The database did not become dramatically slower.</p><p>The model did not change.</p><p>But the request sent to the model became much larger.</p><p>That gives us a hypothesis:</p><blockquote><em>A retrieval-configuration change increased prompt size and contributed to higher first-content latency.</em></blockquote><p>It does not yet prove the cause.</p><p>The next step is a controlled comparison:</p><ul><li>Use the same evaluation questions.</li><li>Hold model selection and output limits constant.</li><li>Compare the old and new retrieval policies.</li><li>Control or record concurrency.</li><li>Record cache conditions where they are observable.</li><li>Measure answer quality as well as latency.</li></ul><p>Reducing retrieved context is not automatically a successful optimization if it also removes the evidence needed to answer correctly.</p><p>The experiment needs both performance and quality criteria.</p><h3>Compare Like With Like</h3><p>An aggregate latency chart mixes workloads.</p><p>A short answer generated from a small prompt is not the same workload as a long answer generated from a large document collection.</p><p>Before concluding that a release caused a regression, I would break down results by a manageable set of dimensions:</p><ul><li>Model or routing destination.</li><li>Input-length range.</li><li>Output-length range.</li><li>Application version.</li><li>Prompt-template version.</li><li>Deployment region.</li><li>Relevant request category.</li></ul><p>Use request-specific details in logs and traces rather than turning every identifier into a metric label.</p><p>For example, a request ID is useful for finding one execution. It is not a useful grouping dimension for an application-wide latency dashboard.</p><p>The objective is to distinguish a genuine slowdown from a change in traffic mix.</p><h3>“What Changed?” Is a Better Starting Point Than “What Looks Abnormal?”</h3><p>When an incident begins after a release, the release is an obvious candidate.</p><p>But a temporal match is not proof.</p><p>The same caution applies to:</p><ul><li>A model update.</li><li>A larger prompt.</li><li>A cache miss.</li><li>A traffic surge.</li><li>A dependency warning.</li></ul><p>I would keep a short hypothesis table during an investigation:</p><p>Hypothesis</p><p>Supporting evidence</p><p>Evidence still needed</p><p>Retrieval change increased prompt-processing cost</p><p>Larger inputs after configuration change</p><p>Controlled old-versus-new comparison</p><p>Queueing increased under load</p><p>Higher first-content latency at higher concurrency</p><p>Serving-side admission timing</p><p>Streaming is buffered downstream</p><p>Backend receives content before client does</p><p>Proxy and client timing comparison</p><p>This makes uncertainty visible.</p><p>It also discourages the familiar incident pattern where a plausible explanation becomes the accepted explanation before anyone tests it.</p><h3>An AI Investigation Assistant Should Have to Show Its Work</h3><p>An LLM can help summarize traces, compare configurations, and generate candidate explanations.</p><p>That is a proposed role — not a claim that it can reliably identify root cause from arbitrary telemetry.</p><p>If I added an AI assistant to this workflow, I would require its output to separate:</p><pre>Observation<br>Hypothesis<br>Supporting evidence<br>Missing evidence<br>Proposed test<br>Potential operational risk</pre><p>A useful response might be:</p><blockquote><em>Input-token counts increased after prompt-template version 12 was deployed. The sampled traces also show higher model-call first-content latency. This is consistent with increased prompt-processing work, but internal queue timing is unavailable. Compare versions 11 and 12 under matched load before attributing the regression to prefill.</em></blockquote><p>That is more actionable than:</p><blockquote><em>Root cause found: prompt too large. Roll back immediately.</em></blockquote><p>The first answer explains what is known and proposes a test.</p><p>The second turns a hypothesis into an operational instruction.</p><h3>A Minimal Investigation Checklist</h3><p>Before buying more hardware or replacing the serving engine, I would check:</p><ol><li>Does the latency measurement represent the user’s experience or only one backend call?</li><li>Can a failed request be followed across retrieval, generation, and streaming?</li><li>Are queueing and execution distinguishable where the infrastructure exposes them?</li><li>Did prompt size, output length, or traffic mix change?</li><li>Are deployments and configuration changes recorded?</li><li>Does the proposed fix preserve answer quality?</li><li>Can the suspected cause be tested without changing several variables at once?</li></ol><p>This does not require collecting everything.</p><p>It requires collecting enough connected evidence to answer the next diagnostic question.</p><h3>Where This Connects to My Book</h3><p>I’m Ahmed Adawy, the author of <em>AI Systems Engineering: From Prototype to Production</em>.</p><p>The book sample explores the inference lifecycle, model-weight and KV-cache budgeting, prefill and decode, and serving metrics. It also includes an educational asynchronous serving simulator. <a href="https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/2174063930/bc0b0f9a-f4a8-4450-844c-c40e84f65ce1/aisystemsengineeringfromprototypetoproduction-sample.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Checksum-Mode=ENABLED&amp;X-Amz-Credential=ASIA2F3EMEYETTIFO4SO%2F20261022%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20261022T195441Z&amp;X-Amz-Expires=2099&amp;X-Amz-Security-Token=IQoJb3JpZ2luX2VjELT%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQCGN1qsEiZKgbPWg%2Btk%2FF%2Fl7qRtdcyQF6k61SLz6RG%2FewIhAOrIJP1KpCxW2byvMzxhI5QoDeJVKhYzrAepwzZnhFLLKoAFCHwQARoMNjk5NzUzMzA5NzA1IgxPon0efnWuiv%2FHR4gq3QQaqUROpQ2tr%2B0IUbtABI54K9ZaPPJfVVQ1aMNCTNPYs4BLlvqi707PuxAdc%2FbDWg1sVeOOxYr3Ndh3%2BVI2mPaStIY6bMXyA%2Bx2M7uKtpT08vSFA3J4mNvSZN3pG8rT2YPW1t3vOJrH89TDRpggk%2B5HLa2MIO%2BYh8csZuZyBFaTmxBhP2oszdE76vITwe10giMRlCfWYnenxShEevLqvnrncBjJYnpBNAWo4gSzsK%2FzaTY0XYEacQw6HKhFs7TPrWdRvV5XXyDe82qjNpLbiv37BepRFGixHhAopLdpPeWWJWLxmDhXf6VbTIq3NrIgasJCOM70JjlML6X1HS3ysu9eq3UuI41jQBCIlZ3hjZUAkFwVUawxNbXKOJ%2B3fcuK0W0cSPYj427pOllBuoEVjKshy%2BvY1AR6goIohswTPkrqRUZJ3QF3Zck1S4Qj7%2B88iT6%2BUVBYCHsAu1TL9bkoT456f964mpVYedkPrzU3JvP59nZYiGIv8hbNvbARSqkJheDj7zh66n6LZxGbMsULBVcK5DHxb%2FghlSmaTE4mXUe0DQwko6zj5m4aSR8SaqXHicyDL3wQqNFp%2FeNBko5UbziMIt2qpD2I1F0A5657pkm96ISTZ4tmTeyiYeP%2FAE6PK03jjKjjmyb50PEnX3dDW2oYhU8IhNOlOkbEyowm2421S5mJULSqAM2nlCeu9Nj3izOCiYE%2FU20phYYcEybK6G9MT3D0H9%2FV%2BwJlUNFa8F7BsMyU8JgvH0EKRdaB0hfxTMMYM11juU4y4tL%2BzhsMKjjf4TnrbZBOjyd0nlzOqTDhyOnWBjqXASMC6MfQueA%2F%2BAx9538QPqloDzS4WcQL7BpbufRcUmipGUNCTLMwIwQ1C0iQcrXAcAYApF9uASdESikk%2FO0cZQnn4eto1NztAtqubyyDjvNzP3r8%2F02aGHOim755f82zWZPQf8wfjfXmox4oBFMbvL0T7ZsU%2FT1Vk3jdEl46oOmhFSuooQC9yYcQRxiy4bUb%2BS9OLGnAHIo%3D&amp;X-Amz-SignedHeaders=host&amp;x-id=GetObject&amp;X-Amz-Signature=1c4b471b55f3e365764641ec358c6e44b61ff6082892e65240f3148da3f43631">ppl-ai-file-upload.s3.amazonaws</a></p><p>That foundation helps frame a useful distinction:</p><p>Understanding how an inference engine works is necessary for diagnosing its behavior — but application latency also includes the systems around it.</p><p>The simulator is an explanation tool, not a production benchmark. Likewise, the example in this article is a diagnostic scenario, not a claim of measured results from a customer deployment.</p><p>If you want to examine the book’s technical level and approach, start with the free sample:</p><p><a href="https://leanpub.com/aisystemsengineeringfromprototypetoproduction">Explore the book and read the sample on Leanpub</a></p><p>Disclosure: this is my book, and I earn from purchases.</p><h3>The Question to Ask Before “Do We Need a Bigger GPU?”</h3><p>A slow answer is an observation.</p><p>“Insufficient GPU capacity” is one possible explanation.</p><p>The engineering work is connecting the two with evidence.</p><p>Before changing hardware, establish where the request waited, what it processed, which dependencies it used, and what changed.</p><p>Otherwise, you may improve a component that was never the bootelneck.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77b5ca4506dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond VRAM: Engineering Reliable LLM Inference Through KV Cache Management, Scheduling, and…]]></title>
            <link>https://medium.com/@anasahmedadawy/beyond-vram-engineering-reliable-llm-inference-through-kv-cache-management-scheduling-and-7d7f2bbedac3?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d7f2bbedac3</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 21:21:53 GMT</pubDate>
            <atom:updated>2026-10-21T21:25:19.184Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond VRAM: Engineering Reliable LLM Inference Through KV Cache Management, Scheduling, and Memory-Aware Architecture</h3><p><em>Why production LLM performance depends on the interaction between memory allocation, request scheduling, prefix reuse, and tail latency.</em></p><h3>Introduction: When a Working Model Becomes a Production Problem</h3><p>Getting a large language model to run is not the same as building a reliable inference system.</p><p>A model may perform well in a notebook, respond quickly to isolated prompts, and fit comfortably inside available GPU memory. Yet the same model can behave very differently when exposed to concurrent users, long-context requests, unpredictable output lengths, and sustained production traffic.</p><p>The failure is often misunderstood as a simple GPU memory problem.</p><p>In reality, inference performance emerges from several interacting systems: model execution, KV cache allocation, request scheduling, memory reclamation, batching, and the distribution of workloads over time.</p><p>The distinction matters because optimizing any one component in isolation can produce misleading results. A configuration that reduces average latency under a controlled benchmark may increase p99 latency under mixed traffic. A cache with an impressive hit rate may consume memory that would otherwise support active requests. A memory allocator that eliminates external fragmentation cannot eliminate the fundamental capacity requirements of long-running sequences.</p><p>The engineering challenge is not merely to reduce memory consumption. It is to make memory utilization, scheduling decisions, and latency behavior predictable under realistic operating conditions.</p><h3>1. Understanding the Memory Cost of Autoregressive Inference</h3><p>During autoregressive generation, a transformer repeatedly processes new tokens while attending to previously computed representations.</p><p>Without a KV cache, the model would repeatedly recompute key and value projections for earlier tokens. Caching these tensors avoids redundant work, but it introduces a growing memory requirement.</p><p>For a conventional decoder-only transformer, an approximate KV cache size is:</p><p>MKV=2LBTHKVDhSM_{\mathrm{KV}} = 2 L B T H_{\mathrm{KV}} D_h S</p><p>Where:</p><ul><li>LL is the number of transformer layers.</li><li>BB is the number of active sequences.</li><li>TT is the number of cached tokens per sequence.</li><li>HKVH_{\mathrm{KV}} is the number of key-value heads.</li><li>DhD_h is the head dimension.</li><li>SS is the number of bytes per stored element.</li><li>The factor of two accounts for keys and values.</li></ul><p>This approximation assumes a conventional attention architecture with a uniform KV representation. Sliding-window attention, hybrid architectures, quantized caches, and implementation-specific layouts can change the actual allocation.</p><p>Consider a model with 32 layers, 8 KV heads, a head dimension of 128, and a two-byte representation.</p><p>The cache requirement per token per sequence is approximately:</p><p>2×32×8×128×2=131072 bytes2 \times 32 \times 8 \times 128 \times 2 = 131072 \text{ bytes}</p><p>That is approximately 128 KiB per token.</p><p>At 8,192 cached tokens, one sequence requires roughly 1 GiB of KV cache, before accounting for allocator metadata, temporary buffers, model weights, and other runtime allocations.</p><p>With multiple concurrent sequences, this requirement scales quickly.</p><p>The important implication is that concurrency and context length are not independent performance variables. They compete for the same finite memory budget.</p><p>A server that handles many short conversations efficiently may struggle when a small number of long-context requests remain active for extended periods.</p><h3>2. Why VRAM Fragmentation Is Only Part of the Problem</h3><p>GPU memory pressure is often discussed as though all allocation failures originate from fragmentation.</p><p>That assumption is too simplistic.</p><p>There are at least three distinct failure modes:</p><p><strong>Capacity pressure:</strong> The total memory required by model weights, active KV state, temporary workspaces, and runtime overhead exceeds the available budget.</p><p><strong>Allocation inefficiency:</strong> The allocation strategy creates unused gaps, internal waste, or reservations that cannot be efficiently repurposed.</p><p><strong>Lifetime mismatch:</strong> Memory remains associated with sequences or cached state longer than the workload can economically tolerate.</p><p>These problems require different interventions.</p><p>Dynamic allocation can reduce waste caused by reserving memory for tokens that have not yet been generated. Paged allocation can make memory easier to share and reclaim. Better eviction policies can shorten the lifetime of reusable cache entries.</p><p>None of these techniques creates additional physical VRAM.</p><p>This distinction is essential when diagnosing a deployment that gradually approaches an out-of-memory condition after hours of operation. The underlying issue might be retained cache state, increasing average sequence length, a changing concurrency distribution, or an allocation leak. A single snapshot of GPU utilization cannot distinguish these possibilities.</p><p>The correct diagnosis requires time-series measurements and workload-aware experiments.</p><h3>3. Paged KV Cache: Changing the Allocation Model</h3><p>Paged attention addresses a central inefficiency of traditional contiguous KV cache allocation.</p><p>Instead of requiring every sequence to occupy one large contiguous memory region, the runtime divides its KV state into fixed-size blocks or pages. Logical token positions are mapped to physical blocks through metadata.</p><p>This allows sequences to grow incrementally and enables physical memory blocks to be managed independently.</p><p>The practical benefits include:</p><ul><li>Less pressure to reserve large contiguous regions for every request.</li><li>More flexible allocation as sequences grow.</li><li>Improved memory sharing and reclamation.</li><li>Better support for variable-length sequences and continuous batching.</li></ul><p>PagedAttention, implemented in systems such as vLLM, illustrates this approach. The underlying design is described in the <a href="https://docs.vllm.ai/en/latest/design/paged_attention/">vLLM PagedAttention paper and documentation</a>.</p><p>However, paging should not be confused with a universal solution to memory pressure.</p><p>Block size matters. Smaller blocks can reduce internal waste and improve the granularity of allocation, but they can increase metadata and management overhead. Larger blocks can improve certain execution patterns while making partial reuse less efficient.</p><p>The optimal configuration depends on the model architecture, attention implementation, workload, and serving runtime.</p><p>More importantly, paging solves an allocation problem. It does not decide which requests should receive priority, how long reusable prefixes should remain resident, or how much capacity should be reserved for active generation.</p><p>Those are separate scheduling and cache-management problems.</p><h3>4. Prefix Caching: Computation Saved, Memory Still Consumed</h3><p>Prefix caching reuses previously computed KV state when a new request shares a compatible prefix with an earlier request.</p><p>This is particularly useful when requests repeatedly contain the same system instructions, document context, or conversation history.</p><p>Without reuse, the inference engine must recompute the prefix during prefill. With reuse, it can avoid some or much of that repeated computation, depending on the matching prefix and implementation.</p><p>The potential benefit is substantial, but the optimization has a cost: reusable state occupies memory that could otherwise serve active sequences.</p><p>A cache hit is therefore not automatically equivalent to a system-level performance improvement.</p><p>Suppose a workload repeatedly uses a long shared system prompt. Keeping that prefix resident may reduce prefill latency and increase effective throughput. Now introduce many long-running requests with different contexts. Those active requests require additional KV blocks, creating pressure on the reusable cache.</p><p>The runtime must decide whether retaining the shared prefix is more valuable than reclaiming its memory for current demand.</p><p>This is a resource-allocation decision, not simply a cache-hit optimization.</p><p>The <a href="https://docs.vllm.ai/en/latest/design/prefix_caching/">vLLM automatic prefix caching documentation</a> describes reuse of KV blocks associated with matching prompt prefixes. NVIDIA’s <a href="https://nvidia.github.io/TensorRT-LLM/features/kvcache.html">TensorRT-LLM KV cache documentation</a> also describes block reuse, retention priorities, and eviction behavior.</p><p>These implementations differ in details, so their configuration options should not be treated as interchangeable. The broader engineering principle remains the same: reuse benefits depend on matching opportunities, memory availability, and the cost of keeping cached state.</p><h3>A better way to evaluate prefix caching</h3><p>Instead of reporting only the cache hit rate, measure:</p><ol><li>Reused prefix tokens per request.</li><li>Prefill time avoided by reuse.</li><li>Time-to-first-token distribution.</li><li>Cache eviction frequency.</li><li>Active-request preemption or admission delays.</li><li>Throughput under memory pressure.</li><li>p95 and p99 end-to-end latency.</li></ol><p>These measurements help determine whether caching improves the actual service rather than one isolated metric.</p><h3>5. Prefill and Decode Are Different Workloads</h3><p>One of the most useful distinctions in LLM serving is the difference between prefill and decode.</p><p><strong>Prefill</strong> processes the input prompt. It can involve substantial parallel computation and often has a significant effect on time-to-first-token.</p><p><strong>Decode</strong> generates subsequent tokens autoregressively. It repeatedly accesses model weights and previously computed KV state, and its performance is strongly influenced by memory bandwidth, active sequence count, scheduling, and output length.</p><p>These phases share infrastructure, but their bottlenecks are not identical.</p><p>A long prompt can create a substantial prefill workload. A large number of concurrently decoding sequences can increase KV cache demand and complicate scheduling. A workload containing both can create interference that is invisible in isolated tests.</p><p>This is why a single aggregate latency number is inadequate for diagnosing a production inference pipeline.</p><p>A useful measurement strategy separates:</p><ul><li>Queueing delay before execution.</li><li>Time spent in prefill.</li><li>Time-to-first-token.</li><li>Inter-token latency during decode.</li><li>Total generation duration.</li><li>End-to-end latency, including queueing and output delivery.</li></ul><p>When prefill and decode are tightly coupled, long prompts can affect the responsiveness of ongoing generations. Techniques such as chunked prefill, scheduling policies that balance prompt processing against decode progress, and disaggregated prefill/decode architectures may help, depending on the workload and infrastructure.</p><p>Disaggregation is not free: it introduces communication, state-transfer, and operational complexity. It should be evaluated against a measured bottleneck, not adopted simply because it is architecturally fashionable.</p><h3>6. Scheduling Under Mixed Workloads</h3><p>Consider a service receiving three request types:</p><ul><li>Short interactive questions requiring low response latency.</li><li>Long-context retrieval requests with expensive prefill.</li><li>Multi-turn sessions with substantial opportunities for prefix reuse.</li></ul><p>A scheduler optimized only for maximum batch utilization may not provide acceptable responsiveness to interactive requests. A scheduler optimized exclusively for short requests may underutilize the hardware or delay long requests indefinitely.</p><p>The system must balance competing objectives.</p><p>A practical design starts by defining service-level objectives for each workload class. Interactive traffic may prioritize time-to-first-token and tail latency, while asynchronous document processing may prioritize throughput and cost per generated token.</p><p>Possible scheduling signals include:</p><ul><li>Estimated remaining sequence length.</li><li>Current KV cache allocation.</li><li>Expected reusable prefix tokens.</li><li>Queueing time.</li><li>Request priority and deadline.</li><li>Recent decode progress.</li><li>Available memory for new sequences.</li></ul><p>These signals should inform admission control and scheduling without creating an unnecessarily complicated policy.</p><p>For example, a scheduler could limit the number of long-context requests admitted concurrently when projected KV demand approaches a safe threshold. It could also distinguish latency-sensitive traffic from background processing.</p><p>However, prefix-aware scheduling requires care. A predicted cache hit is not necessarily a guaranteed hit, and delaying requests to improve reuse can increase queueing latency.</p><p>The goal is not to maximize reuse at any cost. It is to optimize useful work completed within the service’s latency and resource constraints.</p><h3>7. Memory-Aware Cache Retention</h3><p>A simple least-recently-used policy is a useful baseline, but recency alone may not reflect the value of retaining a prefix.</p><p>A frequently reused system prompt may justify a longer lifetime than a one-off context. Conversely, a large cached prefix that is unlikely to be reused soon may consume valuable capacity.</p><p>A more informed retention strategy can consider:</p><ul><li>Reuse frequency.</li><li>Prefix size.</li><li>Time since the last reuse.</li><li>Estimated probability of future reuse.</li><li>Cost of recomputing the prefix.</li><li>Memory required by active requests.</li><li>Cost of moving state between GPU and host memory.</li></ul><p>One conceptual measure is the expected benefit per unit of retained memory:</p><p>Vi=Pi⋅CiMiV_i = \frac{ P_i \cdot C_i }{ M_i }</p><p>Where:</p><ul><li>PiP_i is the estimated probability of reuse within a chosen time horizon.</li><li>CiC_i is the estimated recomputation cost avoided.</li><li>MiM_i is the memory occupied by the cached entry.</li></ul><p>This is a prioritization heuristic, not a complete eviction algorithm. It omits factors such as reuse timing, contention, transfer cost, and the interaction between shared prefixes.</p><p>Still, it illustrates an important principle: cache retention should reflect expected system value rather than popularity alone.</p><p>An implementation could combine reuse estimates with memory pressure and workload priorities. The policy should remain measurable and explainable; otherwise, complexity may exceed the benefit.</p><h3>8. Host-Memory Offloading: Extending Capacity With a Transfer Cost</h3><p>When GPU memory is constrained, moving reusable KV state to host memory can preserve opportunities for future reuse.</p><p>The trade-off is straightforward.</p><p>Keeping a cache entry on the GPU avoids transfer latency but consumes scarce accelerator memory. Offloading frees GPU capacity but requires copying state back when it is needed again.</p><p>Whether offloading helps depends on several variables:</p><ul><li>Interconnect bandwidth and latency.</li><li>Size of the cached state.</li><li>Probability and timing of reuse.</li><li>Cost of recomputing the prefix.</li><li>Contention for host memory and transfer bandwidth.</li><li>Whether transfers overlap with useful GPU work.</li></ul><p>The relevant comparison is not simply GPU memory versus system memory. It is the cost of restoring cached state versus recomputing it, combined with the opportunity cost of retaining that state on the GPU.</p><p>If a prefix is unlikely to be reused, offloading it may merely postpone eviction. If it is expensive to recompute and likely to be reused, offloading may provide a net benefit.</p><p>This is particularly important on systems where CPU-GPU transfers are expensive or where host-memory pressure introduces a second bottleneck.</p><p>A sensible evaluation should compare GPU-resident caching, host offloading, and direct eviction under the same workload trace.</p><h3>9. A Benchmark That Reflects Production Reality</h3><p>Many inference benchmarks are useful for measuring kernel performance but insufficient for evaluating a complete serving system.</p><p>A benchmark that sends identical prompts at a fixed rate may overstate prefix-cache benefits. A benchmark with uniform sequence lengths may conceal scheduler instability. A short test may never expose gradual memory pressure or long-lived cache retention.</p><p>A more informative benchmark should reproduce the characteristics of the intended workload.</p><h3>Workload dimensions</h3><p>Vary the following independently where possible:</p><ul><li>Prompt length distribution.</li><li>Output length distribution.</li><li>Request arrival rate.</li><li>Concurrency.</li><li>Shared-prefix frequency.</li><li>Prefix length and reuse interval.</li><li>Mixture of interactive and long-running requests.</li><li>Duration of sustained load.</li></ul><p>Then introduce combinations that reflect real traffic.</p><p>For example, compare a homogeneous short-request workload with a mixed workload in which short interactive requests compete with long-context generations.</p><h3>Metrics that matter</h3><p>MetricWhat it revealsTime-to-first-tokenInitial responsiveness, including relevant queueing and prefill effectsInter-token latencySmoothness and responsiveness during generationp95/p99 latencyTail behavior under contentionCompleted requests per secondEffective serving throughputOutput tokens per secondGeneration capacityKV cache occupancyPressure on cache capacityPrefix reuse rateFrequency of successful reuseEviction rateCache churn and retention pressureGPU memory utilizationOverall accelerator memory consumptionQueueing delayScheduling and admission pressureError and OOM rateReliability under sustained demand</p><p>The measurement definitions must be consistent. For example, time-to-first-token should specify whether it includes queueing, and throughput should be reported alongside the workload’s prompt and output length distributions.</p><h3>Compare controlled configurations</h3><p>A useful experiment matrix might include:</p><ol><li>Dynamic paged allocation without cross-request prefix caching.</li><li>Paged allocation with prefix caching enabled.</li><li>Prefix caching with different block sizes or retention policies.</li><li>Host-memory offloading where supported.</li><li>Scheduling adjustments for mixed context lengths.</li><li>Admission control under constrained KV capacity.</li></ol><p>Keep the model, hardware, precision, arrival trace, and service settings consistent unless the experiment explicitly varies them.</p><p>Run tests long enough to observe cache turnover and sustained memory behavior. Repeat trials to account for variability.</p><p>Most importantly, report measured results rather than assuming that a particular optimization must improve performance. The winner may differ between workloads.</p><h3>10. An Operational Strategy for Diagnosing Memory Bottlenecks</h3><p>When a production LLM service begins showing rising latency or memory pressure, I would investigate it in a defined order.</p><p><strong>Step 1: Establish the baseline.</strong></p><p>Record GPU memory usage, active sequences, request lengths, queueing delay, time-to-first-token, inter-token latency, and throughput over time.</p><p><strong>Step 2: Separate capacity from allocation inefficiency.</strong></p><p>Determine whether memory is consumed by model weights, active KV state, reusable cache blocks, temporary workspaces, or other allocations. Do not assume that every memory problem is fragmentation.</p><p><strong>Step 3: Separate prefill from decode.</strong></p><p>Identify whether latency rises during prompt processing, ongoing generation, or admission into the scheduler.</p><p><strong>Step 4: Examine cache behavior.</strong></p><p>Measure prefix reuse, cache residency, eviction frequency, and the amount of prefill work actually avoided.</p><p><strong>Step 5: Reproduce the failure with a representative trace.</strong></p><p>Include the same distribution of prompt lengths, output lengths, concurrency, and request arrivals that produced the problem.</p><p><strong>Step 6: Change one major variable at a time.</strong></p><p>Compare allocation strategy, block configuration, retention policy, and scheduling adjustments under controlled conditions.</p><p><strong>Step 7: Validate reliability under sustained load.</strong></p><p>Check for increasing tail latency, unstable throughput, excessive cache churn, and out-of-memory failures over time.</p><p>This approach avoids a common mistake: optimizing the most visible metric before identifying the mechanism responsible for the failure.</p><h3>11. The Deeper Architectural Lesson</h3><p>Reliable LLM inference is a resource-management problem as much as it is a model-execution problem.</p><p>The model defines computational requirements, but the serving system determines how those requirements compete for finite resources across time.</p><p>Paged allocation improves how KV state occupies memory. Prefix caching reduces repeated computation. Retention policies determine which reusable state survives. Scheduling determines which requests receive resources and when. Offloading changes the trade-off between capacity and transfer cost.</p><p>These mechanisms are interdependent.</p><p>Increasing prefix retention can reduce prefill work but leave less memory for active sequences. Increasing concurrency can improve throughput until memory pressure or scheduling interference increases tail latency. Smaller allocation blocks can reduce waste while adding management overhead. Offloading can preserve reusable state while introducing transfer costs.</p><p>Consequently, no single metric or configuration is sufficient to characterize the quality of an inference system.</p><p>The right optimization target is the complete workload: its latency objectives, request distribution, memory budget, reuse patterns, and reliability requirements.</p><h3>Conclusion: Optimize the System, Not the Cache Hit Rate</h3><p>When an LLM inference pipeline struggles under production traffic, the first question should not be, “How do we save more VRAM?”</p><p>A better question is: “Which combination of memory allocation, cache retention, and scheduling decisions is preventing the system from meeting its workload objectives?”</p><p>That question leads to more useful measurements and better engineering decisions.</p><p>Start by separating prefill from decode, distinguish capacity pressure from allocation inefficiency, measure the real value of prefix reuse, and evaluate tail latency under sustained mixed workloads.</p><p>Then select optimizations based on observed bottlenecks rather than isolated benchmark wins.</p><p>The objective is not to keep every reusable token in memory, maximize concurrency, or minimize GPU utilization. It is to deliver predictable latency and high useful throughput while keeping memory consumption and operational complexity under control.</p><p><strong>A production-grade inference pipeline is not the one that achieves the highest cache hit rate in a benchmark. It is the one that continues to meet its service objectives when the workload stops being predictable.</strong></p><h3>Further Reading</h3><ul><li><a href="https://docs.vllm.ai/en/latest/design/paged_attention/">PagedAttention and vLLM design documentation</a></li><li><a href="https://docs.vllm.ai/en/latest/design/prefix_caching/">Automatic Prefix Caching in vLLM</a></li><li><a href="https://nvidia.github.io/TensorRT-LLM/features/kvcache.html">NVIDIA TensorRT-LLM KV Cache System</a></li><li><a href="https://nvidia.github.io/TensorRT-LLM/latest/legacy/advanced/kv-cache-reuse.html">NVIDIA TensorRT-LLM KV Cache Reuse</a></li></ul><p>Artificial Intelligence, LLM, MLOps, GPU, Software Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d7f2bbedac3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong.]]></title>
            <link>https://medium.com/@anasahmedadawy/your-ai-assistant-remembered-the-fact-it-still-got-the-answer-wrong-69ed31892210?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/69ed31892210</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 21:06:47 GMT</pubDate>
            <atom:updated>2026-10-18T21:06:47.001Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why reliable AI memory needs more than embeddings, longer context, or a better retrieval score.</h3><p>An assistant remembers that Sarah works at Acme.</p><p>Then Sarah changes jobs.</p><p>A month later, you ask:</p><p>“Where was Sarah working when we met?”</p><p>The assistant confidently answers with her new employer.</p><p>It retrieved a real fact. It identified the right person. It may even have selected the most recently updated record.</p><p>And it still answered the wrong question.</p><p>This is a useful way to think about AI memory failures: sometimes the problem is not forgetting. It is remembering something without preserving the conditions under which it is relevant.</p><p>For an engineering design, I would treat <strong>validity</strong> as a first-class concern — not metadata added after retrieval.</p><h3>One Sentence Can Contain Two Different Timelines</h3><p>Consider this message:</p><blockquote><em>“I moved to Pune in March.”</em></blockquote><p>Suppose the user says it in June.</p><p>There are at least two distinct times to represent:</p><ul><li>When the move occurred.</li><li>When the system learned about the move.</li></ul><p>These timestamps answer different questions.</p><p>“Where did I live in April?” asks about the world.</p><p>“What did you know about my location in April?” asks about the system’s knowledge at that point.</p><p>Treating both as a single message timestamp loses that distinction.</p><p>This is the motivation behind a bitemporal representation: separate valid time — when a fact held in the world — from transaction or ingestion time — when the system recorded it. Graphiti’s architecture explicitly distinguishes these timelines and tracks validity and record lifecycle timestamps.[<a href="https://arxiv.org/html/2501.13956v1">arxiv</a>]</p><p>This is not a claim that every assistant needs a temporal graph database. It is a claim that the data model needs to distinguish questions the product promises to answer.</p><h3>“Outdated” Does Not Mean “Useless”</h3><p>Imagine these statements arriving over time:</p><blockquote><em>“I work at Acme.”</em></blockquote><blockquote><em>“I’m joining Nova next month.”</em></blockquote><blockquote><em>“My start date moved back two weeks.”</em></blockquote><blockquote><em>“Actually, I decided to stay at Acme.”</em></blockquote><p>A memory pipeline that simply overwrites current_employer risks collapsing several different concepts:</p><ul><li>An existing state.</li><li>An intention.</li><li>A revised plan.</li><li>A canceled transition.</li></ul><p>My proposed design rule is simple:</p><p>Do not turn every newly extracted sentence into an immediately valid replacement fact.</p><p>A plan is not an event. A correction is not always a new state. A later message can change how an earlier message should be interpreted.</p><p>Temporal memory systems such as Graphiti preserve superseded facts through invalidation rather than simply deleting them, allowing current and historical relationships to remain distinguishable.[<a href="https://www.getzep.com/platform/graphiti/">getzep</a>][<a href="https://help.getzep.com/graphiti/getting-started/overview">help.getzep</a>]</p><p>But preserving history is only part of the design. The reader must also choose the right historical state for the question.</p><h3>Store Evidence Separately From Current Beliefs</h3><p>I would separate three layers in a memory implementation.</p><h3>1. Source evidence</h3><p>What was actually said?</p><p>Keep the source message, speaker, timestamp, and enough surrounding context to interpret it.</p><h3>2. Extracted assertions</h3><p>What does the system think the source claims?</p><p>For example:</p><pre>{<br>  &quot;subject_id&quot;: &quot;person_sarah_01&quot;,<br>  &quot;predicate&quot;: &quot;employed_by&quot;,<br>  &quot;object_id&quot;: &quot;company_acme_01&quot;,<br>  &quot;assertion_type&quot;: &quot;reported_state&quot;,<br>  &quot;recorded_at&quot;: &quot;2026-06-10&quot;,<br>  &quot;valid_from&quot;: null,<br>  &quot;valid_to&quot;: null,<br>  &quot;source_message_id&quot;: &quot;message_184&quot;<br>}</pre><p>This is an illustrative schema, not a complete temporal database design.</p><p>The null dates matter. If the source does not establish when employment started, the extractor should not invent a start date.</p><h3>3. Derived views</h3><p>What should the assistant currently believe?</p><p>A current-state view can select the assertions relevant to “Where does Sarah work now?”</p><p>A historical view can select evidence relevant to “Where did Sarah work when we met?”</p><p>These views should point back to the underlying evidence.</p><p>The architectural goal is to make interpretations revisable without losing the information used to produce them.</p><h3>Identity Needs the Same Discipline</h3><p>Now introduce two people named Sarah.</p><p>One is your colleague. The other is your sister.</p><p>A name alone cannot distinguish them in this example. But “name plus relationship” is not necessarily a permanent identifier either: colleagues become friends, managers change, and relationships can overlap.</p><p>I would use stable internal entity IDs, with names and relationships stored as evidence-backed attributes.</p><p>For ambiguous references, the write path should have more than two options.</p><p>Instead of only:</p><pre>Create entity<br>or<br>Merge into existing entity</pre><p>allow:</p><pre>Create entity<br>Link with sufficient evidence<br>Hold as unresolved<br>Ask for clarification</pre><p>A system that can preserve uncertainty does not have to turn every ambiguous mention into a confident merge.</p><h3>Retrieval Should Select a State, Not Just a Sentence</h3><p>Consider four questions:</p><ol><li>Where does Sarah work?</li><li>Where did Sarah work last spring?</li><li>What did Sarah say about changing jobs?</li><li>What did we believe about Sarah’s employer before her correction?</li></ol><p>They concern the same person and topic, but they ask for different evidence.</p><p>For this design, I would represent query intent explicitly enough to distinguish:</p><ul><li>Current state.</li><li>State at a specified time.</li><li>Events or changes.</li><li>Previously recorded beliefs.</li></ul><p>The retrieval pipeline could then look like this:</p><pre>Question<br>   ↓<br>Entity resolution<br>   ↓<br>Temporal and question-intent interpretation<br>   ↓<br>Candidate retrieval<br>   ↓<br>Validity and provenance checks<br>   ↓<br>Evidence selection<br>   ↓<br>Answer or abstention</pre><p>This is a proposed decomposition, not an argument that every box requires a separate model call.</p><p>The important part is that “most similar” and “appropriate evidence” are not treated as interchangeable.</p><h3>Cross-References Should Not Become Conflicting Copies</h3><p>Suppose a fact concerns both Sarah and Acme.</p><p>It should be discoverable through either entity.</p><p>For a structured implementation, I would prefer one canonical fact record referenced by both entity indexes rather than two independently editable copies:</p><pre>Sarah index ──┐<br>              ├── fact_271<br>Acme index ───┘</pre><p>That gives corrections one authoritative target.</p><p>Materialized copies may still be useful for performance, but then their consistency needs to be an explicit engineering responsibility.</p><p>A chapter metaphor can make navigation intuitive. It should not accidentally dictate an update model that is difficult to maintain.</p><h3>A Good Retrieval Score Can Hide a Broken Write Path</h3><p>Imagine a memory system with three stages:</p><pre>Conversation → Extraction → Filing → Retrieval</pre><p>A wrong answer could originate anywhere:</p><ul><li>The extractor missed the relevant statement.</li><li>The entity resolver chose the wrong person.</li><li>The fact was assigned to the wrong attribute.</li><li>A correction was applied too broadly.</li><li>Retrieval selected the wrong time period.</li></ul><p>Improving the retriever will not reveal which of these happened.</p><p>Before choosing the next model to train, I would run an ablation:</p><p>ConditionExtractionFilingAHuman-verifiedHuman-verifiedBModel-generatedHuman-verifiedCHuman-verifiedModel-generatedDModel-generatedModel-generated</p><p>Keep the retrieval procedure fixed.</p><p>Condition A is a reference condition for retrieval with verified inputs — not a theoretical upper bound.</p><p>The other conditions help investigate which upstream errors are responsible for the observed failures.</p><p>The evaluation should report those failures separately, rather than reducing every error to “memory recall.”</p><h3>Test the Questions the System Should Refuse to Answer</h3><p>A useful test set should contain more than questions with known, neatly stored answers.</p><p>I would include:</p><ul><li>An unknown start date.</li><li>Two unresolved people with the same name.</li><li>A job offer that was never accepted.</li><li>A correction with no stated effective date.</li><li>Conflicting statements from different speakers.</li><li>A question whose answer was never provided.</li></ul><p>For each case, define acceptable behavior in advance.</p><p>Sometimes the correct output is a fact.</p><p>Sometimes it is a qualified answer.</p><p>Sometimes it is:</p><blockquote><em>“I know Sarah changed employers, but I don’t have enough information to determine where she worked on that date.”</em></blockquote><p>An evaluation that always requires a factual answer cannot tell you whether the system handles missing evidence appropriately.</p><h3>Measure the Assistant, Not Only the Memory Index</h3><p>Retrieval evaluation is useful, but it does not automatically establish the performance of the complete assistant.</p><p>For a proposed system, I would track at least three groups of measurements.</p><h3>Write-path measurements</h3><ul><li>Extracted-fact accuracy.</li><li>Entity assignment accuracy.</li><li>Attribute assignment accuracy.</li><li>Correction handling.</li><li>Update latency.</li></ul><h3>Read-path measurements</h3><ul><li>Relevant evidence retrieval.</li><li>Correct temporal selection.</li><li>Entity-confusion errors.</li><li>Unsupported-answer frequency.</li><li>Answerable versus unanswerable behavior.</li></ul><h3>End-to-end measurements</h3><ul><li>Final answer quality.</li><li>Tool calls per question.</li><li>Retrieved context tokens.</li><li>Response latency.</li><li>Recovery after incorrect writes.</li></ul><p>A selective reader may reduce context size, but that remains an empirical question until the full reading loop is integrated and measured.</p><p>Likewise, an extra retrieval step might save prompt tokens while adding latency. The relevant trade-off belongs in the end-to-end results, not only in an architectural diagram.</p><h3>Memory Is a Data-Lifecycle Problem</h3><p>Semantic retrieval remains useful. It can help locate evidence when the user’s wording differs from the stored text.</p><p>But my design recommendation is to place it inside a broader lifecycle:</p><pre>Capture<br>→ Interpret<br>→ Resolve identity<br>→ Record provenance<br>→ Establish validity<br>→ Retrieve<br>→ Verify<br>→ Correct</pre><p>Existing systems already address parts of this lifecycle. Graphiti documents temporal invalidation and historical relationships; Mem0 exposes a memory-history interface for tracking how an individual memory has changed. Neither should be reduced to “just a pile of embeddings.”[<a href="https://help.getzep.com/graphiti/getting-started/overview">help.getzep</a>][<a href="https://docs.mem0.ai/api-reference/memory/history-memory">docs.mem0</a>]</p><p>The useful engineering question is therefore not simply:</p><p>“How do we give an assistant memory?”</p><p>It is:</p><p>“What evidence does this assistant preserve, how does it revise that evidence, and under what conditions is it allowed to use it?”</p><p>An assistant that remembers yesterday’s fact is useful.</p><p>An assistant that can explain whether the fact is still true — and recognize when it cannot tell — is a more demanding system to build.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69ed31892210" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications]]></title>
            <link>https://medium.com/@anasahmedadawy/your-ai-prototype-is-not-a-production-system-the-engineering-behind-reliable-ai-applications-5e0e205dfe67?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e0e205dfe67</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 03:37:15 GMT</pubDate>
            <atom:updated>2026-10-18T03:37:15.363Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications</h3><p><em>From containerization and deployment pipelines to observability, failure recovery, and scaling — what it actually takes to operate AI software in the real world.</em></p><p><strong>By Ahmed Adawy</strong></p><p>An AI application can look impressive during a demonstration and still be a long way from production-ready.</p><p>The model returns a useful answer. The API responds correctly. The retrieval pipeline finds relevant documents. Everything works when one developer runs the application locally.</p><p>Then the environment changes.</p><p>A model provider becomes slow. Several requests arrive simultaneously. A deployment introduces an incompatible dependency. The database reaches its connection limit. A background task fails after partially processing a document. An application restart interrupts work that was never persisted.</p><p>None of these problems necessarily means the underlying AI model is bad.</p><p>They reveal something more fundamental: the model is only one component of a software system.</p><p>The engineering challenge is to make the entire system predictable enough to deploy, diagnose, secure, scale, and maintain.</p><p>That distinction is the central idea behind production AI engineering.</p><p>My book, <strong>The AI Engineer — Volume II: Deploy. Scale. Operate.</strong>, focuses on the engineering practices that become essential after an AI application moves beyond the prototype stage.</p><p><a href="https://leanpub.com/theaiengineervolumeiideployscaleoperate">Explore the book on Leanpub</a></p><h3>1. The Real Architecture Is Larger Than the Model</h3><p>Consider a document intelligence service that accepts a file, extracts its text, generates embeddings, retrieves relevant information, and sends selected context to an LLM.</p><p>The visible feature might be summarized as:</p><pre>Document → Embeddings → Retrieval → LLM → Answer</pre><p>But a deployable application has a much larger execution path:</p><pre>Client Request<br>      |<br>      v<br>API Gateway / Application API<br>      |<br>      v<br>Authentication and Input Validation<br>      |<br>      v<br>Application Service<br>      |<br>      +------&gt; Database<br>      |<br>      +------&gt; Object Storage<br>      |<br>      +------&gt; Embedding Provider<br>      |<br>      +------&gt; Vector Index<br>      |<br>      +------&gt; LLM Provider<br>      |<br>      v<br>Response and Operational Telemetry</pre><p>Every external dependency introduces a potential failure boundary. Every additional processing step can contribute latency. Every stored document introduces questions about access control, retention, and data integrity.</p><p>The system therefore needs to answer questions that a successful model call cannot answer:</p><ul><li>What happens when an upstream service times out?</li><li>How are duplicate requests handled?</li><li>Which component is responsible for retrying a failed operation?</li><li>How do we know whether a deployment is healthy?</li><li>Can we identify the slowest stage of a request?</li><li>What happens to work that is interrupted by a restart?</li><li>How do we prevent logs from exposing sensitive information?</li></ul><p>These are architectural questions, not merely operational details.</p><h3>2. Containerization Is About Reproducibility, Not Just Packaging</h3><p>A Python application depends on more than its source files. Its behavior can depend on the Python runtime, package versions, operating-system libraries, environment variables, filesystem assumptions, and startup configuration.</p><p>Docker provides a way to package an application and its dependencies into a container image. That image becomes a versioned artifact that can move through development, testing, and deployment environments.</p><p>A minimal FastAPI example illustrates the idea:</p><pre>FROM python:3.12-slim</pre><pre>ENV PYTHONDONTWRITEBYTECODE=1<br>ENV PYTHONUNBUFFERED=1</pre><pre>WORKDIR /app</pre><pre>COPY requirements.txt .</pre><pre>RUN pip install --no-cache-dir -r requirements.txt</pre><pre>COPY . .</pre><pre>EXPOSE 8000</pre><pre>CMD [&quot;uvicorn&quot;, &quot;app.main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8000&quot;]</pre><p>This assumes the application entry point is app/main.py, the FastAPI instance is named app, and the dependency file contains the required packages.</p><p>Build and run it:</p><pre>docker build -t ai-service .<br>docker run --rm -p 8000:8000 ai-service</pre><p>The commands are straightforward. The engineering decisions around them matter more.</p><p>For example, a production image should use controlled dependency versions, avoid copying secrets into the image, exclude irrelevant files through .dockerignore, and consider running as a non-root user. The build process should also account for vulnerability scanning and the maintenance of base images.</p><p>Containerization does not make software automatically secure or reproducible. It gives the team a consistent unit to build, inspect, test, distribute, and deploy.</p><p>That consistency is what makes it valuable.</p><h3>3. CI/CD Should Verify the Application, Not Merely Deploy It</h3><p>A deployment pipeline is often described as a sequence of automation steps:</p><pre>Commit<br>  ↓<br>Tests<br>  ↓<br>Build<br>  ↓<br>Publish Image<br>  ↓<br>Deploy</pre><p>That is a useful starting point, but a successful deployment command is not proof that the application is healthy.</p><p>A stronger pipeline separates several concerns:</p><ol><li><strong>Code quality:</strong> static checks and formatting rules.</li><li><strong>Behavior:</strong> unit and integration tests.</li><li><strong>Artifact integrity:</strong> build and inspect the container image.</li><li><strong>Security:</strong> scan dependencies and images where appropriate.</li><li><strong>Deployment:</strong> release a known artifact to the target environment.</li><li><strong>Verification:</strong> check application health and important behavior.</li><li><strong>Recovery:</strong> define what happens if verification fails.</li></ol><p>AI applications need particular care around testing.</p><p>A test suite that calls a live LLM for every assertion can become slow, expensive, and nondeterministic. Many application behaviors can instead be tested using controlled fake clients.</p><p>For example, a service should be testable independently of the external model provider:</p><pre>from typing import Protocol<br></pre><pre>class TextGenerator(Protocol):<br>    def generate(self, prompt: str) -&gt; str:<br>        ...<br></pre><pre>class SummaryService:<br>    def __init__(self, generator: TextGenerator) -&gt; None:<br>        self.generator = generator</pre><pre>    def summarize(self, text: str) -&gt; str:<br>        if not text.strip():<br>            raise ValueError(&quot;Text must not be empty&quot;)</pre><pre>        return self.generator.generate(<br>            f&quot;Summarize the following text:\n{text}&quot;<br>        )<br></pre><pre>class FakeGenerator:<br>    def generate(self, prompt: str) -&gt; str:<br>        return &quot;Deterministic test summary&quot;<br></pre><pre>service = SummaryService(FakeGenerator())</pre><pre>result = service.summarize(&quot;A document about distributed systems.&quot;)</pre><pre>assert result == &quot;Deterministic test summary&quot;</pre><p>This example tests application logic without making a network request or consuming model tokens.</p><p>It does not verify the real provider’s behavior, so provider integration tests are still needed. The point is to test each boundary at the appropriate level.</p><p>A dependable pipeline combines fast deterministic tests with targeted integration tests and deployment checks.</p><h3>4. Latency Must Be Measured Across the Entire Request</h3><p>Suppose an AI endpoint takes 2.4 seconds to return a response.</p><p>That number alone does not tell us what needs to be optimized.</p><p>The request may spend time validating input, querying a database, generating an embedding, retrieving documents, waiting for an LLM, and serializing the result.</p><p>A simplified latency model is:</p><p>Ttotal=Tvalidation+Tdatabase+Tembedding+Tretrieval+TLLM+TresponseT_{\text{total}} = T_{\text{validation}} + T_{\text{database}} + T_{\text{embedding}} + T_{\text{retrieval}} + T_{\text{LLM}} + T_{\text{response}}</p><p>This is an approximation. Parallel operations, queueing, retries, and overlapping work can make real request timing more complicated.</p><p>Still, the model helps establish an important rule: measure individual stages before optimizing the system.</p><p>If the LLM accounts for most of the request time, optimizing a small Python function may have almost no effect on the user’s experience.</p><p>If database connection acquisition dominates the delay, adding more application workers could make the problem worse by increasing pressure on the database.</p><p>If embedding generation is repeated for identical inputs, caching may help — provided the cache key, model version, and invalidation behavior are designed correctly.</p><p>Performance engineering begins with a hypothesis, a measurement, and a controlled change. It should not begin with adding infrastructure simply because the application feels slow.</p><h3>5. Retries Can Recover From Failure — or Amplify It</h3><p>Network calls fail. Providers throttle requests. Temporary errors occur.</p><p>Retries can help, but they are not automatically safe.</p><p>Imagine a service receiving 100 requests per second. If every request fails and each request retries three times, the upstream dependency may receive several times the original traffic precisely when it is already struggling.</p><p>A basic exponential backoff strategy increases the delay between attempts:</p><p>dn=min⁡(dmax⁡,d0⋅2n)d_n = \min(d_{\max}, d_0 \cdot 2^n)</p><p>Here, d0d_0 is the initial delay, nn is the retry index, and dmax⁡d_{\max} is the maximum delay.</p><p>In practice, randomized jitter is often added to reduce synchronized retries from many clients.</p><p>Retries should also be bounded by an overall deadline. The application needs to distinguish transient failures from permanent errors, respect provider rate limits, and avoid retrying non-idempotent operations without a safe duplication strategy.</p><p>For operations that create records, charge money, enqueue jobs, or trigger external side effects, idempotency can be especially important.</p><p>The correct question is not simply, “Should we retry?”</p><p>It is: “Which failures are retryable, how many attempts are acceptable, what is the total time budget, and what prevents a retry from duplicating work?”</p><p>That is production engineering.</p><h3>6. Observability Is How You Find Out What Actually Happened</h3><p>A log message saying Request failed is rarely enough to diagnose a distributed application.</p><p>Useful observability combines three complementary signals.</p><p><strong>Logs</strong> provide event-level context: which operation failed, what type of error occurred, and which request was involved.</p><p><strong>Metrics</strong> reveal trends: request rate, error rate, latency percentiles, resource usage, queue depth, and dependency failures.</p><p><strong>Traces</strong> help follow a request through multiple components and identify where time was spent.</p><p>For an AI service, additional measurements can include:</p><ul><li>Model-provider latency and timeout rate.</li><li>Input and output token counts.</li><li>Embedding generation duration.</li><li>Retrieval duration and result counts.</li><li>Retry frequency and rate-limit responses.</li><li>Background-job completion and failure rates.</li><li>Estimated cost per request, where reliable usage and pricing data are available.</li></ul><p>These measurements help distinguish different failure modes.</p><p>A rising response time with normal database latency may indicate a model-provider issue. Increasing queue depth may indicate that background workers cannot keep up with incoming work. A growing error rate after deployment may point to a configuration or dependency change.</p><p>Observability does not prevent every failure. It reduces the time required to understand and respond to failures.</p><p>One caution is essential: operational visibility must not become uncontrolled data collection. Avoid logging credentials, access tokens, full private documents, or sensitive prompts without a justified and carefully governed reason.</p><h3>7. Kubernetes Is a Tool, Not a Production Strategy</h3><p>Kubernetes can manage containerized workloads, schedule services, support health-based restarts, and provide mechanisms for scaling and service discovery.</p><p>But adopting Kubernetes does not automatically make an application reliable.</p><p>A service can still have poor health checks, unsafe database migrations, missing alerts, weak access controls, or no tested recovery procedure while running inside a sophisticated cluster.</p><p>The operational complexity also matters. Teams need to understand resource requests and limits, deployment configuration, service networking, secrets, storage, observability, and the consequences of scaling.</p><p>For a small application, a managed application platform may be the more sensible choice. For a system with multiple workloads, complex scheduling needs, established operational practices, and a clear requirement for orchestration, Kubernetes may be appropriate.</p><p>The decision should follow the problem.</p><p>A useful engineering principle is to adopt the simplest infrastructure that satisfies the application’s actual reliability, security, and scaling requirements — and to revisit the decision when those requirements change.</p><h3>8. Scaling Is About Bottlenecks and Trade-Offs</h3><p>Adding more application instances increases available compute capacity only when the application and its dependencies can use that capacity effectively.</p><p>A stateless API may scale horizontally with relatively little difficulty. A database with a limited connection pool may not. An external LLM provider may enforce request or token limits. A vector database may become the dominant source of latency.</p><p>Scaling decisions must therefore consider the entire request path.</p><p>Common techniques include:</p><ul><li>Caching repeated, valid computations.</li><li>Moving long-running tasks into background workers.</li><li>Improving database indexes and query plans.</li><li>Reusing connections safely.</li><li>Applying concurrency limits to expensive dependencies.</li><li>Batching operations when the workload supports it.</li><li>Reducing unnecessary payload sizes.</li><li>Measuring resource usage and queueing under representative load.</li></ul><p>Each technique has costs.</p><p>Caching creates invalidation and consistency problems. More workers can increase database contention. Batching may improve throughput while increasing individual request latency. Aggressive concurrency can exhaust memory or trigger provider throttling.</p><p>The target is not maximum throughput at any cost. It is a system that meets its performance and reliability requirements at a sustainable operational cost.</p><h3>9. Production Readiness Is a Set of Verifiable Conditions</h3><p>Before calling an AI application production-ready, I would want clear answers to questions like these:</p><p><strong>Deployment</strong></p><ul><li>Can the same artifact be promoted through environments?</li><li>Are configuration and secrets handled outside the application image?</li><li>Can a release be verified and, if necessary, rolled back?</li></ul><p><strong>Reliability</strong></p><ul><li>Are timeouts and retry policies bounded?</li><li>What happens when a dependency becomes unavailable?</li><li>Can interrupted background work be recovered safely?</li></ul><p><strong>Security</strong></p><ul><li>Are authentication and authorization enforced?</li><li>Are inputs validated and resource limits defined?</li><li>Are sensitive records and credentials protected?</li></ul><p><strong>Observability</strong></p><ul><li>Can the team investigate a failed request?</li><li>Are latency, errors, and resource saturation measured?</li><li>Do alerts identify actionable problems rather than generate noise?</li></ul><p><strong>Performance</strong></p><ul><li>Have representative workloads been measured?</li><li>Are the main bottlenecks understood?</li><li>Can the system handle expected concurrency without exhausting dependencies?</li></ul><p><strong>Maintenance</strong></p><ul><li>Are tests automated?</li><li>Are architecture and operational assumptions documented?</li><li>Can another engineer understand, deploy, and troubleshoot the system?</li></ul><p>A checklist cannot guarantee reliability, but it can expose missing decisions before users discover them through failures.</p><p>Production readiness is not a label awarded because an application has Docker, Kubernetes, or a CI/CD pipeline. It is the result of engineering choices that can be explained, tested, and verified.</p><h3>10. The Engineering Journey Behind This Book</h3><p>These topics are connected, and they become much more useful when learned as parts of a complete system.</p><p>Containerization establishes a reproducible runtime. CI/CD turns changes into controlled releases. Cloud deployment introduces operational constraints. Observability helps diagnose behavior. Security protects the service and its data. Performance measurements guide scaling decisions. Architecture connects these practices into a maintainable application.</p><p>That progression is the focus of my book:</p><p><strong>The AI Engineer — Volume II: Deploy. Scale. Operate.</strong></p><p>The volume covers Docker and containerization, Linux and production operations, Git and GitHub workflows, CI/CD, cloud deployment, Kubernetes, observability, production security, scaling, architecture patterns, and production-readiness practices. It also includes a final engineering challenge and reference appendices.</p><p>It is intended for Python developers, backend engineers, AI engineers, and software developers who want to understand what it takes to operate AI applications beyond a local development environment.</p><p>The goal is not to collect tools or copy infrastructure configurations without understanding them. It is to build the engineering judgment needed to choose the right approach, recognize failure modes, and make systems easier to operate.</p><p><strong>Read the book on Leanpub:</strong></p><p><a href="https://leanpub.com/theaiengineervolumeiideployscaleoperate">https://leanpub.com/theaiengineervolumeiideployscaleoperate</a></p><h3>Final Thought</h3><p>A model can produce an impressive answer in a few seconds.</p><p>A dependable AI product requires much more: controlled releases, secure data handling, measurable performance, observable failures, and recovery mechanisms that work when conditions are less than ideal.</p><p>That work is not separate from AI engineering.</p><p>It is part of AI engineering.</p><p><strong>Build beyond the prototype. Deploy deliberately. Measure what matters. Design for failure.</strong></p><p><em>Ahmed Adawy is the author of The AI Engineer — Volume II: Deploy. Scale. Operate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e0e205dfe67" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From AI Prototype to Production: The Engineering Skills That Separate AI Developers from AI…]]></title>
            <link>https://medium.com/@anasahmedadawy/from-ai-prototype-to-production-the-engineering-skills-that-separate-ai-developers-from-ai-4c310b5d7e08?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c310b5d7e08</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 17:33:13 GMT</pubDate>
            <atom:updated>2026-10-16T17:33:13.194Z</atom:updated>
            <content:encoded><![CDATA[<h3>From AI Prototype to Production: The Engineering Skills That Separate AI Developers from AI Engineers</h3><h3>From AI Prototype to Production: The Engineering Skills That Separate AI Developers from AI Engineers</h3><p><strong>By Ahmed Adawy</strong></p><p>The AI industry has made it remarkably easy to build a prototype.</p><p>A few lines of Python. An API key. A call to a large language model. Perhaps a vector database, a simple retrieval pipeline, and a user interface.</p><p>Within hours, a developer can demonstrate an application that summarizes documents, answers questions, generates code, or searches a knowledge base.</p><p>But there is a fundamental difference between building an AI demonstration and engineering a system that people can depend on.</p><p>What happens when the input contains thousands of documents instead of ten?</p><p>What happens when two requests attempt to update the same resource?</p><p>What happens when an external model provider becomes unavailable?</p><p>What happens when latency increases, database connections become exhausted, credentials are exposed, or a deployment introduces a regression?</p><p>And perhaps the most important question:</p><p><strong>Can your AI application remain secure, observable, maintainable, and reliable after it leaves your development environment?</strong></p><p>These are not merely machine learning problems. They are software engineering, systems architecture, infrastructure, and operational engineering problems.</p><p>This is where AI engineering becomes a discipline of its own.</p><h3>1. The Real Challenge Is Not Calling an LLM</h3><p>Large language models have changed how developers build software, but they have not eliminated the engineering principles that make software dependable.</p><p>In many cases, they have made those principles even more important.</p><p>Consider a document question-answering application.</p><p>At first glance, its architecture appears straightforward:</p><ol><li>Accept a document.</li><li>Extract its text.</li><li>Generate embeddings.</li><li>Store the vectors.</li><li>Retrieve relevant passages.</li><li>Send the retrieved context to an LLM.</li><li>Return an answer.</li></ol><p>The demonstration is simple. The production system is not.</p><p>A real implementation must account for document validation, processing failures, duplicate submissions, metadata consistency, access control, embedding model changes, retrieval quality, token limits, provider timeouts, retries, caching, monitoring, and cost management.</p><p>Each component introduces assumptions. Each assumption can become a failure mode.</p><p>For example, a retrieval system might return semantically similar passages that belong to a document the current user is not authorized to access. The vector search can be mathematically correct while the application is fundamentally insecure.</p><p>Likewise, an LLM can generate a fluent answer from incomplete context. The response may look convincing even when the underlying retrieval process failed.</p><p>The lesson is simple:</p><p><strong>An AI application is not just a model. It is a system of interacting components, data flows, dependencies, and failure modes.</strong></p><p>Engineering that system requires a broader skill set than prompt writing or API integration alone.</p><h3>2. Build the Software Engineering Foundation First</h3><p>Before optimizing a model or experimenting with a new retrieval strategy, an AI engineer needs to understand the application surrounding it.</p><p>Python remains an essential tool, but production Python engineering goes beyond knowing the language syntax.</p><p>It involves organizing code into meaningful modules, defining interfaces, managing dependencies, separating configuration from implementation, handling errors explicitly, and designing components that can be tested independently.</p><p>A useful architectural principle is separation of concerns.</p><p>For example, an AI application might separate its responsibilities into these layers:</p><pre>Client<br>  |<br>  v<br>FastAPI Application<br>  |<br>  v<br>Application Services<br>  |<br>  +---- Authentication and Authorization<br>  |<br>  +---- Document Processing<br>  |<br>  +---- Retrieval and Ranking<br>  |<br>  +---- LLM Provider Integration<br>  |<br>  v<br>Persistence and External Services</pre><p>This structure is not valuable merely because it looks organized.</p><p>It gives engineers clear places to enforce security rules, test business logic, replace dependencies, and investigate failures.</p><p>A retrieval algorithm should not need to know how HTTP requests are authenticated. An API endpoint should not contain every detail of document processing. A model provider should not determine the entire architecture of the application.</p><p>These boundaries make systems easier to reason about and evolve.</p><p>The first stage of AI engineering is therefore not choosing the most sophisticated model. It is learning how to build a sound software system around the model.</p><h3>3. Design APIs and Data Systems for Real Workloads</h3><p>AI applications frequently expose functionality through APIs.</p><p>FastAPI provides a practical Python framework for building these interfaces, but using a framework is only the beginning.</p><p>A production API must define clear contracts for incoming requests and outgoing responses. It must validate data, communicate errors consistently, enforce permissions, manage dependencies, and avoid exposing internal implementation details.</p><p>Database design matters just as much.</p><p>Imagine a knowledge platform that stores users, documents, processing jobs, extracted text, and references to generated embeddings.</p><p>The system must preserve relationships between these entities. It must distinguish between a document that has been uploaded and one that has finished processing. It must handle failures without leaving records in contradictory states.</p><p>PostgreSQL can provide persistent storage and transactional guarantees, but engineers still need to design the schema, select appropriate constraints, understand indexing, and reason about transaction boundaries.</p><p>Authentication is another critical boundary.</p><p>Identifying a user is not the same as authorizing an action. A valid session does not automatically grant permission to retrieve every document in the system.</p><p>For AI applications, these distinctions are especially important because retrieved information may be passed directly into model context.</p><p>A reliable architecture treats authorization as part of the data access path, not as an optional check added at the user interface.</p><h3>4. Understand Embeddings, Semantic Search, and RAG</h3><p>AI engineers also need to understand the mechanics behind the AI features they build.</p><p>Embeddings represent information as numerical vectors. These representations allow systems to compare items according to the patterns captured by an embedding model.</p><p>For vectors uu and vv, cosine similarity is commonly expressed as:</p><p>cos⁡(u,v)=u⋅v∥u∥∥v∥\operatorname{cos}(u,v)= \frac{u\cdot v}{\|u\|\|v\|}</p><p>This measures the cosine of the angle between two nonzero vectors. It is frequently used to estimate directional similarity between embeddings.</p><p>However, the equation alone does not produce a good semantic search system.</p><p>The quality of retrieval depends on several interacting decisions:</p><ul><li>Which embedding model is appropriate for the data?</li><li>How should documents be divided into chunks?</li><li>What metadata must accompany each chunk?</li><li>How are candidate passages ranked?</li><li>How are access permissions enforced?</li><li>How is retrieval quality evaluated?</li><li>What happens when the corpus or embedding model changes?</li></ul><p>Retrieval-Augmented Generation, or RAG, adds another stage to the process.</p><p>Instead of relying exclusively on information encoded in model parameters, a RAG application retrieves relevant external information and supplies it as context for generation.</p><p>A simplified pipeline looks like this:</p><pre>Question<br>   |<br>   v<br>Query Embedding<br>   |<br>   v<br>Candidate Retrieval<br>   |<br>   v<br>Filtering and Ranking<br>   |<br>   v<br>Context Construction<br>   |<br>   v<br>LLM Generation<br>   |<br>   v<br>Answer and References</pre><p>Every stage affects the final result.</p><p>Poor chunking can fragment important information. Weak retrieval can omit the passage that contains the answer. Excessive context can increase cost and distract the model. Missing access controls can expose information. Unclear evaluation criteria can make improvements difficult to measure.</p><p>A serious AI engineer must therefore reason about the entire retrieval and generation pipeline, rather than treating RAG as a single API call.</p><h3>5. Reliability Is a Design Property</h3><p>A production system should not depend on the assumption that every component will always work.</p><p>External model providers can time out. Networks can fail. Documents can be malformed. Database queries can exceed their expected duration. Background jobs can be interrupted.</p><p>Reliability engineering begins by making these possibilities explicit.</p><p>For example, consider a document-processing workflow:</p><pre>UPLOADED<br>    |<br>    v<br>PROCESSING<br>    |<br>    +---- Success ----&gt; INDEXED<br>    |<br>    +---- Failure ----&gt; FAILED<br>                           |<br>                           v<br>                    Retry or Review</pre><p>The state model helps define what the application should do when processing succeeds, fails, or needs to be attempted again.</p><p>Retries require particular care. Retrying a transient network error may be useful, but blindly repeating every operation can waste resources or produce duplicate effects.</p><p>Engineers need to distinguish transient failures from permanent ones, establish reasonable timeouts, and consider idempotency where operations may be repeated.</p><p>Testing is equally important.</p><p>Unit tests can validate isolated business rules. Integration tests can verify interactions between components. API tests can examine request validation and authorization behavior. Failure-oriented tests can check what happens when dependencies return errors or stop responding.</p><p>The goal is not simply to increase a coverage percentage.</p><p>The goal is to establish evidence that the system behaves as intended under both normal and abnormal conditions.</p><h3>6. Performance Engineering Means Measuring the Whole Pipeline</h3><p>AI application performance is rarely determined by one component alone.</p><p>A request may involve:</p><ul><li>API request handling.</li><li>Database queries.</li><li>Embedding generation.</li><li>Vector retrieval.</li><li>Context preparation.</li><li>An external LLM request.</li><li>Response serialization.</li></ul><p>Each stage contributes to the total latency.</p><p>A useful first step is to measure the stages independently instead of guessing which component is slow.</p><p>Caching may reduce repeated computation. Background jobs may move expensive document processing outside the request-response cycle. Database indexes may improve retrieval of structured metadata. Batching may improve throughput for suitable workloads.</p><p>But each optimization introduces trade-offs.</p><p>Caching requires a strategy for invalidation. Background processing requires job-state management and failure recovery. Batching can improve throughput while increasing individual request latency. More aggressive concurrency can overload downstream services.</p><p>Performance engineering is therefore a process of measurement, diagnosis, and controlled change.</p><p>A system that is fast in a local demonstration may behave very differently under concurrent requests or a growing document collection.</p><p>The engineer’s job is to understand why.</p><h3>7. Building the Application Is Only Half the Journey</h3><p>Even a well-designed application remains unfinished if it cannot be deployed and operated reliably.</p><p>This is the point where infrastructure and operations become part of AI engineering.</p><p>Docker helps package an application and its dependencies into a reproducible environment. Linux knowledge helps engineers understand the systems on which applications run. Git and GitHub support collaboration and version control. Continuous integration and continuous deployment workflows can automate checks and release processes.</p><p>Cloud infrastructure introduces additional considerations: configuration management, networking, permissions, resource allocation, availability, and cost.</p><p>Kubernetes can help orchestrate containerized workloads, but adopting it also introduces operational complexity. It is most useful when its capabilities solve real deployment, scaling, or resilience requirements.</p><p>Observability connects the application to its operational behavior.</p><p>Logs provide event details. Metrics help quantify performance and resource use. Traces can help follow requests across service boundaries.</p><p>Together, these signals help answer questions that cannot be resolved by looking at source code alone:</p><ul><li>Which stage is responsible for increasing latency?</li><li>Are failures isolated to a specific dependency?</li><li>Is the application exhausting database connections?</li><li>Did a new release change error rates?</li><li>Are background jobs accumulating faster than workers can process them?</li></ul><p>Security must continue throughout this process. Production credentials, API endpoints, stored documents, user permissions, and deployment workflows all require deliberate protection.</p><p>The objective is not to use every infrastructure tool available. It is to understand the operational responsibilities that come with running a real system.</p><h3>8. A Structured Learning Path for AI Engineers</h3><p>These topics are connected, which is why learning them as isolated tricks can leave important gaps.</p><p>A more coherent progression moves from software foundations to AI application development and then to production operations.</p><p>This is the idea behind my two-volume series, <em>The AI Engineer</em>.</p><h3>Volume I: Build the Foundations</h3><p><strong>The AI Engineer Bootcamp in a Book</strong> focuses on the engineering foundations required to build AI-powered applications.</p><p>The book’s scope includes:</p><ul><li>Professional Python engineering and project structure.</li><li>FastAPI and backend application design.</li><li>PostgreSQL and persistent data.</li><li>Authentication, authorization, and security.</li><li>Testing, debugging, and reliability.</li><li>Embeddings and semantic search.</li><li>Retrieval-Augmented Generation (RAG).</li><li>LLM application architecture.</li><li>AI processing pipelines.</li><li>Caching, background jobs, and performance engineering.</li></ul><p>The central theme is that AI functionality needs a well-engineered application around it.</p><p>Explore Volume I:</p><p><a href="https://leanpub.com/theaiengineerbootcampinabook">https://leanpub.com/theaiengineerbootcampinabook</a></p><h3>Volume II: Deploy, Scale, Operate</h3><p><strong>The AI Engineer — Volume II: Deploy. Scale. Operate.</strong> continues the journey into deployment and production operations.</p><p>Its focus includes:</p><ul><li>Docker and containerization.</li><li>Linux and production operations.</li><li>Git, GitHub, and engineering workflows.</li><li>Continuous integration and continuous deployment.</li><li>Cloud deployment.</li><li>Kubernetes and cloud-native engineering.</li><li>Observability and monitoring.</li><li>Production security.</li><li>Scaling strategies and architecture patterns.</li><li>Production readiness and an engineering capstone.</li></ul><p>The emphasis shifts from designing the application to understanding how to deploy, monitor, protect, and operate it.</p><p>Explore Volume II:</p><p><a href="https://leanpub.com/theaiengineervolumeiideployscaleoperate">https://leanpub.com/theaiengineervolumeiideployscaleoperate</a></p><p>Together, the two volumes present a progression from AI application foundations toward the engineering practices involved in running production-oriented systems.</p><p>They are intended for Python developers, backend engineers, and aspiring AI engineers who want to connect application development with infrastructure and operational thinking.</p><h3>9. The Mindset That Matters Most</h3><p>The most valuable skill in AI engineering is not memorizing a list of frameworks.</p><p>Frameworks change. Models improve. Infrastructure platforms evolve. New libraries simplify tasks that once required substantial custom code.</p><p>What remains valuable is the ability to reason about systems.</p><p>Can you identify the boundaries between components?</p><p>Can you explain where a failure originated?</p><p>Can you distinguish a retrieval problem from a generation problem?</p><p>Can you evaluate a performance improvement rather than merely assume it worked?</p><p>Can you protect data when requests pass through multiple services?</p><p>Can you deploy a change while preserving a reliable way to detect and recover from regressions?</p><p>These questions reveal the difference between making an AI feature work once and engineering a system that can be maintained over time.</p><p>AI engineering brings together software design, data management, retrieval, model integration, testing, security, performance, infrastructure, and operations.</p><p>No single tool replaces that understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Cncng5l-Ajj5ul3ilgFog.png" /></figure><h3>Conclusion: Engineer the System, Not Just the Demo</h3><p>The rapid growth of generative AI has lowered the barrier to experimentation. That is a good thing. More developers can explore ideas, build prototypes, and create useful applications.</p><p>But experimentation and production engineering have different requirements.</p><p>A prototype demonstrates possibility. A production system must also account for failure, security, performance, maintenance, and operational responsibility.</p><p>That is why the path toward AI engineering should extend beyond model APIs and prompts. It should include the software foundations that make applications maintainable, the AI concepts that make retrieval and generation understandable, and the operational practices that make deployment sustainable.</p><p>My two-volume series, <em>The AI Engineer</em>, is organized around that progression: first, build the foundations; then, learn the engineering practices involved in deployment, scaling, and operations.</p><p><strong>Start with Volume I: Build the Foundations</strong></p><p><a href="https://leanpub.com/theaiengineerbootcampinabook">https://leanpub.com/theaiengineerbootcampinabook</a></p><p><strong>Continue with Volume II: Deploy. Scale. Operate.</strong></p><p><a href="https://leanpub.com/theaiengineervolumeiideployscaleoperate">https://leanpub.com/theaiengineervolumeiideployscaleoperate</a></p><p>The goal is not simply to build something that works on your machine.</p><p>It is to develop the engineering judgment to understand the system, improve it, and prepare it for the realities of production.</p><p><em>Build the foundations. Understand the system. Engineer for production.</em></p><p>— Ahmed Adawy</p><p><strong>Topics:</strong> Artificial Intelligence, AI Engineering, Python, FastAPI, Large Language Models, LLM Applications, Retrieval-Augmented Generation, Semantic Search, Embeddings, Backend Engineering, Software Architecture, Docker, Kubernetes, CI/CD, Cloud Computing, Production Systems</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c310b5d7e08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From AI Prototype to Production System: The Engineering Challenges Nobody Can Ignore]]></title>
            <link>https://medium.com/@anasahmedadawy/from-ai-prototype-to-production-system-the-engineering-challenges-nobody-can-ignore-432b07b0411d?source=rss-1f846f48bb7d------2</link>
            <guid isPermaLink="false">https://medium.com/p/432b07b0411d</guid>
            <dc:creator><![CDATA[​Ahmed Adawy | AI Systems & Python]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 11:35:08 GMT</pubDate>
            <atom:updated>2026-10-10T11:35:08.619Z</atom:updated>
            <content:encoded><![CDATA[<h3>Docker, CI/CD, Kubernetes, Observability, Security, and the Architecture Behind Reliable AI Applications</h3><p>An AI application can work perfectly in a notebook, return impressive results during a demonstration, and still fail within minutes of reaching production.</p><p>The model may be accurate. The prompts may be carefully designed. The retrieval pipeline may return relevant documents. Yet the system can collapse under concurrent requests, expose sensitive information through poorly designed access controls, exhaust its database connections, or become prohibitively expensive when inference traffic increases.</p><p>This is the uncomfortable distinction between building an AI feature and engineering an AI system.</p><p><strong>Production AI engineering is not simply the process of deploying a model. It is the discipline of making an entire software system reliable, observable, secure, maintainable, and economically sustainable under real operating conditions.</strong></p><p>That distinction becomes increasingly important as applications combine large language models (LLMs), vector databases, embedding services, external APIs, background workers, persistent storage, and user-facing interfaces.</p><p>Each component introduces dependencies. Each dependency introduces failure modes. And every architectural decision affects the behavior of the complete system.</p><p>Let’s examine what it actually takes to move an AI application from a promising prototype to a production-ready platform.</p><h3>1. The Architecture Problem: An AI Application Is a Distributed System</h3><p>Consider a knowledge assistant that allows users to upload documents and ask questions about their contents.</p><p>A simplified request lifecycle might look like this:</p><ol><li>The client sends a question to a FastAPI backend.</li><li>Authentication middleware identifies the user.</li><li>The application generates a query embedding.</li><li>A vector index retrieves semantically similar document chunks.</li><li>A filtering layer verifies which retrieved documents the user is authorized to access.</li><li>The application constructs a context window.</li><li>An LLM generates an answer.</li><li>The backend records relevant operational metadata and returns the response.</li></ol><p>Even this relatively simple workflow crosses multiple system boundaries.</p><p>The embedding service can time out. The vector database can become overloaded. The retrieved context can exceed the model’s token budget. The LLM provider can return a rate-limit response. The client can disconnect while the model is still generating output.</p><p>A request that appears to be one operation from the user’s perspective is actually a chain of dependent operations with different latency distributions, resource requirements, and failure characteristics.</p><p>If the application performs all of these operations synchronously, its response time can be approximated as:</p><p>Trequest=Tauth+Tembed+Tretrieve+TLLM+ToverheadT_{\text{request}} = T_{\text{auth}} + T_{\text{embed}} + T_{\text{retrieve}} + T_{\text{LLM}} + T_{\text{overhead}}</p><p>This approximation assumes sequential execution. Parallel operations require a different model, but the principle remains: every critical dependency contributes to the user-visible experience.</p><p>More importantly, a slow dependency can consume resources throughout the entire request lifecycle.</p><p>If an LLM call takes 12 seconds, the backend may hold a request slot, memory, tracing context, and possibly other resources while waiting for a response. Under sufficient concurrency, the application can exhaust its capacity even when the CPU is not particularly busy.</p><p>The solution is not automatically to add more servers.</p><p>First, identify the bottleneck. Then determine whether the operation belongs on the synchronous request path at all.</p><h3>2. Latency, Concurrency, and the Hidden Cost of Blocking Operations</h3><p>One of the most common mistakes in Python backend engineering is confusing asynchronous syntax with asynchronous behavior.</p><p>Consider this simplified example:</p><pre>from fastapi import FastAPI<br>import requests</pre><pre>app = FastAPI()</pre><pre>@app.get(&quot;/generate&quot;)<br>async def generate():<br>    response = requests.post(<br>        &quot;https://example.com/llm&quot;,<br>        json={&quot;prompt&quot;: &quot;Explain vector embeddings&quot;},<br>        timeout=30,<br>    )<br>    return response.json()</pre><p>The endpoint is declared with async def, but requests.post() is blocking.</p><p>While the function waits for the network operation, it blocks the event loop thread. Under concurrent traffic, this can prevent unrelated asynchronous operations from progressing efficiently.</p><p>An asynchronous HTTP client provides a better foundation:</p><pre>import httpx<br>from fastapi import FastAPI, HTTPException</pre><pre>app = FastAPI()</pre><pre>@app.get(&quot;/generate&quot;)<br>async def generate():<br>    try:<br>        async with httpx.AsyncClient(timeout=20.0) as client:<br>            response = await client.post(<br>                &quot;https://example.com/llm&quot;,<br>                json={&quot;prompt&quot;: &quot;Explain vector embeddings&quot;},<br>            )</pre><pre>            response.raise_for_status()<br>            return response.json()</pre><pre>    except httpx.TimeoutException:<br>        raise HTTPException(<br>            status_code=504,<br>            detail=&quot;The upstream AI service timed out&quot;,<br>        )</pre><pre>    except httpx.HTTPError:<br>        raise HTTPException(<br>            status_code=502,<br>            detail=&quot;The upstream AI service failed&quot;,<br>        )</pre><p>This illustrates the principle, not a complete production implementation. A real service should generally reuse a managed HTTP client rather than create one for every request, and it must also account for provider-specific rate limits, retries, cancellation, and response validation.</p><p>Asynchronous I/O helps when a workload spends significant time waiting for network or other asynchronous operations. It does not make CPU-heavy work disappear, increase an upstream provider’s capacity, or guarantee lower latency under every workload.</p><p>For CPU-intensive processing, you may need process workers, task queues, specialized execution services, or a different computational strategy.</p><p>For LLM applications, you should also distinguish between several latency measurements:</p><ul><li><strong>Time to first token (TTFT):</strong> How long the user waits before generation begins.</li><li><strong>Inter-token latency:</strong> How quickly subsequent tokens arrive.</li><li><strong>End-to-end latency:</strong> How long the complete operation takes.</li><li><strong>Queueing delay:</strong> How long a request waits before processing starts.</li></ul><p>These measurements describe different bottlenecks. A system can have acceptable average response times while delivering a terrible experience to users at the tail of the latency distribution.</p><p>That is why production engineering requires percentiles, not just averages.</p><h3>3. Docker: Reproducibility Before Scalability</h3><p>A system that works only in its developer’s environment is not ready for deployment.</p><p>Python applications are particularly vulnerable to environment differences: interpreter versions, system libraries, package versions, environment variables, file paths, and native dependencies can all affect runtime behavior.</p><p>Docker addresses part of this problem by packaging an application and its runtime dependencies into an image.</p><p>A simplified Dockerfile might look like this:</p><pre>FROM python:3.12-slim</pre><pre>ENV PYTHONDONTWRITEBYTECODE=1<br>ENV PYTHONUNBUFFERED=1</pre><pre>WORKDIR /app</pre><pre>COPY requirements.txt .</pre><pre>RUN pip install --no-cache-dir -r requirements.txt</pre><pre>COPY app/ ./app/</pre><pre>RUN useradd --create-home appuser<br>USER appuser</pre><pre>EXPOSE 8000</pre><pre>CMD [<br>  &quot;uvicorn&quot;,<br>  &quot;app.main:app&quot;,<br>  &quot;--host&quot;,<br>  &quot;0.0.0.0&quot;,<br>  &quot;--port&quot;,<br>  &quot;8000&quot;<br>]</pre><p>This example establishes a working baseline, but a hardened production image requires additional decisions.</p><p>You should consider:</p><ul><li>Pinning dependencies and using reproducible builds.</li><li>Separating build-time and runtime dependencies.</li><li>Running as a non-root user.</li><li>Keeping secrets outside the image.</li><li>Scanning dependencies and images for vulnerabilities.</li><li>Using health checks and appropriate shutdown behavior.</li><li>Avoiding unnecessary files and build tools in the final image.</li></ul><p>Containerization does not automatically make an application secure or reproducible. It provides a controlled packaging mechanism that must be combined with sound build and deployment practices.</p><p>A particularly important rule is to treat container images as immutable release artifacts.</p><p>Build an image, identify it by a version or digest, test it, and promote that same artifact through environments. Rebuilding independently for each environment can introduce subtle differences between what was tested and what was deployed.</p><h3>4. CI/CD: Turning Engineering Discipline into an Automated Process</h3><p>Manual deployment is manageable until it becomes inconsistent.</p><p>Someone forgets to run a test. A dependency changes without adequate validation. A production environment receives a different artifact from the one tested locally. A database migration introduces an incompatibility with the currently running application.</p><p>Continuous integration and continuous deployment help reduce these risks by making validation and release processes repeatable.</p><p>A production-oriented pipeline might follow this sequence:</p><pre>Developer Push<br>      |<br>      v<br>Static Checks and Linting<br>      |<br>      v<br>Unit and Integration Tests<br>      |<br>      v<br>Dependency and Security Scans<br>      |<br>      v<br>Build Container Image<br>      |<br>      v<br>Image and Configuration Validation<br>      |<br>      v<br>Deploy to Staging<br>      |<br>      v<br>Smoke Tests and Health Checks<br>      |<br>      v<br>Production Deployment<br>      |<br>      v<br>Monitor and Verify</pre><p>Each stage should answer a specific question.</p><p>Does the code satisfy quality requirements? Do the tests pass? Does the built image start correctly? Can the service access its required dependencies? Does the deployed version behave as expected?</p><p>For AI systems, testing needs to go beyond whether the API returns HTTP 200.</p><p>A response can be syntactically valid but semantically useless. Retrieval can return irrelevant chunks. An authorization defect can expose documents belonging to another user. A prompt change can increase token consumption without improving answer quality.</p><p>A mature CI/CD process therefore combines traditional software tests with AI-specific evaluations.</p><p>For example, a retrieval evaluation dataset can measure recall at kk, mean reciprocal rank, and the relevance of retrieved passages. A generation evaluation suite can examine groundedness, answer correctness, refusal behavior, and regressions against a representative set of questions.</p><p>Not every quality dimension can be reduced to one metric. The goal is to make important behavior measurable and to detect regressions before they become production incidents.</p><h3>5. Background Jobs: Protecting the Request Path</h3><p>Not every operation should happen while a user waits for an HTTP response.</p><p>Imagine uploading a 300-page PDF. The application must extract text, normalize content, divide it into chunks, generate embeddings, and write vectors to an index.</p><p>Performing the entire pipeline synchronously can create long-running requests, increase timeout failures, and consume resources needed by other users.</p><p>A better architecture separates the submission of work from its execution:</p><pre>Client<br>  |<br>  v<br>FastAPI Upload Endpoint<br>  |<br>  +----&gt; Store Original Document<br>  |<br>  +----&gt; Create Processing Job<br>                 |<br>                 v<br>              Job Queue<br>                 |<br>                 v<br>           Background Worker<br>                 |<br>        +--------+---------+<br>        |        |         |<br>        v        v         v<br>     Extract   Chunk     Embed<br>        |        |         |<br>        +--------+---------+<br>                 |<br>                 v<br>          Update Search Index<br>                 |<br>                 v<br>           Record Job Status</pre><p>The API can return a job identifier and let the client check processing status.</p><p>This design improves responsiveness, but it introduces new responsibilities:</p><ul><li>Jobs may be delivered more than once.</li><li>Workers may crash after completing an operation but before acknowledging the job.</li><li>Tasks may fail because of temporary provider errors.</li><li>The queue can grow faster than workers can process it.</li><li>Documents may be updated or deleted while processing is in progress.</li></ul><p>Consequently, reliable background processing requires idempotency, bounded retries, dead-letter handling, job state transitions, and clear operational visibility.</p><p><strong>At-least-once delivery is not the same as exactly-once execution.</strong> A practical system should assume that a job can be retried and design its side effects accordingly.</p><p>For example, use stable document and processing-version identifiers so a retry can recognize an existing result instead of blindly creating duplicate index entries.</p><h3>6. Observability: Knowing Why the System Is Failing</h3><p>A production service must provide enough evidence to explain its behavior.</p><p>Three complementary signals form the traditional foundation of observability:</p><p><strong>Metrics</strong> reveal trends and aggregate behavior. Request rates, error rates, latency percentiles, queue depth, database connection utilization, token consumption, and cost per successful request are useful examples.</p><p><strong>Logs</strong> capture discrete events. Structured logs can include a request identifier, operation name, error category, duration, and relevant dependency information.</p><p><strong>Traces</strong> connect operations across service boundaries. A trace can show that an API request spent 80 milliseconds authenticating, 40 milliseconds retrieving documents, and 9 seconds waiting for an LLM provider.</p><p>Without these signals, developers often end up debugging production through guesswork.</p><p>Consider an endpoint whose p95 latency increases from 1.5 seconds to 8 seconds. The aggregate metric tells you that something changed, but it does not explain why.</p><p>Distributed traces may reveal that the vector database remains healthy while the LLM provider’s response time has increased. Queue metrics may show that background ingestion is consuming database capacity. Structured error logs may identify a surge in upstream rate-limit responses.</p><p>Together, these signals turn a vague complaint into an actionable diagnosis.</p><p>AI observability needs additional care because prompts, retrieved documents, and generated responses can contain personal or confidential information. Logging every request body is not a responsible default.</p><p>Instead, define explicit policies for data minimization, redaction, retention, access control, and trace sampling. Track token counts, latency, model identifiers, retrieval statistics, and error categories where appropriate without unnecessarily storing sensitive content.</p><p>Observability is not simply about collecting more data. It is about collecting the right data safely.</p><h3>7. Security: Retrieval Is Also an Authorization Problem</h3><p>Retrieval-augmented generation introduces a security issue that is easy to underestimate.</p><p>Suppose a knowledge platform stores documents from thousands of users in a shared vector index. A query retrieves semantically similar chunks, and the application sends those chunks to an LLM.</p><p>If access control is applied only at the user interface, the system can return information from documents the user should never see.</p><p>Semantic similarity does not imply permission.</p><p>The retrieval layer must enforce authorization using trusted ownership metadata, tenant identifiers, document permissions, and appropriate filtering strategies. The application should not rely on an LLM prompt instructing the model to ignore unauthorized information after that information has already entered its context.</p><p>A secure design should consider:</p><ul><li>Authentication and server-side authorization.</li><li>Tenant isolation and document-level permissions.</li><li>Secrets management and least-privilege credentials.</li><li>Input validation and upload restrictions.</li><li>Rate limiting and abuse prevention.</li><li>Protection against prompt injection and malicious retrieved content.</li><li>Auditability without excessive sensitive-data retention.</li><li>Safe handling of tool calls and external actions.</li></ul><p>Prompt injection deserves particular attention. A retrieved document can contain instructions that attempt to manipulate the model into disclosing information or invoking tools. Treat retrieved content as untrusted data, not as an authority that can override system policies.</p><p>Where an AI application can execute actions, security must extend beyond text generation. Validate tool arguments, enforce authorization independently of model output, and require additional confirmation for consequential operations when appropriate.</p><p>The governing principle is straightforward: <strong>the model can help interpret a request, but the application must enforce the rules.</strong></p><h3>8. Kubernetes and Scaling: More Replicas Do Not Solve Every Problem</h3><p>As traffic grows, horizontal scaling may become necessary. Kubernetes provides mechanisms for scheduling containers, managing replicas, performing rolling deployments, and recovering from certain classes of failure.</p><p>However, increasing the number of application replicas does not automatically increase the capacity of every dependency.</p><p>If 20 API replicas share a database with a limited connection budget, scaling the API can make database contention worse. If each replica launches too many concurrent LLM requests, the system may hit provider rate limits faster. If workers consume jobs faster than the database can handle writes, additional workers may increase failures rather than throughput.</p><p>Scaling is a system-wide capacity problem.</p><p>A useful starting point is to identify the limiting resource:</p><ul><li>CPU-bound computation.</li><li>Memory pressure.</li><li>Database throughput or connection saturation.</li><li>Vector-search latency.</li><li>External API quotas.</li><li>Queue processing capacity.</li><li>Network bandwidth.</li><li>Model inference capacity and cost.</li></ul><p>Then choose a response appropriate to that bottleneck.</p><p>Possible strategies include caching, connection pooling, bounded concurrency, batching, query optimization, asynchronous processing, workload isolation, and backpressure.</p><p>Backpressure is particularly important. When downstream capacity is exhausted, the application needs a controlled way to slow admission, reject excess work, or defer processing instead of allowing unbounded queues and memory consumption.</p><p>Kubernetes can help manage workloads, but it cannot replace capacity planning, application-level limits, or a sound dependency architecture.</p><p>It is also important to distinguish liveness from readiness. A process can be alive but unable to serve useful traffic because a required dependency is unavailable. Liveness and readiness checks should reflect different operational questions, and overly aggressive probes should not turn a temporary dependency problem into a restart storm.</p><h3>9. Reliability Engineering: Plan for Failure Before It Happens</h3><p>A system is not production-ready merely because its normal path works.</p><p>What happens when the database is unavailable for 20 seconds? What happens when the LLM provider responds with a 429? What happens when a deployment interrupts active requests? What happens when the queue contains more work than the system can process before a deadline?</p><p>These are not exceptional questions. They are part of the system’s operating model.</p><p>A reliability strategy should define timeouts, retry policies, concurrency limits, and fallback behavior.</p><p>Retries deserve special attention. Retrying a failed operation can improve resilience when the failure is transient, but uncontrolled retries can amplify an outage.</p><p>If a provider is already overloaded, thousands of clients repeatedly retrying immediately can increase pressure on the same failing dependency.</p><p>Use bounded retries, exponential backoff, jitter, and retry budgets where appropriate. Do not retry every error indiscriminately, and be careful with operations whose side effects are not idempotent.</p><p>Circuit breakers can temporarily stop requests to an unhealthy dependency, while bulkheads can isolate resource consumption between different workloads.</p><p>For user-facing AI applications, degradation strategies can also improve resilience. Depending on the product, the system might return a clear temporary error, offer cached information, use an approved fallback model, or allow a request to be queued for later processing.</p><p>A fallback must preserve security and correctness requirements. Returning stale or incomplete information without communicating that limitation can be worse than returning an explicit failure.</p><p>Finally, define service-level indicators and objectives. For example, measure the percentage of eligible requests completed successfully within a target latency, rather than treating uptime alone as a complete description of reliability.</p><h3>10. The Economics of Production AI</h3><p>Reliability is only part of the problem. A system can be technically functional and still be economically unsustainable.</p><p>Consider a retrieval-augmented generation request. Its cost may include:</p><p>Crequest=Cembedding+Cretrieval+Cinput tokens+Coutput tokens+CinfrastructureC_{\text{request}} = C_{\text{embedding}} + C_{\text{retrieval}} + C_{\text{input tokens}} + C_{\text{output tokens}} + C_{\text{infrastructure}}</p><p>This is a simplified accounting model. Actual costs depend on provider pricing, caching, infrastructure utilization, batching, and whether some services are shared across requests.</p><p>The important point is that every architectural choice has an economic consequence.</p><p>A retrieval system that supplies too much context may increase input-token costs without materially improving answer quality. An unnecessarily expensive model may be used for tasks that a smaller model can handle. Repeatedly embedding identical text may waste compute. Poor caching can increase latency and infrastructure utilization simultaneously.</p><p>Optimization should begin with measurement.</p><p>Track cost per successful task, token consumption by operation, cache hit rates, retrieval quality, latency percentiles, and failure rates. Then experiment with one change at a time.</p><p>For example, reducing the number of retrieved chunks might lower token consumption, but it could also reduce answer quality. The correct decision requires evaluating both cost and retrieval effectiveness.</p><p>The objective is not to minimize cost at any price. It is to achieve the required quality and reliability within a sustainable operating budget.</p><h3>11. A Practical Production Readiness Checklist</h3><p>Before deploying an AI application to real users, review the system across several dimensions.</p><p><strong>Application engineering</strong></p><ul><li>Are dependencies pinned and builds reproducible?</li><li>Are API contracts validated and documented?</li><li>Are blocking operations and resource limits understood?</li><li>Are database migrations tested?</li></ul><p><strong>AI quality</strong></p><ul><li>Is retrieval evaluated against representative queries?</li><li>Are generation regressions measured?</li><li>Are model and prompt changes versioned?</li><li>Are token budgets and model timeouts enforced?</li></ul><p><strong>Security</strong></p><ul><li>Is authorization enforced at the data-access layer?</li><li>Are secrets managed outside source code and container images?</li><li>Are tenant boundaries tested?</li><li>Are retrieved documents and model outputs treated as untrusted?</li></ul><p><strong>Operations</strong></p><ul><li>Are logs, metrics, and traces available?</li><li>Are alerts connected to actionable failure conditions?</li><li>Are retries bounded?</li><li>Are backup, restoration, and recovery procedures tested?</li></ul><p><strong>Deployment and scaling</strong></p><ul><li>Is the release artifact tested before promotion?</li><li>Are readiness and liveness checks appropriate?</li><li>Are database connections and concurrent requests bounded?</li><li>Are queue depth and provider quotas monitored?</li><li>Is rollback possible when a release introduces a regression?</li></ul><p><strong>Economics</strong></p><ul><li>Is cost per successful task measured?</li><li>Are token usage and external service costs visible?</li><li>Have realistic concurrency and failure scenarios been tested?</li></ul><p>A checklist cannot prove that a system is reliable. It provides a structured way to expose assumptions, identify gaps, and prioritize engineering work.</p><p>The strongest evidence comes from tests, measured behavior, operational exercises, and documented decisions.</p><h3>12. From Individual Tools to a Complete Engineering System</h3><p>Docker, FastAPI, PostgreSQL, vector databases, CI/CD pipelines, Kubernetes, and observability platforms are useful technologies. But knowing how to configure each tool is not the same as knowing how to engineer the system that connects them.</p><p>The real work lies in understanding the interactions.</p><p>A retry policy affects provider costs. A database connection pool affects concurrency. A retrieval filter affects data security. A deployment strategy affects compatibility with database migrations. A logging policy affects privacy. A caching decision affects both latency and freshness.</p><p>Production AI engineering is therefore an exercise in systems thinking.</p><p>You must reason about dependencies, failure boundaries, resource constraints, security guarantees, and operational trade-offs. You must also know how to test assumptions instead of relying on architectural diagrams that describe only the happy path.</p><p>For developers who want to explore these ideas as a connected, project-based engineering journey, I have developed <strong>The AI Engineer — Volume II: Deploy. Scale. Operate.</strong></p><p>The book covers containerization, Linux operations, Git and GitHub workflows, CI/CD, cloud deployment, Kubernetes, observability, production security, scaling strategies, architecture patterns, and a final engineering challenge designed to bring the concepts together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P0qwhPDrPT1aNtWK9fk2aw.png" /></figure><p>You can find the book here:</p><p><a href="https://leanpub.com/theaiengineervolumeiideployscaleoperate"><strong>The AI Engineer — Volume II: Deploy. Scale. Operate.</strong></a></p><p>The objective is not to memorize a collection of commands. It is to develop the engineering judgment needed to understand how production AI systems behave, why they fail, and how to design them to recover and improve.</p><h3>Final Thoughts</h3><p>The next stage of AI engineering is not simply about making models more capable. It is about building dependable systems around those capabilities.</p><p>A production AI application must do more than generate an impressive answer. It must handle concurrent workloads, respect access boundaries, recover from dependency failures, expose enough information for diagnosis, control costs, and evolve without breaking the users who depend on it.</p><p>That requires disciplined software engineering as much as AI knowledge.</p><p>Build the feature. Test the behavior. Measure the system. Design for failure. Automate the release. Protect the data. Monitor the outcome.</p><p><strong>The difference between an AI demo and a production AI system is the engineering that makes it dependable.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=432b07b0411d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>