<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sunil Thakur on Medium]]></title>
        <description><![CDATA[Stories by Sunil Thakur on Medium]]></description>
        <link>https://medium.com/@sunil.baidyanath?source=rss-14a25a7d44f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aX973lWOrUpBcoam1XfTsA.jpeg</url>
            <title>Stories by Sunil Thakur on Medium</title>
            <link>https://medium.com/@sunil.baidyanath?source=rss-14a25a7d44f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:13:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sunil.baidyanath/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an AI Governance Framework for the Enterprise]]></title>
            <link>https://medium.com/berkeleyischool/building-an-ai-governance-framework-for-the-enterprise-5206c0e4fe06?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5206c0e4fe06</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <category><![CDATA[responsible-ai]]></category>
            <category><![CDATA[berkeleyischool]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 12:52:32 GMT</pubDate>
            <atom:updated>2026-08-18T15:53:15.716Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building an Enterprise AI Governance Framework: From Trusted Data to Trusted Autonomy</h3><p>A practical operating model for scaling AI responsibly through governance, reusable capabilities, and human-centered autonomy</p><figure><img alt="An image showing an enterprise AI governance ecosystem that depicts how AI governance builds trust and scales autonomy." src="https://cdn-images-1.medium.com/max/1024/1*FvX3083ZRz5u2XdhsAcurg.png" /><figcaption>Enterprise AI Governance Ecosystem</figcaption></figure><p>Artificial Intelligence is no longer simply a technology initiative — it is becoming part of the operating system of modern enterprises. What began with predictive models has evolved into Generative AI copilots and is now accelerating toward autonomous agentic systems capable of executing end-to-end workflows.</p><p>The challenge for enterprises is therefore evolving.</p><p>It is no longer limited to <strong>“How do we adopt AI?”</strong></p><p>It now extends to:</p><p><strong>“How do we scale AI without losing trust, control, accountability, and business value?”</strong></p><p>Most organizations are confronting the same paradox:</p><blockquote><strong><em>AI capabilities are advancing faster than enterprise governance is evolving.</em></strong></blockquote><p>Without a coherent governance and operating model, AI adoption fragments into disconnected use cases, inconsistent controls, duplicated technology, and unmanaged risk. The result is predictable — slower scaling, regulatory hesitation, employee uncertainty, and limited realization of business value.</p><p>The answer is not more controls.</p><p><strong>It is a governance model designed to make adoption of responsible AI faster, safer, and more scalable.</strong></p><h4>Why Traditional Governance Breaks Down</h4><p>Traditional enterprise governance largely evolved around deterministic software, structured data, predefined business rules, and human decision-making.</p><p>AI challenges this traditional control-driven governance model.</p><figure><img alt="A two-column table comparing AI-native enterprises to traditional enterprises." src="https://cdn-images-1.medium.com/max/1024/1*j0PLwq25_KI6P7rTuINOfA.png" /><figcaption>Traditional vs AI-Native Enterprises</figcaption></figure><p>The challenge therefore shifts from governing systems and processes to governing data, decisions, capabilities, and increasingly autonomous actions.</p><p>This article synthesizes these ideas into an integrated enterprise AI governance model spanning Data, AI, and Agent Governance, together with the operating architecture, adoption practices, and transformation journey required to evolve from AI experimentation toward an autonomous enterprise.</p><h3>The Core Idea: Governance is a Trust System, Not a Control System</h3><p>Enterprise AI governance should not be viewed as a compliance layer placed around AI. It should be viewed as a mechanism for creating sufficient trust to allow AI to scale.</p><figure><img alt="A layered view of the enterprise AI governance model showing how an autonomous enterprise governs to create business value." src="https://cdn-images-1.medium.com/max/1024/1*mY1w3B9QrLTpqem2uNlRoQ.png" /><figcaption>Enterprise AI Governance Model</figcaption></figure><p>At its core, AI governance is about establishing trust at three levels that map directly to three foundational governance domains:</p><ul><li><strong>Trust in information → </strong>Data Governance</li><li><strong>Trust in decisions → </strong>AI Governance</li><li><strong>Trust in actions → </strong>Agent Governance</li></ul><p>These three governance domains provide the foundation; maturity then extends that trust progressively from individual AI decisions to coordinated autonomous operations.</p><p>Together, they form the backbone of an AI-native enterprise.</p><p>The higher the level of autonomy, the greater the requirement for identity, authorization, observability, human accountability, and recoverability.</p><p>But governance alone is not enough.</p><p>To scale AI, enterprises must also rethink how AI capabilities are built, reused, and delivered.</p><h3>The Enterprise AI Governance Operating Model</h3><p>A scalable AI enterprise requires a layered governance model that separates concerns while maintaining integration across data, models, and agents.</p><h4>1. Data Governance: Trust in Information</h4><p>Data is the foundation of all AI systems.</p><p>This layer ensures that enterprise data is accurate, secure, traceable, and usable for AI and analytics.</p><p>Key capabilities include:</p><ul><li>Data ownership and stewardship</li><li>Data quality management</li><li>Metadata and lineage tracking</li><li>Master data management</li><li>Privacy and regulatory compliance</li></ul><blockquote><strong>Key question: </strong>Can we trust the data?</blockquote><h4>2. AI Governance: Trust in Decisions</h4><p>As organizations deploy machine learning and generative AI, governance must ensure that models are explainable, fair, and aligned with business and regulatory expectations.</p><p>Key capabilities include:</p><ul><li>Responsible AI principles</li><li>Bias detection and fairness controls</li><li>Model validation and testing</li><li>Explainability frameworks</li><li>Human oversight mechanisms</li></ul><blockquote><strong>Key question: </strong>Can we trust the model’s decisions?</blockquote><h4>3. Agent Governance: Trust in Actions</h4><p>With the rise of agentic AI, systems are no longer just recommending decisions; they are executing them.</p><p>This introduces a new governance frontier: controlling autonomous behavior.</p><p>Key capabilities include:</p><ul><li>Agent authorization and permissions</li><li>Tool and API access control</li><li>Runtime monitoring and auditing</li><li>Memory and context governance</li><li>Human-in-the-loop escalation</li></ul><blockquote><strong>Key question: </strong>Can we trust autonomous actions?</blockquote><h3>Governance Structure: Who Decides What?</h3><p>Governance becomes effective only when decision rights are explicit.</p><p>The enterprise structure for AI governance separates strategic direction, domain governance, AI governance, and autonomous-action governance, while keeping business owners accountable for outcomes.</p><figure><img alt="A hierarchical view of the enterprise AI governance structure showing different governance entities, their responsibilities, and how they interconnect." src="https://cdn-images-1.medium.com/max/1024/1*i_ippiyL969a0LZChBGhTg.png" /><figcaption>Enterprise AI Governance Structure</figcaption></figure><p>The governance structure includes the following entities:</p><ul><li><strong>Executive AI &amp; Data Council:</strong> Defines strategy, risk appetite, funding priorities, and enterprise AI roadmap.</li><li><strong>Data Governance Council:</strong> Oversees data standards, quality, stewardship, and compliance.</li><li><strong>AI Governance Council: </strong>Reviews AI use cases, model risk, fairness, and explainability.</li><li><strong>Agent Governance Board:</strong> Approves autonomous agents, tool access, runtime policies, and escalation thresholds.</li></ul><p>The important distinction is that governance bodies establish boundaries and decision rights while business owners remain accountable for the business outcomes those systems produce.</p><h3>The Enterprise AI Operating Architecture: From Data to Business Value</h3><p>Governance ensures trust. Architecture enables scale.</p><p>Governance establishes the boundaries within which AI can operate.</p><p>Architecture provides the mechanisms through which those capabilities are delivered at scale.</p><h4>From Use Cases to Capabilities</h4><blockquote>AI does not scale through isolated use cases. It scales through reusable capabilities.</blockquote><p>A scalable enterprise AI architecture should connect trusted data and context to reusable AI capabilities, shared platforms, and ultimately business use cases.</p><p>The objective is to move from isolated AI applications to an enterprise AI operating architecture in which capabilities can be reused, governed, and continuously improved across the organization.</p><p>The AI operating architecture extends the traditional enterprise stack with AI-specific layers that connect the underlying technology platform to business use cases. Three layers are particularly relevant: the Enterprise AI Platform, Enterprise AI Capability Layer, and Enterprise Context Layer.</p><figure><img alt="A layered view of the enterprise AI operating architecture showing the individual operating layers and how they stack one over another and provide business value." src="https://cdn-images-1.medium.com/max/1024/1*P4AQtzeeWEI6kRf-rM5yxA.png" /><figcaption>Enterprise AI Operating Architecture</figcaption></figure><p>The Enterprise AI Platform provides the technical foundation for building, deploying, operating, and governing AI — including model abstraction, routing, evaluation, monitoring, security, and lifecycle management. This platform may be provided by cloud and AI vendors, built internally, or implemented as a hybrid capability.</p><p>The platform enables AI; the enterprise capability and context layers make that AI reusable and relevant to the business.</p><h4>The Enterprise AI Capability Layer</h4><p>The capability layer defines reusable AI capabilities that can be applied across domains.</p><p>These include:</p><ul><li>Retrieval (enterprise knowledge access)</li><li>Summarization (information compression)</li><li>Classification (intelligent routing)</li><li>Recommendation (next-best-action systems)</li><li>Generation (content creation)</li><li>Reasoning (complex decision making and inference)</li><li>Orchestration (workflow and agent coordination)</li></ul><p>Abstracting recurring AI patterns into reusable enterprise capabilities reduces duplication, accelerates delivery, and allows governance and security controls to be applied consistently across multiple use cases.</p><h4>The Enterprise Context Layer</h4><p>The context layer provides governed access to enterprise resources that serve as trusted context for the AI systems to produce relevant results and increasingly act safely.</p><p>These resources include:</p><ul><li>Enterprise knowledge</li><li>Data with provenance</li><li>Tool and API registries</li><li>Prompt libraries</li><li>Agent memory and state</li></ul><h3>Scaling AI Adoption</h3><p>Technology alone does not drive adoption.</p><p>Successful organizations adopt a set of operating principles that balance experimentation with control.</p><p>Several operating principles have proven effective in accelerating AI innovation and adoption:</p><h4>1. Democratize AI Access</h4><p>Provide broad access to governed AI capabilities so employees can experiment and discover new use cases. Broad access should be paired with <strong>self-service tools, approved prompts, reusable patterns, and low-risk workflows</strong>.</p><h4>2. Crawl → Walk → Run</h4><p>Start with low-risk experimentation and <strong>progressively increase complexity and autonomy</strong> as evidence and controls mature.</p><h4>3. Abstract Common Patterns</h4><p>Turn recurring AI patterns into reusable enterprise capabilities rather than building every use case from scratch. By standardizing these patterns, organizations can <strong>reduce duplication, simplify governance, accelerate deployment, and improve consistency</strong>.</p><h4>4. Embed Governance into the Platform</h4><p>Make governance increasingly automated through identity, access controls, registries, approved patterns, monitoring, and policy enforcement.</p><blockquote><em>This combination allows enterprises to move quickly without creating a governance process that becomes a bottleneck to innovation.</em></blockquote><h3>Human-Centered Autonomy: Matching Autonomy to Risk</h3><p>Not every AI system should operate with the same degree of autonomy. The appropriate operating model depends on the potential impact of the decision, the sensitivity of the data, and the consequences of an incorrect action.</p><p>A useful enterprise principle is:</p><blockquote>Automate where risk is low, involve humans where risk is material, and keep humans accountable where consequences are significant.</blockquote><figure><img alt="A diagram showing how, with increasing autonomy, human engagement evolves from leading the task to overseeing." src="https://cdn-images-1.medium.com/max/1024/1*Qf78aJQi1jE0kgPtUMN_mg.png" /><figcaption>Human-Centered Autonomy Model</figcaption></figure><p>The objective is not to remove humans from the process. It is to move humans toward activities in which judgment, accountability, creativity, and strategic decision-making create the greatest value.</p><p>As trust, performance, controls, and recoverability improve, organizations can progressively move from human-led work toward exception-based autonomous operations.</p><blockquote>Autonomy is not a binary technology choice. It is an organizational capability that must be earned through performance, trust, and governance maturity.</blockquote><h3>Risk and Security: Controls Must Scale with Autonomy</h3><p>Enterprise AI introduces new categories of risk that must be managed holistically:</p><ul><li><strong>Information Risk:</strong> data quality, privacy, lineage, provenance, access control, data leakage</li><li><strong>Decision Risk:</strong> bias, hallucinations, explainability, model performance, drift, inappropriate decisions</li><li><strong>Agent Risk:</strong> unauthorized or unintended actions, excessive autonomy, tool misuse, privilege escalation, prompt injection</li><li><strong>Operational Risk:</strong> business disruption, process failure, financial impact, cascading failures, cost overruns</li><li><strong>Regulatory Risk:</strong> compliance, consumer protection, privacy obligations, auditability, transparency</li></ul><blockquote>The key principle is proportional governance. That is, <strong>governance intensity should increase with business impact and autonomy.</strong></blockquote><h4>Security and Identity as a Cross-Cutting Control Plane</h4><p>Security cuts across Data, AI, and Agent governance rather than operating as a separate governance silo.</p><p>Identity, authentication, authorization, privacy, access control, monitoring, and auditability provide the control plane through which trusted AI operates.</p><p>As agents increasingly act across applications and on behalf of users, these controls become substantially more complex. Agent identity, delegated authority, provisioning, runtime intervention, secure agent-to-application interactions, and recoverability become first-class governance concerns.</p><h3>The Transformation Journey</h3><p>Enterprise AI maturity is not achieved through a single technology deployment. It is a progressive build-up of trust, capabilities, and operating discipline.</p><figure><img alt="A roadmap diagram that presents the AI transformation journey for an enterprise from data-ready to an autonomous enterprise." src="https://cdn-images-1.medium.com/max/1024/1*zqUhdjkgmlw6QxWCqWR3Jg.png" /><figcaption>AI Transformation Roadmap</figcaption></figure><p>Enterprises evolve through a structured AI maturity journey in the following phases:</p><ol><li><strong>Data Ready:</strong> Establish trusted data foundations and governance structures.</li><li><strong>AI Ready:</strong> Enable responsible AI adoption and model governance.</li><li><strong>Agent Ready: </strong>Deploy governed autonomous agents and runtime controls.</li><li><strong>Autonomous Enterprise: </strong>Scale autonomous operations with exception-based human oversight and enterprise-wide agent ecosystems.</li></ol><p>Each phase expands the organization’s ability to apply AI at scale by progressively building trust:</p><ul><li>Trust in data</li><li>Trust in decisions</li><li>Trust in actions</li><li>Trust in autonomous operations</li></ul><h4>AI Governance is an Executive Discipline</h4><p>AI governance cannot be delegated entirely to technology or compliance. It requires coordinated ownership across business, data, technology, AI, risk, security, and compliance.</p><ul><li>Business leaders own outcomes.</li><li>Technology provides the platform.</li><li>Data leaders establish information trust.</li><li>AI leaders establish responsible AI practices.</li></ul><p>Risk, compliance, and security define the boundaries within which innovation can safely operate.</p><blockquote><strong>Executive leadership ultimately determines the organization’s appetite for risk in exchange for innovation and business value.</strong></blockquote><h3>Conclusion: From Governance to Trusted Autonomy</h3><p>Enterprise AI is a transformation in how organizations make decisions, execute work, and create value.</p><p>The journey is one of progressively earned trust:</p><p><strong>Trusted Data → Trusted AI → Trusted Agents → Trusted Autonomy</strong></p><blockquote>The organizations that succeed will not necessarily be those that deploy the most AI. They will be those that build the operating model capable of scaling AI <strong>safely, responsibly, and economically</strong>.</blockquote><p>Governance is therefore not the mechanism that slows AI adoption.</p><p><strong>It is the mechanism that makes greater AI adoption possible.</strong></p><p>And as AI moves from recommending to acting, the next challenge is no longer simply responsible AI.</p><p><strong>It is governing the autonomous enterprise.</strong></p><h3>References</h3><ol><li><strong>Gartner.</strong> <em>Artificial Intelligence Requires an Extended Governance Framework.</em> April 2024.<br><a href="https://www.gartner.com/en/documents/5335063?utm_source=chatgpt.com">Gartner — Artificial Intelligence Requires an Extended Governance Framework</a></li><li><strong>Mastercard.</strong> <em>Securing the Digital Ecosystem with AI.</em><br>Mastercard’s responsible-AI and AI governance lifecycle, including purpose evaluation, data evaluation, risk assessment, validation, and production monitoring.</li><li><strong>Bojinov, Iavor; Lakhani, Karim R.; Lane, David.</strong> <em>JPMorganChase: Leadership in the Age of GenAI.</em> Harvard Business School Case 325–066, April 2025.<br><a href="https://store.hbr.org/product/jpmorganchase-leadership-in-the-age-of-genai/325066?utm_source=chatgpt.com">Harvard Business School — JPMorganChase: Leadership in the Age of GenAI</a></li><li><strong>JPMorgan Chase.</strong> <em>Leadership in the Age of GenAI: Mary Callahan Erdoes and Mike Urciuoli present Harvard case study.</em><br>Useful as the corporate perspective accompanying the HBS case.<br><a href="https://privatebank.jpmorgan.com/nam/en/about-us/mary-callahan-erdoes-and-mike-urciouli-present-hbs-case-study?utm_source=chatgpt.com">JPMorgan Chase — Leadership in the Age of GenAI</a></li><li><strong>Ivey Publishing.</strong> <em>Customer-Centric Design with Artificial Intelligence: Commonwealth Bank.</em><br>Case study covering Commonwealth Bank’s Customer Engagement Engine, AI-driven personalization, cross-channel engagement, leadership sponsorship, and frontline adoption.</li><li><strong>Commonwealth Bank.</strong> <em>Reimagining Banking.</em><br>Commonwealth Bank’s account of its AI-driven customer experience strategy and Customer Engagement Engine.<br><a href="https://www.commbank.com.au/articles/newsroom/2023/05/reimagining-banking.html?utm_source=chatgpt.com">Commonwealth Bank — Reimagining Banking</a></li><li><strong>Celent.</strong> <em>Commonwealth Bank: Personalized Engagement at Scale.</em><br>Useful reference for the <strong>Next Best Conversation</strong> and coordinated, cross-channel customer engagement model.</li><li><strong>Iansiti, Marco; Lakhani, Karim R.</strong> <em>Competing in the Age of AI: Strategy and Leadership When Algorithms and Networks Run the World.</em> Harvard Business Review Press, 2020.<br><a href="https://hbr.org/2020/01/competing-in-the-age-of-ai?utm_source=chatgpt.com">Harvard Business Review — Competing in the Age of AI</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5206c0e4fe06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/berkeleyischool/building-an-ai-governance-framework-for-the-enterprise-5206c0e4fe06">Building an AI Governance Framework for the Enterprise</a> was originally published in <a href="https://medium.com/berkeleyischool">BerkeleyISchool</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Model to Production: A Stepwise Framework for Deploying Machine Learning System at Scale]]></title>
            <link>https://medium.com/berkeleyischool/from-model-to-production-a-stepwise-framework-for-deploying-machine-learning-system-at-scale-7da028ab2d6e?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7da028ab2d6e</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[berkeleyischool]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 19:05:49 GMT</pubDate>
            <atom:updated>2026-06-22T21:53:10.774Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Notebook to Production: A 5-Step Framework for Deploying ML Systems at Scale</h3><h4>A practitioner’s guide using FastAPI, Pydantic, Poetry, Docker, Kubernetes, Redis, Istio, Prometheus, Grafana, and Kiali</h4><figure><img alt="a graph showing: From notebook to production, a 5-Step Deployment Framework for a ML Model" src="https://cdn-images-1.medium.com/max/1024/1*szclEUK4AlSEGoZrW4yjjw.png" /></figure><p>Most machine learning teams do not struggle with training models. They struggle with <strong>operationalizing them</strong>.</p><p>A model that performs well in a notebook still needs to become a production-grade inference system that can:</p><ul><li>serve predictions reliably</li><li>validate requests and responses</li><li>scale elastically</li><li>recover from failures</li><li>support controlled rollouts</li><li>remain observable in production</li><li>meet latency and uptime SLAs</li></ul><p>This is the gap between <strong>model development</strong> and <strong>business value realization</strong>.</p><p>For ML engineers, MLOps teams, and AI platform leaders, this is where deployment architecture becomes critical.</p><p>In this article, I present a <strong>five-step deployment framework</strong> that progressively transforms a trained model into a cloud-native, enterprise-ready inference system.</p><p>The framework is model-agnostic.</p><p>Whether you are deploying:</p><ul><li>a scikit-learn pipeline</li><li>an XGBoost classifier</li><li>a transformer model</li><li>an NLP inference service</li><li>a computer vision model</li></ul><p>the deployment principles remain the same.</p><blockquote>The goal is to build a <strong>repeatable production framework</strong> that teams can standardize across use cases.</blockquote><h4>The Production ML System Architecture</h4><p>With this deployment framework, we will engineer a <strong>consumable ML platform layer</strong> that enables downstream applications, business workflows, and end-user experiences to interact with the model safely and reliably.</p><p>Going beyond a single prediction endpoint, a production-ready machine learning system must be designed to support enterprise-grade operational capabilities, including:</p><ul><li><strong>low-latency inference</strong> for real-time prediction use cases</li><li><strong>reproducible deployment</strong> across development, staging, and production environments</li><li><strong>scalable orchestration</strong> to handle changing traffic volumes</li><li><strong>production-grade traffic routing</strong> for safe access and controlled exposure</li><li><strong>caching optimization</strong> to improve latency and reduce repeated compute</li><li><strong>autoscaling</strong> based on demand patterns</li><li><strong>monitoring and alerting</strong> for system health and operational visibility</li><li><strong>performance and load testing</strong> to validate production readiness</li></ul><p>The diagram below depicts a high-level view of an enterprise-grade ML system architecture with production-level operational capabilities.</p><figure><img alt="a flow chart going from box to box showing ML System Architecture" src="https://cdn-images-1.medium.com/max/1024/1*hGBJGi4FieynWQXZasAtQw.png" /><figcaption>ML System Architecture</figcaption></figure><p>Such a system is best built through a deliberate, stepwise progression, with each stage incrementally introducing operational capabilities and advancing the maturity of the overall system.</p><h3>The 5-Step Deployment Framework</h3><p>Production deployment should not be treated as a single action.</p><p>It should be treated as a <strong>capability maturity path</strong>.</p><p>The framework progresses through five engineering stages:</p><p><strong>Build → Package → Scale → Deploy → Observe</strong></p><p>Each step introduces a distinct layer of production capability.</p><p>The diagram below depicts this layered maturity path, as capabilities are built through each step.</p><figure><img alt="a chart showing ML Deployment Steps" src="https://cdn-images-1.medium.com/max/1024/1*wAjnXwkUmHS5lxAuPoLDCQ.png" /><figcaption>ML Deployment Steps</figcaption></figure><p>This framework provides a <strong>clear, disciplined path to production</strong>, enabling ML systems to evolve from experimental prototypes into <strong>robust, scalable, and enterprise-ready services</strong>.</p><blockquote>Adopting a layered deployment strategy is not just a matter of organization; it is a deliberate design choice that improves reliability, reduces risk, and accelerates production readiness.</blockquote><p>This progressive framework offers three key advantages:</p><p><strong>1. Reduced Validation Scope</strong><br>Each stage introduces a bounded set of capabilities, allowing teams to validate one concern at a time.</p><p>The system evolves incrementally through these steps:</p><ul><li>inference correctness</li><li>reproducible packaging</li><li>orchestration and scaling</li><li>cloud deployment and traffic control</li><li>performance under load</li></ul><p>This significantly reduces debugging complexity.</p><p>Issues can be isolated to a specific layer instead of the entire system; a critical advantage when moving from experimentation to production.</p><p><strong>2. Progressive Systems Maturity</strong><br>Each step builds on the previous one, forming a clear maturity path:</p><p><strong>Local Development → Containerized Workload → Local Cluster → Production Cloud Cluster → Platform → Observable Production System</strong></p><p>This progression separates concerns cleanly:</p><ul><li>Early stages focus on building and packaging (notebooks, APIs, containers) in a local environment</li><li>Mid stages introduce orchestration and scaling via a local cluster</li><li>Later stages transition to production-grade cloud infrastructure and platform capabilities</li></ul><p>This maturity path aligns directly with the deployment framework:</p><ul><li><strong>Build / Package: </strong>Local development and containerization</li><li><strong>Scale: </strong>Running on a local cluster</li><li><strong>Deploy: </strong>Promotion to a production cloud cluster</li><li><strong>Observe: </strong>Evolving into a fully observable, platform-backed system</li></ul><p>Rather than over-engineering upfront, capabilities are introduced at the right stage, progressing from isolated development to a shared, production-ready platform.</p><p>This approach improves efficiency, reduces operational risk, and mirrors how mature ML systems evolve in practice.</p><p><strong>3. Operational Roadmap with Measurable Milestones</strong><br>Each step serves as a concrete milestone with clear validation criteria, enabling:</p><ul><li>structured progression to production</li><li>reduced deployment risk</li><li>alignment across engineering, data science, and platform teams</li></ul><p>Together, these advantages make the framework more than a sequence of deployment steps, and it becomes a structured approach to building production-grade ML systems.</p><p>This deployment approach is not only technically sound but also <strong>predictable, measurable, and repeatable</strong>.</p><p>Let’s walk through how production maturity evolves in practice across these deployment steps.</p><h3>Step 1 — Build the Model Serving API</h3><p>Before a model can be served, the first engineering objective is to establish a <strong>reproducible environment</strong> and a <strong>serving layer</strong> that exposes the model outside of a notebook.</p><p>Together, these form the first <strong>production-ready interface</strong> between the ML artifact and downstream consumers.</p><p>The initial priority is ensuring the system behaves consistently across:</p><ul><li>Machines</li><li>Teams</li><li>Deployment stages</li></ul><blockquote>Many ML systems fail not because of poor models, but due to <strong>fragile and inconsistent environments</strong>.</blockquote><p>Common failure points include:</p><ul><li>Package version drift</li><li>Dependency conflicts</li><li>Python runtime mismatches</li><li>Inconsistent library behavior</li></ul><p>To eliminate these risks, <strong>deterministic dependency management</strong> must be enforced from the outset.</p><p>With a stable environment in place, the next step is to build a <strong>model-serving API layer</strong> that defines a clear contract between the model and its consumers.</p><p>A lightweight, production-ready framework such as <a href="https://fastapi.tiangolo.com/"><strong>FastAPI</strong></a> is well-suited for this purpose due to:</p><ul><li>Low-overhead REST serving</li><li>Automatic API documentation</li><li>Strong request validation</li><li>Native async support</li><li>Seamless integration with Python ML libraries</li></ul><h4>Key Engineering Considerations</h4><ol><li><strong>Dependency Management with Poetry</strong></li></ol><p><a href="https://python-poetry.org/docs/">Poetry</a> enables deterministic dependency management through lock files that fully capture the dependency graph.</p><p>Install and sample setup:</p><pre># install poetry<br>curl -sSL https://install.python-poetry.org | python3<br><br># add dependencies required<br>poetry add fastapi uvicorn<br><br># Add dev dependencies<br>poetry add --group dev pytest</pre><p>This produces a reproducible lock file, which is essential for:</p><ul><li>Team collaboration</li><li>CI/CD pipelines</li><li>Consistent deployments</li><li>Environment parity</li></ul><p><strong>2. Request/Response Validation</strong></p><p>Use <a href="https://pydantic.dev/docs/validation/latest/get-started/">Pydantic</a> models to enforce strict input and output contracts. Reference <a href="https://pydantic.dev/docs/validation/latest/concepts/models/#basic-model-usage">pydantic models</a> to create an appropriate model for the request (data input) and the response (data output)</p><p>Sample request/response model:</p><pre>from pydantic import BaseModel, field_validator, ConfigDict<br><br>class Request(BaseModel):<br>  feature_1: float<br>  feature_2: str<br><br>  model_config = ConfigDict(extra=&quot;forbid&quot;)<br><br>  @field_validator(&quot;feature_1&quot;)<br>  @classmethod<br>  def validate_field1(cls, value):<br>    if not value:<br>      raise ValueError(&quot;Invalid value for feature_1&quot;)<br>    return value<br><br>class Response(BaseModel):<br>  field1: float</pre><p>This ensures the model only processes valid inputs and returns well-structured responses.</p><p>FastAPI and Pydantic have <a href="https://fastapi.tiangolo.com/tutorial/body/?h=pydantic#import-pydantics-basemodel">native integration</a>, making the validation seamless and efficient.</p><p><strong>3. Health Endpoint</strong></p><p>Expose a dedicated health endpoint to verify service availability:</p><pre>@app.get(&quot;/health&quot;)<br>def health():<br>    return {&quot;status&quot;: &quot;healthy&quot;}</pre><p>This endpoint becomes critical for <strong>readiness and liveness checks</strong> in later deployment stages.</p><p><strong>4. Model Initialization Strategy</strong></p><p>Load the model once during application startup rather than on every request.</p><p>This avoids repeated disk and memory overhead, significantly improving performance.</p><p>The code block below shows how to use the lifespan mechanism to configure model loading at the application startup:</p><pre>from contextlib import asynccontextmanager<br><br># Create lifespan function to load the model<br>@asynccontextmanager<br>async def lifespan_mechanism(app: FastAPI):<br>    global model<br>    model = load(&quot;model_pipeline.pkl&quot;)<br>    yield<br><br><br># Create the FastAPI app with this lifespan function<br>app = FastAPI(lifespan=lifespan_mechanism)</pre><blockquote>In production ML systems, models should not be treated as static code dependencies bundled within the application.</blockquote><blockquote>Instead, they should be managed as external, versioned artifacts loaded at runtime from a centralized storage location such as object storage or a model registry.</blockquote><p>This decouples the model lifecycle from the service lifecycle, enabling:</p><ul><li>independent model updates,</li><li>easier rollbacks,</li><li>improved traceability, and</li><li>smaller, more portable deployment artifacts.</li></ul><p>By adopting this approach early, the serving layer becomes more flexible and production-ready, allowing teams to evolve models without requiring full application redeployments.</p><p>This approach is especially important for <strong>large transformer and NLP models</strong>, where initialization cost is high.</p><p><strong>5. Feature Management and Transformation Consistency</strong></p><p>In production ML systems, not all features can or should be computed at inference time. Some features are computationally expensive or depend on historical aggregation, making real-time calculation impractical.</p><p>In such cases, these features must be precomputed and stored in a <strong>feature store</strong>, where they can be efficiently retrieved during inference. This ensures low-latency predictions while maintaining access to rich, high-value features.</p><p>Equally important is ensuring consistency in data transformations across training and inference pipelines. Any mismatch between how features are processed during training and how they are handled at inference time can lead to training-serving skew and degraded model performance.</p><p>To address this, feature transformation logic should be standardized and reused across both pipelines. Many ML libraries provide pipeline abstractions that encapsulate preprocessing steps, ensuring that the same transformations applied during training are consistently applied during inference.</p><blockquote>Together, feature stores and standardized transformation pipelines form a critical foundation for building reliable, production-grade ML systems.</blockquote><p>Validate the model serving API locally using a simple request:</p><pre>curl -X POST &quot;http://localhost:&lt;port-number&gt;/inference-api&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;&lt;input_data&gt;&#39;</pre><p>The system becomes:</p><ul><li><strong>Consumable</strong> via a defined API</li><li><strong>Testable</strong> through structured requests and responses</li><li><strong>Service-oriented</strong> in design</li></ul><blockquote>This step establishes the serving contract boundary between the ML artifact and downstream systems.</blockquote><p>At this stage, the system evolves from an experimental script into a <strong>production-capable service</strong>:</p><ul><li>Reproducible across environments</li><li>Accessible through a well-defined API</li><li>Validated and reliable</li><li>Ready for integration with downstream systems</li></ul><p>This represents the <strong>first maturity milestone</strong> in building a production-grade ML system.</p><h3>Step 2— Containerize and Operationalize the Service</h3><p>Once inference works locally, the next engineering challenge is runtime portability.</p><p>The objective is to ensure the service runs consistently across all environments:</p><ul><li>Local</li><li>Development</li><li>Staging</li><li>Production</li></ul><p>This is where <a href="https://docs.docker.com/get-started/introduction/">Docker</a> becomes essential that builds an immutable contaier image with a given <a href="https://docs.docker.com/reference/dockerfile/">Dockerfile</a>.</p><blockquote>By packaging the service as a container image, we create a <strong>standardized, portable runtime artifact</strong> that behaves identically across environments.</blockquote><h4>Key Engineering Considerations</h4><ol><li><strong>Multi-Stage Docker build</strong></li></ol><p>Multi-stage builds are a best practice for production-grade containerization.</p><p>They separate:</p><ul><li>Build-time dependencies (e.g., compilers, tooling)</li><li>Runtime dependencies (minimal environment required to execute the service)</li></ul><p>Example Dockerfile structure:</p><pre>## Build stage<br>FROM python:3.13-slim AS build<br><br>## Prod stage<br>FROM python:3.13-slim AS prod</pre><p>Keeping the production image minimal provides several advantages:</p><ul><li>Reduced image size</li><li>Lower attack surface</li><li>Faster cold start times</li></ul><p>For Python projects, consider using <strong>Poetry</strong> in the build stage while ensuring only necessary artifacts are copied into the final image.</p><p>You can reference <a href="https://gabnotes.org/posts/lighten-your-python-image-with-docker-multi-stage-builds">this tutorial</a> to create a multistage build with Poetry.</p><p>Follow Dockerfile <a href="https://docs.docker.com/build/building/best-practices/">best practices</a> to optimize layer caching and minimize unnecessary dependencies for reducing image sizes appropriately.</p><p><strong>2. Container Health Checks</strong></p><p>A production-ready container must expose its runtime health.</p><p>Adding a HEALTHCHECK instruction enables the container runtime to monitor service availability:</p><p>Sample HEALTHCHECK command in Dockerfile:</p><pre>HEALTHCHECK --interval=5m --timeout=3s \<br>  CMD curl -f http://localhost:8000/health</pre><p>This ensures:</p><ul><li>Unhealthy containers are detected automatically</li><li>Orchestrators (e.g., Kubernetes) can take corrective action</li><li>Service reliability improves under failure conditions</li></ul><p>However, this mechanism is primarily useful for local development or standalone container environments. Kubernetes does not rely on Docker HEALTHCHECK instructions for production workloads.</p><p>In production, health management should be implemented using Kubernetes probes:</p><ul><li><strong>Liveness probes</strong> to detect and restart unhealthy containers</li><li><strong>Readiness probes</strong> to control traffic routing</li><li><strong>Startup probes</strong> for slow-initializing workloads</li></ul><p>This shifts health responsibility from the container runtime to the orchestrator, providing more precise and scalable control in distributed systems.</p><p><strong>3. Production-Ready Application Serving</strong></p><p>For serving a FastAPI application in production, it is important to use a production-grade ASGI server rather than development-only commands such as fastapi run. Servers like Uvicorn are designed to handle real-world workloads and are better suited for containerized environments.</p><p>A typical production configuration uses Uvicorn to serve the application within the container:</p><p>For deploying a FastAPI service, a production-grade ASGI server such as <a href="https://uvicorn.dev/"><strong>Uvicorn</strong></a><strong> is </strong>more suitable instead<strong> </strong>of development-only commands (e.g., fastapi run) in production environments.</p><p>A performant server configuration, such as U<strong>vicorn, </strong>is better suited for containerized workloads.</p><p>Example command in Dockerfile:</p><pre>CMD [&quot;/app/venv/bin/uvicorn&quot;, &quot;src.main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;]</pre><p>Refer to <a href="https://pytutorial.com/deploy-fastapi-with-docker-for-production/">this</a> documentation to deploy FastAPI app appropriately with Docker.</p><p>Once the container image is built, the application can be validated by running the container locally and testing the inference endpoint.</p><p>You can test the system by building the docker image and then starting the docker container</p><pre># Build the application<br>docker build -t app .<br><br># Run the application<br>docker run -p 8000:8000 app<br><br># Test the application<br>curl -X POST &quot;http://localhost:8000/inference-api&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;&lt;input_data&gt;&#39;</pre><p>This step introduces the <strong>minimum viable operational layer</strong> required for production deployment.</p><p>It transforms the ML service into:</p><ul><li><strong>Portable</strong> → runs consistently across environments</li><li><strong>Reproducible</strong> → deterministic builds and deployments</li><li><strong>CI/CD compatible</strong> → seamless integration into deployment pipelines</li></ul><p>This step builds the immutable runtime artifact that can be spun up on appropriate infrastructure to serve requests on demand.</p><p>At this stage, the system produces an <strong>immutable runtime artifact; </strong>a container image that can be deployed on any compatible infrastructure.</p><p>This artifact serves as the foundation for:</p><ul><li>Scalable deployments</li><li>Automated pipelines</li><li>On-demand service provisioning</li></ul><blockquote>In essence, this step converts the application from a local prototype into a <strong>deployable, production-ready service unit</strong>.</blockquote><p>At this point, the problem shifts from correctness to operational resilience.</p><h3>Step 3— Optimize Performance with Caching and Orchestration</h3><p>Once the service is portable, the next engineering concern is operational performance and resilience at scale.</p><p>In this step, two critical capabilities are introduced:</p><ul><li><strong>Caching with Redis</strong></li><li><strong>Container orchestration with Kubernetes</strong></li></ul><p>At this stage, the system begins to transition into a <strong>production-grade deployment</strong>.</p><h4>Key Engineering Considerations</h4><p><strong>1. Redis Caching</strong></p><p>Inference workloads often experience repeated requests. Recomputing predictions for identical inputs leads to unnecessary latency and compute overhead.</p><figure><img alt="flow chart showing Request Flow With and Without Redis Caching" src="https://cdn-images-1.medium.com/max/1024/1*tHmq1vYSxyx6Pmrx260TUA.png" /></figure><p>Integrate Redis into the model serving API:</p><pre>app = FastAPI(...)<br><br>app.add_middleware(<br>    NaiveCache,<br>    redis_host=REDIS_HOST_URL,<br>    redis_port=REDIS_HOST_PORT,<br>)  </pre><p>For local validation, run Redis using Docker:</p><pre>docker run -d --name temp-redis -p 6379:6379 redis</pre><p>Redis will then be accessible atlocalhost:6379.</p><p>Caching improves:</p><ul><li>Latency</li><li>Throughput</li><li>Compute efficiency</li></ul><p>Because a cache is a stateful (albeit weakly consistent) system, it introduces considerations around freshness and correctness. Cached predictions can become stale as models are updated or underlying data distributions shift.</p><p>A simple strategy is to associate each cache entry with a TTL (time-to-live). This ensures entries expire automatically and are recomputed periodically, bounding staleness without requiring explicit invalidation logic.</p><p>For stronger consistency, consider cache invalidation strategies such as:</p><ul><li>Versioned cache keys (e.g., include model version or feature schema hash)</li><li>Explicit invalidation on model deployment</li><li>Selective eviction for known-changing inputs</li></ul><blockquote>In practice, most inference systems balance performance and correctness by combining TTL-based expiration with lightweight versioning.</blockquote><p><strong>2. Kubernetes for Orchestration</strong></p><p>With multiple components (including inference service and Redis cache), orchestration becomes essential.</p><p><a href="https://kubernetes.io/docs/tutorials/kubernetes-basics/"><strong>Kubernetes</strong></a> provides the control plane to define, manage, and scale these components while handling dependencies between them.</p><p>Core Kubernetes resources include:</p><ul><li>Deployments</li><li>Services</li><li>Namespaces</li><li>Scaling policies</li><li>Pod lifecycle management and recovery</li></ul><p>At this stage, define:</p><ul><li>One deployment and service for the model API</li><li>One deployment and service for Redis</li></ul><blockquote>This separation ensures <strong>independent scaling and lifecycle control</strong> for each component.</blockquote><p><strong>3. Namespace Isolation</strong></p><p>For better organization and multi-team collaboration, isolate resources within a dedicated namespace:</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: demo<br>  labels:<br>    name: demo</pre><p>Namespace isolation provides:</p><ul><li>Logical separation of resources</li><li>Improved governance</li><li>Reduced risk of cross-environment interference</li></ul><p><strong>4. Readiness, Liveness, and Startup Probes</strong></p><p>Health checks are a critical production capability that ensures system stability and reliability. We can use the /health endpoint for these probes.</p><p>Add Readiness Probe to control traffic routing:</p><pre># Readiness Probe<br>readinessProbe:<br>  httpGet:<br>    path: /health<br>    port: 8000<br>  initialDelaySeconds: 5<br>  periodSeconds: 5<br>  failureThreshold: 3</pre><p>Add Liveness Probe to detect and restart failed containers:</p><pre># Liveness Probe<br>livenessProbe:<br>  httpGet:<br>    path: /health<br>    scheme: &#39;HTTP&#39;<br>    port: 8000<br>  initialDelaySeconds: 10<br>  periodSeconds: 10<br>  failureThreshold: 3</pre><p>Add Startup Probe to handle slow initialization safely:</p><pre># Startup Probe<br>startupProbe:<br>  httpGet:<br>    path: /health<br>    scheme: &#39;HTTP&#39;<br>    port: 8000<br>  failureThreshold: 30<br>  periodSeconds: 5</pre><p>These probes ensure:</p><ul><li>Traffic is routed only to healthy pods</li><li>Failed pods are automatically restarted</li><li>Slow-starting applications are handled gracefully</li></ul><p><strong>5. Local Deployment with Minikube</strong></p><p>Use <a href="https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download">Minikube</a> to validate the setup in a local Kubernetes environment.</p><p>You can access the services using:</p><ul><li><strong>minikube tunnel</strong></li><li><strong>kubectl port-forward</strong></li></ul><p>Test the application</p><pre># Start local kubernetes<br>minikube start<br><br># Apply configurations<br>kubectl apply -f namespace.yaml<br>kubectl apply -f deployment-redis.yaml<br>kubectl apply -f deployment-model-api.yaml<br>kubectl apply -f service-redis.yaml<br>kubectl apply -f service-model-api.yaml<br><br># Start a tunnel for the model api service<br>minikube service model-api-service -n demo<br><br># Test the service<br>curl -X POST &quot;http://host-ip:port/inference-api&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;&lt;input_data&gt;&#39;</pre><p>This step establishes a strong operational backbone by enabling:</p><ul><li>Self-healing through automated restarts</li><li>Service discovery across components</li><li>Horizontal scaling of workloads</li><li>Fault tolerance in distributed systems</li></ul><p>At this stage, MLOps maturity increases significantly.</p><p>The system evolves from a simple deployment into a <strong>reliable, orchestrated service</strong> capable of handling real-world workloads.</p><p>This step lays the foundation for:</p><ul><li>Scalability</li><li>Resilience</li><li>Orchestration</li><li>Service discovery</li><li>Fault tolerance</li></ul><blockquote>At this stage, the system gains the <strong>core operational capabilities required for production readiness</strong>.</blockquote><p>This is the moment where deployment becomes platform engineering.</p><h4>Step 4— Production Cloud Deployment and Traffic Governance</h4><p>This step transitions the system from a local deployment into enterprise-grade production operations.</p><p>It brings together the foundational capabilities required to operate at scale:</p><ul><li>Cloud deployment</li><li>Traffic management</li><li>Autoscaling</li><li>Observability</li><li>Performance validation</li></ul><p>In this article, we use <strong>AWS EKS</strong> as the reference platform; however, the approach is equally applicable to other managed Kubernetes services such as <strong>Azure AKS</strong> and <strong>GCP GKE</strong>, as well as proprietary or self-managed Kubernetes environments.</p><p>For AWS EKS, you must have an active AWS account and be authenticated with your EKS cluster to push and deploy container images. Similarly, when using other platforms, appropriate authentication and permissions are required to publish container images and deploy them to the target Kubernetes environment.</p><p>This adds managed Kubernetes capabilities while reducing infrastructure management overhead.</p><h4>Key Technology Considerations</h4><ol><li><strong>Container Registry with ECR</strong></li></ol><p>Container images must be versioned and centrally stored to ensure consistent and reliable deployment.</p><p>A best practice is to tag images using the Git commit hash:</p><pre>GIT_HASH=$(git rev-parse --short HEAD)</pre><p>Commit-based tagging provides several advantages:</p><ul><li><strong>Traceability</strong> across deployments</li><li><strong>Reliable rollback</strong> to known states</li><li><strong>Reproducibility</strong> of environments</li></ul><p><strong>2. Environment Overlays with Kustomize</strong></p><p>Production systems typically span multiple environments, each with distinct operational characteristics:</p><ul><li>dev → lower replica count, faster iteration</li><li>stage → production-like validation</li><li>prod → autoscaling, traffic controls, and reliability guarantees</li></ul><p>Managing these environments manually leads to duplication and inconsistency.</p><blockquote><a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/">Kustomize</a> solves this by enabling reusable base configurations with environment-specific overlays.</blockquote><p>This approach allows teams to:</p><ul><li>Maintain a single source of truth</li><li>Apply targeted configuration changes per environment</li><li>Reduce configuration drift across deployments</li></ul><p><strong>3. Autoscaling with Horizontal Pod Autoscaler (HPA)</strong></p><p>The <strong>Horizontal Pod Autoscaler (HPA)</strong> dynamically adjusts the number of running pods based on system demand.</p><p>Example configuration:</p><pre>minReplicas: 1<br>maxReplicas: 40<br>targetCPUUtilizationPercentage: 50</pre><p>This configuration scales the deployment based on CPU utilization, increasing replicas when target CPU usage exceeds 50% and scaling down when demand decreases.</p><p>You can refer to the <a href="https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/">HPA documentation</a> for more details and tailor the configuration to your specific workload and scaling requirements.</p><p>Scaling decisions can be driven by:</p><ul><li>CPU utilization</li><li>Memory usage</li><li>Custom metrics</li><li>Request volume</li></ul><blockquote>Autoscaling is essential for <strong>inference workloads with variable traffic</strong>, ensuring the system can scale efficiently while maintaining performance and cost control.</blockquote><p><strong>4. Traffic Management with Istio</strong></p><p>In enterprise systems, <strong>traffic control and ingress management</strong> are critical engineering concerns.</p><p><strong>Istio</strong> provides a robust service mesh layer that enables:</p><ul><li>Intelligent ingress routing</li><li>Automatic retries and fault tolerance</li><li>Fine-grained traffic splitting</li><li>Canary deployments</li><li>Built-in telemetry and observability</li></ul><p>You can install <a href="https://istio.io/latest/">Istio</a> with the following command:</p><pre>istioctl install --set profile=demo -y</pre><p>Enable automatic <a href="https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/">sidecar injection</a> to inject the Istio sidecar proxy automatically for our workloads:</p><pre>kubectl label namespace default istio-injection=enabled</pre><p>This configures Istio to automatically attach an Envoy-based sidecar proxy to each pod, enabling traffic management, observability, and security features without modifying application code.</p><blockquote>However, Istio’s architecture is evolving away from the per-pod sidecar model toward a more decoupled data plane using <a href="https://istio.io/latest/docs/ambient/overview/">ambient mesh</a>.</blockquote><p>In this model, a shared node-level proxy (ztunnel) handles traffic, reducing per-pod overhead and improving scalability while preserving core service mesh capabilities.</p><p>With Istio, you can define <a href="https://istio.io/latest/docs/reference/config/networking/virtual-service/"><strong>VirtualService</strong></a> resources to control how traffic flows through your system. You should also add <a href="https://istio.io/latest/docs/reference/config/networking/gateway/"><strong>Gateway</strong></a> to accept traffic for the given URL.</p><p>This is a key mechanism for:</p><ul><li>Safe model rollouts</li><li>Progressive delivery strategies</li><li>Minimizing deployment risk</li></ul><p>Follow <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html">this</a> documentation to connect kubectl to an EKS cluster.</p><p><strong>5. Production Deployment via CI/CD</strong></p><p>In production, deployments should be performed through a CI/CD pipeline rather than manual kubectl commands. This ensures consistency, auditability, and controlled rollouts.</p><p>Configure your pipeline (e.g., GitHub Actions, GitLab CI) to authenticate to the Kubernetes cluster using AWS IAM (or respective access control mechanism of your production environment) with OIDC, avoiding long-lived credentials or static kubeconfigs.</p><p>A typical workflow looks like:</p><pre># Use environment specific context<br>kubectl config use-context &lt;EKS-CONTEXT&gt;<br><br># Apply all configurations<br>kubectl apply -k &lt;prod-config.yaml&gt;</pre><p>In this setup:</p><ul><li>The CI/CD runner assumes an IAM role via OIDC</li><li>The IAM role is mapped to Kubernetes RBAC for controlled access</li><li>Deployments are triggered on merge to main (or a release branch)</li></ul><p>Once deployed, you can validate the application:</p><pre># Get the visrtual service URL<br>kubectl get virtualservice -n demo<br><br># Test the service<br>curl -X POST &quot;http://&lt;virtual-service-url&gt;/inference-api&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;&lt;input_data&gt;&#39;</pre><p>While kubectl is still used under the hood by the pipeline, direct manual usage should be minimized in production. This reduces configuration drift and enforces a single, repeatable deployment path.</p><blockquote>This step introduces the essential building blocks required for enterprise deployment.</blockquote><p>These building blocks include:</p><ul><li>Managed Kubernetes infrastructure (EKS)</li><li>Controlled and observable traffic flows</li><li>Dynamic scaling aligned with workload demand</li><li>Environment-aware configuration management</li></ul><blockquote>A system is not production-ready because it is deployed. It is production-ready because it is measurable under stress.</blockquote><h3>Step 5 — Validate Production Readiness with Observability and Load Testing</h3><p>No deployment framework is complete without rigorous operational validation.</p><p>At this stage, the system must be <strong>validated under realistic traffic conditions</strong> and instrumented for full operational visibility.</p><p>This final layer validates operational fitness, ensuring the system behaves reliably and predictably under real-world workloads.</p><h4>Key Engineering Considerations</h4><ol><li><strong>Enable Observability with Istio Addons (Prometheus, Grafana, and Kiali)</strong></li></ol><p>A production-grade system requires deep visibility into runtime behavior.</p><p>Istio provides preconfigured manifests for its core observability stack, enabling rapid setup.</p><p>You can install these addons with the following commands</p><pre># Prometheus<br>kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/addons/prometheus.yaml<br><br># Grafana<br>kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/addons/grafana.yaml<br><br># Kiali<br>kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/addons/kiali.yaml</pre><p>You can access Prometheus, Grafana, and Kiali by using the following command:</p><pre>kubectl port-forward svc/prometheus -n istio-system 9090:9090<br><br>kubectl port-forward -n prometheus svc/grafana 3000:3000<br><br>kubectl port-forward svc/kiali -n istio-system 20001:20001</pre><p>You can then access these tools with their respective URL path:</p><ul><li>Prometheus: <a href="http://localhost:9090/">http://localhost:9090</a></li><li>Grafana at <a href="http://localhost:3000.">http://localhost:3000</a></li><li>Kiali at <a href="http://localhost:20001,">http://localhost:20001</a></li></ul><p>These tools collectively provide:</p><ul><li>Metrics collection (Prometheus)</li><li>Visualization and dashboards (Grafana)</li><li>Service mesh topology and traffic insights (Kiali)</li></ul><p><strong>2. Load Testing with k6</strong></p><p>To validate system behavior under realistic demand, use <a href="https://grafana.com/docs/k6/latest/set-up/install-k6/"><strong>k6</strong></a> to simulate traffic patterns that resemble production workloads.</p><pre>stages: [<br>  { duration: &#39;1m&#39;, target: 100 },<br>  { duration: &#39;5m&#39;, target: 100 },<br>  { duration: &#39;5s&#39;, target: 0 }<br>]</pre><p>This enables teams to evaluate critical performance indicators:</p><ul><li>p50 latency</li><li>p90 latency</li><li>p99 latency</li><li>throughput</li><li>failure rate</li></ul><p><strong>3. Production Observability Metrics</strong></p><p>Observability becomes essential at scale. The system must be continuously monitored across key dimensions:</p><ul><li>Request rate</li><li>Error rate</li><li>Latency percentiles</li><li>Pod scaling</li><li>Cache efficiency</li></ul><p>These metrics provide actionable insight into both performance and system health.</p><p>This step adds organization-scale capabilities, including:</p><ul><li>Enterprise-grade deployment patterns</li><li>Managed scaling and autoscaling strategies</li><li>Production ingress and traffic management</li><li>Deployment governance and operational controls</li></ul><p>This step evolves the system into:</p><ul><li>Cloud-native</li><li>Observable</li><li>SLA-capable</li><li>Enterprise-ready</li></ul><p>This stage represents full production readiness backed by operational confidence.</p><p>It validates not only the deployment infrastructure but the <strong>entire ML system architecture</strong>, ensuring it can sustain real-world workloads at scale.</p><blockquote>At this point, the system is no longer just deployed; it is <strong>proven, measurable, and production-ready</strong>.</blockquote><h3>Deployment Framework as a Maturity Path</h3><p>This framework is more than a deployment checklist; it is a practical <strong>maturity path for ML systems engineering</strong>.</p><blockquote>This framework is effective because it reduces deployment complexity into <strong>bounded engineering milestones</strong>.</blockquote><p>Instead of debugging an entire production stack at once, teams validate one maturity layer at a time.</p><figure><img alt="ML System Maturity Path" src="https://cdn-images-1.medium.com/max/1024/1*hy5f7YSQIDMvx8c6CQJPPw.png" /><figcaption>The ML System Maturity Path</figcaption></figure><p>This is what makes the framework repeatable and scalable across organizations.</p><h4>Key Takeaways</h4><p>Several principles emerge from this framework that are broadly applicable to real-world ML deployments:</p><ul><li>Production ML systems are defined not just by model quality, but by <strong>infrastructure, reliability, and operational design</strong></li><li>A stepwise deployment approach significantly reduces engineering risk by validating one production capability at a time</li><li>Technologies such as FastAPI, Docker, Kubernetes, Redis, Istio, Prometheus, and Grafana are not isolated tools; together, they form a <strong>composable production system</strong></li><li>Caching, orchestration, autoscaling, and routing should be treated as <strong>core system requirements</strong>, not post-deployment optimizations</li><li>Observability is essential for trust, SLA adherence, and operational confidence in production environments</li></ul><h4>Final Thought</h4><p>The real challenge in machine learning is not prediction. It is production reliability.</p><p>Machine learning creates potential at the point of development.</p><p>Business value is realized only when that potential is translated into <strong>reliable, scalable, and observable execution in production</strong>.</p><blockquote>The teams that operationalize models effectively are the ones that consistently turn experiments into measurable impact.</blockquote><p>This framework is designed to make that transition structured, repeatable, and engineering-driven; enabling teams to spend less time asking <em>“How do we deploy this model?”</em> and more time focusing on <em>“What business problem do we solve next?”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7da028ab2d6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/berkeleyischool/from-model-to-production-a-stepwise-framework-for-deploying-machine-learning-system-at-scale-7da028ab2d6e">From Model to Production: A Stepwise Framework for Deploying Machine Learning System at Scale</a> was originally published in <a href="https://medium.com/berkeleyischool">BerkeleyISchool</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Building a Flight Delay Prediction System Taught Me About Real-World Data Science]]></title>
            <link>https://medium.com/berkeleyischool/what-building-a-flight-delay-prediction-system-taught-me-about-real-world-data-science-93ce67359979?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/93ce67359979</guid>
            <category><![CDATA[applied-machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-sicence]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 00:07:51 GMT</pubDate>
            <atom:updated>2026-05-06T22:51:16.143Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reflections and lessons from a large-scale team project in the <em>Machine Learning at Scale</em> course at UC Berkeley</h4><figure><img alt="A graphic detailing lessons learned building a flight delay prediction system at scale. The graphic shows a plane with a board showing flights and their status." src="https://cdn-images-1.medium.com/max/1024/1*AroSOZoQ2tMqFi4QZSpDcg.png" /><figcaption>ML at Production Scale</figcaption></figure><p>When our team started building a flight delay prediction system, I assumed the hardest part would be the model.</p><ul><li>Pick the right algorithm.</li><li>Tune the hyperparameters.</li><li>Optimize the best metric.</li></ul><p>More specifically, we framed the problem as a <strong>binary classification task</strong>: predicting whether a flight would be delayed by <strong>15 minutes or more at departure (Yes/No)</strong>, rather than estimating the exact delay duration.</p><p>But after working through <strong>31 million flight records</strong>, integrating <strong>NOAA weather data</strong>, engineering <strong>237 features</strong>, debugging leakage, handling imbalance, and building a <strong>stacking ensemble with temporal validation</strong>, I realized something far more important:</p><blockquote><strong><em>real-world data science is rarely about the model alone.</em></strong></blockquote><p>It is about designing systems that remain reliable under scale, drift, imbalance, operational constraints, and future uncertainty.</p><p>This project taught me some of the most important lessons about building <strong>production-grade machine learning systems</strong>.</p><p>Here are the key ones.</p><h3>1. Great models start with better data, not better algorithms</h3><p>The single biggest improvement in this project did not come from switching algorithms.</p><p>It came from <strong>rebuilding the data foundation</strong>.</p><p>Instead of relying on the available On Time Performance with Weather (OTPW) dataset, we built the <strong>feature-rich custom dataset</strong> by integrating:</p><ul><li>U.S. DOT flight records</li><li>high-resolution NOAA weather observations</li><li>airport metadata</li><li>network topology information</li></ul><p>This resulted in a unified dataset of <strong>over 31 million flights from 2015–2019</strong>.</p><p>One of the most important design decisions was enforcing strict <strong>temporal causality in feature construction</strong>.</p><p>For every flight, all features were computed using information available only up to the <strong>2-hour pre-departure cutoff</strong>.</p><p>This ensured that no post-cutoff or near-departure leakage entered the feature space, and that every signal reflected only <strong>historical and observed system behavior at prediction time</strong>.</p><p>This strict temporal boundary significantly improved feature reliability and made the learning problem consistent with real-world deployment conditions.</p><p>Building this custom dataset <strong>enabled us to engineer specialized features</strong> such as network-level delay propagation, accumulated weather impacts, and weather spillover effects across connected airports.</p><blockquote>This reinforced one of the most important lessons in data science: <strong><em>model performance is often capped by data quality long before it is capped by model complexity.</em></strong></blockquote><p>The performance jump from this improved data pipeline was larger than many of the subsequent model-level improvements.</p><h3>2. Domain-driven feature engineering beats blind feature expansion</h3><p>This project was a masterclass in how domain knowledge shapes predictive power.</p><p>We engineered <strong>237 features</strong> across five carefully designed families:</p><ul><li>temporal</li><li>operational</li><li>meteorological</li><li>graph-based</li><li>time-series</li></ul><p>This was not random feature expansion.</p><p>Every feature family was grounded in how delays actually happen in the airline ecosystem.</p><p>For example:</p><h4>Operational features</h4><p>These turned out to be some of the strongest predictors.</p><p>Features such as:</p><ul><li>recent 1-hour departure delay</li><li>3-hour congestion volume</li><li>aircraft tail recent delay</li><li>route-level average delay</li></ul><p>captured <strong>delay propagation and system congestion effects</strong>.</p><p>These signals were often stronger than static airport or carrier attributes.</p><h4>Meteorological features</h4><p>Single weather observations were useful.</p><p>But the stronger signals came from <strong>cumulative and composite weather effects</strong>.</p><p>We created <strong>composite features such as a weather severity index</strong>, which captured the compounding impact of multiple adverse conditions on delay probability.</p><p>Rolling aggregates over <strong>3h, 6h, and 12h windows</strong> provided much stronger predictive signals than point-in-time measurements.</p><p>This was most evident for:</p><ul><li>precipitation</li><li>low visibility</li><li>high winds</li><li>humidity persistence</li></ul><p>Another important learning came from missing data.</p><p>Several weather variables were <strong>sparsely populated but operationally critical</strong>. Instead of applying generic imputations, we used <strong>domain-driven imputation logic</strong> grounded in the meteorological context.</p><p>One of the most interesting findings was that <strong>weather effects were rarely local</strong>. Poor weather conditions at neighboring and upstream airports materially increased the delay risk.</p><p>This reinforced a broader systems insight: <strong><em>flight delays are network events, not isolated airport events.</em></strong></p><p>That understanding materially strengthened the predictive power of the weather feature set.</p><h4><strong>Graph features</strong></h4><p>These features helped capture the structural role of airports within the national flight network to model system-wide connectivity effects.</p><p>Centrality-based measures such as</p><ul><li>PageRank,</li><li>hub score,</li><li>authority score, and</li><li>betweenness centrality</li></ul><p>provide a system-level perspective on delay dynamics, shifting the analysis from isolated airport behavior to network-driven interactions.</p><p>The lesson here was clear: <strong><em>effective feature engineering starts with understanding how the system behaves, not just what columns exist.</em></strong></p><h3>3. Leakage is not a modeling problem — it is a systems problem</h3><p>One of the most valuable lessons from this exercise was how easily leakage can enter a pipeline.</p><p>In temporal prediction systems, leakage rarely announces itself.</p><p>It often emerges through subtle design choices rather than obvious errors.</p><p>This project made that particularly clear when we observed how early <strong>resampling decisions and preprocessing choices interacted with temporal cross-validation</strong>.</p><p>When imbalance handling was applied globally before fold creation, it unintentionally influenced how data was distributed across time-aware splits.</p><p>While this was not classical target leakage, it blurred the boundary between <strong>training data construction and validation structure</strong>, which is a core principle in time-series modeling.</p><p>This reinforced a broader point: <strong><em>leakage is not only about features; it is also about pipeline design</em></strong></p><p>A full leakage audit became one of the most important phases of the project.</p><p>We explicitly checked for:</p><ul><li>target leakage</li><li>temporal leakage</li><li>train-test contamination</li><li>future information bleed</li></ul><p>This led to several key controls:</p><ul><li>strict <strong>2015–2018 train / 2019 test split</strong></li><li>only pre-departure features</li><li>fold-consistent transformations</li><li>train-only preprocessing</li><li>no post-event variables</li></ul><p>The most important realization was that validation integrity depends on <strong>how the dataset is constructed over time,</strong> not just how models are evaluated.</p><p>This project fundamentally changed how I think about validation.</p><blockquote><strong><em>Good validation is not just model evaluation — it is architecture validation.</em></strong></blockquote><h3>4. Scale changes how you think as a data scientist</h3><p>Working with <strong>31 million records</strong> changes everything.</p><p>At this scale, many workflows that work perfectly in notebooks become impractical.</p><p>The challenge shifts from pure modeling to <strong>computational strategy</strong>.</p><p>This forced us to design around:</p><ul><li><strong>PySpark pipelines optimized for cluster resource constraints</strong>, including intelligent partitioning to process manageable chunks of data at a time and frequent checkpointing to break long lineage chains and reduce memory pressure during distributed execution</li><li><strong>Vectorized feature storage</strong>, enabling efficient handling of high-dimensional feature sets and consistent feature representation across models</li><li><strong>Persistent caching of fold datasets, fold models, and out-of-fold predictions</strong>, which became especially important for efficiently building the stacking ensemble without repeatedly recomputing expensive transformations</li><li><strong>Distributed transformations and joins</strong>, designed to keep large-scale preprocessing and feature engineering operations scalable and fault-tolerant</li><li><strong>Spark–PyTorch interoperability</strong>, allowing deep learning models such as LSTM to integrate seamlessly with PySpark-based datasets and distributed preprocessing workflows</li></ul><p>Even generating out-of-fold predictions for stacking became an engineering challenge.</p><p>The lesson was:</p><blockquote><strong><em>production-grade data science requires engineering discipline.</em></strong></blockquote><p>At scale, algorithmic correctness alone is not enough.</p><p>Efficiency, reproducibility, and memory strategy matter just as much.</p><h3>5. Imbalance handling is a business decision disguised as a modeling decision</h3><p>Flight delays were not evenly distributed.</p><p>The data exhibited a natural <strong>80/20 class imbalance</strong>, with roughly 80% on-time flights and 20% delayed flights.</p><p>For Logistic Regression, MLP, and LSTM, we applied <strong>fixed random undersampling on the full 2015–2018 training set before temporal cross-validation folding</strong>, forcing a near <strong>1:1 class balance</strong>.</p><p>This improved recall significantly.</p><p>But it also surfaced a deeper structural issue in how imbalance handling interacts with temporal validation.</p><p>Because resampling was applied <strong>before fold creation</strong>, the downstream folds were drawn from a dataset that had already been globally rebalanced across the entire training horizon.</p><p>This introduced a subtle distortion:</p><p>earlier temporal folds were no longer strictly reflective of their original time-specific class distributions.</p><p>That meant imbalance handling was not just affecting training stability; it was also influencing <strong>how the model perceived temporal structure during validation</strong>.</p><p>An ideal design would instead apply imbalance handling <strong>within each fold</strong>, preserving the natural evolution of class distributions over time while still controlling class bias locally.</p><p>XGBoost followed a different strategy.</p><p>It trained on the full dataset and used <strong>class-weighted loss</strong>, avoiding any data removal while still prioritizing minority-class errors.</p><p>This contrast highlighted an important production insight: <strong><em>resampling strategies do not just affect class balance; they reshape the validation process itself.</em></strong></p><p>This reinforced that imbalance handling is not only a modeling choice, but also a <strong>validation design decision tied directly to temporal integrity</strong>.</p><p>More importantly, it reinforced that imbalance handling must align with business priorities:</p><ul><li>false negatives = missed delays = high operational cost</li><li>false positives = false alerts = operational overhead</li></ul><p>That is why we optimized around <strong>F2 score and recall</strong>, not accuracy.</p><h3>6. The best production model is often an ensemble of perspectives</h3><p>No single model consistently dominated.</p><p>Each model captured different aspects of the delay-generation process.</p><p><strong>Logistic Regression </strong>provided:</p><ul><li>strong interpretability</li><li>stable linear baseline</li><li>very high recall</li></ul><p><strong>XGBoost</strong> captured:</p><ul><li>nonlinear interactions</li><li>threshold effects</li><li>complex feature dependencies</li></ul><p><strong>LSTM a</strong>dded:</p><ul><li>temporal persistence</li><li>sequential delay memory</li><li>short-term propagation patterns</li></ul><p>This led to the final design: <strong><em>stacking-based ensemble learning.</em></strong></p><p>The ensemble combined all three models using a l<strong>ogistic regression meta-learner trained on out-of-fold predictions</strong>.</p><p>This was one of the most important modeling lessons from the project: <strong><em>different models fail differently and that is exactly why ensembles work.</em></strong></p><p>The final ensemble achieved:</p><ul><li><strong>F2 = 0.8311</strong></li><li><strong>Precision = 0.6253</strong></li><li><strong>Recall = 0.9057</strong></li></ul><p>making it the strongest production candidate.</p><h3>7. The hardest production problem is drift, not training</h3><p>The biggest gap in the final model was not recall.</p><p>It was <strong>precision collapse under temporal drift.</strong></p><p>We observed nearly a <strong>25-point precision drop from validation to test</strong></p><p>caused by:</p><ul><li>temporal distribution shift</li><li>changing operational patterns</li><li>concept drift between years</li></ul><p>This was perhaps the most important production lesson.</p><p>A strong model today may degrade tomorrow.</p><blockquote>This means: <strong><em>production-grade ML systems must be adaptive, not static.</em></strong></blockquote><p>This is where future work naturally evolves toward:</p><ul><li>threshold recalibration</li><li>drift monitoring</li><li>adaptive retraining</li><li>online threshold optimization</li></ul><p>This project made one thing very clear: <strong><em>model deployment is the beginning of the ML lifecycle, not the end.</em></strong></p><h3>Final Reflection</h3><p>Looking back, this project was never really about predicting flight delays.</p><p>It was about understanding what happens when machine learning moves from controlled environments into real systems with scale, noise, constraints, and time dependency.</p><blockquote>The biggest lesson from this project was simple and yet fundamental: <strong><em>machine learning success comes from systems thinking.</em></strong></blockquote><p>The strongest gains did not come from any single model. They came from how the system was designed end-to-end:</p><ul><li><strong>better data design</strong> at scale (31M+ flights with weather alignment)</li><li><strong>compute-optimized PySpark pipelines</strong> built for distributed execution</li><li><strong>domain-driven feature engineering</strong> across operational, weather, and network signals</li><li><strong>strict leakage prevention</strong> through time-aware validation design</li><li><strong>business-aligned threshold selection</strong> driven by recall-heavy objectives (F2 optimization)</li><li><strong>adaptive validation</strong> using sliding temporal windows</li><li><strong>complementary model fusion</strong> through stacking (LR + XGBoost + LSTM)</li></ul><p>Each component contributed to the final performance. No single component was sufficient on its own.</p><p>The algorithm was only one part of the solution.</p><p>Everything else, data, validation, imbalance strategy, thresholds, and architecture, determined whether the model would actually work in a realistic setting.</p><p>For data scientists building production systems, this is the real shift: <em>move from </em><strong><em>“which model is best?”</em></strong><em> to </em><strong><em>“which system makes the best decisions under real-world constraints?”</em></strong></p><p>That mindset changed how I approach machine learning.</p><p>And ultimately, it was the most valuable learning from this entire project.</p><p>Happy Modeling!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93ce67359979" width="1" height="1" alt=""><hr><p><a href="https://medium.com/berkeleyischool/what-building-a-flight-delay-prediction-system-taught-me-about-real-world-data-science-93ce67359979">What Building a Flight Delay Prediction System Taught Me About Real-World Data Science</a> was originally published in <a href="https://medium.com/berkeleyischool">BerkeleyISchool</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise Agentic AI: From Foundations to Real-World Systems]]></title>
            <link>https://medium.com/@sunil.baidyanath/enterprise-agentic-ai-from-foundations-to-real-world-systems-cd78cca7581c?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd78cca7581c</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 00:40:06 GMT</pubDate>
            <atom:updated>2026-01-02T00:40:06.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CN92plCWvmXwWmSdjms0g.png" /></figure><p>For decades, enterprises believed the next technology breakthrough would deliver a lasting competitive edge — faster computing, better software, cheaper infrastructure, or smarter systems. Generative AI shattered that assumption.</p><p>When intelligence can be deployed with a prompt, intelligence itself stops being rare. The advantage no longer lies in having better models, but in something far more difficult: <strong>embedding intelligence into the enterprise ecosystem</strong> — orchestrating decisions, automating execution, and enforcing accountability across real business systems.</p><p>The race is no longer about generating smarter outputs. It is about <strong>operationalizing intelligence</strong> across the enterprise in a way that is reliable, responsible, and scalable — without compromising trust, control, or accountability.</p><h4>Context</h4><p>Early enterprise adoption of generative AI focused on copilots, chat interfaces, and content generation. These tools delivered real productivity gains, but they remained largely assistive — dependent on human prompts, approvals, and manual handoffs.</p><p>As a result, many initiatives stalled at experimentation. They improved individual productivity but rarely penetrated core workflows or mission-critical systems. At the same time, a new class of AI systems has begun to emerge: <strong>agentic AI</strong>.</p><p>Agentic systems extend beyond content generation to plan tasks, coordinate actions, interact with tools and data, and adapt across multi-step workflows. Rather than supporting isolated activities, they operate across enterprise environments with varying degrees of autonomy — reshaping how work itself can be executed.</p><p>This marks a clear inflection point:</p><ul><li>From AI that responds → to AI that operates</li><li>From isolated tasks → to end-to-end workflows</li><li>From experimentation → to enterprise execution</li></ul><h4>Motivation</h4><p>Enterprises sit on vast amounts of proprietary data, institutional knowledge, and operational processes — yet much of this value remains locked behind silos, manual workflows, and brittle automation.</p><p>Agentic AI offers a path to:</p><ul><li>Unlock complex, context-dependent workflows</li><li>Reduce operational friction across teams and systems</li><li>Augment human judgment rather than replace it</li><li>Scale intelligence while preserving trust, control, and accountability</li></ul><p>But increased autonomy also introduces increased risk. Agentic systems raise new challenges around governance, safety, observability, and organizational readiness — challenges that cannot be solved by model selection alone.</p><p>Realizing the promise of agentic AI requires new architectures, operating models, skills, and governance patterns — and a clear understanding of <strong>when not to delegate autonomy</strong>.</p><h4>Posts in this Series</h4><p>This series explores <strong>Agentic AI from an enterprise lens</strong>, focusing not only on what is possible, but on what is <em>practical, safe, and sustainable</em>.</p><ol><li><a href="https://medium.com/@sunil.baidyanath/agentic-ai-in-the-enterprise-cc0d458cb0de">Agentic AI in the Enterprise</a></li><li>Building Agentic AI Systems in Enterprises (Upcoming)</li><li>Designing a Sample Agentic AI System: A Reference Implementation</li><li>Agentic AI in Healthcare</li><li>Agentic AI in Education</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd78cca7581c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI in the Enterprise]]></title>
            <link>https://medium.com/@sunil.baidyanath/agentic-ai-in-the-enterprise-cc0d458cb0de?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc0d458cb0de</guid>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 22:07:23 GMT</pubDate>
            <atom:updated>2026-01-02T00:42:59.881Z</atom:updated>
            <content:encoded><![CDATA[<h4>What agentic AI really is, where it creates value, and how enterprises should approach it</h4><p><em>This is the first post of the series </em><a href="https://medium.com/@sunil.baidyanath/enterprise-agentic-ai-from-foundations-to-real-world-systems-cd78cca7581c"><em>Enterprise Agentic AI: From Foundations to Real-World Systems.</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8XgSRy8TAYszcvGIISEHA.png" /><figcaption>Image Generated by OpenAI</figcaption></figure><p>Generative AI is entering its next phase. Beyond copilots and single-turn interactions, enterprises are beginning to explore systems that can take initiative and are capable of translating intent into coordinated actions across complex workflows.</p><p>Agentic AI represents this evolution. Unlike traditional generative AI, which reacts to prompts within a limited context, agentic systems are designed to pursue objectives, sequence tasks, invoke tools, and operate across enterprise systems with constrained autonomy. For enterprises, this marks a fundamental shift:AI is no longer just supporting human work; it is beginning to participate directly in getting work done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9LvIeO7j86-vfCKSkLDeQ.png" /><figcaption>Agentic AI in the Enterprise</figcaption></figure><p>This shift offers the potential for significant gains in speed, efficiency, and scale but it also introduces new challenges. Increased autonomy raises critical questions around trust, control, skills, and organizational readiness. Not every workflow benefits from delegation, and not every organization is prepared to manage systems that can act independently.</p><p>Understanding where agentic AI creates real value and how to adopt it responsibly is now a pressing concern for enterprise leaders.</p><p>This article focuses on the <em>what</em> and <em>why</em> of agentic AI in the enterprise:</p><ul><li>What “agentic” actually means in practice</li><li>How agentic systems differ from traditional generative AI</li><li>Where they create real enterprise value — and where they don’t</li><li>What this shift implies for skills, roles, and operating models</li></ul><h4>What does “Agentic” mean in Agentic AI?</h4><p>“Agentic” refers to the ability to act with purpose and autonomy toward a goal. Rather than behaving as a reactive and assistive tool, agentic AI functions more like a collaborator that is capable of reasoning, troubleshooting, and following through to complete workflows.</p><p>In practice, this represents a clear transition:</p><ul><li>From answering questions → to accomplishing tasks</li><li>From isolated responses → to end-to-end workflows</li><li>From human-driven orchestration → to delegated execution</li></ul><p>Agentic AI does not replace human judgment. Instead, it reduces the need for constant direction, enabling AI systems to perform meaningful work rather than merely assist with it.</p><blockquote>In short, agentic AI shifts AI from <em>answering questions</em> to <em>getting work done</em>.</blockquote><h4>Why Agentic AI Matters Now</h4><p>Agentic AI is rapidly emerging as a foundational capability for enterprises seeking to reduce operational costs while unlocking new sources of value.</p><p>Rather than treating AI as a passive assistant, organizations are beginning to adopt operating models in which humans and AI agents work side by side; accelerating execution, improving decision quality, and delivering more responsive, connected experiences.</p><p>Agentic AI enables organizations to build autonomous agents that can continuously pursue defined objectives by:</p><ul><li>Decomposing goals into tasks</li><li>Selecting appropriate tools and data sources</li><li>Executing multi-step workflows</li><li>Monitoring outcomes and adapting actions</li></ul><p>This evolution moves generative AI from <em>assistive</em> to <em>operational</em>.</p><p>Agentic systems <strong>combine the flexibility of large language models with the rigor of traditional software</strong>, allowing workflows to be designed and decisions to be made based on real-time data. They also <strong>enhance human–machine collaboration</strong> by enabling agents to observe, reason, and act alongside humans to improve operational efficiency.</p><p>Crucially, agentic AI has become viable for enterprises because following three key capabilities matured at the same time:</p><ul><li>Function calling and tool orchestration</li><li>Retrieval-Augmented Generation (RAG) at enterprise scale</li><li>Improved planning and reasoning in modern LLMs</li></ul><p>Together, these advancements enable AI systems to do more than generate responses and take meaningful action alongside humans within real enterprise environments.</p><h4>How Agentic AI Differs from Traditional GenAI</h4><p>Traditional generative AI is reactive. It generates content based on prompts and relies on humans to drive workflows.</p><p>On the contrary, agentic AI is goal-driven and proactive. It perceives its environment, reasons about actions, executes tasks, and adapts based on feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZmrtzGeUvvvinMG7n0EIg.png" /><figcaption>Differences between Traditional GenAI and Agentic AI</figcaption></figure><p>Key differences include:</p><ul><li><strong>Autonomy:</strong> Limited vs. delegated autonomy</li><li><strong>Memory:</strong> Stateless or short context vs. persistent state</li><li><strong>Primary Use:</strong> Content creation vs. goal execution</li><li><strong>Complexity:</strong> Single tasks vs. multi-step workflows</li><li><strong>Enterprise Impact:</strong> Localized vs. system-wide</li></ul><p>This increased autonomy makes agentic AI transformative — but also increases risk, making governance, safety, and control essential.</p><h4>What is an Agentic AI System?</h4><p>An agentic AI system is a generative AI–powered system designed to achieve specific goals with minimal human supervision. It is composed of one or more AI agents that emulate human decision-making to solve problems in real time.</p><blockquote>An AI agent is an autonomous machine learning entity capable of planning, acting, and adapting in pursuit of an objective.</blockquote><p>Rather than viewing an agent as a tool, it is more useful to think of it as a digital operator with a defined role, authority, and constraints. At a conceptual level, an <strong>AI</strong> <strong>agent combine goal-setting, reasoning, memory, and action</strong>.</p><p>A typical agentic AI system is designed to:</p><ul><li>Pursue a defined objective</li><li>Plan and sequence actions</li><li>Use tools, APIs, and data sources</li><li>Evaluate intermediate results</li><li>Adapt behavior dynamically</li></ul><p>In enterprise systems, these capabilities are rarely monolithic; they are instead distributed across platforms, controls, and operating models.</p><p>Unlike traditional automation, which relies on deterministic rules, agentic AI systems operate using probabilistic reasoning that allows them to handle ambiguity, exceptions, and changing conditions.</p><h4><strong>Core Characteristics of an Agentic AI System</strong></h4><p>While “agentic” describes <em>intent</em>, these core haracteristics describe <em>capability </em>of an agentic AI system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*5mXt4bpiM1rt3zGfuoBY5A.png" /><figcaption>Characteristics of an Agentic AI System</figcaption></figure><p><strong>1. Goal Orientation: </strong>Agentic systems operate toward defined outcomes, not isolated responses. Every action is evaluated based on progress toward a specified goal.</p><p><strong>2. Bounded Autonomy: </strong>Agents operate independently, but never without limits. Their authority is explicitly delegated through policies, permissions, and constraints ensuring autonomy is intentional and controlled.</p><p><strong>3. Reasoning &amp; Planning: </strong>Agentic systems can plan, sequence, and execute actions across time. They chain reasoning, tool use, and validation into coherent, end-to-end workflows rather than single-turn interactions.</p><p><strong>4. Tool Invocation: </strong>Agentic AI does not live in a chat window. Agents interact directly with enterprise systems that includes calling APIs, querying databases, navigating codebases, and triggering workflows.</p><p><strong>5. Context Awareness: </strong>Agents maintain memory across steps, tracking progress, decisions, and intermediate results. This continuity allows course correction and grounding in real enterprise environments.</p><p><strong>6. Evaluation &amp; Adaptation: </strong>When conditions change, data is missing, or tools fail, agents adjust their approach rather than stopping or immediately escalating. This adaptive loop enables resilience and sustained execution.</p><p>Together, these capabilities enable agentic AI systems to function as <em>digital operators </em>for<em> </em>executing meaningful work within real enterprise systems rather than merely generating outputs.</p><h4><strong>Core Operational Loop of an Agent</strong></h4><p>Agentic AI systems operate as continuous, goal-driven loops rather than linear pipelines. Across this loop, orchestration and governance act as persistent guardrails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T52Xd3ZnIYKhUKEp9AgtgQ.png" /><figcaption>Operational Loop of an Agent</figcaption></figure><p>At a high level, an agent’s operational cycle includes:</p><ul><li><strong>Perceive:</strong> Gather signals from users, APIs, databases, events, and telemetry</li><li><strong>Reason:</strong> Interpret intent, constraints, and context</li><li><strong>Plan:</strong> Decompose objectives and define executable strategies</li><li><strong>Decide:</strong> Select actions while enforcing policy and authorization</li><li><strong>Execute:</strong> Interact with systems within defined boundaries</li><li><strong>Observe:</strong> Capture outcomes, errors, and side effects</li><li><strong>Evaluate:</strong> Assess progress, confidence, and compliance</li><li><strong>Adapt:</strong> Refine plans, retry, escalate, or terminate execution</li></ul><p>This cyclical model guided by governance enables reliable operation in dynamic enterprise environments.</p><h4>Common Agent Types in the Enterprise</h4><p>As agentic AI adoption expands, enterprises benefit from a clear and pragmatic taxonomy of agent types.</p><p>Not all agents operate with the same level of autonomy, decision authority, or business impact. Distinguishing among agent types helps organizations deploy the right level of intelligence for the right kind of work while maintaining appropriate controls as risk and complexity increase.</p><p>In practice, enterprises should adopt agent types progressively, increasing autonomy only as governance, observability, and organizational maturity evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__wj7IJIhcOvVT8jbvYuJg.png" /><figcaption>Agents Types in the Enterprise on the Spectrum of Autonomy and Risk/Impact</figcaption></figure><p><strong>1. Task Agents</strong></p><p>Task agents execute well-defined, bounded actions such as retrieving data, validating inputs, generating reports, or performing single-step operations. They operate with minimal autonomy and typically sit at the lower end of enterprise risk and impact.</p><p><strong>Best suited for:</strong> deterministic subtasks, human augmentation, copilots, and early-stage agent adoption.</p><p><strong>2. Workflow Agents</strong></p><p>Workflow agents coordinate multi-step processes across tools and systems. They decompose objectives into tasks, manage dependencies, handle exceptions, and recover from partial failures. Compared to task agents, they operate with higher autonomy and business impact.</p><p><strong>Best suited for:</strong> end-to-end business processes, IT operations, onboarding flows, and automation that goes beyond traditional RPA.</p><p><strong>3. Decision-Support Agents</strong></p><p>Decision-support agents analyze data, monitor signals, and surface insights or recommendations without directly executing actions. They maintain context over time and support human judgment with evidence-backed reasoning, often operating in moderate-to-high impact domains.</p><p><strong>Best suited for:</strong> analytics, forecasting, compliance interpretation, knowledge work, and executive decision support.</p><p><strong>4. Supervisory / Meta-Agents</strong></p><p>Supervisory agents operate at the highest levels of autonomy and enterprise impact. Rather than executing tasks, they oversee other agents; monitoring performance, detecting failures or drift, enforcing policies, and deciding when to retry, reroute, or escalate.</p><p><strong>Best suited for:</strong> multi-agent systems, enterprise-scale deployments, resilience, governance enforcement, and operational safety.</p><p><strong>5. Human-in-the-Loop (HITL) Agents</strong></p><p>Human-in-the-Loop is not a peer agent type in the same sense as task or supervisory agents. Instead, it is a governance and interaction pattern that can be applied across agent types based on enterprise risk, regulatory requirements, and organizational maturity.</p><blockquote>Both decision-support and workflow agent types overlap with HITL controls but while the former requires humans to act, the later acts unless humans intervene.</blockquote><p>HITL mechanisms introduce human oversight at defined checkpoints — such as approvals, explanations, exception handling, or confidence-based escalation. These become more prominent as agents move into higher-impact and higher-risk domains.</p><p><strong>Best suited for:</strong> regulated workflows, high-stakes decisions, compliance-heavy environments, and trust-critical operations.</p><p>Key agentic capabilities such as conversational interaction or adaptive behavior cut across these agent types rather than defining them. In mature enterprise deployments, <strong>task and workflow agents operate under supervisory and HITL controls</strong>, forming layered systems that balance autonomy with accountability.</p><p>This progression allows enterprises to scale agentic AI responsibly, increasing automation and intelligence while preserving trust, control, and alignment with business outcomes.</p><h4>High-Value Enterprise Use Cases</h4><p>Agentic AI delivers the most value in goal-driven, multi-step, tool-heavy workflows where context and adaptation are important. Below are the <strong>highest-value enterprise problem areas</strong> where agentic AI is already delivering impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*cUk9jXjofmN26ax9_-OkFw.png" /><figcaption>High-Value Enterprise Use Cases</figcaption></figure><p><strong>1. Knowledge Operations &amp; Decision Support</strong></p><p>Enterprises struggle with fragmented knowledge, rapid information change, and slow synthesis across internal and external sources.</p><p>Agentic AI enables continuous research, monitoring, and synthesis via maintaining living knowledge bases, producing role-specific briefings, and surfacing regulatory, market, or competitive signals as they evolve. These agents reduce time-to-insight while preserving traceability, source attribution, and decision context over time.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI answers questions on demand; agentic AI continuously monitors, updates, and curates knowledge. RPA cannot reason across unstructured sources or adapt to changing information.</p><p><strong>2. IT, Platform &amp; Site Reliability Operations</strong></p><p>Modern IT environments generate overwhelming volumes of alerts, logs, and telemetry.</p><p>Agentic AI performs incident triage, correlates signals across systems, conducts root-cause analysis, and executes remediation workflows such as rollbacks or scaling while adapting to partial failures. This reduces MTTR and operator fatigue without eliminating human oversight.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI can explain incidents, but cannot act. RPA breaks when conditions change. Agentic systems reason under uncertainty and adapt remediation strategies dynamically.</p><p><strong>3. Business Process Automation (Beyond RPA)</strong></p><p>Traditional automation fails when workflows involve exceptions, ambiguity, or cross-system coordination.</p><p>Agentic AI extends automation to complex processes such as onboarding, claims handling, procurement, and order fulfilment reasoning through edge cases, branching logic, and recovery paths while escalating to humans when required.</p><p><strong>Why agentic (not GenAI/RPA):</strong> RPA follows rigid rules; GenAI generates content. Agentic AI manages end-to-end workflows that require judgment, sequencing, and exception-handling.</p><p><strong>4. Governance, Risk, and Compliance (GRC)</strong></p><p>Compliance is continuous, evidence-heavy, and context-dependent.</p><p>Agentic AI interprets policies, monitors workflows for violations, gathers audit evidence, and performs ongoing risk assessments with built-in human review checkpoints. Agents operate within strict guardrails, ensuring transparency and accountability.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI explains policies; RPA checks static rules. Agentic AI continuously enforces, monitors, and adapts compliance actions as conditions change.</p><p><strong>5. Data, Analytics &amp; Information Management</strong></p><p>Data teams face scale, fragmentation, and quality challenges.</p><p>Agentic AI automates data quality checks, metadata generation, semantic search, lineage analysis, and hypothesis-driven exploration across datasets while preserving governance and provenance.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI summarizes dashboards; RPA moves data. Agentic AI reasons across data assets, validates assumptions, and adapts exploration based on findings.</p><p><strong>6. Engineering &amp; Developer Productivity</strong></p><p>Large codebases and rapid release cycles strain engineering teams.</p><p>Agentic AI coordinates refactoring, dependency upgrades, testing, security scanning, and remediation planning across repositories maintaining context across changes rather than operating file by file.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI writes code snippets; RPA automates scripts. Agentic AI manages multi-step engineering workflows with dependency awareness and validation loops.</p><p><strong>7. Customer Operations &amp; Service Management</strong></p><p>Customer journeys span channels, systems, and time.</p><p>Agentic AI triages cases, coordinates resolution across platforms, manages escalations, detects issues proactively, and supports long-lived interactions while ensuring policy-compliant responses and human hands-off when needed.</p><p><strong>Why agentic (not GenAI/RPA):</strong> Chatbots handle conversations; RPA routes tickets. Agentic AI orchestrates end-to-end resolution with memory, judgment, and coordination.</p><p><strong>8. Finance &amp; Revenue Operations</strong></p><p>Finance workflows demand accuracy, controls, and contextual interpretation.</p><p>Agentic AI supports invoice exception handling, revenue leakage detection, forecasting scenarios, and contract compliance operating within approval boundaries and audit requirements.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI explains numbers; RPA posts transactions. Agentic AI reasons through financial exceptions while maintaining governance and traceability.</p><p><strong>9. Security Operations (SecOps)</strong></p><p>Security teams face fast-moving, adversarial threats.</p><p>Agentic AI correlates alerts, ingests threat intelligence, investigates incidents, and executes containment actions — while maintaining clear decision trails and human oversight.</p><p><strong>Why agentic (not GenAI/RPA):</strong> GenAI summarizes threats; RPA executes fixed playbooks. Agentic AI adapts response strategies as attackers evolve.</p><p>Not all enterprises should adopt these use cases simultaneously. The right starting point depends on <strong>organizational maturity, risk tolerance, competitive pressure, and growth ambition</strong>. Enterprises should start where <strong>value is high and risk is manageable</strong>, then progress toward domains where <strong>agentic autonomy becomes a competitive differentiator</strong>.</p><blockquote>Enterprises that align adoption with maturity rather than ambition alone will scale agentic AI safely, sustainably, and ahead of competitors.</blockquote><p>Early-stage adopters should begin with <strong>low-risk, high-leverage domains</strong> such as <em>Knowledge Operations, Developer Productivity,</em> and <em>Data Management, </em>while advanced adopters may deploy agentic systems in <em>Finance, Revenue Operations,</em> and <em>Security</em>.</p><h4>When Not to Use Agentic AI (Guardrails)</h4><p>AI agents are powerful, but they’re not magic. Agentic AI is not appropriate when:</p><ul><li><strong>Deterministic precision is required</strong></li><li><strong>Error tolerance is near zero</strong></li><li><strong>Legal accountability cannot be delegated</strong></li><li><strong>Workflows are simple and linear</strong></li><li><strong>Observability and rollback are insufficient</strong></li><li><strong>Ownership and escalation paths are unclear</strong></li></ul><p>In these cases, traditional automation or human-in-the-loop systems remain the better choice.</p><h4>Organizational Implications</h4><p>Adopting agentic AI reshapes not only technology stacks, but how work is designed, executed, and governed. Enterprises must align roles, skills, and operating models to support a hybrid human–agent workforce.</p><p>This evolution spans three dimensions:</p><ul><li><strong>New roles</strong> to define and supervise agent behavior</li><li><strong>New skills</strong> to work fluently with autonomous systems</li><li><strong>New operating models</strong> aligned to outcome-driven execution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6F7yI0qRMFALwdpYyEecfQ.png" /><figcaption>Layered Organizational Model for Agentic AI</figcaption></figure><p>The challenge is not simply adding AI specialists or retraining existing teams. It is redesigning the enterprise to support distributed autonomy where responsibility is shared between humans and machines and where performance is measured by outcomes, resilience, and trust rather than task completion alone.</p><p><strong>Emerging New Roles</strong></p><p>Agentic AI introduces new responsibilities that cut across strategy, engineering, and operations. These roles are best understood as part of a coordinated system rather than isolated job titles.</p><p>1. AI Product Owners / Agent Owners</p><ul><li>Define agent objectives, success metrics, and business value.</li><li>Decide where autonomy is appropriate and where human oversight is required.</li><li>Act as accountable owners for agent outcomes in production.</li></ul><p>2. AI Governance, Risk &amp; Compliance Specialists</p><ul><li>Establish standards for acceptable agent behavior, safety, privacy, and fairness.</li><li>Monitor regulatory exposure and enforce policy controls across agents.</li><li>Serve as custodians of trust, transparency, and auditability.</li></ul><p>3. Agent Designers &amp; Orchestrators</p><ul><li>Translate business goals into agent roles, constraints, and workflows.</li><li>Design single-agent and multi-agent coordination patterns.</li><li>Ensure agents align with enterprise policies, data contracts, and APIs.</li></ul><p>4. AI Platform &amp; AgentOps Engineers</p><ul><li>Build and maintain the agentic platform, orchestration layers, and tool gateways.</li><li>Implement execution monitoring, lifecycle management, and resilience.</li><li>Often include AgentOps engineers, LLM reliability engineers, and AI SREs.</li></ul><p>5. Prompt &amp; Workflow Engineers</p><ul><li>Design structured prompts, role definitions, and task decomposition strategies.</li><li>Encode business heuristics, constraints, and escalation logic into workflows.</li><li>Continuously refine agent behavior based on real-world feedback.</li></ul><p>6. AI Operations Teams</p><ul><li>Operate agent fleets in production environments.</li><li>Handle incident response, rollback, and performance tuning.</li><li>Coordinate upgrades, configuration changes, and controlled experiments.</li></ul><p>7. Observability &amp; Trust Teams</p><ul><li>Monitor agent decisions, actions, and outcomes end-to-end.</li><li>Detect drift, misuse, bias, or abnormal behavior.</li><li>Provide explainability, reporting, and audit support for stakeholders.</li></ul><p><strong>New Skill Sets Required</strong></p><p>Adopting agentic AI is less about hiring a single new role and more about raising the baseline capabilities across the organization. Enterprises must invest in the following key capabilities.</p><ol><li>AI &amp; Agentic Technical Fluency</li></ol><ul><li>Prompt and workflow design for multi-step reasoning</li><li>API, tool, and system integration</li><li>Agent lifecycle management and observability</li><li>Model evaluation, validation, and performance optimization</li><li>Orchestration of single and multi-agent systems</li></ul><p>2. Data &amp; Knowledge Mastery</p><ul><li>Data quality assessment and semantic consistency</li><li>Knowledge modeling, metadata, and ontologies</li><li>Provenance, freshness, and governance-aware data usage</li><li>Cross-domain data integration to support contextual reasoning</li></ul><p>3. Human–Agent Interaction Design</p><ul><li>Knowing when to delegate, supervise, or intervene</li><li>Designing clear escalation paths and HITL checkpoints</li><li>Teaching agents domain context, constraints, and heuristics</li><li>Interpreting agent decisions and correcting behavior</li></ul><p>4. Governance &amp; Ethical Judgment</p><ul><li>Risk assessment and maturity-based controls</li><li>Bias detection and mitigation</li><li>Privacy, security, and regulatory compliance awareness</li><li>Designing safety guardrails proportional to agent autonomy</li></ul><p>5. Leadership &amp; Cultural Adaptability</p><ul><li>Vision and strategies for hybrid human–agent workflows</li><li>Change management and continuous upskilling</li><li>Culture to foster experimentation, feedback, and continuous learning</li><li>Accountability for outcomes, not just outputs</li></ul><p><strong>Evolving Operating Model</strong></p><p>Traditional IT or business operating models are no longer sufficient because agentic systems introduce distributed autonomy, persistent workflows, and outcome-oriented execution. Agentic AI adoption requires enterprises to evolve their operating model upgrading the way work is structured, decisions are made, and performance is measured.</p><p>1. Hybrid Human–Agent Collaboration Model</p><p>Agents become <strong>digital coworkers</strong>, not tools.</p><ul><li>Agents own execution within defined bounds, clear responsibilities, outcome goals, and performance metrics.</li><li>Human roles evolve from task execution to orchestration, supervision, and strategic judgment.</li><li>Clear contracts define what agents can do independently and when humans step in.</li></ul><p>2. Outcome-Oriented Performance Metrics</p><p>Performance evaluations shift from individual task throughput to agent management outcomes such as workflow success rates, error recovery, and human escalation handling.</p><p>New metrics may include:</p><ul><li>Percentage of workflows completed autonomously without errors</li><li>Time and cost savings from agent-executed processes</li><li>Accuracy, compliance, and policy adherence</li><li>Effectiveness and efficiency of human escalations</li></ul><p>Leaders will increasingly measure team performance by how well they manage and augment agents rather than how many tasks they personally complete.</p><p>3. AI Fluency Across the Organization</p><p>Agentic AI cannot be confined to a center of excellence, and even though not every employee will be a technical expert, all professionals will need AI fluency:</p><ul><li>Non-technical teams must understand the capabilities and limits of agents.</li><li>Technical teams must be able to troubleshoot agent reasoning and outcomes.</li><li>Managers must know how to interpret agent outcomes and risks.</li><li>Employees must recognize when agent autonomy is appropriate and when it is not.</li><li>Executives must identify opportunities to pair human judgment with agent automation.</li><li>Business leaders must interpret performance feedback and adjust workflows accordingly.</li></ul><p>This shared fluency provides a competitive advantage in speed, adaptability, and trust, enabling competitive agility and growth.</p><p>4. Organizational Design for Agentic Systems</p><p>Enterprises move toward a <strong>fluid, outcome-oriented structure </strong>where teams combine:</p><ul><li>Human domain expertise</li><li>Technical expertise in agents and data</li><li>Governance oversight embedded into processes</li></ul><p>Decision rights, accountability, and team boundaries shift accordingly often from hierarchical command chains to aligned, cross-functional squads that manage end-to-end, agentic workflows.</p><p>Adopting agentic AI requires organizations to rethink roles, redefine operating models, and build new capabilities that span technical expertise, governance responsibility, and human-AI collaboration. Enterprises that invest in these areas and not just in technology stand to unlock the most value while maintaining resilience, trust, and long-term strategic advantage.</p><blockquote>Agentic AI delivers the most value when humans define goals and constraints, and agents handle execution.</blockquote><h4>Conclusion: Discipline Enables Scale</h4><p>Agentic AI is a powerful evolution in enterprise automation but it is not a shortcut to transformation.</p><p>Adopting and scaling it responsibly requires:</p><ul><li>Clear business intent and boundary design</li><li>Strong architectural foundations</li><li>New skills and operating models</li><li>Ongoing human oversight and accountability</li></ul><p>Enterprises that treat agentic AI as just another generative feature will struggle. Those that recognize it as a new operational paradigm blending autonomy with control can unlock durable, defensible advantage.</p><blockquote>The future of enterprise AI is not fully autonomous. It is <strong>responsibly delegated</strong>.</blockquote><p><em>In the next part, we move from what agentic AI is to how it should be built — exploring architecture, governance, risks, and adoption paths for enterprise-scale deployment.</em></p><h4>References</h4><ol><li><strong>What Is Agentic AI, and How Will It Change Work? <br></strong><em>Harvard Business Review (Dec 2024) — Intro to agentic systems &amp; enterprise implications.</em><br><a href="https://hbr.org/2024/12/what-is-agentic-ai-and-how-will-it-change-work">https://hbr.org/2024/12/what-is-agentic-ai-and-how-will-it-change-work</a></li><li><strong>The next phase of AI is agentic, and it starts with data architecture.<br></strong><em>TechRadar — Architectural view on agentic systems</em> <a href="https://www.techradar.com/pro/the-next-phase-of-ai-is-agentic-and-it-starts-with-data-architecture">https://www.techradar.com/pro/the-next-phase-of-ai-is-agentic-and-it-starts-with-data-architecture</a></li><li><strong>Agentic AI: 4 reasons why it’s the next big thing in AI research.<br></strong><em>IBM Think — Technical &amp; research drivers</em> <a href="https://www.ibm.com/think/insights/agentic-ai">https://www.ibm.com/think/insights/agentic-ai</a></li><li><strong>Agentic AI: From Experimentation to Enterprise Scale.<br></strong><em>LinkedIn Pulse — Practitioner view on scale and governance</em> <a href="https://www.linkedin.com/pulse/agentic-ai-from-experimentation-enterprise-scale-arvind-gupta-5yrmf/">https://www.linkedin.com/pulse/agentic-ai-from-experimentation-enterprise-scale-arvind-gupta-5yrmf/</a></li><li><strong>What is agentic AI?<br></strong><em>IBM Think — Overview of agentic capabilities and business impact</em><br><a href="https://www.ibm.com/think/topics/agentic-ai">https://www.ibm.com/think/topics/agentic-ai</a></li><li><strong>Anthropic — Building Effective Agents (2024)</strong><br><em>Foundational paper on agent design patterns, tool use, planning, reflection, and failure modes.</em><br><a href="https://www.anthropic.com/research/building-effective-agents">https://www.anthropic.com/research/building-effective-agents</a></li><li><strong>OpenAI — Function Calling &amp; Tool Use Documentation (2024–2025)</strong><br><em>Canonical reference for structured tool invocation, agent control, and safe execution.</em><br><a href="https://platform.openai.com/docs/guides/function-calling">https://platform.openai.com/docs/guides/function-calling</a></li><li><strong>LangChain &amp; LangGraph Technical Documentation (2024–2025)</strong><br><em>Practical implementation of agent orchestration, stateful workflows, DAGs, and guardrails.</em><br><a href="https://docs.langchain.com/">https://docs.langchain.com</a><br><a href="https://langgraph.langchain.com/">https://langgraph.langchain.com</a></li><li><strong>Microsoft — AutoGen: Enabling Multi-Agent Conversations (2024)</strong><br><em>Real-world multi-agent collaboration patterns in enterprise workflows.</em><br><a href="https://microsoft.github.io/autogen/">https://microsoft.github.io/autogen/</a></li><li><strong>Gartner — Top Strategic Technology Trends: Agentic AI (2025)</strong><br><em>Gartner research on enterprise adoption trends</em><br><a href="https://www.gartner.com/en/research/methodologies/gartner-hype-cycle">https://www.gartner.com/en/research/methodologies/gartner-hype-cycle</a></li><li><strong>Harvard Business Review — What Is Agentic AI, and How Will It Change Work? (2024)</strong><br><em>Conceptual grounding for task vs. decision-support agents and human–AI collaboration models.</em><br> <a href="https://hbr.org/2024/12/what-is-agentic-ai-and-how-will-it-change-work">https://hbr.org/2024/12/what-is-agentic-ai-and-how-will-it-change-work</a></li><li><strong>OWASP Top 10 for Large Language Models (2024–2025)</strong><br><em>Threat taxonomy including prompt injection, tool misuse, and agent escalation risks.</em><br><a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">https://owasp.org/www-project-top-10-for-large-language-model-applications/</a></li><li><strong>MITRE ATLAS — Adversarial Threat Landscape for AI Systems</strong><br><em>Authoritative mapping of real-world AI attack vectors.</em><br><a href="https://atlas.mitre.org/">https://atlas.mitre.org/</a></li><li><strong>EU AI Act — Final Text (2024)</strong><br><em>Regulatory grounding for autonomy, accountability, and human oversight.</em><br><a href="https://data.europa.eu/eli/reg/2024/1689/oj">https://data.europa.eu/eli/reg/2024/1689/oj</a></li><li><strong>OECD AI Principles &amp; Updates (2024–2025)</strong><br><em>International AI policy and ethical principles context</em><br><a href="https://www.oecd.org/going-digital/ai/principles/">https://www.oecd.org/going-digital/ai/principles/</a></li><li><strong>UNESCO Guidance on AI Ethics (Updated 2024)</strong><br><em>High-level ethical standards for AI</em><br><a href="https://unesdoc.unesco.org/ark:/48223/pf0000380455">https://unesdoc.unesco.org/ark:/48223/pf0000380455</a></li><li><strong>Stanford AI Index Report 2025</strong><br><em>Evidence on hallucination rates, enterprise failure causes, and data quality impact.</em><br><a href="https://hai.stanford.edu/research/ai-index">https://hai.stanford.edu/research/ai-index</a></li><li><strong>Berkeley RISELab — RAG &amp; Data Quality Research (2024)</strong><br><em>Why poor retrieval breaks agent reasoning.</em><br><a href="https://rise.cs.berkeley.edu/publications/">https://rise.cs.berkeley.edu/publications/</a></li><li><strong>Hugging Face — Open LLM Leaderboard &amp; Evaluation Reports</strong><br><em>Model capability variance and benchmark limitations.</em><br><a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard">https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard</a></li><li><strong>Harvard Business Review — Scaling Generative AI for Value: Data Leader Agenda for 2025</strong><br><em>Why most AI initiatives fail without operating model change.</em><br><a href="https://d1.awsstatic.com/psc-digital/2024/gc-600/cdo-biz-value/CDO-Agenda-2025-ScalingGenerativeAIforValue.pdf">https://d1.awsstatic.com/psc-digital/2024/gc-600/cdo-biz-value/CDO-Agenda-2025-ScalingGenerativeAIforValue.pdf</a></li><li><strong>MIT Sloan Management Review — Gaining Real Business Benefits From GenAI: An MIT SMR Executive Guide</strong><br><em>How to use generative AI to deliver the greatest business value.</em><br><a href="https://sloanreview.mit.edu/article/gaining-real-business-benefits-from-genai-an-mit-smr-executive-guide/">https://sloanreview.mit.edu/article/gaining-real-business-benefits-from-genai-an-mit-smr-executive-guide/</a></li><li><strong>World Economic Forum — AI Governance &amp; Policy Outlook (2025)</strong><br><em>Global perspective on AI accountability and trust.</em><br><a href="https://www.weforum.org/reports">https://www.weforum.org/reports</a></li><li><strong>Infutrix — 9 Problems Agentic AI Is Solving for Enterprises</strong><br><em>Expanded enterprise use case list</em><br><a href="https://infutrix.com/blog/9-problems-agentic-ai-is-solving-for-enterprises">https://infutrix.com/blog/9-problems-agentic-ai-is-solving-for-enterprises</a></li><li><strong>FullStack Labs — 5 Real World Problems Agentic AI Is Solving Today</strong><br><em>Practical use cases and examples</em><br><a href="https://www.fullstack.com/labs/resources/blog/5-real-world-problems-agentic-ai-is-solving-today">https://www.fullstack.com/labs/resources/blog/5-real-world-problems-agentic-ai-is-solving-today</a></li><li><strong>MindDB — Types of AI Agents<br></strong><em>Unlocking the Power of the different types of AI Agents</em> <a href="https://mindsdb.com/blog/types-of-ai-agents">https://mindsdb.com/blog/types-of-ai-agents</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc0d458cb0de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Challenges and Risks for Enterprise Adoption of GenAI]]></title>
            <link>https://medium.com/@sunil.baidyanath/challenges-and-risks-for-enterprise-adoption-of-genai-90d87a2cc5c2?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/90d87a2cc5c2</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <category><![CDATA[innovation]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 00:45:29 GMT</pubDate>
            <atom:updated>2025-12-11T19:16:33.472Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Enterprises Still Struggle With Generative AI Adoption and How to Overcome It</h4><p><em>This is the fourth post of the series </em><a href="https://medium.com/@sunil.baidyanath/adopting-generative-ai-in-an-enterprise-706367bac03a"><em>Adopting Generative AI in an Enterprise</em></a><em>. The previous post in the series is </em><a href="https://medium.com/@sunil.baidyanath/building-a-simple-rag-chatbot-using-llamaindex-and-mesop-92cae73a6a9d"><em>Building a Simple RAG Chatbot using Llamaindex and Mesop.</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tkEcH_8_cCdJ33J5OhxL-w.jpeg" /><figcaption>Image generated by Copilot</figcaption></figure><p>Generative AI (GenAI) is no longer an experimental innovation — it has matured into a transformative capability reshaping product design, customer engagement, knowledge work, and enterprise automation. Yet despite rapid advancements, GenAI remains fundamentally different from traditional AI, requiring organizations to recalibrate their approaches to <strong>data governance, risk, operational models, talent, and architecture</strong>.</p><p>While traditional AI typically operates in tightly controlled environments using curated datasets and governance frameworks, GenAI relies heavily on <strong>externally trained foundation models</strong>, vast volumes of <strong>unregulated data</strong>, and probabilistic behaviors. This results in unpredictable variability in output quality, reliability, and safety. Enterprises must therefore adjust their expectations, systems, and governance models to safely and effectively adopt GenAI.</p><p>Growing regulatory pressure through the EU AI Act, U.S. Executive Orders on AI, NIST AI Risk Management Framework, and emerging global guidelines adds further complexity. These emerging global standards place greater responsibility on organizations deploying GenAI. As a result, successful GenAI adoption requires not only new technologies but also <strong>new enterprise operating disciplines</strong>.</p><h4>Key Concerns</h4><p>Enterprises continue to adopt Generative AI cautiously due to concerns around value, accuracy, transparency, cost, and compliance. The following factors remain the most critical barriers according to 2024–2025 studies from McKinsey, Gartner, and the Stanford AI Index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fkQYyXRD9wvK-LlLl9Fag.png" /><figcaption>Enterprise Concerns for GenAI Adoption</figcaption></figure><p><strong>Value Realization</strong><br>While GenAI POCs are abundant, <strong>enterprise-scale value is still elusive</strong>. Recent industry data shows:</p><ul><li>Only <strong>18% of GenAI use cases</strong> implemented in 2024 yielded measurable ROI (McKinsey Global AI Survey 2024).</li><li>Fewer than <strong>10% of enterprises</strong> have deployed GenAI beyond early pilots (Gartner AI Hype Cycle 2025).</li></ul><p>The lack of structured prioritization frameworks leads enterprises to invest in GenAI “because they should,” rather than to address validated business problems. Without alignment to core KPIs, most implementations fail to scale or deliver bottom-line impact.</p><p>Enterprises must prioritize use cases that meet three criteria:</p><ol><li><strong>Business-critical problem</strong> with measurable outcomes</li><li><strong>Operational feasibility</strong> within existing processes</li><li><strong>Data maturity</strong> to support reliable GenAI behavior</li></ol><p>GenAI succeeds when it starts from business outcomes and not model capability.</p><p><strong>Accuracy and Appropriateness</strong><br>Despite advancements in 2024–2025 such as fine-tuning, domain-specific RAG, and trust layers, LLMs still prioritize <strong>plausibility over factual accuracy.</strong></p><p>The risk of incorrect or inappropriate outputs increases as:</p><ul><li>Hallucination rates continue to range from <strong>3–15% depending on domain</strong> (Stanford AI Index 2025).</li><li>Safety alignment varies significantly across models and vendors.</li></ul><p>This inconsistency can undermine customer interactions, regulatory compliance, and internal decision-making specially in high-stakes environments such as finance, healthcare, legal, and security. Enterprises must therefore invest heavily in <strong>layered</strong> <strong>guardrails, human-in-the-loop validation, and rigorous outcome monitoring</strong>.</p><p><strong>Transparency and Trust</strong><br>LLMs remain black-box systems. Their training data, model internals, and update cadence are often opaque to and beyond the control of enterprise customers. Common concerns include:</p><ul><li>Limited model explainability</li><li>Difficulty attributing model decisions</li><li>Vendor-controlled versioning and updates</li><li>Unclear training data provenance</li></ul><p>As a result, enterprises struggle to adopt GenAI for <strong>regulated, high-stakes</strong> functions. The 2025 regulatory focus on “<strong>explainability obligations</strong>” further amplifies the need for transparency, provenance, and lineage in LLM systems to foster trust.</p><p><strong>Cost</strong><br>GenAI costs have become a top inhibitor of enterprise scale. According to IDC’s 2025 GenAI Spending Guide:</p><ul><li>Cloud GenAI spending grew <strong>582%</strong> from 2023 to 2024</li><li>Context window expansions drastically increase inference costs</li><li>Enterprises underestimate total GenAI spend by <strong>2–3x</strong></li></ul><p>Key cost drivers include:</p><ul><li>Token-based billing for inference and fine-tuning</li><li>Context-heavy RAG pipelines</li><li>Heavy RAG workloads</li><li>GPU scarcity and premium pricing</li><li>Specialized talent requirements</li></ul><p>Scaling GenAI without cost governance and optimization can erode the expected ROI of GenAI investments.</p><p><strong>Legal and Compliance</strong><br>GenAI regulations are evolving rapidly, making compliance highly dynamic. Major developments include:</p><ul><li>The <strong>EU AI Act</strong> imposes strict requirements for high-risk AI systems</li><li>The <strong>U.S. AI Executive Order (2024)</strong> mandates safety reporting and transparency</li><li>NIST and ISO are releasing new AI risk auditing standards</li><li>Several countries require <strong>data provenance disclosures</strong></li></ul><p>Because of this fluid regulatory landscape, enterprises often limit customer-facing GenAI and focus on <strong>internal productivity</strong> use cases until regulatory frameworks stabilize.</p><p><strong>Privacy and Security</strong><br>GenAI systems expand the attack surface in ways traditional AI did not. These include:</p><ul><li>Model poisoning</li><li>Prompt injection</li><li>Retrieval poisoning</li><li>Model hijacking</li><li>Data leakage through third-party APIs</li><li>Model inversion attacks</li><li>Cross-model safety inconsistencies</li><li>Unauthorized propagation of confidential information</li></ul><p>A 2025 IBM Security report states that <strong>AI-related cyber incidents increased 110%</strong> year-over-year. Modern security architectures must ensure data flow control, encryption, auditing, and isolation for safe GenAI integration.</p><p><strong>Enterprise Control, Ethics, and Governance</strong><br>The greatest organizational tension stems from the lack of <strong>control and guarantees</strong> over model behavior. Since LLMs are externally developed and as they evolve rapidly, often without transparent change logs, enterprises struggle with:</p><ul><li>Maintaining ethical consistency</li><li>Ensuring predictable outputs</li><li>Guaranteeing model compliance</li><li>Rapid remediation during incidents</li></ul><p>Therefore, an effective GenAI governance must extend beyond traditional data governance, covering <strong>model behavior, configuration, operations, dependency management</strong>, and human oversight.</p><h4>Barriers</h4><p>Even once concerns are addressed, several <strong>structural barriers</strong> slow or prevent GenAI adoption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RfDMWC1z4-4ANCDqUL5bYw.png" /><figcaption>Enterprise Barriers for Adopting GenAI</figcaption></figure><p><strong>Lack of Clear and Short-term ROI</strong><br>GenAI requires upfront investment in data, infrastructure, and skills. Without a clear ROI framework or use-case prioritization, initiatives lose momentum. Enterprises must adopt value frameworks such as:</p><ul><li><strong>GenAI Opportunity Scor</strong>e (Feasibility × Impact × Confidence)</li><li><strong>Flow-based productivity measurement</strong></li><li><strong>Business-aligned success metrics</strong> (time saved, CSAT, revenue lift)</li></ul><p><strong>Organizational Inertia and Resistance</strong><br>Gartner’s 2025 research indicates that <strong>change management, not technology</strong>, is the top failure point for GenAI programs. Resistance stems from:</p><ul><li>Fear of job displacement</li><li>Misunderstanding of capabilities</li><li>Overly bureaucratic decision pathways</li><li>Fatigue from frequent technology waves</li></ul><p>Executive sponsorship, clear transformation narratives, and ongoing workforce upskilling are essential for the success of any enterprise initiative.</p><p><strong>Misaligned Incentives</strong><br>Traditional KPI structures reward <strong>risk avoidance, process adherence, and stability</strong>, not innovation. GenAI requires:</p><ul><li>Faster experimentation cycles</li><li>Cross-functional collaboration</li><li>Tolerance for iteration and controlled risk-taking</li></ul><p>Organizations must reshape incentives to reward teams for innovation, experimentation, and responsible adoption.</p><p><strong>Fragmented and Slow Decision Making</strong><br>Large enterprises often struggle with distributed ownership and siloed decision-making, resulting in:</p><ul><li>Duplicated efforts</li><li>Misaligned priorities</li><li>Conflicting policies</li><li>Redundant tooling</li></ul><p>A <strong>federated-but-guided operating model</strong> or central governance with distributed innovation is emerging as the 2025 best practice.</p><p><strong>Siloed Enterprise Systems</strong><br>GenAI deliver maximum value when paired with high-quality enterprise data and capabilities. Many organizations still grapple with:</p><ul><li>Disconnected business systems</li><li>Non-standardized data semantics</li><li>Legacy platforms without APIs</li><li>Weak knowledge management</li><li>Fragmented data pipelines</li></ul><p>Without a unified data fabric and capabilities, GenAI effectiveness is significantly reduced. This can result in poor value propositions, potentially harming business values, revenue streams, and customer trust.</p><p><strong>Poor Quality Data</strong><br>Data quality is now directly tied to the quality and performance of GenAI systems. Poor quality data produces unreliable, biased, or harmful outputs, compromising customer experience, business decision-making, and operational workflows</p><p>The Stanford AI Index 2025 states that <strong>41% of LLM failures</strong> in enterprises are caused by upstream data issues and not model issues.</p><p>Enterprises must invest in data governance, canonical definitions, semantic layers, and automated quality monitoring.</p><p><strong>Lack of Enterprise Capabilities and Skills</strong><br>GenAI introduces new engineering disciplines, including:</p><ul><li>RAG architecture</li><li>Agentic systems</li><li>Prompt &amp; context engineering</li><li>Trust &amp; safety engineering</li><li>Model evaluation &amp; alignment</li><li>AI Ops and observability</li></ul><p>Most enterprises face a severe talent shortage. Upskilling programs, internal AI academies, and strategic hiring are now essential.</p><h4>Risks</h4><p>As GenAI adoption accelerates, risk exposure grows. 2024–2025 incidents show material increases in risk associated with GenAI usage, posing challenges for GenAI adoption that are legal, safe, secure, and compliant. These risks include data loss, unintentional sharing of confidential information, and inappropriate use of information.</p><p>Common risks associated with GenAI adoption can be categorized into the following major groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aTN5EV9BN9lHpe5exBBJaQ.png" /></figure><ul><li><strong>Misuse</strong>: This occurs when GenAI capabilities are used unethically and illegally for harmful purposes such as scams and misinformation campaigns.</li><li><strong>Misapply</strong>: With GenAI prioritizing plausibility over accuracy, it often generates inaccurate outputs, which can lead users to misapply the output. With GenAI becoming multimodal, the risk of misapplication rises further.</li><li><strong>Misrepresentation</strong>: These risks are caused when GenAI output created by a third party is purposefully used and disseminated despite questions about its credibility and authenticity.</li><li><strong>Misadventure</strong>: This happens when a GenAI output is accidentally consumed and shared by users who are not aware of its inauthenticity.</li></ul><p>GenAI adoption expands cybersecurity risks that include:</p><ul><li><strong>Prompt injection</strong> allows malicious actors to manipulate the GenAI system’s output by crafting input text, causing the LLM to follow the attacker’s instructions.</li><li><strong>Retrieval poisoning</strong>, where the data inside a RAG system is intentionally manipulated so the LLM retrieves and trusts incorrect, misleading, or malicious information.</li><li><strong>Model hijacking</strong>, where an attacker covertly inserts behaviors, instructions, or payloads through training data, weights, prompts, or retrieval pathways, causing the LLM to produce attacker-controlled outputs.</li><li><strong>Sensitive data leakage</strong> in the training data causes significant privacy risks</li><li><strong>Dataset poisoning</strong>, where attackers introduce malicious data into the training set to manipulate the output of a GenAI system.</li><li><strong>Vulnerable AI-generated code</strong> leading to potential security breaches and a buggy system</li><li><strong>Supply chain risk</strong> from third-party models causes enterprises to unknowingly inherit embedded vulnerabilities, malicious backdoors, compromised training data, or unsafe model behaviors.</li><li>Difficulty enforcing uniform safety policies across models and vendors</li></ul><p>Lack of transparency in GenAI makes it challenging to identify and mitigate risks, leading to mistrust among users. Regulatory risks are also rising as governments aggressively expand AI oversight.</p><h4>Challenges for Successful Adoption</h4><p>Organization planning to adopt GenAI often struggle with this central question: <strong>Where should we start, and how do we measure success?</strong></p><p>The core challenges include aligning GenAI to business value, preparing the organization culturally, and building a strong technical foundation.</p><p>These five pillars consistently determine successful GenAI adoption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jov6UJ7KR5cuKsg9QLnHUw.png" /><figcaption>Challenges for Successful AI Adoption</figcaption></figure><p><strong>Business Focus<br></strong>For successful GenAI adoption, it is crucial to align the initiative with specific business goals and establish measurable outcomes. A comprehensive plan detailing the objectives, rationale, and implementation strategy for the GenAI adoption journey is essential.</p><p>Successful GenAI adoption requires:</p><ul><li>Clear business objectives</li><li>Measurable outcomes</li><li>Cross-functional alignment</li><li>Stakeholder validation</li><li>Use-case evaluation criteria</li></ul><p>Initiatives must start with tangible, outcome-driven goals, not technology experimentation.</p><p><strong>Executive Support<br></strong>Executive sponsorship remains the strongest predictor of success. Executive support is crucial for preparing the infrastructure, readying the organization, acquiring the necessary talent for implementation, and ensuring a successful transition.</p><p>Effective leaders must:</p><ul><li>Champion cultural adoption</li><li>Allocate funding</li><li>Remove organizational roadblocks</li><li>Understand GenAI’s complexity and risks</li><li>Communicate the transformation vision</li></ul><p>Organizations with strong executive backing are <strong>3× more likely</strong> to scale GenAI (McKinsey 2024).</p><p><strong>GenAI Foundation<br></strong>Successful GenAI adoption necessitates a robust foundation comprising capabilities that enable and accelerate implementation. Enterprises need a robust foundation comprising:</p><ul><li>High-quality, governed data</li><li>Scalable vector databases &amp; RAG pipelines</li><li>Secure &amp; cost-optimized cloud infrastructure</li><li>Knowledge graphs and semantic layers</li><li>Orchestration frameworks for multi-agent workflows</li><li>AI governance, model lifecycle, safety, and risk controls</li><li>Skilled engineering and business teams</li></ul><p>Without this foundation, GenAI becomes unreliable, unscalable, and unsafe.</p><p><strong>Accountability and Ethics<br></strong>GenAI lacks causal reasoning and cannot reliably make ethical or context-sensitive judgments. Enterprises must implement an effective AI operating model to remain vigilant against the potential risks of unethical and irresponsible usage. Such an operating model should include:</p><ul><li>Mandatory human oversight</li><li>Escalation mechanisms for harmful outputs</li><li>Enforceable ethical guidelines</li><li>Transparent and auditable workflows</li><li>Lifecycle tracking for model decisions and data usage</li></ul><p>GenAI should augment and not replace judgment, context, or accountability.</p><p><strong>GenAI Governance<br></strong>With the rapidly evolving GenAI landscape, enterprises need a flexible governance mechanism to protect their employees, customers, and brand value. Enterprises must set up a comprehensive governance system to foster trust-based engagement with their users.</p><p>Clear policies and guidelines must be established to govern the acquisition and usage of data and models within business domains. Modern GenAI governance includes policies, guidelines, and controls for:</p><ul><li>Model lifecycle management</li><li>Vendor model evaluation</li><li>Data usage permissions and lineage</li><li>Regulatory compliance</li><li>Safety and red-teaming</li><li>Monitoring and observability</li><li>Change control for model updates</li></ul><h4>Conclusion</h4><p>In 2025, GenAI has evolved from basic chat interfaces to sophisticated agentic systems capable of reasoning, planning, and automating end-to-end workflows. But this increased capability brings higher stakes — and greater responsibility.</p><p>Enterprises cannot adopt GenAI with the same mindset or frameworks used for traditional AI. Success requires:</p><ul><li>A strong business-aligned strategy</li><li>Robust data, security, and governance foundations</li><li>Clear ROI measurement</li><li>Enterprise-wide education and cultural change</li><li>Ethical safeguards and human oversight</li><li>A modern AI operating model</li><li>Continuous alignment with regulatory expectations</li></ul><blockquote>Above all, <strong>the human element remains central</strong>. GenAI augments decision-making but does not replace experience, judgment, or accountability. Organizations that effectively integrate GenAI with human expertise will gain competitive advantage through higher productivity, improved customer outcomes, and accelerated innovation.</blockquote><p>The future belongs to enterprises that embrace GenAI with discipline, governance, and purpose — turning potential into measurable business value.</p><h4>References</h4><ol><li>QuantumBlack, AI by McKinsey, The State of AI in 2024: GenAI Adoption Spikes and Starts to Generate Value. <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-2024">https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-2024</a></li><li>Gartner AI Hype Cycle 2025. <a href="https://www.gartner.com/en/articles/hype-cycle-for-artificial-intelligence">https://www.gartner.com/en/articles/hype-cycle-for-artificial-intelligence</a></li><li>IDC Worldwide AI and Generative AI Spending Guide. <a href="https://info.idc.com/DA-Product-Overview-AI-and-GenAI-Spending-Guide.html">https://info.idc.com/DA-Product-Overview-AI-and-GenAI-Spending-Guide.html</a></li><li>Stanford Artificial Intelligence Index Report 2025, Stanford Institute for Human-Centered AI (HAI), 2025. <a href="https://hai.stanford.edu/assets/files/hai_ai-index-report-2025_chapter4_final.pdf?utm_source=chatgpt.com">https://hai.stanford.edu/assets/files/hai_ai-index-report-2025_chapter4_final.pdf</a></li><li>EU AI Act — Final Text (2024), European Parliament &amp; Council of the European Union. <a href="https://data.europa.eu/eli/reg/2024/1689/oj">https://data.europa.eu/eli/reg/2024/1689/oj</a></li><li>U.S. Executive Order on Safe, Secure, and Trustworthy AI (Oct 2024), The White House, 2024. <a href="https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/">https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/</a></li><li>NIST AI Risk Management Framework (RMF) 1.0, National Institute of Standards and Technology, 2023–2024 updates. <a href="https://www.nist.gov/itl/ai-risk-management-framework">https://www.nist.gov/itl/ai-risk-management-framework</a></li><li>ISO/IEC 42001:2023 — AI Management System Standard, International Organization for Standardization, 2023. <em>(Subscription / purchase required)</em> <a href="https://www.iso.org/standard/42001">https://www.iso.org/standard/42001</a></li><li>IBM Security X-Force Threat Intelligence Index 2025, IBM Security, 2025. <a href="https://www.ibm.com/thought-leadership/institute-business-value/en-us/report/2025-threat-intelligence-index?utm_source=chatgpt.com">https://www.ibm.com/thought-leadership/institute-business-value/en-us/report/2025-threat-intelligence-index</a></li><li>OWASP Top 10 for Large Language Models (LLM) — 2024–2025 Edition, OWASP Foundation, 2024. <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">https://owasp.org/www-project-top-10-for-large-language-model-applications/</a></li><li>MITRE ATLAS (Adversarial Threat Landscape for AI Systems), MITRE Corporation, continuously updated (referenced 2024/2025). <a href="https://atlas.mitre.org/">https://atlas.mitre.org/</a></li><li>Gartner CIO and Technology Executive Survey 2025, Gartner Research, 2025. <em>(Proprietary / subscription required)</em> <a href="https://www.gartner.com/en/insights/ai-journey">https://www.gartner.com/en/insights/ai-journey</a></li><li>Deloitte State of AI in the Enterprise — 6th Edition (2024), Deloitte Insights, 2024. <a href="https://www.deloitte.com/ce/en/services/consulting/research/state-of-generative-ai-in-enterprise.html">https://www.deloitte.com/ce/en/services/consulting/research/state-of-generative-ai-in-enterprise.html</a></li><li>PwC AI Predictions Report 2025, PricewaterhouseCoopers, 2025. <a href="https://www.pwc.nl/en/services/artificial-intelligence/competing-in-the-age-of-ai.html">https://www.pwc.nl/en/services/artificial-intelligence/competing-in-the-age-of-ai.html</a><a href="https://www.pwc.com/gx/en/issues/data-and-analytics/artificial-intelligence-study.html">l</a></li><li>OpenAI Research &amp; System Cards (GPT-4 Turbo, o1/o3, GPT-5 preview), OpenAI, 2024–2025. <a href="https://openai.com/research">https://openai.com/research</a></li><li>NVIDIA AI Index + Hopper/Blackwell Architecture Whitepapers, NVIDIA, 2024–2025. <a href="https://docs.nvidia.com/nvidia-secure-ai-with-blackwell-and-hopper-gpus-whitepaper.pdf">https://docs.nvidia.com/nvidia-secure-ai-with-blackwell-and-hopper-gpus-whitepaper.pdf</a></li><li>LangChain &amp; LangGraph Technical Reports, LangChain, 2024–2025. <a href="https://www.langchain.com/">https://www.langchain.com/</a></li><li>Google DeepMind — Safety Alignment Research 2024–2025, Google DeepMind Publications. <a href="https://www.deepmind.com/research">https://www.deepmind.com/research</a></li><li>Anthropic — Claude Safety Reports (2024–2025), Anthropic Research. <a href="https://www.anthropic.com/research">https://www.anthropic.com/research</a></li><li>HuggingFace Open LLM Leaderboard &amp; Evaluation Reports (2024–2025), HuggingFace. <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard">https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard</a></li><li>Accenture Technology Vision 2024–2025 (GenAI Special Edition), Accenture Research, 2024–2025. <a href="https://www.accenture.com/us-en/insights/technology/technology-trends-2025">https://www.accenture.com/us-en/insights/technology/technology-trends-2025</a></li><li>World Economic Forum (WEF) — AI Governance &amp; Policy Outlook 2024–2025, WEF, 2025. <a href="https://www.weforum.org/stories/2025/10/measurement-momentum-agile-governance-ai/">https://www.weforum.org/stories/2025/10/measurement-momentum-agile-governance-ai/</a></li><li>OECD AI Principles &amp; Regulatory Updates (2024–2025), Organisation for Economic Co-operation and Development. <a href="https://www.oecd.org/going-digital/ai/principles/">https://www.oecd.org/going-digital/ai/principles/</a></li><li>UNESCO Guidance on AI Ethics (Updated 2024), UNESCO. <a href="https://unesdoc.unesco.org/ark:/48223/pf0000380455">https://unesdoc.unesco.org/ark:/48223/pf0000380455</a></li><li>LinkedIn Global Skills Report 2025 — AI &amp; Data Skill Gaps, LinkedIn, 2025. <a href="https://business.linkedin.com/talent-solutions/global-talent-trends">https://business.linkedin.com/talent-solutions/global-talent-trends</a></li><li>World Economic Forum — Future of Jobs Report 2025, WEF, 2025. <a href="https://www.weforum.org/publications/the-future-of-jobs-report-2025/digest/">https://www.weforum.org/publications/the-future-of-jobs-report-2025/digest/</a></li><li>MIT Sloan Review — Five Trends in AI and Data Science for 2025. <a href="https://sloanreview.mit.edu/article/five-trends-in-ai-and-data-science-for-2025/">https://sloanreview.mit.edu/article/five-trends-in-ai-and-data-science-for-2025/</a></li><li>Why Most AI Transformations Fail Before They Start. <a href="https://www.russellreynolds.com/en/insights/articles/why-most-ai-transformations-fail-before-they-start">https://www.russellreynolds.com/en/insights/articles/why-most-ai-transformations-fail-before-they-start</a></li><li><a href="https://datasciencedojo.com/blog/challenges-in-rag-based-llm-applications/">https://datasciencedojo.com/blog/challenges-in-rag-based-llm-applications/</a></li><li><a href="https://medium.com/@tubelwj/considerations-for-data-processing-in-rag-applications-9d06b895574e">Considerations for Data Processing in RAG Applications | by Gen. David L. | Medium</a></li><li><a href="https://hbr.org/2024/05/4-types-of-gen-ai-risk-and-how-to-mitigate-them">https://hbr.org/2024/05/4-types-of-gen-ai-risk-and-how-to-mitigate-them</a></li><li><a href="https://www.lexisnexis.com/community/insights/professional/b/industry-insights/posts/challenges-generative-ai-?srsltid=AfmBOoqDdXZ_jU9iWjn63ity5r3RkC5xpvSM88FFZo8N9CvjIAudVU3S">https://www.lexisnexis.com/community/insights/professional/b/industry-insights/posts/challenges-generative-ai-?srsltid=AfmBOoqDdXZ_jU9iWjn63ity5r3RkC5xpvSM88FFZo8N9CvjIAudVU3S</a></li><li><a href="https://www.datanami.com/2024/01/22/top-10-challenges-to-genai-success/">https://www.datanami.com/2024/01/22/top-10-challenges-to-genai-success/</a></li><li><a href="https://www.cio.com/article/3478772/6-hard-truths-of-generative-ai-in-the-enterprise.html#:~:text=Managing%20costs%20is%20a%20huge%20issue&amp;text=Organizations%20also%20have%20to%20factor,to%20train%20models%2C%20Higginson%20says">https://www.cio.com/article/3478772/6-hard-truths-of-generative-ai-in-the-enterprise.html#:~:text=Managing%20costs%20is%20a%20huge%20issue&amp;text=Organizations%20also%20have%20to%20factor,to%20train%20models%2C%20Higginson%20says</a>.</li><li><a href="https://gradientflow.com/genai-enterprise-adoption/">https://gradientflow.com/genai-enterprise-adoption/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90d87a2cc5c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adopting Generative AI in an Enterprise]]></title>
            <link>https://medium.com/@sunil.baidyanath/adopting-generative-ai-in-an-enterprise-706367bac03a?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/706367bac03a</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 01:42:32 GMT</pubDate>
            <atom:updated>2025-12-11T19:05:06.569Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GwkoJ2yH_hHsCTxVpZgZlg.jpeg" /><figcaption>Image generated by Copilot</figcaption></figure><p>The history of industrial revolutions demonstrates how technological innovations, from the steam engine to personal computers, have disrupted human economies by transforming business practices. The latest technology poised to induce such a significant shift in the business world is generative artificial intelligence (GenAI).</p><h4>Context</h4><p>Recognizing the immense potential of GenAI, many organizations rushed to adopt it for creating new revenue streams, reducing operational costs, and engaging customers more effectively.</p><p>However, these early adopters soon realized that GenAI democratizes AI capability for their competitors as well, who will inevitably catch up, eroding any competitive advantage they accrued earlier.</p><p>Further, GenAI is different from traditional AI in the way that it is far less controlled and regulated, and a hasty implementation can expose an organization to serious risks, causing tangible loss to business, reputation, and customer trust.</p><p>Therefore, enterprises must identify and assess business opportunities for GenAI adoption while also identifying major risks and challenges with necessary controls and measures for successful adoption for a tangible competitive advantage.</p><h4>Motivation</h4><p>Organizations generate and collect vast amounts of data, yet only a small fraction is utilized to support customer and business functions.</p><p>This is primarily due to data being trapped in siloed systems. The lack of visibility and access to insights within this extensive corporate data significantly hampers an organization’s ability to build and sustain a competitive advantage in the industry.</p><p>Building a data-driven AI-enabled application has the following advantages:</p><ul><li>Enterprises can dramatically <strong>improve the value proposition for their proprietary datasets,</strong> thereby attaining considerable competitive advantage.</li><li><strong>Promote data equity in organizations,</strong> bridging the gap in data accessibility and democratizing data-driven decision-making.</li><li><strong>Unlock business patterns and insights</strong> lost in enterprise data silos to understand the business, customers, challenges, and opportunities better.</li><li>Empower businesses in <strong>leveraging their data for enhanced customer engagement and experience</strong> by enabling them to offer more targeted and personalized products and services.</li><li>Ensure all <strong>data-driven decisions and outputs are trustworthy, accurate, and inclusive </strong>of the relevant data available to an enterprise.</li></ul><p>This series examines generative AI (GenAI) from an enterprise perspective, offering a primer for organizations seeking to adopt GenAI to achieve a sustainable competitive advantage.</p><h4>Posts in this Series</h4><ol><li><a href="https://medium.com/@sunil.baidyanath/large-language-models-in-an-enterprise-c94bbecb7710">Large Language Models in an Enterprise</a></li><li><a href="https://medium.com/@sunil.baidyanath/an-overview-of-a-rag-application-d8c144e4ef64">An Overview of a RAG Application</a></li><li><a href="https://medium.com/@sunil.baidyanath/building-a-simple-rag-chatbot-using-llamaindex-and-mesop-92cae73a6a9d">Building a Simple RAG Chatbot using Llamaindex and Mesop</a></li><li><a href="https://medium.com/@sunil.baidyanath/challenges-and-risks-for-enterprise-adoption-of-genai-90d87a2cc5c2?postPublishedType=repub">Challenges and Risks for Enterprise Adoption of GenAI</a></li><li>Enterprise Framework for GenAI Application (Upcoming)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=706367bac03a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Simple RAG Chatbot using Llamaindex and Mesop]]></title>
            <link>https://medium.com/@sunil.baidyanath/building-a-simple-rag-chatbot-using-llamaindex-and-mesop-92cae73a6a9d?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/92cae73a6a9d</guid>
            <category><![CDATA[llamaindex-rag]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[mesop]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[azureopenai]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 18:26:04 GMT</pubDate>
            <atom:updated>2025-12-11T19:13:33.260Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is the third post of the series </em><a href="https://medium.com/@sunil.baidyanath/adopting-generative-ai-in-an-enterprise-706367bac03a"><em>Adopting Generative AI in an Enterprise</em></a><em>. The previous post in the series is </em><a href="https://medium.com/@sunil.baidyanath/an-overview-of-a-rag-application-d8c144e4ef64"><em>An Overview of a RAG application</em></a><em>. The next post in the series is </em><a href="https://medium.com/@sunil.baidyanath/challenges-and-risks-for-enterprise-adoption-of-genai-90d87a2cc5c2?postPublishedType=repub"><em>Challenges and Risks for Enterprise Adoption of GenAI</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_bh7X8amxmUsuUaHzI1qg.jpeg" /><figcaption>Image generated by Copilot</figcaption></figure><p>There has been an explosion of documents-based chatbots using RAG approach in last one year.In the previous parts, we discussed about RAG and basic architecture of a RAG application. In this part, we will build a simple RAG chatbot using Llamaindex and Mesop.</p><p><a href="https://www.llamaindex.ai/">Llamaindex</a> is an advanced framework for building LLM applications that require efficient data lookup. With it’s powerful index and search capabilities and seamless integration with LLMs, it’s ideal for building knowledge-enriched applications such as RAG systems.</p><p>Install required llamaindex libraries using following commands.</p><pre>pip install llama-index<br>pip install llama-index-embeddings-azure-openai<br>pip install llama-index-llms-azure-openai</pre><p><a href="https://google.github.io/mesop/">Mesop</a> is a python-based UI framework from Google allowing for rapid development of LLM-based web apps. With declarative UI, a wide array of useful UI components, and seamless integration with JavaScript libraries, and LLM-ready UI capabilities, it’s preferred choice for building LLM applications.</p><p>Install this package using the command below.</p><pre>pip install mesop</pre><h4>Create Model Deployments</h4><p>For our chatbot, we will use <strong>gpt-3.5-turbo</strong> LLM and <strong>text-embedding-ada-002</strong> embedding model both of which are available with <strong>Azure OpenAI</strong>. Follow the link below to create deployments of these models for using them in our chatbot app.</p><p><a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal">How-to: Create and deploy an Azure OpenAI Service resource - Azure OpenAI</a></p><p>Once model deployments are created successfully for both llm and embedding model, please note down the azure deployment endpoint, api keys, and deployment name for the models.</p><p>It’s a good practice to use sensitive info such as API keys securely and so let’s add them to environment variables. We will pick up while initializing these models later.</p><h4>Configure Application Settings</h4><p>Before building knowledge base and RAG engine for our chatbot, we need to setup the context by configuring different RAG components such as chunking, embedding, and LLM that will be used in the application.</p><p><strong>Specify application settings</strong></p><p>It’s a good practice to organize all the settings for different application components in a configuration file. You may choose to create one or more configuration files depending on the number of components and complexity of their configurations. For this sample application, let’s create a JSON file <strong>config.json</strong> with following content.</p><pre>{<br>  &quot;document_location&quot;: &quot;&lt;local_document_path&gt;&quot;,<br>  &quot;index_location&quot;: &quot;&lt;local_index_path&gt;&quot;,<br>  &quot;embedding&quot;: {<br>    &quot;source&quot;: &quot;AzureOpenAI&quot;,<br>    &quot;model&quot;: &quot;text-embedding-ada-002&quot;,<br>    &quot;deployment_name&quot;: &quot;&lt;model_deployment_name&gt;&quot;,<br>    &quot;azure_endpoint&quot;: &quot;&lt;azure_deployment_endpoint&gt;&quot;,<br>    &quot;api_version&quot;: &quot;2023–07–01-preview&quot;,<br>    &quot;chunk_size&quot;: 1<br>  },<br>  &quot;llm&quot;: {<br>    &quot;source&quot;: &quot;AzureOpenAI&quot;,<br>    &quot;model&quot;: &quot;gpt-3.5-turbo&quot;,<br>    &quot;deployment_name&quot;: &quot;&lt;model_deployment_name&gt;&quot;,<br>    &quot;azure_endpoint&quot;: &quot;&lt;azure_deployment_endpoint&gt;&quot;,<br>    &quot;api_version&quot;: &quot;2023–12–01-preview&quot;,<br>    &quot;temperature&quot;: &quot;0.0&quot;,<br>    &quot;max_tokens&quot;: 256<br>  },<br>  &quot;chunking&quot;: {<br>    &quot;chunk_overlap&quot;: 20,<br>    &quot;chunk_size&quot;: 1024<br>  },<br>  &quot;prompt_helper&quot;: {<br>    &quot;context_window&quot;: 4096,<br>    &quot;num_output&quot;: 256<br>  }</pre><p>Please don’t forget to put correct local folder path for documents and index locations. We have added embedding model for information retrieval and the LLM for response generation.</p><blockquote>Please note that we need to specify model deployment names in addition to model names along with the azure deployment endpoint to correctly configure the embedding model and the LLM.</blockquote><p>You should have got these values if you successfully created model deployments as described earlier.</p><p><strong>Load configuration</strong></p><p>Create a python module <strong>rag.py</strong> and add following code to load configuration from settings file and environment variables.</p><pre># Load configurations<br>with open(os.path.join(basedir, ‘settings.json’)) as fp:<br>  config = json.load(fp)<br><br># Load api keys<br>load_dotenv()<br>llm_api_key = os.getenv(&quot;LLM_API_KEY&quot;)<br>embedding_api_key = os.getenv(&quot;EMBEDDING_MODEL_API_KEY&quot;)</pre><p><strong>Configure LLM</strong></p><p>With the configuration file ready, let’s initialize our chatbot components starting with the LLM. Add following code to create the LLM component for the chatbot.</p><pre>llm=AzureOpenAI(<br>  deployment_name=config[&quot;llm&quot;][&quot;deployment_name&quot;],<br>  model=config[&quot;llm&quot;][&quot;model&quot;],<br>  api_key=llm_api_key,<br>  azure_endpoint=config[&quot;llm&quot;][&quot;azure_endpoint&quot;],<br>  api_version=config[&quot;llm&quot;][&quot;api_version&quot;],<br>  temperature=config[&quot;llm&quot;][&quot;temperature&quot;],<br>  max_tokens=config[&quot;llm&quot;][&quot;max_tokens&quot;]<br>)</pre><p>The <strong>temperate</strong> attribute defines the randomness of LLM output and can have its value between 0.0 and 2.0. While with the lower value (&lt;1.0), the LLM output is more deterministic and repetitive, with higher value (&gt;1.0), the output is more random and creative. Most llms have their default value set to 1.0 balancing their output between being predictive and creative.</p><p>The <strong>max_tokens</strong> attribute specifies the maximum number of tokens that LLMs can generate as their output or in other words, the length of their output text. Though the number of tokens a model can output depends on its architecture, max_tokens can be used to further configure the model behavior for influencing the response.</p><p>Both these parameters can be used to control the quality of the LLM output.</p><blockquote>While configuring max_tokens, keep in mind that prompt tokens + max_tokens ≤ model capacity.</blockquote><p><strong>Configure embedding model</strong></p><p>Add the code given below to to create the embedding model.</p><pre>embed_model = AzureOpenAIEmbedding(<br>  azure_deployment=config[&quot;embedding&quot;][&quot;deployment_name&quot;],<br>  model=config[&quot;embedding&quot;][&quot;model&quot;],<br>  api_key=embedding_api_key,<br>  azure_endpoint=config[&quot;embedding&quot;][&quot;azure_endpoint&quot;],<br>  api_version=config[&quot;embedding&quot;][&quot;api_version&quot;],<br>  chunk_size=config[&quot;embedding&quot;][&quot;chunk_size&quot;]<br>)</pre><p>The optional parameter <strong>chunk_size</strong> specifies the chunk size the model uses to generate embeddings. If the chunk size is too small it might not have enough to provide any context while too large a chunk size might overwhelm the model with irrelevant data affecting the information retrieval.</p><p>The optimal chunk size vary depending on the embedding model being used and for the model text-embedding-ada-001, it’s 256 or 512 tokens.</p><p><strong>Configure text splitter</strong></p><p>We will use the LlamaIndex Sentence Splitter for parsing documents into manageable chunks. It can be configured with chunk_size and chunk_overlap parameters.</p><p>The below code configures the sentence splitter to be used for chunking.</p><pre>text_splitter=SentenceSplitter(<br>  chunk_size=config[&quot;chunking&quot;][&quot;chunk_size&quot;], <br>  chunk_overlap=config[&quot;chunking&quot;][&quot;chunk_size&quot;]<br>)</pre><p><strong>Initialize the Settings for RAG application</strong></p><p>Llamaindex provides a Settings class that can be used to hold the various components required by the RAG application such as llm, embedding model, and text splitter.</p><p>Add following code block to initialize the Settings for our chatbot.</p><pre>Settings.llm = llm<br>Settings.embed_model = embed_model<br>Settings.text_splitter = text_splitter<br>Settings.context_window = config[&quot;prompt_helper&quot;][&quot;context_window&quot;]<br>Settings.num_output = config[&quot;prompt_helper&quot;][&quot;num_output&quot;]</pre><p>The parameters <strong>context_window</strong> and <strong>num_output</strong> are prompt helper arguments to ensure that the input prompts to the LLM have enough space for generating a specific number of tokens.</p><h4>Building searchable knowledge base</h4><p>RAG depends on a knowledge base to provide precise and relevant answers to user queries. This knowledge base is supported by semantic search to enable efficient information retrieval. To facilitate semantic search, the source documents are processed into an indexed vector and stored in a vector store.</p><p>LlamaIndex provides wide array of components to vectorize text documents and for storing them in vector stores such as ChromaDB or FAISS. We can use appropriate LlamaIndex components to then query the vectorized docs in the store to extract relevant text chunk for a given user query.</p><p>The sections below describe steps to create a vector index using LlamaIndex.</p><p><strong>Load documents</strong></p><p>Llamaindex provides an extensive library of readers for loading data from variety of sources however, the simplest way to load files from local filesystem is to use the class SimpleDirectoryReader.</p><pre>documents = SimpleDirectoryReader(input_files=[soure_location]).load_data()</pre><p>Here, we have used the input_files parameter to restrict the data loading to only the specific list of files passed. You may follow this <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/simpledirectoryreader/">link</a> to further customize the behavior of this reader component per your specific data loading requirements.</p><p>You can learn more about the various other document readers available in llamaindex <a href="https://docs.llamaindex.ai/en/stable/api_reference/readers/">here</a>.</p><p><strong>Convert documents into vector embeddings</strong></p><p>The VectorStoreIndex class from Llamaindex provides an easy way to load a list of documents and convert them into indexed list of embeddings.</p><p>The following code creates indexed embeddings from a give list of documents.</p><pre>index = VectorStoreIndex.from_documents(documents)</pre><p>The from_documents method splits the loaded documents into chunks and convert them into appropriate embeddings using the Settings components configured above.</p><p><strong>Store embeddings</strong></p><p>By default, the VectorStoreIndex component stores all its data in the memory. However, we may prefer to save the indexes in external store for future usage effectively decoupling index creation from RAG call. Llamaindex supports number of vector stores, and you can configure StorageContext class to specify your choice of vector stores and then use the StorageContext components to store embeddings to a physical store.</p><p>Use the following code to store the index on a local filesystem.</p><pre>index_path = join(self.index_store_location, index_name)<br>index.storage_context.persist(persist_dir=index_path)</pre><p>If using the same directory path to store more than one index, you can use the index_name parameter to be able to separate one from another.</p><h4>Building RAG engine</h4><p>RAG engine is the heart of our chatbot that processes the user query, retrieves relevant info from the knowledge base, and then uses the LLM to generate appropriate response.</p><p>Llamaindex supports two types of engines: query engine for direct and specific queries and chat engine for dynamic conversational interactions. For building our chatbot, we will use chat engine.</p><p><strong>Load knowledge base</strong></p><p>If the index representing our knowledge base isn’t available in the memory, we need to load index before creating the chat engine. Let’s load the index we saved earlier in the local filesystem by adding the code below.</p><pre>storage_context = StorageContext.from_defaults(persist_dir=index_location)<br>index = load_index_from_storage(storage_context)</pre><p>Please note that the Storage Context should be configured exactly as it was while building and storing the index.</p><p><strong>Setup chatbot persona</strong></p><p>In RAG systems, system prompt is used to set the context, tone, style, and boundary of responses by LLM defining its purpose and guiding its behavior throughout the interaction. The system prompt below defines the purpose and persona of our chatbot.</p><pre>system_prompt = (<br>  &quot;I am a nutritional and dietary consultant who helps people find best nutritional and dietary guidelines&quot;<br>  &quot;based on the document provided.&quot;<br>  &quot;I am professional and friendly able to hold a warm conversation.&quot;<br>)</pre><p><strong>Setup context</strong></p><p>The context for the RAG chat engine can be defined using context template that instructs the LLM for how to use the contextual info provided to the LLM in the request. Here, we are using a very basic context template.</p><pre>context_template=(<br>  &quot;Here are the relevant documents for the context:\n&quot;<br>  &quot;{context_str}&quot;<br>  &quot;Given the context information and not prior knowledge, answer the query&quot;<br>)</pre><p><strong>Create RAG engine</strong></p><p>Llamaindex chat engine enables us to create a chatbot using our proprietary knowledge base. Llamaindex offers different types of chat engine, and we can configure the chat engine of our choice using the chat_mode parameter.</p><p>We are using Context Plus Context chat engine that is suitable for questions directly related to the knowledge base and general interactions. You may follow this <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/modules/">link</a> to know more about different types of chat engines available with Llamaindex.</p><pre>engine = index.as_chat_engine(chat_mode=”condense_plus_context”,<br>  similarity_top_k = 1,<br>  system_prompt = chatbot_persona,<br>  context_template = context_template,<br>  verbose = False<br>)</pre><p>The similarity_top_k parameter specifies the number of results to fetch from the knowledge base for a given query.</p><p><strong>Add chat function</strong></p><p>Add the code below to create chat function for our chatbot. We can use the stream_chat function of Llamaindex chat engines to provide streaming support for our application.</p><pre>def chat(self, query: str, history: list[ChatMessage]) -&gt; Iterable[str]:<br>  messages.extend([ChatMessage(role=&quot;user&quot;, content=message.content) for message in history])<br>  messages.append(ChatMessage(role=&quot;user&quot;, content=query))<br>  <br>  response = engine.stream_chat(query, chat_history=messages)<br>  <br>  for text in response.response_gen:<br>    yield text</pre><p>The chat history parameter allows for making the interaction with LLM stateful providing more context aware and engaging interaction.</p><blockquote>Please note that <strong>mel</strong> module from Mesop provides the resources to maintain all user queries in the history and then passes that along to the chat function when calling the function for each user query entered in the chatbot window.</blockquote><h4>Create RAG App</h4><p>Now with the indexed knowledge base and RAG engine ready, it’s time to test our RAG chatbot using a simple Chat UI. Google has recently released its Pythonic UI framework Mesop and I found it to be easy, intuitive, and powerful to use. With a utility package mel, you need just a few lines of code to spin up a functional chatbot.</p><p>Add the following code to initialize the chatbot UI.</p><pre>def page():<br>  mel.chat(transform, title=&quot;RAG Chat&quot;, bot_user=&quot;Bot&quot;)</pre><p>And this code block to enable Mesop UI to call our chat engine.</p><pre>def transform(message: str, history: list[mel.ChatMessage]):<br>  for resp in rag.stream_chat(message, history):<br>    yield resp</pre><p><strong>Run the Code</strong></p><p>We can run the code by firing the following command from the command shell. Please ensure that the python module chatbot.py is accessible to the command shell.</p><pre>mesop chatbot.py</pre><p>If everything goes well, you should see following output on your command shell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*LwPcygfwMnVAHopo35RhTQ.png" /></figure><p>Navigate to localhost:32123 to see following chat window there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcM77tvgVIJshCBNTXXTpg.png" /></figure><p>And as you enter the query, the chatbot uses the Azure OpenAI GPT-3.5 llm to provide response after querying the index available to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxQTXUxWiEJ5OBVvMkx2-A.png" /></figure><p>We have used the starter utility provided with Mesop to quickly build this chatbot however you can use advanced Mesop UI features and capabilities to build highly customized chatbot UI pertaining to your requirements and choices.</p><h4>Additional Resources</h4><p>A sample project has been made available as public repo on GitHub and you can find the it here.</p><p><a href="https://github.com/sunil-baidyanath/sample-rag-app">GitHub - sunil-baidyanath/sample-rag-app</a></p><p>This example project provides the basic implementation of RAG chatbot as discussed in this article and can be used for reference purposes only. Please also note that this project uses Azure Blob Storage to store the knowledge base and Azure VM to run the chatbot application and the source code has been updated to support this capability.</p><h4>References</h4><ol><li><a href="https://docs.llamaindex.ai/en/stable/examples/customization/llms/AzureOpenAI/">https://docs.llamaindex.ai/en/stable/examples/customization/llms/AzureOpenAI/</a></li><li><a href="https://google.github.io/mesop/components/chat/#overview">https://google.github.io/mesop/components/chat/#overview</a></li><li><a href="https://docs.llamaindex.ai/en/stable/understanding/">https://docs.llamaindex.ai/en/stable/understanding/</a></li><li><a href="https://realpython.com/build-llm-rag-chatbot-with-langchain/">https://realpython.com/build-llm-rag-chatbot-with-langchain/</a></li><li><a href="https://machinelearningmastery.com/building-a-simple-rag-application-using-llamaindex/">https://machinelearningmastery.com/building-a-simple-rag-application-using-llamaindex/</a></li><li><a href="https://medium.com/@vikrambhat2/building-a-rag-system-and-conversational-chatbot-with-custom-data-793e9617a865">https://medium.com/@vikrambhat2/building-a-rag-system-and-conversational-chatbot-with-custom-data-793e9617a865</a></li><li><a href="https://blog.tobiaszwingmann.com/p/custom-llm-chatbot-on-your-enterprise-data">https://blog.tobiaszwingmann.com/p/custom-llm-chatbot-on-your-enterprise-data</a></li><li><a href="https://www.aporia.com/learn/build-a-rag-chatbot/">https://www.aporia.com/learn/build-a-rag-chatbot/</a></li><li><a href="https://cohere.com/blog/rag-chatbot">https://cohere.com/blog/rag-chatbot</a></li><li><a href="https://medium.com/@yingbiao/chatbot-with-llm-and-rag-in-action-575382df4323">https://medium.com/@yingbiao/chatbot-with-llm-and-rag-in-action-575382df4323</a></li><li><a href="https://aws.amazon.com/blogs/machine-learning/build-powerful-rag-pipelines-with-llamaindex-and-amazon-bedrock/">https://aws.amazon.com/blogs/machine-learning/build-powerful-rag-pipelines-with-llamaindex-and-amazon-bedrock/</a></li><li><a href="https://pub.towardsai.net/a-complete-guide-to-rag-and-llamaindex-2e1776655bfa">https://pub.towardsai.net/a-complete-guide-to-rag-and-llamaindex-2e1776655bfa</a></li><li><a href="https://huggingface.co/learn/cookbook/en/rag_llamaindex_librarian">https://huggingface.co/learn/cookbook/en/rag_llamaindex_librarian</a></li><li><a href="https://www.llamaindex.ai/blog/building-a-serverless-rag-application-with-llamaindex-and-azure-openai">https://www.llamaindex.ai/blog/building-a-serverless-rag-application-with-llamaindex-and-azure-openai</a></li><li><a href="https://docs.llamaindex.ai/en/stable/optimizing/production_rag/">https://docs.llamaindex.ai/en/stable/optimizing/production_rag/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92cae73a6a9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Overview of a RAG Application]]></title>
            <link>https://medium.com/@sunil.baidyanath/an-overview-of-a-rag-application-d8c144e4ef64?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8c144e4ef64</guid>
            <category><![CDATA[embedding-model]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[chunking-strategies]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 18:23:57 GMT</pubDate>
            <atom:updated>2024-11-08T03:33:48.086Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This is the second post of the series <a href="https://medium.com/@sunil.baidyanath/adopting-generative-ai-in-an-enterprise-706367bac03a">Adopting Generative AI in an Enterprise</a>. The previous post in the series is <a href="https://medium.com/@sunil.baidyanath/large-language-models-in-an-enterprise-c94bbecb7710">Large Language Models in an Enterprise</a>. The next post in the series is <a href="https://medium.com/@sunil.baidyanath/building-a-simple-rag-chatbot-using-llamaindex-and-mesop-92cae73a6a9d">Building a Simple RAG Chatbot using Llamaindex and Mesop</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVMXspreeNWSq2rmgQdeHg.jpeg" /><figcaption>Image generated by Copilot</figcaption></figure><p>A RAG-based LLM application uses pre-formatted <a href="https://en.wikipedia.org/wiki/Knowledge_base">knowledge base</a> maintained in a <a href="https://en.wikipedia.org/wiki/Vector_database">vector database</a> for generating more appropriate and context-aware responses to user queries. The vector database stores the knowledge in the form of <a href="https://en.wikipedia.org/wiki/Word_embedding">embeddings</a> that are numerical representation of the raw data. A special type of algorithms called embedding models are used to generate embeddings for a given raw dataset. The vectorized knowledge base allows for efficient and precise information retrieval enabling an LLM application to augment response generation.</p><h4>A Simple RAG Application</h4><p>The diagram below depicts a RAG application with its key components and activities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8fsDo9hokaiZqdjtVxRPYA.png" /><figcaption>A RAG Application</figcaption></figure><p>A RAG application is primed with specific domain knowledge vectorized using a given embedding model. The application uses this knowledge to answer user queries in following steps:</p><ol><li>Upon the receipt of a user query, the application translates the query to appropriate search phrase using a specific embedding model.</li><li>The application then uses semantic search to retrieve relevant data for the given search phrase from the knowledge base.</li><li>With the retrieved information, the application augments the user query and passes this augmented query to the LLM as RAG call.</li><li>The LLM uses the input provided in the RAG call to generate appropriate response which is then processed by the application per specified quality criteria before passing that on to the end user.</li></ol><blockquote>The embedding model used to translate the user query to search phrase is same as the one used for vectorizing the knowledge base that is target of the search.</blockquote><h4>How a RAG Application Work?</h4><p>A RAG-based LLM application has three key phases:</p><ol><li>Retrieval: retrieves relevant information from a vectorized knowledge base.</li><li>Augmentation: the query input for the LLM is augmented with the information retrieved.</li><li>Generation: the LLM uses the input query augmented with additional relevant info to generate an accurate and contextually relevant response.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPlIAhkiIsAn_fixWTo4Gw.png" /><figcaption>Working of a RAG App</figcaption></figure><p><strong>Information Retrieval</strong></p><p>In the retrieval phase, application search and retrieve information relevant to a given user query.</p><p>The RAG system retrieves the relevant information from the knowledge base and augments the user query with this info providing the LLM with the context.</p><p>This information retrieval can be from both a public knowledge base hosted over the Internet, or a more controlled proprietary knowledge base hosted in closed enterprise setting supporting specific security and reliability requirements. The data retrieval process can be made more efficient by employing semantic search over the vectorized knowledge base.</p><p>Usage of semantic search allows multidimensional data analysis, improving our understanding of data relationships based on context.</p><p><strong>Query Augmentation</strong></p><p>The external knowledge retrieved is added to the user query and then the augmented RAG query is passed to the LLM for response generation.</p><p>The RAG query also includes specific prompts providing instruction to the LLM for generating response aligned to guidelines and templates for specific tasks and requirements.</p><blockquote>A RAG query thus can be thought of consisting of three components: Context, Role, and User query. The context serves as the knowledge repository, the role specifies the purpose of the RAG system, and the user query represents the information requirement of the user.</blockquote><p>The query augmentation is the critical step in a RAG flow providing the LLM with the context that not only expands the knowledge of LLM but also improves its understanding of the specific ask.</p><p>The LLM uses the RAG query to create contextually aware content customized to specific requirements.</p><p><strong>Response Generation</strong></p><p>The LLM processes RAG query components to generate an appropriate response for the user query.</p><p>It uses the user query to understand the information requirement of the user, then extracts necessary information from the context, and uses its own internal knowledge to synthesize an engaging answer that is tailored to the given conversation context using the role.</p><p>The LLM then pass this response along with citation to the specific data sources it referenced for the response generation.</p><blockquote>In a nutshell, grasping the anatomy of RAG queries and data retrieval via semantic search empowers us to unlock the potential of the RAG technology, facilitating efficient knowledge search and context-aware content generation.</blockquote><h4>Functions of a RAG Application</h4><p>A RAG-based application involves three main work streams:</p><ol><li>Build knowledge base for preparing data, vectorize the information, and maintain the knowledge base in a vector store.</li><li>Respond to user query for processing user query and generate response.</li><li>Maintain conversation context for maintaining conversation context to provide stateful conversation experience over stateless GPT calls.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*3FLtoFQsw5I3vUqeWllw1w.png" /><figcaption>Rag Work Streams</figcaption></figure><p><strong>Build/maintain knowledge base</strong></p><p>This work stream involves building and maintaining a vectorized knowledge base supporting semantic search.</p><p>This builds the library for the RAG system augmenting its knowledge base with external knowledge required to answer user queries accurately and with the ability to explain its responses.</p><p>This work stream has following key activities:</p><ul><li>Prepare data by extracting relevant information from the source data systems</li><li>Break data into smaller chunks</li><li>Create embeddings</li><li>Store embeddings in a vector database</li><li>Build searchable knowledge base</li></ul><p><strong>Respond to user queries</strong></p><p>This work stream defines the core of a RAG application and includes RAG query flow.</p><p>It includes processing user query, retrieving relevant information from knowledge base, augmenting the user query, passing this augmented query to the LLM, and then returning the response from LLM to the end user.</p><p>This work stream has following key activities:</p><ul><li>Setup appropriate prompts</li><li>Build query engine</li><li>Convert query into embeddings</li><li>Retrieve relevant info from knowledge base</li><li>Call the LLM with the relevant documents and get the response</li></ul><p><strong>Maintain conversation contexts</strong></p><p>This work stream maintains the state of conversation across multiple RAG calls.</p><p>LLM calls are stateless and therefore to provide a continuous and consistent conversation experience, each RAG call should include relevant context information. Conversation context acts as memory for a RAG application. User queries and responses from LLM are summarized and saved to be used for augmenting the subsequent user query.</p><p>This work stream has following key activities:</p><ul><li>Customize prompts to suit the conversation context</li><li>Setup LLM for conversation</li><li>Summarize conversation history</li><li>Maintain conversation summary</li><li>Build and maintain user profile</li></ul><h4>Elements of a RAG Application</h4><p>This section describes components that form the building blocks for a RAG-based application. These components support the three RAG work streams described above.</p><p><strong>Data Loading</strong></p><p>Data loading is ingestion of structured and/or unstructured data from various data sources.</p><p>These loaders load data from an array of data formats such as HTML, PDF, codebase, and many others from different locations including document stores, databases, and enterprise APIs. All this data is converted into document objects containing both text and associated metadata.</p><p>Following considerations should be followed while designing a data ingestion pipeline:</p><ul><li>Appropriate data sources portfolio that is just enough for the required use cases with push mechanisms to load data changes.</li><li>Comprehensive data governance with support for auditing and cataloging data sources, redacting sensitive data, and establishing foundation for context training.</li><li>Extensive data quality checks assessing data for diversity, size, biases, and noise.</li><li>Real-time data cleansing to filter out noise, typos, duplicates, and sensitive data in real-time.</li></ul><p>Data loading should ensure that final data set is clean, deduped, and correct semantic representation of the knowledge it refers to.</p><p><strong>Chunking</strong></p><p>Chunking is segmentation of input text into concise, meaningful units that enables the retrieval system to locate relevant contextual passages for appropriate response generation.</p><p>The quality and structure of these chunks are critical to the efficacy of a RAG system and therefore an appropriate chunking method should be picked to maximize the effectiveness of a RAG system. Common chunking methods splits documents based on their content structure or semantic structure. Other advanced chunking methods combine different chunking approaches or adjust chunk size based on real-time factors like query complexity or document attributes. You can learn more about chunking methods <a href="https://www.pinecone.io/learn/chunking-strategies/">here</a>.</p><p>Evolving an optimal chunking strategy should include following key considerations:</p><ul><li>Assess application requirements and content characteristics.</li><li>Find optimal chunk size by experimenting with different chunk size and evaluating against precision and recall value for retrieval.</li><li>Balance relevance and efficiency. While larger chunk sizes can enhance performance, an excess of context might introduce noise.</li><li>Prepare the data considering targeted embedding model and vector store.</li></ul><p>Other chunking strategies to consider include utilizing smaller chunks and retrieving adjacent chunk content or maintaining multiple embeddings for a document.</p><p><strong>Embedding</strong></p><p>Embeddings are numerical representation of text capturing its semantic meaning and thereby making it easier to find similar pieces of text quickly.</p><p>This allows for conceptualization of the knowledge represented by the dataset within a vector space. Embedding uses tokenization to first split a text into set of tokens before mapping them into vector space using an appropriate embedding model. Embedding is critical for facilitating semantic search and choice of the embedding model significantly impacts retrieval and quality scores.</p><p>Off-the-shelf embedding models can generate embeddings for data chunks that work well for most use cases. However, for a specific domain, these models may not adequately represent the domain in the vector space, resulting in poor retrieval quality.</p><p><strong>Vector stores</strong></p><p>Vector stores provide efficient storage and retrieval for embeddings and thereby play a pivotal role in a RAG-based application. These vector stores are essential for ensuring efficient data retrieval, contextual augmentation, and the overall performance of a RAG-based application by enhancing the contextual relevance and accuracy of the generated output.</p><p>Given below are the key advantages of vector stores in the realm of RAG-based systems:</p><ul><li>Overcome limitations with structured database.</li><li>Efficiently store data in vector form.</li><li>Efficient retrieval of relevant contextual information.</li><li>Support for clustering and analysis in multidimensional space.</li></ul><p>Vector databases store information in the form of multi-dimensional vectors and serve as repositories for embeddings that represent the semantic meaning of text. With their ability to efficiently retrieve data precisely and quickly, these databases enable RAG systems to search and retrieve relevant information based on contextual relevance.</p><p>Common vector databases include ChromaDB, FAISS, Pinecone, Qdrant, and Weaviate.</p><p><strong>Retrievers</strong></p><p>Retrievers are core component in a RAG system facilitating the efficient retrieval of relevant information from large knowledge bases to augment a RAG call.</p><p>Retrievers use semantic search to efficiently find and extract relevant information from the knowledge base based on the meaning of the context keywords provided via prompts in the RAG call. Acting as a bridge between the LLM and external knowledge base, retrievers enable a RAG system to access and incorporate relevant contextual information for augmenting the RAG call and enriching the content generation.</p><p>There are different types of retrievers depending on how they calculate similarity between the query and the indexed documents. They can be keyword-based or embedding-based but dependent on the type of vector store.</p><p>You can learn more about retrievers <a href="https://docs.haystack.deepset.ai/docs/retrievers">here</a>.</p><p><strong>Prompt Engineering</strong></p><p>Prompt engineering is the technique to influence content generation in LLM to get responses optimized and tailored for a given conversation scenario.</p><p>Common prompt engineering techniques includes:</p><ul><li>Zero-shot learning: Prompt without any example</li><li>One-shot learning: Prompt with one example providing context or suggesting response format</li><li>Few-shot learning: Prompt with few (2–5) examples to make the model understand the pattern or style of response</li><li>Chain-of-thought prompting: Prompt asking the model to provide stepwise detail for thought process.</li><li>Iterative prompting: Iteratively guiding the model to desired output by using the output of current prompt to create more refined next prompt.</li><li>Negative prompting: Prompt specifying LLM what not to do.</li><li>Hybrid prompting: Combining different techniques for example chain-of-thought and iterative prompting to get more specific or creative output.</li><li>Prompt chaining: Breaking down a complex user query into more than one simple prompts, and then chaining the individual outputs together to form the final output.</li></ul><p>While crafting prompts using any of these techniques, we can use the COSTAR framework that provides a structured methodology for creating effective prompts. By breaking down a prompt into distinct building blocks addressing different aspects, COSTAR allows for methodically refining and optimizing prompts ensuring accurate, relevant, and coherent response from LLMs.</p><p>Learn more about COSTAR framework <a href="https://vivas.ai/blogs/mastering-prompt-engineering-a-guide-to-the-co-star-and-tidd-ec-frameworks">here</a>.</p><p><strong>Context Awareness</strong></p><p>LLMs are stateless and have no native awareness of the conversation. You need to provide the chat history data to the LLM so it can become conversationally aware. Chat history can be saved and then added to a user prompt for providing the reference to the ongoing conversation for LLMs.</p><p>There are various forms via which chat history can be passed on to the LLM and they include:</p><ul><li>Overall chat history: The entire chat history of this conversation.</li><li>Overall chat summary: The entire chat history summarized using LLM.</li><li>Chat history within a given time window: Part of chat history within given time window.</li><li>Chat summary within a given time window: Part of chat summary within given time window.</li><li>Knowledge graph from chat history: Knowledge graph extracted from the chat history.</li><li>Entity models from chat history: Entity model extracted from the chat history.</li></ul><h4>Conclusion</h4><p>RAG has revolutionized the adoption of GenAI and organizations are rushing to build RAG applications to modernize their service portfolio. The LLM technology landscape offers a number of frameworks to help accelerate GenAI adoption with rich feature sets and ready-to-use RAG components. Common such frameworks include Langchain, Llamaindex, and Haystack.</p><h4>References</h4><ol><li><a href="https://docs.llamaindex.ai/en/stable/getting_started/concepts/">High-Level Concepts — LlamaIndex</a></li><li><a href="https://www.chatbees.ai/blog/rag-llm">https://www.chatbees.ai/blog/rag-llm</a></li><li><a href="https://aws.amazon.com/what-is/retrieval-augmented-generation/">What is RAG? — Retrieval-Augmented Generation AI Explained — AWS (amazon.com)</a></li><li><a href="https://cloud.google.com/use-cases/retrieval-augmented-generation?hl=en">https://cloud.google.com/use-cases/retrieval-augmented-generation?hl=en</a></li><li><a href="https://datasciencedojo.com/blog/guide-to-retrieval-augmented-generation/">https://datasciencedojo.com/blog/guide-to-retrieval-augmented-generation/</a></li><li><a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG">https://research.ibm.com/blog/retrieval-augmented-generation-RAG</a></li><li><a href="https://meetcody.ai/blog/rag-api-definition-meaning-retrieval-augmented-generation-llm/">https://meetcody.ai/blog/rag-api-definition-meaning-retrieval-augmented-generation-llm/</a></li><li><a href="https://winder.ai/big-data-llms-retrieval-augmented-generation-rag/">Big Data in LLMs With Retrieval-Augmented Generation (RAG) (winder.ai)</a></li><li><a href="https://towardsdatascience.com/routing-in-rag-driven-applications-a685460a7220">https://towardsdatascience.com/routing-in-rag-driven-applications-a685460a7220</a></li><li><a href="https://medium.com/langstream/building-a-q-a-rag-chatbot-with-conversational-awareness-263c0f7c69f2">https://medium.com/langstream/building-a-q-a-rag-chatbot-with-conversational-awareness-263c0f7c69f2</a></li><li><a href="https://medium.com/@ranadevrat/creating-a-chatbot-using-open-source-llm-and-rag-technology-with-lang-chain-and-flask-ca2f5511e82e">https://medium.com/@ranadevrat/creating-a-chatbot-using-open-source-llm-and-rag-technology-with-lang-chain-and-flask-ca2f5511e82e</a></li><li><a href="https://blog.ml6.eu/semantic-search-a-practical-overview-bf2515e7be76">https://blog.ml6.eu/semantic-search-a-practical-overview-bf2515e7be76</a></li><li><a href="https://www.promptingguide.ai/research/rag">https://www.promptingguide.ai/research/rag</a></li><li><a href="https://generativeaiassociation.org/articles/grounding-llms-responses-with-factual-data-retrieval-augmented-generation-rag/">https://generativeaiassociation.org/articles/grounding-llms-responses-with-factual-data-retrieval-augmented-generation-rag/</a></li><li><a href="https://www.k2view.com/blog/gartner-llm/#Gartner-LLM-tip-5-Choose-your-technology">https://www.k2view.com/blog/gartner-llm/#Gartner-LLM-tip-5-Choose-your-technology</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8c144e4ef64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Large Language Models in an Enterprise]]></title>
            <link>https://medium.com/@sunil.baidyanath/large-language-models-in-an-enterprise-c94bbecb7710?source=rss-14a25a7d44f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c94bbecb7710</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm-function-calling]]></category>
            <category><![CDATA[gpt]]></category>
            <dc:creator><![CDATA[Sunil Thakur]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 18:18:35 GMT</pubDate>
            <atom:updated>2024-11-08T03:31:44.563Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This is the first post of the series <a href="https://medium.com/@sunil.baidyanath/adopting-generative-ai-in-an-enterprise-706367bac03a">Adopting Generative AI in an Enterprise</a>. The next post in the series is <a href="https://medium.com/@sunil.baidyanath/an-overview-of-a-rag-application-d8c144e4ef64">An Overview of a RAG application</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kiVp9YavrCKczyAkAj9A2g.jpeg" /><figcaption>Image generated by Copilot</figcaption></figure><p>Since its launch in 2022, ChatGPT has revolutionized the <a href="https://en.wikipedia.org/wiki/Generative_artificial_intelligence">Generative Artificial Intelligence</a> (GenAI) space and brought in a paradigm shift in adding AI capabilities to business functions particularly the ones targeting end users in a more engaging way.</p><h4>ChatGPT Shifts the Spotlight to Large Language Model</h4><p>Created by <a href="https://openai.com/">OpenAI</a>, <a href="https://en.wikipedia.org/wiki/ChatGPT">ChatGPT</a> is an AI-powered chatbot that uses <a href="https://en.wikipedia.org/wiki/Large_language_model">large language models</a> to enable human-like conversations in a dialogue format on wide range of topics.</p><p>Large language models (LLMs) are <a href="https://en.wikipedia.org/wiki/Deep_learning">deep learning</a> AI models that are pre-trained on vast amount of data and can understand as well as create natural language content. These models use <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">transformer </a>architecture to learn the patterns and structures of human language over vast amounts of natural language data and then use that information to generate responses to user prompts.</p><p>ChatGPT uses a specific type of LLM called <a href="https://en.wikipedia.org/wiki/Generative_pre-trained_transformer">Generative pre-trained transformers</a> (GPTs). Based on the transformer architecture, GPTs are pre-trained on large <a href="https://en.wikipedia.org/wiki/Data_sets">data sets</a> of unlabeled text and can generate novel human-like content. Though the first GPT was introduced by OpenAI in 2018, the term ‘GPT’ generally refers to <a href="https://en.wikipedia.org/wiki/Foundation_model">foundation models</a> released by different companies such as GPT-n series models from OpenAI.</p><p>ChatGPT is built on OpenAI’s pre-trained proprietary GPT foundation models and is fine-tuned for conversational applications. Besides its ability to understand, generate, and refine human-like text, it can also provide a personalized interaction with human users by adapting to specific conversational context.</p><p>The wild popularity of ChatGPT has aroused deep interest in enterprises to explore opportunities for applying LLMs for upgrading their capability mix to attain sustainable competitive advantage.</p><h4>Common LLM Use Cases</h4><p>These LLMs are becoming increasingly important in a variety of applications such as natural language processing, language translation, code and text generation, and many more.</p><p>Common enterprise use cases for LLMs include:</p><ul><li>Chatbot and virtual assistants</li><li>Code generation and debugging</li><li>Sentiment analysis</li><li>Text classification and clustering</li><li>Language translation</li><li>Summarization and paraphrasing</li><li>Content generation</li></ul><p>While LLMs are powerful and can create compelling content, enterprises must note that they are not trained to be fact machines and therefore the veracity of the content they generate cannot be trusted.</p><p>It necessitates the need to always fact check and understand the responses generated by LLMs before passing them on for consumption by an end user or process.</p><h4>Limitations of LLMs</h4><p>As enterprises rush to adopt LLMs for building new generation of intelligent services and products for their users, LLMs still have critical limitations discouraging their enterprise-wide adoption across industries.</p><p>Some of these limitations that restraint organizations for building proprietary LLM-powered products include:</p><ul><li><strong>Hallucinations:</strong> It refers to instances where the model confidently and authoritatively generates response that is either inaccurate or completely fabricated.</li><li><strong>Limited Knowledge:</strong> Though GPT models are trained on vast amount of publicly available data, the models don’t have access to any events or happenings post the training period. This knowledge cut-off limits the usage of these models only for use cases that doesn’t require latest knowledge accrued post the training.</li><li><strong>Lack of proprietary domain knowledge:</strong> With no access to domain specific proprietary data not available or only sparsely available in public, GPT models either have no knowledge or worse eschewed knowledge for these domains. This very often results in either incorrect or highly biased responses making such models ineffective for domain-specific tasks.</li><li><strong>Lack of explainability:</strong> GPT models cannot produce references to specific piece of knowledge that it used to derive the content it generates. However, this explainability is required in critical domains such as healthcare where it’s important to substantiate the response generated with relevant detail.</li><li><strong>Personalization concerns:</strong> GPT models are stateless by design and therefore cannot be made context aware and personalized for given user or context.</li></ul><p>These limitations raise serious concerns for potential risks if not addressed effectively before going for enterprise adoption of LLMs.</p><h4>Augmenting LLMs</h4><p>GPT models address these limitations by allowing for augmenting the foundation models with proprietary knowledge base. There are two approaches for augmenting the foundation GPT models: Grounding and Finetuning.</p><p><strong>Grounding</strong></p><p>Grounding is conditioning and contextualization of existing LLMs with custom dataset and documents to generate a stateful response.</p><p>Retrieval Augmented Generation (RAG) with prompt engineering allows for grounding LLMs to provide responses based on proprietary knowledge base and in a personalized conversational context.</p><p><em>Use cases</em></p><ul><li>Useful for grounding LLM generations on specific information to reduce hallucinations.</li><li>Useful for conditioning LLM generations on some context or state.</li><li>Useful for leveraging data that can’t be incorporated as training data such privacy-sensitive data and fast-changing data.</li></ul><p><em>Limitations</em></p><ul><li>It relies on the LLM to be “smart enough” to use the given context correctly.</li><li>The effectiveness of RAG is limited by the availability of right data.</li></ul><p><strong>Finetuning</strong></p><p>Finetuning involves retraining an LLM with custom data that may change model internally including its weights and architecture. LLMs are finetuned only when we want to change it for performance, behavior, cost profile and output.</p><p><em>Use cases</em></p><ul><li>Teaching LLMs new tasks that can’t be done well via prompt engineering or demonstrations.</li><li>Custom output formats</li><li>Learning text generation for underrepresented domains and/or underrepresented style</li></ul><p><em>Limitations</em></p><ul><li>LLMs can’t reliably remember new knowledge with finetuning unless training from scratch which is not practically feasible.</li><li>Finetuning is also a very time and resource intensive technique that comes with severe restrictions with data governance.</li></ul><p>Both grounding and finetuning are valid approaches for using LLMs effectively with custom data.</p><blockquote><strong>However,</strong> <strong>finetuning is a significant investment so we should consider leveraging existing foundation models with proper grounding to support our specific needs</strong>.</blockquote><p>Also, as these models become more and more powerful with ever increasing size of their training dataset, the need for finetuning will gradually decrease except for some niche domains for which enough data is not available in public domain.</p><h4><strong>Approaches for Grounding LLMs</strong></h4><p>There are two common approaches for grounding LLMs to adapt them for a specific business context.</p><ol><li><strong>Function calling:</strong> LLM builds grounding by calling appropriate business functions for a given user query.</li><li><strong>Retrieval Augmented Generation (RAG):</strong> Relevant grounding is passed to the LLM along with user query by searching the vectorized knowledge base.</li></ol><p>In summary, while RAG pushes grounding information to LLM, function calling enables LLM to pull in grounding information for making the relevant contextual knowledge available for appropriate response generation for a given user query.</p><p><strong>Function Calling</strong></p><p>The Function Calling feature enables LLMs to use external tools and systems to get specific knowledge for answering user queries.</p><p>The LLMs are provided with list of function descriptions during initialization. Upon a user query, the LLM identifies an appropriate function to call for getting the specific information which it then uses to generate response for the user. These functions are designed to access enterprise information systems for providing relevant information to the LLM for generating appropriate response to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*MikGKEWgHdkj62qWyYBXEA.png" /><figcaption>Grounding LLM using Function Calling</figcaption></figure><p>Function calling supports integration with enterprise service APIs and more appropriate for fast changing data or for triggering specific action in the enterprise such as sending notification, updating status, or submitting a form.</p><blockquote>Please note that an LLM doesn’t directly call a function, it only returns the specification for the function it finds more appropriate for given user prompt based on the function documentations fed to it.</blockquote><p>The returned function specification primarily includes name of the function, and the list of parameter values to be passed to the function. The host application implementing the function calling uses this information to call the function appropriately.</p><p><strong>RAG</strong></p><p>RAG pattern helps foundation model pretrained on general data to generate domain-specific responses by enriching the model with additional proprietary data.</p><p>In this approach, LLMs use a search API to find the most relevant content for a given user query. The search API implements semantic search function over a vectorized knowledge base for the given business domain.</p><p>The RAG pattern includes three steps:</p><ul><li><strong>Retrieval</strong>: The process of fetching relevant data from external sources. LangChain simplifies this step with a variety of document loaders and integrations.</li><li><strong>Augmentation</strong>: Enhancing the LLM’s responses by incorporating retrieved data, ensuring more accurate and contextually relevant outputs.</li><li><strong>Generation</strong>: The LLM generates responses based on both the original query and the augmented data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*kRxHktjqo0i0ZpHK_hlVFA.png" /><figcaption>Grounding LLM using RAG</figcaption></figure><p>RAG supports loading external data from variety of sources including document repositories, databases, and APIs to build a searchable knowledge base and is best suited for relatively slow changing or static business data.</p><p><strong>Combining RAG and Function Calling</strong></p><p>Enterprises can combine the capabilities of RAG and function calling to build a comprehensive decision support system.</p><blockquote>While RAG can support decision making with the insights derived from massive amount of enterprise data, function calling can enable integration with enterprise applications to augment the decision with latest knowledge from an application or trigger an action in the enterprise via an application.</blockquote><p>Function calling can also be used to get information from an application which can then be used by RAG to augment the user query for finding more targeted information from the vectorized knowledge base.</p><p>The exact orchestration of RAG and function calling for a given use case will depend on the data flow requirements and workflow integration needs of the use case under consideration.</p><h4>References</h4><ol><li><a href="https://www.gartner.com/en/topics/generative-ai">https://www.gartner.com/en/topics/generative-ai</a></li><li><a href="https://developers.google.com/machine-learning/resources/intro-llms">https://developers.google.com/machine-learning/resources/intro-llms</a></li><li><a href="https://aws.amazon.com/what-is/large-language-model/">https://aws.amazon.com/what-is/large-language-model/</a></li><li><a href="https://medium.com/towards-generative-ai-applications/introduction-to-llms-and-their-applications-f68d31c8d0fd">https://medium.com/towards-generative-ai-applications/introduction-to-llms-and-their-applications-f68d31c8d0fd</a></li><li><a href="https://www.linkedin.com/pulse/three-techniques-adapt-llms-any-use-case-baseten/">https://www.linkedin.com/pulse/three-techniques-adapt-llms-any-use-case-baseten/</a></li><li><a href="https://medium.com/@vipra_singh/building-llm-applications-introduction-part-1-1c90294b155b#4d28">https://medium.com/@vipra_singh/building-llm-applications-introduction-part-1-1c90294b155b#4d28</a></li><li><a href="https://www.promptingguide.ai/applications/function_calling">https://www.promptingguide.ai/applications/function_calling</a></li><li><a href="https://arxiv.org/pdf/2307.06435">https://arxiv.org/pdf/2307.06435</a></li><li><a href="https://www.elastic.co/what-is/large-language-models">https://www.elastic.co/what-is/large-language-models</a></li><li><a href="https://ashishjaiman.medium.com/large-language-models-llms-260bf4f39007">https://ashishjaiman.medium.com/large-language-models-llms-260bf4f39007</a></li><li><a href="https://techcommunity.microsoft.com/t5/fasttrack-for-azure/grounding-llms/ba-p/3843857">https://techcommunity.microsoft.com/t5/fasttrack-for-azure/grounding-llms/ba-p/3843857</a></li><li><a href="https://www.linkedin.com/pulse/rag-vs-function-calling-fine-tuning-detailed-advanced-jagadeesan-lye8c/">https://www.linkedin.com/pulse/rag-vs-function-calling-fine-tuning-detailed-advanced-jagadeesan-lye8c/</a></li><li><a href="https://thenewstack.io/a-comprehensive-guide-to-function-calling-in-llms/">https://thenewstack.io/a-comprehensive-guide-to-function-calling-in-llms/</a></li><li><a href="https://zilliz.com/blog/harnessing-function-calling-to-build-smarter-llm-apps">https://zilliz.com/blog/harnessing-function-calling-to-build-smarter-llm-apps</a></li><li><a href="https://raga.ai/blogs/llm-grounding-techniques">https://raga.ai/blogs/llm-grounding-techniques</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c94bbecb7710" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>