In a recent post on this channel, “Smart contracts for AI agents”, we described embedding governance directly into each agent through a machine-readable manifest and a runtime sidecar proxy. We noted how it addresses four scaling problems: framework-selection paralysis, compliance drift, reactive validation, and vendor lock-in. The smart contract defines what the agent is, and the sidecar enforces it.

However, an agent may operate fully within its declared boundaries while still producing weak outputs, for example, by falling short of quality targets, or drifting in patterns that emerge only across dozens of runs and several campaigns. The smart contracts approach can tell you that your agent used an unapproved model or accessed restricted data, but it cannot tell you about the quality of the emails your sales agent writes on your behalf. Are they any good? Are they improving over time?

What is missing is a way to accumulate learning at scale. That requires another layer, positioned above enforcement, which converts production traces into organizational knowledge. This article describes what a learning layer looks like and how to build it as shared infrastructure rather than a one-off feature bolted onto each agent.

Why learning gets trapped in silos

Enterprises running multiple agents tend to converge on the same organizational problem. One team builds an evaluation rubric for a sales outreach agent. Another adds observability to a marketing campaign agent. A third writes its own reflection workflow for customer care. Each team reinvents the same cycle: generate output → score it → detect what went wrong → propose a fix → review and approve it → apply it.

Their fixes stay local to their agents. A regression test written for the B2B sales agent never reaches the consumer marketing agent. A quality pattern that recurs in both sales and marketing sits in two separate dashboards. There isn’t a portfolio-level view of which agents are improving, which are degrading, and what failure modes they share.

The underlying issue is that improvement is treated as a feature of individual agents, not as a common layer that every team and campaign can reuse. This builds on an idea we introduced in our original article, “Agentic AI Mesh”, which described feedback loops where “agents configured to critique outputs of other agents… can be used to signal positive and negative performance, triggering the automated development of configuration variations that can be tested and scored.”

What that article didn’t specify was the shared infrastructure, such as approval gates, versioning, and cross-campaign visibility, that is needed to operationalize that loop, and that is what we cover here.

A shared reflection infrastructure

Agentic reflection is the process by which a system designed to evaluate agent performance reviews an agent’s outputs against predefined quality criteria, identifies errors or reasoning gaps, and proposes improvements. The learning part happens when improvements are approved, captured, and reused so that mistakes are not repeated. Building this as shared infrastructure requires the same three structural properties that make the smart contract architecture work.

• Declarative: Evaluation logic, reflection prompts, approval rules, and improvement steps are written down and versioned rather than embedded in ad hoc scripts that only the team that wrote them can access.
• Modular: Generation, evaluation, signal detection, reflection, and application are separate stages that can change independently. A team that wants to swap the evaluation model does not need to touch the generation pipeline.
• Reusable: The loop, the deterministic checks, and the audit trail are common to every agent, and the scouts flag when the same type of failure recurs in more than one agent. Approved changes are versioned and visible across the portfolio, so a fix found by one team can be reviewed and adopted by another.

A worked example using Kedro

We built an example of this approach using Kedro, an open-source Python framework, which offers a modular pipeline structure and explicit data contracts to provide a set of building blocks. Using Kedro to take care of the surrounding orchestration enabled us to focus on the reflection and learning workflow.

To show how this pattern works in practice, we built an example that applies the same reflection and learning workflow across three business functions (B2B sales, consumer marketing, and customer care) in a telecommunications company.

Each function uses different data sources, prompts, and evaluation criteria. The improvement lifecycle is identical for all three:

• Generate: run the AI workflow and capture outputs.
• Evaluate: score outputs against quality criteria using a versioned rubric and an LLM judge.
• Identify signals and patterns: detect recurring issues across runs, such as tone drift, hallucinations, missing required content, or score regressions.
• Reflect: generate targeted recommendations such as prompt updates, skill edits, or evaluation logic changes.
• Review and approve: a human reviewer inspects the proposed changes and approves or rejects them through an explicit approval gate. Nothing goes live without it.
• Apply: version and commit the approved changes, and record each one in an audit log.

In the example, the generate, evaluate, detect, reflect, and apply stages are separate pipelines, with review and approval acting as an approval gate. The pipelines, deterministic checks, and approval gate are shared, while the prompts, style guides, rubrics, and LLM judge dimensions change by use case.

Reflection also proposes new regression cases derived from the failures it found. Once approved, they are committed to that agent’s evaluation dataset (alongside any approved changes to the prompt, skills, or other configurable components) then every change is recorded in a single audit log. The signal moves between agents. When the same type of failure recurs in more than one agent, the scouts record it in a shared signal index and flag it as a cross-unit pattern, rather than leaving it as two unrelated findings in two isolated dashboards. A reviewer can then decide whether a fix belongs to one agent or should be adopted by others.

Reflection maps onto a Shape-Control-Operate (SCO) lifecycle. Prompts and evaluation criteria are defined in the Shape phase, and evaluation scoring runs in the Control phase. The evidence gathered while agents Operate then feeds approved changes back into the Shape-phase artifacts, which closes the loop.

How the evaluation step works

Evaluation uses a fixed set of benchmark cases but does not rely on a single correct or “golden” output. Each case, stored in Langfuse and cached locally, pairs a scenario — customer, product, and case ID — with a rubric defining required and forbidden content, expected call to action, tone, and other agent-specific criteria.

For each case, the generation pipeline produces an output, which is then scored in two ways:

• Deterministic checks validate objective requirements such as subject presence, body length, CTA presence, and fabricated SKU-like tokens.
• An LLM judge evaluates qualitative dimensions such as writing quality, personalization, and groundedness. These dimensions can vary by agent (for example, empathy or compliance).

A case passes when the average score across all scorers meets the configured threshold. Results are captured at both case and aggregate levels and logged to Langfuse, enabling teams to compare runs after prompt changes, inspect traces, and feed the findings back into the reflection loop.

The business impact of the solution is cumulative. Every run, signal, and approved change is recorded, to extend the learning history of each reflection. Approved improvements become reusable patterns that other teams can review and adopt. Over time, the system compounds learning so that new deployments benefit from previous ones.

For a deeper walkthrough of the approach and how it works in practice, watch our recent Kedro Coffee Chat.

Kedro as the foundation for agentic reflection

Each stage is a separate Kedro pipeline with explicit inputs, outputs, and Kedro Data Catalog declarations. Every run is traceable. Kedro acts as a runtime implementation of several smart contract dimensions: data boundaries (what each stage can read and write), audit trail (full run history and I/O contracts), and governance (nothing advances without a declared output and an approval step).

Kedro provides modular, independently versioned stages, and a traceable run history by design. In this example, human approval is an explicit step in the Reflection Hub app, which also embeds Kedro-Viz, a live graph view of the pipeline that anyone can inspect without running it.

In our article “Generative AI workflows need engineering discipline to scale beyond the demo”, we showed how Kedro brings reproducibility, observability, and evaluation to gen AI pipelines. This article further adds the layer above that to show how Kedro helps capture improvements for reuse.

Summary

An agent can be fully compliant with an approved model, correct data boundaries, and pass every guardrail, but it can still write outputs that are mediocre or deteriorating, in ways no manifest is built to detect. Gaps can be filled, but if the fixes are then trapped in silos, the organization repeats the same mistakes.

Reflection can turn failures into approved, versioned changes by flagging patterns that recur across agents, so a fix can be considered across the portfolio rather than staying local to the team that found it. Frameworks and deployment patterns in enterprise AI will keep changing, but building portable contracts, embedded enforcement, and a shared improvement loop will give organizations a foundation to improve their agents rather than just watch over them.

Find out more about Kedro

Kedro is an open-source project supported by a full-time development team. It is part of the LF AI & Data Foundation, with a neutral, vendor-independent governance model that supports long-term sustainability and community-driven development. Beyond the core framework, Kedro is supported by an active open-source community. Contributors and users share patterns, extensions, and real-world experience. A growing list of blog posts, videos, and projects that use Kedro can be found in the awesome-kedro GitHub repository.

Kedro is also backed by extensive documentation. This includes tutorials, example projects, coffee chats, and code snippets ranging from introductory features to more advanced use cases. Teams can get started quickly with a ready-to-use project template, which includes sample datasets and enables users to train their first model with just a few commands from the terminal. If you check it out, drop us a comment below to let us know how you get on!

Authored by Alice Cima and Ravi Kumar Pilla, with additional insights from Rory Walsh, Dante Gabrielli, Joseph Perkins, Stephen Xu, and Sallah Kokaina. The editor was Jo Stichbury.

Using Kedro to build reusable reflection systems for enterprise AI agents was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Standard Retrieval-Augmented Generation (RAG) works well for factual lookup. For example, if you use semantic similarity to retrieve a document, you’ll usually find the correct passage. However, it struggles when the lookup requires reasoning across relationships, say if the answer depends on facts that are distributed across records that share little lexical or semantic similarity with each other.

As an illustration for healthcare, a search might want to discover the risk of a particular medication to patients with a certain condition. A typical query might be: “For a chronic kidney disease patient, which medication may not be safe and interact with their poor kidney function?”

A dense retriever might return the passages whose embeddings sit the closest to the query vector, like a chunk about chronic kidney disease and another about drugs with contraindications for renal patients. But to respond to this question as accurately as possible, we’d have to gather information from separate records, like a diagnosis note, a lab report, or a drug prescription. None of these directly references the others, and there is no structure that encodes the intersection of all of these pieces of data.

GraphRAG addresses this by pairing semantic search with a knowledge graph that represents the structure explicitly: entities such as conditions, medications, and test results become nodes, and the relationships between them become edges. Before passing context to the language model, the system traverses the relationships connected to the retrieved entities so the model can reason across linked nodes rather than treating each document in isolation.

In this article, we describe how we used Kedro to build a demo of a simple reference implementation.

The basics

In our example, the GraphRAG system maintains three representations of the same underlying data, which need to stay consistent with each other as source data changes. The representations are:

• a knowledge graph for relationship traversal
• a vector index for semantic retrieval
• a relational store for aggregate statistics.

Data engineering matters as much as retrieval because of the need for consistency. Inconsistent data arises, for example, when data teams use a collection of scripts, run in undocumented order, with no guarantee that the graph, the vector index, and the analytics store are produced from the same source run. When the knowledge graph becomes stale relative to the embeddings, the retrieval and graph traversal return contradictory context. As a result, the model resolves the inconsistency silently, and may resolve it incorrectly.

Incremental updates compound the challenge. Rebuilding the full graph each time new records arrive is impractical at production scale. A production system needs targeted, merge-capable updates that leave unchanged nodes and edges untouched while incorporating new information: a classic data engineering pipeline problem.

An example pipeline

Using Kedro, we built a five-stage pipeline demo over a synthetic healthcare dataset of 55,500 patient records covering conditions, medications, insurers, admission types, diagnostic results, and demographics. The full source is on the Kedro Academy GitHub repository. While this example is based on healthcare data, none of this structure is specific to healthcare, and the example is deliberately simple to keep it self-contained and easy to run. The entity extraction, graph building patterns, and incremental merge logic apply to any domain with typed entities and relationships.

The pipeline stages are as follows:

• Clean and summarize: The first stage normalizes raw records, removes duplicates, and computes summary statistics per entity. We used patient counts per condition, average billing amounts per admission type, and result distributions across demographic groups. This stage stores them in SQLite and reuses them as retrieval context in later stages.
• Build the knowledge graph: The second stage extracts entities and typed relationships. Nodes represent conditions, medications, insurers, admission types, test results, and blood types. Edges carry a relationship label (TREATED_WITH, COVERED_BY, ADMITTED_AS, SHOWS_RESULT, ASSOCIATED_WITH) and a weight recording how frequently each relationship appears in the data. The stage produces a NetworkX JSON graph with 30 nodes and 120 weighted edges, and an interactive D3.js visualization of the entity network.
• Incrementally update the graph: Rather than rebuilding from scratch on each data refresh, the next stage merges new records into the existing graph using create-or-update semantics. It adds new entities, increments edge weights for existing relationships, and leaves unchanged portions untouched. In this demo, each run of the stage merges a fixed batch of 5,000 records rather than the full corpus, and it updates only the knowledge graph. We designed it to model a real production constraint: while rebuilding a relatively small graph like this is cheap and resolves in a few seconds, a production-sized knowledge graph based on a larger, more complex set of data would take much longer.
• Generate vector embeddings: For each entity, the next stage generates a natural-language summary of its characteristics and connections, then embeds those summaries using OpenAI’s text-embedding-3-small model. This stage stores the embeddings in ChromaDB for semantic similarity search at query time.
• Answer questions: The final stage runs an AI agent backed by GPT-4o. When a query arrives, the agent embeds it and queries ChromaDB for the most similar entity summaries. Starting from those results, it traverses one hop across the knowledge graph to collect neighboring entities and the relationships between them. That combined context (semantic matches plus explicit graph structure) goes to the model. Standard RAG asks, “What is most similar to this query?” This stage adds, “What else is connected to those results?” This is a hybrid GraphRAG pattern: vector retrieval followed by graph traversal.

All five stages write to three different backends, and Kedro’s Data Catalog manages loading and saving for all of them from a single configuration file. Every dataset (the NetworkX graph, the ChromaDB collection, the SQLite statistics table, the LangChain prompt template) is declared in catalog.yml so there is one place to see where each input and output lives and there are no separate scripts to reconcile manually. A full default run builds all three stores from the same source data. The incremental update refreshes only the knowledge graph; keeping the vector index and statistics in step with it would require extending their pipelines in the same way.

Each stage is an independent Kedro pipeline with explicit inputs and outputs. The graph construction logic, the embedding model, and the retrieval strategy sit in separate modules. Swapping the embedding model requires changing one node and its catalog entry, while the graph and statistics stages are unaffected. Kedro-Viz renders the full data lineage across all five stages, and a published Kedro-Viz build can be viewed without any installation on the viewer’s side.

This approach also means it is possible to treat generative AI components as first-class Data Catalog entries, rather than scripts or loose objects spread around the project. It makes them easier to manage and keep track of, and they can be used as input or output for nodes like any other entry. Through the use of Kedro, generative AI datasets receive easy credentials management and follow the same load/save contract as other Kedro datasets.

The graph_update pipeline is excluded from the default run and must be invoked explicitly. It writes to the same catalog entry as graph_construction, so running both in the same default pipeline would create a DAG conflict. Full rebuilds and incremental updates are mutually exclusive operations, so making that explicit in the pipeline registry prevents silent errors.

Choosing between standard RAG and GraphRAG

GraphRAG is not a replacement for standard RAG. For factual retrieval, a vector index is simpler to build, cheaper to operate, and easier to debug.

The case for a knowledge graph becomes clear when questions require multi-hop reasoning: finding conditions that share treatments, identifying which medications cluster around a particular insurer, or following relationship chains across several entity types. Healthcare illustrates the pattern, but the same logic applies wherever the connections between entities carry as much meaning as the entities themselves, for example, in areas as diverse as finance, legal, or supply chain management.

A reasonable approach is to begin with vector search and add a knowledge graph once relationship-based reasoning becomes a core requirement rather than an occasional edge case.

Resources

If you want to go deeper on GraphRAG itself: the concept was popularized by Microsoft Research in From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Edge et al., 2024), with an accompanying open-source implementation at microsoft/graphrag. For a broader view of the field, Graph Retrieval-Augmented Generation: A Survey (Peng et al., 2024) covers the range of approaches beyond the pattern used here. Neo4j also has a practical, framework-agnostic tutorial on building a RAG system on a knowledge graph.

Kedro is an open-source project supported by a full-time development team. It is part of the LF AI & Data Foundation, with a neutral, vendor-independent governance model that supports long-term sustainability and community-driven development. Beyond the core framework, Kedro is supported by an active open-source community. Contributors and users share patterns, extensions, and real-world experience. A growing list of blog posts, videos, and projects that use Kedro can be found in the awesome-kedro GitHub repository.

Kedro is also backed by extensive documentation. This includes tutorials, example projects, coffee chats, and code snippets ranging from introductory features to more advanced use cases. Teams can get started quickly with a ready-to-use project template, which includes sample datasets and enables users to train their first model with just a few commands from the terminal. If you check it out, drop us a comment below to let us know how it goes!

This article was authored by Laura Couto and Alice Cima with additional insights from Bruce Philp and Joseph Perkins. The editor was Jo Stichbury.

Building production-ready GraphRAG systems with Kedro was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our recent blog post, “When everyone can build, design is the differentiator”, we introduced QuantumBlack Design System (QBDS). We made the argument that an AI-native design system is more than a component library and its purpose is as an infrastructure that lets agentic developers make fewer guesses. The article described the system and our philosophy.

Over the past few weeks, we have re-platformed the entire UI of one of our products onto QBDS, one that previously had a production dashboard built on an older shadcn layout that predated QBDS. In numbers, it runs to 21 pages, 13 modals, 40 UI primitives and somewhere around 96 supporting components. A refactor of this size used to mean months of redlines and handoff documents. We’ll describe what we discovered from using QBDS, both from the engineering side and the design side.

The setup

We used a small stack since QBDS supplies the components, the tokens, and the agent guidance described in the earlier QBDS article linked above. We used Figma to hold the target designs, and Claude Code for the implementation, connected through the Figma MCP server. We crafted an internal QuantumBlack front-end developer skill to package QBDS guidance in the form an agent can act on. That skill lives on Brix, our inner source platform for sharing artefacts.

The workflow

Our front-end engineer runs a page migration roughly as shown in the diagram.

1. Spin up the app locally with hot reloading.
2. Give Claude the context: the PR, a screenshot of the current screen, the target Figma node, and the skill.
3. Instruct the agent to ask when something isn’t trivial.
4. Review every change as it lands live, through hot reload.
5. Run unit tests and lint (Claude fixes most failures itself), then draft the PR, run CI/CD, and run manual Playwright E2E tests on a local minikube.
6. Use a review bot initially, then have a human review the result before the merge.

Step 3 is more than a throwaway. For example, when Claude migrated the Secrets page, the Figma node left two questions open: how row actions should render, and what to do with two modals the design didn’t cover. Rather than making a guess, it set out the gaps, weighed each option against the pages we’d already migrated, and asked for clarification. We answered in two lines and it then completed the build.

The agent noticed that one option “diverges from other migrated tables,” which means that the agent had to know what the other migrated tables do. This knowledge came about because QBDS encodes patterns and relationships, not only components, which is the benefit of using QBDS over a plain component library.

In time terms: a simple table page with an existing sibling pattern took 10 to 15 minutes end to end, with tests and build included. A complex screen from scratch took 4 to 5 hours. If we had worked on this migration by hand, both would have taken days.

What the design side measures

Our designer’s comparison is against a similar product we shipped last year, before QBDS or any of this tooling existed:

• First-pass fidelity to Figma: around 20 percent previously, 70–80 percent now we’re using this workflow.
• Review rounds per screen: four or more previously, and the result often still wasn’t right. One or two now, and the result is correct.
• Component reuse: we now reuse about 90 percent from stock QBDS components. The other 10 percent is custom, but assembled from QBDS micro-elements as a starting point.
• QA and handoff: three-plus hours per screen previously, much of it spent writing Figma documentation so engineering could interpret intent, and we sometimes also needed a meeting to decide. With the new workflow we now need twenty minutes to an hour. This is a key result because previously handoff was a translation problem from human to agent. Now we can use QBDS, the system documentation is already readable by the agent, so most of the laborious translation work is now unnecessary.

The broader role of a designer

It’s difficult to capture this in a metric, but ultimately, designers on our team don’t live in Figma any more. We’ve expanded our AI skills and ways of working, so depending on what we’re trying to solve, that might mean working in Figma, Cursor, or Claude Code, or moving between them. The choice of tool becomes more intentional, based on what each part of the problem actually needs.

With QBDS in the codebase, we can work against the same system the product actually ships with. Through the Figma MCP, our QBDS skills and Code Connect, we carry context between design and code instead of recreating it at every step. When something is easier to explore visually, we work in Figma. When we want to experience it in the real product, we move into code.

That changes the traditional design-to-development handoff. With AI, designers can take an idea further, from exploring the experience to composing it with real components, implementing the front end, and experiencing the design in the actual product.

AI doesn’t get everything right by default, and the quality of the result depends heavily on the context and tools we give the agent. Explicitly calling the relevant QBDS skill and verifying that the agent is using the MCP correctly can make the difference between something that only looks close and something that follows the design system.

The difference became clear when we ran the same task twice: recreating the product experience in Figma with an agent. In the first attempt we didn’t explicitly call our QBDS skill. In the second, we called the skill and verified the MCP usage. The improvement in component usage, icons, styles and overall fidelity was immediately apparent. The full prompt sequence behind this exercise is in our prompt playbook.

The new design flow

The workflow is much more flexible for us as we can move between design and code depending on the requirements.

0. Feature request. The starting point varies. A product manager might create a ticket, a designer might identify an opportunity, or a developer might already have a working draft that needs UX refinement.

1. Ideation and prototyping. We move between the Figma MCP, QBDS skills, QBDS Code Connect, Cursor, Claude Code and Vercel. Ideas can start directly with AI and quickly become working prototypes. When something is too complex for the AI to resolve well, we design it in Figma and bring that context back into code through the MCP. With Vercel, we can also create shareable versions early and test ideas with stakeholders before treating them as finished.

2. Code to Figma, and design refinement. Sometimes the working prototype becomes the starting point for deeper design refinement. We bring it back into Figma, where we can work with more precision on the parts that need it, while keeping the experience connected to the same design system.

3. The new design handoff. The handoff is no longer necessarily a finished set of screens in Figma. Depending on the feature, we might keep the key components in Figma while handing over a working prototype in a dedicated repository or branch.

Figma remains important for giving designers visibility and control over the components we already have, and a place to explore specific design changes. The working implementation carries those decisions closer to the actual product. Engineering then brings the deeper code expertise to validate the implementation, and make sure what ultimately ships is robust and production-ready.

The design system becomes the shared medium and, with AI assistance, directs the flow from intent to implementation, improving throughput along the way. As one product owner in QuantumBlack Labs explains,

“Before, we would use designers to prototype, then front end engineers, when we could get them, to build. Now, the agentic developer builds the actual code that will be used as part of the actual feature. We are all more efficient.”

Humans stay firmly in the loop

• Hidden style bugs. There are always a few, like a kebab menu with a mysterious white border inherited from a parent. Claude fixes them reliably once a human spots them and points at the exact component. Early screens: 5–6 per page. Now that shared components are migrated: 1–2.
• Interaction states. Static elements are easy, but anything with hover, focus, or conditional states still needs a designer’s eye.
• Functional regressions. A UI update can silently drop behavior, not just pixels, and a screen can look exactly right while a function has gone missing. So beyond the test suite, we explicitly verify that existing functionality survived the redesign.
• Tests. Nearly every migration touches them. A failure means either dropped functionality (a real bug) or stale selectors (housekeeping), and distinguishing between them requires a human.
• Component duplication. An agent hyper-focused on one task can overlook components that already exist and quietly build a second version. We track what gets implemented during design updates, refactor periodically, and document the frontend architecture in a reference file the agent reads (CLAUDE.md, in our case) so it knows what's already there.
• E2E stays manual: Playwright, local minikube, human at the keyboard.

Our learnings

We learned quickly to scope our prompts to what exists: whole pages when a sibling pattern exists, chunks when it doesn’t. It also became apparent that we needed to explicitly instruct the agent to ask when something was non-trivial.

Other practical learnings included keeping hot reload running so that verification was continuous rather than an event. Finally, we learned to treat test failures as a useful diagnostic rather than a set of chores.

The following are our key takeaways:

• An AI-friendly design system made the process faster and the output better. Screens that took 2+ weeks now take hours to days, and land closer to spec.
• Bugs that took 2 days to resolve (5 days at the worst points last year) now get fixed in minutes, because the agent can trace a style to its source.
• The highest-value design work this year was in making the system itself unambiguous, because every product and every AI tool inherits its clarity, or its mess.

Open source for reusability

Apart from our internal skill, everything described in this article, from the design system, the agent guidance baked into it, and the runtime whose UI we rebuilt with it, is code you can clone today.

That’s a deliberate choice because here at QuantumBlack, AI by McKinsey, we work across many clients, industries and brands, so anything we build has to prove itself beyond a single product. Open-sourcing QBDS is how we roll: the system has to be clear enough that either a human or an AI agent can pick it up without us in the room. The agent can only be as good as we teach it.

You don’t have to take our numbers on faith. Installation details for QBDS are in the QBDS documentation, so why not give the design system and runtime a try for yourself? Leave us a note in the comments if you do to let us know how you get on.

Authored by Natalia Kurakina, with contributions from Péter Kismarczi, Carolina Staudinger, and Joseph Perkins. The editor was Jo Stichbury.

What changes when your design system can talk to an AI? was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data estates rarely become complex by design. They grow through acquisitions, new platforms, local workarounds, and repeated efforts to connect, consolidate, or retire legacy systems. While such changes are often necessary, each one opens the door to duplicated data, undocumented transformation logic, and inconsistent definitions. Even when individual datasets are high quality, teams may no longer know the lineage of data, that is, where a value originated, nor how it changed between systems, or which version should be treated as authoritative.

Most enterprises still build lineage manually. Teams piece it together from institutional knowledge, often relying on the few subject-matter experts who remember how a dataset was assembled. Because documentation is frequently incomplete and becomes outdated quickly, analysts spend disproportionate time finding and validating data. Projects are stalled, data is duplicated rather than reused, and different teams repeatedly investigate the same relationships.

Everyone has a reason to invest in data lineage

Until recently, the firms that invested seriously in lineage were regulated firms such as banks, insurers, and healthcare organizations, and they did it because their regulators required it. Since 2020, those sectors have paid more than $1.4B in finalized regulatory penalties, often due to deficient monitoring and reporting frameworks, and failures to meet standards such as BCBS 239. The response was to throw people at the problem. In complex data environments, tracing the path of a single element from report to data source can take weeks, largely because the relevant knowledge is scattered across ETL tools, data catalogs, and the memories of individual employees. Firms outside regulated industries looked at that cost and reasonably decided they could live without lineage.

Unresolved lineage stands in the way of both regulatory compliance and the cost reductions that come from a modernized legacy data estate. The scale of the underlying spend explains why: procuring data sources and maintaining data governance can account for roughly a fifth of total IT spend at a large institution, with more than half of that going toward data controls and manual governance activity alone.

By improving lineage and reducing the effort required to trace, validate, and remediate data issues, organizations may be able to realize an estimated $30–100M in cost savings. The harder-won benefit is trust: executives get a defensible reason to believe the dashboards they already use and the agents they are being asked to rely on.

An AI agent asked to answer a question from enterprise data has to know which source is authoritative and what happened to a value between systems. Without lineage, the agent either guesses or the organization refuses to trust what it returns. Every sector deploying agents now needs lineage across a far wider slice of the estate than a regulatory report ever covered.

Manual lineage can’t keep pace

Most lineage tools available today are point solutions which trace how data moves within a single system, leaving organizations struggling when faced with more complex enterprise estates. It’s not unusual for an estate to comprise dozens of systems, built years or decades apart, on a wide variety of legacy platforms and programming languages with undocumented logic. This helps explain why up to three-quarters of organizations still rely on manual work to trace the lineage of their most critical data, an incredibly slow and expensive process. Manually tracing a single data element end-to-end across legacy and cloud systems typically has a direct cost of one to three million dollars.

Recent advancements in AI that have emerged over the past year enable the interpretation of code and metadata alongside the raw data itself, detecting connections across any environment or platform. These automated tools can complement the lineage tools organizations already have, closing gaps between them and accelerating those tasks that used to require an army of analysts.

This rest of this article explains how AI for data lineage (AI4DL) from QuantumBlack, AI by McKinsey, addresses a shared bottleneck by accelerating lineage creation, to give organizations a clearer view of where data comes from and how it changes across systems.

AI4DL accelerates one core task across four common use cases

AI4DL is an accelerator that combines AI-based detection methods with human validation to identify complete, end-to-end lineage across whatever mix of systems and technologies a client happens to run. It’s designed to capture lineage faster and more completely, engendering greater trust than any manual approach can.

AI4DL is an accelerator for creating lineage output that is reusable, and provides many different types of business value, depending on the use case.

• Regulatory traceability: AI4DL gives compliance teams real and up-to-date visibility into how data flows, transforms, and depends on other data across systems.
• Improved data and business understanding: AI4DL exposes origins, dependencies, transformations, and consumption patterns across the data estate.
• Migration readiness: AI4DL accelerates the mapping between legacy and modern data models, cutting the effort required for platform and code migrations.
• AI and analytics implementation: AI4DL gives AI agents validated context for finding, interpreting, and using enterprise data.

Together, these benefits build a level of trust that enables a regulator, a migration lead, and a data engineering team to point to the same underlying lineage graph and trust it for their own purposes, rather than maintaining separate, often incomplete versions of the truth.

Productivity is the shared economic benefit across all four use cases. By reducing manual lineage documentation, investigation, and validation, AI4DL frees data, engineering, governance, and subject-matter experts to focus on higher-value work.

AI detection and human checks

AI4DL works by running various detection methods in parallel, each tuned to a different signal in the data estate:

• LLM-based metadata inference reads data catalogs (table and column names, descriptions, constraints, tags, owners) to infer likely lineage.
• LLM-based code review parses ETL and ELT code, from SQL to PySpark to DAGs, to derive column-level lineage and summarize what each transformation actually does.
• Statistical attribute matching suggests column similarities between different datasets using data profiling and distribution similarity.

The same approach used for a single COBOL program, for example, can be scaled to two thousand or more, with the same degree of accuracy and trust.

Whatever combination of methods applies, the results are never taken at face value. The tool flags low confidence decisions and human experts review the resulting lineage graphs, so that they flag inconsistencies before anything is treated as fact. This human check is what allows AI4DL’s output to be treated as audit-grade rather than merely directional, a critical distinction in regulated industries.

The results

Example: A major Latin American bank pursuing an AI-first strategy was being held back by data quality issues with ten years of real-time credit card and investment data, more than 34 billion rows spread across 2,000-plus COBOL programs and 50-plus DB2 tables. By decomposing that metadata and code into end-to-end lineage and layering AI models on top, the bank cut lineage creation time from ten weeks to a few hours, freeing roughly 50,000 combined hours for its data governance and engineering teams.

Example: A European insurer faced a Databricks migration that was stalled by a lack of documentation on more than 100 legacy SQL scripts and the departure of the people who understood them. AI4DL parsed the scripts and traced their dependencies, validating the resulting lineage against manually produced lineage to reach about 98 percent accuracy, de-risking a migration that had been stuck for months.

Example: A European bank faced a data migration with an original estimated effort of five people working full time for one year. It used AI4DL to automatically map about half its target features, for a 65 percent efficiency gain against the original baseline. Confirming those source-to-target mappings, which ended up being 70 percent correct, took a subject-matter expert roughly a quarter of their time over one month. This saved the bank months of time and effort from the specialist team the bank had originally budgeted for.

In each case, the win wasn’t simply speed. It was that everyone involved, whether at the bank, the insurer, or on the regulatory side, could point to a specific, auditable trail for how a number in a report traced back to its source, something none of them could do with confidence before.

AI4DL doesn’t require a rebuild

AI4DL runs across source systems, whether mainframe, on-premises or cloud, making it possible to implement without requiring a client to rebuild their technology stack. Deployment can be cloud-native, hybrid, or on-premises, expanding to match whatever architecture a client already runs. The solution traces how fields are derived, transformed, and consumed, so teams get one consolidated map instead of disconnected documentation per system or layer.

Start with one use case, prove the value, scale the recipe

Teams that get data lineage right unlock benefits well beyond the initial use case, and provide a genuine foundation for the AI initiatives that data problems have been quietly undermining. The technology behind that shift has been proven at scale across financial services and other regulated sectors, delivering results in a fraction of the time required by manual methods.

The right way to start is with a pilot focused on a single domain or product. Clients can enter through whichever use case has the clearest current economics, such as a regulatory report, a difficult data domain, a migration wave, or a bounded agent workflow. Ideally this is in a domain where success can be measured and the approach refined before being codified into a repeatable recipe. Once that recipe works, scaling AI-driven lineage across an entire enterprise becomes a matter of repetition rather than invention.

AI4DL makes lineage something that can be created quickly and reused across the work that depends on it, from regulatory reporting to data migration and AI use cases. It creates a foundation that supports a variety of business use cases, including regulatory and business reporting, data migration, and agentic AI implementation.

When lineage is no longer a slow, manual effort, it stops being a one-off exercise and becomes part of the infrastructure teams rely on day to day.

The broader AI4Data portfolio addresses the full data lifecycle and has now been deployed with clients more than 100 times. Find out more from QuantumBlack’s ongoing series on data transformation that showcases products in the AI4Data portfolio, and from the following articles:

• AI4Data Quality: The product cleans data using a customizable AI based toolkit that can detect and correct data quality issues.
• AI4Data Quality Unstructured: An extension of our AI4DQ product, this asset uses a customizable AI based toolkit to detect and correct issues in unstructured datasets.
• AI4Data Products: The solution generates ETL/ELT code automatically from table schemas, cutting engineering time by up to 80 percent and embedding best practices in the generated pipelines.
• AI4Data Extraction: The product transforms un-structured documents (PDFs, PPTs, PNGs) into re-usable structured data assets for downstream use in agentic workflows

Do you work on a team that regularly grapples with issues of data lineage? We’d love to hear about your experiences, challenges, and solutions in the comments. And if you’d like to know more about AI4DL or hear relevant use cases in more detail, you’re invited to get in touch with us directly.

Authored by Bridget Bassi, Andreea Popa, and Alan Conroy, with additional input from Kayvaun Rowshankish, Sebastian Schöbl, Nikhil Rao, Nishant Kumar, Dana Mekler, Esther Leong, Season Yang, and Filipe Oliveira. The editors were Carl Alviani and Jo Stichbury.

Solving the data lineage problem that’s holding enterprise AI back was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here at QuantumBlack, we’ve had an internal design system for a while. It’s powered multiple products and gives our development teams a consistent language to help them ship faster. But it was built for a different era. The original architecture worked well for designers and developers, but it wasn’t set up for AI-assisted development. Now we’ve rebuilt it to make it easier for both human team members and AI agents to use, customize, and extend.

We are sharing part of the new design system infrastructure as an open-source project, giving our clients and other external teams a practical way to adopt it in their own products.

In this article, we explain what makes a design system AI-native, share what we learned rebuilding ours on shadcn and Tailwind, and describe where we think design infrastructure goes next.

Our belief: the design system is where design scales

AI makes interface implementation faster and more widely available. That puts more weight on decisions generation alone cannot make: whether an interface works, whether it is appropriate for its users, and whether it should be built at all. Good judgment, critical design thinking, taste, and trust have become more important than ever.

If design is the differentiator, then a design system is the way to scale it. In the past, design systems were mainly component libraries that kept products consistent. Today they’re more fundamental as a shared infrastructure that captures an organization’s design intelligence and makes it available to every product, every team, and, increasingly, every AI tool.

The best design systems enforce consistency and help humans and AI build better products and solutions together.

Unlike most design systems, ours isn’t built for a single product or company. We deploy design and code across a variety of McKinsey clients, industries, and brands, which means our system has to scale. It has to capture design decisions, foundations, principles, guidelines, structure, and intent that work in different contexts.

We’ve spent years on the closely related design problem of explaining AI/ML reasoning to people so they trust it. Building an AI-native design system reverses that challenge, so that we are now teaching AI to understand and apply human design judgment.

Design infrastructure, not a component library

An AI-native design system is more than a collection of components; it includes the infrastructure that helps AI build the right thing. That means going beyond reusable UI to defining a system that encodes design values and the intent behind them.

Every component, token, guideline, and connection between design and code must be explicit. The less AI has to guess, the more consistent, explainable, and production-ready its output becomes. A clean, well-structured, well-documented design system is now one of the highest-value investments a design team can make, because every AI tool in the organization inherits its quality…or its mess.

This benefits people too. The process of making a system understandable to AI forces greater clarity between designers and engineers, creating a shared source of truth that everyone can build from. That clarity serves the AI as well as every human designer and engineer who has to understand the system, not to mention every user who consumes its outputs.

Good structure also makes AI more efficient and cost-effective. Better structure means fewer iterations, fewer unnecessary tokens, improved output accuracy and consistency, and overall lower cost to reach a production-ready result. We designed for this deliberately, down to letting the system steer work toward the correct technical path for the job, rather than defaulting to a heavier setup that the task may not require.

QuantumBlack Design System brings together design tokens, components, documentation, and agent skills into a single foundation. The more knowledge it captures, the better every product and every AI workflow built on top of it becomes. It is executable knowledge, not just documentation. Agents can learn the principles and use the implementation directly, turning design guidance into something they can both reference and reason with.

We treat agent guidance as part of the design system. Alongside design tokens and components, we package the project context, implementation guidance, and reusable capabilities that help agents navigate the system consistently. Rather than relying on handcrafted prompts, agents inherit the same shared knowledge and conventions that our teams use.

What we learned along the way

Building a system for AI means making different trade-offs from those we would have made a few years ago. These are best summarized through a series of principles we used to guide our efforts:

• Build with AI, not against it: Instead of maintaining a fully bespoke component library, we built on foundations that AI already understands, layering our design system on top. Rather than building custom components from scratch and sharing npm packages which are hard for LLMs to read and edit, we rebuilt our design system on shadcn and Tailwind and incorporated our design intelligence alongside it.
• Composition over configuration: Not every product should look the same. Rather than rigid, feature-rich components, we rebuilt around composition and flexible slots. We created smaller, modular pieces for people or AI to assemble as necessary for different use cases.
• Structure over documentation: We found that richer, more explicit tokens that capture design intent, guidelines on component relationships, and usage rules were more valuable than an exhaustive catalog of components. AI performs better and reasons more effectively when knowledge is defined and structured well.
• Consistency with room to adapt: The system needs to serve different use cases and users. We built different levels of flexibility into the system so AI can distinguish between these use cases. For example, quick experimental offline prototyping and production-grade application work are two different tasks running on the same system.

What’s next? 🔮

QuantumBlack Design System is the start of a new way of building products here at QuantumBlack Labs. Everything we learn becomes a replicable playbook we can bring to our clients.

As models improve, we expect some of today’s scaffolding to become unnecessary. We will continue testing which design rules agents need explicitly, which they can infer reliably, and where designers and engineers still need tighter controls. The interesting work for AI-era designers and engineers is to keep finding the next novel combination.

Find out more about QuantumBlack Design System from our open-source GitHub repository and learn how to install it from QBDS documentation.

Authored by Stephanie Kaiser, with contributions from Joseph Perkins, Li Nguyen, Dan Dumitriu, Pruthvi Prakasha, Tynan DeBold, Kate Soliman, and feedback from Stephen Xu, Dante Gabrielli and Ottis Kelleghan. The editor was Jo Stichbury.

When everyone can build, design is the differentiator was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

In McKinsey’s recent State of AI 2026 global survey, 89 percent of respondents said that their organizations regularly use AI in at least one business function. Yet only 37 percent attributed an EBIT impact to AI, most of them under 5 percent of EBIT. McKinsey’s “Seizing the agentic AI advantage” report calls the mismatch between widespread use and limited bottom-line impact the gen AI paradox.

A previous article on this blog about enterprise agentic platform architecture provides a useful framing for this imbalance: horizontal solutions such as copilots and chatbots may be easy to deploy, but they’re loosely coupled to core processes and so don’t move the dial. In contrast, vertical solutions embedded into specific workflows can be transformative. For large organizations with the budget and executive sponsorship for AI at scale (itself a non-representative slice of the market), technology is rarely the binding constraint. For these companies, capable models exist, frameworks are mature, and cloud infrastructure is available. What keeps stalling is the move from “interesting proof of concept” to “production system that moves P&L in the right direction.”

This article captures key lessons about the engineering and delivery practices that separate initiatives that move the P&L from those that don’t. We’ll give an engineering perspective on principles that other McKinsey authors have framed strategically and architecturally. It runs in parallel to the six lessons described in the earlier McKinsey article “One year of agentic AI”.

The case studies

Two McKinsey engagements anchor most of what follows. We’ll refer back to them as Case 1 and Case 2.

Case 1: Manufacturing service operations

A global manufacturer working with our team and AWS deployed four AI solutions across an end-to-end service journey, and saw meaningful improvements in key service metrics, with clear P&L impact over the program’s measurement window. Each solution addressed a different step and used a technique suited to the problem: LLMs for unstructured text, optimization algorithms for resource allocation, and ML models for recommendation and prediction, with agentic orchestration over the top.

Case 2: Enterprise agentic platform

A global manufacturing conglomerate, working with our team, has built and launched its first enterprise agentic AI platform at scale, currently released to more than 10,000 account executives and strategy leaders. The platform is the venue through which dozens of agentic use cases are delivered across business units — the architectural side of the “codify and replicate” challenge in Lesson 6.

Both engagements started as proofs of concept, and both could easily have stayed there. We mention them upfront not as a pair of victory laps, but because the lessons that follow draw on the details of their delivery, as well as other engagements where we did not get these things right the first time.

The shape of pilot purgatory

The phrase “pilot purgatory” is one that our McKinsey colleagues have used before, for example in “How COOs maximize operational impact from gen AI and agentic AI”. That article describes organizations developing gen AI tools “that have little effect beyond saving workers a few minutes a day.” We use the term as shorthand for a pattern that’s pointed to by several independent data points, even though each measures something slightly different:

• Project abandonment. Gartner predicts that over 40 percent of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear value, and weak risk controls. The prediction also notes a tendency among vendors to engage in “agent washing,” rebranding existing tools as agentic. Gathered data already points this way: S&P Global Market Intelligence found the share of organizations abandoning most of their AI initiatives rose to 42 percent in 2025, from 17 percent a year earlier, with the average organization scrapping 46 percent of its proofs of concept before they reached production (note that this is self-reported, and “most” initiatives is not “all”).
• POC-to-production conversion. IDC research on enterprise gen AI programs describes a funnel in which a typical enterprise identifies hundreds of potential use cases, launches dozens of proofs of concept, and puts fewer than six into production.
• Maturity. In McKinsey’s Superagency research (a US-weighted C-suite survey), only about 1 percent of respondents describe their gen AI rollouts as “mature” (full source and caveats in “Why knowing what to do is not enough” below).

The metrics are not strictly comparable, but together they describe an industry that is much better at starting AI work than finishing it. The engagements we see exhibit a consistent pattern: a proof of concept lands well, stakeholders are excited, the real work begins, and progress stalls. There are several recurring symptoms:

• The demo-to-production gap. A model that works on curated data but doesn’t survive real-world inputs with incomplete records, inconsistent formats, and edge cases nobody anticipated because the training data was too clean.
• The adoption cliff. A technically impressive system that nobody uses, because end users were not involved in the design and it does not fit their workflow.
• The accountability vacuum. Data science built the model, engineering built the infrastructure, the business unit owns the process, and a vendor manages the platform. So when the system underperforms, everyone points at someone else.
• The “next model” trap. Instead of fixing the deployment, the team pivots to a new model, framework, or vendor. Each pivot resets the clock, and the pilot stays a pilot.
• The skills mismatch. A team can say it “knows AI” because it built a prototype that calls an LLM API. Production AI engineering has a much wider scope, which overlaps with rather than replaces MLOps and traditional software engineering. We return to that distinction in Lesson 5.

These are mostly socio-technical failures rather than algorithmic ones: a mix of workflow design, ownership, executive alignment, and engineering discipline. Better algorithms alone do not solve them.

Our lessons from production

1. Redesign the workflow alongside the model.

This is the lesson McKinsey colleagues have already amplified, and we build on it here. In “One year of agentic AI,” the first of the six lessons is “It’s not about the agent; it’s about the workflow.” The State of AI survey shows that intentional workflow redesign and senior-leader ownership correlate strongly with meaningful business impact, though this is an association across self-reported high performers, not a measured intervention effect. The broader “rewiring” thesis is the subject of McKinsey’s popular technology playbook “Rewired”.

A common mistake in AI deployment—and one we have made ourselves more than once—is treating it as a technology insertion problem. The team takes an existing process, bolts an AI model onto one step, and expects transformation. This is the horizontal-solution pattern called out in a recent QuantumBlack platform architecture article, in which copilots and chatbots ride on top of existing workflows, making them easy to ship and scale, but leaving them loosely coupled to the process they are meant to improve. In Case 1, the obvious horizontal approach would have been a single chatbot frontline teams could query when stuck: a reasonable pilot, but one that provides just a fraction of its potential impact. Our QuantumBlack colleagues reached the same conclusion from the software-development angle in Agentic workflows for software development. Agents earn their value only when they run inside redesigned, convention-driven workflows rather than bolted onto the old ones, and doing that well “requires organizational change that goes beyond technical tooling.”

In Case 1, we worked with the client to redesign the end-to-end service journey and put four AI solutions into production across it. On its own, no single solution would have moved the needle. Together, in the client’s reporting, they were associated with the service-metric improvement and P&L impact noted earlier. We treat those as program-attributable rather than a clean causal claim. AI was one factor among several, and the client’s own teams did the integration work that made the savings stick. This pattern appears in the research too: enterprise-level benefit shows up more often alongside vertical, workflow-embedded redesign than horizontal point automation.

Internal AI teams often struggle to break out of pilot mode because they are usually organized around a single function or technology layer. Seeing the full workflow and having a mandate to redesign it from end to end requires a cross-functional perspective and executive sponsorship that pilot teams frequently lack. The problem is not the teams so much as an operating model built for tools rather than transformation.

The teams that ship redesign the whole workflow, not just the step where the model sits.

2. Measure in P&L rather than accuracy.

Model accuracy is necessary, but it’s not the full story. A 95 percent accurate model that nobody uses delivers no business value; a lower-accuracy model embedded in a workflow people actually follow can move an operation. The shift that matters is from “does this model work?” to “does this system move the business?”

In Case 1, the C-suite tied a portion of their own performance targets to the AI program’s outcomes. In our experience this is a stronger forcing function than any technical KPI. Optimization arguments become business arguments, and trade-offs are resolved in days rather than quarters. We do not claim the incentive structure caused the results, but it visibly changed the conversation.

That kind of alignment might sound obvious, but it’s also rare, and the State of AI 2026 survey suggests that the organizations capturing value are the ones that instrument for it. AI high performers — the roughly 6 percent of respondents who attribute at least 5 percent of EBIT to AI and report ‘significant’ value from it — are about twice as likely as others to say their organizations have defined processes to measure the impact of their AI initiatives. That is correlation, not proven causation, and it’s important to note that high performers are doing several things right at once, with practices that reinforce one another. But the correlation matches our own observation that teams who successfully move the P&L tend to be the ones measuring in P&L terms from the start.

Many pilots are still measured in technical metrics (F1 scores, latency, throughput) that the business cannot easily act on. Show accuracy charts to a CFO trying to understand cost-to-serve and the pilot gets another quarter of funding without anyone learning anything new. Closing that gap takes someone who can translate technical architecture into financial outcomes for the CFO, then turn those outcomes into system requirements for engineering. Many organizations have people who can do one or the other, but the combination of skills is rarer. This becomes especially important in the second or third quarter when sponsorship is renegotiated.

The teams that ship measure success in business outcomes and translate P&L goals into engineering requirements.

3. Design for adoption.

The hardest engineering problem in AI deployment is getting people to use it well, and this is hard to get right on the first attempt. In an earlier engagement we assumed operators would adopt a separate tool once we showed what it could do. In practice, usage barely moved. The pivot that mattered was rebuilding an integration so the AI’s recommendations appeared inside the tools they already used, where following a recommendation took fewer steps than ignoring it. Usage increased sharply over the following months.

In Case 1 we built that lesson in from the outset, and the four solutions reached high active-usage rates across the targeted frontline population within the measurement window. We do not put a single number on that outcome here, because adoption-rate definitions vary by tool, role, and time window. The result was not accidental because adoption was treated as a first-class engineering requirement. Our experience suggests a few key principles:

• Build alongside end users. Frontline operators were in the room during design. We watched them work and mapped their tools, frustrations, and workarounds, then designed the systems to fit the workflow they already had rather than replace it.
• Optimize for trust, not just accuracy. Operators with decades of experience rarely trust a black box, so the systems show their reasoning: why an action was recommended, what data was used, the confidence level. Transparency is an engineering decision, and a costlier one than it looks. It has to be built into the model interface, evaluation harness, and UI in a coordinated way.
• Ship incrementally. Each system went live independently, proved its value, and built confidence for the next. By the time the last launched, the organization had already internalized AI as part of how it works.

Adoption, measured properly, is a form of behavior change. The test we look for is whether people reach for the AI over their existing process when both are available, and keep doing so once launch attention fades.

The teams that ship design for adoption with end users and fit AI into the tools they already use.

4. Own the outcome end-to-end.

Pilots stall most often where ownership is fragmented. When the model is built by one team, the infrastructure by another, the frontend by a third, and integration managed by a vendor, nobody owns the outcome and decisions get relitigated. At scale, one engineering group needs to own enough of the stack — architecture, code, infrastructure, and the technical relationship with the client’s governance functions — to defend the design end-to-end. Not everyone must build everything, but someone has to be answerable for the whole picture, and reaching that arrangement often requires reorganization that can be uncomfortable in the short term.

At enterprise scale, technical governance reviews are inevitable: security teams audit the architecture, compliance questions data flows, senior stakeholders ask why one approach was chosen over another. The team that can speak credibly to architecture, coding standards, test coverage, and security practices retains trust. The team that defers to a vendor, or escalates to someone who was not in the room when the decisions were made, quickly loses it.

We have seen what this looks like when it goes wrong. In one earlier engagement we accepted what looked like a clean split, with a vendor owning the inference layer, a partner the data pipeline, and our team the application layer. In practice, though, every governance review revealed an ownership gap: a security question that needed three answers, an integration bug that needed three reproductions before anyone could diagnose it. The program was stalled not by technology, but coordination. The uncomfortable fix involved re-scoping to take operational accountability for the full stack, holding the vendor and partner to a defined interface as upstream suppliers rather than parallel owners. That broke a few expectations, but ended the governance stalls within the next quarter.

One note, for transparency: McKinsey supports its clients with exactly this kind of integrated capability, but there are legitimate alternatives, such as investing in platform teams, hiring senior AI engineering leadership, building deeper hyperscaler partnerships, or setting up a structured handover from a consulting partner into in-house ownership. Indeed, many of our engagements are completed this way. The point is that someone, internal or external, must own the architecture, the code, and the governance conversation as a single point accountability, rather than three.

The teams that ship give one engineering group clear accountability for the architecture, code, infrastructure, and governance relationship.

5. Staff for production AI engineering.

Production AI engineering is not a wholly new discipline. It overlaps heavily with mature MLOps and production-grade software engineering, and most of the strongest practitioners we work with come from one of those backgrounds. What is different is the surface area that LLM-based and agentic systems introduce on top:

• rigorous evaluation and output validation at every step
• model selection accounting for cost, latency, and capability constraints
• streaming and retry patterns tuned for token-by-token responses
• observability
• migration paths when the underlying model changes every few months.

Instincts about systems differ because AI failure modes are not the same as those in deterministic software, nor in classical ML pipelines.

Two recent QuantumBlack pieces map this concretely. “Generative AI workflows need engineering discipline to scale beyond the demo” argues that what breaks in production “is almost never the quality of the underlying model, but the way the system is structured,” and treats reproducibility, observability, and evaluation as first-class properties, not afterthoughts. “Agentic workflows for software development” shows the same discipline at the level of a single workflow: bounded agent execution wrapped in deterministic orchestration, with automated evaluation at every step. The engineers who do this well treat those practices as the job, not as overhead.

Teams that staff for “software engineers who have used an LLM once” tend to discover this the hard way, in production. Many are able adapt if given time to learn the new failure modes, with support from a senior practitioner who has shipped LLM systems at scale. But the adjustment happens somewhere, and a pilot is a cheap place to learn it. A high-visibility production rollout is not.

A related challenge is domain expertise. Engineers can write prompts, but they are rarely best placed to know what a good output looks like in a given business context. A subject matter expert can tell you whether a recommendation is reasonable for an aftermarket service call, but is not usually the right person to build the evaluation harness around it.

Domain experts take charge of prompts, evaluation criteria, and output validation; engineers scale and harden what works. Treating prompt engineering as a form of software engineering, rather than a domain task with engineering infrastructure around it, is a common and costly mistake.

The teams that ship pair production AI engineers with domain experts who own prompts, evaluation criteria, and output validation.

6. Codify and replicate.

Production is not the finish line, and AI projects are successful only if they keep working after they are deployed. We recently described this dynamic in our article about live AI operations, “AI value: The growing AI operations challenge”.

The real value multiplier, in our experience, is turning a single successful deployment into a repeatable approach. “One year of agentic AI,” cited in Lesson 1, puts a sharper version of this in its fifth lesson: “…the best use case is the reuse case.” Companies often build a unique agent or pipeline for each task, when the same components — ingesting, extracting, searching, analyzing — recur across many tasks and can be assembled from parts already built.

The most common first instinct is documentation, in the form of detailed pages on architectural decisions, integration choices, and lessons. But a few months later, few outside the original team have read it, and the next engagement rebuilds the same things from scratch.

What works better is packaging the decisions as runnable scaffolds — template repositories, evaluation harnesses, integration adapters for a new team to clone and adapt — with a much shorter narrative on what the patterns are and when not to use them. Engineers are far more likely to reach for the code their colleagues have already written than the documentation about it. In Case 1 we codified the most reusable artifacts as runnable assets with a thin narrative wrapper: architectural patterns for multi-agent orchestration over enterprise data, integration approaches for common back-end systems, the frontline adoption playbook, and the governance model used to clear internal risk and compliance.

The approach has since shortened the path, through people and patterns more than transplanted code. With an overlapping team, the disciplines from Case 1 — a habit of building evaluation harnesses, orchestration conventions, and shared McKinsey engineering tooling — carried into Case 2 and cut time-to-first-deployment for new use cases, even where little of the implementation transfers directly. Case 1 is a reference point for later teams to start from, rather than a blank page. The recent QuantumBlack platform architecture article frames this as building for composable and compostable assembly: modular blocks that combine into complex systems and can be replaced without redesign. That is the bar that Case 2 is being built to meet.

There is a difference between building a single instance and building a capability. Under delivery pressure, AI teams usually do the former: ship, declare victory, start the next project. Codification asks for abstraction, time spent on assets that will not directly benefit the current engagement, and an honest retrospective about which parts of the last build were actually transferable and which were incidental. That last part is the hardest, because it asks the engineers who built something to argue against parts of it. It is easy to skip, yet it is the difference between a team that delivers projects and one that builds a capability the next program can stand on. Knowing what to codify comes from having shipped across industries and use cases more than once.

The teams that ship turn reusable decisions into runnable scaffolds that later teams can clone and adapt.

7. Build business-led capability.

To escape pilot purgatory, build AI as a capability that the business owns and runs, instead of a finished system handed over to it. Lesson 5 argued that domain experts, not engineers, should own prompts, evaluation criteria, and output validation; this lesson is about the operating model around them.

In the programs that scale, business and domain leaders take ownership of the product build, setting requirements, shaping and refining prompts, and validating successive versions of the output so that hard-won domain expertise is encoded in the system rather than approximated by engineers.

Sustaining that at enterprise scale means investing in AI literacy across a broad base of business staff, so they can work shoulder-to-shoulder with technologists instead of lobbing requirements over a wall. In our experience this business–technology divide is where many programs quietly stall, and the organizations that succeed are dissolving it deliberately. The roles blend over time, and the capability to run AI becomes the enterprise’s own rather than a vendor’s or a central lab’s. This is the least engineering-centric of our lessons and, for that exact reason, the one most often left to chance.

The teams that ship make AI a business-owned capability, not a system handed over by technologists or suppliers.

Why knowing what to do is not enough

None of these seven lessons is a secret, and most technology leaders could read the list and nod. The lessons are facets of an integrated system, and each depends on the others, which is a point that Rewired and State of AI make at the strategic level. We restate them here, but from an engineering and delivery perspective.

Workflow redesign needs executive sponsorship, which needs P&L-linked KPIs, which need enough understanding of the workflow to know what to measure.

Adoption depends on trust, which depends on owning the governance relationship, which depends on a production AI engineering skill set that goes beyond “has used LLMs.”

Codification depends on having shipped enough to recognize what generalizes.

And underneath all of them sits business-led capability: the domain ownership and AI literacy that let an organization own this itself rather than depend on outside help.

The industry data broadly supports this, with some caveats. In McKinsey’s US-weighted Superagency in the workplace survey (which includes 238 C-level executives), only one percent of C-suite respondents describe their gen AI rollouts as “mature” (fully integrated into workflows, driving substantial outcomes). Another striking number, albeit a US-heavy one based on self-reporting against an ambitious bar, comes from the global State of AI 2026 survey, which shows 44 percent of overall respondents scaling AI (up from 38 percent the previous year). Among organizations with over $1B+ in revenue, the share scaling agentic AI increased from 27 percent in 2025 to 40 percent in 2026.

Cost is a material part of why so few get there. Gartner’s analysis of gen AI deployment approaches puts deployment at US$5–20 million for business-model-innovation cases (a gen AI category, not an agentic-specific figure); we see comparable ranges by analogy, well above most pilot budgets. Outside the largest revenue tier, that shifts where the bar for “worth doing at scale” sits, and why staged investment with frequent value checkpoints matters.

The organizations we have seen escape pilot purgatory rarely do it with a single existing team on its own. They combine three things: an internal core that owns the AI agenda over time, external partners (whether consulting, hyperscaler, vendor, or open-source) who bring in patterns from other deployments, and an explicit plan for bringing the capability in-house as the program matures. The cost of learning by trial and error often dwarfs the cost of borrowing pattern recognition from outside. We support clients with one version of that outside help, but the broader claim does not depend on who supplies it.

In summary

This article explains the gap between AI’s promise and its production reality and why, more often than not, it comes down to delivery rather than to better models or newer frameworks. The seven lessons from teams that ship are as follows:

• Redesign the workflow alongside the model.
• Measure in P&L rather than accuracy.
• Design for adoption.
• Own the outcome end-to-end.
• Staff for production AI engineering.
• Codify and replicate.
• Build business-led capability: the business owns the build, and AI literacy is spread across the organization.

Agentic systems, which autonomously plan, use tools, and chain actions across workflows, stress every lesson above. And the bar is rising: the State of AI 2026 reports that 20 percent of respondents have now reached the point of scaling AI agents.

Workflow redesign now crosses system boundaries, ownership blurs across many models and tools, and a mistaken agent takes a bad action rather than just returning a bad answer. McKinsey’s “Seizing the agentic AI advantage” frames that shift strategically and the QuantumBlack platform architecture piece answers it architecturally. Case 2 is our attempt at the “dozens across an enterprise” version, and we expect to write more as it matures.

One fair pushback on the “AI is the easy part” framing is worth examining: AI is not easy in any absolute sense, because building, evaluating, and tuning modern models is hard, and the field moves quickly. What we mean is that relative to redesigning workflows, securing executive alignment, engineering for adoption, defending the architecture in governance, hardening for production, and codifying the result, the model-building part is the more solved element for most enterprise use cases. And that cross-functional work is growing more demanding, not less, as systems become more autonomous and deployments more interconnected — engineering many organizations have not had to do at this scope before.

Are you interested in finding out more about what we’ve learned in McKinsey engagements? Leave us a comment or reach out to us, and follow this publication for further insights, deep-dives, and case-studies.

This article was written by James Mortemore with contributions from Subu Narayanan. It was edited by Jo Stichbury.

Escaping AI Pilot Purgatory: From POC to P&L was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

You’re using AI in production. What keeps it delivering value?

AI projects are successful only if they keep working after they are deployed.

It may sound obvious, but one of the persistent obstacles to realizing value from AI implementation is a tendency to treat its deployment as a finish line, not a starting point. Organizations often invest heavily up front in AI systems, and then expect the same teams and processes that built and launched them to keep them running, if they invest in this stage at all. But running AI solutions reliably, at scale, with maintainable systems and predictable outcomes is a specialized discipline which requires its own expertise, processes, operational cadences, and technology.

To consider what this looks like in practice, imagine an insurer deploying a hybrid workforce with AI agents working alongside humans. The AI handles most customer calls through generated voice and an agentic workflow that decides claim validity, payout, and messaging. At launch, everything looks promising. But then a customer complains that the agent was rude. Another cannot reach the helpdesk. Another receives a payout ten times higher than expected. Another is denied despite a clearly valid contract.

This is the point where hard questions arise. Who or what gets alerted? How much business value is at risk? How do you diagnose root causes across the model, tools, data, and workflows? How do you detect issues before they affect customers? Failure to address these questions leads to problematic issues for the organization, such as reputational risk and customers losing trust.

In agentic systems, the blast radius expands because the system acts autonomously, not just predictively. Many of the most common failure modes may not even show up in demos or test environments. Examples include:

• tools breaking when upstream APIs change
• retrievals that miss critical context as data evolves
• performance degrading as models and workloads evolve
• policy violations that surface in edge cases
• spiking costs as usage patterns shift or platforms refresh pricing
• growing latency as workflows become more complex

These are the problems that Live AI Operations (LiveOps) is designed to solve. LiveOps, developed by QuantumBlack, AI by McKinsey, is an end-to-end AI operations capability that comprises a set of practices to ensurethat AI continues to deliver value in production. The “end-to-end” scope of LiveOps means that it considers all aspects of operating AI post-deployment, with the wide range of actions it drives, including:

• AI performance management and value realization
• monitoring and observability for quality, drift, cost, latency, safety, and business outcomes
• governance, risk management, and human oversight where needed
• production hardening and readiness
• incident response and triage
• remediation through hotfixes, prompt and workflow updates, retraining, model upgrades, and rollback
• controlled model upgrades with change and release management
• knowledge continuity through runbooks and decision records

Put simply, LiveOps transforms a one-time AI deployment into a dependable, long-term service, with clear accountability for how it operates and a reliable path to recovery when things go wrong, while freeing up developer capacity to focus on delivering new product releases on time.

AI systems will degrade without AI operations in place

Even the best-performing AI system will degrade after launch. They may do so quietly over time, or loudly and abruptly, in some cases almost immediately after being deployed. Performance often degrades through small changes that accumulate: data shifts, new edge cases, outdated evaluations, rising costs, and declining response quality. When degradation is abrupt, value can fall off a cliff, for example, due to a sudden, significant shift in data due to a major event or change in operating conditions.

Beyond the technical impact, degradation erodes trust. Users see inconsistent results and revert to old workflows, while stakeholders lose confidence as business value fades. Without an operations discipline in place, teams can end up stuck in a pattern of repeatedly patching systems to try to achieve their original value goal instead of continuously improving them and realizing further value.

Relying on AI builders to handle operations only goes so far

Without LiveOps, your best developers can quickly become full-time firefighters, while users lose confidence and scaling hits a ceiling fast.

Agentic AI makes this even harder. Teams can easily get pulled into unstructured data cleanup, prompt and workflow tuning, fixing tool chain breakages, and repeated validation of what agents are allowed to do. That work is not optional if the organization expects stable performance and responsible AI use in production. The fact that agentic capabilities can be implemented so quickly can also put pressure on teams to speed up release times, often at the cost of stability and without sufficient consideration of the true business value added.

Gen AI and agentic workflows raise the stakes

As AI systems shift from providing model predictions to delivering automated decisions and agentic actions, the number of moving parts compounds. Retrieval, tools, workflows, voice, policies, and external APIs are all needed, and all create more failure modes. Personalization and context open up new kinds of risk exposure related to privacy, bias, safety, and reputation.

When organizations struggle to achieve AI impact at scale, it can often be traced to the gap between what was working in a single pilot compared to what is needed for a sustainable system to run at scale. In production, small changes can have outsized, compounding effects, as systems drift from the conditions under which they were validated.

A number of pressure points commonly show up in production across the full AI stack:

• Agentic AI systems can behave unpredictably as prompts, tools, models, and context evolve. Unstructured data introduces inconsistencies and bias that drive downstream mistakes and create operational drag.
• Maintaining quality, managing latency, optimising costs, and achieving reliability becomes an ongoing operational challenge.
• Even when the technology performs well, poor change management can slow user adoption and limit business impact.
• Organizations need the right skills in place to operate and troubleshoot complex, ever-evolving technologies. Human intervention in tandem with tools and agents remains essential to delivering consistently good output.
• When new privacy, fairness, and security risks arise, ethical and regulatory considerations can suddenly become paramount.

The QuantumBlack LiveOps approach

QuantumBlack, AI by McKinsey, has made advances in this space for years, working with clients across industries and organizations of all sizes to sustain AI value. We’ve developed the frameworks and tooling that makes AI value sustainable and scalable.

Our LiveOps framework is designed as a multi-phase operations approach that overlays on the existing well-established AI lifecycle of Build and Launch → Operate → Maintain and Improve.

During the Build and Launch stage, LiveOps requirements such as scalability and reliability are considered from the start, along with a clear set of readiness validation checks prior to models, agents, and/or workflows being deployed to production.

During the Operate stage, the AI workflow is triggered and runs to produce an output. This is validated from a technical and business perspective (e.g., validation rules, guardrails, evaluations, key performance indicators). Ongoing monitoring with logging and observability is essential for tracking AI performance, costs, and overall business value and surfacing issues which may risk value. User feedback is vital at this stage to understand AI performance and impact and identify areas to fix or improve.

During the Maintain and Improve stage, interventions are implemented to address issues detected during the operate stage and enhancements are delivered to increase value.

Four critical operations elements to get right from the start

To effectively deliver on AI operations goals, it’s critical to start early on before the solution is live. Doing so means focusing on four key elements throughout the lifecycle, starting during the Build phase.

For each element, we’ve provided a brief example from a recent project with a large-scale enterprise client, working to maintain their Gen AI-driven solution, followed by key takeaways.

1. Evaluations

Automated, systematic performance assessments across development, deployment, and post-production; golden datasets; user feedback

Example: We found it was vital to set up a clear process for gathering user feedback and to share this as part of onboarding. Feedback from user evaluations was then applied directly to prompts during finetuning.

Takeaway: Setting user feedback loops early ensures feedback can be captured and used from day one.

2. Architecture, scalability, and tooling

Token limits; disaster recovery; client tech-stack compatibility

Example: We had to scale existing functionality as adoption increased, taking into account tool and plugin constraints which could have required architectural changes. An increase in users brought in more perspectives on the best way to show outputs, leading to further prompt refinement.

Takeaway: When selecting an architecture, take future scale into consideration and plan accordingly.

3. Observability

Logging across all layers; tracking both solution performance and business impact (FinOps)

Example: We used a ‘single pane of glass’ monitoring view to track key metrics, including cost-related business KPIs, failed responses in the application, and query response/load times. This monitoring triggered interventions such as refactoring and architectural improvements.

Takeaway: Identify early what will need monitoring and tracking, so that logging and observability requirements can be built for from the outset.

4. Model changes and reliability

Multi-model strategy; planning for model drift and provider updates; primary/secondary model fallback

Example: We faced issues related to LLM outages that required adopting a multi-agent setup, with a secondary LLM to fall back on. Data drift validation enabled us to identify issues through alerts and address them before the pipeline ran, avoiding any impact for business users.

Takeaway: Plan for LLM availability issues, and decide in advance how to catch problems fast enough to prevent them from impacting business users.

All elements are relevant throughout the lifecycle, although some assume greater prominence in certain phases. For example, observability is most critical in the Operate stage, while model changes and reliability take a central position in the Maintain and Improve stage.

In practice, there are other aspects that require attention, including continuous checks and regression tests, intelligent alerting and routing, automated diagnostics, runbook guidance, and evidence capture for audits and governance. Automation reduces manual effort in the highest volume operational tasks and helps teams respond consistently. Agentic support can help triage drift, watch cost and latency, enforce evaluation gates, capture incident context, and guide responders through runbooks.

Following this approach enables a highly streamlined, tech-enabled operations process to run AI at scale in production, with expert operations input applied as needed, shortening the time from issue detection to resolution by making diagnostics and response more precise and repeatable.

Summary

AI value can erode after launch, and with it, the trust users have in it. Teams can quickly find themselves consumed by reactive fixes as they deal with changes, outages, degraded outputs, stakeholder disappointment, and falling adoption.

LiveOps is the AI operations discipline that keeps AI delivering value after launch through monitoring, observability, incident response, change control, and capability building. Gen AI and agentic AI can provide tremendous value, but they also bring new operating challenges such as higher inherent inaccuracy risk, reliability edge cases, higher operating costs, and more end-to-end complexity.

QuantumBlack, AI by McKinsey, brings expertise and insights developed while operating AI solutions across industries, and have the people, processes, and tools needed to sustain and scale ML, gen AI and agentic solutions end-to-end. Our aim is to help clients maximize the value of their AI solutions over time by enhancing their AI operations capabilities, bridging skillset gaps at launch, reducing adoption risk with production-ready AI solutions, and offering clients access to our latest innovative products for AI operations, built based on our learnings to-date

By assessing AI production-readiness, defining the necessary operating workflows, and standing up the right teams, technologies, and processes, we can ensure that your implementation delivers sustained value over time. If you are launching or scaling ML, Gen AI or agentic AI solutions in production, we can help. Reach out to us to learn more, and follow this publication for further insights, deep-dives, and case-studies.

Authored by Andrew Ferris, Kate Soliman, Parmeet Kumar and Bruce Philp. Edited by Carl Alviani, Alexandra Quinn and Jo Stichbury.

AI Value : The growing AI operations challenge was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

As AI agents move out of the realm of experimentation and into core enterprise workflows, the question of how to govern them effectively becomes critical. Companies are deploying agents across teams and runtime environments, making software delivery, customer operations, risk management, and other workflows more productive and efficient. But as deployments expand, the concern shifts from how to build capable agents to how to govern them consistently at scale.

The primary challenge is fragmentation. Teams are constantly evaluating new solutions, deciding what to adopt and what to replace, and this tends to make the underlying solution stacks transient and heterogeneous, with agents operating under different governance policies and monitoring practices. A cobbled-together approach of this sort can work with a dozen agents, but becomes harder to sustain with a hundred or more. As agents gain access to enterprise workflows, sensitive data, and operational decision-making, regulatory scrutiny compounds, making the issue even more urgent.

In our work developing Agentic AI Mesh architecture, we have found a way to address this challenge using a distributed governance model that equips each agent with an identity and compliance framework. Using a standard open specification that represents the agentic harness, we created smart contracts that embed governance within each agent. At runtime, these contracts can power an agentic service mesh that runs alongside the main agent, monitoring and evaluating its behavior, and enforcing policy and guardrails. This is one approach among several: centralized control planes and framework-native governance are others, and each has different tradeoffs. What follows is the path we took and what we learned from it.

The governance gaps that prevent agentic AI from scaling

Unless explicit steps are taken to prevent it, fragmentation tends to set in as enterprises scale up agent deployment. This manifests in several ways:

• At the configuration layer, different frameworks are applied, each with its own proprietary schemas, runtime abstractions, and orchestration patterns. This leads to a lack of standardization.
• At the operational layer, guardrails are developed on an ad-hoc basis, often embedded directly into individual applications rather than managed through reusable governance layers. This limits runtime control.
• At the validation layer, evaluation and quality assurance controls are often implemented late in the process, frequently in reaction to observed misbehaviors in production. This relegates validation to a reactive process, rather than a predictive, pre-deployment one.
• Combined, these gaps make frameworks and orchestration models difficult to transfer efficiently, leading to lock-in that can slow enterprise-wide scaling efforts.

In our experience, building governance in rather than bolting it on has held up better as deployments grow. When built into the agentic architecture, governance can actually accelerate scaling and innovation by:

• preventing operational inconsistencies
• making agents more modular and reusable
• reducing the cost of migrating to better solutions as technology evolves

Towards built-in agentic governance using smart contracts

In modern microservices environments, a service mesh separates operational concerns (security, observability, resiliency, policy enforcement) from application logic itself. We applied a similar architectural principle to our AI agents, introducing a sidecar proxy that operates alongside each agent at runtime. The proxy was responsible for enforcing governance policies, validating runtime behavior on sample traces, monitoring execution, maintaining auditability, and metering guardrails independently of the orchestration framework.

Under this model, the smart contract and sidecar create a built-in distributed governance layer that can operate across different environments. This approach offers several advantages over traditional centralized control models:

• governance becomes attached to the agent, making it self-auditable, rather than tied exclusively to the platform
• runtime validation and policy enforcement remain consistent even when orchestration frameworks change
• operational controls became built-in across the lifecycle especially as part of the CI/CD pipelines as well as probed within live environments

The practical shift is from an external control plane that governs agents from the outside to an embedded control plane that travels with each agent. That is what contributed to make our agents more self-standing, reusable, and auditable across use cases, building trust at the level of the individual agent.

The smart contract defines. The sidecar enforces.

The smart contract represents the agentic harness manifest, carrying the agent’s identity, model choices, tools, memory, guardrails, evaluation rules, observability controls, compliance expectations, and runtime requirements.

Based on this, the agent sidecar proxy:

• continuously enforces or verifies agent behavior against the harness manifest.
• operationalizes these controls during execution by validating approved models and tools’ use, alerting or enforcing permissions and policies, monitoring runtime behavior, and triggering human-in-the-loop requests.
• runs evaluations on a sample of production traces, manages authentication, exports audit evidence, and eventually detects or escalates contract violations.
• operates as a deterministic enforcement layer, evaluating runtime behavior against the declared contract and executing predefined actions such as allow, block, throttle, log, or escalate

We also used the sidecar in an agentic mode, in which it selectively invoked evaluators to assess behaviors that are difficult to capture through static rules alone. For example, a local evaluator or lightweight LLM-as-a-judge reviewed a configurable sample of interactions (e.g., five percent of traffic) and helped flag cases where observed behavior appeared to deviate from the declared contract.

Case Example: A leading technology provider has deployed reliable, reusable agents to large government entities, intending them to scale rapidly across administrative workloads, citizen facing workflows, risk reporting, and software development. As part of this effort, the team decides to direct some of the traffic to a local LLM to ensure data residency without needing a full platform to monitor and control the agent.

Applying a Story Grooming agent, they move several capabilities previously embedded in the application — including model configuration, prompts, tool definitions, and operational controls — into a declarative smart contract. The team can then isolate and tailor the agent, optimizing it on a smaller self-hosted LLM focused on specialized tasks that don’t require hyperscaler models. With roughly 15% of traffic routed locally, the team can restrict data residency on some part of the SDLC flow within built-in metrics and evaluations to ensure that the agent behaves as expected.

​Five principles for operating at scale

This architecture builds on principles introduced in our earlier work on the Agentic AI Mesh, extending them through Smart Contracts for AI Agents and portable runtime enforcement. Five core principles inherited from the original Agentic AI Mesh hold this architecture together — and each one directly addresses a scaling bottleneck:

• Composability means that any agent, regardless of framework or runtime, can be introspected and discovered in a trust network without requiring system-wide rework.
• Distributed intelligence lets tasks be decomposed and resolved by networks of cooperating agents, each with compliance baked in. Instead of a single monolithic orchestrator, we use a web of capable, governed agents collaborating at the task level, with each sidecar ensuring its agent stays within declared boundaries.
• Layered decoupling separates an agent’s logic, memory, orchestration, and interface functions into declared layers, maximizing both modularity and portability. When one layer needs to change because of a new model, a different memory backend, or an updated set of tools, the rest stays untouched.
• Vendor neutrality means every agent can be re-targeted to any framework, updated or replaced as the technology evolves, without locking you into a single vendor’s ecosystem. The same manifest that defines an agent for a framework can generate runnable code to another one.
• Governed autonomy keeps agent behavior proactively controlled through embedded policies, evaluations suite and idempotent escalation mechanisms. Agents operate with significant freedom, but within boundaries that are declared in the manifest, enforced by the sidecar policy engine, and auditable — even across heterogeneous runtimes.

Case Example: A Telecom player is embarking on a full agentic transformation journey where agentic workloads are expected to span multiple cloud providers and runtimes. The organization opts for a distributed governance architecture, with the objective of increasing the governance coverage from 10% to +80%, where each agent will be attached with a side agent mesh proxy to enforce required policies and report to a central agentic governance plane.

The SCO framework

This approach does carry overhead: organizations take on sidecar infrastructure, policy consistency, and governance ownership across teams. To support that, we used a Shape-Control-Operate (SCO) framework:

• Shape: During the shaping phase, organizations define the operational characteristics of the agent: model selection, tool configuration, prompts, memory architecture, evaluation criteria, and API interfaces. This is also the phase when security and governance controls are introduced, including policy definitions, access restrictions, PII safeguards, and prompt injection protections.
• Control: The control phase focuses on validation and auditability. Schema validation, runtime health checks, evaluation scoring, and compliance testing help ensure that agents meet operational and governance requirements before deployment. This phase can also support alignment against emerging regulatory and security standards, including internal governance baselines and external frameworks.
• Operate: Once deployed, the operating phase focuses on runtime reconciliation and continuous monitoring. The system evaluates whether deployed agents continue to align with their declared specifications, policies, and operational constraints. Drift detection, runtime monitoring, audit logging, and policy enforcement remain active throughout execution.

Model choices, memory boundaries, tool permissions, evaluation standards, runtime policies and escalation rules are declared during the design stage, rather than discovered after deployment. Together, these lifecycle stages establish a continuous governance loop capable of evolving alongside the agent ecosystem itself.

Summary

The enterprise agentic AI landscape is still evolving rapidly. Frameworks, orchestration models, and deployment patterns are likely to change significantly over the coming years. The governance challenge, though, looks structural rather than temporary. Centralized platform models give strong integrated controls but introduce ecosystem dependency. Many frameworks maximize flexibility but leave governance responsibilities fragmented across teams and implementations.

We observed that built-in governance provides a practical tradeoff that also fits, when necessary, with a central platform governance. The broader implication is that enterprise AI governance starts to look like modern distributed systems of governance, with portable governance layers that enforce policies consistently across changing runtime environments.

Reach out to us to learn more, and follow this publication for further insights, deep-dives, and case-studies about enterprise-grade agentic AI.

This article was authored by Sallah Kokaina, Ali Ustun, Tom Isherwood, Giuseppe Sorrentino, Jalaj Singh, Samarth Gupta, and Prakhar Mathur. The editor was Carl Alviani.

Smart contracts for AI agents was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

As any powerful new technology goes through its adoption curve, there arrives a moment when the technology itself stops being the differentiator. With cloud computing, it happened when AWS made world-class infrastructure available to everyone, shifting the advantage from those who could access high-quality infrastructure to those who knew what to do with it. A similar drama played out with databases, with mobile platforms, and with SaaS.

Now it’s happening with AI. As foundation models gradually converge in capability, the gap between best and second-best LLM is narrowing toward irrelevance. Tooling is standardizing as well, with agents that required months to deploy now running in days. Our experience suggests that a technical edge that was meaningful 18 months ago can often now be considered merely table stakes.

Despite this convergence, we’re still seeing wildly different outcomes between organizations as they deploy AI. They may all be using frontier models, employing skilled people, and staying on top of the dizzying pace of new tool releases, yet the value they realize varies. Some are achieving outcomes that compound over time: faster decisions, more consistent outputs, and workflows that improve as they scale. Many more, though, are seeing their early productivity gains plateau.

The organizations pulling ahead are not, by and large, the ones that have deployed the fastest or spent the most on infrastructure. They’re the ones that have figured out what to put inside the agents. Prompt quality is a part of this, as is knowing how to manage context windows. But for many organizations, a key differentiator is the ability to encode existing expertise.

The encoded knowledge problem predates AI

Most organizations know what “good” looks like within their category. If they’re competitive, they’ve spent years defining best practices, documenting their processes, and accumulating experience. However, many struggle to consistently apply this knowledge at the point of execution, especially as they scale up.

Sometimes this is because teams are too siloed, causing them to interpret guidance differently. Sometimes it’s because they’re repeatedly rethinking solved problems, just for the sake of doing something. Failure to codify knowledge, however it manifests, is a familiar problem.

For a long time, organizational knowledge was primarily tacit, living in the judgment and experience of individuals. This kind of knowledge is powerful, but nearly impossible to scale: if a senior analyst leaves, their methodology often leaves with them.

The shift toward explicit knowledge such as documents, playbooks, and standard operating procedures has made knowledge easier to share, but not necessarily easier to apply. These are all artifacts, which rely on human interpretation, and thus introduce variability at the point of use. A playbook may describe a process, but whether that process is followed depends on leadership, organizational culture, and individual initiative.

Software brought encoded knowledge that could be embedded into systems and made executable and repeatable. This is far more reliable but comes with trade-offs. Encoding knowledge in software means translating human expertise into technical formats, which tends to distance it from the domain experts who understood it best, making change slower and more expensive.

Each of these steps improves access to organizational knowledge and makes reuse easier, but none have fully solved the application problem.

Agent Skills: the executable expertise layer

With the arrival of AI, we’re now seeing the value of agent skills, a more structured approach to encoded knowledge.

In this context, a skill is a set of instructions that define how a task should be done, packaged in a reusable, versioned format. Skills enable organizations to capture procedures, constraints, contextual knowledge, and evaluation criteria across workflows, in a way that AI agents can consistently apply instead of inferring individual prompts.

Having access to skills shifts an agent role from one of reasoning and execution to something more like orchestration. It can now interpret intent and then invoke the appropriate skill, ensuring that its execution follows a defined, repeatable pattern. The result is a system that is not only more efficient, but significantly more reliable.

Crucially, however, skills need the context of the situation they’re operating within. For example, in a software development life cycle, this context might include repositories, architecture decision records, or architecture diagrams. In the broader architecture of agentic systems, skills are just one of several “context assets” that shape how agents behave, reason, and execute. Applying feedback to the right asset is critical. Skills must work in concert with system prompts, knowledge, handoff templates, tools, and memory.

Three current shifts are making the skills-based model viable:

• Converging foundation models are elevating the value of embedded knowledge over reasoning power.
• Natural language interaction is making powerful processing tools directly available to subject matter experts.
• Skills can be defined and refined largely by experts themselves, allowing knowledge to be encoded directly.

In regulated industries like banking and life sciences, skills offer the crucial additional benefit of traceability. With methodology encoded into a versioned, auditable format, the actions of agents leave auditable compliance trails, in sharp contrast to the often opaque workings of systems based on prompts alone. When a regulator asks how a conclusion was reached, the answer is largely summarized in the skill version that governed the workflow, rather than being reconstructed from memory.

How do skills work?

A skill is a structured knowledge document that encodes expert methodology, compliance requirements, and quality standards into a reusable, versioned format. Neither a prompt nor code, it is an explicit specification of how work should be done, written in natural language that agents read and execute at runtime. Note that as an AI-assisted workflow, skills support compliance activities. They do not replace legal, regulatory, or human review requirements.

This sample file (anonymized) shows how a typical skill is structured:

---
name: compliance-due-diligence-methodology
version: 1.2.0
domain: AML-KYC
regulatory-framework: EU-AMLD5 <-- Example
requires-skills: []
---

## 4-Step Due Diligence Research Process

### Step 1: Check internal policy documents
Objective: Understand regulatory requirements before analysis begins.
Tools: policy-lookup, grep-policy

### Step 2: Review existing analysis reports
Objective: Identify prior conclusions with evidence citations.
⚠️ Do not stop here — proceed to Step 3 for independent verification.

### Step 3: Deep dive into source documents
Objective: Extract direct evidence from original filings with page numbers.
Tools: pdf-extract-vision, pdf-keyword-search

### Step 4: Web research for current information
Objective: Verify current status; filed documents may be outdated.
Tools: web-research, perplexity-search
⚠️ This step is mandatory, not optional.

Skills sit between the agent and the domain rather than the agent and the model. The model provides reasoning capability, while the skill provides the expertise layer that makes that reasoning consistent and auditable. Multiple skills can be composed: an agent reads a methodology skill, a validation skill, and a reporting skill, and each governs a different dimension of its behavior.

Skills are stored in version control alongside the codebase, with a changelog that documents every modification. When regulations are updated, the relevant skill is updated, reviewed, and redeployed. And because they are written in natural language, this process often takes just hours, not the weeks or months typically required for code changes.

Skills-based AI in banking: Four agent skills that can halve compliance review time

The combination of reliability, efficiency, and traceability afforded by skills-based AI brings value to any industry, but its impact is especially pronounced in banking. The need for regulation and compliance in banking is absolute; without it, the trust required to complete transactions is undermined, which affects the sector’s entire value proposition.

Every compliance decision in banking comes with a paper trail: beneficial ownership declarations, corporate registry extracts, annual reports, sanctions screening results, among countless other records. A single customer due diligence case can touch dozens of documents across multiple jurisdictions, and every conclusion must be traceable, defensible, and aligned with current regulation.

For years, compliance teams have had two tools for making sense of this complexity.

• Rule-based systems bring consistency, but break down when faced with messy data or changes in regulation.
• Hard-coded logic offers predictability, but cannot process a narrative filing, and takes months to update when a threshold shifts.

Large language models brought flexibility but introduced several new problems: outputs that varied from case by case, no systematic audit trail, and a tendency to hallucinate the very thresholds they were meant to enforce.

Neither traditional approaches nor LLMs encode the expertise of analysts. In the traditional approach, this expertise rested with the analysts and not the infrastructure, while LLM approaches seek to simulate that expertise but often fail. The structured reasoning, evidence hierarchy, and quality checks that analysts use to turn a pile of documents into a defensible decision are the bedrock on which compliance relies.

AI skills offer a third way that explicitly encodes expertise in a way that experts themselves can review, approve, and update, producing reusable assets that are incorporated directly into agent workflows. Applied to banking compliance, skills transform the process into something deterministic and auditable.

In our work with the banking sector, we have developed and refined a variety of skills, and identified four that have a dramatic impact on the quality of the banking compliance workflow. When implemented together, their benefit is undeniable:

• An order-of-magnitude reduction in policy violation errors through guardrail skills.
• Significant improvement in first-pass report quality via structured methodology skills.
• 100 percent audit trail coverage with evidence-based reasoning.

Below we’ll describe the skills that drove these results and explain the design principles behind them.

The four skills that changed the workflow

Skill 1: Research methodology

This foundational skill, shown in the example above, defines the four-step due diligence process and makes each step mandatory with explicit tooling requirements. It also introduces an evidence confidence framework, where every finding is rated HIGH (multiple independent sources), MEDIUM (single authoritative source), LOW (indirect inference), or N/A (cannot determine). This forces analysts to quantify the strength of their conclusions rather than simply assert them.

After introducing this skill, the majority of cases completed all four steps, and human evaluators reported more than a two-fold improvement in the evidence quality.

Skill 2: Policy validation guardrails

The second skill acts as a quality gate between workflow stages. Rather than checking outputs at the end, it validates them before each subsequent step, blocking non-compliant work from progressing. It covers four validation layers:

• data integrity: required fields, type validation, cross-field consistency.
• business logic: beneficial ownership threshold calculations, indirect ownership chain multiplication.
• policy compliance: documentation requirements by risk level.
• output quality: report structure, citation format, professional language standards.

Critically, when validation fails, the skill generates specific remediation guidance instead of simply flagging the error. For example:

“Indirect ownership for John Doe: calculated as 20 percent but should be 30 percent. Chain: John (60 percent of CompanyB) → CompanyB (50 percent of Target) = 0.60 × 0.50 = 30 percent.”

Before implementing this skill, a significant portion of reports required late-stage rework due to policy violations. Applying the skill reduced the rework rate to the point of virtually eliminating the manual review burden, allowing analysts to reclaim almost half of their workweek for higher-value tasks.

Skill 3: Report writing quality

The third skill defines the exact structure every compliance report must follow:

• nine mandatory sections in a fixed order.
• a target length of 250–400 lines.
• citation format requirements (page numbers for PDF sources, access dates and publication dates for web sources).
• explicit prohibitions on speculative language, unsupported claims, and informal tone.

By using this skill to enforce structural and formatting standards, analysts were able to assemble reports that were virtually complete on their first pass. This cut the average report length by half and significantly reduced the number of revision cycles, freeing them from tedious formatting and allowing more time for substantive review.

“Documentation quality improved significantly,” observed one reviewer, “Consistent format across all submissions.”

Skill 4: Open questions and assumptions tracking

The fourth skill addresses a less visible but equally damaging problem: undocumented reasoning.

Prior to skill implementation, when regulators asked how a conclusion was reached, the analytical trail was frequently unrecoverable and highly inconsistent. This skill requires agents to document every knowledge gap actively searched for but not found, and every logical inference made without direct evidence, at the end of every analysis. It further requires agents to provide an explicit impact assessment for each instance and recommend follow-up actions.

Without this skill, almost half of reports were challenged for insufficient justification; with the skill implemented, we eliminated almost all gaps in analytical reasoning.

The validation architecture

These four skills do not operate independently, but are embedded in a workflow built around validation gates. Each gate checks that outputs meet the standards defined in the guardrails skill (#2 above) and blocks progression if they fail. This leads to a reliable pattern of:

extract → validate → analyze → validate → generate report → validate

This chain creates a traceable audit trail explaining what an agent concluded, which version of which skill it followed, and which validation it passed.

We run this workflow on an orchestration framework such as Argo (which we use along with ARK), with skills mounted as versioned ConfigMaps in Kubernetes; the same infrastructure we use to manage application deployments. Skills are therefore moving through the same review and deployment pipeline as code and using the same change controls. They can still be authored and updated by domain experts, though, without engineering involvement.

Principles for teams building skills-based workflows

Several design decisions drove this implementation’s success. Their absence is what typically causes skills-based approaches to underperform.

Single responsibility: Each skill covers one aspect of the workflow. A multi-aspect skill that encodes methodology, validation, and reporting simultaneously quickly becomes difficult to maintain and audit. With single responsibility skills, it’s immediately clear what to modify when a regulatory update requires change.

Explicit over implicit. Vague instructions produce variable behavior. “Do thorough research” leads to inconsistent results; “check these four sources, in this order, and document why each step was performed” does not. The most effective skills are prescriptive about process, not just intent.

Validation as architecture, not afterthought. Embedding validation between stages, throughout the workflow, catches errors when they are cheapest to fix. In practice, we found that the policy guardrails skill was just as important to reliable results as the methodology skill.

Skills as living assets. A skill that is not maintained drifts from current requirements. The teams that sustain improvement treat skills the way they treat software: versioned, tested before deployment, updated when requirements change, and reviewed on a defined cadence. A self-improving AI agent evaluates its own output quality, logs structured data, and auto-updates its skills over time without requiring human intervention for every cycle. To support this capability safely, teams must manage AI agent skills with the same rigor they apply to traditional APIs and software, through versioning and automated rollback capabilities. A systematic approach ensures that as an agent dynamically updates its own behavior and context, the organization maintains a traceable, stable, and compliant history of its evolving expertise.

A new organizational discipline

As skills become more central to the way AI agents execute their work, they begin pointing to a new kind of discipline. In much the same way that agile methodologies transformed how organizations built and maintained software, skills are now reshaping how organizations manage and evolve their knowledge.

Skills are more flexible and easily updated than traditional codebases, but also more transparent and predictable than ad-hoc prompting. This creates a shift that is both technical and organizational, changing the way teams capture, share, and refine their expert knowledge. The organizational knowledge base persists and compounds over time, rather than dispersing when individuals move on.

To realize these benefits, organizations must prioritize the task of making implicit knowledge explicit. Defining what tasks must be completed is only the first part; the more subjective steps of defining how they should be completed, and what constitutes a good outcome, must also be clearly defined.

This can present a challenge to organizations that view documentation as a one-time exercise rather than an integral part of the workflow. Knowledge that’s captured once and not maintained leads to skills that drift and eventually become irrelevant. Organizations that build systematic processes for capturing, validating, and updating expertise will find their advantage compounds.

Ensuring Responsible AI-Assisted Workflows

As we noted above, AI-assisted workflows do not replace legal, regulatory, or human review requirements. While skills-based agents create powerful efficiencies, their implementation requires a robust governance framework to be effective and responsible. Our experience has shown that success is dependent on three key principles:

• Humans remain in control.
• Continuous monitoring and testing
• Explicitly defined boundaries.

By building these guardrails, organizations can harness the full value of agentic AI while maintaining the highest standards of compliance and operational resilience. Further insights can be found in a recent McKinsey article titled “Deploying agentic AI with safety and security: A playbook for technology leaders”.

The question leaders should be asking

The first wave of AI adoption was about deployment: which tools, where to start, how to manage change. That question is largely answered.

The more important question now is: how do we ensure that what our best people know can be applied consistently and at scale, in every workflow?

Organizations that succeed in answering the question will build libraries of reusable skills that capture how they operate, as well as what they produce. Those that do not may find themselves with capable agents that produce inconsistent results. Ultimately, this reinforces a fact that has been true across the business world since long before the arrival of AI: an organization’s most valuable assets are not the technologies it employs, but the expertise of its people.

Reach out to find out more about our work with agentic AI.

This article was authored by Iris Roelens, Debanjan Banerjee, John Law-Lund, Anasthasia Manu, Frederik Van Hecke, Rory Walsh, Carl Alviani and Jo Stichbury. Additional graphics were supplied by Kasper Janehag.

Agent skills turn human expertise into AI advantage was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most remarkable qualities of generative AI (gen AI) and agentic systems is the speed with which they can progress from an impressive demo to real-world usage. Shipping to production is exciting for any developer, but it’s also often the point at which unanticipated friction emerges.

When agentic logic is integrated into a broader data pipeline, it becomes harder to follow what the system is doing and why. Prompts, tools, and routing logic between agents are often embedded throughout the pipeline in places that assume deterministic execution. The result is often failures that are difficult to explain and runs that are hard to reproduce. This can make even small changes feel risky because the outcome is no longer purely deterministic.

The source of this trouble is almost never the quality of the underlying model, but the way the system is structured. Structure is what makes the system understandable and evolvable for the engineers building it, and for anyone who needs to review it, trust it, or influence what it does.

We’ve seen this before

If this sounds familiar, it should. Development of early machine learning (ML) pipelines went through the same pattern: experimentation-first workflows, hidden states, fragile handoffs from notebooks to deployment, and systems that worked…until they didn’t. The teams that scaled ML reliably solved these problems through standardization, by creating clear structures for how work is executed, and building visibility into every run.

Gen AI has reached a similar inflection point, just much faster. Agentic systems are often released to production before teams have put basics like reproducibility, observability, and maintainability in place. Because of this, today’s agentic AI integrations are often plagued by recurring problems, amplified by dynamic execution paths and non-deterministic outcomes.

Many of our clients have found that the requirements for building agentic workflows (orchestration, observability, governance, version control of steps, and so on) overlap significantly with what existing data pipeline and ML infrastructure already provides. Rather than building agentic platforms on top of cloud infrastructure or custom setups, they are building them on top of their existing data platforms.

We have been through this process too. Kedro, an open-source tool first launched by QuantumBlack Labs in 2019, is a pipeline authoring framework that, with minimal adaptation, can run complex workflows combining agentic and deterministic steps. Kedro complements existing agent frameworks by enforcing system-wide structure, managing environment-specific configuration, and guaranteeing reproducible pipeline execution across environments.

In a recent article we argued that agentic systems require deterministic orchestration wrapped around bounded agent execution. That same architectural principle applies here: for a pipeline to be understandable, evolvable, and robust, it needs a backbone that controls what runs, in what order, with what configuration, while letting agent loops handle the reasoning within individual steps. Kedro operationalizes that deterministic backbone by providing a consistent way to:

• create pipelines
• manage configuration across environments
• make execution repeatable
• version the artifacts that matter, from datasets and prompts to model outputs.

Critically, Kedro brings software engineering best practices including modularity, separation of concerns, and testability into systems that include non-deterministic components.

In the real world, a workflow may have deterministic steps like data preparation, feature engineering, or model scoring sitting alongside less deterministic agentic steps like retrieval, prompting, tool use, and planning. Kedro lets teams run these side-by-side in the same pipeline without needing to change their stack or bolt on a separate orchestration layer just for gen AI and agentic AI. We’ve expanded on our explanation on where Kedro fits in a recent blog post about Kedro in the modern data and AI tooling landscape.

Building gen AI systems on explicit engineering principles

When gen AI systems break, it’s typically because the surrounding system isn’t designed to safely carry dynamic, non-deterministic behavior. What begins as a promising notebook prototype can quickly evolve into tightly coupled logic, hidden configuration, and behavior that is difficult to explain. These qualities can become entrenched once the system is running in production. Changes start to feel risky, failures become ambiguous, and a once robust demo suddenly feels fragile when running as part of a production pipeline.

This entrenchment is one of the primary causes of what’s been called the “gen AI paradox” in McKinsey & Company implementation studies: systems are widely deployed, but struggle to deliver reliable impact at scale. Without the right architectural foundations, observability, process design, and accessible, well-governed data products, this outcome is nearly inevitable.

Structured and modular workflows

Rather than treating engineering discipline as something to add later, Kedro treats it as part of the design from day one, ensuring that intelligence sits inside a system built to support it.

To give a concrete example, imagine a customer support assistant for an insurance company. The process begins when a user asks a question, prompting the system to interpret intent, retrieve relevant information, and generate a response. While this may sound like a simple chain of actions, it requires a multi-step agentic workflow built on a system with branching logic, state transitions, tool calls, and data dependencies. This is when structure stops being optional.

In this example, the solution can be organized into two pipelines, one for intent understanding, another for response generation. The agent framework handles reasoning and tool usage, while Kedro manages the surrounding system of configuration, datasets, credentials, and repeatable execution.

While this example is relatively simple, it demonstrates the importance of designing agentic workflows from the outset as part of a system, rather than loosely connecting scripts that gradually accumulate complexity.

Underlying this approach, and Kedro’s core functionality, are production engineering principles that should be familiar:

• Explicit, modular pipelines instead of loosely connected scripts: agent logic, context assembly, and data processing are clearly separated.
• Environment-aware configuration rather than hard-coded parameters: models, prompts, and tracing are defined declaratively and can vary safely across environments.
• Centralized credential management: external services and tools are configured consistently and securely.
• Reproducible, traceable execution across environments: each run captures the exact configuration, prompt versions, and dependencies that shaped the outcome.

Since agent logic sits on top of this structure, switching gen AI frameworks becomes a matter of updating the nodes that invoke those services, while the overall pipeline design remains intact.

This approach creates a separation of responsibilities. Agent frameworks handle dynamic decision-making and conversational state, while Kedro manages the operational backbone: data access, configuration, credentials, session state, and execution. The result is a stable system that can be changed without becoming brittle.

Transparent agent configuration

Beyond structure, there is a question that becomes increasingly important as AI agents proliferate: Why did this agent make that decision?

Agent behavior depends on multiple moving parts, including:

• the selected model
• the active prompt version
• registered tools and their parameters
• runtime configuration.

In many systems, reasons for agentic decision-making are scattered across framework internals and implicit defaults, and when something goes wrong, it’s unclear which inputs actually shaped the outcome. Kedro addresses this by treating agent setup itself as a first-class system component.

Instead of assembling context implicitly inside agent code, Kedro introduces a dedicated pipeline step that explicitly declares the model, prompts, and tools the agent will use before any reasoning occurs.

These inputs are loaded, validated, and versioned like any other pipeline inputs. At runtime, they are assembled into a single, well-defined context object and passed downstream. From that point forward, the agent consumes a clearly defined configuration. It does not need to know how its context was assembled, and there is no hidden execution layer coupling it to a specific framework.

This approach shifts agent setup from implicit runtime wiring to explicit system design. The configuration that shapes behavior is visible before execution begins, making decisions easier to understand and safer to evolve.

System-level visibility

Agentic systems are dynamic by design. They branch, select tools, and transition between states based on intermediate outputs. To safely evolve these systems, teams need to clearly understand how the system is structured and how decisions propagate through it.

Many agent frameworks, such as LangGraph, recognize this need and provide visualization features to inspect reasoning paths and state transitions. This kind of visibility is essential for debugging and iteration within the agent loop.

Kedro extends this idea to the system level through its native visual interface, Kedro-Viz, which provides a graphical representation of pipeline structure, execution flow, and dependencies. Deterministic and agentic components are inspectable within the same architecture. And because structure and configuration are explicit, agent setup appears directly in the pipeline graph alongside traditional data steps. Gen AI-specific components such as agent context, prompts, LLM configuration, and tracing can be visually distinguished from standard processing nodes, clarifying the boundary between reasoning and transformation.

Since prompts are treated as datasets, they can be inspected and compared without running the pipeline. This enables teams to review exactly what the agent will be prompted with before deploying a change. Agent nodes are also inspectable. Their internal structure, including states and transitions defined by the chosen agent framework, can be explored without executing the full workflow.

The impact of this design is subtle but significant. Agent behavior moves from opaque runtime execution to visible system architecture. Teams can understand how the system is assembled before evaluating its performance, and they can review changes as structural modifications rather than unpredictable side effects.

Together, modular workflows, transparent agent configuration, and system-level visibility form the engineering foundation of a production-ready gen AI system.

Once these principles are in place, the challenge shifts from internal clarity to completing the engineering loop, connecting observability, evaluation, and lifecycle tooling into a single, disciplined workflow without fragmenting the system.

Closing the engineering loop with ecosystem integrations

So far, we’ve focused on the structural and operational properties that Kedro makes explicit in a gen AI project, to help create stability. But in addition to internal discipline, production systems must also support observability, evaluation, and controlled evolution. This enables models, prompts, and tools to change over time without forcing teams to redesign their architecture each time.

Rather than rebuilding these capabilities from scratch, Kedro connects specialized ecosystem tools into the same structured workflow. The goal is not to own every layer of the gen AI stack, but to make external platforms feel like natural extensions of a coherent system design.

Observability as a system property

When something goes wrong in a gen AI system, it’s not enough for teams to know that something failed; they need to understand exactly what happened. This becomes critical when:

• a small prompt change alters behavior in production
• an agent selects the wrong tool at scale
• costs or latency spike unexpectedly
• model updates subtly shift output quality.

These issues rarely come from a single component. They emerge from how the system behaves as a whole.

Kedro approaches observability by making the key ingredients of a gen AI system explicit before execution. Instead of instantiating models, prompts, and tracing implicitly inside agent code, these properties are declared as part of the project configuration.

This has two important effects. First, behavior becomes explainable. Each run carries a clear record of which model, prompt version, and tools were used, and these are tied to a reproducible pipeline execution.

Second, observability tools can integrate cleanly at the system level. Instead of embedding logging or tracing logic into agent implementations, Kedro connects pipeline execution to specialized platforms like Langfuse or Opik. This ensures that:

• traces are captured automatically
• prompts are versioned and linked to runs
• agent behavior can be inspected end-to-end in dedicated UIs
• evaluations and comparisons remain connected to structured executions.

In this design, observability becomes a property of the architecture. The agent focuses on reasoning, and the surrounding workflow ensures every run is traceable and comparable.

Evaluation as an essential part of production gen AI

Observability explains what happened. Evaluation answers whether the system is improving or regressing over time.

In production gen AI systems, evaluation involves a wide range of metrics that go well beyond the monitoring performed in ML systems. These include:

• output quality and consistency
• prompt regressions after updates
• changes in agent decision patterns
• tool usage accuracy
• operational metrics such as cost and latency.

Across the gen AI ecosystem, one pattern is consistent: dataset-based evaluation forms the foundation for assessing output quality, consistency, and correctness over time.

This approach aligns naturally with Kedro’s design. Evaluation datasets are catalog entries, declared in configuration, synced between local files and remote platforms, and managed the same way prompts and other datasets are. During development, teams author evaluation items locally and iterate quickly. In staging or production, the remote platform becomes the source of truth. For reproducibility, a dataset can be pinned to a point-in-time snapshot. All of this is done through configuration, without changing pipeline code.

This connects to platforms like Langfuse, where experiment execution, scoring, and regression analysis happen outside Kedro. Kedro handles the structured, reproducible runs; the evaluation platform handles the rest. The loop between experimentation, observability, and evaluation closes inside the same system.

Flexibility without fragmentation

In gen AI and agentic AI ecosystems new frameworks and tools appear constantly, and teams need the freedom to adopt them without rewriting their systems. This is why Kedro is designed to be deliberately framework-agnostic. Specifically, this means:

• agent logic can be implemented using different frameworks or custom approaches
• tools remain plain Python functions
• configuration and execution stay consistent
• prompt management, tracing, and evaluation are handled through integrations rather than embedded framework logic.

Because there is a clear boundary between the responsibilities of agent frameworks and the system-level structure Kedro provides, teams can change how agents are implemented without redesigning the surrounding workflow. This enables pipelines to remain intact, while inputs, outputs, and operational guarantees stay the same.

This flexibility extends across the full lifecycle from development to observability and evaluation without fragmenting the system.

From opaque experiments to trustworthy systems

As a gen AI solution graduates from a promising demo to a real-world system, its success depends on whether it can be:

• understood by the whole team (not just its original authors)
• observed clearly in production
• evaluated systematically over time
• evolved safely as tools and standards shift.

There’s no doubt that the rapidly evolving gen AI tools available to us offer incredible resources in terms of intelligence and high-quality decision-making on demand. Kedro acts as a partner to these tools, helping ensure that the data pipelines used by gen AI systems are reliable. As a deterministic operating system that can incorporate non-deterministic agents, this makes it, in many cases, the crucial link in the chain of capabilities that enable AI solutions to generate real, lasting value.

Find out more about Kedro

Kedro is an open-source project with broad industry adoption. It records over seven million downloads annually, has more than ten thousand GitHub stars, and is used by teams across sectors to build production-grade data and machine learning systems.

Kedro is supported by a full-time development team and is part of the LF AI & Data Foundation, providing a neutral, vendor-independent governance model that supports long-term sustainability and community-driven development. Beyond the core framework, Kedro is supported by an active and growing open-source community. Contributors and users share patterns, extensions, and real-world experience, helping the project evolve in response to practical needs. A growing list of blog posts, videos and projects that use Kedro can be found in the awesome-kedro GitHub repository.

Kedro is also backed by extensive documentation. This includes tutorials, example projects, coffee chats, and dedicated sections for each part of the tool, with code snippets ranging from introductory features to more advanced use cases. Teams can get started quickly using a ready-to-use project template, which includes sample datasets and enables users to train their first model with just a few commands from the terminal.

To find out more about Kedro, reach out to us on Slack.

This article was written by Elena Khaustova and Merel Theisen, with additional contributions from Elijah Koh, Alice Cima, Joseph Perkins, Carl Alviani, and Jo Stichbury.

Generative AI workflows need engineering discipline to scale beyond the demo was originally published in QuantumBlack, AI by McKinsey on Medium, where people are continuing the conversation by highlighting and responding to this story.