An exploration of the engineering principles behind trustworthy generative AI, from evidence-grounded evaluation and organizational constraints to the emerging possibilities of quantum machine learning.

The AI Industry Has a Trust Problem That Better Models Alone Cannot Solve

Something unusual is happening in artificial intelligence.

We have reached a point where developers can build an AI assistant in an afternoon, connect it to enterprise documents, give it access to tools, and deploy it behind a polished interface.

The assistant can summarize contracts, analyze financial reports, generate code, retrieve information, and even execute multi-step workflows.

On paper, the system looks impressive.

Then someone asks a question it cannot answer confidently.

The assistant produces a convincing explanation, cites the wrong document, or confidently fills a gap with information that was never there.

Suddenly, the conversation changes.

The question is no longer whether the AI system is capable.

The question becomes whether anyone should trust it.

This distinction is becoming one of the most important challenges in production AI engineering.

Organizations are investing in increasingly capable language models, sophisticated retrieval systems, and autonomous agents. Yet many still struggle with the same fundamental problems: inconsistent answers, unverifiable reasoning, poorly defined expectations, and systems that perform brilliantly in demonstrations but unpredictably in real environments.

What if the biggest obstacle to reliable AI is not the intelligence of the model but the architecture surrounding it?

And what if solving that problem requires rethinking not only how we build AI systems but also how humans interact with them?

These questions sit at the center of a conversation between Sanjana Gupta, Relationship Lead at Packt, and Daniel Buchta, CEO of Avenue78, a mathematician and technology architect with more than 25 years of experience building complex IT systems.

Drawing on his background in mathematics, generative AI, enterprise architecture, and quantum computing, Daniel offers a perspective that challenges several assumptions about where AI engineering is heading.

His central argument is straightforward:

The future of AI will not be determined by what systems can generate. It will be determined by what people can reliably trust them to do.

1. The Architecture of AI Has Changed, but Our Engineering Assumptions Haven’t

To understand why trust has become such a significant problem, we first need to understand how AI has changed the relationship between software and its users.

Traditional software systems were built around explicit instructions.

A developer defined the logic, a database stored structured information, and an application executed predetermined operations.

If a user clicked a button, the system followed a known sequence.

If a query ran against a database, the expected output was generally deterministic for a given state of the data.

This made software behavior relatively straightforward to test.

Generative AI introduces a fundamentally different interaction model.

Instead of describing every operation through code, users can express their intentions through natural language.

The system interprets meaning, draws on context, and generates a response.

Daniel explains this shift through three characteristics that he believes make modern AI particularly aligned with human interaction.

Perception: AI systems increasingly operate through iterative feedback

Many modern AI workflows are built around a repeated cycle.

The system receives information, interprets the current situation, performs an action, and uses the resulting feedback to determine what happens next.

This is particularly visible in agentic systems.

An AI agent might retrieve a document, examine its contents, identify missing information, call another tool, and revise its response.

The workflow is no longer simply input followed by output.

It becomes an iterative process.

However, this does not mean every generative model continuously learns from its interactions. In many deployed systems, the model’s parameters remain fixed while the surrounding application manages state, memory, and feedback.

That distinction matters because reliability depends on understanding which parts of the system can actually adapt.

Semantics: Meaning has become part of the programming interface

Previously, interacting with complex software required understanding its functions, interfaces, or query languages.

Now, natural language can act as an interface to computation.

A user can request:

“Compare the last three quarterly reports, identify changes in operating expenses, and explain the largest discrepancies.”

A sufficiently equipped AI system can interpret that request, retrieve relevant information, and coordinate multiple operations.

But semantic flexibility introduces ambiguity.

What counts as a discrepancy?

Which reports should be treated as authoritative?

Should the system compare absolute values or percentage changes?

What happens if the documents disagree?

These are not merely language-model questions.

They are architecture questions.

Attention: Systems can prioritize relevant information

Transformer-based models use attention mechanisms to represent relationships between tokens and contextual information.

This has contributed significantly to the capabilities of modern language models.

But from an engineering perspective, relevance remains a difficult problem.

A model may process a large amount of context without correctly prioritizing the information necessary to answer a specific question.

That is why context selection, retrieval quality, and information organization remain critical even as context windows expand.

Daniel connects these ideas to a principle that has influenced his approach to architecture throughout his career: the Anna Karenina principle.

Inspired by Tolstoy’s famous observation about happy and unhappy families, the principle suggests that successful systems depend on several necessary conditions being satisfied together.

A failure in any important condition can undermine the entire system.

For AI engineering, the implication is significant.

A highly capable model cannot compensate indefinitely for poor data, inadequate evaluation, unclear requirements, or missing organizational knowledge.

A reliable AI product must be aligned with its purpose at every layer, not merely at the model layer.

2. The Missing Layer in Enterprise AI: An Architecture of Trust

During the conversation, Sanjana asks Daniel to describe a real project, from the initial business problem to its implementation.

Rather than beginning with a model, framework, or infrastructure decision, Daniel starts with a more fundamental question.

What would make generative AI genuinely useful and acceptable to people?

His answer is trust.

He draws a comparison with technologies that became essential because people developed confidence in using them.

Banking systems made it possible to manage money digitally.

Search engines became a trusted starting point for finding information.

Generative AI is entering a more sensitive territory.

People are increasingly using it to interpret information, support decisions, and participate in their thinking processes.

That creates a different kind of responsibility.

When an AI system helps summarize an article, a small error may be inconvenient.

When it influences a financial assessment, clinical workflow, or business decision, the consequences can be much more serious.

Daniel describes an ongoing project at Avenue78 built around a reusable trust-oriented architectural core.

At the time of the interview, the team was developing three initial applications:

Privacy: An application intended to address privacy concerns when interacting with public-facing technology.

Shield: A component focused on protecting structured environments and information.

Studio: An environment intended to support the creation of AI assistants.

Daniel did not disclose the underlying proprietary architecture or provide independent performance results. The important insight from the conversation is therefore the design philosophy rather than a verified implementation benchmark.

Instead of treating trust as a feature added after development, the team is attempting to make it a foundational architectural concern.

This raises an important question for enterprise AI teams.

What would a trust-first AI architecture actually require?

Consider an enterprise knowledge assistant.

A conventional implementation might connect a language model to a vector database, retrieve relevant documents, and generate responses.

A trust-first implementation needs to ask additional questions.

Can the system identify where every important factual claim originated?

Can it distinguish retrieved evidence from generated interpretation?

Does it respect the user’s access permissions?

Can it recognize when the available information is insufficient?

Can a user challenge an answer and inspect the evidence?

Can the organization investigate what happened when a response is wrong?

These requirements suggest that trust is not one technical component.

It is an architectural property emerging from several controls working together.

A practical trust-oriented architecture might include:

• Identity and authorization: Determine which users and agents can access particular resources.
• Context management: Supply relevant, current, and appropriately scoped information.
• Evidence and provenance: Preserve links between generated claims and their supporting sources.
• Evaluation: Test whether the system meets defined accuracy, reliability, and usefulness criteria.
• Observability: Record retrieval behavior, tool calls, failures, and relevant execution traces.
• Human oversight: Introduce review or approval when actions carry meaningful risk.
• Failure handling: Allow the system to express uncertainty, abstain, retry safely, or escalate.

These are practical engineering implications of Daniel’s trust-first argument, rather than a disclosed blueprint of Avenue78’s product.

The distinction matters.

A system can have excellent retrieval performance and still expose confidential information.

It can produce accurate answers and still execute an unauthorized action.

It can cite genuine documents and still misinterpret their contents.

Trust is not simply the absence of hallucinations. It is the ability to understand, constrain, and verify the behavior of the entire AI system.

3. Why AI Architecture Is Becoming Easier to Build but Harder to Get Right

One of Daniel’s more provocative observations is that generative AI is making certain aspects of system architecture simpler.

That might sound surprising.

After all, modern AI systems can involve embeddings, vector databases, orchestration frameworks, tool integrations, monitoring pipelines, evaluation systems, and multiple models.

How can architecture be getting simpler?

Daniel’s argument is that many implementation patterns are becoming increasingly accessible and reusable.

Developers no longer need to invent every component from scratch.

Retrieval-augmented generation, agent orchestration, and automated data-processing workflows have established implementation patterns.

But as the mechanics of building become easier, the difficulty moves elsewhere.

It moves toward defining the problem, supplying meaningful context, managing expectations, and deciding what successful behavior looks like.

Consider two teams building an AI assistant for financial research.

The first team begins by choosing a model, selecting a vector database, and designing a multi-agent workflow.

The second begins by asking:

What decisions will the assistant support?

Which information sources are acceptable?

How recent must the information be?

What happens when sources disagree?

Which errors are tolerable?

What evidence must accompany each conclusion?

Both teams may eventually use similar technical components.

But the second team has a clearer basis for determining whether its system works.

This is the distinction between building an AI application and engineering a reliable AI product.

Daniel emphasizes the importance of context and metadata.

A language model may be capable of interpreting a financial document, but it still needs to know whether the document is current, which organization it belongs to, what period it covers, and whether the user is authorized to access it.

Without that information, even a sophisticated model may produce a plausible but inappropriate response.

The lesson for AI architects is that automation does not eliminate the need for architecture.

It changes where architectural judgment is most valuable.

The challenge increasingly becomes designing the conditions under which AI-generated behavior can be trusted.

4. The Most Dangerous AI Failure May Be an Answer Nobody Can Verify

When Sanjana turns the conversation toward production evaluation, Daniel identifies a problem that many AI teams encounter after deployment.

Traditional systems are often evaluated against expected outputs.

A calculation should return a particular value.

A database query should retrieve a defined set of records.

An API should produce a response that conforms to its contract.

Generative AI complicates this approach because the same request can produce different wording across multiple executions.

Two answers can be semantically equivalent without being textually identical.

Conversely, two answers can sound equally convincing while only one is factually correct.

This makes simple string matching inadequate for many generative AI applications.

Daniel argues that one of the first questions evaluators should ask is:

Where did the information come from?

For a document-grounded assistant, that means the ability to trace claims back to their evidence.

Not merely identifying the document that was retrieved, but locating the relevant passage.

For example, imagine an AI assistant answering:

“The company’s operating expenses increased by 18% in the second quarter.”

A useful evaluation process should establish whether the cited source actually supports that figure.

It should determine whether the calculation is correct, whether the comparison period is appropriate, and whether any relevant qualifications were omitted.

A citation that merely points to the company’s annual report is not sufficient evidence.

The citation must support the specific claim.

Why citations are necessary but not sufficient

Daniel places considerable emphasis on source attribution as the starting point for evaluating hallucinations.

This is especially valuable in retrieval-based systems.

However, hallucinations are not limited to missing citations, and the presence of a citation does not establish correctness.

An AI system can misread a valid source.

It can combine facts from unrelated passages.

It can draw an unsupported conclusion from accurate information.

It can also generate an incorrect answer even when the necessary context is available.

A more complete evaluation strategy therefore needs to examine several dimensions.

Groundedness: Is the response supported by the available evidence?

Factual correctness: Are the claims accurate?

Retrieval quality: Did the system retrieve the information required to answer the question?

Task completion: Did the response actually solve the user’s problem?

Safety and authorization: Did the system remain within its permitted boundaries?

Consistency: Does performance remain acceptable across different prompts, inputs, and repeated runs?

For production teams, this suggests a practical evaluation workflow.

Start with representative tasks collected from actual users.

Define expected behavior and acceptable failure conditions.

Create reference answers or evaluation criteria where possible.

Test retrieval and generation separately.

Inspect unsupported claims.

Then repeat the tests as models, prompts, documents, and tools change.

The most important part is that evaluation must reflect what users consider successful.

Daniel repeatedly returns to the importance of defining expectations with customers and end users.

Without agreement on what a good answer looks like, a team can spend months optimizing metrics that do not correspond to real-world value.

The goal of evaluation is not to prove that an AI model can answer questions. It is to establish when its answers are dependable enough to use.

5. The Hidden Trade-Off: Would You Rather Have One Perfect Task or 100 Good-Enough Ones?

At this point, the conversation moves away from infrastructure and toward something less frequently discussed in technical architecture reviews.

Human expectations.

Daniel describes a recurring pattern he has observed when organizations adopt generative AI.

At first, people become excited by its possibilities.

They imagine AI automating entire workflows, replacing repetitive tasks, and dramatically increasing productivity.

Then they begin testing it against their own work.

The model makes mistakes.

It overlooks context.

It fails to reproduce exactly how an experienced professional would approach a problem.

Enthusiasm turns into disappointment.

Daniel describes this progression using the familiar imagery associated with the Dunning–Kruger effect: an early peak of confidence followed by a period of disillusionment.

His analogy is not a formal psychological model of AI adoption, but it captures a recognizable organizational challenge.

Teams often move from expecting AI to do everything to dismissing it because it cannot do everything perfectly.

The difficult work begins between those extremes.

Daniel frames the trade-off through a deliberately provocative comparison.

Would you prefer to complete one task perfectly yourself, or allow AI to complete 100 tasks at an acceptable level of quality?

The numbers are illustrative, not a measured productivity result.

The question exposes an important tension.

People are accustomed to evaluating their own work against standards they understand.

When AI performs the same task differently, they may interpret that difference as failure.

But the appropriate standard depends on the consequences of the task.

For a low-risk activity such as drafting alternative headlines, generating many usable options may be more valuable than producing one meticulously crafted response.

For a medical decision, financial authorization, or security-sensitive action, an approximate answer may be entirely unacceptable.

This is why AI adoption requires more than persuading employees to accept imperfect outputs.

It requires organizations to classify tasks according to risk.

Some tasks can tolerate variation.

Some require review.

Others require deterministic controls or should remain outside the system’s authority.

The real trade-off is not simply perfection versus productivity.

It is how much autonomy a system should receive given the cost of being wrong.

For AI product leaders, this is a crucial design decision.

The right answer may be full automation for one workflow, human approval for another, and no AI involvement for a third.

6. The Production Failure That Changed Daniel’s Approach to Engineering

One of the most revealing moments in the interview arrives when Sanjana asks Daniel about a project that did not go according to plan.

Instead of offering a polished success story, he describes a major failure from earlier in his career.

Around two decades ago, Daniel was leading the architecture of a large IT system in Slovakia.

The team was preparing a significant upgrade.

He had confidence in the architecture, the implementation, and his own experience.

But during the transition, critical assumptions about compatibility between the old and new versions proved incorrect.

Production systems began failing.

The upgrade created serious operational problems.

Looking back, Daniel identifies two interconnected causes.

He underestimated testing.

And he allowed his confidence in his own expertise to prevent him from questioning important assumptions.

The technical failure was significant.

But the professional lesson was even more enduring.

Experience does not remove uncertainty.

In fact, confidence accumulated through previous successes can sometimes make uncertainty harder to recognize.

Daniel also recalls a later machine learning project in which he attempted to handle too much of the work himself.

He took responsibility for the mathematics, modeling, and broader implementation.

Because he appeared capable of managing everything, the organization did not provide the team support the project required.

Eventually, the workload became unsustainable.

His conclusion was that individual expertise cannot replace the combined capabilities of a properly structured team.

These experiences are particularly relevant to modern AI engineering.

A single developer can now assemble an impressive AI prototype using APIs, frameworks, and existing models.

But a production system may require data engineering, security, infrastructure, evaluation, domain expertise, governance, and operational support.

The ability to build a prototype alone does not mean one person can safely own the entire lifecycle.

What these failures teach AI teams

The first lesson is to test assumptions, not just functionality.

A system may pass ordinary tests while failing during migration, under unexpected inputs, or when interacting with dependencies that behave differently from what the team anticipated.

The second is to design for failure before deployment.

What happens when a model produces an unsupported answer?

What happens when a tool call fails?

What happens when a retrieval system returns outdated information?

What happens when a new model version changes behavior?

The third is to make uncertainty visible.

Teams should be able to question architectural decisions without those questions being interpreted as a challenge to individual competence.

And the fourth is to recognize that production reliability is a team responsibility.

The most dangerous assumption in engineering is not that something might fail. It is believing that experience has made failure unlikely enough to ignore.

7. The Constraint Most AI Teams Underestimate Isn’t Technical

Latency matters.

Infrastructure costs matter.

Scalability, throughput, data quality, and model performance all matter.

But Daniel argues that some of the most consequential constraints in generative AI projects are organizational.

Specifically, they involve knowledge that never reaches the engineering team.

Imagine building an AI assistant for a customer support organization.

The engineering team has access to documentation, historical tickets, and internal knowledge articles.

It builds a retrieval system and evaluates responses against the available material.

Initial results look promising.

But experienced support agents know something the documentation does not capture.

Certain customer complaints require escalation regardless of what the knowledge base recommends.

Some policies have exceptions.

Some troubleshooting steps are technically correct but inappropriate for particular customer situations.

If those unwritten rules never enter the system’s design, the AI assistant will not reliably follow them.

The problem is not necessarily a weak model.

It is missing organizational context.

Daniel describes an earlier project in which his team was instructed not to communicate directly with end users and customers.

Without that access, the team could not adequately understand the real requirements or the constraints shaping the users’ work.

He considers that separation a decisive reason the project failed.

This leads to one of the strongest lessons from the conversation.

You cannot engineer a trustworthy AI system around a workflow you do not fully understand.

For AI teams, user involvement should therefore extend beyond initial requirements gathering.

Domain experts need to participate in defining evaluation criteria, reviewing outputs, identifying edge cases, and refining workflows.

They also need to understand what the AI system can and cannot do.

Otherwise, the engineering team may optimize for technical success while the users experience operational failure.

Start with the problem traditional software cannot solve well

Daniel offers another principle from his most successful projects.

Find the one important problem that the existing approach cannot adequately solve.

Then build around that.

This is a useful corrective to the tendency to introduce AI simply because the technology is available.

A deterministic rules engine may be better for applying a small set of clearly defined business policies.

A conventional database query may be better for retrieving an exact value.

A standard workflow automation tool may be more reliable for executing a predictable sequence.

But an AI system may offer meaningful value when the task requires interpreting diverse unstructured information, navigating ambiguous language, or synthesizing evidence across many sources.

For example, extracting relevant obligations from thousands of contracts with inconsistent formats may be difficult to achieve through rigid rules alone.

A language-model-based workflow could help interpret and organize that information, provided its outputs are verified against the source documents.

The important question is not whether AI can perform a task.

It is whether AI provides a meaningful advantage that justifies its additional uncertainty, cost, and governance requirements.

When that value is clear, organizations have a stronger reason to share knowledge, contribute resources, and participate in making the system successful.

8. Beyond Classical AI: Could Quantum Machine Learning Change the Equation?

Toward the end of the conversation, Sanjana introduces a different dimension of Daniel’s work: quantum computing and quantum machine learning.

The transition may initially seem unexpected.

After discussing trust, evaluation, and organizational constraints, why move toward quantum computing?

For Daniel, the connection lies in three limitations he believes will influence the future of AI.

Resources. Privacy. Comprehension.

These represent different challenges, and quantum technologies may eventually contribute to addressing some of them.

However, it is important to distinguish promising research directions from capabilities demonstrated in practical AI systems.

Resources: Can quantum computing reduce computational bottlenecks?

Modern AI systems can require substantial computational resources.

Training large models demands significant processing capacity, memory, and energy.

Inference workloads also become expensive as applications scale.

Quantum computing uses a fundamentally different computational model based on quantum states and operations.

For certain classes of mathematical problems, quantum algorithms offer theoretical speedups over known classical approaches.

Daniel sees this as an opportunity to rethink how computationally intensive AI tasks might be performed.

He also discusses the possibility of substantial energy savings.

That possibility should be treated cautiously.

Quantum hardware has its own significant operating requirements, including control systems and, for many platforms, cryogenic cooling.

A theoretical computational advantage does not automatically translate into lower end-to-end energy consumption.

Nor is quantum computing known to provide a general speedup for all machine learning workloads.

The meaningful research question is narrower:

Can specific quantum algorithms solve useful AI-related problems more efficiently than the best available classical alternatives, once the complete system cost is considered?

That remains an active area of investigation.

Privacy: Quantum technology creates both opportunities and threats

Daniel also highlights the relationship between quantum computing and privacy.

Quantum communication techniques, including quantum key distribution, offer ways to detect certain forms of interception under appropriate conditions.

At the same time, sufficiently capable fault-tolerant quantum computers could threaten widely used public-key cryptographic systems through algorithms such as Shor’s algorithm.

This creates a dual challenge.

Quantum technology may enable new security mechanisms while also requiring organizations to migrate away from vulnerable cryptographic approaches.

For AI systems handling sensitive enterprise information, this matters because privacy is not limited to what the model generates.

It includes how information is transmitted, stored, accessed, and protected throughout the infrastructure.

Quantum communication does not, by itself, guarantee privacy for an entire AI application.

But developments in quantum security could influence how future systems protect sensitive information.

Comprehension: Could quantum representations improve machine learning?

The third area is the one Daniel finds especially compelling.

He is interested in whether quantum mathematical structures could provide new ways to represent and process complex relationships.

Classical machine learning already uses high-dimensional vector spaces to represent information.

Quantum systems operate using mathematical structures such as complex Hilbert spaces, where states and transformations follow quantum mechanical principles.

Quantum machine learning researchers are investigating whether these structures can support useful approaches to classification, optimization, and representation learning.

Daniel sees potential for these methods to contribute to future AI systems with richer ways of handling information.

He also discusses his research interests in quantum language models.

However, the possibility of better mathematical representations should not be confused with demonstrated improvements in semantic understanding.

There is not yet established evidence that quantum language models generally comprehend human language better than leading classical models.

The potential is intriguing precisely because the research questions remain open.

9. Why Classification May Be an Important Test for Quantum AI

When Sanjana asks where quantum computing could make a meaningful difference, Daniel points toward optimization and classification.

Classification is fundamental to machine learning.

A financial model may estimate whether a borrower belongs to a particular risk category.

A medical model may classify patterns associated with a disease.

A pharmaceutical application may attempt to distinguish promising molecular candidates from less suitable ones.

These tasks often involve finding meaningful boundaries in complex datasets.

Daniel uses an analogy from Cinderella to explain the intuition behind quantum classification.

Imagine a large collection of mixed objects that must be separated into two groups.

A classical approach processes the information using conventional computational operations.

A quantum approach may encode information into quantum states and use quantum operations to transform those representations.

In certain mathematical settings, this can lead to different computational strategies.

But the analogy has an important limitation.

Quantum superposition does not mean that a quantum computer can simply inspect every possible answer simultaneously and return the correct classification instantly.

Quantum algorithms must be designed so that useful information can be extracted through measurement.

Their advantages depend on the structure of the problem, how data is encoded, the cost of preparing quantum states, and the algorithm being used.

This is why quantum machine learning research frequently investigates whether proposed advantages survive realistic implementation costs.

For example, a quantum classifier might perform well on a carefully designed dataset but lose its advantage once the overhead of encoding large amounts of classical data is included.

The engineering challenge is to identify problems where quantum methods offer measurable benefits over strong classical baselines.

What should AI engineers pay attention to?

For practitioners, the immediate opportunity is not to replace production machine learning pipelines with quantum systems.

It is to understand which research developments could eventually affect practical architectures.

Three questions are particularly useful.

First, does a proposed quantum approach demonstrate an advantage on a relevant problem rather than only a synthetic benchmark?

Second, does that advantage remain after accounting for data preparation, noise, hardware limitations, and measurement overhead?

Third, can the result be reproduced and compared fairly against modern classical methods?

These questions separate engineering progress from technological optimism.

Daniel’s enthusiasm reflects the potential he sees in quantum mathematical approaches.

The practical challenge is turning that potential into systems with measurable advantages under real-world constraints.

And that brings the discussion back to where it began.

Trust.

A future quantum-enhanced AI system would still need evaluation, evidence, security, and clear operating boundaries.

A faster or mathematically more sophisticated system is not automatically a more reliable one.

10. What Production AI Teams Should Do Differently Today

Daniel’s conversation offers a broader lesson that extends beyond any single model, framework, or emerging technology.

Reliable AI requires an alignment between what the technology can do, what the organization needs, and what users can verify.

For engineering teams, that translates into several practical priorities.

Begin with a decision, not a model

Identify the actual business problem and the decision or workflow the AI system is intended to support.

Determine why a conventional solution is insufficient.

Then establish what measurable improvement AI is expected to deliver.

Define trust before deployment

Decide what evidence users need, what errors are unacceptable, and what the system should do when it is uncertain.

Trust requirements should influence architecture from the beginning.

Treat context as an engineering dependency

Relevant information, metadata, permissions, and domain knowledge must be available to the system in a usable form.

A larger context window does not compensate for incomplete or poorly governed information.

Make evaluation continuous

Use representative production scenarios, source-grounded tests, regression suites, and feedback from actual users.

Evaluate both the model’s output and the behavior of the surrounding application.

Design for human participation

Bring domain experts into the development process.

Allow them to identify exceptions, validate outputs, and help define success.

Do not assume that documentation captures every important operational rule.

Engineer failure paths deliberately

A production AI system should know when to stop, ask for clarification, defer to a human, or decline an action.

The ability to fail safely is part of reliability.

Explore emerging technologies with measurable criteria

Quantum machine learning and other new computational paradigms deserve serious research.

But production adoption should depend on reproducible advantages, not promises alone.

Together, these principles point toward a different definition of AI maturity.

It is not the number of models an organization has deployed.

It is not the complexity of its agent architecture.

And it is not the apparent intelligence of its demonstrations.

It is the organization’s ability to deliver dependable outcomes while understanding and controlling the risks.

The Real Breakthrough Will Be When AI Earns the Right to Be Trusted

There is a recurring idea throughout Daniel’s conversation with Sanjana.

Technology succeeds when it solves an important problem that people genuinely need solved.

But for generative AI, solving the problem is only part of the challenge.

The system must also establish enough confidence for people to use its results.

That confidence cannot come from impressive demonstrations alone.

It must come from evidence.

From well-defined expectations.

From careful evaluation.

From architectures that recognize uncertainty.

And from organizations willing to share the knowledge required to make their systems work.

Quantum computing may eventually change how certain computational problems are approached.

New language models will continue expanding what AI applications can accomplish.

Agentic systems will likely take on increasingly complex workflows.

But none of those developments removes the fundamental engineering responsibility to establish when a system is dependable.

Daniel’s experience offers a reminder that applies equally to conventional software, today’s generative AI, and tomorrow’s emerging architectures.

The most consequential failures often begin with assumptions nobody thought to question.

And the most valuable systems are not necessarily the ones capable of doing the most.

They are the ones that solve meaningful problems within clearly understood boundaries.

Perhaps the next great leap in artificial intelligence will not be an AI that can answer every question.

It will be an AI that knows when its answer can be trusted, can show us why, and knows what to do when it cannot.

This article is adapted from an expert conversation hosted by Sanjana Gupta, Relationship Lead at Packt, with Daniel Buchta, CEO of Avenue78, exploring production AI architecture, trust, evaluation, real-world engineering constraints, and the future of quantum machine learning.

The architectural examples and engineering recommendations expand on themes from the interview. Forward-looking quantum applications are discussed as research possibilities rather than established production capabilities.

The Architecture of Trust: Why Reliable AI Requires More Than Intelligent Models was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Robert Hawker helped uncover a critical enterprise data failure, improve data quality from 73% to 93%, and rethink what it takes to build trusted analytics with Microsoft Fabric and Power BI.

The Data Problem Nobody Saw Coming

A manufacturing company was on the verge of losing essential supplier services. Payments were delayed, supplier complaints were mounting, and the finance team was struggling to keep up. Senior leadership believed the organization needed more people to manage the growing backlog.

But the real problem wasn’t staffing, financial systems, or even the payment process itself.

It was a missing email address.

Following a hurried data migration, more than 90% of supplier records were missing the email addresses required to send remittance advices. The system was generating payment notifications, but suppliers weren’t receiving them. Without those notifications, suppliers struggled to reconcile incoming payments against outstanding invoices.

Queries multiplied. The payments team became overwhelmed. Payment backlogs grew. Eventually, suppliers threatened to withdraw services, including the electricity supply to a major manufacturing plant in the UK.

A seemingly minor data quality issue had escalated into a serious operational risk.

The most troubling part? The organization had spent considerable time treating the symptoms without recognizing the underlying cause.

This is the kind of problem that Robert Hawker, Microsoft Fabric and Power BI Consultant and Trainer at MELORO LIMITED, has encountered while helping organizations navigate enterprise data governance and analytics challenges.

In a conversation with Apeksha Shetty, Portfolio Manager at Packt, Hawker explores why organizations continue to struggle with unreliable reporting, inconsistent business metrics, and poor analytics adoption despite investing heavily in modern platforms.

Drawing on real-world experiences, including a data quality initiative that improved average overall data quality from 73% to 93%, he examines how organizations can move beyond reactive data fixes and establish sustainable governance practices.

And his perspective challenges a familiar assumption: the biggest obstacle to better analytics isn’t always the technology. It’s the business problems hidden beneath the data.

1. Why Data Quality Problems Often Disguise Themselves as Business Problems

The supplier payment crisis reveals a recurring weakness in enterprise data management.

Organizations often recognize data quality issues only after they begin affecting business operations.

An incorrect customer address becomes a delivery failure. Incomplete product information creates inventory discrepancies. Inconsistent financial definitions lead to conflicting management reports.

By the time these problems reach leadership, the conversation is rarely about data quality.

It’s about operational efficiency, customer satisfaction, compliance, or rising costs.

Hawker’s experience demonstrates why identifying the underlying cause matters more than immediately addressing the visible symptoms.

At the manufacturing organization, the initial response focused on the payments team’s inability to manage supplier queries.

But adding resources would not have prevented the same queries from recurring.

The missing remittance advice information had created a process failure that continued generating additional work.

From Operational Crisis to Measurable Data Improvement

Once the root cause became clear, data quality became a board-level priority.

Hawker helped establish a structured program focused on identifying, measuring, correcting, and continuously monitoring data quality.

Working with business process owners and data owners, the team developed rules reflecting the organization’s actual operational requirements.

They also built a tool capable of measuring quality across different data objects, quality dimensions, companies, and countries.

This provided visibility into where problems existed and whether corrective actions were producing sustained improvements.

The results were significant.

Average overall data quality improved from 73% to 93% during Hawker’s tenure, contributing to better operational and analytical outcomes.

However, the improvement wasn’t simply the result of introducing another technology tool.

It came from changing how the organization understood and managed its data.

The Practical Challenge: Business Owners Don’t Have Time to Define Every Rule

One of the biggest obstacles was obtaining input from employees who understood the business processes.

The organization was still establishing its systems and workflows, leaving business users with limited time to participate in detailed governance exercises.

Rather than waiting for stakeholders to document every requirement, Hawker’s team took a different approach.

They extracted data from the ERP system, profiled it, and identified potential business rules.

These proposed rules were then presented to process owners for validation and refinement.

This reduced the effort required from stakeholders while allowing the team to establish an initial set of quality rules quickly.

The approach illustrates an important principle for data engineering teams: business experts don’t always need to create requirements from scratch.

Sometimes, the most effective approach is to use the data itself to identify potential problems and then work with stakeholders to determine which ones matter.

The objective isn’t to make every field perfect. It’s to ensure the data is sufficiently reliable for the business processes that depend on it.

2. The Power BI Adoption Paradox: Better Dashboards, Same Old Spreadsheets

The same disconnect between technology and business requirements appears in analytics adoption.

Organizations invest in Power BI, establish reporting environments, and publish dashboards across departments.

Yet employees continue using Excel.

Leadership questions why the new reporting platform isn’t delivering the expected value.

Is the technology inadequate? Are employees resistant to change? Does the organization need additional training?

Hawker suggests that the problem often begins much earlier, during report development.

In his experience, one of the most common reasons for poor Power BI adoption is that report developers aren’t sufficiently connected to the business users they’re supporting.

Developers understand the datasets, relationships, and semantic models.

Business users understand the decisions they need to make.

But the two groups don’t always collaborate closely enough to translate those requirements into useful reporting.

When a Technically Correct Report Still Fails

Consider a sales manager investigating declining performance in a particular product category.

The analytics team delivers a dashboard showing revenue trends, regional sales, and monthly growth.

The calculations are accurate, and the visualizations are clear.

But the manager needs to understand whether the decline is caused by inventory shortages, pricing changes, or weakening customer demand.

The report contains information, but it doesn’t answer the question driving the investigation.

This illustrative example captures the problem Hawker identifies.

A technically correct dashboard isn’t necessarily a useful dashboard.

The solution begins with understanding how employees make decisions and what information those decisions require.

Hawker recommends starting with conversations involving the people who should benefit most from the reporting.

Their experiences often reveal the underlying issue.

Slow or outdated reports may indicate inefficient semantic models, failed refreshes, or poorly managed Fabric capacity.

Incorrect numbers may point to data quality problems or insufficient validation.

Multiple reports covering the same subject may indicate weak analytical governance.

And employees who prefer Excel may simply be relying on a workflow that already meets their needs.

Why Forcing Power BI Adoption Rarely Works

Organizations sometimes treat spreadsheet usage as evidence of resistance to modernization.

But employees need a meaningful reason to abandon a familiar and reliable process.

A Power BI report should offer something better: reduced manual work, stronger security, more accessible information, or improved analytical capabilities.

Hawker also cautions against restricting familiar capabilities too quickly.

Preventing users from exporting Power BI data to Excel may eventually support security and governance objectives.

But introducing such restrictions before employees trust the new reporting environment can create frustration and resistance.

The lesson is straightforward.

Analytics adoption isn’t achieved by forcing employees to use a platform. It’s achieved by making that platform more useful than the alternatives.

3. When Five Reports Produce Five Different Answers

Poor adoption is only one symptom of a broader analytical governance problem.

Another emerges when different departments develop independent reporting products for the same business subject.

A finance team creates a revenue dashboard.

Sales builds another.

Operations develops its own version.

Each report uses slightly different calculations, filters, or reporting periods.

When leadership requests a single revenue figure, the organization produces conflicting answers.

The issue isn’t necessarily that any individual report is technically broken.

It’s that the organization has allowed multiple definitions of the same business metric to develop independently.

Hawker sees this as one of the strongest arguments for enterprise semantic modeling.

Building a Shared Foundation for Business Metrics

Rather than allowing every department to recreate important calculations, Hawker recommends maintaining a relatively small number of centrally managed semantic models.

These models should be organized around meaningful business domains and contain reusable metrics, relationships, and logic.

For a streaming organization, for example, a model covering subscribers and engagement may be more useful than a collection of generic departmental models.

Reports built on top of the shared model can answer different business questions while relying on consistent underlying definitions.

This approach reduces duplication and improves confidence in reporting.

But it also raises an architectural question.

How much control should remain centralized, and how much freedom should business users have?

Hawker’s answer is to separate the responsibilities.

Centralize enterprise data engineering and semantic modeling. Decentralize report development where business users have the knowledge and skills to create useful analytical products.

Why Centralized Data Engineering Matters

Enterprise data engineering teams should manage the ingestion, transformation, integration, and preparation of important organizational datasets.

This becomes particularly valuable when business processes depend on information from multiple systems.

Manufacturing planning, for example, might require sales pipeline information from CRM, inventory data from ERP, and production lead times from operational systems.

Without coordinated engineering, different teams may repeatedly recreate these integrations.

A centralized approach establishes reliable datasets that can support multiple business requirements.

Giving Business Users Freedom Without Losing Consistency

Once trusted datasets and semantic models are available, business users can develop their own reporting products.

They can explore information, answer department-specific questions, and respond quickly to changing requirements.

However, that flexibility requires a degree of maturity.

Report developers must understand the shared models and avoid introducing conflicting versions of established metrics.

When a new metric is required, it should ideally be incorporated into the enterprise semantic model rather than repeatedly recreated inside individual reports.

Hawker also believes organizations should preserve opportunities for experimentation.

If a business user develops a valuable new data engineering or modeling approach using Microsoft Fabric, the organization should have a way to bring that innovation into the centrally managed environment.

The goal isn’t to prevent employees from experimenting.

It’s to ensure that successful innovations can become trusted, reusable enterprise capabilities.

4. The Governance Dilemma: How Much Control Is Too Much?

Governance is essential for trustworthy analytics.

But excessive governance can become another obstacle to adoption.

A report developer may spend two weeks building a useful reporting product, only to discover that the approval process requires another six weeks.

By the time the report is certified, the original business requirement may have changed.

Hawker identifies this as a common governance anti-pattern.

Organizations introduce controls to improve reporting quality, but those controls become so demanding that developers and business users struggle to deliver value.

The solution isn’t to eliminate governance.

It’s to make governance proportionate to the importance and risk of the reporting product.

Different Decisions Require Different Levels of Assurance

A prototype used to explore an idea doesn’t carry the same risk as a report supporting a major financial decision.

Similarly, internal operational reporting may not require the same level of assurance as information submitted to regulators or shared with customers.

Hawker recommends classifying reporting content according to its intended use.

Exploratory reports and small-team reporting can receive lighter controls appropriate to their limited impact.

Reports used for important internal decisions require stronger review, testing, and documentation.

External or regulated reporting requires the highest level of assurance.

The underlying quality principles remain consistent, but the depth of review changes.

For example, internal reporting may reasonably combine data from enterprise platforms with supplementary spreadsheets.

For external reporting, Hawker recommends using governed system-sourced data, with data engineering teams responsible for managing its quality.

Highly regulated industries may require additional oversight.

In pharmaceutical organizations, reporting associated with GxP compliance may need involvement from the quality function.

Why Certification Should Build Confidence, Not Create Delays

Hawker recommends a formal quality assurance process that includes peer review, testing, documentation, and consistent report design.

Reports that meet the relevant requirements can receive Power BI endorsement levels such as promoted or certified.

These classifications help employees identify reporting products that have undergone appropriate review.

But the process must remain practical.

If certification takes significantly longer than development, users may lose interest or seek alternatives.

There is also a technical reason to maintain good development standards.

Poorly optimized reports and semantic models can consume excessive Fabric capacity, affecting performance and cost.

Governance therefore protects more than the accuracy of business information.

It also supports the efficiency and sustainability of the analytical environment.

Effective governance should make trustworthy reporting easier to deliver — not harder to adopt.

5. The Overlooked Reason Trusted Reports Don’t Get Used

Even when reports are accurate and well governed, employees may struggle to find the information they need.

Large organizations can accumulate hundreds or thousands of reporting assets.

Some are actively maintained.

Others are outdated.

Several may cover similar business subjects.

Without clear guidance, employees may not know which reports are authoritative.

Hawker recommends creating a curated reporting catalogue.

This should go beyond a technical inventory generated from Fabric administration tools.

A useful catalogue explains what reports contain, which business metrics they use, who owns them, and when they should be used.

The difference is significant.

A technical inventory tells employees what exists.

A curated catalogue helps them determine which reporting product they can trust.

Hawker also recommends embedding reports into established business procedures.

In organizations with detailed standard operating procedures, such as pharmaceutical companies, the relevant Power BI reports should be referenced directly within those workflows.

This makes reporting part of everyday operations rather than a separate activity employees must remember to perform.

Another important element is a strong Power BI community.

By bringing report developers and semantic model developers together, organizations can improve knowledge sharing, encourage collaboration, and identify opportunities to standardize useful reporting practices.

The broader lesson is that analytics adoption depends on discoverability, usability, and trust — not simply the availability of dashboards.

6. Migrating to Microsoft Fabric? Don’t Rebuild the Problems You’re Trying to Escape

For organizations beginning a Microsoft Fabric migration, Hawker highlights a mistake that can undermine the value of the entire project.

Teams often treat migration as a technical replication exercise.

Existing pipelines are recreated.

Transformations are transferred.

Semantic models and reports are rebuilt to resemble their previous versions.

But the existing platform may have been designed years earlier, when the organization’s business requirements were very different.

Reproducing the same architecture without questioning its relevance can carry old inefficiencies into the new environment.

Hawker recommends using migration as an opportunity to reassess what the business actually needs.

Make the Architectural Decisions Early

Before large-scale development begins, organizations should agree on important architectural principles.

These include where data transformations will occur, how engineering environments will be layered, and when Fabric Lakehouses, Warehouses, or Materialized Lake Views should be used.

Establishing these principles early helps prevent different teams from developing incompatible patterns.

Once the architecture is agreed, Hawker recommends an agile implementation approach.

Requirements should be discovered, documented, built, and tested in manageable sprints.

Instead of waiting for a large migration to finish, teams can release smaller increments of production-ready functionality that deliver business value.

Why the Location of Data Transformations Matters

Hawker recommends preparing dimensions and facts upstream so they are ready for star-schema semantic modeling.

Where appropriate, notebooks and Lakehouses can handle transformation work before the data reaches Power BI semantic models.

This reduces the need for extensive downstream transformation logic and can support Direct Lake semantic models.

Direct Lake can also reduce dependence on the conventional scheduled refresh process associated with Import models.

However, Hawker emphasizes that technical decisions must reflect the organization’s skills.

A team experienced in Power Query but unfamiliar with PySpark may need a different implementation approach.

The best architecture is not necessarily the most sophisticated one.

It’s the architecture the organization can develop, govern, and maintain effectively.

7. The Fabric Cost Problem That Begins With Development Decisions

Microsoft Fabric brings multiple analytical capabilities into a unified environment.

But that flexibility also makes workload efficiency an important operational concern.

Hawker emphasizes that capacity management should begin on day one.

A poorly optimized notebook or semantic model may appear manageable during a pilot.

As data volumes increase, however, inefficient processing can become increasingly expensive.

He offers a straightforward example.

A notebook performing relatively simple transformations on a few thousand rows should not consume 10,000 capacity units every day without prompting an investigation.

If it does, the implementation may contain inefficiencies that will become more costly as usage grows.

Why Early Monitoring Matters

Hawker recommends implementing Fabric Unified Admin Monitoring (FUAM) early in the platform’s lifecycle.

The solution provides visibility into tenant activity and inventory, helping administrators understand how the environment is being used.

Establishing monitoring from the beginning also helps preserve historical information.

Because Microsoft’s audit logs retain activity for a limited period, organizations that delay monitoring may lose useful visibility into earlier platform activity.

This information can help teams investigate governance issues, understand usage patterns, and manage platform operations more effectively.

The larger principle applies to cloud analytics generally.

Capacity optimization isn’t something to address only after costs become a problem. It’s part of building reliable, production-ready data systems.

8. Building a Microsoft Fabric Roadmap That Delivers Value Early

Hawker cautions against applying a single implementation roadmap to every organization.

A business migrating an established data platform has different requirements from one building a data infrastructure for the first time.

Similarly, an organization adopting Fabric only for Power BI reporting will have different priorities from one implementing its full data engineering capabilities.

However, several principles remain relevant across implementations.

The journey begins with establishing administration and governance responsibilities, creating initial capacity, and confirming the business scope.

Organizations should also define architectural principles and complete necessary security and governance assessments.

Next comes requirements discovery.

Business stakeholders and technical teams need to agree on reporting requirements, enterprise data models, and the source systems required to support them.

Initial connections to ERP, CRM, and other systems can then be established in development environments.

This stage may involve resolving network dependencies, configuring gateways, and understanding source-system schemas.

Once connectivity is established, teams can begin ingesting data, documenting transformations, and preparing analytical structures.

After the required governance approvals, production connectivity and incremental development can proceed.

Hawker recommends delivering in short sprints, with small tranches of functionality reaching production and generating value early.

This approach provides opportunities to validate assumptions, identify problems, and improve the implementation before extending it across the enterprise.

Rather than measuring success solely by migration milestones, organizations can evaluate whether each release improves reporting, data reliability, or business decision-making.

9. The Most Important Governance Decision: Where Do You Start?

Perhaps the most valuable lesson from Hawker’s experience is that data governance shouldn’t begin with an attempt to fix every dataset.

Organizations often start by identifying incomplete fields, inconsistent values, and outdated records.

But without a clear business priority, these programs can become expensive exercises with uncertain outcomes.

Hawker recommends beginning with the organization’s strategy.

What is the business trying to achieve?

Which obstacles are preventing progress?

And how many of those obstacles are connected to data, even if they don’t initially appear to be?

This approach makes it possible to link governance initiatives directly to business outcomes.

Choose Quick Wins That Build Long-Term Capabilities

Hawker recommends evaluating potential initiatives based on their benefits and the effort required to deliver them.

High-value, lower-effort improvements are often useful starting points.

But he also emphasizes the importance of stakeholder support.

An early initiative involving a committed business owner can help demonstrate success and build confidence in the broader governance program.

The important condition is that the initiative must contribute to the organization’s longer-term strategy.

One of Hawker’s experiences illustrates this distinction.

In a previous role, a proposal had been developed to pay an external supplier for a one-time cleanup of supplier data using standard industry rules.

Rather than proceeding with that approach, Hawker redirected the investment toward establishing a sustainable data quality process and supporting tool.

The resulting capability used rules tailored to the organization’s requirements and supported continued quality improvement.

A one-time cleanup might have improved the data temporarily.

The repeatable process helped the organization maintain improvements over time.

The strongest governance initiatives don’t simply correct today’s errors. They reduce the likelihood of those errors becoming tomorrow’s business problems.

10. What Data Leaders Should Take Away

Across the conversation, Hawker’s experiences reveal a consistent pattern.

Organizations often invest in technology before fully understanding the business problems the technology is supposed to solve.

They build dashboards without establishing consistent metrics.

They introduce self-service analytics without defining the responsibilities of central and business teams.

They implement governance controls without considering the impact on adoption.

And they migrate platforms without questioning whether the existing architecture still meets business needs.

The alternative is a more deliberate approach.

Start with the business outcome.

Identify the data dependencies that influence it.

Establish quality rules and ownership.

Create reusable engineering and semantic modeling foundations.

Give business users the freedom to develop reports within those trusted structures.

And measure whether the resulting environment improves decisions, processes, and operational efficiency.

For organizations working with Power BI and Microsoft Fabric, this means asking questions that go beyond technical implementation.

Are employees using the reports because they trust them?

Can different departments agree on the meaning of important business metrics?

Are data quality issues identified before they disrupt operations?

Can new reporting products move through governance without unnecessary delays?

And is the platform designed to remain efficient as data volumes and usage grow?

These questions provide a stronger measure of analytics maturity than the number of dashboards published or datasets migrated.

The Final Lesson: Your Biggest Data Problem Might Not Look Like a Data Problem

The manufacturing company in Hawker’s case initially saw a supplier payment crisis.

It didn’t see incomplete data.

Organizations struggling with Power BI adoption often see a technology or training challenge.

They may not recognize the disconnect between business requirements, shared metric definitions, and reporting design.

And companies moving to Microsoft Fabric may see an opportunity to modernize infrastructure without recognizing the need to reconsider how their data environment is governed and maintained.

These challenges share a common lesson.

Modern technology can make data easier to access, process, and analyze.

But it cannot independently establish the business meaning, ownership, and trust that reliable decisions require.

Robert Hawker’s experience demonstrates why the most effective data initiatives begin with the problems organizations are trying to solve — not the tools they want to implement.

The supplier payment crisis eventually helped turn data quality into an executive priority.

The resulting program improved average overall data quality from 73% to 93%, showing what becomes possible when an organization moves beyond temporary fixes and establishes sustainable practices.

For data engineers, BI developers, analytics leaders, and organizations adopting Microsoft Fabric, the takeaway is clear.

The real measure of an analytics platform isn’t how many dashboards it produces. It’s whether the organization can trust the information enough to act on it.

About the Expert

Robert Hawker is a Microsoft Fabric and Power BI Consultant and Trainer at MELORO LIMITED. His experience spans enterprise data governance, data quality improvement, analytics adoption, semantic modeling, and Microsoft Fabric implementation. He is also the author of Practical Data Quality.

About the Host

Apeksha Shetty, Portfolio Manager at Packt, hosts this Expert Edge conversation, exploring the practical challenges organizations face when building trustworthy enterprise data environments and scaling analytics with Microsoft Fabric and Power BI.

A Missing Email Field Nearly Shut Down a Factory. The Real Problem Was Data Governance. was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS GenAI expert Evandro Franco explores the architectural decisions and engineering practices required to take agentic applications from prototypes to reliable enterprise systems.

Imagine spending weeks building an AI agent that works exactly as intended.

It understands natural-language requests, retrieves relevant information, calls external APIs, and completes complex tasks with minimal human intervention. Your development team is impressed. Stakeholders see the potential. The prototype gets approved.

Then someone asks a deceptively simple question:

“Can we deploy this for 10,000 users?”

Suddenly, the conversation changes.

What happens when hundreds of requests arrive simultaneously? What if an external API fails halfway through a workflow? How do you determine why an agent made a particular decision? Who pays when one department generates ten times more model calls than another? And how do you know whether the agent is still performing correctly after deployment?

These are not theoretical concerns. They are the engineering challenges emerging as enterprises move beyond GenAI experimentation and begin deploying autonomous applications into production.

In a conversation with Nilesh Kowadkar from Packt, Evandro Franco, Senior GenAI Data Scientist at Amazon Web Services (AWS), shared what he has observed while working with enterprise customers and focusing on Amazon Bedrock AgentCore.

His central observation reveals an important shift in the AI industry: organizations are increasingly moving from experimenting with agents to confronting the realities of operating them.

And the hardest part is no longer simply getting an agent to work.

It is making sure that the agent continues to work reliably, securely, and efficiently when the real world becomes unpredictable.

The AI Agent That Worked Perfectly — Until Everyone Started Using It

Consider an enterprise building an AI-powered customer support agent.

During development, the agent performs three straightforward operations:

1. Understands a customer request.
2. Retrieves relevant information from internal systems.
3. Generates a response or invokes a tool to complete the task.

With a handful of test users, everything looks promising.

But production introduces conditions that a controlled demonstration rarely captures.

A request might require several model invocations. Some users ask simple questions, while others trigger complex reasoning workflows. External tools introduce additional network latency. A downstream service may become temporarily unavailable. Traffic can increase dramatically during peak hours.

Now imagine the same agent handling thousands of concurrent requests.

Even if the underlying language model performs well, the application surrounding it may struggle.

Evandro highlighted scaling, resilience, and latency as three recurring challenges among enterprises moving their GenAI systems into production.

The distinction matters because model intelligence and application reliability are two different engineering problems.

A powerful model does not automatically provide predictable response times, graceful failure recovery, or efficient resource utilization.

Why latency becomes an architectural problem

Traditional applications often follow relatively predictable execution paths.

An agentic application may behave differently.

A single request can trigger multiple reasoning steps, tool calls, retrieval operations, and additional model invocations before producing a final answer.

For example:

User request → Agent reasoning → Knowledge retrieval → API call → Additional reasoning → Response

Every step introduces potential latency.

If one external API takes several seconds to respond, the entire workflow can slow down. If the agent repeatedly invokes tools because it cannot resolve an ambiguous result, response times become even less predictable.

The problem is not necessarily that the model is too slow.

It may be that the application is doing too much work.

Evandro pointed to smaller models and code optimization as potential ways to address latency challenges.

This suggests a broader engineering principle: not every operation in an agentic workflow needs the same level of model intelligence.

For example, a simple classification task might be handled by a smaller model, while a more complex reasoning task could be escalated to a more capable one.

Other practical improvements include reducing unnecessary tool calls, introducing appropriate timeouts, caching reusable results, and separating synchronous tasks from operations that can run asynchronously.

These are architectural strategies rather than specific implementations described in the interview, but they illustrate the trade-offs Evandro identified.

The engineering question is no longer simply, “Which model gives the best answer?” It is, “Which combination of models, tools, and execution strategies produces an acceptable answer within our latency and cost constraints?”

When Your AI Agent Becomes a Frankenstein of APIs, Tools, and MCP Servers

One of the most revealing moments in the conversation came when Evandro described the growing complexity of enterprise agents.

As agents accumulate APIs, tools, and Model Context Protocol (MCP) integrations, he compared them to a Frankenstein — a collection of interconnected components that becomes increasingly difficult to troubleshoot.

It is a useful description of a problem many AI engineers will recognize.

A modern enterprise agent might interact with a CRM, a database, a document repository, a search service, an internal application, and multiple external APIs.

Each integration adds functionality.

But every integration also adds another potential failure point.

Imagine an agent tasked with generating a quarterly financial report.

It retrieves sales figures from a database, obtains customer information from a CRM, queries an analytics service, and uses an LLM to summarize the results.

The final report contains incorrect revenue figures.

Where did the failure occur?

Was the database query wrong? Did the CRM return outdated information? Did the model misinterpret the retrieved data? Was a tool response truncated? Or did an earlier reasoning step introduce an incorrect assumption?

Without adequate observability, engineers may see only the incorrect final answer.

They have no reliable way to reconstruct how the agent arrived there.

Observability needs to follow the entire agent workflow

Traditional application monitoring typically focuses on metrics such as CPU utilization, request latency, error rates, and service availability.

Those metrics remain important for AI agents, but they do not fully explain agent behavior.

Engineers also need visibility into the decisions and interactions that occur within an execution.

A practical observability approach can capture:

• Execution traces: The sequence of model invocations, tool calls, and workflow transitions.
• Tool-level latency: Which external services contribute most to response delays.
• Failure information: Where timeouts, invalid responses, or unsuccessful tool executions occur.
• Model usage: How many model invocations and tokens are consumed by a task.
• Outcome signals: Whether the agent successfully completed the intended operation.

Consider a trace showing that an agent called the same search tool five times before returning an answer.

That behavior may indicate poor retrieval quality, ambiguous tool descriptions, or an ineffective planning strategy.

Without tracing, the team may assume that the language model itself is responsible for the delay.

With tracing, engineers can investigate the actual bottleneck.

This is why observability becomes more than a debugging convenience.

It is the mechanism that makes agent behavior understandable enough to improve, govern, and trust.

The Hidden Enterprise Challenge: Who Can Access What, and Who Pays?

Not every obstacle to production AI involves model performance.

Some of the most difficult challenges emerge from enterprise governance requirements.

Evandro described how large organizations often need granular controls beyond what their underlying cloud infrastructure provides by default.

For example, an organization may need to allocate AI-related costs across different business units, restrict access to particular resources, and apply internal policies within a shared environment.

These requirements become especially complicated when multiple teams use the same agent platform.

Consider a financial services company deploying an internal AI assistant.

The finance department needs access to financial reporting systems. The HR department needs access to employee-related information. The engineering team requires access to technical documentation.

All three departments may use the same underlying AI infrastructure.

But they should not have identical permissions.

An agent authorized to retrieve engineering documentation should not automatically gain access to confidential employee records simply because both systems are reachable through the same platform.

Similarly, the organization needs to understand which department is responsible for the infrastructure and model usage costs generated by each workflow.

Why cloud-level permissions may not be enough

Cloud infrastructure can provide account-level and resource-level security controls.

However, enterprises may also require application-specific authorization.

For example, two users within the same application may have different permissions to invoke a particular tool or retrieve a specific category of information.

This creates a need for additional governance layers that understand user identity, organizational context, tool permissions, and data-access policies.

A production architecture may therefore need to answer questions such as:

• Can this user invoke this particular agent capability?
• Is the agent authorized to access the requested dataset?
• Which tools can the agent execute on the user’s behalf?
• How should usage costs be attributed to a department or cost center?
• Can administrators audit sensitive operations?

Evandro explained that enterprises sometimes build additional platform capabilities on top of AWS services to meet such requirements.

That approach can provide flexibility, but it also introduces engineering responsibilities.

Teams must maintain the controls, ensure their availability, and keep them consistent as the application evolves.

The takeaway is straightforward:

An AI agent’s ability to perform an action should never be confused with its authorization to perform that action.

Production readiness requires both.

Building the Agent Is Easy. Keeping It Reliable Is the Hard Part.

Evandro drew a parallel between today’s agentic AI challenges and the earlier evolution of machine learning operations.

In traditional machine learning, training a model was only one part of the problem.

Teams also needed to deploy models, monitor performance, manage infrastructure, handle changes in data, and maintain acceptable prediction quality over time.

Agentic AI is following a similar pattern.

A successful prototype proves that an agent can complete a task under certain conditions.

A production-ready system must demonstrate that it can perform consistently across a much broader range of conditions.

That includes conditions the original developers may not have anticipated.

Resilience must be designed, not assumed

Imagine an AI agent that processes insurance claims.

It retrieves policy information, validates submitted documents, checks internal records, and prepares a recommendation.

Now suppose the document retrieval service becomes temporarily unavailable.

A poorly designed agent may continue reasoning with incomplete information and generate a confident but unsupported recommendation.

A more resilient architecture would recognize that a required dependency failed.

Depending on the business workflow, it might retry the operation, return a controlled error, request human review, or pause execution until the necessary information becomes available.

The important point is that failure should not silently become an incorrect business decision.

This is where production engineering practices become essential.

Timeout policies, bounded retries, fallback strategies, idempotent operations, and human escalation paths can help manage failures in complex workflows.

Not every application needs every mechanism. The right approach depends on the consequences of failure and the reliability requirements of the system.

But these decisions need to be made deliberately.

An agent should not be considered production-ready simply because its happy-path demonstration succeeds.

The Most Important Question: How Do You Know Your Agent Is Actually Good?

Perhaps the most consequential topic in the conversation was evaluation.

Evandro emphasized that organizations need to assess whether agents are accurately performing the tasks they were designed to complete.

This becomes especially important because evaluating agents is fundamentally different from checking whether a traditional application returned the expected output.

A conventional function might have a deterministic result.

Given the same input, it should return the same output.

An AI agent may generate different wording, choose different tool sequences, or follow alternative reasoning paths while still completing the task successfully.

This makes correctness harder to define.

Why a convincing answer is not necessarily a correct answer

Consider an AI-powered data analyst.

A user asks:

“What caused revenue to decline last quarter?”

The agent queries several datasets and produces a polished explanation.

It identifies regional underperformance, changing customer behavior, and reduced conversion rates.

The answer sounds plausible.

But was the underlying analysis correct?

Did the agent query the right tables? Did it apply the correct date filters? Did it confuse revenue with bookings? Did it overlook a missing dataset? Did it draw conclusions that the data could not support?

A fluent explanation can hide an incorrect analytical process.

This is why evaluating only the final response may be insufficient.

What should an evaluation framework measure?

A practical agent evaluation strategy can examine several dimensions.

1. Task success

Did the agent accomplish the user’s intended objective?

For a support agent, this might mean resolving the issue correctly rather than merely producing a helpful-sounding response.

2. Tool-use correctness

Did the agent select appropriate tools, provide valid arguments, and interpret their outputs correctly?

An agent that reaches the correct answer through unsafe or unauthorized tool execution may still represent a serious failure.

3. Factual accuracy

Are the agent’s conclusions supported by the available evidence?

This is particularly important in data analysis, financial workflows, and enterprise knowledge retrieval.

4. Reliability under variation

Does the agent perform consistently when user instructions are ambiguous, input data is incomplete, or external tools behave unexpectedly?

5. Operational performance

Does the workflow complete within acceptable latency and cost limits?

An agent may be accurate but operationally unsuitable if each task requires excessive model calls or takes too long to complete.

These dimensions are practical extensions of the evaluation challenge discussed by Evandro, rather than a formal framework he prescribed during the conversation.

Together, they demonstrate why evaluation cannot be treated as a final checkbox before deployment.

Evaluation should continue after deployment

A useful starting point is to build a representative test dataset containing realistic user requests, expected outcomes, and challenging edge cases.

Teams can run these evaluations before releasing changes to models, prompts, tools, or orchestration logic.

They can also monitor production outcomes to identify new failure patterns that were not represented in the original tests.

For example, an agent may initially perform well on straightforward requests but struggle when users begin asking multi-step questions that require combining information from several systems.

Those failures can become new evaluation cases.

Over time, the evaluation suite becomes a record of the problems the system must continue to solve.

The goal is not to prove that an agent never fails. It is to make failures measurable, reproducible, and actionable.

Can Managed Infrastructure Solve the Production AI Problem?

As the conversation turned toward production architecture, Evandro discussed his work with Amazon Bedrock AgentCore.

AgentCore is AWS’s managed platform for building and operating agentic applications.

Evandro explained that the team is focused on addressing the operational challenges enterprises encounter when taking agents into production.

The motivation is clear.

When organizations build their own agent platforms, they assume responsibility for infrastructure and operational capabilities that may sit outside their core business objectives.

They need to think about scaling, availability, monitoring, security, and integration with existing enterprise systems.

Managed services can reduce parts of that operational burden.

But there is an important distinction.

Managed infrastructure can simplify operating an agent. It cannot automatically make the agent’s decisions correct.

A platform may help teams manage execution environments and operational concerns, while application developers remain responsible for defining appropriate tool permissions, evaluating task outcomes, handling domain-specific failures, and ensuring that the system meets business requirements.

For engineering teams, the real architectural decision is therefore not simply whether to use a managed service.

It is determining which responsibilities should be delegated to the platform and which must remain within the application.

That distinction becomes especially important as agentic systems evolve from isolated experiments into shared enterprise capabilities.

A Practical Production-Readiness Playbook for AI Engineers

The challenges Evandro identified point toward a useful way to structure the journey from prototype to production.

Rather than treating deployment as a single milestone, engineering teams can approach it as a series of reliability decisions.

The table is an editorial synthesis of the engineering themes raised in the conversation.

It also reveals something important about production AI.

These challenges are interconnected.

Reducing latency by switching to a smaller model may affect task accuracy. Adding more tool integrations may improve functionality while increasing security and observability requirements. Introducing retries may improve resilience but increase execution time and cost.

Every architectural decision creates trade-offs.

The objective is not to optimize a single metric in isolation.

It is to build a system that meets the combined reliability, performance, security, and business requirements of its intended environment.

The Next Competitive Advantage in AI Won’t Be Better Demos

The industry has made extraordinary progress in lowering the barrier to building AI agents.

Developers can now connect models to tools, retrieve enterprise information, orchestrate workflows, and create useful prototypes with relatively little infrastructure.

But that accessibility can create a misleading impression.

If an agent can be built quickly, it may appear that deploying it at scale should be equally straightforward.

Evandro’s observations suggest otherwise.

The difficult engineering work begins when the application must operate under real-world constraints.

It must serve many users, respect organizational boundaries, recover from failures, provide useful diagnostic information, control costs, and demonstrate that its outputs are sufficiently reliable for the tasks it performs.

In other words, agentic AI is encountering the same fundamental lesson that shaped earlier generations of enterprise software:

A working feature is not the same as a reliable system.

The most valuable AI engineering skills may increasingly involve everything surrounding the model — architecture, security, observability, resilience, performance optimization, and evaluation.

These capabilities are what transform an impressive demonstration into an application that an organization can responsibly depend on.

And that leads to the question every AI engineering team should ask before its next production release:

If your agent fails tomorrow, will you know what happened, why it happened, and how to prevent it from happening again?

If the answer is no, the next priority may not be a more powerful model.

It may be better engineering.

About the expert

Evandro Franco is a Senior GenAI Data Scientist at Amazon Web Services (AWS), with approximately six years of experience at Amazon working in machine learning. His recent work has focused on Amazon Bedrock AgentCore and the operational challenges associated with enterprise agentic AI applications.

About the conversation

This article is adapted from an expert conversation between Evandro Franco and Nilesh Kowadkar of Packt, exploring enterprise GenAI adoption, production architecture, observability, scaling, security, and agent evaluation.

Production AI Agent Engineering: Solving Latency, Reliability, and Observability Challenges was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dr. Firat Gonen explains why leaderboard success doesn’t guarantee production reliability.

From Kaggle leaderboards to real-world credit decisions, Dr. Firat Gonen reveals why building successful machine learning systems has less to do with sophisticated algorithms and more to do with data reliability, explainability, and engineering decisions that survive production.

Expert: Dr. Firat Gonen, Independent Executive Advisor — AI & Data Strategy, AlphaSights | Triple Kaggle Grandmaster
In conversation with: Sanjana Gupta, Relationship Lead, Packt

The Model Was Brilliant. The Production System Wasn’t.

Imagine spending months building a machine learning model.

Your dataset is clean. Your features are carefully engineered. Your validation scores are impressive. The model consistently outperforms your baseline, and your team is confident it is ready for deployment.

Then it goes live.

A data pipeline breaks. An external API returns incomplete information. Customer behavior shifts. Predictions that looked reliable during testing begin to deteriorate.

Suddenly, the model that performed exceptionally well in development is struggling to make dependable decisions in the real world.

What went wrong?

Perhaps nothing was fundamentally wrong with the model.

The problem was everything surrounding it.

This is the uncomfortable reality of production machine learning: a highly accurate model is not necessarily a reliable business system.

And few people understand that distinction better than Dr. Firat Gonen.

An AI and data strategy executive and triple Kaggle Grandmaster, Firat has worked across fintech, insurance, telecommunications, and retail, helping organizations move beyond experimental machine learning into systems that deliver measurable business value.

In a conversation with Packt’s Sanjana Gupta, he shared the engineering decisions, production failures, and practical lessons behind an AI-powered credit risk platform designed to evaluate thousands of small businesses every day.

His central argument challenges a common assumption in machine learning:

The hardest part of ML is rarely training the model. It is building a system that continues making trustworthy decisions after deployment.

The Credit Scoring Problem Nobody Could Solve With Traditional Data

Consider two businesses applying for financing.

The first is an established corporation with years of audited financial statements, credit bureau records, and predictable revenue.

The second is a growing small business with limited formal credit history, fluctuating cash flow, and a complex network of suppliers and customers.

Traditional credit scoring systems have a much easier time evaluating the first.

But does the absence of a lengthy financial history necessarily make the second business a bad credit risk?

Not at all.

That was the fundamental challenge Firat encountered while leading the development of an AI-driven risk scoring platform at Figopara, a Turkish fintech company focused on SME financing.

Small and medium-sized enterprises often lack the historical financial records that conventional credit models rely upon.

As Firat explained during the conversation, the core problem is information asymmetry.

Lenders need reliable evidence of creditworthiness, but traditional financial records may reveal only a fraction of how a small business actually operates.

A company might have limited credit bureau history while maintaining consistent supplier payments, healthy invoice growth, and dependable transaction relationships.

Traditional models can struggle to recognize these signals.

The solution was to rethink what constituted useful financial information.

Rather than relying exclusively on established credit indicators, Firat’s team developed a machine learning platform that incorporated alternative data sources.

These included:

• Invoice flows: How invoices are generated, settled, and accumulated over time.
• Payment behavior: Patterns in how businesses meet financial obligations.
• Transaction networks: The relationships and interactions between businesses.
• Supplier relationships: Signals of commercial stability and dependency.
• Cash flow volatility: How financial activity changes across different periods.
• Sector behavior: How a company’s financial patterns compare with its industry.

Individually, these signals might reveal only part of the picture.

Together, they could provide a richer representation of business health.

And this is where machine learning offered an advantage over conventional analytics.

Why Traditional Analytics Wasn’t Enough

Traditional credit models frequently depend on established financial ratios, historical defaults, and predefined decision rules.

These methods remain valuable, particularly when data is structured, consistent, and historically representative.

But small businesses do not always behave in ways that fit neatly into predefined categories.

A business might experience seasonal invoice growth, irregular payment cycles, and changing supplier relationships without necessarily becoming less creditworthy.

Another might report relatively stable financial figures while its transaction network reveals growing vulnerabilities.

Capturing these relationships requires looking beyond isolated financial indicators.

Machine learning can identify nonlinear interactions between variables, allowing models to recognize patterns that simpler rule-based systems may overlook.

For example, cash flow volatility might carry a different risk implication depending on the company’s sector, payment history, and supplier network.

The challenge is not merely identifying whether a variable has increased or decreased.

It is understanding how several variables interact to influence risk.

Firat’s team built hundreds of behavioral and financial features to capture these relationships.

The platform was designed to evaluate financing applications at scale, typically scoring thousands of SMEs daily.

But as the system grew, a more difficult question emerged.

How do you turn hundreds of useful predictive signals into a system that lenders, engineers, and regulators can actually trust?

That question would shape nearly every architectural decision.

The Architecture: Building a Decision System, Not Just a Model

One of the most revealing moments in the conversation came when Sanjana asked Firat to walk through the system’s architecture.

His explanation showed how far a production ML application extends beyond model training.

The platform needed to collect information from multiple sources, transform that information into usable features, generate predictions quickly, and integrate the results into real financing decisions.

Its architecture followed a practical end-to-end workflow.

1. Data ingestion: Bringing fragmented information together

The first challenge was collecting data from internal transaction databases, invoice platforms, bank integrations, and external financial providers.

Each system had its own data structures, update patterns, and integration requirements.

This created an immediate reliability problem.

Even the most sophisticated prediction model cannot function consistently when its input data is missing, delayed, or incorrectly processed.

Firat identified data integration as one of the most difficult aspects of the entire project.

The team needed to combine fragmented information into a dependable foundation before meaningful predictions could be generated.

2. Centralized storage and feature engineering

Once ingested, the information flowed into a centralized data warehouse.

A feature engineering layer then transformed the raw information into hundreds of financial and behavioral indicators.

These features represented the underlying signals that the risk models would use.

But feature engineering introduced another trade-off.

More features could potentially improve predictive performance.

They could also create more dependencies, greater maintenance requirements, and additional opportunities for data failures.

A feature that improves offline accuracy but frequently fails during live scoring may ultimately make the production system less useful.

3. Model training: Choosing practicality over complexity

The team experimented with supervised learning approaches, with gradient boosting models forming an important part of the solution.

These models were particularly well suited to structured financial data.

Rather than pursuing architectural complexity for its own sake, the team focused on models that offered strong predictive performance while remaining operationally manageable.

4. Real-time scoring and business integration

The trained models were exposed through an API layer and integrated into the company’s financial decision engine.

When an SME applied for financing, the system could generate the necessary features and produce a risk score as part of the decision process.

This made latency an important architectural constraint.

A risk model might be extremely accurate, but if it takes too long to return a result during an application, it can become an operational bottleneck.

5. Monitoring, drift detection, and retraining

Finally, the platform monitored model performance, changing data patterns, and portfolio outcomes.

This was especially important because the Turkish financial market could experience significant volatility.

A model trained on historical business behavior could gradually become less reliable as economic conditions changed.

Monitoring and retraining therefore became essential parts of maintaining the system.

Firat described the result as something much bigger than a machine learning model.

He called it an entire decision infrastructure.

That distinction matters.

The model produces a prediction.

The surrounding infrastructure determines whether that prediction arrives on time, reflects current conditions, can be explained, and is useful enough to influence a financial decision.

The Trade-Off Most ML Teams Eventually Face: Accuracy vs. Explainability

What if a more sophisticated model delivers slightly better predictions but nobody can adequately explain its decisions?

For many ML applications, that question is inconvenient.

For financial services, it can become a serious operational and regulatory concern.

Credit decisions must be understandable to risk teams, management, and regulators.

A lender cannot simply rely on a complex prediction without being able to investigate the factors influencing it.

This constraint influenced Firat’s modeling strategy.

Deep neural networks were a possible option, but their additional complexity did not necessarily justify the explainability challenges for this particular structured-data problem.

Instead, the team prioritized gradient boosting models and used SHAP-based interpretation techniques to understand the contributions of different features.

SHAP can help explain why a model assigns a particular prediction by estimating how individual features contribute to its output.

That does not automatically make a model fair, compliant, or causally interpretable.

But it provides a useful layer of transparency for investigating predictions and communicating model behavior.

The team’s philosophy was straightforward:

A small improvement in predictive accuracy may not be worth sacrificing the reliability and transparency required to operate the model responsibly.

There was another practical consideration.

The project was developed in a startup environment where budgets mattered.

Rather than investing heavily in proprietary tooling, the team relied on Python and open-source machine learning libraries.

This was not simply about reducing expenses.

It was about choosing technology that matched the problem.

Structured financial data did not automatically require deep learning architectures designed for images, audio, or other complex inputs.

For this use case, tree-based models offered a compelling balance between predictive power, interpretability, and operational simplicity.

The lesson for ML engineers is important.

The best architecture is not necessarily the most advanced one. It is the architecture that satisfies the real constraints of the business.

The Failure That Changed How the Team Thought About Monitoring

Even with a carefully designed architecture, the system encountered a familiar production problem.

Data drift.

In the early versions of the platform, the team observed that some financial and behavioral features began behaving differently during periods of market volatility.

The model had learned relationships from historical information.

But the underlying economic environment was changing.

Payment behavior shifted. Financial patterns evolved. Relationships that had previously been predictive became less dependable.

The result was a modest degradation in predictive performance.

This exposed an important limitation of conventional model development.

A model can be trained correctly, validated carefully, and deployed successfully while still becoming less reliable over time.

The problem is not necessarily a coding error.

It may be that the world the model learned from no longer behaves in the same way.

For Firat’s team, this meant revisiting how training data was selected, how validation was performed, and how models were monitored and retrained.

It also raised a deeper question.

How do you know when an ML system is failing before that failure becomes a business problem?

Technical metrics tell only half the story

Most machine learning teams are comfortable monitoring technical measures such as precision, recall, or F1 score.

These metrics help evaluate how well a model performs against labeled outcomes.

But they are not always meaningful to the people responsible for business decisions.

A risk manager may not immediately understand the operational significance of a change in F1 score.

A collections team is more likely to care whether the platform is identifying risky accounts effectively.

Leadership may want to understand whether automated decisions are reducing manual workloads while maintaining acceptable portfolio outcomes.

Firat emphasized the need to connect technical performance with business-oriented measures.

In practice, this means evaluating models through two complementary lenses.

Technical performance: Is the model still producing accurate, stable, and useful predictions?

Business performance: Are those predictions improving the decisions and outcomes the organization actually cares about?

Neither perspective is sufficient on its own.

A model might perform well on a technical metric while creating operational friction.

Conversely, an apparently healthy business metric might conceal a deteriorating model that has not yet produced visible downstream consequences.

Production monitoring becomes more valuable when both perspectives are considered together.

What Actually Improved After Deployment?

The most important outcome of the project was not simply a better prediction score.

It was a more scalable financing decision process.

According to Firat, the platform improved the company’s ability to predict SME credit risk and automated substantial portions of the evaluation workflow.

That reduced manual review requirements, helped limit human errors, and allowed the company to evaluate many applications in parallel.

It also supported broader access to financing for small businesses that were difficult to assess through traditional credit models.

Firat did not provide a specific percentage improvement in default prediction accuracy or a quantified reduction in manual processing during the interview.

But he emphasized the operational shift.

A human analyst can review a limited number of applications at a time.

A properly engineered decision platform can process many applications concurrently, while applying consistent scoring logic and retaining the ability to monitor outcomes.

This illustrates a critical difference between a successful ML experiment and a successful ML product.

The experiment demonstrates predictive capability.

The product converts that capability into a repeatable business process.

And the value comes not only from making better predictions, but also from making useful decisions faster, more consistently, and at greater scale.

The Kaggle Paradox: Why Winning Models Can Fail in a Week

Perhaps the most surprising insight came when the conversation turned to Firat’s experience as a triple Kaggle Grandmaster.

Kaggle is one of the world’s best-known platforms for competitive machine learning, bringing together practitioners who compete to solve difficult prediction problems.

Firat has spent years participating in that community, learning modeling techniques, studying other competitors’ solutions, and improving his ability to extract performance from data.

But he was quick to distinguish competition success from production engineering.

In a typical machine learning competition, participants receive a defined dataset.

The objective is clear.

The evaluation metric is established.

And the challenge is to build a model that achieves the strongest possible predictive performance.

Competitors may spend enormous effort improving results by tiny margins.

Production environments are fundamentally different.

Datasets are not necessarily clean.

Inputs are not static.

APIs fail.

Business requirements change.

Stakeholders need explanations.

And a model’s usefulness depends on much more than its leaderboard performance.

Firat summarized the difference with a striking observation:

“A model that can win a Kaggle competition, it may not survive one week in production.”

Consider what happens when a highly optimized model depends on a feature that becomes unavailable.

Or when its input distribution changes unexpectedly.

Or when its predictions cannot be explained to the people making the final business decisions.

None of those problems necessarily appears in an offline competition.

Yet any one of them can undermine a production deployment.

This does not diminish the value of competitive machine learning.

Firat credits Kaggle with strengthening his modeling foundations and teaching him to learn from talented practitioners.

But it highlights a distinction that every aspiring ML engineer should understand.

Competition teaches you how to build a stronger model. Production teaches you how to build a system that can survive.

If He Built the System Again Today, What Would Change?

Looking back, Firat identified several improvements he would consider if he were rebuilding the platform.

The first would be investing earlier in a more mature machine learning platform layer.

That would include capabilities such as automated feature storage, model registries, version management, and stronger monitoring infrastructure.

These tools become increasingly important as an organization moves from managing one model to operating multiple models across different business processes.

Without consistent infrastructure, teams can struggle to track which model is deployed, which features it uses, how its performance has changed, and when it should be retrained.

A more mature platform helps make these operations systematic rather than dependent on individual engineers.

Where would LLMs fit?

Firat also sees opportunities for large language models to help process unstructured financial information.

For example, LLM-based components could potentially extract relevant information from financial reports, documents, and other text-heavy sources.

That information could then complement the structured financial features used in conventional risk models.

The opportunity is not necessarily to replace a reliable credit scoring model with a generative AI system.

It is to use newer AI capabilities where they solve a specific data-processing problem.

For a regulated financial application, extracted information would still require appropriate validation, governance, and integration before influencing decisions.

This reflects the same principle that guided the original platform.

Use the technology that solves the problem, rather than redesigning the problem around the newest technology.

And despite the advances in ML platforms and generative AI, Firat believes the fundamentals remain unchanged.

Successful machine learning systems require dependable data infrastructure, sound modeling decisions, and close collaboration with the business.

The Next Generation of ML Engineers Will Need More Than Modeling Skills

As the conversation approached its conclusion, Sanjana asked a forward-looking question.

What skill will define the strongest applied machine learning engineers over the next three years?

Firat’s answer was not a particular framework, model architecture, or programming language.

It was the ability to combine foundational machine learning knowledge with automation and production engineering skills.

The distinction between a data scientist and an ML engineer is becoming increasingly important.

Knowing how to train a model remains valuable.

But organizations also need practitioners who understand how models interact with data systems, software infrastructure, operational processes, and business requirements.

An engineer who can produce a convincing prototype has solved only part of the challenge.

An engineer who can turn that prototype into a reliable production component brings a different level of value.

Firat also stressed the importance of understanding what happens beneath the abstractions.

It is easy to import a library, call a model, or integrate an AI service.

It is much harder to diagnose a failure when the underlying mechanism is poorly understood.

As automation makes model development more accessible, that deeper understanding may become even more important.

The engineers who stand out will be those who can explain not only what their systems produce, but also how they work, why they fail, and how to make them dependable.

Five Lessons for Anyone Building Production ML Systems

Firat’s experience offers a practical framework for ML practitioners navigating the transition from experimentation to production.

1. Design around the decision, not the algorithm.
Start by understanding what the business needs to decide, what information is available, and what constraints the system must satisfy. Model selection should follow those requirements.

2. Treat data pipelines as critical infrastructure.
An accurate model cannot compensate for unreliable inputs. Data ingestion, integration, feature consistency, and API reliability deserve as much attention as model development.

3. Optimize for explainability and operational stability.
Additional complexity is justified only when its benefits outweigh the cost of maintaining, explaining, and governing the system.

4. Monitor changing behavior, not just model availability.
A model can remain technically operational while becoming less accurate. Track changing data patterns, prediction quality, and relevant business outcomes.

5. Measure success through real-world impact.
Offline accuracy matters, but so do decision speed, operational efficiency, stakeholder trust, and the ability to maintain performance as conditions change.

These principles are relevant far beyond credit scoring.

They apply to fraud detection, customer churn prediction, demand forecasting, recommendation systems, insurance analytics, and many other applications where machine learning influences real decisions.

The Real Competitive Advantage Isn’t a Better Model

The machine learning industry has become remarkably good at building models.

Powerful open-source libraries, pretrained architectures, automated experimentation tools, and increasingly accessible compute have reduced many of the barriers to developing sophisticated predictions.

But building models has never been the entire challenge.

The harder problem is making them useful under real-world conditions.

Firat’s SME financing platform demonstrates what happens when machine learning is approached as an engineering and business discipline rather than an isolated modeling exercise.

The team did not simply train a risk classifier.

It brought fragmented data together, engineered meaningful financial signals, selected interpretable algorithms, integrated predictions into live financing workflows, and adapted the system as market behavior changed.

The result was a platform that helped improve risk evaluation while supporting faster, more scalable financing decisions.

And perhaps the most valuable insight is that none of this depended on using the most complex model available.

It depended on making the right engineering trade-offs.

For ML practitioners, the question worth asking is no longer just:

How accurate can I make this model?

It is:

Can I build a system that continues delivering trustworthy decisions when the data changes, the APIs fail, the business scales, and the real world refuses to behave like my training dataset?

That is where machine learning stops being an experiment and starts becoming valuable infrastructure.

This article is adapted from an expert conversation hosted by Sanjana Gupta at Packt with Dr. Firat Gonen, Independent Executive Advisor — AI & Data Strategy at AlphaSights and triple Kaggle Grandmaster. The discussion explores production machine learning, SME credit risk modeling, explainability, data drift, architectural trade-offs, and the evolving skills required of applied ML engineers.

A Kaggle Grandmaster’s Warning: Your Best Model Might Fail was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Somil Rastogi, Principal Solutions Architect at SAS, joins Deepesh Patel, Relationship Lead at Packt, to explore why multi-agent AI systems become harder to control as they scale, and how better orchestration, shared context, and architectural decisions can turn promising prototypes into reliable enterprise applications.

Imagine you’ve built three AI agents.

The first retrieves information from your enterprise knowledge base. The second analyzes it and recommends an action. The third executes that action through a connected business application.

Individually, each agent performs well. Your tests pass. The demo impresses stakeholders.

Then you connect them.

Suddenly, the retrieval agent returns incomplete context. The analysis agent confidently fills in the gaps. The execution agent acts on that recommendation before anyone notices the problem.

Three capable agents. One unreliable system.

This is the uncomfortable reality behind the excitement surrounding multi-agent AI. While frameworks are making it easier to build autonomous agents, getting those agents to collaborate reliably introduces an entirely different class of engineering problems.

And those problems become significantly more serious when agents move beyond generating responses and begin interacting with enterprise data, business workflows, and production systems.

In a Packt expert conversation hosted by Deepesh Patel, Relationship Lead at Packt, Somil Rastogi, Principal Solutions Architect at SAS, brings an enterprise architecture perspective to the challenge of designing multi-agent AI systems.

The discussion centres on a question that every AI engineer and solution architect should be asking:

If individual agents are becoming more intelligent, why does the architecture connecting them remain so difficult to get right?

To understand the answer, we need to look beyond models and prompts and examine what actually happens when autonomous components must work together.

1. Why Adding More Agents Doesn’t Necessarily Make Your System Smarter

The appeal of multi-agent AI is easy to understand.

Instead of relying on a single model to perform an entire workflow, developers can assign specialized responsibilities to different agents.

One agent handles research. Another performs analysis. A third plans actions. A fourth verifies the results.

It resembles the way human teams divide responsibilities.

But there’s a critical difference.

Human teams rely on shared understanding, established processes, accountability, and the ability to clarify ambiguous instructions.

AI agents don’t automatically possess these coordination mechanisms.

They operate within the information, tools, permissions, and instructions their architecture provides.

Consider a financial research application.

A data retrieval agent collects market information. A forecasting agent generates projections. A reporting agent produces an executive summary.

What happens if the retrieval agent provides outdated financial data?

The forecasting agent might generate a mathematically valid prediction using incorrect inputs. The reporting agent could then present that prediction as a credible business insight.

Each component may appear to have completed its assigned task successfully.

Yet the overall system has failed.

This is one of the most important distinctions in multi-agent engineering:

Individual agent performance and end-to-end system reliability are not the same thing.

The architecture must account for what happens between agents, not just what happens inside them.

That includes how tasks are delegated, how outputs are validated, how context is transferred, and how the system responds when an agent makes a mistake.

For enterprise architects, the challenge is not simply creating more specialized agents.

It is establishing the mechanisms that allow those agents to behave as a coherent, dependable system.

2. The Orchestration Problem: Who Decides What Happens Next?

Suppose a customer asks an AI-powered financial assistant to analyze their portfolio, identify potential risks, and recommend adjustments.

The request involves several activities.

The system must retrieve portfolio information, analyze market exposure, evaluate risk, and generate a recommendation.

But who decides which agent performs each task?

Who determines whether the output is good enough to proceed?

And what happens when two agents disagree?

This is where orchestration becomes central to multi-agent architecture.

There are several ways to coordinate agents, each with different implications for reliability and control.

A central supervisor controls the workflow

In a supervisor architecture, one coordinating component delegates tasks to specialized agents.

For example, a supervisor might send a research task to one agent, pass its output to an analytical agent, and then request validation before producing a final response.

This provides a relatively clear execution structure.

It also gives developers a central point for enforcing workflow rules.

However, it introduces a dependency on the supervisor’s decisions. If the supervisor routes tasks incorrectly or fails to recognize an invalid result, the entire workflow can be affected.

Agents operate in a predefined sequence

A sequential architecture works more like a production line.

Each agent completes a defined task and passes its output to the next stage.

This can be effective for document processing, reporting, and other workflows where the order of operations is predictable.

But it also means that an early mistake can influence every subsequent step.

Without intermediate validation, errors can travel through the workflow unnoticed.

Agents collaborate dynamically

More flexible architectures allow agents to communicate, exchange information, and decide how to approach a task.

This can be valuable for complex research or planning problems where the next step cannot always be determined in advance.

However, flexibility introduces uncertainty.

Agents may repeat work, disagree over responsibilities, or enter unnecessary communication loops.

The result can be higher costs, longer execution times, and more difficult debugging.

The architectural decision therefore depends on the nature of the problem.

The goal isn’t to maximize agent autonomy. It’s to introduce enough autonomy to solve the problem while retaining control over how the solution is produced.

For developers, this means choosing orchestration patterns based on business requirements rather than selecting the most sophisticated framework available.

3. Your Agents May Be Sharing Information, but Are They Sharing the Same Context?

One of the less obvious challenges in multi-agent systems is maintaining a consistent understanding of the task.

Imagine an insurance claims application with three agents.

The first extracts information from a submitted claim. The second checks policy eligibility. The third recommends the next action.

Now imagine the policy information changes between the second and third stages.

The eligibility agent may be operating on an older version of the policy, while the recommendation agent has access to newer information.

Both agents could produce reasonable outputs based on what they know.

But the system may arrive at an inconsistent decision.

This is not primarily a language-model problem.

It is a data consistency and state-management problem.

As multi-agent applications become more sophisticated, developers must decide what information agents should share, where that information should be stored, and which source should be considered authoritative.

Shared memory isn’t the same as reliable memory

Giving agents access to a common memory store might seem like an obvious solution.

But shared memory introduces questions of its own.

What information should be retained?

Which agent can update it?

How are conflicting updates resolved?

What happens when stored information becomes outdated?

And how can developers distinguish between verified enterprise data and an interpretation generated by another agent?

These distinctions matter because an agent-generated summary is not necessarily equivalent to an authoritative business record.

For example, a customer-support agent might summarize a previous interaction as “the customer is eligible for a refund.”

But that summary should not automatically override the organization’s actual refund policy or transaction history.

A more dependable architecture separates temporary conversational context from persistent business state and authoritative data sources.

It also defines explicit rules for reading, updating, and validating shared information.

The important question isn’t how much information an agent can remember. It’s whether the agent is using the correct information when making a decision.

This becomes especially important in enterprise environments where agents interact with databases, analytics platforms, document repositories, and continuously changing operational systems.

4. The Enterprise Data Problem: An Agent Is Only as Reliable as the Information It Uses

One of the biggest differences between an impressive AI prototype and a production-ready enterprise application is the environment in which it operates.

A prototype might use a small collection of documents and a few carefully designed examples.

A production application may need to work across thousands of documents, multiple databases, legacy systems, internal APIs, and business-specific access policies.

The difficulty is not simply retrieving information.

It is retrieving the right information, interpreting it correctly, and using it within the appropriate business context.

Consider a multi-agent system designed to support enterprise risk analysis.

One agent retrieves operational data. Another analyzes historical patterns. A third evaluates potential risks. A fourth prepares recommendations.

If the underlying data is incomplete or inconsistent, the system may produce convincing but misleading conclusions.

Adding more reasoning agents does not fix unreliable source data.

In fact, it can make the problem harder to detect because each agent may transform the original information before passing it downstream.

This is where techniques such as retrieval-augmented generation, structured data access, metadata management, and deterministic validation become important.

Rather than allowing agents to rely entirely on generated interpretations, architects can connect them to governed enterprise information sources.

For example:

• A retrieval agent can locate relevant documents and preserve source references.
• An analytical component can execute structured queries against validated datasets.
• A policy engine can evaluate deterministic business rules.
• A reasoning agent can interpret the combined evidence and explain the result.
• A validation component can check whether the proposed action satisfies the required constraints.

These responsibilities do not necessarily require five separate LLM-powered agents.

Some may be better implemented as conventional software services.

That distinction is essential.

Not every intelligent workflow component needs to be an AI agent. Sometimes the most reliable part of an agentic architecture is the part that doesn’t use an LLM at all.

The practical solution is to combine probabilistic reasoning with deterministic systems wherever correctness and accountability matter.

5. When an Agent Makes a Mistake, How Does the System Know?

Traditional software applications are generally built around defined execution paths.

Developers know which functions are called, what inputs they expect, and how failures should be handled.

Agentic systems introduce more variability.

An agent may choose a different tool, interpret an instruction differently, or generate an unexpected intermediate result.

In a multi-agent workflow, that variability can influence the behaviour of other agents.

Consider an automated incident investigation system.

A monitoring agent detects an anomaly.

A diagnostic agent investigates potential causes.

A remediation agent recommends corrective action.

What happens if the diagnostic agent incorrectly identifies the root cause?

Should the remediation agent proceed?

Should another component verify the diagnosis?

Should the system request human approval?

These questions reveal why validation and failure recovery need to be built into the architecture.

A successful response isn’t enough

Many AI applications are evaluated primarily on whether they produce a correct final answer.

For multi-agent systems, that approach is incomplete.

A workflow might produce the correct answer while making unnecessary tool calls, ignoring a required validation step, or exposing information to an agent that should not have access to it.

Conversely, a workflow might fail to complete a task but correctly recognize the failure and escalate it to a human.

From an enterprise perspective, that second outcome may be preferable to an incorrect autonomous action.

Evaluation therefore needs to consider more than answer quality.

Developers should examine whether agents perform their assigned responsibilities correctly, whether information is transferred accurately, whether tools are used appropriately, and whether the overall workflow satisfies business requirements.

The system should also make failures observable.

Engineers need to understand which agent acted, what information it received, which tools it called, and why the workflow reached a particular outcome.

This requires tracing and logging across the entire agent execution process.

Without that visibility, debugging becomes difficult because the source of an error may be several agent interactions away from the final result.

A reliable multi-agent system isn’t one that never fails. It’s one that can detect failures, contain their impact, and recover without creating a larger problem.

6. More Agents, More Tokens, More Latency: The Economics of Multi-Agent AI

There is another consequence of multi-agent architecture that often becomes visible only after deployment.

Cost.

Every agent interaction can introduce additional model inference, context processing, retrieval operations, and tool execution.

A workflow that appears simple from the user’s perspective may involve multiple rounds of internal communication.

Imagine a research agent collecting information, a planning agent reviewing it, and a validation agent requesting revisions.

The research agent retrieves more information.

The planning agent updates its recommendation.

The validation agent requests another check.

The process repeats.

While each interaction may be relatively inexpensive, the cumulative cost can grow quickly.

Latency becomes another concern.

A user asking a straightforward question may not be willing to wait while several agents exchange messages and repeatedly reconsider their decisions.

This creates a practical engineering trade-off.

More agent collaboration may improve certain complex outcomes, but it can also increase processing time and operational cost.

The solution is not necessarily to reduce every workflow to a single agent.

It is to make autonomy selective.

Developers can establish limits on retries, define clear termination conditions, use smaller models for simpler tasks, and route predictable operations through conventional software.

They can also evaluate whether an additional agent produces a measurable improvement in task success.

If it doesn’t, that agent may be adding complexity without adding value.

The most sophisticated multi-agent system is not necessarily the most effective one. The better architecture is the one that achieves the required outcome with acceptable reliability, latency, and cost.

7. The Security Challenge Changes When Agents Can Take Action

An AI assistant that generates an incorrect response can create confusion.

An autonomous agent that executes an incorrect action can create operational consequences.

This difference becomes critical when agents gain access to enterprise systems.

Imagine an agent responsible for reviewing customer accounts.

It may need permission to retrieve account information and identify discrepancies.

But should it be allowed to modify account balances?

Should a procurement agent be able to approve purchases?

Should a data-analysis agent have permission to update production databases?

These decisions cannot safely depend on an LLM interpreting instructions correctly every time.

They require enforceable security controls.

A well-designed agentic architecture limits each agent’s permissions to the operations necessary for its role.

It also separates recommendations from actions that carry significant business consequences.

For example, an agent may be permitted to prepare a payment recommendation but require an authorized employee to approve the transaction.

Similarly, an agent can identify a potentially suspicious claim without automatically rejecting it.

This approach allows organizations to benefit from automation while maintaining control over sensitive operations.

Another concern is the trustworthiness of external information.

Agents frequently retrieve documents, websites, messages, and tool outputs.

Those sources may contain misleading or malicious instructions.

If an agent treats retrieved content as authoritative instructions, an attacker may be able to influence its behaviour.

Architectures therefore need to distinguish between information the agent should analyze and instructions the agent is authorized to follow.

The underlying principle is straightforward:

An agent’s ability to reason about an action should never be confused with its authority to perform that action.

As multi-agent systems become more deeply integrated into enterprise workflows, this distinction will become increasingly important.

8. Do You Really Need a Multi-Agent System?

After exploring orchestration, shared memory, validation, cost, and security, one question deserves particular attention.

What if the best solution is not a multi-agent system at all?

The growing popularity of agentic AI frameworks can encourage developers to break every workflow into multiple autonomous components.

But distributing work across agents introduces communication overhead and new failure modes.

For many applications, a single agent with well-designed tools may be sufficient.

For others, a deterministic workflow with one reasoning component may offer better reliability.

Multi-agent systems become more valuable when a problem genuinely requires specialized reasoning, independent subtasks, different tool permissions, or coordinated analysis across multiple domains.

Consider two examples.

Scenario A: Summarizing a customer support ticket

The application retrieves the ticket, examines relevant support history, and generates a summary.

A single agent with retrieval capabilities may be enough.

Introducing separate retrieval, interpretation, summarization, and review agents could increase complexity without meaningfully improving the result.

Scenario B: Investigating a complex enterprise security incident

The application must examine logs, correlate alerts, investigate potential causes, evaluate remediation options, and produce an auditable recommendation.

Here, specialized components may offer meaningful advantages because the task involves distinct capabilities, tools, and validation requirements.

Even in this scenario, the architecture should determine which responsibilities need autonomous reasoning and which can be handled by deterministic services.

The key is to start with the problem rather than the framework.

Before adding another agent, developers should be able to explain what unique responsibility it performs, why that responsibility benefits from autonomous reasoning, and how its performance will be evaluated.

If those questions cannot be answered, the additional agent may not be necessary.

9. What Does a Production-Ready Multi-Agent Architecture Actually Need?

For engineering teams moving from prototypes to enterprise deployment, the practical challenge is translating these principles into a repeatable architecture.

A production-ready system needs several capabilities working together.

Clear task boundaries

Each agent should have a defined responsibility, with explicit expectations for its inputs, outputs, and available tools.

Controlled orchestration

The workflow should establish how tasks are assigned, when agents can act, and what conditions trigger retries, escalation, or termination.

Reliable context management

Agents need access to relevant information without losing track of authoritative data, shared state, or previous decisions.

Governed tool access

Permissions should be enforced outside the model, particularly when agents interact with sensitive data or execute business actions.

Validation and evaluation

The architecture should check intermediate outputs and assess both individual agent performance and end-to-end workflow outcomes.

Operational observability

Engineers should be able to inspect agent decisions, trace tool calls, measure latency, and identify the source of failures.

Cost and resource controls

The system should limit unnecessary interactions and use the appropriate level of model capability for each task.

These requirements may sound familiar to experienced software architects.

That’s because many of the hardest multi-agent AI challenges resemble problems that distributed systems engineering has been addressing for years.

State management, fault tolerance, access control, observability, and coordination are not new concerns.

What changes with agentic AI is that some components make probabilistic decisions rather than following entirely predetermined execution paths.

That makes architectural discipline more important, not less.

10. The Next Phase of Agentic AI Is About Trust, Not Just Autonomy

The excitement surrounding agentic AI often focuses on increasingly autonomous systems that can plan, reason, and act with minimal human involvement.

But enterprise adoption depends on something more fundamental.

Trust.

Organizations need confidence that an AI system will use appropriate information, follow business policies, respect permissions, and behave predictably enough to operate within critical workflows.

That does not mean every decision must be deterministic.

It means the surrounding architecture must establish clear boundaries for autonomous behaviour.

For some applications, agents may be trusted to complete tasks independently.

For others, the appropriate role of an agent may be to gather evidence, analyze alternatives, and recommend an action for human approval.

The right level of autonomy depends on the consequences of failure.

An agent that organizes internal research documents presents a different risk profile from one that initiates financial transactions or modifies customer records.

Treating both systems as equally autonomous simply because the technology permits it would be a mistake.

The more useful question is how much autonomy produces meaningful value without introducing unacceptable operational risk.

This is where enterprise architecture becomes the foundation for sustainable agentic AI adoption.

The Bigger Picture: Stop Building More Agents. Start Building Better Systems.

The transition from single-agent applications to multi-agent systems represents an important development in AI engineering.

But it also exposes a fundamental misconception.

Connecting several intelligent agents does not automatically create an intelligent system.

The real engineering work happens in the architecture connecting those components.

It happens in the orchestration logic that determines what happens next.

In the data architecture that ensures agents operate on trustworthy information.

In the validation mechanisms that detect errors before they propagate.

And in the security controls that determine what an agent is allowed to do.

For AI engineers, the challenge is evolving from prompt design and model integration toward building reliable, observable, and governable systems.

For solution architects, it means applying established software engineering principles to applications that incorporate probabilistic reasoning and autonomous decision-making.

And for organizations investing in agentic AI, it means evaluating success through measurable outcomes rather than the number of agents deployed.

The enterprise architecture perspective brought into focus through Somil Rastogi’s expertise at SAS offers a useful way to examine this transition.

As AI systems become more capable, the importance of their surrounding architecture does not diminish.

It grows.

Because the question that ultimately matters isn’t how many agents an organization can deploy.

It’s whether those agents can work together reliably enough to be trusted with real business responsibilities.

And that may be the defining challenge of the next generation of enterprise AI.

About the Expert

Somil Rastogi is a Principal Solutions Architect at SAS, with a focus on enterprise technology and solution architecture. His expertise provides an enterprise-oriented perspective on agentic AI architecture, multi-agent systems, and the challenges of designing scalable AI applications.

About the Host

Deepesh Patel is a Relationship Lead at Packt, connecting industry experts and practitioners to explore emerging technologies, practical engineering challenges, and the ideas shaping modern software development.

Packt Expert Conversations brings together industry practitioners and technical leaders to explore the engineering decisions behind emerging technologies, helping developers move beyond the hype and build solutions that work in the real world.

Join 150K+ data scientists, AI engineers, BI leaders, analysts, and finance professionals staying ahead of the trends shaping the future of data and AI.

👉 Subscribe to DataPro: [Packt DataPro | Substack]
👉 Subscribe to BIPro: [Packt BI-Pro | Substack]
👉 Subscribe to AlgoFinance: [Packt AlgoFinance | Substack]

Adding More AI Agents Won’t Fix Your Workflow. Here’s What Might. was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

How one enterprise transformed hundreds of thousands of unstructured insurance documents into an LLM-ready data platform, reduced manual processing time by 97%, and discovered why clean data alone is not enough for production AI.

Interview hosted by Vaideeshwari Roshan, Portfolio Manager, Packt | Featuring Jennifer Ratten, VP, Application Engineering | Data Strategy and Practice, NFP, an Aon company

Your dashboards are accurate. Your SQL queries return the right numbers. Your data pipelines are running smoothly.

Then you connect a large language model, ask it a seemingly straightforward question, and suddenly the answers are incomplete, inconsistent, or simply wrong.

What changed? The data didn’t. The way it needed to be understood did.

This is one of the most overlooked challenges in enterprise AI. Data engineered for reporting, analytics, and business intelligence is not necessarily ready for retrieval, reasoning, or conversational AI.

And the consequences extend far beyond a disappointing chatbot response. In industries where accuracy, compliance, and trust matter, a poorly grounded answer can undermine an entire AI initiative.

In a conversation hosted by Vaideeshwari Roshan, Portfolio Manager at Packt, Jennifer Ratten, VP of Application Engineering, Data Strategy and Practice at NFP, an Aon company, shared what it actually takes to build an LLM-ready data foundation.

Drawing on a production implementation involving hundreds of thousands of insurance documents, Ratten explained how her team combined document intelligence, retrieval-augmented generation (RAG), semantic modeling, structured databases, and AI agents to transform a highly manual process into a scalable AI-enabled workflow.

The result was striking: approximately 97% less manual processing time, achieved by rethinking how data was prepared, structured, validated, and retrieved.

But the most valuable lesson wasn’t about the model or the technology stack.

It was about understanding the data well enough to make AI trustworthy.

The Business Problem: Thousands of Health Plans, Hundreds of Thousands of Documents, and Too Much Manual Work

Before moving into technology, Ratten spent nearly two decades working with insurance agencies. Her career began on the business side, where operating within a small organization made efficiency a necessity rather than an aspiration.

She started automating repetitive tasks through macros and reporting, gradually moving into analytics, Power BI, and Microsoft’s Power Platform. As AI capabilities evolved, she began applying them to operational problems that traditional automation could not easily solve.

Following her company’s acquisition, she transitioned fully into technology, focusing on building and scaling solutions using Microsoft Fabric and related technologies.

That combination of business experience and technical expertise became particularly important when tackling one of her most demanding projects.

The organization was processing hundreds of thousands of unstructured documents, including PDFs, Word files, and text documents containing detailed health insurance plan information.

Each document could contain different terminology, layouts, coverage rules, and benefit structures.

Insurance agents were responsible for reviewing these documents manually, extracting relevant details, and entering the information into multiple downstream systems.

The challenges were predictable but significant:

• Manual extraction: Employees spent substantial time locating and interpreting information buried inside documents.
• Inconsistent terminology: Similar benefits could be described differently across providers and plan types.
• Frequent updates: Changes to health plans required information to be reviewed and updated repeatedly.
• Limited scalability: Growing document volumes increased operational effort and the risk of errors.
• Fragmented access: Information existed across documents and systems rather than within a consistent, queryable knowledge foundation.

The system supported an operational environment of roughly 70,000 employees, with individual employees responsible for managing thousands of health plans.

At that scale, even minor inefficiencies could become substantial operational bottlenecks.

The goal was not simply to automate document reading.

It was to build a system that could extract, validate, standardize, compare, and retrieve insurance information reliably, while allowing users to interact with that information through natural language.

Figures shared by Jennifer Ratten. The 99% figure represents a project requirement, not a reported final accuracy result.

Why One LLM Was Never Going to Solve the Problem

One of the first architectural decisions was recognizing that the solution could not rely on a single LLM prompt or a standalone RAG application.

The system needed several complementary AI capabilities working together.

Retrieval-augmented generation grounded responses in relevant source documents, reducing dependence on the model’s internal knowledge.

Document classification helped distinguish between different formats, plan categories, and content structures.

Section-level summarization extracted and normalized important details from specific benefit categories, while higher-level summarization made broader plan information easier to interpret.

Conversational AI allowed users to ask questions without navigating multiple documents or manually constructing database queries.

And AI agents supported more complex workflows involving reasoning, plan comparisons, and response generation.

These were not isolated features bolted onto a document repository. They formed an integrated data processing and reasoning architecture.

That distinction matters.

A chatbot might answer a question about a single document reasonably well. A production insurance application must identify the correct plan, retrieve the relevant coverage section, distinguish similar terminology, compare information against other sources, and produce an answer consistent with the underlying records.

Each step introduces another opportunity for error.

The challenge was not getting an LLM to read documents. It was ensuring that every piece of information reaching the LLM was relevant, interpretable, and trustworthy.

Inside the Architecture: From Raw Documents to AI-Ready Intelligence

Ratten described an architecture that separated document ingestion, semantic preparation, structured storage, and AI-driven reasoning.

Documents first entered blob storage, where they were parsed and normalized.

The team then applied LLM-driven chunking logic designed to accommodate approximately 25 different document layout variations across insurance plan types.

This was a critical departure from generic document-processing approaches.

Rather than splitting text into arbitrary segments, the pipeline needed to preserve the meaning and relationships between insurance benefits, coverage conditions, and plan categories.

Once extracted and normalized, structured information was stored in Cosmos DB and Fabric SQL databases.

These data stores provided a consistent foundation for downstream querying and AI workflows.

On top of that foundation, agents built using Microsoft AI Foundry handled reasoning, comparisons, and final response generation.

Python and SQL formed the core of the transformation, validation, and orchestration work, with R Studio used selectively for analytical tasks.

The architecture reflects an important principle for modern data engineering: LLM readiness is not a final transformation applied before inference. It is an end-to-end design requirement.

The Chunking Problem: Why Generic RAG Failed to Deliver Accurate Answers

Chunking is often treated as a routine implementation detail in RAG systems.

Choose a chunk size, add overlap, generate embeddings, and move on.

For Ratten’s team, that approach was not sufficient.

Early experiments with generic chunking produced weaker accuracy because the structure of the documents carried important business meaning.

A health insurance document is not simply a collection of paragraphs. It contains relationships between plan types, coverage categories, conditions, exclusions, and benefit structures.

Splitting the document without respecting those relationships can remove the very context needed to answer a question correctly.

The team found that section-based chunking delivered substantially better results.

Instead of organizing content around arbitrary token boundaries, documents were divided into meaningful sections, such as hospital services or prescription coverage.

Those sections were then refined according to the structural differences between document types.

This approach made retrieval more aligned with how insurance professionals actually reason about coverage.

For example, a question about prescription benefits should retrieve the relevant prescription coverage section for the correct plan, not a semantically similar passage from another plan or benefit category.

The distinction sounds simple.

In production, it can determine whether the system delivers a reliable answer or a misleading one.

When the data existed, but retrieval still failed

One of the clearest examples emerged during user acceptance testing.

Users were asking for information about a specific health plan, but the system returned results covering multiple plan types.

The information was present in the underlying dataset.

The model was not necessarily inventing facts.

The failure was in how the data had been chunked and indexed.

The original indexing strategy did not sufficiently distinguish between plan categories, allowing unrelated plan information to enter the retrieval results.

The team addressed the issue by introducing additional indexing layers aligned with specific plan types.

This exposed a problem that many AI teams misdiagnose.

When an LLM produces the wrong answer, the immediate instinct is often to change the model, rewrite the prompt, or adjust the generation parameters.

But sometimes the model is simply receiving the wrong context.

Before optimizing generation, teams need to verify that retrieval is selecting the correct business entities and relationships.

Accuracy Wasn’t a Final Test. It Was Built into the Pipeline.

Given the sensitive and regulated nature of health insurance information, the project had demanding operational requirements.

Accuracy needed to meet a minimum target of 99%. Responses had to arrive within seconds or minutes rather than hours. Security, stability, and personalization were also essential.

Different users needed different types of outputs, and the system had to support those requirements without compromising consistency.

Interestingly, cost was not the primary constraint.

The organization was prepared to invest in the infrastructure required to deliver a reliable solution.

That shifted the engineering focus toward correctness, performance, and operational stability.

Rather than waiting until the end of development to evaluate the system, Ratten’s team introduced approximately 15 checkpoints throughout the pipeline.

These checkpoints tracked retrieval accuracy, latency, and hallucination rates before tuning, during the addition of processing layers, and after optimization.

The goal was to identify regressions as soon as they appeared.

For example, a transformation might improve retrieval precision but increase latency. A new indexing strategy might make some plan categories easier to find while reducing accuracy for others.

By measuring continuously, the team could understand the consequences of individual architectural changes.

This is a valuable lesson for any organization moving from AI prototypes to production.

A successful demo shows that a model can produce an answer.

A production evaluation strategy must establish whether the system consistently retrieves the right information, respects business constraints, and remains reliable as data changes.

Batch vs. Real Time: Not Every Document Needs Immediate Processing

Another important architectural decision involved separating stable reference data from information that required immediate processing.

The team maintained what Ratten described as a core universe of plans, refreshed through weekly batch processing.

This provided a consistent baseline for the majority of queries.

Real-time processing was reserved for point-of-service scenarios where users uploaded new or updated documents and needed those documents processed and compared against the existing knowledge base immediately.

This hybrid approach balanced freshness with operational stability.

The team also had to address external benchmarks that were sometimes one or two years behind current information.

Rather than treating outdated benchmarks as current truth, the system incorporated equivalency disclosures so stakeholders could understand the limitations of comparisons.

That decision highlights a broader point: making data LLM-ready also requires making its limitations visible.

A model may generate a fluent comparison, but fluency does not make outdated or incomplete reference information reliable.

The Hardest Decisions Were About Data Meaning, Not Model Selection

Some of the most consequential engineering work happened before information ever reached the LLM.

The team needed to determine which entities mattered, which attributes were essential, and what minimum information a document needed to contain before it could be considered usable.

Not every health plan document included every required element.

Instead of forcing incomplete documents through the same processing path, the team established minimum viability criteria.

These criteria helped distinguish plans that could be reliably compared from those lacking sufficient information.

The team also had to resolve semantic differences in insurance terminology.

Two documents could describe equivalent benefits using different language, while similar terms could refer to materially different coverage conditions.

Without normalization, those differences could distort retrieval and comparisons.

Another improvement came from isolating anomalous benefit structures and excluding them from early processing stages.

The team deliberately began with a narrower dataset and expanded coverage gradually as the system became more stable.

This required accepting a trade-off: less breadth and less-than-perfect real-time completeness in exchange for higher accuracy and more dependable performance.

For AI engineering teams under pressure to support every use case immediately, that is a particularly relevant lesson.

A smaller, well-understood knowledge base can be more useful than a comprehensive dataset whose semantics cannot be trusted.

Why Your BI Data Model May Be the Wrong Starting Point for LLMs

Perhaps the most important misconception Ratten challenged was the assumption that clean, well-modeled BI data is automatically ready for AI.

Traditional BI systems are optimized for reporting, aggregation, filtering, and consistent analytical calculations.

Star and snowflake schemas serve those requirements well.

But LLM-driven applications introduce different needs.

They must interpret natural-language questions, identify relevant entities, understand relationships, retrieve supporting context, and reason across information that may not fit neatly into a conventional reporting structure.

A data model that answers how many employees are enrolled in a particular plan? may not be sufficient to answer how does this plan’s prescription coverage compare with another plan under a specific set of conditions?

The second question requires more than accurate tables.

It requires semantic context, relationships, and an understanding of business meaning.

This does not mean traditional BI models should be abandoned.

It means they may need additional semantic layers, transformations, metadata, and retrieval structures before they can reliably support LLM applications.

Ratten identified conventional star and snowflake schemas as early signals that additional transformation work may be necessary.

The problem is not that these schemas are poorly designed.

It is that they were designed to solve a different class of problem.

If She Were Building It Again, She Would Start with Ontologies

When asked what she would change if she were rebuilding the system today, Ratten’s answer was particularly revealing.

She would introduce ontologies from the beginning.

An ontology defines the important concepts in a domain and the relationships between them.

In a health insurance context, that could mean explicitly representing relationships between plans, coverage categories, benefits, eligibility conditions, and providers.

Instead of depending entirely on document text and implicit associations, the system could use a more formal semantic foundation.

That could make it easier to incorporate additional data sources, maintain consistent definitions, and support more sophisticated reasoning.

It could also help address some of the challenges encountered during the original implementation, including inconsistent terminology, plan-specific indexing, and complex comparisons.

This points toward an important evolution in enterprise AI architecture.

The next challenge is not merely making documents searchable.

It is making the underlying business knowledge understandable to machines.

Ontologies, knowledge graphs, and semantic modeling can help provide that foundation, particularly in domains where relationships and context determine the correctness of an answer.

Five Lessons for Data Engineers Building Production LLM Systems

Ratten’s experience offers practical guidance for teams building AI-enabled data platforms.

1. Model business meaning before optimizing retrieval.

Identify the entities, relationships, terminology, and rules that determine whether an answer is correct. A technically clean dataset can still be semantically ambiguous.

2. Design chunking around document structure.

Generic chunk sizes are a starting point, not a universal solution. Preserve the sections, categories, and contextual relationships users depend on.

3. Evaluate the entire pipeline, not just model output.

Track retrieval accuracy, latency, and hallucination rates throughout development. Introduce checkpoints to catch regressions before they reach production.

4. Prioritize reliability over immediate completeness.

Define minimum viability criteria, isolate anomalous records, and expand coverage gradually. A trustworthy narrow solution is a stronger foundation for scale.

5. Keep domain experts involved throughout development.

Many production failures emerge from misunderstood business rules rather than model limitations. Stakeholder knowledge must inform architecture, testing, and validation.

The Bigger Picture: AI-Driven Data Engineering Is Really About Trust

There is a tendency to measure progress in enterprise AI by the sophistication of the models being deployed.

Larger context windows, more capable agents, faster inference, and increasingly advanced reasoning systems attract attention.

But Ratten’s experience reveals a different reality.

The success of an enterprise LLM application often depends on the less visible engineering decisions beneath it.

How are documents classified? Which entities are important? Where should content be split? What metadata is required? How do we distinguish similar plan types? When is information too incomplete or outdated to support a reliable comparison?

These are fundamentally data engineering and business modeling questions.

The approximately 97% reduction in manual processing time demonstrates what becomes possible when those questions are addressed systematically.

Yet the deeper achievement was creating a foundation that could transform fragmented documents into information suitable for retrieval, comparison, and conversational interaction.

For organizations already investing in modern data platforms, the message is clear.

Being data-rich is not the same as being AI-ready.

A well-governed warehouse, accurate dashboard, or mature analytics pipeline is a valuable starting point. But production LLM systems demand another layer of engineering: one that preserves context, encodes business meaning, evaluates retrieval quality, and makes uncertainty visible.

And as Ratten’s experience shows, getting that foundation right can matter far more than choosing the latest model.

The next time an enterprise LLM gives the wrong answer, the most useful question may not be, Which model should we use instead?

It may be:

Did we ever make the data understandable enough for the model to answer correctly?

About the Expert

Jennifer Ratten is VP, Application Engineering | Data Strategy and Practice at NFP, an Aon company. A data, AI, and platform architect and transformation leader, she brings nearly two decades of insurance industry experience into enterprise technology. Her work focuses on building scalable data and AI solutions, including implementations using Microsoft Fabric and the Microsoft Power Platform. She also shares practical implementation insights through technology conferences and community engagement.

About the Host

Vaideeshwari Roshan is a Portfolio Manager at Packt. In this expert conversation, she explores the architectural, operational, and business challenges of making enterprise data LLM-ready, from document processing and semantic modeling to retrieval accuracy and production-scale AI systems.

Editorial note: This article is adapted from the supplied interview with Jennifer Ratten, hosted by Vaideeshwari Roshan. The narrative has been reorganized for clarity and readability. All project figures and implementation details are based on the interview, and no reconstructed statements are presented as verbatim quotations.

Your Data Works for BI. So Why Does Your LLM Keep Getting It Wrong? was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some academic journeys begin with a clear plan. Mine began with a question that refused to leave me alone: How do we intervene in a system to achieve a desired effect? That question followed me across continents, across disciplines, and across decades — from aeronautical engineering in Cairo to stochastic adaptive control in Europe and Morocco, and eventually to causal inference and Bayesian networks in Southern California.

A Selection of Early Publications

Only in hindsight do I see the thread connecting it all. Every phase of my work — optimal control, learning automata, stochastic processes, modeling biological morphogenesis, market price formation, and communication flow control — was really an exploration of how systems behave, adapt, and respond when we act upon them. Causality was always present. I simply didn’t yet have the vocabulary for it.

The First Half: Learning to Control What We Don’t Fully Know

My early academic life was devoted to understanding how to control systems whose dynamics were uncertain or partially unknown. I focused on finite Markov chains and semi‑Markov processes, developing active, recursive learning mechanisms that unified control and estimation in real time.

This work introduced several ideas that today echo in reinforcement learning:

• Randomized control policies to ensure continual exploration
• Reward gradient formulas to quantify how policy changes affect long‑run performance
• Recursive stochastic gradient algorithms that adapt policies safely toward optimality

At the time, these were simply solutions to problems I found intellectually irresistible. Today, they resemble early versions of concepts that underpin modern adaptive systems.

The figure below illustrates block diagrams from my book Learning Systems, depicting supervised learning of Bayes’ rule, a game between multiple learning automata, a flow‑control system regulating server activity, and a learning automaton.

The Turning Point: Discovering the Language of Causality

Everything changed in 1986.

While teaching control systems and AI at California State University Long Beach, I attended a lecture by Judea Pearl on Bayesian networks. That single talk reframed my entire intellectual world. It connected the systems I had spent decades modeling with a new, powerful idea: causal structure.

I realized that the question I had been chasing — how do we intervene? — was the beating heart of causal inference.

From that moment on, my research, teaching, and industry work shifted toward causal modeling, Bayesian networks, and decision analysis. When Packt approached me in 2023 to write a book on causal inference with Bayesian networks, I knew it was time to synthesize everything I had learned across my eclectic academic journey.

Three Years, Hundreds of Papers, and One Unifying Vision

Writing Causal Inference with Bayesian Networks took three years of deep reading, coding, experimentation, and careful structuring. My goal was ambitious:

Build a book that teaches causal inference not as abstract theory, but as a practical, computational discipline grounded in Bayesian networks, influence diagrams, and relational databases.

The book is organized into three parts:

1. Representation (Chapters 1–5)

Foundations of probability, Bayesian networks, structural causal models, and relational database representations.

2. Inference (Chapters 6–11)

Join tree clustering, probabilistic inference, causal identifiability, potential outcomes, and machine‑learning meta‑learners for heterogeneous treatment effects.

3. Applications (Chapters 12–14)

Real‑world case studies in economics, epidemiology, and social science.

One of the book’s most distinctive contributions is an algorithm that compiles influence diagrams into Query Composition Graphs (QCGs) — a reusable, SQL‑driven workflow for computing maximum expected utility (MEU). This bridges causal modeling with production‑grade decision systems.

About the Book

Causal Inference with Bayesian Networks Build Bayesian Networks and Causal Inference Models with R and Python by Yousri El Fattah and Reza Bagheri

This book provides:

• A unified introduction to Bayesian networks and structural causal models
• Hands‑on inference using R and Python
• Practical methods for causal identifiability and treatment‑effect estimation
• A novel SQL‑based workflow for MEU computation using QCG
• Real‑world applications in economics, epidemiology, and social science

Why I Wrote This Book

Because causal inference is no longer optional.

Whether you work in AI, data science, economics, epidemiology, or decision analysis, understanding why things happen — and how to intervene — has become essential. Bayesian networks and influence diagrams offer a structured, transparent way to reason under uncertainty.

My hope is that this book gives readers not just knowledge, but tools they can use immediately:

• Build Bayesian networks from scratch
• Perform probabilistic and causal inference
• Implement SCMs and meta‑learners
• Deploy causal models inside relational databases
• Compute MEU policies using QCGs

After more than two years of collaboration with my co‑author, Reza Bagheri, the book is now live on Amazon. It represents the culmination of my academic journey — and the beginning of yours.

This book is not just a technical guide — it is the culmination of a lifetime spent asking how systems respond when we intervene, and how understanding causality can help us make better decisions.

Book Link

👉 https://packt.link/QNELF

About the Author

Yousri El Fattah is the founder of Causal Computing and a researcher whose career spans aeronautical engineering, stochastic adaptive control, learning automata, and causal inference. He has held academic appointments across Egypt, Europe, Morocco, and Southern California, and has contributed foundational work on randomized control policies, recursive learning algorithms, and causal modeling. His current work focuses on Bayesian networks, influence diagrams, and decision systems that bridge theory with real‑world applications.

From Control Theory to Causal Inference: A Three‑Decade Journey That Led to My Book was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Maxime Labonne, Head of Post-Training at Liquid AI, explains why benchmark scores rarely tell the whole story, why evaluation should start before training, and how data teams can build LLM systems that survive contact with real users.

An LLM tops a benchmark.

It scores well on coding, reasoning, instruction following or general knowledge. The evaluation dashboard looks strong. The team ships it.

Then users arrive.

They phrase requests differently from the evaluation set. They introduce edge cases nobody anticipated. They expect outputs in particular schemas. They combine constraints. They send inputs that look nothing like the carefully prepared examples used during development.

And suddenly, the model that looked excellent on paper does not look quite so capable in production.

This gap between benchmark performance and production performance is becoming one of the most important engineering problems in applied AI.

In a conversation hosted by Sanjana Gupta, Relationship Lead at Packt, Maxime Labonne, Head of Post-Training at Liquid AI, explores why teams need to rethink how they evaluate language models, where automated evaluation works, why smaller specialized models can sometimes outperform much larger ones, and what ML teams should prioritize if they want more reliable AI systems.

Labonne’s argument is not that benchmarks are useless.

It is that a benchmark should be treated as a signal, not a verdict.

For data scientists, ML engineers and AI practitioners building systems for production, that distinction changes almost everything.

The benchmark says the model is better. Your users may disagree.

The traditional model-selection workflow feels reassuringly quantitative.

Run several candidate models against MMLU, HumanEval, a leaderboard or another standardized evaluation. Compare the numbers. Pick the model with the strongest result.

But Labonne points to an uncomfortable problem: the ranking produced by an evaluation does not necessarily reflect how people experience the models in practice.

A model can perform exceptionally well on a standardized test while another is preferred by developers for actual programming work.

That does not necessarily mean the benchmark is wrong.

It means the benchmark measures only part of the system’s behavior.

LLMs now operate across coding, extraction, reasoning, summarization, agents, RAG, tool use, structured generation and countless domain-specific tasks. Expecting one score to represent performance across that entire space is increasingly unrealistic.

Labonne therefore recommends treating evaluations as multiple signals rather than searching for one definitive number.

The same caution applies to human preference systems and automated judge models. Human evaluations introduce their own incentives and biases. LLM-as-a-judge systems can automate comparison at scale, but judge models also have biases.

There is no perfect evaluator.

For practitioners, the more useful question becomes:

What evidence would convince us that this model works for the specific job we are asking it to perform?

That question leads to a very different evaluation strategy.

Why good benchmark performance can collapse in production

One of the most dangerous evaluation failures happens before deployment.

Suppose your training and evaluation examples come from essentially the same distribution.

The evaluation set is technically separate. There is no obvious leakage. Your metrics look excellent.

But both datasets were generated using similar assumptions, similar pipelines or similar source material.

The model can become extremely good at the world represented by your test set without becoming equally good at the world represented by your users.

Labonne highlights contamination as an obvious version of this problem. If a model has effectively encountered the evaluation data during training, extraordinarily high benchmark scores deserve scrutiny.

But the problem can be subtler.

Even without direct contamination, training and evaluation distributions can resemble one another too closely.

Production users do not care about your data distribution.

They will use the system in ways you did not expect.

They will submit strange inputs, omit context, combine requirements, misunderstand instructions and discover failure modes that never appeared in the development environment.

A model can therefore pass an evaluation while the evaluation itself fails to represent reality.

That is why production evaluation should not simply ask:

Did the model answer our test questions correctly?

It should ask:

Does our test set resemble the situations this system will actually encounter?

Start with evaluation, not training

This is perhaps the most actionable idea from Labonne’s interview.

Many teams follow a familiar sequence:

collect data → train or fine-tune → evaluate → discover problems → rebuild

Labonne argues for reversing part of that process.

Define the evaluation first.

Consider a seemingly straightforward task such as structured data extraction.

You give the model an article or paragraph and expect JSON.

At first glance, the task sounds clear.

It is not.

What fields should the JSON contain?

Can objects be nested?

Which fields are optional?

Which data types are permitted?

How should missing values be represented?

What happens when the source contains contradictory information?

What should the model do when the requested information does not exist?

Before generating thousands of training examples, the team needs to define what success actually means.

Once those expectations become an evaluation dataset, something important happens: the evaluation becomes the specification for the system.

It tells engineers what to optimize.

It tells data teams what examples need to exist.

And it gives stakeholders something concrete to approve.

Labonne even recommends separating the processes used to produce training and evaluation data where possible. Different people, processes or code paths can reduce the risk of accidentally building an evaluation that merely reproduces the assumptions embedded in training.

The lesson for data teams is simple:

If your evaluation is built from the same assumptions as your training data, it may validate your pipeline rather than challenge your model.

Evaluation is also a stakeholder-alignment tool

There is another reason to build evaluations early that has little to do with model metrics.

It forces teams to agree on what the product is supposed to do.

Imagine showing a customer or product team an evaluation set before training:

Here are 100 situations we believe the model needs to handle. Here are the expected behaviors. Do these represent your use case?

Suddenly, abstract requirements become concrete.

Stakeholders can identify missing scenarios.

They can challenge assumptions.

They can decide whether a behavior is acceptable.

They can expose edge cases while changing them is still inexpensive.

Labonne argues that this is far easier at the beginning of a project than after the model has been trained and integrated.

For practitioners accustomed to software engineering, the idea resembles test-driven development.

But for LLM systems, the evaluation suite does more than test code.

It defines expected model behavior.

And once that definition exists, constructing the right training dataset becomes much easier because the team knows what it is actually optimizing for.

Automate what can be measured objectively

Building better evaluations does not mean manually reviewing every output.

Labonne strongly supports automated evaluation because it changes the economics of experimentation.

ML teams operate under a time budget.

If evaluating every model takes days of manual review, fewer models can be trained and fewer ideas can be tested.

Automating repeatable checks reduces that cycle time.

The saved time can then be spent where Labonne believes it matters most: experimentation and data quality.

Some failures are particularly suitable for automation.

A model falling into a repetition loop, for example, should not require a human reviewer to discover it every time. If an output repeats the same phrase thousands of times, an automated evaluator should catch it.

Likewise, many instruction-following requirements are deterministic.

If the instruction says:

Return five bullet points in Italian.

A program can check whether:

• the response contains five bullet points;
• the required format was followed;
• the output satisfies the language requirement.

The evaluator does not need another giant model to decide whether a bullet point exists.

This distinction is increasingly important as teams experiment with reinforcement learning with verifiable rewards (RLVR).

If a machine can verify the requirement, let it

Labonne contrasts verifiable rewards with approaches that depend heavily on learned reward models.

Consider an instruction:

“Answer using bullet points.”

A Python function can inspect the response and determine whether the requirement was satisfied.

Pass the check: reward.

Fail the check: no reward.

More complicated instructions can support partial credit.

If the model must produce bullet points and answer in Italian, satisfying one constraint but not the other could receive an intermediate reward.

This works particularly well for tasks with objectively testable outcomes:

Instruction following. Mathematical answers. Structured output. Formatting constraints. Function calls. Schema compliance.

For data practitioners, the implication is significant.

Before reaching for another LLM to evaluate an LLM, ask:

Can this requirement be expressed as code?

If yes, deterministic verification may be cheaper, faster and easier to debug.

But not everything valuable is verifiable

Now ask the model to write a good poem.

A Python function can check whether the response contains four stanzas.

It cannot reliably tell you whether the poem is moving.

This is the boundary of purely deterministic evaluation.

Open-ended tasks such as writing quality, tone, creativity and some forms of reasoning cannot always be reduced to binary checks.

In those situations, Labonne suggests bringing reward or judge models back into the evaluation architecture.

The useful pattern is therefore not:

deterministic evaluation versus model-based evaluation.

It is:

use deterministic evaluation wherever the requirement is objectively verifiable, and model or human judgment where it is not.

A mature evaluation system can combine all three.

That gives practitioners a useful hierarchy:

Code where possible. Models where necessary. Humans where judgment genuinely matters.

The evaluation suite should evolve with the model

Evaluation is not something a team finishes.

New models reveal new failure modes.

Users uncover behaviors the development team never anticipated.

Labonne gives repetition or “doom loops” as an example: a model may start repeating the same words or sentences endlessly. Once a team discovers a failure like this, it should become part of the evaluation suite.

That creates an important engineering cycle:

discover failure → encode failure → test automatically → prevent regression

Over time, the evaluation harness becomes a record of what the team has learned about the system.

This is much closer to production software testing than leaderboard chasing.

The best evaluation suite is therefore not necessarily the one containing the most famous benchmarks.

It is the one that increasingly reflects your model, your application and your users.

Small models change the economics of production AI

The conversation also moves beyond evaluation into another shift that data practitioners should be watching: the growing usefulness of small language models.

At Liquid AI, Labonne works with models ranging from hundreds of millions to several billion parameters, including models designed to run locally on phones, laptops and other edge hardware.

Why does that matter?

Because cloud-scale LLMs introduce constraints that benchmark leaderboards rarely capture:

Cost. Latency. Connectivity. Privacy. Hardware requirements.

A frontier cloud model might produce the strongest general-purpose answer.

But what happens when the application needs to operate without internet connectivity?

What happens when sensitive information cannot leave the device?

What happens when latency needs to be extremely low?

Or when an agentic workflow consumes enormous numbers of tokens?

At that point, “best model” stops being a meaningful question without context.

The real question becomes:

Best model for what operational constraint?

A smaller model can beat a larger model when the task becomes narrow

Labonne describes a particularly revealing example involving in-car function calling.

A client required a very specific function-calling format.

A large general-purpose model struggled with the task.

A much smaller model was fine-tuned specifically on examples matching that format.

According to Labonne, the specialized model achieved over 90% accuracy, compared with roughly 10–20% for the GPT-4 system tested at the time.

The point is not that small models are universally better.

Labonne explicitly notes that general-purpose tasks become harder for smaller models.

The lesson is about specialization.

When the problem is sufficiently narrow, a small model can be trained directly around the required behavior.

For practitioners, that suggests another reason why application-specific evaluation matters.

A generic benchmark might strongly favor the larger model.

Your private evaluation might strongly favor the smaller one.

And if the smaller model also delivers lower latency, local inference and dramatically lower inference cost, the production decision can look very different from the leaderboard.

Agentic AI makes model economics impossible to ignore

This trade-off becomes particularly important for agents.

Agentic systems can be extremely token hungry.

A conventional chatbot may make one model request and return one answer.

An agent can reason, call a tool, inspect the result, generate another request, call another tool, revise its plan and repeat the process.

Token consumption compounds.

Labonne expects this to strengthen the case for local and smaller models, particularly for tasks where frontier-model quality is unnecessary.

That does not mean cloud models disappear.

Instead, production architectures may become more selective.

Use a specialized local model for predictable, repetitive or narrow operations.

Escalate to a larger cloud model when the problem actually requires the additional capability.

For data and AI teams, model routing may therefore become as important as model selection.

The future of LLM evaluation is probably not one benchmark

So what happens as models become more capable?

Labonne expects evaluation to move away from dependence on a handful of dominant benchmarks toward many specialized signals.

This is already happening.

Researchers and practitioners are building evaluations for increasingly narrow domains and behaviors.

Individually, each benchmark sees only a small part of the model.

Collectively, they can provide a much richer picture.

This matters because improvements between successive frontier models may not always appear as dramatic gains on everyday tasks. Instead, Labonne argues that much of the progress comes from models becoming competent across an expanding range of domains.

The capability surface gets wider.

Evaluation therefore has to get wider too.

The interesting infrastructure opportunity is an aggregation layer capable of combining many heterogeneous signals into a more useful picture of model behavior.

Until then, practitioners should resist compressing model quality into one leaderboard position.

What should an ML team do in the next 30 days?

Labonne’s recommendation is surprisingly simple:

Start with evaluations.

Not after the next fine-tuning run.

Not immediately before launch.

Now.

For a team building an LLM application, a practical 30-day version of that advice could look like this:

Week 1: Define production behavior

Write down the tasks the system actually needs to perform. Include output formats, constraints, edge cases and unacceptable failures.

Week 2: Build a private evaluation set

Create examples representing realistic user behavior rather than merely reproducing the training distribution. Separate evaluation-data creation from training-data creation wherever practical.

Week 3: Automate the obvious

Turn deterministic requirements into executable checks: schema validation, formatting, function calls, numerical correctness, repetition detection and instruction compliance.

Use model-based or human evaluation for qualities that cannot be verified reliably with code.

Week 4: Connect evaluation to experimentation

Run every meaningful model, prompt, fine-tuning or data change through the same harness.

When production reveals a new failure, add it to the suite.

The goal is not a perfect benchmark.

It is an evaluation system that becomes harder for your model to fool and increasingly representative of what users actually need.

If the budget is limited, invest in data and evaluation

When asked where a team with limited resources should invest first, Labonne returns to the fundamentals.

Data quality is one of the strongest drivers of model performance.

But data alone is not enough.

Without evaluation, you cannot reliably determine whether better data actually produced a better system.

Training itself is becoming increasingly automated. Tooling can handle much of the mechanical work.

The harder questions remain human ones:

What data represents the problem?

What behavior counts as success?

What failures matter?

What should be measured?

And does the evaluation actually resemble production?

That makes data and evaluation two sides of the same engineering problem.

Stop asking which model is best

Perhaps the most useful takeaway from the conversation is a change in the question practitioners ask.

Instead of:

Which LLM has the highest benchmark score?

Ask:

Which model performs reliably on the distribution, constraints and failure modes that matter to this application?

That change sounds small.

In practice, it affects model selection, dataset design, fine-tuning, evaluation infrastructure, deployment architecture and cost.

Public benchmarks remain useful. Human preference remains useful. LLM judges remain useful. Private prompts and “vibe checks” remain useful too. Labonne himself says that before shipping, one of his final signals is a private collection of prompts designed to probe different model capabilities.

None of these signals needs to become the truth.

They become more valuable when they are combined.

And the closer evaluation gets to the messy reality of production, the more useful it becomes.

Because the hardest test for an LLM is not MMLU, HumanEval or a leaderboard.

It is what happens after real users start typing.

About the expert

Maxime Labonne is Head of Post-Training at Liquid AI and holds a PhD in artificial intelligence. His work spans LLM post-training, evaluation and efficient language models. He has created widely used LLM resources and models on Hugging Face and is the author of LLM Engineer’s Handbook and Hands-On Graph Neural Networks Using Python. His open LLM course on GitHub has attracted more than 39,000 stars.

About the host

Sanjana Gupta is Relationship Lead at Packt. She hosted this conversation with Maxime Labonne on the widening gap between benchmark performance and production reliability, exploring LLM evaluation, verifiable rewards, automated evaluation, small language models and the practical changes AI teams can make to build more dependable systems.

Why Most LLM Benchmarks Fail in Production: Maxime Labonne on the Future of AI Evaluation was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Robert Hawker explains why trusted analytics begins with business priorities, governed semantic models, and controls that help people move faster rather than hold them back

A company can invest heavily in Microsoft Fabric and Power BI, migrate data, build semantic models, publish dashboards, and train users, yet still watch employees return to their Excel trackers.

It is tempting to call this resistance to change. Often, it is something more fundamental: the analytics environment has not earned users’ trust.

Reports may be slow, incomplete, duplicated, or disconnected from the decisions people actually need to make. Governance may arrive too late, after conflicting metrics have already spread, or too forcefully, turning every improvement into a lengthy approval process. In both cases, the technology is visible, but the business value is not.

In a conversation hosted by Apeksha Shetty, Portfolio Manager at Packt, Microsoft Fabric and Power BI consultant, trainer, author, and experienced data and analytics leader Robert Hawker shared a more practical way to approach the problem. His central argument is that successful analytics transformation does not begin with a platform feature. It begins with the outcome an organization needs, the data problems blocking it, and a governance model designed to produce lasting improvement.

Start governance where the business already feels the pain

There is no universal first step for enterprise data governance. The right starting point depends on the organization’s industry, strategy, and immediate pressures.

In financial services, a regulatory return may set the priority. During a major transformation, attention may need to shift to new processes, new systems, and the quality of migrated data. Sometimes the trigger is an operational emergency, such as suppliers threatening action because data failures have prevented timely payment.

When no urgent driver is obvious, Hawker recommends starting with the organization’s strategy. Ask leaders what is preventing them from delivering their objectives, without limiting the discussion to issues already labelled as data problems. A deeper analysis often reveals that apparently operational or commercial blockers have data at their root.

That connection matters. Stakeholders are more likely to support governance when they can see how it improves the probability of achieving an outcome they already care about.

Prioritization can then follow a familiar benefit-versus-effort assessment. Quantitative and qualitative value should be weighed against the cost and complexity of resolving each issue, with high-benefit, lower-effort opportunities rising to the top. But the calculation is not purely mechanical. An early initiative backed by a supportive stakeholder can create proof, credibility, and an internal advocate for the wider programme.

The important distinction is between a quick fix and a quick win. Hawker recalls a proposal to pay a supplier for a one-off cleanse of supplier data using generic industry rules. Instead, he redirected the investment into the foundations of a repeatable data quality process and a tool containing rules specific to the organization. The initiative delivered near-term improvement while building a capability that could sustain it.

The lesson is simple: even early wins should point towards the long-term operating model.

When dashboards are ignored, diagnose the experience before blaming the users

Low Power BI adoption can look like a technology problem, a data problem, a governance failure, or a cultural issue. Frequently, several are present at once.

Hawker says the most common cause is a gap between report developers and business users. Developers understand the data and may have built the semantic model, but they are not close enough to the operational decision to create a useful product. Business users understand what the analysis needs to accomplish, yet may lack access to governed data or the Power BI skills required to build it themselves.

The diagnosis should therefore begin with wide-ranging conversations, especially with the people whose roles should benefit most from the reporting.

What users say provides clues about where to look:

“Reports are slow or out of date.” Poor semantic modelling, many-to-many relationships, failed refreshes, insufficient operational oversight, or a throttled Fabric capacity may be involved. What appears to be a performance complaint can expose gaps in skills and governance.

“The figures are wrong or important data is missing.” The source data may be unreliable, but governance is also implicated. Releasing a report should include an explicit assessment of whether its data quality is known and adequate for the intended decision.

“There are five reports for the same topic, and I do not know which one to trust.” This is a sign of weak analytical governance. Decentralized development without shared definitions and endorsement creates competing versions of the truth.

“My Excel tracker is more reliable.” This may indicate a cultural barrier, but dismissing the user would be a mistake. If an established process provides dependable information, the new solution must offer a compelling improvement, such as a better outcome, stronger security, or less manual work.

Adoption is not achieved by asking people to stop using familiar tools. It is achieved by giving them something demonstrably more useful and trustworthy.

What happens when poor data becomes an operational emergency

Hawker has seen the consequences of low-quality data move far beyond reporting.

At one organization formed through acquisition, hurried migrations had left critical records incomplete. More than 90% of the field used to store suppliers’ remittance-advice email addresses was blank, even though a different field for purchase-order emails had been populated.

From the operators’ perspective, remittance advices were being created. In practice, they were not reaching suppliers. Suppliers could not easily match incoming payments to outstanding invoices, so queries flooded the payments team. Staff became consumed by resolving those queries, a payment backlog developed, and some suppliers considered stopping services because they had not been paid. The risk extended as far as the power supply to a large UK manufacturing plant.

Senior leaders were initially told that the team did not have enough people to handle the workload. The underlying problem was not headcount. It was data.

The crisis elevated data quality to the board and executive committee agenda. With funding secured, the organization worked with process and data owners to establish business rules, then built a tool to measure, correct, and maintain data quality. Scores could be analysed by data object, quality dimension, company, and country, allowing progress to be tracked rather than assumed.

During Hawker’s time there, the average overall data quality score rose from 73% to 93%, improving both operational processes and analytical outcomes.

The hardest part was not building the measurement tool. It was finding people with enough knowledge of the newly established processes and enough time to define the rules. To reduce the burden, the team profiled data extracted from the ERP system, inferred possible rules, and presented process owners with a practical starting point for review. This was faster than asking busy experts to begin with a blank page.

It also illustrates why broad data quality programmes frequently lose momentum. A programme should begin in an area connected to organizational strategy and supported by a stakeholder willing to help the team navigate its first implementation. Scope without sponsorship creates activity. Focus with sponsorship creates evidence.

Trusted self-service requires a deliberate division of responsibility

“Self-service” is often used as though it has one agreed meaning. It does not.

For one organization, it may mean allowing people to run centrally developed reports. For another, it means enabling trained users to build reports from a governed semantic layer. Before designing the platform, leaders need to define which version of self-service they are trying to enable.

Hawker recommends a model with three distinct layers of responsibility.

Centralize enterprise data engineering

A central team should manage the ingestion, transformation, and preparation of strategically important datasets. Its role includes connecting enterprise systems such as ERP, CRM, and HR platforms and creating cross-system data products where business processes require them.

Manufacturing planning, for example, may require sales-pipeline data from a CRM to be combined with inventory levels and production times from an ERP system. That integration is an enterprise concern, not something every report developer should rebuild independently.

Centralize enterprise semantic modelling

Organizations should aim for a relatively small number of reusable Power BI semantic models aligned to meaningful business subject areas. The names and boundaries must reflect the business itself. A streaming company, for example, might need a model centred on “subscribers and engagement” rather than a generic “sales” model.

These centrally managed models should contain governed metrics that multiple reports can reuse. When calculations live in the shared semantic layer, consistency becomes part of the architecture instead of something every development team must negotiate later.

Enable self-service report development

Business users can then develop reporting products on top of those models. They are often closest to the requirement and best positioned to translate it into a useful report, provided they receive sufficient training in the model’s relationships, measures, and intended use.

This freedom requires discipline. When a new enterprise metric is needed, report developers should request its addition to the shared semantic model rather than quietly recreating it inside an individual report.

Hawker does not advocate shutting off Fabric capabilities to prevent experimentation. Innovation should remain possible, especially as AI makes advanced capabilities more accessible. The governance boundary appears when an experiment demonstrates enterprise value. At that point, it should be handed to the central data engineering and modelling teams so it can be hardened, supported, and reused.

Build a governed path from discovery to trust

A trusted self-service platform needs more than architecture. Users must be able to find the right content, understand it, and judge whether it is suitable for an important decision.

Hawker’s adoption and governance playbook includes four connected elements:

Create a curated reporting catalogue. Scanner API output alone is not enough. The catalogue should provide business definitions for reports and metrics, explain ownership, and help users distinguish current, trusted content from duplication.

Embed reports into the work itself. In organizations with detailed standard operating procedures, such as pharmaceutical companies, relevant Power BI reports should be referenced within those procedures rather than treated as a separate destination employees must remember to visit.

Establish proportionate quality assurance. Peer review, testing, and documentation should lead to clear Power BI endorsement, such as promoted or certified status. Senior stakeholders should understand those levels and question important content that lacks them. Quality assurance builds decision confidence and protects Fabric capacity from inefficient development.

Invest in the community. A strong Power BI community spreads learning among report developers and creates a working relationship between those developers and the teams responsible for semantic models.

Templates for accessibility, colour, branding, navigation, and slicer behaviour can also reduce reinvention and raise the baseline quality of reporting.

The danger is allowing governance to become so restrictive that people route around it. Disabling export to Excel may offer security and adoption benefits in the long term, but imposing the restriction before users have a workable alternative can alienate them. The same applies to endorsement: if a report takes two weeks to build and another six weeks to approve, the process has made trusted delivery impractical.

Good governance creates a shorter, clearer route to trusted content. It should not make unofficial content the fastest way to get work done.

Apply controls according to the consequence of being wrong

The baseline does not need to change dramatically across most organizations. Content relied upon for decisions should receive appropriate peer review, testing, and documentation. These practices protect both information quality and the Fabric capacity from excessive consumption caused by inefficient artefacts.

Highly regulated sectors may require additional teams and formal checks. In a pharmaceutical company, for example, the quality function may need to participate when analytics contributes to GxP compliance.

Within the same organization, however, not every report carries the same risk. Hawker recommends classifying content according to its intended use:

Team-level or exploratory content, including sprint reporting and prototypes intended to provoke discussion

Internal decision content used to guide significant operational or strategic choices

External content shared with customers, auditors, regulators, or other outside stakeholders

Internal and external reporting may both require review, testing, and documentation, but the evidence and data controls should be stricter for external use. Supplementing governed platform data with a spreadsheet might be acceptable for some internal reporting. For external reporting, the data should come through managed systems and platforms, with data engineering responsible for curating the feed and controlling its quality.

This consequence-based model preserves flexibility without treating trust as optional.

A practical first-year roadmap for Microsoft Fabric

There is no single Fabric roadmap. A company migrating from Databricks faces a different programme from a greenfield organization, while a business using Fabric only for semantic modelling and reporting needs a different architecture from one moving its entire data engineering estate.

For an organization migrating an existing platform and adopting Fabric for both data engineering and Power BI, Hawker recommends an agile, incremental approach.

Months 1–3: Establish the operating foundations

Appoint Fabric administration and governance roles, create an initial capacity, and confirm the business scope. A small pay-as-you-go capacity that can be turned off outside working hours may be appropriate at the beginning.

Agree the architectural principles before sprint delivery accelerates. Decide where transformations will occur, how many engineering layers are needed, when to use Lakehouses or Warehouses, and whether capabilities such as Materialized Lake Views belong in the design. Complete the necessary architecture, privacy, and security reviews early.

Capacity management should begin on day one. If a notebook consumes 10,000 capacity units each day to perform simple transformations on only a few thousand rows, that inefficiency will become more expensive as data volumes grow.

Hawker also recommends implementing Microsoft’s free Fabric Unified Admin Monitoring solution early. It captures tenant activity and inventory, creating a history that cannot be reconstructed indefinitely from Microsoft’s limited audit-log retention.

Months 3–6: Discover and design around current needs

Run requirements workshops to define reporting outputs and the enterprise data model. Revalidate business needs rather than reproducing the old platform exactly. A platform designed several years ago reflects an earlier version of the organization; migration is an opportunity to remove obsolete logic and address new priorities.

Make initial development connections to source systems and resolve network, gateway, and access constraints. Ingest the required data into development and test environments, understand the source schemas, and document the transformations needed to produce target dimensions and facts.

Months 6–12: Deliver thin slices of production value

After governance documentation is approved and development work has been proven, connect to production sources and move into sprint-based delivery. Each tranche should carry a small, coherent set of requirements through development, testing, and production so the programme demonstrates value early and learns continuously.

Where skills and architecture permit, Hawker recommends using notebooks and Lakehouses to prepare star-schema-ready facts and dimensions. This reduces downstream transformation inside semantic models and supports Direct Lake models, avoiding the scheduled refresh pattern associated with Import mode. The recommendation must still match the team’s capabilities; an organization with strong Power Query skills but limited PySpark experience may reasonably choose a different path.

Across the 12 months, the aim is not to complete a technology migration in isolation. It is to establish an operating model in which architecture, governance, capacity oversight, data quality, and business delivery evolve together.

Governance should make trusted analytics easier

The common thread across Hawker’s advice is that governance earns support when it solves a visible business problem.

It prevents a missing email field from becoming a supply-chain crisis. It turns five competing reports into one discoverable, endorsed source. It allows business users to build what they need without recreating enterprise logic. It catches wasteful Fabric workloads before scale makes them expensive. And it gives decision-makers a reason to trust the information in front of them.

The goal is not maximum control. Nor is it unrestricted self-service. It is a deliberate system of shared responsibility: central teams manage enterprise data and definitions, business users apply their domain knowledge, and proportionate controls reflect the consequence of getting a decision wrong.

Organizations that begin there are more likely to achieve what their Fabric and Power BI investments promised in the first place: not more dashboards, but better decisions.

Join 150K+ data scientists, AI engineers, BI leaders, analysts, and finance professionals staying ahead of the trends shaping the future of data and AI.

👉 Subscribe to DataPro: [Packt DataPro | Substack]
👉 Subscribe to BIPro: [Packt BI-Pro | Substack]
👉 Subscribe to AlgoFinance: [Packt AlgoFinance | Substack]

About the conversation

This expert conversation was hosted by Apeksha Shetty, Portfolio Manager at Packt, with Robert Hawker, Microsoft Fabric and Power BI consultant and trainer, author, and experienced data and analytics leader.

Your Power BI Problem May Not Be Power BI was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lorenzo Toni, Lead Data Scientist — AI Product Analytics at Amazon, explains why traditional A/B testing can miss what is really happening inside AI products, and how causal inference can reveal the downstream effects that engagement metrics hide.

An AI feature can pass an experiment and still make the product worse.

Engagement can rise. Session duration can increase. Users can interact with the feature more frequently. An A/B test can even produce statistically convincing results.

And yet, a few weeks later, conversion starts falling.

Users stop returning.

Trust begins to erode.

That is the measurement problem AI product teams increasingly need to confront.

Traditional digital products generally give users a relatively deterministic experience. Change a button, ranking algorithm, checkout flow, or interface and teams can establish a reasonably clean treatment and control.

Generative AI changes that assumption.

The same feature can produce different answers for different users, and even different answers for the same user. A seemingly insignificant prompt adjustment can alter the length, tone, relevance, accuracy, and usefulness of an interaction.

So what exactly should teams measure?

And more importantly, how do you know whether an AI feature actually caused the outcome you are seeing?

That question was at the centre of a conversation between Nilesh Kowadkar, Relationship Lead at Packt, and Lorenzo Toni, Lead Data Scientist — AI Product Analytics at Amazon, focused on experimentation and causal inference for AI products. Drawing on a generalized e-commerce case study, Lorenzo unpacked why apparently positive metrics can be misleading and why AI teams need to look beyond conventional A/B testing.

When More Engagement Is Actually a Warning Sign

Consider an e-commerce platform introducing an AI chatbot designed to help customers discover products.

Instead of navigating through search results, users could ask questions, compare prices, describe what they wanted, and receive recommendations conversationally.

A change was made to the prompt powering the chatbot.

Initially, the numbers looked promising.

Engagement increased. Users performed more actions. They also spent longer interacting with the chatbot.

Under a conventional product analytics framework, those signals could easily be interpreted as evidence that the new experience was working.

Then, roughly two weeks later, another pattern emerged.

Chatbot interactions began declining.

More importantly, overall website conversion started falling.

The metric that initially appeared particularly encouraging, time spent with the chatbot, turned out to be negatively correlated with conversion.

Users were not necessarily spending longer because the experience was valuable.

Some were spending longer because the chatbot was taking too long to help them reach a decision.

A ten-minute conversation might look like excellent engagement on a dashboard. To the customer, it might mean:

I could have found this myself in thirty seconds.

That distinction matters.

Because when AI introduces friction instead of removing it, the damage may extend beyond a single failed interaction. Users learn from that experience. The next time they need something, they may bypass the chatbot entirely.

The apparent engagement win can therefore become a trust problem.

The Real Problem Wasn’t the Chatbot. It Was What the Chatbot Was Optimizing For.

Digging deeper revealed another issue.

The chatbot was largely recommending top-ranked, popular products.

At first glance, that seems perfectly rational.

Popular products already convert well. Recommend them more frequently and customers should be more likely to buy them.

Except that assumption ignores why someone chooses an AI assistant instead of ordinary search.

A shopper typing pen into a search box probably expects popular pens.

Someone opening an AI chatbot may instead ask:

What is the best gel pen for writing on rough paper?

That is a fundamentally different intent.

The value of an LLM-powered shopping experience comes from its ability to understand those more specific needs and personalize the discovery process, rather than simply reproduce the first page of search results.

This changed the experimentation question.

Instead of asking:

Does the new prompt increase chatbot engagement?

The team needed to ask:

What kind of conversational behaviour actually helps users make better decisions?

Experimenting With Different AI Behaviours

Lorenzo worked with engineering and data teams to test alternative experiences.

The existing prompt became the baseline.

Another version made the chatbot more inquisitive, encouraging it to gather more information about the customer’s needs before recommending something.

A further approach explored what Lorenzo described as alternative “gems”: products that might not dominate the rankings but had strong reviews, good conversion performance, and a closer fit with a customer’s specific requirements.

The experiments also extended across regions using a switchback approach.

Rather than permanently assigning one experience to a group, different versions could be switched on and off across time periods. A user interacting with the system at one point might encounter one LLM behaviour and later encounter another.

This provided another way of understanding how changes to the AI experience affected behaviour while accounting for the dynamic nature of the system.

But product quality was only one side of the equation.

A more sophisticated recommendation experience could require access to a much larger product catalogue and richer context. That means additional memory, infrastructure, retrieval complexity, and ultimately cost.

The winning experiment therefore could not simply be the one with the highest conversion rate.

It had to be statistically meaningful, commercially valuable, and operationally practical.

Where Traditional A/B Testing Starts to Struggle

A/B testing remains enormously useful.

When teams can create clean treatment and control groups and isolate a specific intervention, it provides an intuitive framework for determining whether a product change caused an outcome.

But AI introduces complications.

Some users may already have experienced the AI feature. Others may use AI frequently outside the product. Some approach a chatbot naturally; others distrust it. One user might ask precise questions while another experiments with vague prompts.

Those differences can influence outcomes independently of the feature being tested.

And because generative AI itself is probabilistic, the underlying experience is not perfectly stable.

Lorenzo points to the assumptions behind traditional controlled experiments, including SUTVA, where treatment effects are expected to be sufficiently isolated and the experimental environment stable enough to interpret the comparison.

AI experiences make that harder.

The interface might remain identical while the generated response changes every time.

That is where causal inference becomes particularly valuable.

Don’t Just Find a Control Group. Build a Better Counterfactual.

Imagine two customers who look nearly identical according to conventional segmentation.

They are similar in age.

They have similar purchasing behaviour.

They spend similar amounts.

They browse similar products.

Yet one regularly uses AI assistants and knows how to ask detailed follow-up questions, while the other barely interacts with AI.

From the perspective of an AI product, those are not necessarily equivalent users.

Lorenzo argues that AI behaviour itself can become another important covariate.

Teams can attempt to identify comparable users not only through demographic and purchasing characteristics but through the way they interact with AI: the questions they ask, how they structure conversations, and how comfortable they are with AI-assisted decisions.

That opens the door to forms of matching and “digital twin” thinking, alongside techniques such as synthetic controls and interrupted time-series analysis.

Instead of relying entirely on a clean traditional control, teams can construct richer counterfactuals:

What would probably have happened to this user if the AI experience had not changed?

That is a much more powerful question than simply asking whether treatment group A converted more frequently than control group B.

Measure the Journey, Not Just the AI Interaction

This shift also changes what product teams consider success.

Suppose a user spends more time with an AI shopping assistant and eventually purchases the recommended product.

Success?

Possibly.

But now imagine that before buying, the same user copies the product name into the conventional search bar, reads the product page independently, and only then completes the purchase.

Conversion happened.

But did the user actually trust the AI recommendation?

Perhaps not.

Conversely, an experienced AI user may completely stop using traditional search because the assistant now satisfies that need.

That could look like cannibalization if teams examine search usage in isolation.

But if conversion remains stable, average order value improves, and time-to-order falls, the story looks very different.

The AI has not necessarily damaged search.

It may have successfully replaced part of it.

This is why Lorenzo argues that teams need to construct a coherent behavioural story around their metrics.

Conversion alone is not enough.

Engagement alone is not enough.

Session duration alone is certainly not enough.

You need to understand how the user moved from intention to interaction to decision.

AI Product Analytics Needs to Explain the “Why”

This becomes particularly important for generative AI products.

Traditional recommendation systems and optimization products already require careful experimentation, but LLM-driven applications add another variable: randomness is part of the product itself.

A product manager cannot simply eliminate that uncertainty.

They have to design measurement around it.

That means defining success metrics, guardrails, and downstream outcomes before declaring an experiment successful.

Instead of reporting:

Conversion increased 4%.

An AI product team should be working toward a richer explanation:

A segment of customers held short, successful AI conversations and purchased the recommended item. Another segment abandoned the experience after longer conversations. Some users received recommendations from the AI but still verified them through conventional search before purchasing.

Now the team knows where to intervene.

If customers abandon recommendations, perhaps recommendation quality needs work.

If conversations become hostile or frustrated, perhaps tone or accuracy needs attention.

If users repeatedly verify AI answers through conventional search, perhaps the underlying issue is trust.

If hallucinations are causing abandonment, retrieval or grounding may need improvement.

The measurement system becomes a diagnostic tool rather than simply a scoreboard.

The Hidden Cost of Getting AI Experiments Wrong

There is another reason experimentation matters so much for AI products.

Bad AI experiences can teach customers not to use your AI.

A poor conventional interface can be redesigned.

A bad AI interaction can create a stronger behavioural response: the user decides the assistant simply cannot be trusted.

As Lorenzo puts it, teams should experiment fast before they roll out fast.

If a deteriorating AI experience reaches users at scale, negative behaviour can spread quickly. Customers who repeatedly encounter inaccurate, irrelevant, or frustrating responses may begin ignoring future AI suggestions altogether.

Fast iteration therefore matters.

But fast experimentation does not mean random experimentation.

Small Prompt Change. Surprisingly Big Experiment.

One particularly important lesson concerns prompt engineering.

Changing a few words can look trivial from an engineering perspective.

Statistically, it may be anything but trivial.

Suppose a team changes its system prompt to instruct:

When customers ask a follow-up question, respond in a more conversational tone.

Then the team exposes 1,000 users to the new version.

Has the experiment actually tested “conversational tone”?

Not necessarily.

LLM outputs are probabilistic. Some responses from the original prompt may already sound conversational. Some responses from the modified prompt may not feel meaningfully different.

The experimental groups can therefore overlap semantically even though the underlying prompts differ.

To isolate a subtle behavioural effect like tone, teams may need far larger samples than expected, richer behavioural classification, and more sophisticated counterfactual methods.

This is an important distinction for AI teams.

You are not simply testing whether Prompt A differs from Prompt B.

You are testing whether the experiences those prompts generate are sufficiently different to cause a measurable change in human behaviour.

The Framework: Hypothesis → Technique → Evidence → Rollout → Next Decision

So what should an AI product team actually do?

Lorenzo’s recommendation is not to abandon A/B testing.

It is to build a broader experimentation discipline around it.

The process starts with a clear hypothesis.

Then choose an experimentation or causal technique appropriate to the question.

Collect enough evidence to distinguish meaningful effects from the randomness inherent in AI interactions.

Roll out deliberately.

And then ask what comes next.

Should exposure increase?

Is more evidence necessary?

Should the experiment move to another user cohort?

Are there particular forms of AI randomness that need further investigation?

The loop becomes:

Generate a hypothesis → choose the right evaluation technique → collect evidence → roll out → decide what needs to be tested next.

That framework gives product managers, data scientists, and engineers a common language for experimentation.

And that may be as important as the statistical techniques themselves.

Causal Inference Is Also a Cultural Shift

There is a final challenge.

A/B tests are relatively easy to communicate.

Treatment beat control.

Conversion increased.

Ship the feature.

Causal inference can be more nuanced. Matching, covariates, counterfactuals, interrupted time series, synthetic controls, and behavioural segmentation require greater statistical understanding.

But the resulting product story can actually be much more intuitive.

Instead of merely knowing what moved, teams begin understanding why it moved, for whom, under what conditions, and what might have happened otherwise.

Lorenzo argues that building confidence in those insights is partly a cultural challenge. Organizations need teams capable of understanding and trusting causal evidence rather than treating a single dashboard metric as the final verdict.

The Metric Is Not the Outcome

AI product teams have become remarkably good at measuring activity.

Tokens generated.

Conversations started.

Messages per session.

Session duration.

AI-assisted conversions.

Click-through rates.

But activity and value are not the same thing.

A customer spending ten minutes talking to your AI might be deeply engaged.

Or they might simply be struggling.

A decline in traditional search might indicate cannibalization.

Or it might mean your AI assistant has successfully replaced a slower workflow.

A conversion following an AI recommendation might demonstrate trust.

Or the customer might have independently verified everything the AI said before buying.

The metric cannot tell you which story is true on its own.

That is the deeper lesson from Lorenzo Toni’s conversation with Nilesh Kowadkar.

As AI becomes embedded in more products, experimentation cannot stop at asking whether a metric went up or down.

Teams need to understand the counterfactual.

They need to measure downstream behaviour.

They need to account for the randomness of AI responses and differences in how people interact with these systems.

And above all, they need to understand the human journey behind the numbers.

Because the most dangerous AI experiment isn’t necessarily the one that fails.

It’s the one that appears to succeed while quietly making the product worse.

Join 150K+ data scientists, AI engineers, BI leaders, analysts, and finance professionals staying ahead of the trends shaping the future of data and AI.

👉 Subscribe to DataPro: [Packt DataPro | Substack]
👉 Subscribe to BIPro: [Packt BI-Pro | Substack]
👉 Subscribe to AlgoFinance: [Packt AlgoFinance | Substack]

Your AI Experiment Says the Feature Is Working. Your Users May Be Telling You Otherwise. was originally published in Packt Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.