AI Lessons From The Work Not The Hype — Part 6
The Most Expensive AI Architecture Is Often the One That Sends Everything to the Model
There is a simple way to make an AI prototype look impressive.
· Give the model more information.
· More documents. More context. More data.
Then ask it to work everything out.
It can produce compelling results.
It can also become a very expensive way to build enterprise AI.
Prototypes can hide production economics
During experimentation, model usage costs may not attract much attention.The number of users is limited. Volumes are manageable. The priority is proving that something works.
Production changes the equation.
Thousands of interactions become hundreds of thousands.
A prompt carrying large amounts of unnecessary context is no longer just technically inefficient. It becomes a recurring operating cost.
We have seen organisations discover this problem only after usage started to scale. In one case we encountered, an AI budget intended to support a much longer period had effectively been consumed by mid-March.
The lesson was straightforward:
A solution that works is not necessarily a solution that works economically.
The model does not need to do everything
Our approach is to reduce the amount of work delegated to an AI model wherever it makes sense. That may sound strange coming from people building AI solutions. But a model should be used for the parts of a problem where it creates real value.
Before information reaches the model, data can be filtered and prepared. Traditional algorithms can handle tasks they perform reliably. Similarity and retrieval logic can be controlled outside the model. Only the information required for a specific interaction needs to be sent.
The result is not “less AI” as a compromise. It is a more deliberately engineered system.
The question we ask is not:
How much can the model do?
It is:
How little does the model need in order to solve this problem well?
Smaller can be the smarter choice
The same principle applies to model selection. The largest available model is not automatically the best model for an enterprise use case.
In the document-focused aviation use case discussed in Part 5, a smaller targeted model was appropriate for the problem. The solution did not need an expensive cloud model carrying unnecessary capability.
The model could run on laptops.
That decision was possible because the use case had been understood first. Model selection followed the problem. Not the other way around.
For enterprise leaders, this matters because model capability is only one part of the decision.
Cost, infrastructure, control and operational requirements matter too.
Today’s model should not become tomorrow’s constraint
There is another risk in giving the model too much responsibility.
Dependency.
Get Agile Actors’s stories in your inbox
Join Medium for free to get updates from this writer.
AI models and provider capabilities are changing quickly. A model that is the obvious choice today may not be the preferred option in the future.
· Commercial terms can change.
· Capabilities can shift.
· Models can be updated or deprecated.
If the entire solution is deeply dependent on one model or provider, changing direction becomes expensive.
We therefore try to separate the wider system from the model wherever practical.
In one insurance implementation, the architecture was designed so the current deployment could be migrated to another cloud environment in under two weeks.
The objective was not to predict which provider would be best several years from now. It was to avoid needing that prediction.
Design for change, not permanence
Traditional enterprise architecture often aims for stability. AI requires a slightly different mindset.
· The business problem may remain.
· The workflow may remain.
· The need for reliable data may remain.
But the model solving one part of the problem will almost certainly evolve.
That means organisations should understand where the model sits in the system, what information it receives, what it produces and how difficult it would be to replace.
This is not simply a technical concern. It is a question of business control.
· Can costs be understood?
· Can providers be changed?
· Can a smaller model be introduced?
· Can part of the workflow move away from AI if a traditional approach becomes more effective?
The more deliberately the system is engineered, the more options the organisation keeps.
Smart AI engineering is economic discipline
Enterprise AI is moving into a phase where “it works” will not be enough.
Leaders will increasingly ask:
· What does it cost at scale?
· What are we dependent on?
· How quickly can we adapt?
Those questions should not arrive after deployment.
They belong in the architecture from the beginning.
· Use AI where AI adds value.
· Use simpler engineering where simpler engineering works.
· Give the model the information it needs — not everything the organisation owns.
And avoid designing the business around a technology choice that may change faster than the business problem itself.
The goal is not to maximise model usage. It is to maximise the value of the system around it.
But even a well-engineered AI system can begin with the wrong decision.
In the final part of AI Lessons From the Work, Not the Hype, we look at why some of the most valuable people in an AI programme are those with enough understanding — and enough trust — to challenge the original brief.
