How I moved beyond synthetic tests to find and fix the “Lost in the Middle” problem in Large Language Models.

Chapter 1: The Worrying Whisper in the Digital Brain

It started with a nagging feeling, a quiet whisper of doubt that many of us in the machine learning world have felt. I am building incredible solutions with Large Language Models (LLMs), feeding them massive contexts — entire books, research papers, and legal documents — and asking them to reason over them. The demos look spectacular.

But what if the LLM, for all its digital brilliance, has a very human flaw? What if it gets bored? What if, in the middle of that vast sea of text, it starts to lose focus?

My concern was specific: I worried that the LLM might not hold onto the most crucial, and often the rarest, pieces of information. If a single, critical fact — a “needle” of information — was buried deep in the middle of a “haystack” of context, would the model find it when I asked? Or would it hallucinate, confidently making up an answer because it had simply forgotten the fact was there? This phenomenon, known as the “Lost in the Middle” problem, isn’t just an academic curiosity; it’s a critical vulnerability that can break the trust I place in these systems.

Chapter 2: The Mirage of the Synthetic Needle

My first step was to research existing solutions. The most common method is the “Needle in a Haystack” test. The concept is straightforward: you invent a piece of nonsensical information (the “needle”), like “The secret to perpetual motion is a squirrel named Bartholomew eating a glowing acorn.” You then insert this sentence into a massive document (the “haystack”) and ask the LLM a question about it.

While clever, I quickly realized this approach was fundamentally flawed for testing our real-world systems. Here’s a deeper look at why I decided against it:

1. It Tests for the Bizarre, Not the Relevant. Real-world “needles” aren’t nonsensical phrases. They are project codenames, specific financial figures, unique legal clauses, or critical technical parameters. These facts are rare and important within their own context, but they are not bizarrely out of place. A model that is good at spotting a sentence about a glowing squirrel might not have the same skill in discerning a subtle but critical detail in a dense financial report. I needed to test for relevance, not absurdity.
2. It Encourages Exact String Matching Over Comprehension. Because the synthetic needle is so unique, the test often degrades into a simple string search. An LLM can score highly just by being good at Ctrl+F. But true intelligence is about comprehension. In a real scenario, an LLM might correctly answer a question by synthesizing information from a sentence without repeating it verbatim. A rigid, synthetic test could falsely mark such a nuanced and intelligent answer as a failure.
3. It Doesn’t Reflect Our Data’s DNA. Every document set has its own unique character and distribution of information. A test based on a generic haystack of Wikipedia articles and a fake needle tells us nothing about how the LLM will perform with our legal contracts or our engineering specs. To have real confidence, the test must be born from the very data it will be used on.

I needed to move beyond this artificial construct. We had to build a system that could discover the natural needles already present in our documents and forge them into a reliable test.

Chapter 3: The Crossroads of Meaning and Machines

This led us to a critical design choice: how do I programmatically identify “rare information” within our own documents? Two paths emerged, representing a classic trade-off between power and pragmatism.

Path 1: The Semantic Search for Outliers (The Gold Standard)

The most powerful approach was to use Sentence Transformers. These are incredible models that convert sentences into high-dimensional vectors, where the distance between vectors represents a difference in meaning. Using this, I could find the true “semantic outlier” — the sentence whose concept was most distinct from the rest of the document. This method is intellectually pure; it understands that “The CEO has resigned” and “The chief executive has departed” are the same concept. It finds rarity based on meaning, not just words.

But this power comes at a steep price:

• Hardware Dependency: These models are massive, often requiring powerful and expensive GPUs to run efficiently. For a testing pipeline that needs to be run regularly, this was a significant barrier.
• Computational Cost & Speed: The process of loading a multi-gigabyte model into memory and generating embeddings for thousands of sentences across a large test suite is slow. It would create a bottleneck, turning a quick validation step into a lengthy, expensive chore.

Path 2: The Pragmatic Hunt for Keyphrases (The Engineering Choice)

The second path was N-gram TF-IDF. This classic, computationally-light technique identifies phrases (n-grams) of 2, 3, or more words that are frequent in one document but rare across a wider corpus of documents.

While it doesn’t understand “meaning” in the same way a Sentence Transformer does, it is exceptionally good at finding the very things that often represent rare and critical information in a corporate or technical setting: unique project names (“Project Phoenix deliverable”), specific jargon, technical identifiers, and key financial figures.

The Decision: I chose the pragmatic path. N-gram TF-IDF was the right engineering decision. It gave us 90% of the value for 10% of the cost. It was fast, ran on any standard CPU, and perfectly identified the lexical outliers that served as our “natural needles.” It was a reminder that the most advanced solution isn’t always the best one.

Chapter 4: Forging the Test: An Automated Assembly Line

With a method to find our needles, it was time to build the factory — an automated pipeline to test any LLM. The process flows like a well-oiled machine, with different LLMs even playing specialized roles.

1. The Miner (N-gram TF-IDF): The process starts by feeding a large document to our n-gram TF-IDF model. It scans the text and identifies the highest-scoring keyphrases. I then extract the full sentences surrounding these phrases to serve as our “ground truth” context.
2. The Questioner (A Small LLM): This ground truth text is passed to a small, fast LLM (like Gemma or a fine-tuned T5). Its only job is to read the text and formulate a clear, direct question that can only be answered by that text. This automates the creation of our entire test suite.
3. The Subject (The LLM Under Test): Now, the main event. I give the large, original document to our primary LLM and ask it the question generated by the Questioner.
4. The Judge (The Evaluator LLM): The answer from our subject LLM is sent to a final, impartial “Evaluator LLM.” The Judge is given the original ground truth text, the question, and the answer. Its task is simple: provide a structured JSON output stating whether the answer was “Correct” or “Incorrect” and a brief reason. This avoids any bias and makes the results easy to parse.

Chapter 5: The Moment of Truth: Why Recall is King

With thousands of automated tests running, the final step is to measure performance. While I track several metrics, for the “Lost in the Middle” problem, one metric reigns supreme: Recall.

• Precision measures how often the model’s answers are factually correct. It’s about not making things up. A failure in precision is a False Positive.
• Recall measures how often the model finds the answer when it’s present in the context. A failure in recall is a False Negative — the model gives up or hallucinates because it couldn’t find the needle.

For this specific test, a False Negative is the cardinal sin. It represents the exact failure mode I am trying to prevent. Therefore, a high recall score is the ultimate proof that the model is successfully retrieving information from the depths of its context. It’s the direct measure of its resistance to the “Lost in the Middle” problem. (Note: In this test design, there are no “True Negatives,” as every question is designed to have an answer, making the recall score an even more focused indicator of performance).

Finally, so bored

Throughout my 13-year journey as a software engineer, I have witnessed the evolution of various technologies, tools, frameworks, and best practices. The future seemed destined for an endless number of software techs. But now, thanks to AI and GPT, those boring struggles in software engineering will soon become a thing of the past.

This is my first blog on Medium. I’ve had so many scattered thoughts that were coming and going like boiling water bubbles, so I thought, “Why not boil every Medium reader’s mind?” What’s the worst that could happen, besides people pointing fingers and calling me silly? At least I’ll have tried my hand at something new to me.

The truth is, I am genuinely bored with typical software engineering tasks. And I am on a journey to kill the boredom. Though I have nothing else to do to earn a living, I can’t help but wonder what if we had Minions to complete general, very common tasks on their own. Each minion is known to do some set of atomic (cannot be broken down further) tasks. Well, I said atomic, but it falls to the reader's imagination. And there are one or many orchestrators for managing and instructing. On the other hand, if it really happens, I’ll be insecure and poor.

Each minion could be known for executing a set of atomic tasks, while an orchestrator manages and instructs them. Perhaps we could build a standard set of minions and share them among ourselves, or we could share the DNA so that anyone could build their own minion. Standardizing it could bring immense benefits, whether globally or within an organization. This DNA could be preserved, while minions have a lifecycle.

Strategy to kill boredom.

I am seriously bored of data engineering, there is no challenge there and of course, I am wrong. But I wish for an AI that can read the design document and build a data pipeline on its own.

One day my lazy mind fell into a dream when I was boring myself. And my mind started thinking, which is of course not new because the mind process from already gathered impressions/data, so I cannot boast “This is my idea….MY PRECIOUS…”. (Gollum in The Lord of the Rings). I was writing the design document and was thinking, Can someone do these things physically?

I wish an AI could read a design document and build a data pipeline on its own. This led me to the idea of breaking down tasks in the design document into atomic rules, especially in the context of data validation in the ETL process. I am narrowing the scope to make my point.

Data validation consists of validation rules on data. To simplify this process, we can:

1. Break down those rules into atomic rules.
2. Group semantically close rules together.

DNA

These atomic rules can act as genetic information for our future minions. We can do the encoding manually and share it in a global repository in an ideal world, which can give foundational knowledge to future AI. Though it looks like current advanced generative ML models like GPT can help us encode this DNA, maybe without manual intervention. The DNA encoding consists of WHAT and HOW

• WHAT: the genetic information like validation rules, factual data, common patterns, and common tasks, the minion is going to do
• HOW: the runnable environment with tools, techniques, lifecycle information, and configuration

Combining WHAT and HOW, completes the DNA encoding of the minion’s behavior. It is required to include a switch on and off mechanism for individual genetic information (atomic rules) so that minions can do specific tasks out of their known tasks.

Cultural encoding is another aspect to consider if we want to restrict ourselves. An outside entity would then have to adhere to this cultural purview to use HOW (preferred environment) and WHAT. Alternatively, the genetic information of WHAT can be shared and enhanced with HOW in another context.

But the common HOW and variety of WHAT globally will give a foundation to self-conscious AI, and for that I am sure about it. And it will happen someday. It is like everyone is eating and drinking Sun, mud, and water but in different forms and different ways.

Minion/Worker

Now, let’s build our minion or worker. We can take this DNA encoding and clone a minion, giving it a physical form, which seems very much possible in a given cultural context (HOW). However, our minion should only perform necessary tasks, which should be a subset of its genetic information (atomic rules). These tasks should be switchable on and off based on design instructions like design documents and, perhaps in the future, more high-level communications like use cases.

Manager

The manager or orchestrator, a smart, non-human entity that can read the design document given by the emperor clones an army from the DNA, and execute Order 66. (Star Wars Order 66)

A non-human manager could potentially organize cloning and complete social activities to achieve all desired goals. A manager can do the following.

• DNA Discovery and Resolution (Discover and gather the common rules)
• Clone the DNA and switch on and off the genetic information (Encapsulate the rules in a runnable environment and switch on and off certain rules)
• Do the activity as needed and die a natural death. (I am thinking of AI's natural death after its goal is reached)

There are many more things here and maybe it will take a lifetime. It is like I am walking the journey, which seems indefinite and gives the illusion that I am about to reach its end. Someday AI will understand and do relevant things on its own, for that future we should start building foundations.

More thoughts are in the next episode. Continued….