Architecting an Understanding Layer for AI-First Systems, and Why It Matters

Most people who build systems on RAG face a familiar problem: the answer is in the documents, but the system cannot find it. Sometimes, the missing piece is nothing but understanding the question.

Imagine a learner finishes a practice quiz and asks: “Can I try again?”

The knowledge base explains quiz retakes. But searching those four words returns instructions for retrying an upload. Without the relevant policy, the answer model says it cannot help.

The information is there. The system needs to connect the question to the quiz, then find the section called “Attempts and retakes”.

I encountered this while building a system that answered from approved documents. A short question missed a section that a fuller question could find. Before changing retrieval, I needed to check what the system thought the user meant.

Retrieval-augmented generation (RAG) brings external evidence into an answer. A poor response can still begin before retrieval, when the request loses its context.

Figure 1. Turn a short request into specific information needs.

Give understanding its own responsibility

The Understanding Layer turns a request and its context into a structured plan. It identifies what the user means, separates questions with several parts, and chooses the resources needed to address them.

In this design, it is one module with a defined input and output: request and context in, plan out. It brings together query rewriting, task decomposition and routing.

The planner never answers. It proposes a plan; the application checks it. Evidence and tool results support the final response.

This applies beyond RAG. On a deployment dashboard, “Can I try again?” could mean checking whether a retry is allowed or starting one. Understanding the request matters before either search or action, whether it arrives through text, voice or an API.

Figure 2. The application validates the plan before using a capability.

Show the planner what is available

The planner needs two descriptions:

• A knowledge map: document versions, section titles, locations and short summaries.
• A capability catalog: available tools, required inputs, prerequisites and whether each tool changes data.

Build the knowledge map when documents are imported. Keep the original passages separately for retrieval. A section key such as “S3.2” gives the planner a precise place to search.

The map helps it navigate; it is not evidence. Generated summaries can miss conditions, so answers must still rely on source passages.

Figure 3. Use the map to find evidence, then use the evidence to answer.

Keep the map small by shortening summaries and selecting relevant source groups. Show only resources the caller may access. Define tools in application code, and update the map when its sources change.

Make the plan concrete

For “Do I have attempts left, and can I retake this quiz?”, the system needs both personal attempt data and a documented policy. An illustrative plan looks like this:

{
  "intent": "check_retake",
  "target": "current_quiz",
  "parts": [
    {"need": "Attempts left", "capability": "quiz.read_attempts"},
    {
      "need": "Retake policy",
      "capability": "knowledge.search",
      "locations": ["S3.2"],
      "search_phrases": ["practice quiz retake attempts limit"]
    }
  ],
  "clarification": null
}

The server resolves “current_quiz” from the authenticated application state and checks access. It also validates the tool names, inputs and section references. The planner cannot establish identity or permission by writing an identifier.

If the quiz or request is unclear, ask a specific question. Checking retake eligibility does not authorize starting another attempt.

Check evidence for each part

Read the remaining attempts through the quiz tool and search documents for the policy. For document questions, combine keyword and vector search. Reciprocal rank fusion can merge their rankings without requiring comparable scores. [1]

Get Injamamul Haque Sonet’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Search the suggested sections too, but keep useful results from the broader search. A wrong location hint should not push out good evidence.

Figure 4. Check each information need separately.

Check whether each part is supported, missing, outdated or conflicting. Apply source-version rules. When current sources still disagree, explain the conflict.

This allows an honest partial answer: the system can report remaining attempts while saying it could not confirm the retake policy. The missing policy may still block starting a new attempt.

Keep permission checks in the application

A valid plan is not permission to act. Requests, documents and tool results can contain malicious instructions. Treat the plan as untrusted input and enforce access in retrieval and tool handlers. OWASP recommends layered defenses and least privilege. [2]

Before an action runs, check permissions, current state, prerequisites and required approvals. Use idempotency controls so a repeated request does not accidentally perform the same action twice.

Figure 5. Application policy controls whether an action can proceed.

Keep logs limited to what you need. Plans and search phrases can contain personal details, even when the original message is not stored.

What testing taught me

Context placement matters. Putting the map between the conversation and the latest question caused unnecessary clarification on follow-ups. Moving it ahead of the conversation, clearly marked as data, helped preserve the connection.

More reasoning did not help in my tests. A higher setting added two to three seconds without improving accuracy. A clearer instruction about follow-ups helped more.

Add hints without replacing evidence. I kept the main search results and added a limited amount of relevant section evidence. For narrowly filtered searches within one document, I used exact distance search after approximate search returned nothing. That was specific to my setup; pgvector also supports iterative scans for filtered approximate search. [3]

Shorten the map gradually. I reduced summaries to 100 characters, then 60, before removing them. Dropping every summary at once removed useful guidance.

Repeat the evaluation. Different questions failed on different runs. One successful run did not establish reliability.

Measure whether the layer earns its cost

Planning adds latency, tokens and another opportunity for error. Limit the number of plan parts and retrieved passages. If nothing has usable evidence, allow one more search within the same access rules, then acknowledge any remaining gap.

Compare your existing pipeline, query rewriting alone and the full planner. Repeat tests with short requests, follow-ups, compound questions and malicious inputs. Measure accuracy, evidence coverage, unnecessary clarification, unauthorized actions blocked, latency and cost.

A planner cannot recover information that was never imported correctly.

Why it matters

Separating understanding from retrieval and execution makes failures easier to diagnose. You can inspect the plan, see where the request went wrong, and improve that stage without rebuilding everything else.

For AI-first systems, this matters whenever interpretation drives product behavior. A capability can only be as good as the request it is given. Make understanding the request a layer of its own.

References

1. Elastic: Reciprocal rank fusion
2. OWASP: LLM Prompt Injection Prevention Cheat Sheet
3. pgvector: Filtering and iterative index scans

This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!