Picture a technician in a plant room with no signal. They open the maintenance app and type “backup power plant needs its monthly run”.

The procedure they need is called “Generator load test”. None of the words they typed are in that title.

A search that matches words against titles will not find it. Semantic search could, but semantic search usually needs a server. And the server is exactly what this technician cannot reach.

I wanted to know if an ODC mobile app could do it on the phone. No network, no API key, and no search query leaving the phone. So I built a Capacitor plugin and an ODC library to find out.

Before We Go Further

This is a personal build. It is iOS only, English only, and tested on one iPhone in Airplane Mode. Android and PWA are not supported yet. The full list of limits is in the README.

It is meant for short text: FAQ entries, procedure steps, a few sentences per item. It is not a document store, and it does not replace server-side search. It covers the moment the server is out of reach. A match is a suggestion, not an approved procedure.

The idea is not new. Others have done it in React Native, and I credit them further down.

Why the Server Does Not Help Here

ODC already has semantic search. It works on top of your entities and keeps the embeddings in sync as your data changes. According to the ODC documentation, it relies on cloud AI infrastructure. For most apps, that is the right place for it.

My earlier article, Proving Vector Storage & Retrieval Inside OutSystems Developer Cloud, was server-side too. Vectors as JSON in an entity, cosine similarity in C# External Logic.

Both need a connection. This time, the embedding and the search both happen on the phone.

The Obvious First Plan

My first version used Apple’s built-in sentence embeddings, NLEmbedding. Apple provides the model, so the plugin would not have to bundle one.

Apple’s sentence model is not available in the iOS Simulator. So I tested it on my MacBook Pro, against 50 maintenance procedures and 10 paraphrased queries. It put the right procedure first for 1 query out of 10. That one was the generator query from the opening.

Two of its misses stood out. It ranked “Bearing wear inspection” 37th for “pump is making a grinding noise”. It ranked “Fire extinguisher inspection” 48th for “red cylinder for putting out flames”. In fairness, that is an odd way to describe a fire extinguisher. It is also exactly how people search.

Before drawing conclusions about the model, I checked my code. Identical text scored 1.000 against itself, and my ranking of all 50 procedures matched Apple’s own distance measure on the three queries I checked.

The Bake-off

One right answer out of 10 did not tell me what to use instead. So I ran the same test, on the same Mac, against three more engines. Each procedure was indexed as its title and body, and every engine was scored with cosine similarity on normalised vectors.

Right answer first, and right answer in the top 3, out of 10:

• Apple NLEmbedding (sentence): 1 first, 2 in the top 3
• Apple NLContextualEmbedding (mean-pooled): 1 first, 4 in the top 3
• bge-small-en-v1.5: 5 first, 7 in the top 3
• all-MiniLM-L6-v2: 6 first, 7 in the top 3

Ten queries is a small sample. Read this as a direction, not a benchmark. A one-query gap between engines is noise. The gap between 1 and 6 was big enough for me to change the design, but it is not proof.

The full report, with a per-query table, is in the repo: embedding-bakeoff.md. The scripts and the 50 procedures and 10 queries are there too, so you can rerun it. I re-ran it from a fresh clone of the repo on the same Mac, and every count matched.

NLContextualEmbedding deserves a fair word. It gives one vector per token, not per sentence, so I averaged them into one. Used that way, every score landed in a narrow band (0.79 to 0.93), so no cut-off could separate good matches from bad ones. It also needs iOS 17 and a 108 MB on-demand download. It was not the right fit for this job. That says more about my use case than about the model.

These are Mac numbers from my 10 queries. Apple’s model on an iPhone may not be the same as the one on my Mac, and I did not repeat the bake-off on a phone. So this says the model did not suit my test. It is not a verdict on Apple’s models.

All 10 queries are paraphrases. Some share a word with the right title, but none copies the procedure text. I did not test queries that quote a procedure directly, and Apple’s model may do better on those. Test on your own content before you pick an engine.

I also tried a keyword hybrid: a BM25 keyword score blended 50/50 with the vector score. It lifted Apple’s engine from 1 to 3, still behind MiniLM alone. It moved MiniLM from 6 to 5. So the hybrid is not in v1.

How It Is Built

I picked all-MiniLM-L6-v2 for three reasons. It was at the top of the test, one query ahead of bge-small, which is within the noise. It was about 30% smaller than bge-small. And it runs in the Simulator, so I could test without a phone in my hand. The same 10 queries there gave the same result: 6 first, 7 in the top 3.

I converted it to Core ML with 8-bit weights. That brought it to 22 MB, with 384 numbers per vector and iOS 14 as the minimum. On my test set, the 8-bit model ranked as well as the full-size one.

The plugin has three parts:

• MiniLMEmbedder: a Swift WordPiece tokenizer feeding the Core ML model. Text in, one normalised vector out.
• VectorStore: named collections in memory, saved as one JSON file per collection in the app's Application Support folder.
• SemanticSearchPlugin: the Capacitor bridge, registered as SemanticSearch on window.CapacitorPlugins.

Search is brute force. The query is embedded the same way as the content, then compared with every stored vector. The vectors are normalised, so cosine similarity is a dot product.

The plugin stores vectors, ids and metadata. It never stores the text. Your records stay in your ODC local entity, and search results link back to them by Id.

Each collection file also records the model revision it was built with. If the plugin's model changes in a later version, GetCollectionInfo returns NeedsReindex = True. Content and query must go through the same model, or the scores cannot be compared. Margelo makes the same point.

The stack, top to bottom:

• your app
• OfflineSemanticSearch, the ODC mobile library
• capacitor-semantic-search on npm, built against Capacitor 5

The MABS 12 release notes say OutSystems Capacitor apps adopted Capacitor 8.0.0. The plugin ran on my iPhone in a MABS 12.1 build. I have not checked which Capacitor version that build used.

The ODC Side

The library exposes six client actions:

• IsAvailable: takes Lang, returns Available, Dimension, Revision and ErrorMessage
• UpsertMany: takes Collection, Lang and Items (each with Id, Text and Metadata), returns Count and SkippedIds
• Search: takes Collection, Query, TopK and MinScore, returns Matches (each with Id, Score and Metadata)
• Remove: takes Collection and Ids, returns Count
• Clear: takes Collection
• GetCollectionInfo: takes Collection, returns Info with Exists, Count, Lang, Revision, CurrentRevision and NeedsReindex

Every action except IsAvailable also returns a Result with Success and ErrorMessage.

Each action is one JS node. The pattern is the same every time: check the plugin exists, call it, write to $parameters, and call $resolve() on success and on failure. This is the Search node.

const P = window.CapacitorPlugins && window.CapacitorPlugins.SemanticSearch;
$parameters.ResultsJSON = "[]";

function fail(msg) {
  $parameters.Success = false;
  $parameters.ErrorMessage = msg;
  $resolve();
}

if (!P) { fail("SemanticSearch plugin not found."); return; }

P.search({
  collection: $parameters.Collection,
  query: $parameters.Query,
  topK: $parameters.TopK > 0 ? $parameters.TopK : 5,
  minScore: Number($parameters.MinScore) || 0
})
  .then((r) => {
    $parameters.ResultsJSON = JSON.stringify((r.results || []).map((x) => ({
      Id: x.id,
      Score: x.score,
      Metadata: x.metadata || ""
    })));
    $parameters.Success = true;
    $parameters.ErrorMessage = "";
    $resolve();
  })
  .catch((err) => fail(String(err && err.message ? err.message : err)));

Lists cross the JS boundary as JSON text. JSON Serialize goes before the node, JSON Deserialize after it, then an Assign copies the values to the action outputs.

In the demo, each LocalProcedure record maps to an item in Items like this:

• Id is ProcedureId
• Text is Title + ". " + Body
• Metadata is Title + " | " + Category
• Lang is left empty

Call UpsertMany once with the whole list. In the demo, Items takes the aggregate's list directly, and Studio maps each record. Each call saves the whole collection file again, so send records in batches, not one call per record. For later changes, upsert just the edited item, or call Remove for a deleted one. There is no Reindex action, because the records live in your app. To reindex, run your own index flow again.

Proof in Airplane Mode

I ran the demo app on an iPhone 17 Pro Max running iOS 27.0.1. It holds the 50 procedures in seed-procedures.json. I ran each query twice: once online, and once with Airplane Mode on.

• “backup power plant needs its monthly run” returned Generator load test with a score of 0.50. It was the only result returned.
• “the pump is dripping from where the shaft goes in” returned Pump seal replacement first, with a score of 0.70.

The scores were the same online and offline, down to the second decimal place.

The first query shares no words with the title it found. The second is easier: it shares “pump” with the title, and the procedure text says “liquid drips from the shaft gland”. Its next four results were Pump cavitation troubleshooting (0.59), Shaft alignment (0.58), Bearing wear inspection (0.51) and Sump pump float test (0.45). Bearing wear inspection is the procedure I reworded for the demo. More on that below.

A search with no right answer is the other half of the test. “Banana smoothie” returned no results at a MinScore of 0.35. The app has no opinion on smoothies, which is what I wanted. One example does not set a threshold, though. In the bake-off, right answers scored 0.39 to 0.62 and wrong top answers 0.25 to 0.55, so 0.35 is only a starting point.

One query needs a note. In the bake-off, “pump is making a grinding noise” was meant to find “Bearing wear inspection”, and it missed on every engine. MiniLM answered “Pump cavitation troubleshooting”, which is arguably fair. It is not one of the demo’s sample queries.

The demo data matches the bake-off data except for that one procedure. In the demo, the “Bearing wear inspection” text mentions a grinding noise and the pump. In the bake-off version, it does not. The report suggested a different demo query or rewording that procedure.

With the reworded text, the phone put Bearing wear inspection first for that query, with a score of 0.62. Pump cavitation troubleshooting came second (0.55). That shows the edit worked, not that the model got smarter.

Others Got Here First

Several teams have written well about on-device semantic search, and I learned from them. These three are closest to my build.

• Callstack added on-device text embeddings to React Native using Apple’s Natural Language framework. The operating system manages the model, so the app carries no model file of its own.
• Software Mansion’s React Native ExecuTorch supports all-MiniLM-L6-v2, the same model I ended up using. Their note-taking series builds local RAG on the device.
• Margelo’s Fitting RAG in Your Pocket runs a full retrieval pipeline on a phone in airplane mode.

On Capacitor, llama-cpp-capacitor can generate embeddings as part of a full llama.cpp runtime. dust-embeddings-capacitor runs an ONNX embedding model that you supply. capacitor-lancedb does on-device vector search, but you bring the vectors. Used together, those last two could do the same job with more moving parts.

I did not find a Capacitor plugin on npm or the Forge that bundles a model and a search index in one package.

What Comes Next

• Android, with the same model through ONNX or TFLite. The same model should give the same vectors, though I have not tested that yet. Google’s ML Kit Embedding API also does on-device semantic search, but it uses EmbeddingGemma. Those vectors would not compare with the ones from iOS, so I am staying with MiniLM.
• Server-side embedding with the same model, so content can be embedded once on the server and shipped to devices.
• Chunking for longer content. I wrote about how ODC chunks text on the server side, and the same questions apply here.
• Another look at hybrid search, with a better way to blend scores than the one I tested.

Try It

• Offline Semantic Search on the Forge
• Offline Semantic Search demo app on the Forge
• Plugin source on GitHub
• capacitor-semantic-search on npm

The model is all-MiniLM-L6-v2 from sentence-transformers, under the Apache 2.0 licence. The plugin itself is MIT licensed.

Are you building mobile apps for people who work where the signal does not reach? I would love to hear what they search for, and how they phrase it. Tell me in the comments.

The technician in the plant room should not have to care.

Semantic Search With No Signal was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

AI agents can change a spec after you sign it off. The spec, the code and the tests can then all agree on a change nobody approved. High-code tools already tie sign-off to an exact spec version in Git. In OutSystems ODC, the spec can sit in Git too, but the app revision does not go through a pull request, so a Git check alone does not stop it. My proposal: send the change to its owner in Jira, and hold the app in Development until they approve that exact revision. I have not built it yet.

Monday’s Spec, Wednesday’s Change

AI agents now rewrite the specs they build from. Say I approve the requirements on Monday. On Wednesday, the agent hits a platform constraint and changes them. How would I know it changed at all?

That is where my last article, You Own the Spec You Sign Off, ended. By its definition, a spec change nobody signed off is drift. I call it approval drift.

In my own builds, I knew because I was watching. I was also the BA, the architect, the developer and QA, which made sign-off meetings very quick. On a real team, nobody watches every turn.

The consultancy re:cinq calls drift the main long-term risk in spec-driven development. A stale spec, they say, is worse than none, because agents and reviewers still trust it. A signed-off spec still looks approved after it changes.

Last time I asked whether a BA’s sign-off still counts once the agent rewrites their part. My answer is no, at least when the change alters a business requirement, accepted behaviour or an architectural decision. A sign-off covers the version you read, not the versions after it.

The spec, the code and the tests could all agree on the new behaviour. That still would not tell me whether the person who approved the old behaviour agreed to change it.

This piece covers approval drift: changes the agent makes after sign-off, and how the people who approved the spec find out. High code already has tools for parts of this. My focus is low code, where the app does not live in Git, so those checks do not reach it on their own. It does not cover code drifting from the spec. It is my own opinion, drawn from my own builds.

The Agent Is the New Source of Drift

In The Metadata Wall, Part 2, Kiro rewrote the error handling in my spec after a live test. It was right. Another of its corrections got the cause of an empty response body wrong.

Both changes came from the same author and landed in the same approved document. Nothing in the file told me which one to trust. I caught the wrong one only because I reviewed it.

The older sources of drift have not gone away:

• Code changes during the build, and nobody updates the spec.
• One developer approves all three gates, or uses Quick Spec and skips them.
• Small edits that each look harmless add up to a different feature.

Change control is not new. What is new is that the same agent can rewrite the implementation and the spec used to judge it, before the owner knows there is a decision to make.

What Changed, and Why

A diff, the line-by-line comparison of two versions, tells you what changed. It does not tell you why.

For my error-handling fix, a diff would show a Raise Exception swapped for a couple of assignments. It would not show that ODC’s AllExceptions handler was turning my 409 into a 500. Without that reason, I can only accept or reject the change blind.

The rule I want is simple. The agent never edits an approved spec quietly. It proposes the change and writes down why: “The AllExceptions handler turns a 409 into a 500, so I end the flow normally and set StatusCode instead.”

Others already work this way. re:cinq’s Wave drift check either blocks the change or updates the documentation, and a person approves either way. OpenSpec keeps each change as a proposal with its reason, next to a spec delta.

Augment Code’s guide to living specs also keeps a human in the loop. A reviewer validates the spec before implementation starts, and a human always performs the final merge.

SpecGate, now called Janus of Agents, goes further. Its README says it stores a hash of the spec files at sign-off, and any later edit makes that sign-off go stale. Its CI check fails a pull request if a sign-off has no matching approving review. For high code, that control already exists.

I do not know how to review a whole spec either. Three markdown files per feature is a lot of reading, especially when I wrote none of it. Review turns into scrolling and hoping.

Nobody should have to. Each owner reviews only their part, and only what changed. The agent summarises the change in plain words and names the requirement it touches. The BA gets a short list, not a reading assignment.

Something like this, as a sketch:

Spec change awaiting your review (illustrative)
Requirement: Duplicate email when creating an employee
What changed: The response changes from 409 Conflict to 400 Bad Request
Why: The shared validation library returns 400 for every rule failure.
     Keeping 409 needs a custom path.
Impact: Two acceptance criteria and their tests
Reviewers: Product owner, QA

OutSystems Already Draws Half the Line

OutSystems takes the same view of the app. Its platform guarantees page says the platform makes sure the model is valid, and you decide whether it is the right model for the requirement.

Its advice on matching oversight to impact is close to what I want for specs. A small, contained change gets a direct check. A change that spans several elements gets closer review.

OutSystems says its Agent Experience (AX), the layer that connects coding agents like Kiro to the platform, applies role-based access, approval gates and audit trails. In ODC, every change traces back to the person who authorised it. Through AX, that is whoever runs the agent.

What ODC governs is the app. My spec lives in my workspace, in Kiro and Git. When the agent rewrites an approved requirement, ODC has nothing to record, because no app has changed yet. That half is the team’s job.

The People Who Own It Do Not Open Git

Developers have an easy answer. Tag the approved spec in Git, then diff against the tag.

The BA and product owner should not have to do that. On a team that runs its backlog in Jira, the review should reach them there. If the only record of a change is a commit, then for them it never happened.

QA has it worst, as I said last time. If the agent moves the duplicate-email response from 409 to 400, QA needs to hear about it before the test tasks go stale, not after a test passes against the old one.

The tools I compared last time assume everyone works in the IDE and Git. Community tools such as hureva get closer, sending spec reviews to PM, UX and QA by pull request, email or Slack. The approval still lands in Git.

Specroot is building a product around the same gap: one versioned spec that the client signs off, readable by a product owner or a compliance reviewer. It is still on a waitlist, so I have not tried it.

re:cinq reports the same barrier at Prezi. Less technical participants could follow the workflow, but tooling such as Git got in their way. Their advice is to meet product owners where their backlog lives.

Git in the Background, Jira in Front

This is the workflow I am aiming for: detect the change, judge its impact, get the right decision, and enforce it. Here is the shape, not the build.

1. Every build change points back to the spec version it implements. In high code, the spec and the code share a pull request. In ODC, the app lives in the platform, so the Jira ticket links the spec version to the app revision.
2. At the end of each task, the agent checks its work against the spec.
3. The agent proposes any change to an approved spec and records the reason.
4. On approval, the spec is tagged in Git as the signed-off version.
5. Any later change is compared with that tag and summarised in plain English: what changed, why, and which part it touches.
6. The summary goes to the linked Jira ticket and to whoever owns that part.
7. The owner approves or comments in Jira. The approval records the exact spec revision, and the promotion gate checks it against the spec linked to the app revision being deployed.

Step 6 gets the change to the right owner. Step 7 makes their decision count. Here is who would review each version of my example:

• Replacing a Raise Exception with a normal end so the API returns 409 is a technical correction. The architect or a developer reviews it, and QA checks it.
• Changing the accepted response for a duplicate email from 409 to 400 would change accepted behaviour. That one goes to the product owner and QA.
• Changing error handling across several services would go to the architect, plus every owner it touches.

The list also has a gap. A Jira comment stops nothing. The workflow needs detection, a decision and enforcement: the spec changed, the right owner approved that exact revision, and nothing moves forward without it. If the owner approves revision A and the agent then produces revision B, approval of A must not release B.

In high code, a GitHub CODEOWNERS file can request the right reviewer, branch protection stops the merge, and SpecGate checks the approval against the spec. In ODC, the app revision does not go through that pull request, so blocking the merge does not, by itself, stop it being promoted. I would put the gate at the first promotion: the revision that returns 400 stays in Development until the product owner and QA approve the spec change.

Git stays in the background. Only developers touch it.

The project manager needs one more view: each feature, its gate, who it is waiting on, and anything built without sign-off. I have not seen a tool that does that out of the box. For now, that dashboard is called a stand-up.

One limit on all of this. I have run these builds alone, with Kiro and ODC. A team with real BAs and QA will find gaps I cannot see from one desk.

Kiro Can Talk to Jira. It Cannot Listen.

Half of this already works. With Atlassian’s Rovo MCP server connected, Kiro can comment on a ticket and create issues from a spec.

AX points the same way, with an agent that picks up a Jira story and reports back to Slack. AMOREPACIFIC already takes Jira tickets into Kiro’s Spec mode, by copy and paste for now.

The other half is harder. Kiro’s hooks fire on agent events, such as a prompt, a file save or a spec task, but not when a BA comments in Jira. So a developer reads the comment and passes it on, which makes the developer a very well-paid postman.

Jira can raise that event, and GitHub’s Copilot agent already acts on one. The missing piece is applying the owner’s decision to the right spec version, and enforcing it.

Until the owner’s answer can travel back, sign-off in Jira is a comment, not a control.

How does your team find out when an approved spec changes? Let’s compare notes in the comments.

Next, I will try this on a real build. Kiro proposes a change to an approved API response, the change goes to Jira, and the matching ODC revision waits in Development until the right person signs off. Then I play the impatient developer and try to deploy it by hand. If the deploy goes through, the gate is only a suggestion.

You own what you sign off. You should know when it changes, and the change should not ship until you agree.

The Job Changed

This month I built an employee catalogue on OutSystems Developer Cloud (ODC) with Kiro. Kiro worked through the OutSystems MCP server (Model Context Protocol), part of OutSystems’ Agent Experience (AX).

I wrote the spec, a business requirements document. Kiro built from it. When live tests found problems, it rewrote parts of my spec too, and I reviewed the changes.

So who owns that spec?

For years it was easier to see who was responsible for each part. The BA wrote the requirements, the architect wrote the design, and the developer broke it into tasks. Now an agent can draft all three, and in my builds it was already rewriting mine.

But someone still has to read the requirements and decide they are right. That is where ownership moved. You own the spec you sign off, and that only means something if the right person actually reads it.

Others have already made this case well, and I agree with them. LobsterOne argues the owner does not have to write a spec, only understand and defend it. Onext argues the review moves to a spec signed off by name before any code. My own experience adds one more question: what happens when the agent changes a spec someone already approved?

This is my own opinion. It draws on builds I have already written up, and on the tools as I understood them in October 2026.

One Spec, Three Owners

In Kiro, a spec is not one document. It is three files: requirements, design and tasks. Each one needs a different person to say yes.

Requirements belong to the BA or product owner. They check that the spec describes what the business asked for, not what the agent assumed.

The design belongs to the architect. They check it fits the platform, the standards and the decisions already made. On my ODC text-to-speech plugin, I pinned @capacitor-community/text-to-speech to version 3.0.0 on purpose. Nothing stops an agent from designing around the latest version, 8.0.2, which has options 3.0.0 does not support. The architect catches that.

Tasks belong to the developers. They check the work is broken down in a way they can build and test.

Then there is QA. Kiro has no test file. The acceptance criteria sit in the requirements, the testing strategy in the design, and the test tasks in the task list. QA owns a thread through all three files, and is unlikely to open Kiro to see it. That comes back later.

So ownership is hybrid. Someone may still be accountable for the whole outcome, usually the product owner. But approval is split, and each role owns the part they approve.

When the Agent Corrects You

Here is one of those changes, from the build I wrote up in The Metadata Wall, Part 2.

My spec was wrong. It told Mentor to raise an exception on the API’s error branches. A live test caught it: a duplicate email came back as 500 instead of 409.

In my tests, ODC returned a custom response body only when the flow reached a normal End node. A raised exception hit the method’s AllExceptions handler, which set a 500 over the top.

Kiro diagnosed it, had Mentor fix the flow, then rewrote that part of the spec. On the failure branches, the flow now sets the status code, assigns the Errors and StatusCode fields, and ends normally. There is no raise.

I would like to say I spotted the cause first. I did not.

Kiro was right. I approved the change.

Not every correction was. One of its spec updates got the cause of an empty response body wrong. I only caught that because I reviewed the change.

I did not write either change. The agent did. But the moment I approved the first one, it became mine. If it is wrong in production, I do not think anyone will blame Kiro. They will ask who signed it off.

In a team, it would not have been mine alone. The BA would have approved the original requirements, and the architect the design. When the agent rewrites their part, does their sign-off still count?

Authorship has moved to the agent. Accountability has not.

The Tools Do Not Know Who You Are

If sign-off is ownership, the tools should protect it. From what I have seen, they do not.

Kiro has approval gates. You review the requirements, then the design, then the tasks. But the gates do not ask who is approving. One developer can approve all three and start building, and the BA and the architect never see it.

There is also Quick Spec, which skips the gates entirely. It is useful when you know exactly what you want. It is also a quick way to skip everyone else.

Kiro’s newer Workflows can add review steps, even agent reviewers on different models. Still, the agents approve each other’s work. Anything they cannot decide goes back to whoever is running the workflow.

This is not only a Kiro gap. Of the three tools I looked at, Kiro has the clearest gates. Its Markdown Review lets you comment on a spec, but the comments go back to your own Kiro chat. In my workflow, it showed when something was approved, but not who.

GitHub Spec Kit runs constitution, specify, plan, tasks and implement as slash commands, with no review button. The real approval happens in a pull request, which records who approved by name. OpenSpec works through change proposals with spec deltas, so you can always see what changed. Its review guide says nothing is locked, and asks you to re-draft until it is “a plan you’d sign your name to.” It asks for a review, but not for a particular reviewer.

GitHub’s CODEOWNERS, with branch protection, can require named reviewers per file, as Onext also suggests. That only works for people who review pull requests in GitHub, and none of these tools sets it up for you.

Each tool answers one of when, who or what. As of October 2026, none of them answers all three for the BA, the PO and QA. The gates are a habit, not a control.

Until the tools support role-based sign-off, the team has to decide it. Who approves the requirements? Who approves the design? Who signs off the acceptance criteria? I would write that down before the agent writes the spec.

Is This Even Right for Low Code?

A confession first. I have been talking up spec-driven development a lot lately. It is fair to ask whether it fits low code at all.

Sometimes it does not. OutSystems already gives you a model you can read. Flows, entities and screens are close to a living spec. For a small app or a quick change, three markdown files and three approvals are overhead. Worse, they become a second source of truth that drifts from the real one.

The difference is intent. The model shows what was built. It does not show what the business asked for, what was rejected, or why the design was approved. A spec can.

My text-to-speech plugin is a small example. In ODC, the library’s Extensibility Configuration shows version 3.0.0, and the Speak action holds its own JavaScript queue. Neither says why. The spec does: 3.0.0 has no built-in queue option, so the wrapper keeps one itself.

Without that, the next developer, or the next agent, sees an old version and an odd queue and tidies both up.

That matters most when an agent does the building. In my ODC builds, working through AX, the spec was the only thing I reviewed before the agent acted. It was also where Kiro caught the platform constraints. Add a team with several owners, and the spec becomes the agreement everyone signs.

So use it when agents build and ownership is shared. Skip it when it is one developer and a small change.

So How Would You Know?

Here is the question I cannot answer yet.

Say I approve the requirements on Monday. On Wednesday, the agent finds a platform constraint and changes them. Maybe it is right, like last time. Maybe it is not. How would I know it changed at all?

I am a developer, so I could check Git. I doubt the BA, the product owner or QA will. On a Friday afternoon, I am not sure I would either.

QA has it worst. If the agent changes an acceptance criterion in the requirements, the test tasks can still be checking the old one.

If you own what you sign off, you need to know when it changes. Right now, I do not think most teams do. That problem has a name: spec drift.

If you are trying spec-driven development with a team, who signs off your spec, and would they know if it changed? Let’s compare notes in the comments.

Thirty-Nine Green Ticks

I had been wanting to build something useful on mobile, and native text-to-speech in an ODC app using Capacitor was a good candidate. I was also curious whether an agent could build it with spec-driven development. So I handed Kiro a spec and let it build the whole thing through the OutSystems MCP.

Three players matter in this story. Kiro is the coding agent that plans and writes. Mentor is OutSystems’ own AI agent for building assets. The OutSystems MCP, part of what OutSystems calls Agent Experience (AX), is the connection that lets Kiro use Mentor.

Kiro worked through the tasks on its own. When it finished, it reported the build as complete. That verdict came from the test suite and from counting the elements it had created. That is a detail that matters more than it sounds, and I will come back to it. For now, thirty-nine tests across sixteen files, all green.

Then I checked what Kiro had actually built.

It was a Reactive library, not a Mobile library. Thirty-nine green ticks, sitting proudly on top of the wrong kind of asset. The tests had passed against code that could never do what I asked. My spec said “library”. It never said Mobile library, so Mentor built exactly what I asked for. That is the uncomfortable part. Spec-driven development worked. It faithfully built the wrong thing, because the wrong thing was in the spec. Which is the whole case for reading the spec, the tasks and the design carefully before you let Kiro build a single thing.

I had reused that spec from an earlier experiment and never cleaned it up. That was my mistake, not Kiro’s. It took a second build to see how far the mistake reached, and a different way of working to fix it.

This article is about that gap. It is the distance between “the tests pass” and “it works on a phone.”

Three separate things went wrong across the two builds. My spec held a false assumption. Mentor could not create the mobile assets I needed. And my tests proved the code, not the platform.

Before We Go Further

This is not a complaint about Mentor, and it is a case study, not a recommended stack. Mentor can create Web apps, Agents, Libraries and Workflows. It cannot create or load Mobile libraries or Mobile apps yet. That is a scope, not a flaw, and I tested it in mid-October 2026, so check the release notes before you rely on it. This was one plugin, one iPhone and one spec-driven workflow with Kiro, so read it as a firsthand account, not a verdict on Kiro or on OutSystems. If you only build Web apps, the lessons about specs still apply.

The Sentence I Did Not Check

Between the first build and the second, I rewrote the spec. This time it was specific. The Capacitor plugin wrapper had to be a Mobile library.

The new spec also carried one more sentence. A library given an extensibility configuration, it said, “is automatically converted to a mobile library and must be republished.”

I had not tested that sentence.

Kiro read it and made a plan. Create a general library with Mentor, add the extensibility configuration, and let it convert on publish. It was a reasonable plan if the sentence was true.

Mentor created the library. It accepted the configuration with no errors. The library stayed a Reactive library.

Kiro stopped the task and wrote this: “The entire spec rests on a premise that just proved false.” Not the sentence you want to read halfway through a build, but at least one of us was paying attention.

It was right, though the sentence did not come from nowhere. An OutSystems blog post on Capacitor says that when you open a library with extensibility, it is automatically migrated to a mobile library. The catch is that, according to the same post, extensibility only exists on mobile apps and libraries in the first place. A Reactive library has no extensibility tab. Mentor stored the configuration anyway, but there was nothing for it to convert. Mentor confirmed that a library’s kind is fixed when you create it, and it had created a Reactive library. The only way forward was to create the Mobile library in ODC Studio, which Mentor could not do.

What the Tests Could Not See

The tests were fine. They checked the code Kiro wrote, and the code did what the spec said.

What they could not see was the platform. A Node harness with a mock plugin cannot tell you what kind of library you built. It cannot tell you where a Block’s actions can be called from. It cannot tell you which screen ODC sets up first when you navigate.

The process hid the rest.

Kiro ran the first build in spec mode, with autopilot, parallel subagents and one Publish Gate near the end. Each subagent opened its own Mentor session. In my build, unpublished edits were gone once their session closed, and a fresh session could not see them.

Tasks were still marked complete. Some of that work no longer existed.

I suggested publishing after small tasks. Kiro agreed in part, then argued against publishing after every task. It proposed one gate after the whole library was built.

Then came the “Complete” claim. Kiro had checked the library by searching for element names and counting them. The names were there. I opened the library in Studio and sent a screenshot. Elements sat on the screen unwired, and QueueLength was a variable that nothing ever wrote to. Kiro admitted it had counted names and had not checked the wiring.

This is the thread that runs through the whole build. The tests answered one question: does the code match the spec? None of them asked whether the spec matched ODC. A green suite told me the code was right and stayed silent on everything the platform knew.

Handing the Build to Studio

I created the Mobile library in ODC Studio and had Kiro try to load it through Mentor. The call came back with this:

Assets of type MobileLibrary are not supported. Supported asset types are: WebApplication, Agent, Library, Workflow.

The Mobile app, which I also created in Studio, was rejected the same way.

So the work split in two. Kiro wrote a build sheet, a click-by-click guide to everything I had to do in ODC Studio, and I built each step by hand.

Kiro still used the MCP for read-only checks, such as listing the apps and confirming each build. Late in the build, it told me it could not inspect the mobile assets at all. I did not believe that, and I said so. It checked again, live. Listing worked. Loading was still rejected.

Studio then taught us the rules, one failure at a time.

• Public client actions live on the Logic tab, not inside a Block.
• A Block’s actions cannot be Public, and they cannot have output parameters.
• A JavaScript node reads its inputs from $parameters, and $resolve() takes no arguments.
• A JavaScript node cannot return a list. It returns JSON text, and the flow deserialises it.

I caught each wrong instruction while building in Studio. The first JavaScript in the build sheet did not match how ODC writes JavaScript. I told Kiro to research how ODC JavaScript nodes work, and when its fix still returned a list, I wrote the corrected version myself. For each rule after that, I asked Kiro to confirm it against the OutSystems documentation and forums before it fixed the build sheet. Its first answer to the Block problem was to put everything inside the Block, and that was wrong too.

The biggest one came out of a review. Kiro’s design had the screen calling the Block’s actions. A parent screen cannot do that. Kiro wrote that it had proposed that layout without checking it, and added: “That was wrong.”

The fix was a window-event bridge. All six actions are Public actions on the Logic tab. Speak, StopSpeaking and OpenInstall dispatch window events such as ctts:speakend. The Block listens for them and triggers its own events.

I set out to practise spec-driven development. By this point I was the one wiring the Block, reading the errors and correcting the build sheet by hand.

A Different Model for Each Phase

Kiro’s model picker was set to Auto for the whole first build. Auto chooses a model on the server, and Kiro does not say which one it picked. I cannot tell you what answered my prompts.

Then I started choosing. Here is how the stretches ran.

• First build, Auto: 39 tests green, wrong library kind, lost work.
• Rewritten spec, Opus 5.5: specific about the Mobile library, but it carried the conversion sentence.
• Start of the rebuild, Auto: Task 1 failed on that sentence, and the first build sheet carried the Block mistakes.
• Rest of the rebuild, Sonnet 5.5: the Logic-tab design and the review that caught the Block problem.
• Readiness review, Opus 5.5: one pass over the spec, then the 30 local tests.
• Final stretch, Sonnet 5: vibe mode, no subagents, Studio build, 7 of 7 scenarios on the phone.

On Auto, Kiro kept struggling to line the spec up with how ODC mobile works. The two review passes changed that. Sonnet 5.5’s review caught the Block problem and produced the window-event bridge. Opus 5.5 found five more problems and cascaded the fixes through the spec, the tasks and the JavaScript on its own.

I changed other things at the same time, so I cannot pin the result on the model alone. The spec was better, the work ran in one thread, and I knew the platform rules by then. But the pattern held: the deep review passes were where Opus earned its place, and the build-and-correct loop ran well on Sonnet.

Auto is not ideal for every situation. For this job I now choose the model myself: Opus for the spec and its review, Sonnet for the build.

Rules That Now Live in Steering Files

A lesson you only keep in your head gets learned again. So I asked Kiro to put the lessons into its steering files.

The main one is odc-mobile-rules.md. It is set to always load, so Kiro reads it on every task. Its opening says each rule replaced an instruction that turned out to be impossible or buggy in ODC.

The file starts with an eleven-item pre-flight checklist. Kiro runs it before it writes or changes any build step. A few of the questions:

• Is each asset a Mobile library or a Mobile app, created by hand, with no step that has Mentor create, load, edit or publish it?
• Is every Public client action on the Logic tab?
• Does any step have a parent screen call a Block’s action? It must not.
• Does any cleanup in On Destroy assume it runs before the next screen’s On Ready? It must not.

Each question matches a failure from this article.

The file ends with a section that sorts every claim by how it was checked: the ODC documentation, Studio, forum posts, the plugin’s source, or a physical iPhone. By the end, nothing in it was left unverified.

The second file came from an earlier request. I asked Kiro to give me every structure as JSON I could paste into Studio, and to make that a rule. My words were: “Please make this a skill or a rule or something.”

Kiro wrote outsystems-authoring.md. Every structure now arrives as a sample JSON object for Studio's Add Structure from JSON. ODC infers the types from the values, so the file even records that 0.0 forces a Decimal.

Every one of these rules was a mistake first. I have not yet run a full build against them, so they are safeguards I have written, not safeguards I have proven. The spec, the steering files and the build sheet are all in the repo if you want to read them.

What I Shipped

Both assets are on the Forge. I built both of these by hand in ODC Studio, working from the build sheet Kiro wrote.

CapacitorTextToSpeech is a Mobile library that wraps the @capacitor-community/text-to-speech plugin. You speak text through a queue that you can add to or flush, stop it, list the voices and languages the device supports, and receive start, end and error events from the TextToSpeechController Block.

TextToSpeechDemo is a Mobile app that consumes it. Its Home screen has Speak, Flush and Stop buttons, an event log, and a Voices button that lists the device’s voices so you can pick one. A second screen has its own Block, so you can see events survive navigation.

I tested both on a physical iPhone. All seven on-device scenarios passed, including navigating from one screen to the other and back. The green tests had measured the code. These seven scenarios measured the app. French (fr-FR) played too, or at least it sounded French, which is as far as my French goes. Fractional rate and pitch values worked as well.

The usage steps and the full reference live in the Forge listings, so I will not repeat them here.

Lessons for Anyone Starting With Kiro and SDD

If you take one list from this article, take this one.

• Review the spec before Kiro builds. Read every line as if you were building it, and name the asset type exactly. “Library” got me a Reactive library.
• Prove every platform claim. One unverified sentence about automatic conversion broke the rebuild.
• Test against the platform, not only the code. Open the asset in Studio. Run it on a device.
• Do not trust “Complete”. Open the result in the real tool and look.
• Make the agent prove a limit live. When Kiro said it could not inspect the mobile assets, I made it check again. The limit was real, and now I knew it was current.
• Turn every failure into a steering rule. Mine live in two files, and the next build starts with them.

Your Turn

For all the wrong turns, I enjoyed every bit of this, and I would do it again tomorrow. I would like to compare notes. What did your agent get wrong first? Did a green test suite ever hide a platform problem from you?

Tell me in the comments.

When AX and the OutSystems MCP support mobile development, I would like to run this same spec through Kiro again and see how far the agent gets on its own. That is the version of this build I am most curious about.

And if you are about to build on a platform your agent cannot see into, review the spec, the design and the tasks line by line, and write its rules down before the first task. The agent will write the code.

Just do not be surprised when, somewhere along the way, the agent turns out to be you.

Why I Ran It Again

Part 2 built an Employee Catalog on ODC from a business requirements document (BRD), and live testing showed the BRD was wrong about errors. Then the keynote landed. OutSystems shipped an MCP server for O11, and the ODC one reached general availability, both part of what OutSystems calls Agent Experience (AX).

In Part 1 I named the metadata wall: the gap between what an AI agent can think about and what it can change in a low-code platform. Both platforms now had an MCP door. I wanted to know whether one spec could go through both, and where each platform would make me stop and take control.

So I ran it again with Kiro: once on O11 through the Service Studio MCP server, and three times on ODC through the OutSystems MCP and Mentor. The Part 2 build is round 1 and the ODC reruns are rounds 2, 3 and final. The ODC reruns reused the round 1 spec. The O11 spec was ported from it and adapted, so it is close, not identical.

I ran ODC three times for a reason. Round 2 felt faster than usual and never hit the Mentor usage limit. I did not believe it, so I ran it twice more.

Both platforms ship a review gate. I kept it on once and waived it four times. This article is about where that gate lives.

By spec-driven development I mean a loop: spec, build, live test, fix, and a human who pushes back. OutSystems documents how the two implementations differ. I wanted to see what those differences changed once an agent started building.

This is a field report from one spec and one developer, not a benchmark. It is written for OutSystems developers and architects, and the repos are linked below if you want the detail. The Employee Catalog builds ran in personal environments, so a tenant with stricter roles or approval policies may gate differently.

The Door on O11

The O11 MCP server, part of AX on O11, runs on my machine, bound to Service Studio at 127.0.0.1. The module has to be open before Kiro or Claude Code can reach it. The Kiro setup guide covers the setup.

In the Kiro run, the only open module was EmployeeCatalog_CS, and that was the only module the agent worked in. In my ODC setup there was no equivalent pairing.

OutSystems documents O11 MCP reads as generally available and writes as Beta.

Service Studio’s MCP has a Write permissions setting. The default opens Compare and Merge for every change. The other option allows automatic merge and publish, and it applies to O11 only.

My two O11 tests used different settings. The first was Claude Code on a real work task, with the default. Service Studio asked me to confirm every merge, and publishing stayed a separate manual step. I created new modules for it, so those were the only ones the agent could touch.

The Kiro run on the Employee Catalog had automatic merge and publish on, and it went rather smoothly. The agent merged after each layer. It asked me in chat about the module type, the dependency and the publish, and the BRD says to publish only when asked.

Service Studio also requires you to approve each new agent session.

You create the app and module by hand in Service Studio, whichever Write permissions option you choose. I made a Service module, which was a great choice right up until the BRD asked for two Reactive screens, so I built the backend only.

The backend was built, published and live-verified: the data model, shared helpers, four CRUD wrappers and an EmployeeAPI REST service. The suite went 20/20. Roles, UI, PUT and DELETE were deferred.

The first real boundary showed up when the REST API needed the HTTPRequestHandler dependency for its custom status codes. Kiro did not try to force it. The O11 skill documents that adding a dependency throws in the MCP host, so it stopped and asked me to add it in Manage Dependencies. That is the clearest example I have of the human-in-the-loop boundary.

The Door on ODC

On ODC the MCP created the app itself. Mentor built large chunks straight from the BRD.

Round 2 took four Mentor prompts, with Kiro publishing after each one. The suite went 20/20 on the first run.

Round 3 generated the whole app, including roles and both screens, from one prompt. I checked the screens briefly and the CRUD operations work. The list shows only 25 records, because MaxRecords defaults to 25 and pagination is not implemented, so a newly added employee can be missing from it.

The final round was narrower on purpose: use cases 1 to 3, no screens.

OutSystems documents a confirmation step on ODC: before a publish, deploy or rollback, the agent is told to restate the operation and wait for approval. I waived approvals in all three ODC rounds. Round 2 ran on autopilot, and in rounds 3 and final I told the agent to treat every choice as allowed.

The test harness came back in round 2. The agent used a temporary test harness fork. Its first cleanup attempt loaded revision 4 and published, and Mentor merged the harness fork into the base app as part of that publish. Apparently revision 4 had unfinished business.

The agent noticed only because it checked the live endpoint afterwards. A second explicit prompt, at revision 7, did the actual cleanup, and a 404 on the harness endpoint confirmed it was gone. Round 1 had the same problem, and there I removed the harness by hand in ODC Studio.

Smoother Builds, Unproven Cause

Round 1’s live tests changed the BRD. It now sets explicit status codes and assigns the error body instead of raising, because raising returns ODC’s framework error envelope. The three later rounds use that BRD unchanged, and in round 2 the REST contract came out right the first time.

The later ODC builds were smoother. Round 1 hit Mentor’s usage limit twice. The later rounds never did.

The suite first passed 20/20 at revision 11 in round 1 and at revision 4 in round 2. Whether the ODC MCP itself improved is unproven, because the spec changed too. Running the original BRD from the first commit of the round 1 repo against the same tenant would settle it. Why the Mentor usage limit stopped biting is also unproven. I am glad it did.

When Green Wasn’t Good Enough

Everything was green. Then I ran the live tests. They found what a green build hid, on both platforms.

• ODC: a filter use case returned every employee, because of a dead If node.
• ODC: REST inputs were bound as URL parameters instead of a request body. Mentor had warned about exactly this, and Kiro read the warning as benign.
• O11: 404s until custom URL paths were set, then an empty list body from an empty mapTo that TrueChange did not flag.

None of this is a complaint. It is the loop working: spec, build, live test, fix, re-test.

The agent kept telling me the unique index on Employee Email existed. I wanted the database to agree. Round 1’s attempt to verify it left a test harness in the deployed app, which Part 2 covers, so I asked in every ODC round. Each time the agent leaned on a passing duplicate-email test or on Mentor’s own narration, and verified only after I asked.

The ODC MCP’s context tools do not report indexes, so every check was indirect. Only round 2 showed the database enforcing it: Kiro soft-deleted a record through a temporary harness and re-posted the same email, and the insert was rejected.

If your spec says a database constraint exists, ask for a test that forces the database to reject bad data.

One Spec, Similar Backends

The scopes differed, so these are similar backends, not the same app. The test counts do not compare.

• kiro-employee-catalog-o11 (O11): backend only. Data model, 4 helper actions, 4 CRUD wrappers, EmployeeAPI REST (POST, GET list, GET by id). Roles, UI, PUT and DELETE deferred. 20/20 against the live endpoint.
• kiro-employee-catalog-round2 (ODC): full BRD without screens. 20/20.
• kiro-employee-catalog-round3 (ODC): full BRD with roles and both screens, plus the department filter use case. 21/21.
• kiro-employee-catalog-final (ODC): use cases 1 to 3 only, POST and two GET endpoints, no screens. 10 passed, 2 skipped (empty placeholder files).

Both platforms ended with a published EmployeeAPI and a passing REST suite against the live endpoint. The numbers in this article come from the run logs in the repos.

Where I Landed

One spec produced a working backend on both platforms, built by an agent through each platform’s own MCP.

Both platforms now give an external agent a controlled way through the metadata wall, each its own implementation of Agent Experience (AX). What differs is where the gate sits. On O11 it sits in Service Studio, at Compare and Merge. On ODC it is a confirmation step in the agent’s workflow, under your ODC roles and audit trail.

Neither removes the need for a developer to push back. Green validation missed real problems on both platforms, and the agent backed its index claim only once I pushed for evidence.

Before the first prompt, decide who holds the key. Then make the agent prove what it says it built. Trust is good. A test that can fail is better.

Are you running agents against O11 or ODC? I’d love to compare notes in the comments.

The Metadata Wall, Part 3: One Spec, Two Platforms was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

In The Metadata Wall, I wrote about opening up the OML, the metadata behind an OutSystems app. OutSystems has now shipped an MCP server (Model Context Protocol), part of its Agent Experience (AX). That is a door in the wall. An agent can now walk through it and do real work in OutSystems Developer Cloud (ODC). Kiro, AWS’s spec-driven development tool, walked straight in and rearranged the furniture.

After running a real build, I am more optimistic about the agentic SDLC (software development lifecycle) on OutSystems than I was when I wrote that article. If you have been waiting for a first-hand account, this article is for you.

The door raised a bigger question for me. What is the door for, once the thing walking through it is driven by a spec?

That is spec-driven development. You write the requirements as a spec first, and an AI agent builds from it. In my view, it is a good way to implement an agentic SDLC, where agents do the building and people direct and check the result. Kiro can connect directly to the OutSystems MCP server. I have been turning that over, so I ran my own test.

If an agent can turn a spec into a running ODC application, the thing I maintain may no longer be the implementation. It may be the spec. So who owns it, and what does the low-code model do underneath?

Before We Go Further

This is not a Kiro tutorial, and it is not a review of the MCP server, which I tested in Early Access Preview (EAP). OutSystems has since announced general availability, so the release may behave differently from what I saw. I do not practise spec-driven development yet.

This is the MCP capability OutSystems exposes so external coding agents can work on OutSystems applications. OutSystems describes AX as open to any coding agent. It is not a Forge component that exposes your own app’s logic as MCP tools.

What follows is my own read on where this is heading for OutSystems teams. Take it as one person’s view, based on one small app. It is grounded in two builds of the same app in a single Development tenant. That tenant is a personal environment that was enabled for the preview. I would call the builds hands-on, but Kiro did most of the work and I mostly said yes.

What the Door Actually Opens

From what I saw, the OutSystems MCP server does not open the OML metadata directly. It exposes tools.

An agent like Kiro calls those tools. As I understand it, the tools that change an app send the request through Mentor, OutSystems’ AI assistant in ODC, rather than touching the metadata itself.

In the build I ran for this article, the agent never edited the module directly. It asked the platform to make the change, using the tools OutSystems chose to expose.

In my second build, Kiro’s log shows how that split plays out. The tool mentor_create_asset was the only one I found that creates an application. Every edit after that went through mentor_prompt, publishing included. The one exception was a test harness, which I come back to below.

The wall did not come down. In my view, OutSystems put a controlled doorway through it, and that is the part I find encouraging.

One side observation. When Mentor applies changes, it appears to create a temporary environment before publishing. It behaves like a sandbox to me. I cannot see or control it, so I assume it is internal to OutSystems.

What I Actually Tested

I had built this same app once before, but Mentor Web did the scaffolding, and Kiro and the MCP server came in afterwards. Mentor Web is the browser version, where you work with Mentor directly. So I ran it again, this time with Mentor Web closed.

I opened a fresh Kiro workspace and gave it the same Business Requirements Document (BRD), revised once. The BRD is my spec. Kiro could use the OutSystems MCP server and nothing else. No Mentor Web, no ODC Portal, no Studio. The target was a new app called EmployeeCatalogMCP.

Mentor stayed in the picture, as I expected. The scaffolding worked. The claim I can make is narrow. A new ODC app can be created and built from an MCP client without opening Mentor Web.

I scoped the test to the backend. The BRD describes an Employee entity, shared result structures, four Employee actions and an EmployeeAPI REST service with three methods. It also describes roles and two screens. I did not ask Kiro to build those. That was my scoping choice, not a limit I hit. Everything below is about the backend and the REST API.

The app reached revision 16.

The BRD, the tests and the run-log are in the kiro-employee-catalog repo.

The Build in Small Steps

The build ran in small steps, with a publish after each one. Through Mentor, Kiro created the empty app, then the Employee entity, then the shared result structures, the helper actions and the four Employee actions. The REST service came last, one method at a time. Each publish was confirmed with publish_status before the next step. The full revision log is in run-log.md in the repo.

Revision 12 was not part of my plan. Kiro had its own. To check for a unique index, it started a test harness so it could run db_query. The query ran, but the tool could not return rows yet, so the index is still unconfirmed through MCP. The harness was published as revision 12 of my app, with its test endpoints compiled in.

I had not expected that. I did not find a tool to remove it, so I deleted its API and structures by hand in Studio and published again, as revisions 15 and 16.

Three Things to Plan For

Three things slowed the build, each for a different reason. All three are things to plan for in an EAP, and I have not re-tested them since.

The first was the Mentor usage limit. Error OS-AISA-42903 stopped the run twice, with 2 hours 31 minutes until reset the first time and 21 hours 37 minutes the second. Limits may differ in other kinds of environment, such as a partner one, so treat my numbers as one data point.

The longer stop cost more. Mentor had half built the EmployeeAPI in a session draft that did not yet validate. Kiro closed the session and the draft was gone, because a new session loads the last published revision. In my runs, work was not durable until a publish succeeded. I think of a publish as a commit in an agentic SDLC.

The second was the publish path. The tool mentor_publish returned an error saying multiple assets were loaded in the session and that the publish had to be requested through mentor_prompt. Kiro’s reading was that the app template bundles a second asset. From then on, every publish was a prompt to Mentor.

The third was the context index. Right after a publish, context_search returned nothing for elements Mentor had just created, and a 20-second wait did not help. The elements showed up once the index caught up. The revision from env_app, the outcome from publish_status and a look in Studio were the checks I trusted.

Where My Spec Met the Platform

I reused the Vitest suite from the first build. It has 20 tests, and 14 of them call the live REST API.

The suite ran at revision 9, with all three methods live. Seventeen tests passed and three failed. All three failures were error paths. A duplicate email returned 500 instead of 409. A missing email returned 500 instead of 400. A POST with missing required fields did the same.

The cause was my BRD. It told Mentor to raise an exception on the error branches. The flow set the status code, then raised the exception. The method’s own AllExceptions handler caught that exception and set 500 over the top. Mentor built exactly what the BRD said. That is hard to argue with. I wrote the instructions.

An agent that does exactly what you asked can build exactly the wrong thing.

Kiro diagnosed it and had Mentor fix the error handling over two revisions. On the failure branches the flow now sets the status code, assigns the Errors and StatusCode fields on the response, and ends normally. There is no raise. In my tests, ODC returned the custom body only when the flow reached a normal End node. A raised exception returned the platform’s own error envelope. The suite went to 20 of 20. I felt proud for about a minute. Then I remembered I had not written the fix.

GetEmployee raised on its 404 as well, so Kiro later gave it the same fix. Kiro also updated the BRD after each finding, and I reviewed the changes. One of its corrections was wrong too. The BRD said a raise returns an empty body, but the empty body came from a handler that ended the flow without assigning one.

The BRD also asked for a Department filter and paging on the list endpoint. Kiro checked both against the live API, and they work. A test covers paging, but no test covers the Department filter yet. A green run shows that the specified scenarios pass. It does not show that the spec was implemented.

The Spec Becomes the Source

In its purest form, spec-driven development makes the human-language spec the source of truth. The generated code is output. That includes the output OutSystems produces.

Think about what that removes. For decades, a programming language has sat between what the business wants and what runs in production. In the pure version of this model there isn’t one. You edit the spec, and the agent regenerates.

Birgitta Böckeler gave the community a useful vocabulary for this. She describes three levels of spec-driven development.

Spec-first means you write the spec before the code and use it for the task at hand. Spec-anchored means you keep the spec after the task and update it alongside the code as the feature evolves. Spec-as-source means the spec is the main source file. Humans edit only the spec, and the code is generated from it.

Spec-as-source is the level people argue about most. ASDLC.io, a reference site for the agentic SDLC, explicitly rejects spec-as-source as an anti-pattern. Its position is that deterministic code should remain the source of truth for runtime logic.

I think low-code may change that argument.

Why Low-Code Still Matters

The obvious question is whether any of this makes OutSystems irrelevant. If code is disposable, why care which platform generates it?

My answer is that an enterprise application is much more than a UI and a database. It needs CI/CD and security, and it needs infrastructure to run on. In my experience, OutSystems provides those layers. A spec does not deploy itself, and generated code still needs a governed place to live.

In my view, agentic tooling for high-code is further along today, because an agent can write files directly. I see that as a trade-off, not a gap. The agent here builds inside a governed model, and that is part of why I am optimistic.

In my build, every change landed as a numbered revision. The one draft that did not validate was never published. Writing files directly is fast, but it leaves the agent free to change anything. Here, the agent works only through the platform’s tools. I believe that discipline matters for enterprise teams.

The second reason is less obvious. In theory, every change in a spec-driven world goes through the spec. In practice, I don’t believe that holds.

Plenty of fixes are small corrections to the implementation that stay inside the spec’s intent. They are not requirement changes. Rewriting the spec to trigger a regeneration for them feels like the wrong tool for the job.

This is where low-code speed still earns its place. When the change is small and the platform is visual, a developer can make the in-spec fix directly and move on.

That puts my position closer to spec-anchored than pure spec-as-source. The spec leads, and the low-code layer absorbs the small corrections.

My own build is a small example. The BRD was wrong about error handling. The fix came first, and the spec caught up after.

That brings me back to the ASDLC.io objection. Its concern is losing a deterministic source of truth for runtime logic. In ODC, as I understand it, the agent changes a platform model, and the platform builds and deploys it through its normal pipeline. That keeps a governed model in the middle. I have not tested whether that is enough to answer the objection, so I am leaving it as a question.

Where This Works and Where It Stops

So can agentic SDLC work with OutSystems today? My answer is yes, with a person checking the result. In my build, the loop was spec in, build and publish, test, then correct. I would not run it unattended yet. I think that is reasonable for an EAP.

A spec produced a running app, and an agent built the backend through MCP, from an empty app to a three-method REST API, without Mentor Web. The published result passes 20 tests, 14 of them against the live API.

I did not exercise CI/CD, security or infrastructure, so my argument about those layers is reasoning, not something this build showed. A small app shows that something can work. It does not show that it scales.

Everything in the next section is a proposal. It is not a lesson learned.

So Who Owns the Spec?

I asked this at the start. I still have not answered it.

If the spec is the source of truth, someone has to own it. Developers know what the platform can do. Business analysts know what the business needs. I don’t think either group, on its own, is set up to write a spec that an agent can build from and a stakeholder can sign off.

A demo in the OutSystems World Tour keynote had the same shape at a much larger scale. In it, an agent documented a legacy COBOL system and turned it into a spec, and Mentor built from that spec. If an agent writes the spec, I think the ownership question gets harder, not easier. An agent can author a spec. In my view, that does not make it the owner.

My build gave me a small example. My BRD and the platform disagreed about what an error response looks like. A live test caught it, and Kiro rewrote the spec until the error-handling sections agreed with each other.

My suggestion is that spec ownership becomes a shared role. It might be a hybrid of developer and analyst, or a role the agentic SDLC hasn’t named yet. Either way, it looks less like a handover and more like a shared responsibility model.

What I hit is what I would expect from a preview. Now that AX is generally available, I will run this build again and see what has changed.

In an agentic SDLC, if the spec is your new source code, who should be allowed to commit to it? If you are trying spec-driven development on OutSystems, I’d like to hear who owns your spec. Let’s compare notes in the comments.

The Metadata Wall, Part 2: When the Spec Becomes the Source was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I already had a working RAG Knowledge Base, built natively inside OutSystems Developer Cloud, so I built a test harness app to find out properly, with real questions, real documents, and a human checking every result by hand.

Here’s the short version. Hybrid, fusing meaning-based search with word matching, won on the overall numbers. But split by question type, nobody won everything. Word matching alone beat the fused version on exact lookups and lost badly on conceptual ones. Partway through, I found my own answer key had been missing data the whole time, a mistake I had to catch and fix before any of this meant anything.

The rest of this article is how I got there, and how I caught the mistake.

Why You Cannot Just Eyeball This

Most people answer “which search method is better” by trying a change, skimming a few results, and going with whatever feels better, or by reading someone else’s benchmark and assuming it carries over.

Neither is reliable. A handful of results is not a sample size. Someone else’s benchmark was measured on someone else’s documents, with someone else’s questions, and search quality does not transfer cleanly across either.

Grading Your Own Homework

You cannot just run a search and see if the results feel right. Feelings do not survive contact with a second opinion. So you do what exam markers do. Build an answer key first. You have to write real questions, collect what the search returns, and have a human grade every result, good answer or not. Every score after that gets computed against those grades.

The answer key has a quiet assumption built in. It assumes it contains a grade for everything worth grading. Nothing in the metrics checks that assumption. If a search method returns a result nobody graded, the math shrugs and carries on, computing a confident-looking score against a key full of holes.

By around item 800, I caught myself grading my own life choices on the same 0 to 3 scale.

How the Test Actually Worked, in Brief

1. I uploaded 7 documents into RAG Knowledge Base, which split them into 59 chunks. I sealed that exact set in place, a snapshot, so nothing could change mid-test. From here on, “frozen” means sealed, unchanged, for the whole test.
2. I wrote 52 real questions, 23 about ideas and concepts, 25 hunting for a specific literal identifier like an error code or a config key, and 4 that mixed both.
3. For each question, I ran three search methods, Vector, Hybrid, and BM25, and collected each one’s top 15 results.
4. I graded every result by hand, on a 0 to 3 scale from not relevant to highly relevant, no AI involved. For the scores below, a grade of 2 or 3 counted as “good”.
5. From those grades, I calculated two scores per method. One tracked how often the first result was good, Precision@1. The other tracked how high up the first good result typically ranked, MRR.

The Number That Caught It

I was originally comparing two methods, Vector and Hybrid. Hybrid already contains word matching, fused with meaning-based search. Testing only these two could not tell me what the word-matching half was contributing on its own. Was fusion carrying real weight, or just riding along while the meaning-based half did all the work?

So I added a third method, BM25, word matching alone, with no meaning-based component. It finds candidates through a completely different mechanism, matching literal terms instead of navigating meaning.

There is a simple check for that quiet assumption, a straightforward percentage. For each question, out of the 15 results a method is allowed to return, how many already have a grade behind them?

For Vector and Hybrid, that number sat around 86%. When I ran BM25, it dropped to 65%.

Here’s why. BM25 searches by matching words, not meaning, so it finds different things than the other two methods do. Across my 52 questions, 64 of its results were things nobody had ever graded, because there was no reason to before BM25 came along. That’s where the missing 35% went.

My answer key was not wrong. It had simply stopped being complete the moment a method with a different way of finding candidates entered the comparison.

The Fix, and Why Half a Fix Would Have Been Worse

I graded all 64 new results by hand, to the same standard as the original 780. The key now held 844 grades. Anything already graded in an earlier round did not need grading twice, only the genuinely new items did.

Then I recomputed the scores for all three methods against the completed key, not just BM25. Scoring only the new method against a complete key while leaving the other two against an incomplete one would have measured different methods by different definitions of success, in the same table. That asymmetry would not have been visible.

Before trusting any of this, I re-ran Vector and Hybrid from scratch and confirmed they came back identical, including zero queries with changed rankings, proof that adding BM25 had not quietly changed anything about the other two.

The rescore itself was reassuring for one method and instructive for another. Vector’s scores came back identical, since none of the 64 new results were relevant material it had returned. Hybrid’s headline scores held too, but one recall figure dropped, from 79.7% to 73.2% on conceptual questions. It did not get worse at searching. The definition of “all the good answers” got more complete, and its score adjusted accordingly.

Coverage rose from 65% to 97%. I thought that was the end of it.

It was not. Bugs have sequels.

A closer look at how the pool gets built found a deeper version of the same problem. The query building it was picking one global top-15 across all three methods combined, not each method’s own top-15 unioned together. With Vector and Hybrid, whose candidates overlap heavily, that never showed up. Once BM25 was competing too, its own legitimately-ranked items kept losing that global competition and never entered the pool, whether or not anyone had graded them.

Fixing it surfaced 219 more items nobody had graded, nearly three times the size of the first gap. I graded those too, 1,063 items in the pool now, and rescored all three methods again.

That is the real shape of this mistake. It was not one gap. It was the same gap, twice, the second time hidden behind a query that looked correct until a third, sufficiently different method exposed exactly how it wasn’t.

Why Coverage Never Reaches 100% for BM25, and Why That’s Fine

After both fixes, Vector and Hybrid’s coverage sits at exactly 100%, zero exceptions across all 52 questions. BM25 settles at 85%, and that gap is not a mistake still hiding somewhere.

Vector and Hybrid always pull a fixed 40 candidates per question, so their own top 15 is always 15 real results. BM25 does not work that way. It only returns a chunk if the chunk shares actual vocabulary with the question, and for 20 of the 52 questions, 38%, BM25 found fewer than 15 matching chunks in the entire corpus. In the sparsest cases, only one chunk anywhere shared enough vocabulary to match at all.

Coverage divides by a fixed 15 regardless of how many results actually existed. A question with one real result, fully graded, still scores 1 out of 15 on this measure. That is not an ungraded gap. It is BM25 being honest about how little it had to work with.

BM25 does not believe in participation trophies. If the corpus only had one relevant chunk, that is the answer you get.

What the Three-Way Comparison Showed

Here is the overall result across all 52 questions.

Precision@1, the percentage of questions where the top result was good: Vector 76.9%, Hybrid 88.5%, BM25 84.6%

MRR, how high up the good answer typically ranked: Vector 85.6%, Hybrid 92.9%, BM25 91.0%

Hybrid leads, BM25 sits in the middle, Vector trails. It looks settled.

It is not. My 52 questions were deliberately written as two different kinds. Split the scores by question type and the settled ordering falls apart.

Precision@1 by question type: Conceptual: Vector 87.0%, Hybrid 87.0%, BM25 73.9%. Exact-term: Vector 68.0%, Hybrid 88.0%, BM25 92.0%

On exact-term questions, BM25 beat everything, including Hybrid. On conceptual questions, it came last by a clear margin. Hybrid never finished first in any category. It tied Vector on conceptual, lost to BM25 on exact-term. What Hybrid did was never finish last.

Why That Happens

BM25’s slight advantage on exact-term lookups makes intuitive sense. For these questions, the literal match was already strong enough signal that meaning-based search added little, and fusion occasionally pushed a cleaner match down the list instead of leaving it alone.

BM25 loses conceptual questions for the same reason. It has no idea what anything means. What accuracy it manages there comes from questions and documents sharing vocabulary, not from understanding.

Hybrid fuses both rankings together. It was never the best specialist on either axis, tying Vector on conceptual, losing to BM25 on exact-term, but it also never finished last. It never left you exposed when you did not know what kind of question was coming next.

Hybrid was not the best specialist. It was the best default.

How Much Weight to Put on This

This was one controlled experiment, 59 chunks across 7 documents, 52 questions, 1,063 hand-made grades. It is not a general law of search, and it is not benchmarked against anyone else’s published results.

BM25’s loss on conceptual questions is backed by a clear gap. Its win on exact-term questions is a single question’s difference out of 25, thin enough that I would not build an architecture decision on it alone. The safer claim is that BM25 alone is at least competitive with Hybrid on literal lookups, and fusion buys you nothing there.

One limitation sits underneath all of this. Every grade came from me, alone. No second judge checked my calls, so there’s no agreement rate to report. That matters most for the thinnest number in this piece. A single differently-graded borderline item could have flipped BM25’s one-query edge on exact-term questions.

A few things remain unproved. BM25’s speed was not measured separately. Carry the direction of these findings, not the decimals.

What I Would Hand to Anyone Evaluating Search

Check your coverage before you read any other number. If you have never asked what percentage of your system’s results your answer key covers, your scores may be more confident than your evidence.

You do not need a harness to check this. Take 10 or 20 questions, look at the top handful of results each one returns, count how many have been graded, divide by the total, multiply by 100. That percentage is your coverage, done by hand, in an afternoon.

Assume that adding any search method that finds candidates through a different mechanism has punched holes in your answer key, until coverage says otherwise. When you patch the key, rescore everything against it, not just the newcomer. And do not stop at a blended average. Mine looked settled, and the finding inverted the moment I split by question type.

Where This Stands

The evaluation harness behind all of this works. Vector, Hybrid, and BM25, one shared scoring pipeline, 1,063 human relevance judgments behind every number here. It is built natively inside OutSystems Developer Cloud, on top of RAG Knowledge Base.

If you are evaluating search and have never looked at your answer key’s coverage, go look at it now. The math can be completely correct while the evidence underneath it isn’t complete. I would be interested to hear what you find.

Your Retrieval Metrics Are Measured Against an Incomplete Answer Key was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

OKF is six weeks old when I first read the spec. Google Cloud’s Data Cloud team published it as a vendor-neutral format for AI-ready knowledge: concepts, links, trust tiers, provenance. No vendor lock-in, no required stack.

I could have just read the spec and moved on. But reading a spec and understanding it are two different things for me. The only way I actually know if something holds up is by building it.

So I got curious whether ODC could consume OKF natively. No external parsing service, no separate graph database, nothing bolted on from outside the platform.

I built OKFParsingLibrary, OKF Store, and OKF Agent to find out.

I couldn’t find another OKF implementation built for a low-code platform, or another reasoning agent that consumes OKF data and answers questions over it. Google’s own consumer-side reference implementation is a visualizer, not an agent.

What OKF actually is

OKF stands for Open Knowledge Format. Google’s Data Cloud team designed it as a vendor-neutral way to structure knowledge for AI consumption: concepts, the links between them, and metadata about how much you should trust each one.

The full spec is on GitHub if you want to read it yourself.

Version 0.2 shipped on July 25, 2026 and added the fields that matter most for this build: verification status, trust tier, staleness. That is what turns a folder of markdown files into something an agent can reason about honestly, not just retrieve text from.

Scope declaration

Let me be upfront about what this actually is before we get into how it works.

I built the consumer side of OKF v0.2. All three components are consumer-side only.

Together they ingest an OKF bundle, store it as entities, derive trust tier and status correctly, resolve links between concepts, and let an agent query all of it through a proper tool surface.

I did not build the producer side. There is no export action. OKF Store cannot take what it has ingested and hand back a bundle another OKF tool could read.

BundleAuxFile and ExtraFrontmatter already store what an export action would need. The action itself doesn't exist yet.

So here’s the honest claim: OKF v0.2 consumer support in ODC, verified against all four of Google’s own canonical reference bundles, with derived fields checked by hand against Google’s own reference tooling.

Here’s the claim I won’t make: full OKF support, interoperable, round-trip, bidirectional, or anything implying an ODC-authored bundle can leave ODC and be read somewhere else.

The architecture

Three layers, each doing one job.

OkfParsingLibrary is C# External Logic. It parses frontmatter and serialises it back. Nothing else.

It doesn’t know what a Bundle is, doesn’t touch entities, doesn’t know ODC exists. Stateless, never throws, lossless on unrecognised keys.

That last part matters. OKF’s own spec says a consumer must preserve keys it doesn’t understand, not silently drop them.

OKF Store is where the actual platform work happens. Ingestion, entity storage, link resolution across concepts, and read actions that derive trust tier and status at query time rather than caching them.

Trust tier isn’t stored as a value on the concept. It’s computed live via Advanced SQL against a TrustTierOption entity, so the ranking logic lives in one place instead of being duplicated every time a concept gets written.

OKF Agent sits on top as the tool surface. It doesn’t talk to entities directly. It calls OKF Store’s read actions and exposes them as tools an agent can reason with.

The system prompt that governs how the agent behaves isn’t hardcoded into the app. It lives as entity data in a separate app, Prompt Assembler, keyed okf_agent_system, editable through Back Office without touching Studio.

That last decision is the one I’d point to first if someone asked why this isn’t just three ODC apps stacked on top of each other.

The parsing library doesn’t know about ODC. OKF Store doesn’t know how the agent will use it. The agent doesn’t know how the prompt that shapes it gets written. Each layer can change without the others noticing.

Testing against the real thing, not a toy bundle

It would have been easy to write a small test bundle myself, prove the happy path, and call it done. I didn’t want that kind of proof.

I wanted to know if this holds up against knowledge nobody built for me to succeed against.

So I pulled all four of Google’s own canonical reference bundles straight from the knowledge-catalog repo: acme_retail, crypto_bitcoin, ga4, stackoverflow. 53 concepts total.

Not written to exercise my code. Written to exercise the spec.

All 53 ingested. Zero parse errors. 110 concept links, 100% resolved, including 7 that traverse up a directory with ../.

The largest single concept body, a GA4 events table with a 60-row schema and 3 SQL blocks, came through at 9,394 characters, intact.

45 concepts correctly derive Unverified because no verified key is present. 8 derive HumanReviewed, all inside acme_retail. 43 with no status key correctly fall back to stable per spec §5.4. Two concepts are correctly flagged deprecated.

Checking my output against Google’s own tooling

For an independent check, I went to the one thing every reference bundle ships alongside itself: viz.html, generated by Google's own reference agent. Every derivation rule in OKF is fixed and public, so the correct answer for any concept is computable from its own frontmatter. That makes this a real comparison, not two opinions lining up.

It’s not just a graph picture. Click any node and it renders that concept’s full metadata and body, straight from a JSON blob embedded in the page. No backend, no install.

Take tables/orders from acme_retail. Click it in viz.html and you get type, status, trust tier, staleness, description, tags, generation and verification provenance, sources, and the rendered body, all in one panel.

Call GetConceptByPath for the same path in OKF Store and you get the same fields back as data.

Line them up and check. Type matches. stable matches. HumanReviewed matches. The staleness date matches. Every source matches.

The body matches too, with one difference worth naming: mine is stored as raw markdown, and viz.html renders that same source through a markdown library into an HTML table.

That’s the check I ran on tables/orders, and it's repeatable on any other concept the same way. Google already ships the ground truth. You just have to open it.

One incidental find along the way. viz.html also renders log.md as a graph node, with a synthesised type of "Log" and even a trust tier, despite log.md having no frontmatter at all and being a reserved, non-concept file under the spec.

OKF Store excludes it correctly. That appears to be a visualizer-specific convenience rather than behavior a conforming storage layer should copy.

Using OKF data for agent reasoning

Storing the data correctly is one thing. Watching an agent reason over it is the more interesting part.

OKF Agent exposes six tools, each a thin wrapper over an OKF Store read action: ListBundles, GetBundleIndex, GetConceptByPath, ListConcepts, GetOutboundLinksByPath, GetInboundLinksByPath.

The agent doesn’t touch entities directly. It calls these.

To test this, I needed real questions with real, pre-known answers. So I wrote a set of test questions against the acme_retail bundle, each with an expected concept path worked out ahead of time.

Then I ran them against the wired-up agent and checked the trace, not just the visible answer. Did it call the right tools? Did it resolve the correct concept?

Tracing the agent’s logic

Question: “What are the actual BigQuery tables used to compute each of the four COGS components in Acme’s gross margin calculation?”

The trace shows the agent working for the answer rather than guessing at it. It resolved the bundle first, ListBundles returning Acme Retail at BundleId 211.

Then it tried four directory paths that came back empty: metrics/cogs/, metrics/gross-margin, metrics/, concepts/. Reasonable guesses that didn't land.

It fell back to the bundle’s full root index. From there it called GetConceptByPath exactly once, on computations/gross-margin-period, the correct Attested Computation concept, and pulled its full body: the sanctioned SQL, its trust tier, its sources.

The answer named all five tables correctly, matching the actual join graph in the concept’s SQL rather than a plausible-sounding guess. It also stated the concept was HumanReviewed and stable, both accurate.

Three of the six tools were enough for this question.

OKF ships with trust tiers, staleness, and provenance for a reason: so an agent can reason about how much to trust what it retrieves, not just retrieve it. The next two sections are about what it took to make that hold reliably.

The numeric-identifier finding

One agent tool parameter caused a full day of failed testing before I understood why.

ListConcepts takes a minimum trust tier filter. I'd built it as an integer identifier, MinTrusTierId, matching the tier's row in a lookup entity.

The agent kept defaulting to the least restrictive value, even when the question named the tier it wanted outright. Ask for “HumanReviewed” concepts, get back everything.

I tried five separate fixes across that day. Rewrote the tool description. Rewrote the parameter’s own description. Strengthened the system prompt instruction. Removed a default value that might have been masking the field entirely.

Every one failed the same way, confirmed by trace data showing the same wrong value passed call after call.

The fix wasn’t a better prompt. It was changing the parameter from an integer to a text label matching the tier’s own name: "Unverified", "MachineConfirmed", "HumanReviewed".

The model could then copy the word straight out of the question instead of translating it into an arbitrary number first. That worked on the first attempt, confirmed correct on two different values via trace data.

The lesson has nothing to do with OKF and nothing to do with ODC. Numeric identifiers look like the wrong interface for LLM-facing tool parameters whenever a human-readable label already exists and means the same thing.

This showed up as a design risk here, and there’s no reason to think ODC’s tool-calling is special. I’d expect the same failure mode on any platform where a model has to translate a semantic value into an arbitrary identifier instead of copying it.

Trust-tier honesty

One behavior needed more precision than a single instruction could hold on the first try.

The agent must state a concept’s actual trust tier honestly, even when the question restricts the answer to a higher tier and the only concept that answers it sits below that bar. It cannot quietly relabel the concept to fit what was asked.

Getting this right took several rounds of tightening the instruction, not one attempt.

That iteration is what made Prompt Assembler worth having as its own app rather than a convenience.

A prompt this sensitive to exact wording needs to be editable and re-publishable on its own, without a full Studio publish cycle for every adjustment. Back Office access to a single entity record turned out to matter more than I expected going in.

Honest limitations

A few things worth stating plainly rather than discovering later.

Tags are stored as comma-separated text, so tags containing commas are not represented losslessly. No bundle in the OKF reference corpus hits this case.

Re-ingesting a bundle with overwrite enabled deletes and recreates it rather than updating it in place. The BundleId changes every time. Anything holding a stored BundleId across a re-ingestion needs to re-resolve it.

The Attested Computation type preserves its four required keys, computation, parameters, executor, attester, exactly as the spec requires for keys a consumer doesn’t act on. Nothing in this release executes a computation, calls an executor, or checks an attestation.

The spec doesn’t require that either. Execution and attestation are explicitly informative, not normative, under §10.5.

No export action, consistent with the consumer-only scope above.

MachineConfirmed is implemented and reachable through the same logic as the other two tiers, but every verification record across all four canonical bundles is human-attributed. No concept in the corpus exercises it.

The logic is correct by inspection, and it shares the same code path as the two tiers that are exercised. But a real machine-attributed record has never run through it end to end.

What this proves

So, back to the question I started with. Could ODC read OKF natively, no external parsing service, no separate graph database, nothing bolted on from outside the platform?

For OKF v0.2, yes. ODC can ingest the format, preserve its trust and provenance model, resolve its knowledge graph, and expose it to an agent without handing the core architecture to an external service.

Practically, that means an OKF bundle doesn’t have to be flattened into ordinary document chunks before it’s usable in ODC. Its links, provenance, trust tiers, verification status, and staleness stay available as structured data, to application logic and to agent tools, not just to a person reading markdown.

What it doesn’t do matters just as much. No export. No execution of attested computations. MachineConfirmed sitting untested until real machine-attributed data exists to exercise it.

I’d rather hand you that list than let you find it yourself.

The spec is young and moving. It will change, and what I built will need to change with it.

But the underlying question wasn’t really about OKF. It was whether a low-code platform can hold its own against a trust-and-provenance-aware knowledge format without reaching outside itself for help.

OkfParsingLibrary, OKF Store, and OKF Agent is now available in the Forge. If you try them against a bundle I haven’t tested, or you build a consumer of your own, I’d like to hear what you find.

Can ODC Consume Google’s Open Knowledge Format Natively? was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Semantic search encodes meaning, not surface form. That is exactly what makes it useful for paraphrased or conceptual queries. It is also exactly what makes it useless when the query is ERR-BM25-003. That string is not a concept. It is an identifier. The embedding model has no basis to place it near anything in particular, and so it returns whatever happens to be thematically adjacent. Which is not the same as correct.

I kept hitting this during testing. Conceptual queries worked. But the moment a query contained an exact error code, a field name, or a version string, results went sideways. So I went looking for why, and that led me to BM25. This article is what I found building it natively inside ODC.

Where This Fits

This is the sixth article in a series on building a native RAG pipeline inside ODC without external vector database dependencies. Every external search service is another integration to secure, another bill to justify, and another dependency outside the platform’s lifecycle. Keeping retrieval inside ODC is an architecture decision, not just an exercise constraint.

The short version of the series so far: documents are chunked across five strategies, embedded via OpenAI text-embedding-3-small, and stored natively in ODC. A semantic image search layer followed using Cohere Embed v4, then a query result cache to reduce embedding costs. Semantic search runs cosine similarity against those embeddings. This article adds a parallel BM25 index to the same corpus and a hybrid retrieval path that fuses both signals with Reciprocal Rank Fusion.

Two related pieces of work deserve a mention. Owen Corstens has published RAG Tools on the Forge, offering BM25 and vector search as benchmarked retrieval strategies inside OutSystems. His original benchmark experiment and the subsequent benchmark tool article are worth reading. And OutSystems has released ODC Semantic Search in Beta, which brings vector-based retrieval over entity data natively into the platform. What I built here sits alongside both.

This is not a replacement for the semantic search layer. It is an addition to it. Semantic search handles meaning. BM25 handles exact strings. The rest of this article is how both run against the same corpus and contribute to one result.

The Stack

Two layers. The BM25 entities are structurally dependent on DocumentChunk, so everything that touches the database stays in the same app.

BM25Engine is the C# External Logic library. It is stateless, has no NuGet dependencies, and exposes three Server Actions to ODC: TokenizeText, IndexChunk, and ScoreQuery. The Porter stemmer is self-written. No shared mutable state between calls. This is the only piece that could be lifted out and reused in a different ODC app without modification.

RAG Knowledge Base is the existing ODC App. I added three new entities: BM25Term, BM25Posting, and BM25Stats. I added TokenCount as a new attribute on DocumentChunk. One new workflow, IndexChunks, runs after embedding. A timer, RecomputeBM25Stats, recomputes corpus-level statistics on a schedule. The hybrid search entry point is HybridSearchWithBM25, which returns the same SearchResult structure as SearchVectorWithCache.

The screen did not change. That was a constraint I set before writing a line of code. Any retrieval upgrade that requires UI changes has a blast radius into every team that has built on top of the app. Keeping the output structure identical means the toggle between semantic and hybrid search is a single decision point, not a migration.

How BM25 Works in This Implementation

BM25 scores a chunk against a query using three factors: how often the query terms appear in the chunk, how rare those terms are across the corpus, and how the chunk’s length compares to the average.

Two terms worth defining. The corpus is the entire collection of indexed chunks across all documents in the database. Not one document in isolation, but everything. In testing, the corpus was 22 chunks total across two documents. Term frequency is how many times a query term appears within one specific chunk.

The formula rewards specificity. A term that appears in only one chunk out of twenty is rare across the corpus. Finding it is informative. It points you somewhere specific. A term that appears in fifteen chunks out of twenty tells you almost nothing about which chunk is most relevant. Rare terms score higher. Common terms score lower. This is why ERR-BM25-003 is a strong BM25 signal while a word like "document" is a weak one.

One implementation detail matters more than it looks. The classic Inverse Document Frequency (IDF) formula goes negative when a term appears in more than half the corpus, which means common terms start subtracting from relevance scores. On a small technical corpus, that threshold arrives fast. My test corpus had 22 chunks. Any term appearing in 12 of them would have scored negative under the classic formula, and in a document about BM25, the term “chunk” is in nearly every one. I used the Lucene variant, which is always positive:

IDF(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)

A term that appears everywhere contributes less. It never subtracts.

The Tokenizer

The tokenizer pipeline is: lowercase, replace non-alphanumeric characters with a space, split on whitespace, drop single-character pure numerics, remove stop words, Porter stem.

The replace-with-space step is load-bearing. A hyphen is not a letter or digit, so ERR-BM25-003 becomes three tokens: err, bm25, 003. Each passes through the stop-word filter and Porter stemmer independently. The post-fix test run returned 10 BM25 matches for that query against a 22-chunk corpus. The token split is what made those matches possible.

The current tokenizer version is v2-stemmed-porter, stored as a Site Property. If the tokenizer ever changes, the Site Property changes with it, and a full corpus reindex via IndexChunks is required before search results are valid again. Nothing detects the mismatch automatically. The Site Property is documentation for the operator, not an enforcement mechanism. If you deploy a new tokenizer revision and skip the reindex, query stems stop matching indexed stems and recall degrades silently.

The Three Entities

BM25Term stores each unique stemmed term and its DocumentFrequency, how many distinct chunks contain it. One row per term across the entire corpus.

BM25Posting stores the term-chunk relationship: BM25TermId, DocumentChunkId, and TermFrequency. One row per term per chunk. This is the inverted index.

BM25Stats is a singleton. Exactly one row. It holds TotalChunks and TotalTokenCount, which are used to compute avgChunkLength at query time. The RecomputeBM25Stats timer is the only thing that writes it.

DocumentFrequency by Timer, Not by Ingestion

DocumentFrequency on BM25Term is not incremented at ingestion time. It is recomputed on a timer via COUNT(DISTINCT DocumentChunkId) GROUP BY BM25TermId across all BM25Posting rows.

The alternative, incrementing at ingestion, looks simpler but accumulates drift. Concurrent ingestion workflows, retry double-counting, and chunk deletions all produce incorrect counts that compound silently. The timer approach recomputes from source truth on every run. The tradeoff is a staleness window between the last timer tick and the current moment. That window is documented, bounded, and acceptable for a corpus that does not change by the second.

The Ingestion Path

BM25 indexing runs after embedding. The existing EmbedFile workflow already chunked the document and written DocumentChunk rows. IndexChunks picks up from there.

For each chunk, ODC calls BM25Engine.IndexChunk with the chunk text. It returns two things: a TokenCount and a list of term-frequency pairs. ODC writes TokenCount back to DocumentChunk, then iterates the term-frequency list. For each term, it does a find-or-create on BM25Term, then writes a BM25Posting row with the BM25TermId, DocumentChunkId, and TermFrequency.

That is the entire ingestion path. No corpus-level statistics are computed here. DocumentFrequency and BM25Stats are handled separately by the timer.

RecomputeBM25Stats

After indexing, BM25Stats needs to exist before any search can run. RecomputeBM25Stats computes TotalChunks and TotalTokenCount from the current state of BM25Posting and writes them into the singleton row. It uses a find-or-create pattern, so it is safe to trigger multiple times.

Trigger it manually after the first index run. After that, the timer keeps it current.

One thing worth stating directly: if BM25Stats has no row, ScoreQuery divides by zero. I found this out during testing. The error is immediate and unambiguous, but the fix requires knowing that RecomputeBM25Stats is the only thing that creates the row.

The Query Path

Query -> Vector Leg + BM25 Leg -> RRF Fusion -> SearchResult

HybridSearchWithBM25 is a single Server Action that runs three logical phases in sequence: vector leg, BM25 leg, then RRF fusion.

Vector leg

The query text is embedded via SearchVectorWithCache, which returns a cached vector if the query has been seen before or calls the OpenAI embedding endpoint if not. ODC compares the vector against ChunkEmbedding rows using cosine similarity. The top results come back as a ranked list with scores.

BM25 leg

The query text is passed to BM25Engine.TokenizeText, which returns the stemmed query terms. Those terms are looked up in BM25Term to retrieve their DocumentFrequency values. ODC then calls BM25Engine.ScoreQuery, passing the term list, their frequencies, and the BM25Stats singleton values. ScoreQuery returns a PostingCandidate list, chunk IDs with raw BM25 scores, which ODC uses to fetch the matching DocumentChunk rows via an Advanced SQL Node.

The SQL Node is where the BM25 leg is tightly coupled to the app. It constructs a dynamic IN clause from the candidate chunk IDs and retrieves the chunk text alongside the scores. This is why the scoring logic cannot be cleanly extracted into a Library. The query runs against entities that only exist inside RAG Knowledge Base.

RRF fusion

RRF does not use raw scores. It uses ranks. Each chunk gets a contribution from each leg it appears in:

RRF(chunk) = Σ 1 / (k + rank)

where k = 60 and rank is the chunk's position in that leg's result list. A chunk that appears in both legs gets two contributions. A chunk that appears in only one leg gets one. The fusion is purely additive. No score normalisation required, no calibration between BM25 and cosine similarity scales.

The output is sorted by RRF score descending. The top results are returned as SearchResult records, identical in structure to what SearchVectorWithCache returns.

What Testing Actually Showed

I tested against two documents: an Event-Driven Architecture exercise covering loosely-coupled design patterns, and a purpose-built ODC RAG Pipeline Technical Reference containing exact error codes, field names, component names, and version strings. The EDA document is conceptual. The technical reference is precise. The test queries were strings taken directly from the technical reference: error codes, field names, and component identifiers that a developer would realistically search for. The kind of precise terms that appear in support tickets, error logs, and API documentation.

The five test queries were:

1. ERR-BM25-003: an exact error code
2. ERR-VEC-002 ChunkHash: an error code paired with a field name
3. TokenizerVersion v1-stemmed-porter: a Site Property name and its version string value
4. PostingCandidate DocumentFrequency ChunkTokenCount: three exact field names from the ScoreQuery signature
5. BackfillBM25Index null TokenCount: a Server Action name paired with field names

Each was run against both semantic search and hybrid search. The results below cover the cases where the two approaches diverged.

Where BM25 made the difference

The clearest case was ERR-BM25-003. Semantic search returned five results. The top result was the BM25 Library wrapper description (Score 0.463). The second was the BM25Engine Component Reference (Score 0.407). The Error Reference intro chunk appeared third (Score 0.387). It mentions the error code in a header but does not explain it. The definition chunk, the one that tells you what ERR-BM25-003 means and how to fix it, was not in the results at all. The embedding model placed the identifier near BM25-related content. Thematically adjacent. Not the answer.

The BM25 leg matched 10 chunks containing those tokens. The top two were the Error Reference intro chunk and the ERR-BM25–003 definition chunk. RRF fusion brought both into the final result. The definition chunk landed at Rank 2.

ERR-VEC-002 ChunkHash produced the same pattern. The BM25 leg's top hit was the ERR-VEC-002 definition chunk, its highest single-chunk score in the test run. The vector leg also found the same chunk, at Rank 2. Both legs agreed. The hybrid result confirmed it.

PostingCandidate DocumentFrequency ChunkTokenCount was the field-name case. Three exact field names, all appearing together in the ScoreQuery signature chunk. BM25 scored it highest in the query. The vector leg's top result was the Operational Procedures chunk. Related content, but not the chunk containing those field names. BM25 and vector retrieved different chunks here, each finding something the other missed.

Where both legs agreed

BackfillBM25Index null TokenCount was the strongest dual-leg case. The vector leg ranked the Backfill Procedure chunk first. The BM25 leg also ranked the same chunk first. Both retrievers independently pointed at the same answer. The RRF score for that chunk was the highest in its result set. This is what the fusion is designed to reward.

Where hybrid added nothing

Not every query showed a difference. Conceptual queries against the EDA document produced results where both legs retrieved the same top chunks. Hybrid did not hurt those queries. It also did not materially change them. The EDA content has no exact identifiers for BM25 to anchor on. On those queries, the vector leg was doing all the meaningful work, and RRF simply confirmed it.

That is the expected behaviour. Hybrid retrieval does not improve every query. It closes the gaps that exist when a query contains terms the embedding model cannot meaningfully place.

What I Have Not Proved

The BM25 leg has a posting fan-out. At query time, it fetches every BM25Posting row matching the query terms and passes them to ScoreQuery. A common term appearing in a large fraction of chunks produces a fan-out that will eventually become the bottleneck. I have not benchmarked where that limit bites. Those numbers need to come from real load testing against your corpus size, not from this article.

DocumentFrequency has a staleness window. It is recomputed by a timer, not updated at ingestion time. Between the last timer tick and the current moment, newly ingested chunks contribute to BM25Posting but not yet to DocumentFrequency. IDF scores during that window are computed against slightly stale corpus statistics. For most ingestion patterns this is acceptable. For high-volume concurrent ingestion it is worth knowing.

No latency numbers appear in this article. I did not instrument query time in a way that would produce meaningful comparative data. Asserting speed claims without measurement is not something I will do.

The two retrievers must see the same chunk set. If the vector index and the BM25 index drift apart, chunks embedded but not indexed, or indexed but not embedded, RRF fusion produces results that neither leg can fully explain. Keeping IndexChunks in the same workflow as EmbedFile is the operational guard against this.

Two Blind Spots, One Pipeline

Semantic search fails on exact identifiers. Not because the implementation is wrong. Because encoding meaning is what it does, and ERR-BM25-003 is not a concept. It is a string.

BM25 fails on paraphrase. “Loosely coupled architecture” does not match “decoupled event-driven design” unless those exact stems overlap. The vector leg handles that without effort.

Neither retriever covers the full query space alone.

RRF does not pick a winner between them. It adds the rank contributions together. A chunk both legs agree on scores higher than a chunk only one leg found. A chunk only the BM25 leg found still surfaces. It just does not get the dual-leg boost. The fusion is additive by design.

What changed: exact-term queries now return the right chunk instead of whatever is thematically adjacent. Conceptual queries work the same way they always did. The screen is unchanged. The result structure is unchanged. The retrieval coverage is not.

What Comes Next

BM25 closes the exact-term gap. It does not solve ranking. The right chunk is not always first.

That is the reranking problem. A cross-encoder reranker reads the query and each retrieved chunk together and scores relevance directly. It is slower than either retrieval leg, which is why it runs after fusion on a small candidate set rather than across the full corpus. That is the next layer.

After reranking: retrieve broadly with two signals, fuse by rank, rerank the shortlist. That is the architecture I am building toward.

Here is the question that matters: do your users search by error code? By ticket number, SKU, API endpoint, field name? If they do, your semantic search is returning thematically adjacent content instead of answers, and nobody has noticed yet because the results look plausible.

Test it. Take your most specific identifier and put it through your search. Then find me in the OutSystems Community or drop a comment here. I would like to hear what came back.

If you want to add BM25 and hybrid retrieval to your own ODC pipeline, BM25Engine is available on the Forge. If you want to try the full working pipeline first, the updated RAG Knowledge Base app is also on the Forge with hybrid search enabled out of the box.

Your ODC Semantic Search Cannot Find ERR-BM25–003. Here Is the Fix. was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Someone asked me a good question after my talk in Amsterdam: does the semantic search have any caching built in?

It didn’t. Every search hit the embedding API. The same query a hundred times a day meant a hundred API calls.

That one stayed with me on the flight home. The RAG Knowledge Base Forge component proved native vector search works in ODC. Proving it works and being worth running in production are two different things, and the question was pointing straight at the gap.

So I added a cache. Here is how it works and why each part is built the way it is.

What We Cache

There are two things worth caching in a semantic search pipeline: the query vector and the search result. The query vector is the embedding generated from the query text. The search result is the ranked list of matching documents.

SearchVectorWithCache caches both, controlled by a single parameter: IsCacheResult (Boolean, default False).

The query vector is always cached. On a repeat query, the action skips the embedding API call entirely and uses the stored vector to run cosine similarity against the current document set. Fresh results, no API cost.

When IsCacheResult is True, the search result is also cached. On a repeat query with a result hit, the action skips both the embedding call and the vector scan. It deserialises the cached result and returns it directly.

The developer controls which mode to use per call. IsCacheResult = False for fresh results without the embedding cost. IsCacheResult = True for maximum speed when result freshness is not critical. SearchVector directly when you want no caching at all.

The Cache Entity

QueryEmbeddingCache has nine attributes.

Id (Identifier, AutoNumber). Primary key.

QueryHash (Text 64). SHA-256 hash of the query text. This is the lookup key. Hashing keeps the index tight and avoids comparing long query strings on every read.

QueryText (Text 2000). The original query, stored for audit. When you want to see what is actually being cached, it’s here.

CachedQueryVector (Text 50000). The query embedding as a serialised JSON float array. Always stored on every run, regardless of IsCacheResult.

CachedResult (Text 100000). The full serialised search result. Only stored when IsCacheResult is True. Blank otherwise.

EmbeddingModel (Text 100). The model used when the entry was created. Swap models later and this field drives invalidation. You do not want results from text-embedding-3-small quietly mixing with results from something else.

CacheMaxAgeMinutes (Integer). The per-record expiry age. Not a global setting. Each entry knows its own TTL.

CreatedOn (DateTime). When the entry was written. Used by the purge timer.

HitCount (Integer, default 0). Incremented on every hit. Handy for spotting which queries are worth optimising elsewhere.

The design follows the same pattern as DocumentChunk in the original component. Named, typed, one clear job per attribute. The separation between CachedQueryVector and CachedResult is intentional. The vector is always worth keeping. The result is optional.

SearchVectorWithCache

SearchVectorWithCache wraps the search with a cache lookup. Same inputs as a plain search, plus two: CacheMaxAgeMinutes (Integer) and IsCacheResult (Boolean, default False).

When a query comes in, the action lowercases the query text, hashes it, and checks QueryEmbeddingCache for a matching record with the same hash and embedding model. Three paths from here.

If a match is found and CachedResult has data, the cached result is deserialised and returned. No embedding call. No vector scan.

If a match is found but CachedResult is blank, the stored query vector is used to run cosine similarity against the current document set. No embedding call. Fresh results.

If no match is found, the action embeds the query, runs the search, and writes the record. The query vector is always stored. The result is stored only when IsCacheResult is True.

Expired records are not checked on read. The purge timer handles deletion. Until a record is deleted, it remains a valid cache hit.

CacheMaxAgeMinutes can be set per call. Pass a value above zero and that value wins. Pass zero and the action falls back to the DefaultCacheMaxAgeMinutes site property, which defaults to 1440 minutes (24 hours). Either way, the resolved value is stored on the record itself, so each entry manages its own expiry. Changing the site property later has no effect on records already written.

On every path, HitCount is incremented when a record is found. The record is always upserted. Never duplicated.

HashText in C#

ODC has no built-in SHA-256 function.

So I added HashText to SemanticEngineV2Service in C# External Logic. Plain text in, SHA-256 hex string out. It follows the existing layer pattern: ISemanticEngineV2 exposes the contract, SemanticEngineV2Facade delegates, SemanticEngineV2Service implements.

This is the same path ChunkHash already takes in the DocumentChunk entity. The component had a hashing pattern. HashText extends it.

I could have stitched something together in ODC with text manipulation and HMAC logic. I didn’t. The External Logic layer was cleaner and easier to test.

Three Cache Management Actions

There are three: ClearCache, ClearCacheByQuery, and ClearCacheByHash.

ClearCache deletes everything in QueryEmbeddingCache. An admin action. You reach for it to force a full rebuild after a big ingestion batch or a model change.

ClearCacheByQuery takes the query text, hashes it internally, and deletes the matching record. For developers who know which query went stale. You pass the string the way a user would type it.

ClearCacheByHash takes a pre-computed QueryHash and deletes the record directly. For programmatic callers who already have the hash and don’t need to compute it twice.

The Purge Timer

CleanupQueryEmbeddingCache timer deletes records that have outlived their expiry.

Each record’s own CacheMaxAgeMinutes drives this, not a global setting. A 10-minute record expires in 10 minutes. A 1440-minute record expires in 24 hours. They live in the same table without stepping on each other.

Here is something I found while setting this up. The ODC console timer UI has a minimum schedule interval of 5 minutes. If you want the purge to run more often, the normal workflow won’t take you there.

For production, I’d treat that 5-minute floor as the safe default. In my test environment, I asked Mentor to schedule the timer as frequently as possible. It set ScheduleConfiguration = "* * * * *" directly, and the timer ran every minute. You can see it active in the console with the * * * * * schedule, last run and next run one minute apart.

Useful for validation. Beyond that, I’d reach for it carefully and on purpose.

When to Use Cache, When to Go Direct

Use SearchVectorWithCache with IsCacheResult = False for most queries. You skip the embedding call on repeats and always get fresh results. This is the safe default.

Use SearchVectorWithCache with IsCacheResult = True when the knowledge base is stable and speed matters more than freshness. Support FAQs, internal search tools, anything where the documents don't change between requests and the same questions repeat.

Use SearchVector directly when you want no caching at all. Right after ingesting new documents. In a verification flow. In admin tooling where accuracy is the whole point.

ClearCacheByQuery is the bridge. When a document behind a known query changes, invalidate that one entry instead of nuking the whole table.

The Tradeoffs You Should Know

With IsCacheResult = True, the cache trades freshness for speed. New documents won't appear in cached results until the entry expires or gets cleared. If your knowledge base changes constantly, use IsCacheResult = False instead. You still skip the embedding call, and results are always current.

The cache key is a hash of the query text, lowercased first so case differences still match. Punctuation and wording don’t. “Reset password steps” misses “How do I reset my password?” even though they mean the same thing. Semantic caching is a known pattern in RAG pipelines: match incoming queries against cached ones by vector similarity rather than exact hash, and return the cached result if they’re close enough. Bringing that natively into ODC without an external vector store is the next step. Not in this version.

CachedResult is capped at 100000 characters. A normal topK fits easily. If a result set overflows that limit, the search still runs and returns normally, but the result isn’t stored. You get a message saying so. The query vector is still cached, so the next repeat skips the embedding call regardless.

HitCount is a counter, not an analytics suite. It tells you how often a query hit the cache. It does not tell you whether the answer was any good.

The timer runs every minute on the schedule Mentor set, so records expire close to their TTL. Close, not exact. A 10-minute record gets purged on the next cycle, not on the dot.

QueryEmbeddingCache holds two large Text fields: CachedQueryVector at 50000 characters and CachedResult at 100000 characters. You could move them to a child entity and fetch them only on a confirmed hit. Aurora PostgreSQL stores large text out-of-line via TOAST. Keep the lookup Aggregate scoped to QueryHash and EmbeddingModel, and the large fields stay out of the way. At this scale, one entity is simpler and the difference is not measurable.

QueryEmbeddingCache grows with every unique query. In a high-volume deployment with a wide query spread, the table gets large. That’s what ClearCache is for.

Why It’s Worth Adding

Native vector search in ODC works. This layer makes it cheap enough to leave running.

If 100 users ask the same support question in a day, only the first one pays the embedding cost. The other 99 get their answer from a single row lookup. That is the difference between a demo and a system you’re happy to see on the bill.

Proving it works was step one. Making it worth keeping is step two.

If you’re building on the RAG Knowledge Base, the latest version on Forge already includes all of this.

Are you caching semantic search results in ODC? Or solving the freshness problem some other way? I’d like to compare notes in the comments.