Technology keeps surprising us and moving human progress forward.

One example is the brain-computer interface (BCI), which tries to decode what the mind is thinking. Think of it as voice-to-text, but for thoughts. You wouldn’t have to speak, you’d just think, and the system would turn that into text or commands. It could even cut down on noise, and people might share things without speaking. But how secure would that be?

The next step is changing what the brain does, and that is where optogenetics comes in.

Our brain has about 86 billion neurons and several hundred trillion connections (synapses) between them. Optogenetics tries to control a chosen few of those neurons using light. It works in three steps:

1. A gene for a light-sensitive protein called an opsin is delivered into specific neurons.
2. Light of a particular color is sent in through a thin optical fiber.
3. Depending on the protein used, the neurons are either activated or silenced.

Is it a remote control like the one for your TV or AC? Not quite. A TV remote has one button and one result. In the brain, you stimulate one set of neurons, but the final outcome depends on how they affect the cells connected to them.

Tamil cinema gave us a version of this idea in 7aum Arivu, where characters control other people’s minds with a gaze. Real science is far more careful. Researchers are exploring whether the same approach could one day help with conditions that involve faulty brain circuits, such as addiction.

Today, this research is mostly at the experimental stage in animals like mice. The challenge is the brain’s complexity. Because neurons are so tightly connected, changing one cell can affect many others, and we can’t always predict the result.

If it matures safely, it could help people with conditions like Parkinson’s and depression. It also raises a bigger question: if we can influence thoughts and behavior, who decides how, and when?

The scientists linked to this field include Karl Deisseroth, Peter Hegemann and Georg Nagel. Nagel and Hegemann discovered Channelrhodopsin-2, a light-sensitive protein that became a key tool in this research.

What do you think, is this hope or risk?

What Is Reasoning?

For people, reasoning means working through a problem step by step to reach a conclusion. You might add up prices, check whether a plan follows from your assumptions, or compare options before choosing one.

Language models can do something similar. Given a hard question, a model can produce intermediate steps before its final answer, and those steps often improve accuracy. The open question is where those steps should happen: in written words, or inside the model’s internal numbers.

An Overview of Text Reasoning

Text reasoning, usually called chain-of-thought (CoT), is the most common approach today. The model writes out its intermediate steps as ordinary text before giving an answer.

Take this question: A bakery sells muffins at $2 and cookies at $1.50. A customer buys 4 muffins and 6 cookies and pays with a $20 bill. What is the change? A text-reasoning model might write:

• Muffins: 4 × $2 = $8
• Cookies: 6 × $1.50 = $9
• Total: $17
• Change: $20 − $17 = $3

Strengths: Every step is visible, so people can check, debug, and trust the reasoning. It is well understood and widely deployed.

Weakness: Every step costs tokens, and long reasoning chains are slow and expensive.

An Overview of Latent Reasoning

Latent reasoning takes a different route. The model still works with vectors internally, as it always does, but between steps it does not turn its internal state into a word. Instead, the hidden vector from one step is fed directly into the next. The model reasons in continuous vector space and produces text only for the final answer.

With the same bakery question, the model’s internal state might carry “muffins total 8,” then “cookies total 9,” then “total 17,” then “change 3,” with none of these written out. The user sees only “$3.”

A widely cited method is COCONUT (Chain of Continuous Thought), from Meta researchers in 2024. It trains a model to use continuous thoughts instead of words, using a curriculum that gradually replaces written reasoning steps with hidden-state steps.

Further Reading

• White paper: A Survey on Latent Reasoning (Zhu et al., 2025)
• COCONUT: Search for the 2024 paper “Training Large Language Models to Reason in a Continuous Latent Space” by Hao et al.

Models That Support It

Latent reasoning is mostly a research area. Examples include:

• Huginn-3.5B, a depth-recurrent model with open weights on Hugging Face (tomg-group-umd/huginn-0125), described in Scaling up Test-Time Compute with Latent Reasoning
• COCONUT-style research models, which are prototypes rather than products
• Hybrid approaches such as Token Assorted, which mix latent and text tokens

I am not aware of a major commercial API that documents purely latent reasoning. Check current provider documentation before assuming otherwise.

The Industry Perspective

Industry interest in latent reasoning comes from three directions:

• Efficiency: Fewer generated tokens could lower serving costs and latency.
• Capability: Some researchers believe continuous states can carry richer intermediate information than single words.
• Concern: Hidden reasoning is harder to monitor. Safety and interpretability researchers have flagged this as a real trade-off.

The field is still early. Most published results use small or medium models and specific benchmarks, so treat broad claims about production benefits with caution.

Text Reasoning vs. Latent Reasoning: The Logical Flow

Consider the same question processed both ways.

Text reasoning:

1. The question is tokenized and embedded into vectors.
2. The model computes an internal state for the first step.
3. It projects that state onto the vocabulary and picks a word, “8.”
4. That word is embedded again and becomes the input for the next step.
5. The process repeats until the model writes “$3.”

Latent reasoning:

1. The question is tokenized and embedded into vectors, as before.
2. The model computes the same internal state for the first step.
3. It does not pick a word. The full vector is fed forward as the next input.
4. The model continues computing from that vector through several latent steps.
5. Only after the planned steps does it write “$3.”

The key difference is the discretization step. Text reasoning collapses each intermediate state into a discrete word, while latent reasoning keeps the continuous state. In both cases, the computation happens in vectors. Text reasoning adds a bottleneck at each step, and latent reasoning removes it.

It’s also worth contrasting latent reasoning with vector databases. A vector database retrieves the stored item nearest to a query, and its vectors don’t change. Latent reasoning computes a new state from the previous one at each step.

Pros and Cons of Latent Reasoning

Pros

• Fewer generated tokens: Intermediate steps aren’t written out, which can cut output-token costs and latency.
• Higher information per step: A continuous vector can carry more than one discrete word.
• Possible parallel exploration: A single vector may represent several candidate paths at once.
• Less need for hand-written reasoning data: Some methods, such as depth-recurrent models, need no bespoke reasoning traces.
• Adaptive compute: Some architectures can spend more computation on harder inputs.

Cons

• Low transparency: Hidden vectors are hard to read, which makes auditing and debugging difficult.
• Accuracy gap: Latent methods have often trailed explicit chain-of-thought, especially on math-heavy tasks.
• Unclear reasoning: Some analyses found weak evidence that the internal steps are truly step-by-step.
• Training complexity: Curricula and gradient handling are more involved than standard fine-tuning.
• Oversight risk: Reasoning that isn’t in text is harder to monitor for problematic plans.

Token Cost: Does Latent Reasoning Save Money?

Partly, and the savings are narrower than they first appear.

Where the savings come from: Text reasoning is billed per output token. If a problem needs 300 tokens of written reasoning and a latent approach needs only 30 tokens for the final answer, the output cost falls sharply.

Where the savings don’t apply:

• Input tokens are still billed. The question and any context cost the same.
• Compute isn’t free. Latent steps still run the model’s layers, and some architectures use as much or more compute per step.
• Pricing may change. If providers can’t meter latent steps as tokens, they may charge by compute time or another measure, and your savings depend on that choice.
• Accuracy affects cost. If a latent approach fails more often and needs retries, the cheaper tokens can be offset by extra calls.

A practical test: Compare cost per correct answer, not cost per request. Also compare against text reasoning with a shorter reasoning budget, which is often the simplest way to cut token spend today.

Bottom line: Latent reasoning can reduce billed reasoning tokens, but it doesn’t eliminate compute or input costs, and it is still experimental. Measure on your own tasks before relying on it.

References

• Zhu et al., A Survey on Latent Reasoning, 2025
• Geiping et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, 2025
• Hao et al., Training Large Language Models to Reason in a Continuous Latent Space (COCONUT), 2024

Design the questions before you build the graph, the same way you design tables before you load rows.

Why knowledge graphs matter

Most of an organisation’s important questions are about connections, not single records. Which customers are linked to the same fraud ring? Which drugs interact with a patient’s current prescriptions? Which projects stall if one supplier or one engineer disappears? A knowledge graph stores those connections directly, so they can be followed, counted and explained.

• It joins data that lives in separate systems. A CRM, an ERP, a document store and a ticketing tool each hold part of the picture. A graph links their entities into one model, so a question can cross all of them in a single query.
• It makes relationships fast to follow. Questions that need four or five hops, such as “suppliers of suppliers” or “friends of friends”, stay quick in a graph where they would need many expensive joins in tables.
• It carries meaning, not just data. Typed nodes and edges say what a thing is and how it relates to others. That shared vocabulary lets people, applications and AI systems read the data the same way.
• It grounds AI in facts. Retrieval over a knowledge graph gives a language model precise, connected facts with a traceable path, which reduces hallucination and makes answers explainable.
• It answers “why”, not only “what”. A path through the graph is an explanation: this product is at risk because this component comes from this factory in this restricted region.

That power is exactly why design matters. A graph is only as useful as the connections it was built to hold.

Why knowledge graphs make data AI-ready

A knowledge graph makes data AI-ready by doing the relating before the language model sees anything. The data arrives pre-related, so the model receives a few connected facts instead of a pile of loosely matching text.

• The data is pre-related. Entities are resolved and linked when the graph is loaded. The model never has to work out that “Acme” in an invoice and “Acme Steel Ltd” in a contract are the same supplier.
• Relationships are found before the LLM is called. A multi-hop question is answered by a graph traversal, which is exact and repeatable. In the bike example, the graph finds the path from Acme Steel to RideShop, and the model only has to explain it.
• The context window stays under control. Only the small subgraph the question touches goes into the prompt, not dozens of document chunks. That means fewer tokens, less noise and lower cost.
• Hallucination goes down. The model answers from explicit facts, and every claim can be traced back to a path in the graph and checked.

One caution keeps this honest. A graph reduces hallucination; it does not remove it, and it only helps when the graph holds the facts the question needs. That is the thesis again: a graph is AI-ready only for the questions it was designed to answer.

A simple supply chain example

Imagine a small bike company. It buys parts from suppliers, builds bikes in its plants, and ships them to customers. A knowledge graph of that business has six kinds of things (nodes) and five kinds of links (edges).

Now ask one simple question: if Acme Steel stops supplying, which customers are affected?. In the graph you just follow the links: Acme Steel supplies the Frame, the Frame is used in the City Bike, the City Bike is in Shipment #4512, and that shipment goes to RideShop. The answer is RideShop, and the path itself explains why.

Notice the order. We did not collect every document about bikes and hope this answer appeared. We started from the question, and the question told us exactly which nodes and edges we needed: Supplier, Part, Product, Shipment and Customer, linked by SUPPLIES, USED_IN, CONTAINS and SHIPS_TO. Plant and MADE_AT are there because a second question needs them: “which customers are affected if the Chennai plant shuts down?” If no question needed a node or edge, it would not be in the graph.

Build the questions first

A knowledge graph is not something you build first and make sense of later. It is closer to designing a relational database: you decide the tables and columns before you load a single row, and you decide them by asking what you need to read back out.

Many knowledge graph projects start the other way round. A team extracts entities from documents, links everything that co-occurs, loads millions of nodes and edges, and then opens a graph explorer hoping the structure will reveal insight. It rarely does. What appears is a dense hairball, a few super-connected nodes, and a long list of questions the graph cannot answer cleanly.

The fix is a change in order. Choose what you will read before you fill the graph. Write down the questions, turn them into queries, and let those queries define the node types, edge types and properties. Only then start loading data.

Both paths have four steps. The difference is where the questions sit: at the end, as a hope, or at the start, as the design brief.

The lesson relational databases already taught us

Nobody designs an RDBMS by creating one giant table, inserting every fact they can find, and working out the columns afterwards. Good database design starts from the access patterns: which screens, reports and APIs will read this data, and what joins and filters they need.

Those access patterns decide almost everything. They decide which things deserve their own table, which attributes become columns, which relationships become foreign keys, and which columns get indexed. A reporting query that groups orders by region forces you to store region somewhere you can join to. A fraud check that looks up every account sharing a phone number forces phone number to be a first-class, indexed value rather than a free-text note.

The schema is a contract with the queries. A knowledge graph needs the same contract. The fact that a graph can hold any node connected to any other node does not mean it should.

Mapping the analogy

Every core design decision in a relational schema has a direct counterpart in a knowledge graph.

The last row matters most. In SQL, a query you did not plan for becomes an expensive multi-way join. In a graph, a question you did not plan for becomes a traversal across edges that were never modelled, or that were modelled under five different names.

The ontology is your schema

In a relational database, the schema of tables, columns and keys is written before any data arrives. In a knowledge graph, that role is played by the ontology: the agreed definition of which classes of things exist, which relationships can connect them, and what each one means.

An ontology does more than a table schema does.

• It defines meaning, not just structure. It says that a Supplier is a kind of Organisation, and that OWNED_BY always points from a subsidiary to its parent. Everyone who reads the graph interprets the data the same way.
• It sets the rules for what can connect. If LOCATED_IN may only link a Facility to a Region, a bad extraction that links a person to a country is caught at load time instead of polluting queries later.
• It keeps sources consistent. When five systems call the same relationship five different names, the ontology picks one and maps the others to it. This prevents the vocabulary drift described below.
• It enables reasoning. Rules such as “ownership is transitive” let the graph infer that a company owns its subsidiary’s subsidiaries, without loading that fact separately.
• It makes the graph reusable. Aligning with shared vocabularies such as schema.org or an industry standard lets other teams and tools understand your graph without a translation layer.

This is where the thesis becomes concrete. Choosing what you will read means writing the ontology first. The questions tell you which classes and relationships you need, and the ontology turns that list into a contract the data must follow.

More on Ontology Read my previous Article

https://medium.com/@dhilip.kumar/from-unstructured-chaos-to-queryable-knowledge-designing-ontology-backed-knowledge-graphs-for-9df031d97257

Query-first design in practice

Now scale the bike example up to a real procurement team that wants a supplier-risk knowledge graph. The tempting start is to ingest every contract, invoice, news article and supplier website, extract every named entity, and see what emerges. The query-first start looks like this.

1. Write the questions in plain language. For example: “Which of our products depend, directly or through a sub-supplier, on a factory in a region with an active trade restriction?” and “Which suppliers share a parent company, so that one failure hits several contracts?”
2. Turn each question into a traversal. The first question walks from Product to Component to Supplier to Facility to Region, then checks a Restriction. The second walks from Supplier up OWNED_BY edges to a common Company.
3. Derive the schema from the traversals. The node types are Product, Component, Supplier, Facility, Region, Restriction and Company. The edge types are CONTAINS, SUPPLIED_BY, OPERATES, LOCATED_IN, SUBJECT_TO and OWNED_BY. Properties are only the ones a query filters or returns, such as Restriction.status and SUPPLIED_BY.share_pct.
4. Decide what each source contributes. Invoices populate SUPPLIED_BY. A corporate registry populates OWNED_BY. News articles are useful only if they can be resolved to a Restriction or a Facility; otherwise they stay out.
5. Load, then test the questions. If a question returns nothing or nonsense, fix the model or the extraction before adding more data.

Each coloured path is one question. Together they produce the full schema, and nothing outside those paths needs to be loaded.

The first question then becomes a short, predictable query:

MATCH (p:Product)-[:CONTAINS*1..3]->(:Component)-[:SUPPLIED_BY]->(s:Supplier)
      -[:OPERATES]->(f:Facility)-[:LOCATED_IN]->(r:Region)-[:SUBJECT_TO]->(x:Restriction {status: 'active'})
RETURN p.name, s.name, f.name, r.name, x.name

Every label and relationship in that query exists because the question needed it. Nothing in the graph is there just in case.

Why “load everything and explore” fails

The build-first approach feels productive because the node count goes up every day. The costs show up later, when someone finally asks a real question.

• Inconsistent relationship names. Without a defined edge vocabulary, the same fact arrives as WORKS_AT, EMPLOYED_BY, MEMBER_OF and RELATED_TO. Every query has to guess them all, and most will miss one.
• Generic “related to” edges. Co-occurrence links say two things appeared together, not how they relate. They inflate the graph and carry almost no meaning a query can use.
• Duplicate entities. “IBM”, “I.B.M.” and “International Business Machines” become three nodes. Entity resolution is only tractable when you know which entity types matter and what their keys are.
• Hub nodes. A node like Country: USA or Topic: Technology ends up linked to millions of others. Traversals through it explode in cost and return noise.
• Missing data you actually needed. The one property a key question depends on, such as an ownership percentage or an effective date, was never extracted because no one knew it mattered.
• No way to say “done”. Without target questions, there is no test for whether the graph is complete, correct or useful. Exploration has no finish line.

All six problems have the same root cause. The graph was filled before anyone decided what it was for.

Where the analogy breaks down

The RDBMS comparison is about discipline, not about rigidity. Graphs differ from relational tables in ways that matter, and pretending otherwise would throw away their strengths.

• The schema can grow cheaply. Adding a new edge type to a graph does not need a migration or a rewrite of existing rows. That is a reason to design for today’s questions and extend later, not a reason to skip design.
• Relationships are first-class. In SQL, a many-to-many relationship needs a join table, and deep chains of joins get slow. In a graph, a five-hop traversal is normal. This lets you plan questions that would be impractical in a relational model.
• Some exploration is legitimate. Community detection, centrality and link prediction can surface patterns nobody thought to ask about. But these algorithms work far better on a graph with clean, typed edges than on a hairball, so they reward up-front design too.
• Ontologies can be shared and reasoned over. Semantic web graphs built on RDF and OWL let you reuse public vocabularies and infer new facts from rules. That adds a layer of meaning tables do not have, but someone still has to choose which classes and properties to adopt.

So the honest version of the thesis is this: a graph’s schema is more flexible than a table’s, but it still has to exist before the data does. Flexibility lowers the cost of changing your mind. It does not remove the need to decide in the first place.

Before you load a single node

Treat the graph like a database you are about to design, and answer these questions first.

• Have we written down the 5 to 10 questions this graph must answer?
• Can each question be expressed as a traversal over named node and edge types?
• Does every node type, edge type and property trace back to at least one of those questions?
• Do we know the unique key for each node type, so duplicates can be merged?
• Do we know which source populates each edge type?
• Do we have a test query per question to run after every load?

A knowledge graph earns its value from the questions it can answer, not from the number of nodes and edges it holds. Decide what you will read, then build the graph that makes reading it easy.

I recently attended a university hackathon. Every single team was building “agents” — LLMs wired up to tools, calling APIs, chaining tasks together. Impressive demos, fast-moving code, a lot of excitement in the room.

But when I asked a few teams a simple question — why did the model Hallucinates and just say that word, and not one of the other 50,000 it could have said? I mostly got toward “it’s just the AI.” The honest feedback was before you build on top of LLMs, understand the fundamentals underneath them.

So here it is, built from the ground up: how a straight line turns into something nonlinear, and how that nonlinear machine ends up predicting the next word by looking at every other word at once. No black box. Just the actual mechanism, one step at a time.

Here’s the picture I want you to walk away with every major leap in AI — from the simplest statistical models of the 1950s to the LLM powering agents has been built on the same quiet trick, dressed up in a bigger costume each time.

To see the trick clearly, picture a talent show judge. A contestant finishes singing, and the judge doesn’t shout “yes” or “no” — they hold up a score. 8.2. , 6.5. Somewhere in their head, that number came from a rough formula: pitch mattered a lot, stage presence mattered a little, that one cracked note cost some points. Add it all up, and you get a score. That judge’s habit — score first, then decide — is the whole story.

Here’s the whole journey in one picture four generations of AI, each one adding a single new idea on top of the last:

Act I: The judge learns to guess a probability

Imagine you’re not scoring a talent show — you’re trying to predict something binary. Will this customer churn? Is this email spam? Will this loan default?

The earliest, simplest way to do this is logistic regression. It works in two steps:

Step 1 — Score it. Take your inputs (income, email length, whatever matters) and combine them into a single number using weights you’ve learned:

score = w₁·x₁ + w₂·x₂ + … + b

If this looks like y = mx + b from high school algebra, that’s because it basically is just with more than one x. This raw number is called a logit. It can be anything: -12, 4, whatever. It’s not a probability yet — it’s just a lean, a hunch, a raw score.

Step 2 — Squash it. A raw score of “4” doesn’t mean much on its own. So you run it through a function called sigmoid, which takes any number and folds it into the range between 0 and 1:

probability = sigmoid(score)

Now “4” becomes something like “0.98” — a 98% chance of churning. That squashing step is the entire magic trick. Everything downstream from here is really just this same move, performed at greater and greater scale.

For the technically inclined: the model is trained by maximum likelihood — adjusting the weights w so the predicted probabilities line up as closely as possible with what actually happened, measured via cross-entropy loss. This training principle — maximize the likelihood of the true outcome — never changes for the rest of this story either.

Act II: The judge stops being one person

Logistic regression has a limit: it can only draw a straight line through your data. Real-world patterns are rarely that polite.

Deep learning solves this — but not by simply “adding more of the same.” Here’s the part that trips people up: if you stack ten linear scoring steps on top of each other with nothing else, you still just get… one linear scoring step. Math folds it right back down. Depth alone buys you nothing.

The actual unlock is inserting a nonlinear twist between each layer — a small function (called an activation function, commonly ReLU) that breaks the straight line. Do this repeatedly, and something remarkable happens: the network starts learning to build features on top of features. In an image model, early layers might learn to detect edges, the next layer combines edges into shapes, the next combines shapes into “this looks like a cat’s ear.”

Then, at the very last layer, the network does exactly what our talent show judge did: computes a score, then squashes it into a probability (using softmax, sigmoid’s cousin, when there are more than two possible outcomes — cat vs. dog vs. bird, instead of just yes vs. no).

The judge didn’t retire. Twenty layers of increasingly sophisticated judges now hand their scores up the chain, and the final judge still just says: “score it, squash it, done.”

This is the linear-to-nonlinear jump in a nutshell: linear alone can only cut the world with straight lines; nonlinearity is what lets the model bend.

Act III: The judge starts listening to the room

Deep networks got powerful, but they had a blind spot: earlier architectures (like RNNs) read text one word at a time, in order, like reading with your finger under each word, forgetting what came ten words ago.

Transformers fixed this with a mechanism called attention. Instead of reading in a strict sequence, every word gets to look at every other word in the sentence at once and decide how much each one matters. In the sentence “the trophy didn’t fit in the suitcase because it was too big,” the model needs to figure out whether “it” refers to the trophy or the suitcase — and attention is what lets the model weigh those relationships directly, regardless of distance.

This is a genuine architectural idea, not just “more layers” or “more scale.” A transformer with attention behaves fundamentally differently from a deep network without it.

And yet — look closely at the final layer of a transformer, and you’ll find our judge again: a score for each possible output, squashed by softmax into a probability distribution.

Act IV: The judge sits in a stadium, and never stops reading

So what actually makes an LLM an LLM, if the architecture (transformer) already existed?

Mostly, it’s two things:

1. Scale. Billions of parameters instead of millions. Hundreds of billions of words of training text instead of a curated dataset.
2. A deceptively simple training objective. Predict the next word. That’s it. Read a sentence, hide the next word, guess it, check the answer, adjust the weights. Repeat this trillions of times across nearly the entire written internet, and the network is forced to develop something that looks a lot like grammar, facts, reasoning patterns, and style — not because anyone taught it those things directly, but because predicting the next word well requires implicitly learning them.

Here’s the part I find genuinely beautiful: at the very last step, when an LLM is deciding what word to say next, it computes a logit for every single word in its vocabulary — tens of thousands of scores, each one informed by attention over every token that came before it — and runs all of them through softmax to get a probability distribution over what to say next. Then it samples from that distribution.

The talent show judge is still there. They’re just now scoring 50,000 contestants simultaneously, thousands of times per second, to write a single paragraph — and each score is shaped by everything said so far, not just the last word.

This is the fundamental loop underneath every agent at that hackathon: think, choose a tool, generate a response — every one of those “decisions” is this same score → squash → sample mechanism, running over and over, one token at a time.

The thread that never broke

Every “revolutionary leap” in this story is really a new way of computing a better, richer, more context-aware score. The final move — squash that score into a probability, using sigmoid or softmax — has not changed since the earliest days of statistics. It’s not a coincidence that we still call the output of the last layer of an LLM a “logit.” It’s the same word, because it’s the same idea.

So the next time you’re at a event and someone’s spinning up an agent in twenty minutes, you’ll know what’s actually happening underneath: a talent show judge, still holding up a scorecard, still doing the same two-step move it always did — just with a much, much bigger room to judge, and every other word in the room whispering in its ear.

Introduction

If you’re building an agentic application, at some point you’ll hit the same wall: your agent needs to reason over a pile of unstructured information — PDFs, wikis, transcripts, tickets, contracts — and a vector store alone doesn’t cut it. Semantic search is great at finding similar text. It’s much weaker at answering questions like “who founded companies headquartered in Cupertino before 1980” — questions that require structure, not just similarity.

That’s where knowledge graphs come in. But a knowledge graph is only as useful as the schema behind it. Nodes and edges without a schema are just a pile of triples with no way to validate, infer, or query consistently. That schema is called an ontology, and the order in which you build your ontology and your graph — and how you keep both current as new data arrives — is the difference between a knowledge base that stays useful and one that quietly rots into noise.

This post walks through a practical pattern for building this system when you’re starting from a genuinely unfamiliar domain: you don’t know your entity types yet, you don’t know your relationships yet, and forcing a schema on day one would just be guessing. We’ll cover:

• Why “discover, then formalize” beats “schema-first” when the domain is unknown
• What an ontology actually is, versus what a knowledge graph instance is (with a worked example)
• The AI-native steps that bridge unstructured vectors to a structured graph
• The two loops that keep the whole system alive: the build loop and the query loop

Why not schema-first?

Traditional knowledge engineering says: define your ontology first, then extract data to fit it. That works when you already understand the domain.In my Previous Article I have mentioned that Sematic Interoperability for Data Product , that is for known domain. It falls apart when you don’t — you end up either guessing at a schema that doesn’t match reality, or spending weeks in workshops with domain experts before writing a single line of extraction code.

The alternative — increasingly practical because of LLMs — is discover-then-formalize:

1. Explore the unstructured corpus first, using vector search, so a human can get a feel for the domain
2. Let an LLM do open, schema-free extraction to surface candidate entities and relationships
3. Induce a starter ontology from those candidates (cluster near-duplicate labels, normalize types, human approves)
4. Re-run extraction against that seed ontology to build a properly typed graph
5. Keep both the ontology and the graph evolving as new data arrives, instead of treating either as “done”

This isn’t a rejection of ontology-first design — it’s a recognition that with LLMs in the loop, you can earn the ontology from real data instead of authoring it from assumptions.

Ontology vs. knowledge graph: getting the layers right

A common point of confusion: are nodes and edges the same thing as classes and relationships? Not quite. They’re two different layers.

The ontology (schema / TBox) defines:

• Classes — the types of things that can exist (Organization, Person, Location)
• Subclass hierarchies — Company is a subclass of Organization
• Relationship types — headquartered_in, founded_by, with domain and range constraints
• Datatype properties — attributes like founded_year, which point to literal values, not other nodes

The knowledge graph (instance data / ABox) is the actual facts:

• Nodes — instances of a class (Apple is an instance of Company)
• Edges — instances of a relationship type (Apple —headquartered_in→ Cupertino)
• Literals — attribute values with no independent identity (1976, attached to Apple via founded_year)

That third category — literals — is worth calling out because it’s the most common mistake in extraction pipelines. A literal has no identity and nothing else ever points to it. An entity like Tim Cook, by contrast, does have identity — he could later connect to other facts (other roles, other organizations), so he should become a node, not a flattened attribute.

Worked example

Ontology:

Knowledge graph instance:

(Simplified for the example — Apple was actually founded by Steve Jobs, Steve Wozniak, and Ronald Wayne. The point here is structural, not historical accuracy.)

The ontology says such a relationship could exist between a Company and a Location; the graph asserts that it does, specifically for Apple. That's the whole relationship between the two layers in one sentence.

The ontology, as a schema:

The knowledge graph, as instance data of that schema:

Notice the 1976 node in the diagram is styled differently on purpose — it's drawn as a box because Mermaid needs something to point an arrow at, but conceptually it isn't part of the graph's node set the way Apple, Cupertino, and Tim Cook are. It has no identity and nothing else will ever point to it.

The AI-native bridge: turning exploration into a schema

The gap in a pure “explore, then build ontology” plan is how human exploration of vectors turns into structured types. Semantic search surfaces similar passages — it doesn’t automatically surface “I need a Contract class and a governed_by relationship." You need an explicit step in between:

1. Ingest and chunk the unstructured sources
2. Embed into a vector store — this becomes the exploration substrate
3. Human exploration — a subject-matter expert queries and browses semantically, gets a feel for what matters
4. Schema-free (open) extraction — an LLM pass extracts candidate entities and relationships with no fixed schema, tagging anything unfamiliar rather than discarding it or forcing a bad fit
5. Ontology induction — cluster and normalize the candidate labels (embedding similarity plus LLM-assisted merging) so “Apple Inc,” “Apple,” and “AAPL” collapse into one entity, and near-duplicate relation labels collapse into one predicate. A human reviews and approves this as the seed ontology.
6. Re-run extraction against the seed ontology, now producing properly typed nodes and edges

Laid out as a sequence, the actors involved and the order they talk to each other looks like this:

From here, the two ongoing loops take over.

Loop one: the build loop

This loop never really finishes — it just slows down as the domain model matures.

Unstructured data splits into two parallel paths:

• Vector store path — chunk and embed for semantic search
• Extraction path — LLM extraction, constrained (or guided) by the current ontology, producing candidate triples

Extracted candidates go to human review, which does two things: approves valid nodes/edges into the ontology and knowledge graph store, and — critically — feeds back to update the ontology itself when something genuinely new shows up (a new class, a new relationship type). That updated ontology then guides the next extraction pass. It’s a loop, not a pipeline stage.

There are two extraction modes worth knowing:

• Closed/constrained extraction — the LLM only extracts instances of classes and relations already in the ontology. Use this once the ontology is fairly stable.
• Open extraction with ontology as guidance — the LLM proposes new candidate types when the text doesn’t fit anything defined yet, tagged separately rather than dropped or force-fit. Use this early on, or during periodic re-audits.

A useful discipline: run a conformance check after every extraction pass — does every triple’s subject/object type actually match its relation’s declared domain and range? LLMs do occasionally hallucinate relation types near ontology boundaries; catching that automatically, before it reaches human review, saves a lot of reviewer time.

Both paths converge into a hybrid retrieval layer — vector store plus ontology-backed graph — which is what the query loop actually talks to.

This loop has no natural end state. It keeps running — just less frequently — for as long as new documents keep arriving.

Loop two: the query loop

Every query — from an agent or a human — runs through this loop:

1. Query arrives
2. Ontology interprets and plans — it doesn’t hold the answer, it holds the map. For “who founded Apple?”, the ontology tells the planner: founded_by exists, domain is Company, range is Person, so look for a Company node matching "Apple" and traverse that edge. This is also where disambiguation happens — if "Apple" could mean the company or something else in your domain, the ontology's type system is what lets the agent ask a sharp clarifying question instead of guessing.
3. Split into two retrieval paths:

• Graph traversal for precise, structured, multi-hop facts
• Vector search for fuzzy, narrative, or free-text context

4. Combine and assemble the answer from both

5. Human review — added conditionally, for ambiguous or high-stakes queries only

The two human-in-the-loop points in this whole system are doing genuinely different jobs, and it’s worth being explicit about which one you mean when you say “HITL”: build-time review is about the correctness of the schema and the data; query-time review is about disambiguation and risk on a specific answer. Conflating them tends to produce a review UI that’s mediocre at both.

This is the consumption pipeline — the part of the system every agent or user-facing query actually runs through, on every request, regardless of what stage the build loop happens to be in.

Putting it together

The full lifecycle, end to end:

The build loop runs continuously in the background, growing slower in cadence as your ontology matures. The query loop runs on every single request, and it’s only as good as whatever state the build loop has produced so far. Neither one is ever “done” — and treating either as a one-time setup step is the most common way these systems degrade over time.

What’s not covered here (and probably should be, next)

This post focused on the design pattern. A few things worth a follow-up if you’re taking this to production:

• Ontology versioning and migration — how to evolve class/relationship definitions without breaking already-extracted data
• Conflict resolution — what happens when two documents assert contradictory facts about the same entity
• Evaluation metrics — how do you actually measure extraction quality and ontology coverage over time, not just eyeball it
• Access control and multi-tenancy — if different consumers should see different slices of the graph
• Tooling choices — graph database options (e.g., property graph vs. RDF/OWL stores), and where GraphRAG-style frameworks fit in versus building this extraction/review loop yourself
• Cost and latency tradeoffs — LLM extraction and re-extraction at scale isn’t free; when to batch vs. stream

Conclusion

The instinct that started this whole discussion — “I don’t know my domain yet, so let me explore first” — turns out to be the right one, as long as it’s paired with a mechanism to turn that exploration into structure. Vectors alone give you similarity, not truth; a knowledge graph without an ontology is just an unconstrained pile of triples wearing a fancier name. The ontology is what turns raw extraction into something an agent can trust, validate, and traverse with confidence.

The bigger shift, though, is realizing this isn’t a project with a finish line. It’s two loops running at different tempos: a build loop that keeps refining the schema and the graph as new documents arrive, and a query loop that leans on whatever state the build loop has produced at that moment — no more, no less. Treat either one as a one-time setup task, and the system quietly drifts out of sync with reality. Treat both as ongoing, and you end up with a knowledge base an agent can actually reason over, not just search.

If you’re building this for a real agentic application, start small: pick one narrow slice of your corpus, run it through the discover-then-formalize sequence end to end, and let the seed ontology be genuinely rough. It’s much easier to extend a small, honest ontology than to unwind a large, wrong one.

Using Ontology-Driven Semantic Contracts to Enable Cross-Domain Data Understanding — Author Dhilip Kumar & Arvind Babu

Abstract

The motivation for a Semantic Interoperability Layer stems from a fundamental shift in how data is consumed. Traditional data contracts were designed for traditional systems they enforce structure and format, which is sufficient when human developers mediate every integration. However, as AI agents become first-class consumers of enterprise data, they require more than structure; they need contextual understanding of how business processes relate to one another (e.g., how procurement connects to inventory, logistics, and demand planning within a supply chain). Therefore, interoperability is no longer optional — it is the foundation that enables both traditional systems and autonomous agents to interpret data with shared meaning. The new semantic contract does not replace the traditional contract; it extends it, serving both paradigms while unlocking the contextual and relational intelligence that agent-driven architectures demand.This work proposes a Semantic Interoperability Layer for data products, where interoperability is achieved through ontology-driven semantic contracts. These contracts extend beyond structural definitions to encode meaning, relationships, and intent, enabling systems to align and interpret data consistently across domains.

In this model:

Semantic contracts define meaning, relationships, and intent — not just structure — serving as the binding agreement between data products and their consumers, whether traditional systems or autonomous AI agents.

Ontologies operate as a three-tier shared abstraction (upper, domain, and bridging), enabling automated translation and alignment across domains without manual mapping

Data products become semantically interoperable by design through ontology-driven annotations at the source, eliminating downstream reconciliation and duplicate transformation logic.

A Context Engine provides relational reasoning over the Knowledge Graph, allowing AI agents to understand how business processes connect without human intervention.

Integration shifts from manual transformation to automated semantic reasoning, where queries are resolved through graph traversal and intent resolution rather than point-to-point data mapping

This approach reduces integration complexity, improves data consistency, and enables both traditional systems and AI agents to consume enterprise data with contextual understanding and semantic clarity. By bridging the needs of human-mediated and agent-mediated architectures through a unified semantic contract, it unlocks scalable cross-domain collaboration in AI-first enterprises.

Problem Statement

Data mesh promised domain autonomy with interoperability. Enterprises got the autonomy. The interoperability never arrived.

Despite significant investment in domain-driven architectures, data product platforms, and contract frameworks, enterprises remain trapped in a structural interoperability model — one that enforces shape compatibility while leaving meaning alignment as an exercise for the consumer. The consequences are architectural, not cosmetic.

The structural contract ceiling

Current data contracts enforce payload shape, field types, and SLA guarantees. They operate entirely at the syntactic layer. No mechanism exists in the contract stack to formally declare what a field means, how concepts relate across domains, or under what business intent a data product should be consumed. A status: “CLOSED” is structurally valid but semantically unresolvable fulfilled, cancelled, expired, or merged and the contract is silent.

Consumer-side semantic debt

Without semantic encoding at the source, every consuming domain builds its own interpretation layer: bespoke mapping, transformation, and reconciliation logic that encodes invisible semantic assumptions. This produces pairwise integration mappings across n domains unversioned, untested, and guaranteed to drift. The enterprise accumulates semantic debt that compounds with every new domain, every new consumer, and every schema evolution.

The agent consumption gap

This gap becomes structurally untenable as AI agents enter the consumption plane. A human analyst tolerates semantic ambiguity through tribal knowledge and manual reconciliation. An autonomous agent cannot. When an LLM-based orchestrator receives a procurement dataset, it needs to know that a purchase order triggers an inventory reservation, generates an accounts payable entry, and constrains a logistics schedule — not through prompt engineering, but through formal ontological declaration. Current architectures deliver well-typed payloads with zero relational context: the agent gets the nouns but not the gramma

Root cause

The enterprise data platform has no semantic layer. Contracts enforce structure. Catalogs provide discovery. Governance defines policy. Nothing formally encodes meaning, cross-domain relationships, or consumption intent. This is the architectural gap the Semantic Interoperability Layer closes.

Observable impact

· Integration overhead scales quadratically: Each new domain requires mapping against every existing consumer

· Data reuse collapses: Products are discoverable but semantically unusable across domain boundaries

· Analytics diverge silently: Same concept, different interpretations, conflicting dashboards

· Trust erodes : Consumers rebuild from source, creating shadow pipelines that undermine the mesh investment

· AI adoption stalls at domain boundaries: Agents reason within one domain but fail across domains where no formal relationships exist

Despite advances in data mesh and domain-driven architectures, enterprises face significant challenges in achieving true interoperability:

• Data products expose schemas without meaning, making interpretation ambiguous
• Cross-domain integration requires manual mapping and transformation efforts
• Semantic inconsistencies lead to duplicate logic and conflicting interpretations
• Data contracts are limited to structure, not intent or context
• AI systems struggle due to lack of unified semantic grounding

As a result:

• High integration overhead
• Low data reuse across domains
• Inconsistent analytics and decision outcomes
• Reduced trust in enterprise data

Objectives & Scope

Primary Aim

To design a Semantic Interoperability Layer that positions ontology-driven semantic contracts between the data product plane and the consumption plane — enabling data products to be consistently interpreted, automatically aligned, and consumed with contextual understanding by both traditional systems and autonomous AI agents, without replacing existing data infrastructure.

Objectives

• Define a semantic contract specification that extends structural contracts with formal semantics encoding meaning, relationships, cardinality constraints, and consumption intent using OWL/RDFS, SHACL, and JSON-LD
• Establish a three-tier ontology model — upper ontology for cross-cutting abstractions, federated domain ontologies owned by domain teams, and a collaboratively governed bridging ontology that formally declares cross-domain relationships (e.g., Purchase Order — generates → Accounts Payable)
• Enable automated semantic alignment through an ontology mapping and alignment engine that eliminates pairwise domain mappings in favour of per-domain ontology alignment
• Instantiate a knowledge graph that materializes the ontology with live enterprise state, providing a queryable substrate for bounded graph traversal and relational reasoning
• Deliver agent-ready data consumption through a Context Engine that resolves queries with relational context — enabling AI agents to reason across domain boundaries using formal ontological assertions rather than prompt-engineered heuristics

Scope

In scope

• Semantic contract framework: specification, validation, and lifecycle management for ontology-driven contracts that extend existing structural contracts
• Three-tier ontology architecture: upper, domain, and bridging ontologies with federated ownership and collaborative governance of cross-domain relationships
• Knowledge graph integration: instantiation of ontological relationships as queryable graph , serving as the foundation for all semantic reasoning
• Context Engine: bounded graph traversal with intent resolution for AI agent consumption, delivering scoped context frames rather than raw payloads
• Integration pattern support: event-driven , API gateway , batch , and stream processing for semantic enrichment and validation
• Supply chain as the primary domain use case: procurement, inventory, logistics, and demand planning demonstrating cross-domain interoperability with finance and operations

Out of scope

• Replacement of existing data platforms, lakehouses, or orchestration infrastructure — the SIL augments the current stack, it does not replace it
• Domain-specific business logic modelling — domain teams retain ownership of internal business rules; the SIL governs only the semantic interface between domains
• Fully automated ontology governance without human oversight : the bridging ontology requires collaborative human governance; automation assists but does not replace domain stewardship
• Natural language processing or LLM-based ontology generation : the ontology is formally authored and validated, not inferred from unstructured text
• Enterprise-wide deployment across all domains simultaneously : the architecture is designed for incremental adoption, starting with the supply chain domain and extending domain by domain

Technical Approach & Innovation

Proposed solution

The solution introduces a five-layer Semantic Interoperability Architecture that positions three new capabilities between the data product plane and the consumption plane: a Virtualization Layer that projects meaning without replicating data, a Semantic Interoperability Layer governed by ontology-driven semantic contracts, and a Knowledge Graph that stores relationships and ontological assertions as the queryable substrate for all downstream reasoning.

Each data product is bound to a formal semantic contract — a declarative artifact expressed in JSON-LD with SHACL validation graphs — that extends the structural schema (Avro, Protobuf) with concept definitions (OWL/RDFS), relationship axioms, cardinality constraints, and cross-domain alignment mappings. These contracts are published and validated at the source, not reconciled at the consumer.

Instead of consumers building bespoke interpretation layers for every domain they integrate with, the architecture shifts semantic reconciliation to a shared infrastructure layer. When the supply chain domain publishes a Purchase Order data product, its semantic contract formally declares that this entity triggers an Inventory Reservation, generates an AccountsPayable entry in the finance domain, and constrains a ShipmentSchedule in the logistics domain. Any consumer — whether a BI platform, an operational application, or an autonomous AI agent — can traverse these relationships through the Knowledge Graph without writing a single line of domain-specific mapping code.

The Virtualization Layer ensures this happens without full data replication. The Knowledge Graph holds relationships and ontological structure; when a Context Engine query needs actual data values (quantities, dates, amounts), those queries resolve downward through virtual SPARQL endpoints (Ontop/Morph-KGC) and federated query engines (Trino/Dremio) against live data products. Data stays in place. Meaning is projected. Only high-frequency subgraphs are materialized on demand.

Methodology

Step 1 — Define the semantic data product model

Extend existing data product specifications to include three layers: structural schema (field types, nullability, enumerations), descriptive metadata (ownership, freshness, lineage), and ontology mapping (concept binding, relationship axioms, domain constraints). In the supply chain domain, this means the PurchaseOrder data product declares not only that it has fields like order_id, supplier_id, quantity, and status, but that PurchaseOrder is a subtype of the upper ontology concept Entity, that supplier_id resolves to the Supplier concept through a placed-with relationship, and that status: CLOSED means fulfilled (not cancelled, not expired) as formally defined in the supply chain domain ontology.

Step 2 — Develop semantic contracts

Author semantic contracts as JSON-LD documents with embedded SHACL constraint graphs. Each contract binds the data product to a specific ontology version, declares its concept mappings, and specifies cross-domain relationships through the bridging ontology. The supply chain PurchaseOrder contract includes the assertion PurchaseOrder — generates → AccountsPayable with cardinality exactly-one and a trigger condition on status transition to APPROVED. This assertion is not documentation — it is a machine-executable ontological declaration that the Alignment Engine can traverse and the Context Engine can reason over.

Step 3 — Implement the three-tier ontology and knowledge graph

Build the ontology in three tiers: an upper ontology defining universal abstractions (Entity, Process, Event, Agent, Location, Measure), federated domain ontologies owned by each domain team (Supply Chain defines PurchaseOrder, Inventory, Shipment, DemandForecast; Finance defines AccountsPayable, CostCenter, JournalEntry), and a collaboratively governed bridging ontology where cross-domain relationships are formally declared. Instantiate the ontology as a knowledge graph (Neo4j/Neptune/Jena) containing typed relationship edges — not replicated data — that represent the live state of enterprise relationships.

Step 4 — Deploy the virtualization layer

Implement virtual SPARQL endpoints using Ontop or Morph-KGC that map R2RML definitions to live data product sources, enabling the Knowledge Graph to resolve entity attributes on demand without data replication. Deploy federated query engines (Trino/Dremio) for cross-domain SQL and GraphQL federation. Configure selective materialization for high-frequency subgraphs — for example, the PurchaseOrder → InventoryReservation → ShipmentSchedule hot path may be materialized for sub-second agent queries, while the long tail of historical relationships remains virtualized.

Step 5 — Enable semantic validation

Implement SHACL/ShEx constraint enforcement at two checkpoints: contract publish-time (validating that the semantic contract is internally consistent and compatible with the bridging ontology) and data ingestion-time (validating that incoming data conforms to the declared semantic constraints). When the supply chain domain publishes a new PurchaseOrder with supplier_id referencing a supplier not registered in the ontology, semantic validation catches this before it enters the consumption plane — shifting quality left rather than reconciling downstream.

Step 6 — Activate translation, alignment, and context services

Deploy the Alignment Engine (RML/R2RML-based) for automated cross-domain semantic translation. Activate the Context Engine for bounded graph traversal with intent resolution — when an AI agent queries “impact of delaying PO-4821,” the engine walks the knowledge graph through bridging relationships (PurchaseOrder → InventoryReservation → ShipmentSchedule → DemandForecast → SLA commitments), resolves “impact” against the supply chain domain’s consequence model, bounds traversal at domain contract boundaries, and returns a scoped context frame. This is formal relational reasoning over a typed, constrained graph — not keyword search, not RAG retrieval.

Step 7 — Integrate with existing data platforms

Connect to existing infrastructure through four integration patterns: event-driven propagation (Kafka/CloudEvents) for real-time CDC with semantic enrichment at stream level, a semantic API gateway (GraphQL-LD) for synchronous queries with contract enforcement, batch ontology sync (Spark/Airflow) for historical backfill and bulk enrichment, and continuous stream validation (Flink/ksqlDB) for inline semantic constraint checking on streaming data products. The SIL augments the existing stack — no platform is replaced.

THE SHIFT (Top-left)

From Structural Integration → Semantic Interoperability

Today

· Schema-driven data exchange

· API-level integration

· Meaning interpreted manually

Emerging

· Ontology-driven data alignment

· Semantic contracts

· Machine-interpretable meaning

Key Insight

Interoperability is shifting from aligning structure to aligning meaning

2. THE PROBLEM (Left-middle)

Why Data Products Still Fail to Interoperate

· Data contracts enforce structure, not meaning

· Cross-domain integration requires manual mapping

· Semantic inconsistencies across domains

· Duplicated transformation logic

· AI systems lack contextual grounding

Result:

· Fragile integrations

· Inconsistent analytics & decisions

· High integration cost

· Low trust in enterprise data

3. THE PROPOSED PARADIGM (Centre — intro)

Semantic Interoperability Layer

Make data products interoperable by design using ontology-driven semantic contracts

4. CORE ARCHITECTURE (CENTER — MAIN VISUAL)

Layered Architecture

[ CONSUMERS ]

AI Systems • Applications • Analytics

↓

[ SEMANTIC INTEROPERABILITY LAYER ]

Semantic Contracts • Ontology Mapping • Alignment Engine

↓

[ ONTOLOGY / KNOWLEDGE GRAPH ]

Domain Concepts • Relationships • Vocabulary

↓

[ DATA PRODUCT LAYER ]

Domain Data Products (Schema + Metadata + Semantics)

↓

[ DATA SOURCES ]

Systems • Streams • Databases

Core Flow

Data Product → Semantic Mapping → Alignment → Consumption

Key Capabilities

· Semantic contract validation

· Ontology-driven translation

· Cross-domain alignment

· Meaning-aware data consumption

Key Insight

Data integration becomes semantic reasoning, not transformation

5. KEY ARCHITECTURAL SHIFTS (Right middle)

· Schema Contracts → Semantic Contracts

· Data Mapping → Ontology Alignment

· Manual Integration → Automated Interpretation

· Metadata → Semantic Context

· Data Exchange → Context Exchange

6. TRADE-OFFS (Right-lower)

· STANDARDIZATION ↔ FLEXIBILITY

o Global ontology vs domain variation

· ACCURACY ↔ COMPLEXITY

o Richer semantics vs modeling effort

· CENTRALIZATION ↔ FEDERATION

o Shared ontology vs domain ownership

· PERFORMANCE ↔ SEMANTIC DEPTH

o Speed vs reasoning richness

7. WHAT THIS ENABLES (Bottom-left)

· Seamless cross-domain interoperability

· Reduced integration and transformation effort

· Consistent data interpretation across systems

· Improved data reuse across domains

· AI-ready, context-aware data consumption

· Higher trust in enterprise data

8. FUTURE OUTLOOK (Bottom-centre)

· Data contracts evolve into semantic, intent-aware contracts

· Ontologies become core enterprise infrastructure

· Integration shifts to semantic alignment platforms

· AI systems rely on context-rich data ecosystems

· Enterprises move from data-driven → meaning-driven architectures

9. DISCUSSION PROMPTS (Bottom-right)

· Can schemas alone ever ensure interoperability?

· Who owns the enterprise ontology?

· How much semantic standardization is needed?

· Can semantic alignment be fully automated?

· How do we scale ontology across domains?

DESIGN GUIDELINES (CRITICAL)

Visual Balance

· 70–75% visuals

· 25–30% text

Colour Coding

· Blue → Data / systems

· Purple → Semantic / ontology

· Orange → Alignment / flow

Layout Tips

· Centre diagram must be largest element (40%)

· Left = problem narrative

· Right = interpretation (shifts + trade-offs)

· Bottom = impact + future

FINAL “WOW” LINE (Add near center or bottom)

“We don’t integrate data — we align meaning. And we don’t copy data — we project it..”

1-LINE PITCH

“We extend data contracts with ontology-driven semantics, so data products become interoperable by meaning, not just structure.”

Ever wondered whether a hobby can eventually become a habit? I was curious about how closely the two are connected and whether they share the same roots. So, I did some digging using a few online research tools — and here’s what I discovered

Hobby: an activity done regularly

Habit: in Latin Habere Meaning ‘have, consist of’

Both done regularly and consistently. Create a hobby to build habits or build a habit to create a hobby.

James Clear breaks habits into four big buckets and here’s the simplest way to understand them:

I. Foundational Concepts of Atomic Habits

The core philosophy of Atomic Habits is that big, meaningful change is the result of small, consistent improvements. Behavior change is viewed as an iterative cycle: Try → Fail → Learn → Try Differently, reinforcing that progress is nonlinear but cumulative.

James Clear emphasizes that behavior does not exist in isolation — it is shaped by the interaction between a person and their environment, expressed through the formula

B = f(P, E)

(Behavior = function of Person + Environment).

Foundational research also distinguishes two major habit types:

• Outcome-Based Habits, which focus on results
• Identity-Based Habits, which focus on who you become

Identity-based habits are ultimately more powerful because they reshape self-perception, making behavior change more sustainable.

II. The Four Steps of Building Habits (The Habit Loop)

The formation and reinforcement of any habit follow a predictable four-step loop. At its core, the reward is what drives the entire cycle, and it emerges only after the interaction of the first three steps: cue → craving → response.

1. Cue — The Trigger
   A cue acts as a reminder or prompt. Its purpose is simple: to help you notice a potential reward.
2. Craving — The Motivation
   A craving creates a powerful desire. Its function is to make you want the reward that follows the cue.
3. Response — The Action
   The response is the behavior itself. Its purpose is to obtain the reward that the craving motivates.
4. Reward — The Outcome
   The reward provides satisfaction and reinforces learning.
   We pursue rewards because they both please us and teach us — and the human brain is wired with extensive neural circuitry that pushes us toward seeking them.

III. The Four Laws of Habit Change

James Clear outlines four practical laws that can be applied to the habit loop, depending on whether you want to build a new habit or break an existing one. These laws provide a simple, actionable framework for shaping behavior with intention.

A. Four Laws for Building Atomic Habits

1. Cue — Make it obvious
   Increase visibility and clarity so the behavior is easy to start.
2. Craving — Make it attractive
   Pair the habit with something pleasurable or motivating.
3. Response — Make it easy
   Reduce friction and simplify the behavior so it’s effortless to perform.
4. Reward — Make it satisfying
   Reinforce the behavior with immediate satisfaction to encourage repetition.

B. Four Laws for Breaking Bad Habits

1. Cue — Make it invisible
   Remove or reduce exposure to triggers.
2. Craving — Make it unattractive
   Reframe the habit to highlight its negative consequences.
3. Response — Make it difficult
   Add friction so the behavior becomes harder to perform.
4. Reward — Make it unsatisfying
   Introduce immediate accountability or negative consequences.

IV. The Power of Visual Cues and Small Changes

Research consistently shows that visual cues play an outsized role in shaping human behavior. Of the body’s 11 million sensory receptors, 10 million are dedicated to sight, meaning that nearly half of the brain’s processing power is devoted to visual input. Because of this dominance, even small visual signals can create disproportionate behavioral change.

A striking example illustrates this principle:

The Schiphol Airport Case: In the 1990s, cleaning staff at Amsterdam’s Schiphol Airport placed a tiny fly-shaped sticker near the center of men’s urinals. This simple visual cue encouraged users to aim more accurately, reducing spillage significantly. The result was measurable and impressive: an 8% reduction in annual restroom cleaning costs — all triggered by a sticker smaller than a coin.

This incident reinforces a key truth of habit formation:
Small habit changes, anchored by clear cues, can create remarkably large benefits — both socially and personally.

From Prompt to Context: The Rise of Context Engineering in Agentic AI

Whenever a meaningful conversation begins, a good presenter sets the context first. Before diving into details, they align everyone’s minds bringing all participants’ “human large language models” into the same mental space so the vision, agenda, and goals move in the right direction.

A Scene That Explains It All

Recently, I watched a movie that beautifully illustrated this idea.

Two childhood friends grew up together in a small village. Years later, one of them moved to the city for family reasons. When he returned to the village for a wedding, he met his old friend again. The friend from the village recognized him instantly, but the one from the city couldn’t recall his name.

Embarrassed, he asked, “What’s your name again?”

Instead of answering directly, the village friend gave him clues where they used to live, the games they played, even the character names they used as kids. Each clue provided context. Gradually, the city friend pieced the memories together and finally remembered his friend’s name.

It was an emotional and nostalgic moment and surprisingly, it reminded me of how Large Language Models (LLMs) work.

To get a precise answer, you don’t just throw in a question. You provide context examples, cues, structure the way a human brain recalls memories and connects meaning.

In this story, the prompt was the question (“What’s your name?”), but the context made understanding possible.

From Prompt Engineering to Context Engineering

In the early phase of Generative AI, everything revolved around prompting. Prompt engineering became an art form carefully crafting questions and instructions to make LLMs respond effectively. But as Agentic AI evolved, something shifted.It was no longer enough to give a single prompt and expect a perfect answer. Real-world systems needed reasoning across multiple turns, longer time horizons, and dynamic knowledge sources.

This led to the rise of Context Engineering the discipline of curating and maintaining the optimal set of information (or tokens) that an LLM uses during inference.

What Is Context Engineering?

Context engineering is about orchestrating everything the model needs to think and act intelligently not just the immediate prompt.

It includes:

• System Instructions: Defining the agent’s goals and rules.
• Memory: Persisting relevant information across interactions.
• Tools and APIs: The capabilities exposed through frameworks like the Model Context Protocol (MCP).
• External Data: Information pulled in real time from documents, sensors, or APIs.
• Message History: Prior conversations that shape the next decision.

An agent running continuously generates more and more data that may influence future reasoning. Good context engineering means selecting the smallest, most relevant set of signals that maximize the model’s performance much like a presenter keeping the discussion focused on what truly matters.

Context is not a monolithic concept but a multidimensional construct that spans several domains:

• Context by Role in Reference and Meaning : Philosophical and Linguistic Contexts define how meaning and reference are established.
• Functional Context : AI and Computational Contexts describe how systems reason, infer, and act using structured assumptions.
• Context in Dynamic and Procedural Contexts emphasize how knowledge evolves and becomes actionable.
• Context in Dialogue and Natural Language Systems: Sensorial and Logical Contexts unify human and machine perception through measurable, temporal, and semantic dimensions.

Together, these frameworks highlight that context is the connective fabric of understanding enabling both human cognition and artificial intelligence to interpret, reason, and act meaningfully within a world rich in ambiguity and intent.

This mind map visually organizes the taxonomy of context across six major domains.

“Context”, branching into six key categories:

1. By Role in Reference & Meaning — Linguistic and philosophical contexts like Objective, Intentional, K-Context, Alternative K-contexts, and Fictional Contexts.
2. Functional in AI & Modeling — Contexts used in computation such as Projection, Approximation, Ambiguity, Mental State, and Problem-Solving Contexts.
3. Dynamic & Procedural — Contexts that evolve with reasoning, including External Knowledge, Contextual Knowledge, and Proceduralized Context.
4. In Dialogue & NLP — Communication-oriented contexts: Dialogical, Situational, Interlocutionary, Domain, and Dialogue Contexts.
5. In Ubiquitous Computing — Sensor-based contexts following the 5W1H model (Who, What, Where, When, Why, How) and their Preliminary, Integrated, and Final stages.
6. By Logical Structure — Formal representations such as Context Sets, CIL Contexts, Contextual Propositions, and Interpretation/Evaluation Contexts.

Conclusion

Overall, the mind map captures how context operates across philosophy, AI, linguistics, logic, and computing, illustrating that context is a multidimensional framework connecting meaning, reasoning, and situational awareness.

References

• Recanati, F. (2004). Literal Meaning. Cambridge University Press. — Discusses intentional vs public aspects of context, context-shifts, and how alternative contexts can supply content for indexicals and demonstratives. (PMC)
• Bonomi, A. (2008). Fictional Contexts. In Context-Sensitivity and Semantic Minimalism (eds. G. Preyer & G. Peter). — Treats fictional contexts where characters (ficta) act as agents and truth is evaluated relative to a story-world. (abonomi.it)
• Gurevych, I., Porzel, R., & Malaka, R. (2006). In Context: Integrating Domain- and Situation-specific Knowledge for Dialogue Systems. In W. Wahlster (Ed.), SmartKom: Foundations of Multimodal Dialogue Systems, Springer. — Provides a table of context types for spoken dialogue systems: dialogical context, ontological/domain context, situational context, and interlocution Ary (user) context, plus the notion of overall dialogue context. (download.mmag.hrz.tu-darmstadt.de)
• Bunt, H. (2000). Dialogue Pragmatics and Context Specification. In Abduction, Belief and Context in Dialogue; Studies in Computational Pragmatics, pp. 81–150. Benjamins. — Discusses what constitutes dialogue context and how linguistic, situational and user-related factors jointly determine interpretation and generation. (ResearchGate)
• Larsson, S., & Traum, D. (2000). Information State and Dialogue Management in the TRINDI Dialogue Move Engine Toolkit. Natural Language Engineering, 6(3–4), 323–340. — Treats dialogue as updates to an information state/dialogue context, integrating linguistic, domain, and user information. (Edinburgh Research)
• Anthropic Applied AI Team. (2025, September 29). Effective context engineering for AI agents. Anthropic. Retrieved from https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

A “stateless agent” in agentic AI is actually a misnomer because true agents, by definition, need some form of memory and adaptability, while stateless architectures simply repeat tasks without learning or accumulating experience. The task-oriented automations operating with context understanding and natural language capabilities, but lacking persistent memory is context-powered programs not an agent here is why,

Stateless Agents vs. Automation

• Stateless agents process each interaction as a standalone event, with no retained history — making their outputs predictable and repeatable for the same inputs.​
• This is ideal for basic, repetitive functions (like answering one-shot queries or running isolated computations), but it isn’t “agentic” in the sense of goal-driven, learning AI.​
• When memory and learning are missing, these systems become automations they execute jobs but do not adapt, improve, or personalize based on prior experience.​

Agent Characteristics and the Role of Memory

• True agents are designed to remember context, update themselves based on feedback, and pursue tasks with dynamic strategies.​
• Stateless systems may have natural language understanding and context processing per request — sometimes leveraging system prompts or instructions — but without memory, they cannot develop “agency” or autonomous capabilities.​

Architectural Implications

• Some stateless frameworks mimic agents: they interface with Model Context Protocols (MCPs), support sophisticated prompting, and parse intent flexibly.
• However, these are not “agents” in a full sense — they’re programs or automations that repeatedly execute mapped instructions for business functions.​
• The absence of memory and autonomous learning means they aren’t agentic, nor are they truly “stateless agents.” Rather, they are stateless processes, agentless automations, or context-powered programs.​

In short, a stateless architecture with natural language and context processing is not truly agentic — it’s better defined as intelligent automation or a process that executes mapped jobs without memory or autonomy.

References

1. https://www.belsterns.com/post/statefulvs-statelesaiagents
2. https://insights.daffodilsw.com/blog/stateful-vs-stateless-ai-agents-when-to-choose-each-pattern
3. https://zbrain.ai/building-stateful-agents-with-zbrain/
4. https://mem0.ai/blog/why-stateless-agents-fail-at-personalization
5. https://codewave.com/insights/agentic-ai-vs-ai-agents-key-differences/
6. https://dev.to/abhishekjaiswal_4896/the-difference-between-ai-agents-and-traditional-ai-models-1aj

Software engineering has undergone many paradigm shifts in its history. From the early days of machine-level programming to object orientation, from client-server computing to the cloud, each era has brought new abstractions and challenges. The latest shift, driven by intelligent agents and natural language interfaces, is a move from contract-driven systems to context-driven systems.

This transition is not just about technology — it changes how we design, validate, and trust software. Contracts remain important, but context is now the driving force that determines how systems reason, adapt, and act.

The Era of Contracts

For most of software engineering’s history, the contract defined how software was built and consumed. A contract was a formal, precise, and deterministic agreement between components — whether between user and interface, two systems, or a developer and the database.

• User Contracts: End users interacted through rigid controls — text boxes, list boxes, checkboxes, and buttons. These controls exposed a contract: the user could only provide information in permitted formats, and the system responded accordingly.
• UI Contracts: In web applications, HTML defined the contract. Each element — forms, buttons, event handlers — was a binding agreement between developers and browsers on what behavior was expected.
• Integration Contracts: As systems grew interconnected, XML schemas, JSON structures, and REST APIs became the lingua franca. Each side had to adhere strictly to schema definitions; otherwise, communication failed.
• Database Contracts: Structured Query Language (SQL) epitomized contract enforcement. To query data, you needed to know the exact schema, table, and column definitions. Even the rise of NoSQL could not fully replace SQL because developers valued this predictable contract.

In this world, if you broke the contract, the system failed. Contracts provided guardrails for reliable execution, but they were rigid and unforgiving.

The Rise of Context

The growth of AI-powered agents has disrupted this model. Instead of relying only on fixed contracts, systems are now expected to interpret context.

• Natural Language First: Users increasingly interact with software through natural language, not predefined controls. A request like “Show me last quarter’s sales in India” does not conform to a strict API contract. Instead, it requires interpretation of intent.
• Context as Meaning: Context is the broader semantic environment surrounding a request. It is not just what the user says, but also how they say it, when they say it, and what tools are available for execution.
• Agents as Interpreters: The software agent now mediates between ambiguous, fluid human intent and strict system contracts. The flow of interaction becomes:
• User context: expressed in natural language.
• Agent context: defined by system prompts, memory, and reasoning strategies.
• Tool context: exposed through specifications like API descriptions or Model Context Protocol (MCP).
• Execution as Mapping: The agent maps human context into system contracts — converting ambiguous intent into SQL queries, API calls, or tool invocations. Contracts still exist, but they are wrapped in an interpretive layer of context.

In essence, contracts have not disappeared. They have been subsumed into a context-driven pipeline, where interpretation precedes execution.

Context → Contract: A New Flow

The modern execution flow in context-driven software looks like this:

1. User Intent: Expressed in natural language.
2. Agent Grounding: The system prompt or reasoning model ensures the agent aligns with the right domain and scope.
3. Tooling Context: Agents check exposed capabilities — schemas, APIs, or knowledge sources — to know what they can leverage.
4. Translation to Contract: The agent maps intent into a tool-specific contract (e.g., creating an SQL query, calling an API, or executing a workflow).
5. Result Interpretation: The output is contextualized back into human-readable form.

This pipeline ensures that context arrives before contract. The system must first understand why and what is being asked before it can execute.

From Deterministic to Semantic Systems

This shift is more than technical — it represents a philosophical change in software design.

• Deterministic Era (Contract-driven): Inputs were clean, structured, and validated. The user pressed a button, the system executed a contract, and the output was deterministic.
• Semantic Era (Context-driven): Inputs are natural, ambiguous, and conversational. Interpretation is required before execution. Contracts remain in place, but they serve the tools, not the user.

This mirrors the transition from command-driven mainframes → GUI-driven desktops → web APIs → context-driven agents. Each wave has democratized access to computation, and each has expanded the role of Real-World Examples

• ChatGPT Plugins and MCP: When a user asks “Find me the top three vegan restaurants nearby”, the AI interprets intent and maps it to a contracts-based tool like Yelp API. Yelp still exposes a predictable API schema, but the user is oblivious — the agent is bridging intent to contract.
• LangChain Agents: In AI agent frameworks, tools expose structured contracts for functions. Agents parse user context and choose which contract to invoke.
• Databases in Agentic Workflows: A user’s natural language query like “Show me customers who purchased more than three times last month” can be transformed into a precise SQL command by the agent, without user knowledge of schema details.interpretation and flexibility.

Implications for Software Engineering

This context-first shift is redefining engineering practices:

• New Design Discipline: Prompt engineering, context scoping, and guardrail design become as critical as schema definition once was.
• Contextual Validation: Instead of only validating data formats, engineers must validate interpretation. Does the agent truly understand the user request correctly?
• Exposed Tool Context: Tools must expose contracts in machine-readable, context-friendly formats like OpenAPI, JSON Schema, or MCP.
• Shifting Roles: Traditional roles (UI developer, API designer, DBA) may converge into Context Architects — professionals who design how context maps into tools.
• Trust and Safety: Since context is probabilistic, misinterpretation risks increase. Systems must include safeguards like grounding checks, system prompts, and rule-based filters.

Opportunities and Risks

• Opportunities:
• Democratization of software usage (anyone can “talk” to systems).
• Greater integration flexibility, since context bridges heterogeneous systems.
• Enhanced personalization, as agents adapt contextually to each user.
• Risks:
• Misinterpretation of context leading to wrong actions.
• Over-reliance on language models without proper validation.

Security hazards if malicious or ambiguous context bypasses safeguards.

Future Outlook

In the years ahead, software may increasingly expose capabilities instead of APIs. Instead of forcing developers or users to learn schemas, systems will declare what they can do, leaving context-driven agents to handle the specifics of execution.

Imagine a world where:

• A finance system no longer publishes SQL schemas but simply offers capabilities — “query transactions,” “generate reports.”
• A logistics system simply exposes affordances like “track shipment” or “optimize route,” and agents orchestrate these capabilities based on context.
• Developers design not only system APIs, but also contextual affordances that intelligent agents can leverage.

The role of software engineering will then expand — from contract writers to context designers.

Conclusion

The shift from contract to context marks a foundational transformation in software engineering. Contracts still underpin systems, holding structure and precision, but they are no longer the first-class citizens. Instead, context — shaped by intent, semantics, and interpretation — leads the way.

In this new era, software engineers are context engineers. Their job is to ensure that the bridges between human intent and system execution are reliable, safe, and meaningful. Software is no longer just about handling precise input — it is about understanding, adapting, and responding to context.