Most AI systems are built around a simple expectation: ask a question and get an answer.

For financial analysis, I started wondering whether that is always the right behaviour.

Sometimes, the better answer is simply — the evidence isn’t there.

I have been building a small IPO analysis system that answers questions using only a company’s regulatory filing. The idea is deliberately narrow. No market opinions, no assumptions, and no filling in missing information from what the underlying model may already know.

The system works from a structured knowledge base created from the filing. Each fact carries its source page, section and disclosure status. The language model sits at the end of the process and narrates only the information that survives the earlier checks.

The model doesn’t get the final say

One lesson from earlier experiments was that asking a model to please stay within the provided context isn’t a particularly strong control.

So I moved that responsibility outside the model.

The current system uses a three-stage pipeline.

First, certain questions such as investment recommendations, future projections and fair-value opinions are blocked deterministically.

For legitimate questions, the model can identify which part of the knowledge base may contain the answer. But that choice is then validated in code. If the requested information does not actually exist, it never reaches the final answering step.

Python also handles the numbers. The language model does not perform arithmetic or invent missing values.

Its job is much smaller: Explain the evidence it has been given.

A surprisingly useful failure

During testing, I deliberately asked questions that a general-purpose model could easily answer from its training data.

A common financial ratio was one example.

But if the relevant definition wasn’t available within the information I had allowed the system to use, it didn’t quietly reach into general knowledge and complete the answer.

It stopped — That was exactly the behavior I wanted.

The distinction becomes even more important with company-specific information. Understanding what a financial ratio means is one thing. Claiming what that ratio is for a particular company is something entirely different.

The second requires evidence.

Why this matters

In many applications, a plausible answer may be good enough.

Financial documents are less forgiving.

A number that looks reasonable but cannot be traced back to the filing can easily be mistaken for a disclosed fact. An estimate can start looking like evidence simply because it was written confidently.

That makes restraint part of the system design.

The goal is not to make the model less capable. It is to decide where that capability is allowed to operate.

Most of my career has involved getting systems to do more. This project has been an interesting reversal: part of the value comes from making the system deliberately do less.

For financial AI, reliability is not only about producing the correct answer.

It is also about controlling what the system is allowed to claim.

And sometimes, “the filing doesn’t provide enough evidence” is exactly the answer the system should give.

Thank you for reading…

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Designing AI Around Evidence, Not Confidence was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you follow AI discussions or news, you’ve probably encountered terms like "LLM," "Claude," "GPT," "Ollama," "LangChain," "RAG," and "GraphRAG"—without a clear sense of where they fit.

It sounds like a lot. It isn’t, actually. The problem isn’t that these things are individually hard — it’s that nobody ever draws you the map of how they fit together. So let’s just draw the map.

Why this feels confusing (it’s not you)

Every one of those words describes a completely different kind of thing. Some are models. Some run models. Some serve models over an API. Some coordinate the whole app. Some are just strategies for finding information before answering a question.

Mixing all of these into one sentence—"we used LangChain with RAG on top of vLLM serving a Llama model”—is like describing a restaurant by saying “the menu, the chef, the delivery app, and the gas line” all in one breath. Technically all real, all connected, and all completely different jobs.

The full stack, top to bottom

Here’s the entire thing. Every AI app, no matter how fancy, is basically this stack:

1. The App — where you type your question. A chatbot, a website, a CLI, whatever.
2. Orchestration — decides what happens next. Should we look something up? Call a tool? Just answer? This is LangChain, LangGraph, or LlamaIndex.
3. Retrieval — goes and finds actual evidence before answering. This is RAG or GraphRAG.
4. Model serving — loads the AI model and runs it efficiently. Ollama, vLLM, SGLang, llama.cpp.
5. The model (LLM) — the actual trained network. Llama, Qwen, Mistral, GPT, Claude.
6. Hardware — CPU/GPU. The thing quietly paying for all of this.

That’s the whole map. Every confusing tool name you’ve ever seen fits into exactly one of these six boxes. Once you know the box, the tool stops being intimidating — it’s just “oh, that’s an orchestration thing” instead of a mystery word.

One honest caveat: real tools don’t always respect these boundaries neatly. llama.cpp does inference and has its own little server built in. vLLM handles serving and dips into orchestration-adjacent territory. Think responsibilities, not strict rulebooks, and you’ll be fine.

Here’s that same map, one card per layer:

Layer 1: The model isn’t the whole app

An LLM — Llama, Qwen, Mistral, GPT, Claude — is just a trained neural network that predicts the next word, over and over, extremely fast. By itself, it’s a file. It can’t take your question, can’t run itself, can’t do anything until something else loads it up and starts feeding it prompts.

This trips people up constantly: “we have access to GPT” does not mean “we have a working app.” You still need everything below it.

Layer 2: Who actually runs the model

This is the model-serving layer — the software between “a trained model exists” and “you can actually talk to it.”

• llama.cpp — runs models locally, even on a regular laptop CPU, especially compressed (“quantized”) versions. Great for offline or budget setups.
• vLLM — built to handle lots of requests at once, production-grade, usually GPU-powered.
• SGLang — optimized for speed and more complex, multi-step generation, also production-grade.
• Ollama — the easy button. Downloads models, manages them, gives you a simple local API to call. It is not the model itself, and it’s not raw low-level inference — it’s the convenience layer sitting on top of it.

Layer 3: Who decides what happens next

LangChain and LangGraph don’t do any of the actual “thinking” — no neural network math happens inside them. They just decide the sequence: check this, call that, then respond.

• LangChain gives you pre-built pieces to plug together — tools, integrations, model calls.
• LangGraph is for when the steps aren’t a straight line — there’s branching, memory, maybe looping back before it’s actually done.

Layer 4: Getting the model real information first

This is where RAG and GraphRAG live, and it’s probably the single most misunderstood part of the whole stack—because neither one is a model, a database, or a serving engine. They’re both just strategies for finding good evidence before the model writes an answer.

RAG (Retrieval-Augmented Generation): search for relevant text — usually using something called vector search, which finds passages that mean something similar to your question — and staple that text onto the prompt before sending it to the model. Simple, effective, and the reason a chatbot can answer questions about stuff way outside what it was originally trained on.

GraphRAG: Instead of grabbing loose, similar-sounding chunks of text, it pulls a connected map of facts—entities and the relationships between them—usually from something called a knowledge graph (Neo4j is a common tool for this). It’s the difference between handing someone a stack of relevant articles versus handing them a diagram that already shows how everything connects.

Here’s the one sentence worth remembering from this entire post:

RAG and GraphRAG never change how the model “thinks.” They only change what it’s given to think with. Same model, same hardware, same underlying math — just better material in front of it.

Layer 5: When AI starts making its own decisions—"agentic AI”

Everything so far has been mostly fixed steps: get a question, maybe retrieve something, send it to the model, get an answer. An agent is different—it's a setup where the model gets to decide, on its own, what to do next: search for something, run a calculation, call another tool, or ask a follow-up question—and keep looping through these choices until an actual goal is met, not just until one step is technically finished.

Agents live in the orchestration layer. They use retrieval, model serving, and the LLM as tools toward a goal, instead of following a fixed script top to bottom. LangGraph shows up a lot here specifically because it’s built for this kind of branching, multi-step, sometimes-looping process.

Quick myths, busted

• “Ollama is an LLM.” Nope — Ollama runs and manages models. The LLM is what it’s running.
• “RAG talks directly to the model’s inner workings.” Nope — RAG just preps a finished prompt, which then goes through the serving layer like any other request.
• “GraphRAG replaces the LLM.” Nope — it just hands the LLM better material to work with.
• “GraphRAG is always the better choice.” Also nope. For a quick, simple fact lookup, plain RAG is faster and cheaper. Save the relationship-graph approach for genuinely connected, multi-step questions.
• “Every tool fits neatly into one box.” Mostly, but not perfectly — remember the caveat above. Use this map as a mental model, not a legal contract.

The whole thing, compressed

LLM = the model. Can’t run itself.

Serving layer (Ollama/vLLM/SGLang/llama.cpp) = runs the model.

Orchestration (LangChain/LangGraph) = decides the steps.

RAG/GraphRAG = fetch good evidence before answering.

Agents = orchestration that makes its own calls, in a loop, toward a goal.

Hardware = pays the electricity bill.

Once this map is in your head, new AI tool names stop being scary. You’ll just ask yourself, "Which of these six boxes does this thing actually belong to?” — and nine times out of ten, you’ll already know the answer.

Final Thoughts

And that’s the stack. The next time someone says, “We’re using RAG with a LangGraph agent on vLLM,” you won’t have to nod thoughtfully and hope nobody asks a follow-up. You’ll know they mean: retrieve the right information, coordinate the workflow, and run the model efficiently.

Go ahead — use “orchestration layer” in a meeting like it has always been part of your vocabulary.

Thank you for Reading 🙌 and Keep an eye out for similar blogs!!

Editorial note: This guide presents a responsibility-based view of AI infrastructure based on my current understanding. It is intended as a learning framework, not a representation of universal industry practice. Because product capabilities evolve quickly, always validate deployment decisions against the latest official documentation.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

A Simple Mental Model for Modern AI Infrastructure was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Whether it’s:

• Presenting your analysis and watching stakeholders disengage one by one
• Or dropping a painfully long Slack message only to get zero reactions

Stumbling at the final hurdle hurts more than the messy middle.

I’ve been there. Hell, I’m still there sometimes — except now, when it happens, I know exactly why.

The real tell-tale sign

One of the biggest signals of an analyst who actually knows what they’re talking about isn’t technical depth. It’s restraint.

Giving more information than is necessary is a sure-fire way to lose a room. Not because people don’t respect the effort, but because they don’t have the bandwidth to take the full journey with you.

Clarity isn’t about saying more — it’s about deciding what not to say.

The temptation: “Let me show you how hard I worked”

When you’ve put genuine hours into something, there’s a gut feeling that you need to show your working.

So you end up talking about:

• What the business problem is
• What you had to do to structure the data
• How every metric is calculated (even the irrelevant ones)
• What you tried that worked
• What you tried that didn’t
• Finally… the insight and recommendation

It’s pretty thorough. But working hard and adding value aren’t the same thing, and “keeping it concise” is one of the cleanest ways to signal the difference.

What’s actually necessary

In most real-world situations, what you really need to land is:

• The business problem
• The outcome that worked (with the metrics that matter)
• Insights & recommendations (next steps)

That’s it (sorry, no elaborate checklist of 30 areas to consider). 90% of stakeholders only want three things:

Context. Proof. Actions.

If you give them that, you win. If you make them dig for it, you lose them.

A quick example: long-winded vs concise

Let’s say conversion dropped last week and you’ve investigated, both updates below describe the same analysis:

Long-winded update (the “no one replies” version)

Hey all — I’ve looked into the drop in conversion last week. I pulled data from GA4 and our internal events tables but there were discrepancies due to event deduping so I used internal events as source of truth. I also filtered suspected bot sessions because they were inflating sessions and skewing the funnel.

I rebuilt the funnel from landing → product view → add to basket → checkout → purchase. I calculated conversion in multiple ways (purchases/sessions, purchases/users, purchases/checkout_starts) to sense check and segmented by device type, browser, channel and new vs returning users.

The drop seems concentrated in mobile iOS Safari and starts at checkout_start where we see about a 14% drop week on week. Desktop seems stable. I checked whether this aligned with the new pricing banner test but it doesn’t fully line up.

I also investigated latency and there was a slight increase. Payment decline logs also increased but I’m not sure if it’s noise. I can run a deeper cohort analysis and set up a dashboard for ongoing monitoring…

It might be correct, but it’s work for the reader. I’ve done this many times before. It isn’t until you step away from your laptop for 10 minutes and come back to read your response that you realise you just overloaded your stakeholder with words. They had to strip out the point you’re trying to make themselves. Not good.

Concise update (Leadership Team-ready)

Conversion dropped 1.8pp last week (6.2% → 4.4%).
Where: Almost entirely caused by mobile iOS Safari. Desktop remains stable.
What changed: Checkout conversion was down 14% WoW on iOS Safari from Tuesday onwards.
Likely driver: Spike in payment declines on that route.

Next steps:

1) Engineering to validate iOS Safari payment flow changes (top priority).
2) Add alerting for checkout completion by device/browser to track changes.
3) If unresolved by end of week, mitigate by routing iOS to alternative payment options.

This does three things instantly:

• sets the scene
• proves the point
• makes the actions obvious

And if anyone wants the “how”, you’ve still got it in your back pocket, but this time — its more of a tailored response to fill in the gaps of knowledge asked by the stakeholder directly. Cool right? you just pre-empted the follow-up questions.

The trick that keeps me honest

Before I send any update now, I ask:

If they only read one line, what do they need to know?

That line goes first, everything else earns its place. As a professional in analytics, being right isn’t the job, but being understood sure as hell is.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Say Less, Deliver More was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

When generative AI entered the mainstream, it did something few technologies have managed to accomplish. Almost overnight, artificial intelligence became part of everyday business conversations. Executives who had never discussed machine learning were suddenly asking about large language models. Product teams began exploring AI assistants, customer support organizations experimented with conversational interfaces, and employees discovered that many of the tasks consuming their day could now be completed with a well-written prompt.

For many organizations, the chatbot became their first meaningful interaction with artificial intelligence. It was intuitive, approachable, and surprisingly capable. Instead of learning a new application or navigating another dashboard, people could simply ask a question in plain language and receive an answer within seconds. It felt natural, almost effortless, and for a moment it seemed as though the future of enterprise AI had finally arrived.

That excitement was both justified and necessary. Chatbots made artificial intelligence accessible in a way that predictive models and machine learning algorithms never had. They demonstrated that AI could move beyond research laboratories and become something employees could interact with every day. They helped organizations imagine possibilities that previously felt too complex or too expensive to pursue.

Yet as the excitement grew, I began noticing something that appeared in conversation after conversation. Regardless of the industry or the size of the organization, discussions about enterprise AI almost always started in the same place.

“How do we build a chatbot?”

It sounds like a perfectly reasonable question, especially given how visible conversational AI has become. But over time I started wondering whether it was also the wrong question.

The more I worked on enterprise AI initiatives, the more I realized that many organizations had unintentionally begun treating the chatbot as the destination rather than the introduction. Artificial intelligence had become synonymous with conversation, even though conversations represented only a tiny fraction of how businesses actually operate.

Most organizations don’t struggle because employees lack someone to answer questions. They struggle because information is fragmented across systems, decisions take longer than they should, repetitive tasks consume valuable time, and critical insights often arrive only after an opportunity has already passed. None of those challenges disappear simply because an employee can chat with an AI assistant.

That realization changed the way I thought about enterprise AI.

I stopped viewing conversational interfaces as the center of an AI strategy and began seeing them as just one possible entry point into something much larger. The real opportunity wasn’t teaching people to ask better questions. It was designing systems capable of answering questions before they even needed to be asked.

Think about how many decisions happen inside an organization on any given day. A purchasing manager decides whether inventory should be reordered. A finance team reconciles supplier invoices. Operations teams monitor products approaching expiration. Customer service representatives search for information spread across multiple systems before responding to a customer. Managers spend valuable time trying to understand why performance changed instead of deciding what should happen next.

These aren’t isolated problems. They are everyday business activities that quietly shape how efficiently an organization operates.

Artificial intelligence has enormous potential to improve each of these decisions, but not because it can generate fluent responses or summarize documents. Its real value lies in its ability to connect information, recognize patterns, reduce repetitive effort, and deliver insights precisely when people need them. In other words, enterprise AI becomes most valuable when it stops acting like a tool people visit and starts behaving like a capability woven into the way the organization works.

That distinction may seem subtle, but I believe it defines the next phase of enterprise AI.

For the past two years, we’ve been fascinated by what AI can say.

The next challenge is discovering what AI can quietly help organizations do.

If there’s one lesson I’ve learned from working on enterprise AI initiatives, it’s that technology rarely determines whether a project succeeds. That may sound surprising in a field obsessed with model performance, benchmark scores, and the latest breakthroughs, but the longer I spend designing enterprise AI solutions, the more convinced I become that organizations rarely struggle because they chose the wrong AI model. They struggle because they try to fit AI into processes that were already broken.

I’ve seen organizations spend months evaluating foundation models while paying very little attention to the quality of the data those models would eventually consume. Teams debated whether one language model reasoned better than another, yet no one could confidently explain where critical business information lived or whether employees trusted the data they were using every day. It reminded me that artificial intelligence doesn’t replace the fundamentals of good system design. If anything, it makes them even more important.

Enterprise AI isn’t simply another application that sits alongside existing business systems. It becomes part of the way decisions are made. When an AI system recommends a purchasing decision, flags an operational risk, summarizes financial information, or identifies an anomaly, people naturally begin relying on those recommendations. At that moment, the conversation is no longer about artificial intelligence. It’s about trust.

Trust is difficult to measure, yet it determines almost everything.

Employees don’t adopt intelligent systems because they are technically impressive. They adopt them because they consistently produce reliable outcomes. Managers don’t change the way they make decisions because an algorithm is sophisticated. They change because the system proves itself over time by helping them make better decisions with greater confidence.

That is why I believe the future of enterprise AI has less to do with creating more capable conversations and more to do with creating more dependable systems.

A well-designed enterprise AI platform should feel less like asking questions to a chatbot and more like working within an organization that simply functions better. Information should appear when it’s needed instead of requiring someone to search for it. Routine tasks should happen in the background without drawing attention to themselves. Business users shouldn’t have to understand prompt engineering to benefit from artificial intelligence. Ideally, they shouldn’t have to think about AI at all.

The most successful technology often becomes invisible. We don’t admire the database every time a transaction is completed, and we rarely think about the cloud infrastructure supporting the applications we use every day. Artificial intelligence will eventually follow the same path. As organizations mature, AI will become another capability quietly embedded within business operations, supporting people without demanding their attention.

That shift also changes the role of architects, engineers, and technology leaders. Our responsibility is no longer limited to implementing intelligent models. We are designing systems that influence how people work, how organizations make decisions, and how businesses create value. Those responsibilities extend far beyond software development. They require us to understand operations, governance, collaboration, and perhaps most importantly, the people who will ultimately rely on the technology.

When viewed from that perspective, enterprise AI stops being a technology initiative. It becomes an organizational capability — one that grows stronger as data improves, processes evolve, and people gain confidence in the systems supporting them.

Artificial intelligence will continue evolving at an extraordinary pace. The models available a year from now will almost certainly outperform the ones we use today, and new capabilities will continue to reshape what organizations believe is possible. That progress is exciting, but it can also become distracting. It’s easy to focus on the next breakthrough while overlooking the foundations that allow innovation to create lasting value.

I don’t believe the organizations leading the next generation of enterprise AI will necessarily be those that adopt every new model first. They will be the ones that understand their own business well enough to know where intelligence truly belongs. They’ll invest in trusted data before chasing benchmarks, design workflows before designing prompts, and view AI as a capability that strengthens people rather than replaces them.

Chatbots have undoubtedly changed the way we interact with technology, and they deserve their place in the evolution of artificial intelligence. But they are only the beginning of a much larger story.

The real transformation happens when intelligence becomes woven into the fabric of an organization — supporting decisions, simplifying complexity, and helping people solve problems without requiring constant attention to the technology itself.

Perhaps that’s the future enterprise AI has been moving toward all along.

Not a future where people spend more time talking to artificial intelligence.

But one where they spend less time thinking about it because it’s already helping them do their work better.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Beyond Chatbots: Designing Enterprise AI Systems That Deliver Business Value was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article explains why that happens and describes the different ways of handling it.

Fact tables and dimension tables

In a data warehouse, data is split into fact tables and dimension tables as a deliberate design choice driven by scale. A fact table can grow to billions of rows, one for every event a business has ever recorded, while the customers or products those events refer to are far fewer and rarely change.

Storing a customer’s full details on every one of their transactions would create two costs: it would multiply storage many times over, and a single change to one detail would mean rewriting every transaction row in which that customer appears. Splitting the two into separate tables avoids both.

Fact tables record events: a sale, a click, a payment. Each happened at a specific moment and is not expected to change once written. A sales fact table might look like this:

Dimension tables record descriptions of the things a fact table refers to: who a customer is, where they live, what category a product belongs to. A simple customers dimension might look like this:

The two tables are connected by customer_id. To analyse sales by city, a query joins the two tables: for every row in the fact table, it takes the customer_id and looks up the matching row in the dimension table to get the city.

The cost of overwriting a dimension

Joining the fact table to the dimension table by customer_id returns the correct city for an old sale only if the customer has not moved since the sale happened. Suppose the customer relocates to New York City, and the dimension table is updated to match:

The join now returns New York City for every one of that customer’s sales, including the sale from 21 June 2025, made while they still lived in San Francisco. A sales-by-city report run before the move correctly showed that sale under San Francisco. The same report, run again now, shows it under New York City instead, even though nothing about the sale itself changed; only the recorded address did.

The underlying issue is that a dimension table built this way can only ever reflect what is currently true about a customer. A query asking what was true in the past gets today’s value instead, because the table no longer holds any other value.

This is one of the oldest problems in data warehousing: how much history to keep for a value that changes, and in what form. The problem has a standard name: a Slowly Changing Dimension, or SCD. A customer’s city is one example of a slowly changing attribute; the same problem applies to a product’s category, a company’s sector, an employee’s department, or any other description that is not fixed forever.

The term comes from Ralph Kimball’s work on dimensional modelling in the 1990s, long before cloud warehouses or lakehouses existed. It has nothing to do with any particular platform: the same eight types apply whether the table lives in Oracle, SQL Server, Snowflake, Databricks, or a spreadsheet.

Consequences for data engineering, analytics, and machine learning

The join described above matters well beyond a single report. Every downstream dashboard, business intelligence (BI) query, and machine learning feature pipeline relies on that same mechanism to look up a descriptive value, such as a city or a name, for a raw ID. How the dimension table handles change determines what every one of those downstream layers can know about the past.

Data engineers make that decision first, often without realising how far it reaches. If the choice is to overwrite, as in the example above, an updated value replaces the old one immediately, at the point of ingestion. From then on, no downstream query can recover what was lost, no matter how it is written.

Analysts and BI teams encounter the same problem through that join: a quarter that previously reported $50,000 in San Francisco sales now reports $30,000, because the join now attributes those same historical sales to a different city.

The same join affects machine learning systems too, but as a feature problem rather than a reporting one. A model trained to predict customer churn needs the join to return what was true about a customer at the time of each past event, not what is true today. If the dimension only holds the current value, the join feeds the model facts that did not exist yet, a problem usually known as data leakage.

There is no single right pattern: the choice depends on which of these failures a business can tolerate. That is a business decision about what the data needs to answer, not a fixed technical rule.

Type 0: Retain original (rarely needed)

The simplest option: the value never changes once it has been written.

Best for: attributes that genuinely should not change after the fact, such as a customer’s original signup date or the marketing channel through which they were first acquired.

Trade-off: if the value turns out not to be truly fixed after all, there is no built-in way to correct it. Most attributes do not fit this pattern, which is why Type 0 is rarely the right choice.

Type 1: Overwrite (no history kept)

This dimension table, shown earlier, has a single row per entity, updated in place, with no history kept. Few teams choose this pattern deliberately; it is simply what happens by default when no extra mechanism is built to preserve history, since each update then has only one existing row to overwrite.

Best for: corrections with no future analytical value, such as a corrected typo, a corrected phone number, or a corrected spelling. Only the current truth matters; the value it held before is irrelevant.

Trade-off: old reports go stale without warning. A report that joined this dimension last year will give a different answer if it runs again today, and there is no way to recover what the data showed at the time.

Type 2: Add a new row (the standard choice)

Instead of overwriting, Type 2 inserts a new row and marks it as current. This is the most common pattern in real data warehouses, and the one most worth understanding. It also involves the greatest implementation complexity: surrogate keys, effective dates, and a way to atomically close out the old row while inserting the new one.

A transaction recorded before the move now joins to the row that was true at that time (San Francisco), while today’s reports use the current row (New York City). The customer_id no longer uniquely identifies one row. The table needs a separate surrogate key for that, plus the date range or the is_current flag, to select the correct version.

In practice, a MERGE step populates this table each time new data arrives:

Customer A’s move follows the rightmost path: the San Francisco row is flagged inactive, and the New York City row is inserted as current. A customer who has not moved follows the left branch instead: no change, no new row.

Best for: point-in-time accuracy and audits, where knowing what was true at a specific moment matters as much as knowing what is true now. It comes up most often in financial services and insurance, where regulators expect a record of what was true when a decision was made; in healthcare, where a patient’s coverage or care details must reflect the state at each visit rather than today; and in retail, where attributing a past sale to the correct price or customer segment depends on knowing what was true when the sale happened. An internal directory or a marketing contact list rarely needs this level of detail; the extra complexity is not justified, and Type 1 is enough.

Trade-off: more keys, more dates, more deliberate joins, and more storage, since a new row is added every time the value changes. Every dimension table needs surrogate keys, date ranges, and a rule for which row applies to which fact.

Type 3: Add a column (one step of history)

A lighter alternative: Type 3 keeps one previous value alongside the current one, in the same row.

Best for: cases where only a before-and-after comparison is needed, not the full chain of every previous change, such as showing a product’s previous price next to its current one.

Trade-off: only one step of history survives. A second change erases the original value the same way Type 1 does.

Type 4: A separate history table (for read-heavy current tables)

Type 4 splits the table in two: a ‘current’ table with one row per entity, and a separate history table holding every past version.

Current table:

History table:

The current table always holds exactly one row per customer and answers almost every query instantly. The history table grows with every change and is only queried when someone specifically needs to know what was true in the past.

Best for: systems where queries access the current table far more often than the history table, such as a customer-profile lookup that almost always needs only the current address, with the full change history queried only occasionally for support or audit purposes. The common case stays fast without paying the cost of scanning years of changes.

Trade-off: two tables to maintain instead of one, and any query that needs both current and historical data has to look in two places instead of one.

Type 6: A hybrid of full history and a current column (most complete, most complex)

Type 6 combines Types 1, 2, and 3: it overwrites a current-value column the way Type 1 does, keeps full history the way Type 2 does, and exposes that current column alongside the history in the same row the way Type 3 does.

The current_city column holds New York City in both rows: every time the customer's city changes, that column is overwritten across all of their rows, past and present. So querying the 2019 row alone shows both what was true then (San Francisco, in city) and what is true now (New York City, in current_city), with no join required.

Best for: cases where both the current value and the full history are needed often enough that paying for an extra join every time is not worth it. It shows up most often in subscription billing, where a customer’s current plan tier is checked on every request for feature access while the full history of plan changes is needed to calculate prorated charges each billing cycle, and in loyalty programmes, where a member’s current tier drives an instant discount on every transaction while the full tier history is kept for eligibility checks and dispute resolution.

Trade-off: the most storage and the most upkeep of any type, since every change has to update both the history and the current-value column, and keep them in sync.

Types 5 and 7: rare in practice

Two further types exist, though rare enough that they are only sketched briefly here.

Type 5 solves a narrower problem: a few attributes, such as a customer’s income band or household size, change far more often than the rest of the customer dimension. Applying Type 2 directly would mean inserting a new row, repeating the customer’s name and every other stable attribute, each time just one of those fast-changing fields shifts.

Type 5 avoids that by splitting the fast-changing attributes into their own, much smaller table: a mini-dimension. The mini-dimension holds one row per distinct combination of those attributes, not one row per customer, so many customers share the same row.

Main customer dimension:

Demographic mini-dimension:

demo_key is a foreign key, just like customer_id connects a fact table to a dimension table. When the customer's income band changes, only demo_key in the main dimension is overwritten, to point at whichever mini-dimension row matches the new combination; the main dimension itself never grows. The pattern is most common in retail loyalty programmes, where income band, household size, or activity tier are resurveyed often, while a customer's name and signup date are not.

Type 7 keeps both a durable key and a surrogate key in the same table: customer_id stays the same for that customer forever (the durable key), while customer_sk is a new surrogate key generated for every new row, the same kind of surrogate key Type 2 uses to tell historical versions apart.

A fact table that stores customer_sk always joins to the row that was true when the event happened: the point-in-time view. A fact table that stores customer_id instead joins to today's row regardless of when the event happened: the current view. This shows up in sales compensation systems, where commission is calculated against the organisational structure at the time of the sale, while executive dashboards roll the same sales up to today's structure.

Weighing trade-offs at a glance

The table below compares the six most common types; Types 5 and 7, being rarer, are not included. A quicker guide for the three most common choices:

• Does the old value matter for any future analysis? If not, Type 1 is sufficient.
• Does accurate point-in-time reporting require the full history of every change? Use Type 2.
• Does the use case need only a comparison between the immediately previous value and the current one? Type 3 is enough, and simpler.

The trade-offs side by side:

Most real warehouses use a mix: Type 1 for fields like spelling corrections, Type 2 for anything tied to financial or compliance reporting, where what the data showed at the time still matters.

Further reading

Ralph Kimball and Margy Ross, The Data Warehouse Toolkit (Wiley), the original source of the SCD terminology and numbering used throughout this article.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Slowly Changing Dimensions: Tracking Historical Data was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most financial institutions have run market analytics in SAS for decades. The results are correct, but the infrastructure is a single server, the code mixes data access and analytics logic in the same script.

The modern alternative is a data lakehouse on Databricks: a platform that combines Delta Lake storage, distributed compute with PySpark, and built-in version history for audit. The question is what the migration actually looks like, end to end.

Three things drive this at enterprise scale. At the compute level, SAS runs on a single server — as data volumes grow, the only option is a larger machine. PySpark distributes across a cluster; the same logic scales without rewriting. At the governance level, every write to a Delta table is a new version, so a regulator asking for the state of any table on a specific date gets a query rather than a search through archived flat files. At the team level, Python and PySpark skills are widely available in a way that makes teams easier to build and sustain.

This project is a personal exploration of that migration, built outside work on Databricks Free Edition (the free tier, without Unity Catalog, Delta Live Tables, or Auto Loader). It runs on real financial data from 2010 to the present, with a tested pipeline and a live Streamlit dashboard showing the Gold layer output. The aim is to show what the patterns look like in practice, not to replicate a production deployment.

What the project covers

Two things, which represent the two sides of any migration.

The first is the infrastructure: a three-layer Delta Lake pipeline (Bronze, Silver, Gold) processing five financial data series: S&P 500, Euro Stoxx 50, VIX (a measure of expected market volatility), US Federal Funds Rate, and ECB (European Central Bank) Deposit Facility Rate. The Gold layer is the consumption-ready layer where final business rules and aggregations are applied: rolling volatility, cross-market correlations, drawdown, and rate regime classifications for both central banks — served through an interactive Streamlit dashboard.

The second is the code: a SAS → PySpark converter that translates legacy SAS scripts to PySpark automatically. Common patterns are handled by a rule engine; anything outside those patterns is sent to an LLM (large language model). The converter is covered in the article: Automating SAS-to-PySpark migration, but it is part of the same project: one cannot move the infrastructure and leave the code behind.

The three-layer pipeline

The medallion architecture separates data into three layers, each with a distinct role. That separation reflects what happens to data as it moves from raw source to analytics-ready output.

Bronze is raw ingestion. Two sources, five tables, no transformation:

• Yahoo Finance (daily): S&P 500 closing prices, Euro Stoxx 50 closing prices, VIX
• FRED (Federal Reserve Economic Data, monthly): US Federal Funds Rate, ECB Deposit Facility Rate

Everything is written to Delta tables exactly as received; Bronze is the permanent record. If a cleaning step in Silver changes later, one can reprocess the full history without going back to the original APIs.

Silver is the data preparation layer. It cleans, joins, and validates before anything downstream uses it. A quality check module runs at every layer transition: row count, null, range, and duplicate checks; if any fail, the pipeline halts rather than passing bad data downstream.

After Silver, the pipeline has a single clean daily table: one row per trading day, all five series aligned, no nulls, log returns computed, quality checks passed.

Gold is the consumption-ready layer where final business rules and aggregations are applied for project-specific use cases. As Databricks defines it, this is the layer where data is organised specifically for how it will be consumed (Databricks: Medallion Architecture). This project’s Gold layer computes rolling realised volatility at 20-day and 60-day windows (annualised), 60-day rolling correlation between US and EU equity returns, S&P 500 drawdown from its 52-week high, rate regime classifications for both central banks, and VIX regime labels. A financial analyst opening the dashboard gets these answers directly, with no further calculation needed.

Each pipeline run appends a record to a pipeline_run_log Delta table: stage, table name, rows written, date range, and timestamp. Every write to Gold is a new Delta version, so any prior state can be queried with VERSION AS OF, which is what a regulatory audit trail requires.

The companion article covers each layer in detail for an example data migration project.

Testing and validation

The converter is tested with 39 automated test scripts covering every supported SAS construct — sort, aggregation, SQL, conditional logic, date functions, and constructs like RETAIN that cannot be translated mechanically and trigger a warning instead — as well as confidence scoring, config-driven library and macro variable resolution, multi-block script conversion, and migration manifest generation. A further 7 cases cover the pipeline submission script — credential loading, correct exit codes on success and failure, and job status handling. The full suite runs automatically on every push to main via GitHub Actions.

The above is generated by running pytest tests/ -v in the terminal from the project root.

Quality checks at every layer transition validate the actual data on Databricks. If any check fails, the pipeline halts before bad data reaches the next layer. This is not something one gets from a traditional SAS setup, where ingestion, cleaning, joining, and analytics typically live in a single script. The medallion architecture makes each transformation an independently testable unit.

What this looks like in production

This project runs on the Free Edition. A production deployment on a full Databricks workspace would add:

The code is written to be Unity Catalog-ready: adding catalog prefixes to table names is the only change needed when moving to a full workspace. The patterns are the same; the infrastructure around them is more robust.

The most significant code-level change in production is Delta Live Tables. Currently, each notebook writes tables explicitly and calls quality check functions. In a full workspace, both collapse into declarative table definitions with @dlt.expect decorators. Less repetitive code, same logic.

The dashboard

The live dashboard is the working end of the pipeline: 15 years of US and European market data, interactive and filterable by period, reading live from Databricks with no local data files or intermediate exports.

The Analytics page shows 15 years of real US and European market data: how the two markets moved together during crises, the ECB’s eight years of negative rates (2014–2022), the fastest rate rises in four decades in 2022, and how quickly both markets recovered after COVID. Interactive date filters can isolate any period.

The Converter page takes SAS code and returns PySpark.

Built with Claude Code

The project was built with Claude Code (Anthropic’s AI coding assistant) as a development collaborator: architecture decisions, the quality checks module, the converter’s rule engine design, and iterative review across all three pipeline layers. The converter uses the same Claude API as its fallback for unrecognised SAS patterns.

Code

Full project on GitHub: Financial Analytics Pipeline on Databricks

Live Streamlit dashboard connected to Databricks

Full project write-up: Financial Analytics Pipeline on Databricks

For the medallion pipeline in detail: A SAS Migration on Databricks: A Hands-On Project

Part 2 — SAS-to-PySpark converter: Automating SAS-to-PySpark Migration: Rule Engine First, AI for the Rest

Built with Databricks Free Edition, PySpark, Delta Lake, Streamlit, and Claude Code. Rate data from FRED (Federal Reserve Economic Data). Equity data from Yahoo Finance.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Migrating Financial Analytics to a Lakehouse on Databricks: A Working Demo was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

What happened?

Mine didn’t. It asked something different…

Can reported revenue be fully attributed to visible transaction components, and if not, what operational control weaknesses are reducing the reliability of transaction-level reporting?

That single question completely changed how I approached this project. Instead of building another sales dashboard, or doing another menu engineering or customer behaviour analysis, I treated a restaurant point-of-sale (POS) dataset as if I were conducting an internal revenue assurance audit.

The result wasn’t just another SQL project. It became an exercise in commercial thinking.

Every SQL project starts with data. Mine started with a business question…

When I first explored the dataset, it would have been easy to analyse revenue trends or identify the best-selling menu items. Those questions are valuable but, they weren’t the most interesting questions.

Instead, I asked:

Could reported revenue be fully reconciled to visible transaction components?

If the answer was no… Why?

That changed everything.

Instead of organising the analysis around tables and joins, I organised it around an audit investigation.

The workflow became:

• Data preparation and validation
• Revenue reconciliation
• Exception analysis
• Root cause assessment
• Management recommendations
• KPI development

Every SQL query had a business purpose.

One lesson surprised me…

The audit identified:

• 494 revenue attribution exceptions
• $340.10 of unattributed revenue within the audit sample

At first glance those two numbers tell very different stories. Almost five hundred exceptions sound serious, whilst three hundred and forty dollars doesn’t. That’s when I realised something. Commercial analysts shouldn’t confuse frequency with financial exposure as those are different dimensions of risk.

That distinction changed the way I interpreted every subsequent finding.

The project wasn’t really about revenue

This might sound strange. Yes, the project identified revenue attribution exceptions. But I don’t think revenue was the most important outcome. The real outcome was confidence. Or more accurately… Confidence in management information.

The analysis identified:

• missing transaction attributes
• inconsistent reporting labels
• incomplete transactions
• a confirmed pricing configuration defect

None of those automatically indicate “lost revenue”. What they indicate is reduced confidence in transaction-level reporting. That’s a different conversation. And, arguably, a more important one.

The “So What?” Test

Every analysis eventually reaches the same point. Someone asks:

So what?

If the answer is simply:

“Interesting.”

The analysis probably isn’t finished. The recommendations should change something.

In this case they focused on:

• strengthening transaction capture
• correcting confirmed configuration issues
• standardising reporting logic
• implementing KPI-based monitoring

Notice what’s missing?

I never claimed these actions would increase revenue because I couldn’t prove that. Instead, the recommendations improve the quality of the information management relies on. That’s what the evidence supported.

What this project taught me

The biggest lesson wasn’t about SQL. It was about restraint. As analysts, we’re often tempted to make bold claims. But good analysis isn’t about saying the most interesting thing. It’s about saying the most accurate thing.

Throughout this project I deliberately avoided describing unattributed revenue as “lost revenue” because the available evidence didn’t support that conclusion. Ironically, I think that made the project stronger.

Final thoughts

When I started this project, I wanted to improve my SQL. When I finished, I realised I’d learned something more valuable. Technical skills help you answer questions, but commercial thinking helps you ask better ones. For me, that’s the difference between writing SQL and delivering analysis that supports decision-making.

The full project is available on Maven Showcase. If you have any questions or would just like to provide feedback for improvement, please feel free to reach out to me in the comments below or feel free to reach out to me on LinkedIn.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

Submit your own writing here if you’d like to become a contributor.

Happy learning!

-Team Maven

Most SQL Projects Analyse Sales… I Audited Them Instead was originally published in Learning Data on Medium, where people are continuing the conversation by highlighting and responding to this story.