Last week, a technology leader who runs finance, HR, and payments systems for a large retailer asked me the question I keep hearing about the context layer: “What’s it going to cost me?” I didn’t have a simple answer. Neither, as far as I can tell, does anyone else yet.

I’ve heard from so many practitioners about the persistent challenge of getting AI right. One team had diligently written skills into markdown files and still could not get its agent to use it reliably for simple tasks like “prep me before this call”. Another fixed an agent that misread its churn scores and still had no production accuracy number to show the fix held.

The bottleneck to enterprise AI has moved from the model (2024) to the harness (2025) to, finally, the context. OpenAI, AWS, Google, Snowflake, Databricks, LangGraph, and a dozen others have put this word at the center of their AI story this year. Yet, despite this, no one agrees about what it means. For some, context means knowledge. For others, it means memory, or skills, or tools and permissions, or the instructions and data model an agent reasons over.

We won’t settle on one definition before October 28, and honestly, I don’t think we need to. What we need are answers to a few practical questions that real practitioners care about, in roughly the same words, regardless of industry.

These are the five questions I most want us to answer at the upcoming Context Conference on October 28th.

1. How do I convince my CFO to invest in a context layer?

AI is the top priority in most boardrooms right now, but it is still barely understood in most parts of the business. What I hear in closed rooms from technologists, architects, and data leaders is some version of the same sentence: “My business does not understand that consistent, reliable AI needs solid foundations, and investing in foundations takes time, money and patience. Heck, they don’t even know the difference between a prompt, a skill, and an MCP server. How do I make a business case?”

The evidence that getting the foundation for AI right pays is getting harder to ignore. A Gartner survey of 353 data and AI leaders, published in April, found that organizations with successful AI initiatives invest up to four times more, as a share of revenue, in data and context foundations than those with poor outcomes, and that the most mature see up to 65% greater business outcomes.

A survey number does not get a budget approved though. The CFO will want a business case. Maybe that’s a strategic case: context is the company’s intellectual property, and it should be owned and portable rather than rebuilt inside every new tool. And maybe a P&L case: lower token consumption, better accuracy and ROI across the other AI projects that depend on the same context, the direct cost of running the system at scale, and how long it takes to get an AI project live.

That is the question the executive session at 11:20 pm ET puts to Cindy Hoots, Leigh-Ann Russell, and Prakash Kota, who have run technology as CIOs at AstraZeneca, BNY, and UKG: why invest in context at all, and how do you build the business case?

2. How do I model context to create reliable AI outcomes?

Most of the companies I talk to are stuck one step earlier, at bootstrapping. How do I get context in at all? Where does it come from, and who writes it? Then they get their first few agents running on it, and all hell breaks loose. The problems that come after bootstrapping get far less attention, and in my experience they are harder.

The clearest example I have seen came from a team at a large consumer internet company. They had already done the work: written definitions, collected examples, specified output formats, and built skills. The context existed. It just lived in six Markdown files.

Yet the agent still would not use it reliably and consistently. “So they tried making the context update itself from usage, and that made things worse: “The context keeps updating. It was going really crazy. So we introduced a human in the loop.”

Enterprise AI requires reliable, consistent execution. We can’t just set up an agent to drive a business process that’s right 97% of the time. I like to joke that even if the agent is wrong, in enterprise, you want it to be consistently wrong.

Every advanced team I speak to these days is learning the hard way that context is not a pile of documents. It is a system that has to help an agent retrieve the right pieces, combine them, decide which one wins when two conflict, and apply them at the right moment. That is at least four separate modeling problems: what context exists, how do I model it (metric, semantic view, ontology, skills), how is it retrieved (yes, every harness retrieves context differently), how is precedence decided, and how do you know an update improved the answer instead of breaking it?

The customers furthest along are already asking the next questions. How are corrections captured from traces, and how do they reach the person who owns the skill? How are context changes versioned? Can we run a regression before publishing? Can we see which agents depend on a definition before we change it? How do we roll back a change?

The goal is context that is composable, bounded, and applied the same way every time, which is a different thing from complete. Context management is a discipline, and most of us have barely started treating it as one. Context has to be managed like code: owned, versioned, reviewed, tested before release, and rolled back when it breaks.

At 11:50 pm ET, Verizon’s context architecture session by Steven Huang is where I expect this to get concrete, after which our team will do a show and tell on everything we’ve learned in the past year.

3. What will it take to finally ship an accurate data agent?

Data agents have been all the rage for the past 12 months but the general word on the street is, they don’t quite work. I’ve seen this play out firsthand, quite painstakingly, in the past 18 months. I believe we’re finally on the precipice of data and analytics agents going from the lab and into production.

Don’t believe me? In the past 9 months, 43 teams have published content about their internal analytics agents. Ian Macomber at Ramp calls it the post-AI data stack: an agent harness between the data stores and the interfaces, fed by company context and a feedback loop, with the data team’s job shifting to defining the one reality the company operates on. His one piece of advice: when models give different answers to the same question, assume the models are never wrong, but instead that your context is just underspecified. And on October 9, LinkedIn’s engineering team published a write-up of its data context layer, which tells people and agents which datasets to trust, which joins and filters are required, and what a row means. After LinkedIn’s analytics agent was connected to their context layer, the share of acceptable answers in one business-area benchmark rose from 25.9% to 79.5%. In a separate comparison, median time to an answer fell from 10.7 to 4.7 minutes.

At Atlan, we have spent this year on the same problem, on our own research bench and with the teams building alongside us. For the first time, I think we know what it takes to ship an analytics agent the business trusts.

On October 28, we will finally show it publicly: the results from our bench, how to model context so the agent actually retrieves it, and the forward-deployed playbook we use to get an analytics agent live. Then we hand the talk to the people doing this in production. Joe DosSantos from Workday, Piyush Bhargava from DocuSign, and Sowmya Parthan from Upwork will sit on the practitioner panel at 12:30 pm ET and talk about what it really takes to make an analytics agent work.

4. Who owns context quality once agents cross team boundaries?

The moment an agent serves more than one team, context stops being a modeling problem and becomes an operating-model problem.

A large healthcare enterprise told us it had no central semantic model and no control layer across the organization. Definitions were spread across platforms and teams. The hard part was not finding the conflicts. It was deciding whether two different definitions of the same term were a mistake, or whether both were valid for different uses.

A global technology company described the same problem at a much larger scale: roughly 1,200 people in one organization building agents for their own use cases. What they wanted was a place where skills could be stored, evaluated, traced, deduplicated, and assigned an owner with authority.

A third customer had already picked a model. Some teams build their own context. Others rely on a central analyst group. A team is allowed to own its context end to end if it meets the CI/CD and evaluation bar.

Enterprise AI is a multiplayer game. Context is like software dependencies. An update that marketing makes to a skill might feed dozens of downstream processes, including outbound emails that the BDRs send, messaging on the website, and product copy. What if a skill owner just updates a skill? How does it affect all the downstream tooling, especially the more autonomous tooling?

We still have to decide who owns a metric definition, who approves a shared skill, who is accountable when a local preference becomes a global instruction, who reviews a change that touches ten downstream agents, and who owns the regression when an update fixes one workflow and breaks another.

Context quality cannot be owned by nobody — but who owns it?

It’s one of the questions behind The AI Context Gap, a new study from MIT Technology Review Insights on what’s blocking AI projects, including who should own the knowledge and rules AI depends on. Everyone who attends the conference gets early access on 28th October.

We’re launching two more things at the conference: AI Context Platform for Dummies, and the forward-deployed playbook our team uses to get agents into production.

5. How do we keep context portable, and whose is it?

Portability is a question about who owns the accumulated intelligence of a business.

A large multi-product company described a situation I think many enterprises will recognize. Its internal agent platforms handled context well within their own user bases. But employees were also working across several other tools, internal and external, and the team worried that a skill handed from one platform to another arrived only partially, with the meaning diluted on the way. The context existed, but it did not survive the boundary between tools.

We saw a sharper version with a company that had moved to a cheaper cloud stack. It still needed its analytics glossary, the business terms, metrics, and narratives its analysts depended on, and it was rebuilding that glossary by hand in a Git repository. Agents could propose merge requests, but someone still had to push them to propagate the information and approve each change. As far as we could tell, none of the old context carried over on its own.

These common challenges should change how we think about a customer’s context. It should not be trapped inside whichever data platform, agent framework, or vendor interface the company happens to use this year.

The customer’s context is the customer’s intellectual property. It should move when they do. And so I would put a few questions to any vendor, including us: Can the customer export its context in a form a human can read? Can one definition be used by agents on different platforms? Does a skill keep its owner, permissions, provenance, and version history after a migration? Does the same business definition have to be maintained in five tools? What happens when the company switches its model provider?

We believe that the real hard work will be building a community and ecosystem for open context. That’s why we are so excited to have Josh Klahr from Apache Ossie (Snowflake), Prajakta Damle from Open Knowledge Format (Google), Dipti Borkar from Microsoft OneLake, and Amandeep Khurana from Anthropic on a panel led by Bob Muglia at 1:20 pm ET on how to build an open context ecosystem.

I do not expect the Context Conference to settle all five. But if one of these questions is the one keeping you up, bring it. The sessions are built around them.

Originally published on Context & Chaos platform.

Context Conference: Come for the chaos. Stay for the context.

Satya Nadella calls the loop that captures a company’s judgment “the new IP of the firm.”. Aaron Levie says context is king for AI agents. Andrej Karpathy says context engineering is the real work.

Context is turning into its own discipline, but it isn’t defined yet. And nobody should get to define it alone. Including us.

That’s why we’re inviting you to Context Conference: the community’s first ever conference on context. The biggest voices in AI, the deepest thinkers on context, and the leaders building AI in production, in one room on October 28, defining the discipline.

The Cats have already registered. Their opinions, as ever, are not always fair.

October 28 · 11 AM to 2 PM ET · Virtual

Save your spot →

The Cats of Context & Chaos

The bottleneck moved from the model (2024), to the harness (2025), to context (2026). The questions moved with it.

That’s all for this edition. Stay curious, keep exploring, and see you all in the next one!

Thanks for reading Context & Chaos! Subscribe for free to receive new posts and support our work.

About Context & Chaos

Context & Chaos isn’t just a newsletter. It’s shared community space where practitioners, builders, and thinkers come together to share stories, lessons, and ideas about what truly matters in the world of data and AI: context engineering, governance, architecture, discovery, and the human side of doing meaningful work.

Our goal is simple, to create a space that cuts through the noise and celebrates the people behind the amazing things that are happening in the data & AI domain.

Whether you’re solving messy problems, experimenting with AI, or figuring out how to make data more human, Context & Chaos is your place to learn, reflect, and connect.

Got something on your mind? We’d love to hear from you.

Satya Nadella’s article on what happens when context compounds went viral viral last week on X, and it’s easy to see why. He introduced a beautiful framework: a continuous learning loop where “human capital” and “token capital” reinforce one another, creating a compounding learning loop that will separate the organizations that thrive from those that commoditize.

It is a brilliant vision. But as those of us building the operational infrastructure for enterprise AI know, there’s a lot of work between this vision and reality. In most organizations today, the “hill-climbing machine” is stalled. The learning loop is hard to build, human capital hard to capture, and the whole system suffers from leaks.

But that doesn’t mean it isn’t worth doing. So I want to talk about what it’ll take to make Satya’s vision a reality — the architectural ins and outs of context engineering for AI.

What “human capital” actually means: the three elements of context

Satya rightly points to “human capital” as one of the core elements of his vision. It’s easy to think of examples of human capital: private evals that know what “correct” means for your business, reinforcement from real organizational traces, a knowledge base that makes institutional expertise queryable.

In just a few examples, we already have a huge range of what human capital looks like and where it lives. And there’s the problem — most enterprise AI initiatives treat “context” (i.e. human capital) as a simple documentation problem, but it’s so much more complex. After all, there’s a reason why context has been a hot topic these days.

What I call context and Satya calls human capital is actually made up of three things:

• Knowledge — the map of your business. This comprises the semantic entities, definitions, metrics, and relationships. It is understanding exactly what “ARR” means in your finance motion versus your sales motion, or what a qualified lead looks like in your ideal customer profile (ICP).
• Expertise — how work actually gets done. These are the procedures and playbooks that encode tacit knowledge, institutional intuition, and the information that lives in people’s heads but never makes it into your core software systems.
• Norms — the programmatic rules of acceptable action. Which data can this agent access? Who approves this workflow? Which customer can be offered which discount? What data can and cannot leave a specific jurisdiction?

The context layer: a unified, actionable, governed place for all human capital

All of this context can’t exist in a prompt or database. It needs to live somewhere where it can be built upon and modified and governed over time, something that agents and humans alike can draw upon to create Satya’s hill-climbing machine. That’s what I began to call the context layer a year ago, and it has three distinct and important layers:

• An AI-ready data and knowledge graph makes structured data and unstructured knowledge machine-readable for agents. Tables enriched with descriptions, join paths, SQL patterns, and usage history. Documents identified, classified, and indexed into a governed knowledge graph. Without this, agents can’t distinguish trusted sources from stale ones.
• Semantics and ontology encode the map of the business with shared definitions, metric logic, and entity relationships. This is where it gets precise. Sales and Finance often disagree on the definition of ARR. When an agent queries “ARR,” which definition does it use? In most enterprises today, the answer is whichever one it happens to retrieve first. The semantic layer makes that question have one canonical answer, and reconciles the two versions when they conflict.
• Skills encode how work gets done. A skill is the new primitive that does for procedural knowledge what code did for logic in software. Each skill is a reusable, versionable, testable unit of how-to. The context layer manages skills the way software engineering manages code: version control, testing, lifecycle management, and clear ownership. The tenth agent inherits what the first nine learned.

The “human capital to token capital” linkage that Satya describes can’t function without all three of these layers. Data without semantics is uninterpretable. Semantics without skills can describe the business, but cannot operate it. Either one, deployed without governance, becomes a mess of contradictory instructions that no enterprise can safely trust.

It’s not just documentation: What it actually takes to build human capital

The second structural bottleneck in building Satya’s learning loop is actually creating human capital or context. The default instinct is to think about this as documentation work: write down what everything means, index the documents, and connect the knowledge base. But that won’t be durable or scalable.

Human capital is dynamic. It can’t be captured passively, and it can’t be a one-time effort. The most valuable institutional knowledge isn’t written down anywhere. It’s in query logs, Slack threads where decisions got made, and the judgment calls the veteran employee makes differently from everyone else.

Here’s what it takes to actually find and document all important human capital, i.e. to populate the context layer:

• Skill-building during agent-building: Do the work manually with the agent once, then use AI to examine the session, extract the repeatable pattern, and write the skill with its triggers and edge cases. You can bootstrap an agent at 50% autonomy this way, and keep improving it toward 90%. Each task teaches the system something.
• Context Mining from systems: Read query history, detect where Sales and Finance are using two different definitions of the same metric, and surface the conflict for a human to resolve. The systems, like Slack, emails, and event logs, already contain most of what you need, if you know how to read them.
• Failure-driven capture: Don’t try to encode every procedure upfront. Run the agent, watch where it fails, and capture the missing context at the point of failure. Each failure is a signal. The library grows where agents actually work, not where someone guessed they would.
• Desktop shadowing: Shadow what top operators actually do on their machines via clicks, keystrokes, and app switches, then reverse-engineer their workflows. This captures the complex desktop processes that systems mining misses entirely.
• Structured AI interviews: The most important procedural knowledge is never logged anywhere. It’s judgment. AI can run structured conversations with employees to surface the exceptions, workarounds, and decision rules that observation can’t reach. “Walk me through the approval process. How do you actually decide?”

The context layer is assembled from a mix of approaches like these, actively running and refined over time. None of them produce finished context on their own. They produce candidates that a context development lifecycle — testing, review, approval, deployment — turns into governed, durable artifacts.

Learning leaks: The key failure point in the hill-climbing machine

The true magic of Satya’s “hill-climbing machine” lies in its compounding mechanism.

Architecturally, there are four types of memory in any agent system. Working memory and episodic memory belong close to the agent — they represent the immediate execution state. Semantic memory and procedural memory belong deep inside the context layer, representing the knowledge the organization wants to preserve, govern, and make portable across different tools.

The compounding happens when episodic experience gets promoted into shared enterprise knowledge. A repeated exception becomes an explicit policy. A resolution path that consistently works becomes a reusable skill. The system should detect the emerging pattern, propose the update, and ask the right owner to certify it. Each certification makes the layer smarter for every future agent.

The key gap in this loop is context drift.

When a CMO updates company positioning or a finance lead tweaks a metric definition, that change creates an immediate blast radius. If that update doesn’t seamlessly propagate, you get a fragmented ecosystem where the social media skill, the SDR pitch skill, and the analyst call skill are all running on different versions of reality.

The context layer can’t just store knowledge. It has to understand the blast radius of every change and route updates to the right owners at the right moment. This requires building in critical tenets like versioning, change tracking, and context propagation. Without them, the learning loop turns into a learning leak.

The ultimate test: achieving ownership and portability

Satya closed his piece with an excellent test: Can a company swap out an underlying generalist model without losing the enterprise veteran expertise it has built?

That’s a great test, but I’d add two more:

Can your AI agents agree on what “ARR” means, trace why they gave different answers last week, and get smarter from the correction, all without anyone touching a model?

Can you take all of your human capital and context from one agent platform or data ecosystem to another without losing a single day of institutional learning?

These are the real tests of enterprise AI today.

If enterprises build their human expertise into a proprietary software, custom GPT stack, or walled-garden orchestration layer, they have only traded model lock-in for platform lock-in. When the next major architectural breakthrough happens outside that ecosystem, their hard-won context will be trapped.

Satya’s warning about the instability of a frontier without an ecosystem and the danger of a few all-consuming models is profoundly right. But the solution isn’t just building local applications on top of big platforms. The solution is creating absolute portability.

Context is your core IP — the digital distillation of your human capital. The organizations that treat context as a durable asset, rather than a consumed resource or a documentation project, are the ones who will harness this powerful compounding loop and climb the hill to Satya’s vision.

Originally published on the Context & Chaos platform.

The Cats of Context & Chaos

That’s all for this edition. Stay curious, keep exploring, and see you all in the next one!

Thanks for reading Context & Chaos! Subscribe for free to receive new posts and support our work.

About Context & Chaos

Context & Chaos isn’t just a newsletter. It’s shared community space where practitioners, builders, and thinkers come together to share stories, lessons, and ideas about what truly matters in the world of data and AI: context engineering, governance, architecture, discovery, and the human side of doing meaningful work.

Our goal is simple, to create a space that cuts through the noise and celebrates the people behind the amazing things that are happening in the data & AI domain.

Whether you’re solving messy problems, experimenting with AI, or figuring out how to make data more human, Context & Chaos is your place to learn, reflect, and connect. Got something on your mind?

We’d love to hear from you.

I have had some version of the same conversation with a CIO almost every day this year. Their team has read about context layers, or seen the term in a Gartner note. They know it matters and that it sits somewhere underneath the agents they are putting into production. Then comes the real question.

• What is it, actually?
• A data catalog with new branding?
• A semantic layer?
• A knowledge graph?
• Memory, decision traces, context graphs?

All of those things matter, but none of them is the context layer. So let me break down the buzzwords and make clear what a context layer actually is, what it is not, and where it fits into the AI platform architecture.

First, what “context” means

An AI agent operating inside a real business needs three kinds of context, and they map directly onto what the layer has to encode.

• Knowledge is the map of the business: entities, definitions, metrics, relationships, glossary terms, what lets a human and an AI agree on what a customer or revenue actually mean.
• Expertise is how work actually gets done: procedures, workflows, playbooks, the know-how behind running the monthly close or triaging a support escalation, today scattered across SOPs, tickets, Slack threads, and the heads of the three people on the team who remember why we do it this way.
• Norms are the rules of acceptable action: policies, permissions, approval paths, compliance constraints. They tell the agent not just what is true and how to do it, but what is allowed: which customer gets which discount, which actions need a human approver, which data cannot leave which jurisdiction.

Hold onto those three words. In the substrate, they reappear wearing different clothes: knowledge becomes data and semantics, expertise and norms together become skills.

What an enterprise context layer actually is

An enterprise context layer is the system that turns knowledge, expertise, and norms into machine-usable context for AI, across the heterogeneous landscape of data, business systems, and AI tools. It exists so agents can operate with shared meaning, the right information at the right moment, and the governance to act safely and consistently. Think of it as a shared enterprise brain: the foundation every agent reaches into, learns from, and contributes back to. That is what makes the tenth agent dramatically better than the first, because each one inherits what the previous ones learned.

The layer has two halves: a core context substrate, the machine-usable substance of context itself in three integrated parts, and five capabilities, the operating system that produces, governs, delivers, and improves that substrate over time.

Part 1: The core context substrate

The substrate has three tightly integrated parts, each answering a question an agent needs answered before it can act. What data can I trust? That is AI-ready data and the knowledge graph. What do things mean, and how are they connected? That is semantics and ontology. How does work get done here, and what is allowed? That is skills. You cannot have one without the others: data without semantics is uninterpretable, semantics without data describes a business nobody can query, and both without skills describe how the company works without being able to operate it.

AI-ready data and the knowledge graph

This is the integrated, trusted, AI-ready representation of an enterprise’s data and knowledge assets. It makes structured data machine-readable, enriching tables with the context agents need such as descriptions, join paths, and the ways humans typically query that data, and it makes unstructured knowledge accessible in a governed way. An underrated piece of this is canonical knowledge: the narrative that defines how the company thinks, sells, builds, and speaks. Strategy documents, brand voice guides, product positioning, org charts, the ideas experienced employees carry in their heads and every new hire absorbs in their first 90 days. A layer that ignores canonical knowledge produces agents that answer questions but have no idea what the company is actually trying to do.

Semantics and ontology, the map of the business

If AI-ready data tells you what exists, semantics and ontology tell you what it means and how it connects. Semantics are the shared definitions of business concepts: glossary entries, metric definitions, the vocabulary every team uses and disagrees on, like what counts as an active customer. Ontology is the structure of how those concepts relate: Customers to Accounts to Transactions, Products to SKUs to Inventory. This is what lets agents reason across the business instead of treating each system as an island; without it, retrieval is brittle and the agent fetches strings it cannot interpret. Ontologies are not new. What is new is that AI can now build and curate this knowledge at the pace the business moves, reading query logs and reconciling conflicting documents, so a living knowledge model is finally tractable at enterprise scale.

Skills, the reusable procedures and norms

Knowing what something means is not the same as knowing what to do next. A semantic layer can tell an agent what gross margin is; it cannot tell the agent how to close the month or which approval path matters in practice. This is where skills come in. Skills make procedural knowledge, the way work gets done and the norms that constrain it, durable and machine-usable.

A skill is the new primitive that does for procedural knowledge what code did for logic in software. That comparison is worth slowing down on, because it is the whole argument. Before software had functions, logic lived as instructions you re-derived every time you needed them: copied between programs, half-remembered, subtly different in each place they ran. The function turned that into something you could name, version, test, and call from anywhere, and the moment logic became a reusable unit, software started compounding instead of being rewritten.

Procedural knowledge in the enterprise is stuck where logic was before functions. The way you actually run the monthly close, qualify a lead, or handle a refund exception lives as prompts in someone’s Notion doc, tribal memory, and steps people improvise differently each time. A skill is that knowledge made durable: a reusable, versionable, testable unit of how-to, with its own triggers, edge cases, and owner. When an organization treats skills as first-class assets rather than scattered prompts, the same thing happens that happened to software. Procedural knowledge stops being re-derived by every agent and every employee, and starts compounding. Two problems make this hard, and both are capabilities in Part 2: building the library, which is a mining problem, and maintaining and governing it as the business changes, which is a lifecycle problem. Everything else is an operating system around this primitive.

Part 2: The five capabilities

The substrate is what the context layer is. The five capabilities are what it does.

Context Mining: most business context was never written down

If you want to understand how a business thinks it runs, read its documentation. If you want to understand how it actually runs, observe its systems. Most business context is hidden across systems of record, data, knowledge, and work, and across runtime signals like query history, agent traces, and human overrides. The first job of the layer is to connect to that fragmented reality and reverse-engineer business operations from it. Mining semantics looks like this: AI reads your SQL query history, notices Sales and Finance define Annual Recurring Revenue differently, and surfaces the conflict for a human to resolve company-wide. AI does the heavy lifting; humans decide.

Mining skills is harder, because most procedural knowledge was never written down. The durable methods are system-led: extracting a skill from an agent session after the work is done manually, building process maps from event logs, capturing context at the point where an agent fails, and running structured AI interviews to surface judgment that observation cannot reach. None produces a finished skill alone. They produce candidates the next capability tests, approves, and deploys.

The development lifecycle: context needs an SDLC of its own

Software engineering created the software development lifecycle to manage how code is built, reviewed, versioned, and deployed. Companies that want a real shot at becoming AI-native need a context development lifecycle to do the same for context: created, tested, approved, deployed, retired. It can no longer live as scattered prompts and isolated team decisions; it has to become a first-class asset that is reusable, reproducible, versioned, and governable. AI handles the building, testing, and reviewing; humans own approval and deployment, deciding whether a candidate becomes canonical, where it propagates, and what it replaces.

Change propagation is the crucial part. If a company redefines its core ICP, that is a structural change to enterprise understanding, not a small edit. A positioning document might be a CMO-approved skill that feeds the social media skill, the SDR pitch skill, and the analyst call skill, so when the CMO updates it, does the change propagate automatically, queue for review by each downstream owner, or leave the old version running until someone certifies it? These are not theoretical questions. They are the difference between a layer that compounds and one that contradicts itself, and the next era of innovation will look much more like organizational design than prompt engineering.

Compounding learning loops: where the tenth agent gets smarter than the first

Memory is one of the muddiest words in the agent stack. The split that matters is architectural. Working memory, the agent’s immediate execution surface, and episodic memory, the structured record of what happened, belong close to the agent harness as an execution state. Semantic memory, the durable knowledge the system preserves, and procedural memory, the rulebook for how work gets done, belong inside the context layer, because they are knowledge and skills the organization wants to preserve, govern, and make portable.

This is where learning loops matter. Traces should not just sit as logs. Through evals, corrections, human review, and certification, temporary experience becomes a durable context: a clarification becomes a saved preference, a repeated exception becomes an explicit policy. Take a contact center agent whose customer mentions a son with a dairy allergy. In the moment it lives in working memory and becomes an episodic trace, but once the system verifies and promotes it, it becomes semantic memory in the customer profile, and future agents never rediscover it. Every interaction makes the layer smarter, and the smarter it gets, the better every future agent performs.

Activation and retrieval: one layer, many dialects

Context is only valuable if it reaches the right human or agent, in the right interface, at the right moment: copilots, search, analytics, workflows, code editors, agent frameworks. There will not be one winner. Some systems consume context through MCP, others through APIs, SQL, vector retrieval, or graph traversal, so the layer cannot be tied to one interface or assume any single standard is always the answer. In the short term that means translation: even inside the Google ecosystem, Looker wants a LookML model while Gemini Enterprise wants a skill file. The winning architecture will not force every ecosystem to speak one language. It will translate canonical context into many local dialects.

Context governance and observability: the difference between infrastructure and a data lake with ambitions

The other four capabilities make the layer functional. Governance makes it trustworthy: without it, context decays into a fog of unverified prompts, drifted definitions, and competing instructions; with it, context becomes infrastructure. Five concerns have to be live across every capability. Quality: is this definition or skill verified by an owner and tested against real cases? Drift: has the world changed underneath it? Lineage: where did it come from, and what depends on it? Versioning: can we roll back, and can we tell when two agents disagree because they are on different versions? Approval: who can merge a change that affects multiple teams, and who certifies a new playbook is safe and reusable? These are organizational design questions, and they decide whether enterprise AI becomes trustworthy at scale. Without clear accountability loops, the layer becomes another data lake, a graveyard of artifacts no one trusts. With them, it becomes the shared brain the rest of the stack reaches into with confidence.

The shape of the market today

Draw the market map of context-layer companies right now and you will find dozens of logos crammed into one category. Some belong there. Most are building something else that happens to have a context layer inside it: agent builders whose layer is scoped to their own vertical, platforms whose layer is scoped to data resident in their product, and specialists who own one component of the substrate, whether memory, process mining, vector retrieval, or semantics. Each is excellent at one piece, and most will be integrated into a context layer rather than become one.

None of this is unhealthy; every early ecosystem looks like this before it consolidates. The context layer will consolidate harder and faster than most, because the whole point of the layer is that it is shared. A Fortune 500 with one layer for its CS agents, another for analytics, a third for memory, and a fourth for process mining does not have a context layer. It has four context islands, and the compounding loop breaks the moment context cannot move between agents. The companies that define this category will integrate across all three substrates and run all five capabilities as one coherent loop.

What it is not

It helps to define the layer by contrast, because the confusion almost always comes from a neighbor concept.

How is a semantic layer different from an enterprise context layer? A semantic layer is a part of the picture, not the whole picture: it is scoped to metrics and dimensions for analytics, while the context layer covers data, semantics, and skills and runs all five capabilities.

Is an enterprise data catalog the enterprise context layer? A data catalog was built for humans, to help analysts find tables; the primary producer and consumer of a context layer is AI, which is why the substrate has to include skills and the activation layer has to speak MCP, vectors, graphs, and APIs, not just power a search box.

Is long-term memory the same as the enterprise context layer? No. Long-term memory is one piece of one capability. The context layer is the broader system that decides what gets promoted from memory into shared, governed enterprise knowledge.

Where this goes next

Here is what I keep coming back to. The companies that win the next decade will not be the ones with the best models, because everyone will have access to the same models. They will be the ones whose context compounds, where the tenth agent is dramatically smarter than the first because the layer underneath it learned something every time.

That is the whole game. The substrate and the five capabilities are how you build for it, and getting the architecture right is the difference between a layer that compounds and four context islands that quietly contradict each other.

The context layer is not a feature you ship. It is the foundation everything else stands on. Let’s build it carefully.

This is the map. Over the coming weeks, this series walks each territory in depth: building the substrate inside a company that already exists, governing the lifecycle when one change ripples across hundreds of downstream agents, and measuring whether any of it is working. The hard part was never the definition. It is the build.

The full canonical piece, with the complete architecture diagram and deeper treatment of each capability is here. This is the Context and Chaos edition.

The Cats of Context & Chaos

Thanks for reading Context & Chaos! Subscribe for free to receive new posts and support our work.

That’s all for this edition. Stay curious, keep exploring, and see you all in the next one!

About Context & Chaos

Context & Chaos isn’t just a newsletter. It’s shared community space where practitioners, builders, and thinkers come together to share stories, lessons, and ideas about what truly matters in the world of data and AI: context engineering, governance, architecture, discovery, and the human side of doing meaningful work.

Our goal is simple, to create a space that cuts through the noise and celebrates the people behind the amazing things that are happening in the data & AI domain.

Whether you’re solving messy problems, experimenting with AI, or figuring out how to make data more human, Context & Chaos is your place to learn, reflect, and connect.

Got something on your mind? We’d love to hear from you.

How to build a business case for a modern data platform?

Our traditional data catalog program failed, what should I do differently to make it succeed?

How do I evaluate a data catalog?

These are just a small sample of questions I heard in our last data leaders community dinner. The one thing learned speaking with dozens of data leaders at our last community event was that the data world is changing so quickly that many teams have to ‘‘figure it out on the go’’. But here’s the deal — there are a ton of resources, learnings, and experiences that already exist in the data community.

It’s just a matter of giving the best and brightest minds in the Atlan community a stage to share these hard-earned insights with their peers. So, that’s what we’ve done!

The Atlan team has been hard at work organizing the first community conference on modern data governance, Re:Govern! On October 5th, we’ll host half a day where an all-star lineup of teams like Autodesk, Elastic, Fox, HelloFresh, Nasdaq, and more exchange the real-life strategies and frameworks, templates, and resources that built the world’s best data governance programs.

I’ve had a chance to see some of the behind-the-scenes content as we prepare this conference for you, and I’m very impressed with the quality of insights being shared. So in this edition of the newsletter, we’re here to share a teaser for four of the amazing data teams joining us at Re:Govern.

Spotlight 1: Elastic’s Approach to Modern Data Governance Through Formalized Data Contracts

Elastic is a legendary technology company that powers search solutions, helps applications run smoothly, and protects against cyber threats for the world’s most respected organizations.

With a rich history in open source and an offering that’s built on a foundation of simplicity, speed, scale, and relevance, and powered by data, Elastic holds its Data & Analytics team to a high standard.

Elastic is distributed by nature, and its team is diverse across experience, geography, and culture, making it important that anything its data team creates and services is relevant and personalized across a spectrum of needs, expectations, and skills.

Their Data Governance strategy reflects this, guided by the principles of Transparency, helping people where data is and who owns it, Accountability, driving ownership, stewardship, and that stakeholders are accountable for the data they use, and Engagement, building a trusting, productive relationship with stakeholders.

Takashi Ueki (Director, Enterprise Data & Analytics) is joining us to share how Elastic is improving how they execute this strategy through formalized data contracts. Sign up for his session here →

Spotlight 2: Building a Business Case for Data Governance at Nasdaq

Nasdaq began its cloud journey with AWS back in 2012. As cloud adoption grew, and internal demand for data grew alongside it, Nasdaq’s data team began building their modern data stack to better meet the needs of their data consumers, implementing AWS, dbt, Monte Carlo, and Atlan over an 18-month modernization process.

Nasdaq operates over 30 exchanges across a variety of asset classes across the globe. They are home to over 4,100 publicly listed companies globally and 73% of all publicly listed tech companies in the US.

Rather than acting as a service center, Nasdaq’s central data team focuses on larger initiatives and acts as a coach and consultant to the teams they partner with. Moving away from a service-led model and into an enablement model demanded meaningful changes not just to their technology, but to team structures and operating models.

For a highly regulated organization like Nasdaq, this cultural, organizational, and technical change was no small feat! And leading the charge is Michael Weiss (Senior Product Manager) and his team, who have added Atlan to their modern data stack to drive education, collaboration, and data discovery.

Joining us at Re:Govern for this session, Michael will share not just his experience, but real-world frameworks and templates that helped him build the business case for a modern data governance practice.

Spotlight 3: Porto’s Migration from Legacy Data Catalog to Active Metadata Platform

The Data Governance team at Porto serves as somewhat of a central platform team, responsible for the myriad data ecosystems at Porto, and ensuring their data lakes and warehouses are well organized, well documented, compliant, and secure.

But the legacy data catalog that Porto once used meant that yielding the maximum possible ROI from their data stack, and achieving their data literacy goals, would be impossible.

Choosing Atlan as their new, modern data catalog, their team quickly got to work building Datapedia, a branded internal catalog whose goal is to democratize knowledge across Porto’s entire business.

Migrating all documentation and functionality from their legacy catalog into Atlan and launching to end users in just six weeks, Porto’s team has since automated the governance of over 1 million data assets, saving their 5-person team 40% of their time, and are nearly one-third of their way to 1,000 users of the tool by 2025!

Danrlei Alves is joining us at Re:Govern to share what it took to make the shift from legacy catalogs to a more modern approach to data governance with an active metadata platform. You can sign up and learn more about the session here →

Spotlight 4: Data Governance for Data Mesh Era at Autodesk

With over 14,000 employees and a revenue of $5 billion, Autodesk’s 60 data teams revolutionized data collaboration and governance by constructing a data mesh framework.

By adopting a data mesh approach, Autodesk aims to empower every team within their organization to function as data publishers, responsible for ingesting and presenting data within their specific domains. Their state-of-the-art data infrastructure comprises AWS, Snowflake, Fivetran, dbt, Atlan, Looker, and Databand, forming the foundation for their data mesh.

Autodesk’s data mesh strategy rests on Zhamak Dehghani’s four principles:

1. Decentralized Domain Ownership: Each team controlled its data.
2. Data as a Product: Data was treated like a valuable product.
3. Self-service Architecture: Teams could manage their data on their own.
4. Federated Computational Governance: There were rules in place to keep things organized.

Mark Kidwell (Chief Data Architect, Autodesk) is joining Re:Govern to share how Autodesk’s data mesh lets 60 different teams create, organize, and use their data more effectively. You can sign up for this session here →

I hope these stories have excited you enough to block your calendars on October 5th to join the community for a fun half-day filled with learning! You can check out the complete agenda and sign up here. :)

At a Glance

• A global provider of employment services sought to improve the creation and distribution of metrics definitions atop their new, modern data stack
• By adopting Atlan as their data catalog, their data team collaborated with subject matter experts to define critical KPIs and make them available to data consumers in both Atlan and Looker
• In less than two years, the organization has defined more than 400 terms for over 500 users, and has automatically mapped more than 75,000 data assets in Atlan

It’s 1993 and you’ve just graduated from college. You’re going job fair to job fair, looking through alumni directories, and constantly carrying a stack of printed resumes on high-weight, cream-colored stock. This was the reality of starting a career before the advent of web-based job boards, an innovation that changed how professionals across the globe built their careers.

With over millions of unique visitors per month to their website and hundreds of thousands of job postings, supported by thousands of employees, the organization sits atop a truly staggering amount of data. And among the key people responsible for stewarding and activating this data is their data leader.

“I feel that I’ve had a magnetic pull to data and analytics since the beginning of my career,” their leader shared. “I’ve always loved the way data can tell a story, but I also recognize how important it is that the person writing the story has the right knowledge, tools, and information to drive it.”

This leader helped to build the foundations of their modern data stack and function, and was responsible for business intelligence and data engineering, and overseeing data strategy and governance.

Summing up how important these functions are to the organization, their data leader explains, “It’s really to provide that foundational data architecture which stores the insights, the source of truth, the clean, governed, trusted data so our business consumers, our customers, and the folks planning the future of our product can access data to make decisions.”

An Event-driven Modern Data Stack

Underpinning their data team is a modern, re-architected tech stack, transformed from a fragmented ecosystem into a one data lake and warehouse. Their team ensured these new systems were structured from the ground up for the purpose of analytics, driving quality in a performant, cost-effective manner.

Much of their modern data stack is constituted of Google Cloud Platform. “We were excited about the managed technology within BigQuery,” their leader shared. “That would allow our teams to really focus on modeling and structuring the data, as opposed to some of the things prior to BigQuery that had to be managed more manually.”

In 2019, the team adopted Looker in record time. “It was a really great partnership between the business and my team. I call it a heroic effort,” their leader shared. Nearly 500 people at the organization use Looker, across a number of business functions.

“We have a huge volume of data and it’s widely recognized that the data in Looker is trusted. That’s a really different position than we were in a handful of years ago, where there may have been three or four different reports to talk about the same data, and not really an understanding of which system was meant to be the right one to go to for a certain purpose.”

The organization’s data stack benefits from an event-driven architecture. While some data pipelines operate on batch, a majority of their enterprise data flows into their data lake in real-time, enabling their data team to decide whether or not data should drive just-in-time use cases. “This makes us much more nimble as a business, and it allows us to support much higher volumes of data than what we were previously handling,” their leader explained.

Finally, to ensure the significant investment the organization has made into data technology yields as much value as possible, their data leader and her team formulated a data governance program to ensure data is fit-for-purpose and supports the broader business strategy.

“Fit-for-purpose means, for me, that the data is high-quality and trusted, that people know what it means when they look at the data,” their data leader explained. “From a more technical standpoint, that means we need to think about the ways we manage our metadata. We need to have a data catalog and business glossary, which we do in Atlan. We need to think about the right use cases for the right data sources and really empower folks to play a role in owning the data through data stewardship.”

The Need for a Data Catalog Emerges

In the midst of the organization’s re-architecture, it became clear to the data leader and her team that a data catalog was a necessary piece of the puzzle. “We were transforming our technology and our business model at the same time,” she shared. “There were a plethora of new terms to be referred to based on different systems and new ways we were doing business. To be able to socialize that knowledge and to agree on what something should be was so important.”

“For instance, there was no absolute, one way to define how many people viewed a job. It may sound silly, but do you count a view if someone scrolls past this tiny job on a webpage? Do you count a view if they only open it up and look at its fullest form? So simple debates like that were necessary to define the common KPIs that we would use to manage our business.”

In the absence of a data catalog, the organization’s data team drove definitions through consensus with business colleagues and subject matter experts, which resulted in 20 definitions stored in a separate glossary. Recognizing that definitions would need to be created and agreed upon at a far higher velocity, the data leader and her team began to actively search for a catalog to accelerate the process. This solution would host a business glossary, and serve as a mechanism for data experts to document what terms meant, break siloed knowledge, and to share it with downstream consumers.

“We needed to get to a data empowerment perspective. Some call it data ownership, where you have data stewards who take responsibility for saying, ‘This is what this data term means. I can explain it. I can share and socialize this,’” their data leader shared. “I needed a tool to do that at scale. We were talking about hundreds of, and eventually thousands of defined terms, and we were just not able to do that in the Confluence tool that we were using earlier.”

Supercharging Documentation with Atlan

After a thorough evaluation of the Data Catalog and Active Metadata Management markets, the organization chose Atlan to serve as the interaction and contextualization layer on top of their Google Cloud Platform-powered data estate.

Key to their data leader and her team’s choice was Atlan’s time to value, with their data estate mapped and visible in mere days. “It was very fast to get started. That was a significant part of why we chose Atlan. The integration with our toolset was very easy to do with a quick connection through a service account,” their data leader shared. “Atlan scans our BigQuery and scans our Looker data. So for us, there’s no entering data assets into Atlan at all. We now have 75,000-plus data assets, and that was all done through a very easy setup.”

With their data estate mapped and visible, and automated lineage making it clear how data traveled through it, the data leader and her team moved to defining the roles that they and their counterparts would play as they scaled their definitions. They quickly worked to define who would serve as data stewards, how data would be grouped and categorized in domains and structured in Atlan, and how their glossary would be structured.

Then, the data leader and a colleague took the 20 terms that were previously defined in slow-moving, large meetings, and populated them into Atlan to familiarize themselves with the tool and prepare for broader adoption. The first cohort of users would be product managers, uniquely familiar with their data, and capable of being the final arbiter of a definition.

To ensure these product managers were both capable and motivated to write these definitions, their data team partnered with Atlan’s customer success team. “There was that burden of a blank page. A tabula rasa challenge of how to get started. So we partnered with Atlan and did an adoption program for about a quarter. Atlan gave us so many great ideas and supported us with trainings, even with prizes. That was an amazing partnership,” their data leader shared.

Between clearly defined roles, the right enablement and support, and the ease of Atlan’s user interface, the pace of creating, finalizing, and communicating definitions skyrocketed. “It took us about as much time to get the original 20 definitions as it did the next 100,” their data leader shared. And after inviting more product managers to participate, the number of defined terms increased to 250.

But the data team and their colleagues weren’t finished. With product owners realizing the benefits of defining terms, and end users benefitting from fast, transparent self-service, the pace of definitions continued to accelerate.

“But the beautiful thing is that actually, we hit our stride. So with no more adoption effort, we reached around 300 terms towards the end of last year (2022), and that was just through our product owners realizing that they needed to document these things for their own benefit, for the business, my team and the data team being another force to remind them, ‘Hey. People are asking about this. Please make sure it’s there as we launch reports trying to get everything documented.’ So that got us up to around 300 terms.”

Now at 400 terms defined, the organization’s data team has built a sustainable foundation for growth, with data stewards continuously updating their assets with new context. “We went from a scenario where it was taking four to five hours to define a term we were documenting in Confluence, and it was impossible to socialize that definition properly, to having a clear, clean definition in Atlan that would be a living, breathing source of knowledge,” their data leader explained. “People can add additional information when it’s found. Now, they can add links to other pages, Jira tickets, and we can link those definitions to where they’re used.”

Extending the Reach of Definitions

Reaching as large a number of users who could benefit from definitions as possible, their data team then focused on making context available where data consumers spend the most time. Hundreds of reports are routinely created on Looker, both by the data team and data-savvy counterparts in the business. Each requires context, including the purpose of the report, who’s responsible for its creation and maintenance, its popularity, and direction on when and when not to use it.

“Atlan has been perfect for us to start storing that knowledge and making it available. When Atlan then went ahead with the Looker plugin, that has just eased the ramp-up for new users tremendously because they’re able to work and live in Looker, which is what we really want to train them on primarily,” their data leader explained.

By having terms and definitions available immediately through Atlan’s chrome plug-in on Looker, the data team drove significant value for a broad spectrum of users. New employees, and even tenured colleagues launching new products and services, have near instantaneous access to context, and can make decisions even faster, without the need to train on, then adopt Atlan’s core user interface.

“That’s the kind of usability that we want to bring to our data platform if you think about our data as a product. My team is developing the data and thinking about our internal customers as our customers. And to make their lives easier, save them time, help them accomplish their full potential when they’re looking at the data is what makes us passionate about our jobs each day.”

Supporting a Full Spectrum of Users

Crucial to the organization’s successful rollout of Atlan was their careful consideration of who would use the platform, both data consumers and data stewards.

For data consumers, their data leader’s goal was to provide a simple, but powerful self-service experience where her colleagues would be confident they’ve found the right information, and knew exactly who to speak to if they needed more information, leading to a virtuous cycle of adoption and saving time.

“They may be able to be curious and explore more on their own, and then go to the expert, rather than feel like they’re taking the time from somebody who’s busy, which we don’t want them to feel. This is the reality of moving to the self-serve dynamic. It’s so positive,” their data leader explained.

For data stewards, success on Atlan means that the data team’s colleagues who own definitions and tribal knowledge see their documentation used frequently, and are either fielding fewer, or better-informed, questions.

“Instead of the person coming for that Basic 101, they’re coming in maybe at the college level, saying, ‘Okay. I know this, but what about if I want to use it this way?’ So that’s where I talk about helping people reach their potential instead of struggling with some of the basics,” their data leader shared. “I’m often in meetings where we’ll hear something coming up in the discussion, and the Atlan link can be published, and people can benefit from the work that the stewards have done.”

Simplifying Data Discovery

A key data consumer at the organization works in Digital Marketing, and has been an early beneficiary of the data team’s good work. Among her responsibilities are press relations, using the organization’s unique position as a broker between employers and candidates to provide novel insights to secure press coverage.

“Sometimes I find data about (candidate) profiles, or job offers, and I’ll send that to create content,” the digital marketer shared. “For example, this month we created a newsletter about jobs in sports, so I found data about people who have that background.”

Prior to adopting Atlan, the process for locating and understanding the organization’s existing data and reports was difficult for the digital marketer and her colleagues. Requests for data from the press meant navigating from Looker report to Looker report to learn if the data existed at all, then escalating to a manager or Data Engineering to find an answer. “If I didn’t find it, or I didn’t know if the data existed or not, that was my first frustration. And after I asked, I’d need to wait for the answer from my coworker and I lost time,” their digital marketer explained.

But when the organization’s data team implemented Atlan, providing hundreds of definitions, mapping their data estate, and making context available where the digital marketer lives, directly in Looker, the process became far simpler.

“After two or three times (using Atlan), I found all my data and all my answers,” the digital marketer shared. “I love to use Atlan because I know where I can find this data, and in which dashboard. I don’t have time to find the category of a dashboard.”

Sharing Deep Context in Looker

Among the biggest beneficiaries of the data stewardship experience on Atlan is one of the organization’s product managers, who is responsible for the organization’s job ingestion and job processing technology, as well as external partner APIs and integrations.

“One of the things that I realized early in my career is what you call something is important and how you define it. And so, I’m a big stickler for agreeing on the terminology and what it actually means,” the product manager shared. “This is a very important aspect, especially when you’re building systems greenfield, to make sure that you align on terminology and that we’re consistent about it.”

To store and distribute these definitions, the product manager historically used Confluence, and created what she calls “bite-sized” videos to provide additional context about design, business process, and the decisions that were made for software and data. And while Confluence still has a place in her toolkit, the introduction of Atlan has extended the benefit of her thorough documentation.

The organization’s data leader introduced the product manager to Atlan after hearing she had already been creating definitions in Confluence, communicating that it was the organization’s new system of record, and the two agreed that the product manager would own all terminology for job ingestion.

The product manager got to work creating succinct, but useful definitions in Atlan, and providing a link to Confluence on each data asset. Now, data consumers can use Atlan’s chrome plug-in to find more information about the data assets the product manager is responsible for, and can find even deeper context if desired, all in Looker, natively.

“It gives you almost an index of things that are related to that term if you want to explore different aspects of it,” the product manager shared. “The ability to connect the different tools and to be able to share the information across tools is really powerful.”

“I think we’re still in the early to mid-days of getting the terminology into Atlan, but I find that now when I go into meetings and I say, ‘Okay, we’re going to define this and we’re going to put it into Atlan,’ it’s more of a status quo thing. This is part of our process and this is going to be part of our process, so let’s align on terminology.”

A Bright Future, at Enterprise Scale

Looking back on what the organization’s data team has been able to accomplish, their leader remains impressed with the scale and criticality of their implementation, “One thing that excites me about what we’re doing is it’s really at the enterprise scale. We’re not just using Atlan for one team as a pilot or for the candidate side or for the employer side. It’s everything that we do.”

Their data team continues to grow their use of the platform, serving as a catalyst that ensures that each new term and report introduced is properly defined and accessible by its subject matter expert. And as their data ecosystem continues to evolve, as systems change, and as new business lines continue to launch, the foundation their data team built using Atlan will continue to pay dividends. “We are now in a much better position to continue to help people understand what they’re really looking at when they consume the data,” their leader shared.

“I think it’s remarkable for me personally, the speed we’ve been able to work (considering) the fact we don’t have a big team dedicated to this. For everybody who’s been involved, whether they’re a steward, or a data producer, or a data consumer, or a business person just inquiring about the data that they see in reports, whatever role you’re playing. I think people can be proud that we made it this far. Not all companies can pull it off. So I’m proud of what we’ve done. We’re only scratching the surface of the value we can get out of the catalog.”

Subscribe to Metadata Weekly to stay updated about upcoming stories, case studies, and best practices from leading data teams.

At a Glance

• Austin Capital Bank, a fast-growing community bank, sought to modernize their data stack to support their evolution into a product centric and digital first bank
• In just 16 months, the bank has adopted Snowflake, dbt, Fivetran, and Thoughtspot, and Atlan as its Active Metadata Management solution of choice
• The bank is now able to launch new products at unprecedented speed and effectiveness, with Atlan supporting customer data querying for a new product with mere hours of implementation effort

A member of the Inc. 5000 list of fastest-growing private companies for seven consecutive years, Austin Capital Bank is a leader in community banking, using technology and data to fuel its nationwide growth.

“We started off several years ago with goals just like any other community bank, serving the Austin community,” shared Ian Bass, Head of Data & Analytics for Austin Capital Bank. “After a few years of being the sponsor bank for a FinTech offering, our founders decided we could build this kind of thing ourselves. So we started building our own digital banking products division, which is where Credit Strong was born.”

Credit Strong represented a new frontier for Austin Capital Bank, a fully digital offering of loans and lines of credit, offered to consumers with no credit history or poor credit to help them improve their scores without the common risk of predatory lending or credit cards with astronomical interest rates.

Transforming from a community bank, offering traditional products like checking and savings accounts with a brick-and-mortar location and online portals, into a digital-first bank with a national customer footprint, was no trivial task. Not only would it demand groundbreaking products and modern user experiences, but also the best and brightest minds in Data & Analytics, with the technology to help them launch and iterate at unprecedented speed.

A Modern Data Stack, Built in just 16 Months

“My role was to build a modern data stack for the bank and for Credit Strong. And when I came on board, the data and analytics stack was non-existent,” Ian explained.

Austin Capital Bank’s transactional data resided in AWS and PostgresSQL databases, and were not well-connected, making something as simple as querying across databases difficult to impossible. To meet the significant demands of a new digital products model, Ian and his team were tasked with a true ground-up build of a modern data stack.

“What I really wanted to build was a NextGen platform that I could use to democratize access to data and analytics,” Ian shared. “So the first step was to virtualize this into an environment where none of those barriers would exist. That was Snowflake.”

Moving from implementing Snowflake, Ian’s team sought a solution to ingest data without the need for significant data engineering resources, choosing Fivetran and launching in just one month as they began to integrate into their core technology and build models. Already existing at the bank were Tableau and PowerBI, serving as a temporary visualization and analysis layer.

Just 16 months later, Austin Capital Bank and Credit Strong would sit atop a thoroughly modern data stack, built to support digital financial products that required far more flexibility than traditional banking systems, and architected to run lean.

“When I came in, none of this existed. And today we have about six different SaaS products that comprise our data stack. We have ThoughtSpot, we use Snowflake, we use Reverse ETL tools. We have a lot of things in place that we can virtualize and handle with a two or three person team, as opposed to needing a big engineering team. We’re paying for these tools, but it’s offsetting costs in that we don’t need to hire teams to do this. We can run really lean, and we can pivot very quickly.”

Ian Bass, Head of Data & Analytics

But while tool after tool was successfully implemented, Ian and his team were spending significant effort responding to requests for data, and had yet to find a way to offer this data in a self-service manner while remaining compliant. Active Metadata Management would prove to be the missing piece.

Choosing Atlan

Inherent to Financial Services offerings is a large amount of sensitive customer data, including addresses, names, birth dates, social security numbers, and account numbers, all valuable to bad actors, and requiring careful stewardship and strict access controls.

“We wanted to make sure that we are being good stewards of our customer’s data. Part of that involves making sure that only the right people within our organization have access to it and that we have the proper controls around it, multi-factor authentication, things like that,” Ian shared.

Their first step was creating masking policies, ensuring that only authorized employees of Austin Capital Bank would be permitted to access sensitive data. Ian’s original plan was to implement masking policies in data access through Snowflake, but found that while his team could enforce these policies using the tool, they would be difficult to maintain, and to be read at-a-glance by colleagues with lower technical aptitude.

“As we thought about data governance more and more, it became clear I wanted a tool to handle that. You really need an interface built on top of Snowflake so that you can easily see who has access to what. And that’s where Atlan came into the picture. I was really taken with the whole concept of DataOps and what that meant, and what we’re driving toward is every team being a data team.”

Ian Bass, Head of Data & Analytics

With the right Active Metadata Management platform, Austin Capital Bank could finally offer self-service to its data consumers without sacrificing strong access and masking policies.

“Atlan became a necessity. That’s how we control access in an easily repeatable fashion,” Ian explained. “And then the fact that we have data lineage on top of it, and the ability to organize all of our information and classifications, and add glossary terms linked to that data? That’s all icing on the cake for us.”

Where data consumers once had to approach Ian and his team for data requests, his team now uses Atlan to build queries that enable their colleagues in teams like Operations to run reports on demand. Moreover, with masking policies in place, this data is not only available via self-service, but is governed in a manner where only authorized users can access it.

“That was a really powerful piece of what Atlan offered for us, and I’ve been very happy with how that’s progressed,” Ian shared.

Enabling Service for a New Product with only Two Hours of Effort

Among the earliest opportunities to put their new data stack into practice was a promising new product named FreeKick. “We believe it’s a first-to-market product, in that it’s designed to help build credit for minors and young adults,” Ian shared.

FreeKick allows parents to safely and reliably establish, build, and protect their children’s credit. Built on top of a digital savings account, FreeKick automatically builds credit for children aged 14 to 25 once activated. Parents simply make a deposit into their FreeKick account and activate credit building. A parent can choose to renew the service annually, or close the account and receive 100% of their deposit back. Establishing a strong credit profile for their children can save their children more than $200,000 over their lifetime.

“We think this is going to be a differentiator in the marketplace,” Ian explained. “There may be other products or options out there, like co-signing on a car note, but there’s not really any other way to do this that avoids risk. If you come in today and deposit $2,500 with our bank, it’s an FDIC-insured deposit and you’re getting a credit building product for your kids, for free. It doesn’t actually cost you any money.”

In FreeKick, a promising business model was emerging for Austin Capital Bank. But crucial to success would be a timely launch before graduation season. And given that FreeKick was a departure from their usual products, the technology Austin Capital Bank would use to service new accounts couldn’t be adapted in time for launch.

“Our internal CRM was not going to be ready because there was new infrastructure and transactional databases to understand. They hadn’t pieced it together yet,” Ian shared.

While it was possible to launch FreeKick without full technical support from their CRM, each time the client service team would need data about account holders, Ian’s team would have to support it.

“My team, and myself, personally, would have had to be on call for the client service team, and ready to run the query at the drop of a hat as soon as they were on the phone with somebody who was asking questions about their account,” Ian explained.

Ian and his team also considered a self-service model while the CRM was enhanced to support requests for client data, but found it infeasible. This approach would demand that the client service team be given access to Snowflake, with the proper roles and policies, then trained to write SQL queries against their data warehouse.

In Atlan, Ian’s team found a solution. Insights, a metadata-based query builder, would enable their client service team to run queries about customer data in a self-service manner, without the need to learn SQL.

“We used Insights to support our client service team, and Atlan made it so much easier and saved a bunch of time for me and my team. It was an easy and effective workaround, until our CRM was ready to handle the new product. So it really gave us peace of mind. It was something that took my team only a couple of hours to put together. We could not have done that without Atlan.”

Ian Bass, Head of Data & Analytics

While only a workaround while their CRM caught up to support the new service model that FreeKick demanded, Atlan Insights ensured that Ian’s team spent just two hours delivering a solution. The alternative would have been painful: Turning their data team into an on-demand service center for basic client data requests, and taking their eye off building a bright, data-driven future for the bank.

Further Enhancements to Support Growth

With the right data technology, activated and accessible using Atlan, Austin Capital Bank is in a stronger position to launch new products and services, and evolve its business model.

“We’re in the phase of figuring out if we want to incorporate these new product launches into our existing model architecture, or if we want to break it out separately,” Ian shared. “In some cases, that could make things a lot simpler for us. Having this virtualized and using dbt and Snowflake makes it simple for us to say ‘Here’s the connections we need to make, here’s all the data and models we have, and how does that fit into the ecosystem?’ And then we know what changes we need to make, and what models we need to create.”

Downstream, Austin Capital Bank’s recent adoption of ThoughtSpot is another point of optimism for Ian and his team, affording the flexibility they need to query and visualize at the speed their business is changing.

“We’re running live queries on top of ThoughtSpot day in and day out, and life is a whole lot simpler and more effective,” Ian explained. “It’s just easy to write up new queries, spin up new live boards, and pivot quickly. If someone wants to drill down on an answer, it’s very simple.”

Additionally, Ian and his team are looking forward to cataloging Thoughtspot assets in Atlan to better understand the impact of changes to their models in their visualization and analysis layer, and completing a comprehensive, single view of their data estate.

To support their marketing campaigns for new products, and to collect the necessary data to evolve these products and campaigns quickly, Ian’s team has quickly added Reverse ETL to their growing set of capabilities. “We could have hired an engineering team to set up all these API connections, but we decided to just pay a small amount of money per year and do it through a tool that I could set up myself in a matter of 30 to 60 minutes,” Ian shared.

The road ahead for Austin Capital Bank remains ambitious, with a project to build a 360-degree view of their customers, inclusive of web activity. Also crucial to support a growing number of deposit-based products will be new metrics and KPIs, supported by Ian’s team and accessible through Atlan.

Perhaps the most exciting to Austin Capital Bank is the introduction of data science and machine learning, improving the way they market and operate. “We want to get into optimization for marketing spend. We want to put fraud detection models in place. We want to make sure that we’re good stewards of our customers’ data,” Ian explained.

Ultimately, Austin Capital Bank sees Atlan as its window to the world, better understanding their fast-evolving technology and processes, and the impacts that changes to them will make.

“I want it all to be integrated, so that we’re able to assess impact,” Ian shared. If we have a change to a field in a table that’s upstream of some model, I’d want to understand where else that gets used. And I’d know if it’s in a downstream model that combines 10 tables that are combined in a view that aggregates data, or if it’s also utilized by an algorithm to optimize marketing spend.”

“If all of our infrastructure is tied into the same place, we can make changes, and before we make those changes, understand where the impact would be so that nothing breaks. All of that will need to be integrated into Atlan.”

Ian Bass, Head of Data & Analytics

A Modern Data Stack, Paying Dividends

The way Austin Capital Bank’s data team operates, thanks in part to its investment in technology and executing the right strategy, is unrecognizable from the start of their modernization journey just 16 months ago.

“With the tools we have in place now, we’re able to measure the impact of product changes right away,” Ian shared. “We can also understand how changes in our marketing strategy impact our business. For example, we did a deep dive on keyword searches that we were spending on and made a substantial change. And we’re not just measuring the effectiveness of that, but we were also able to predict what would happen. We’re following pretty closely in-line with what that prediction was.”

But beyond the obvious benefits their technology, data team, and process offer, the culture of Austin Capital Bank, from client-facing employees to executives, is rapidly changing toward being data-driven.

“I have these tools, but the President of Credit Strong and the CEO of Austin Capital Bank also have access to these tools. They can see the effectiveness of changes within hours or days of implementing. It’s something we didn’t have in place before. We don’t have to make these new constructs and connections, they’re in place already and we can pivot very quickly.”

Ian Bass, Head of Data & Analytics

Subscribe to Metadata Weekly to stay updated about upcoming stories, case studies, and best practices from leading data teams.

At a Glance

• Contentsquare, a leading digital experience platform, sought to launch a data governance program after years of significant growth
• Choosing Atlan, Contentsquare launched their program with a single source of context, transparency, and interaction across a diverse range of users
• Having successfully launched data governance, Contentsquare now benefits from a fully mapped data estate, supports crucial KPIs with single owners and shared definitions, has streamlined collaboration, and has propagated knowledge and standards of data quality across its assets

Contentsquare is a company that lives and breathes data. With over 1,000 customers and over 1,600 employees, their digital experience analytics platform provides rich context and insights into behavior, feelings, and intent at each touchpoint in a customer journey.

With $1.4B in funding, and now in its Series F round, Contentsquare’s meteoric growth has been enabled by data. And at the center of making that data accessible, understandable, and trustworthy are their Data Governance team. Composed of Kenza Zanzouri, Data Governance Strategist, and Otavio Leite Bastos, Global Data Governance Lead, Contentsquare’s team led an Atlan Masterclass to share how Data Governance is capable of accelerating critical business outcomes.

The Importance of Data Governance

Contentsquare’s standards are high. They expect data governance to act as an accelerator, not a hindrance, to their Analytics and Business Intelligence responsibilities.

“Data Governance can help us have a faster time to value,” Otavio explained. “From the time you commit to an analytics project until you deliver it, you can make it faster. If dashboards go live faster, and if analysis can be faster, then decision-making is faster. We can also promote faster onboarding for new data professionals, and speed up remediations of any data downtime that we might have.”

With ambitions as high as Contentsquare’s, their team is mindful of striking a delicate balance between being a business enabler, and building a strong analytics foundation to avoid the pitfalls that a governance function is expected to address.

Contentsquare’s team understood the potential consequences if their commitment to data governance was insufficient, or went unsponsored altogether.

“We might have two different people calculating the same metric using two different methods or formulas, leading to different decisions. We can also have data flows that are poorly known, so we don’t master our data pipelines. Being compliant with data protection guidelines and regulations would be very challenging,” Otavio shared. “And data reliability would be at risk. So you would have data consumers calculating and re-calculating metrics to compare with dashboards, not trusting the dashboards that are live.”

Data Governance at Contentsquare

Contentsquare’s data team consists of 10 people, reporting to the CIO, and segmented into three teams. Data Analytics consists of a Lead and four Analysts, Data Engineering consists of a Lead with two Engineers, and Data Governance consists of a Lead and a Strategist.

“We are the heads of KPI standards. For any formula, for any method of calculating metrics, it’s up to us to orchestrate and lead people to identify the standards for the most important metrics at the company. Second is data quality, testing data to see if it’s behaving correctly, and as expected,” Otavio shared. Also crucial for the Data Governance team is data protection, ensuring that best practices are perfectly executed, and regulations are adhered to.

Living at the center of each of the Data Governance team’s responsibilities is Atlan, advancing their charter of accessibility, understandability, and trustworthiness. “It’s our home for every KPI and dashboard that we have at the company,” Otavio explained. “Our purpose is to make data accessible. Data must be simple, understandable. And data must be trustworthy.”

Contentsquare’s Modern Data Stack

Contentsquare’s data stack includes source systems like Salesforce, Workday, and Hubspot flowing through Matillion, their integration layer. Snowflake serves as their data storage layer, with a data lake containing raw data, a data warehouse with transformed data, and data marts directly connected to Tableau and Google Data Studio for visualization and analysis. Finally, Monte Carlo serves as Contentsquare’s data quality tool.

Contextualizing and activating this data stack, and serving at the center of their data governance function, is Atlan.

“Data domain owners can both read data from Atlan and write data to Atlan. Same for KPI owners, who can read any information, but can also enrich Atlan with documentation,” Otavio explained. “We have a data catalog. We have a glossary that’s home for our most important KPIs and metrics. And we can catalog every dashboard that we have at the company, and do automatic data lineage, which is very nice.”

And to ease collaboration across a spectrum of users, Contentsquare has integrated Atlan with Slack, enabling seamless communication about their data assets.

Treating Data Output as a Data Product

Contentsquare views data governance as a way to bring clarity to a complex, difficult-to-understand landscape of assets, ownership, and context. “You might find in any company across the globe that there are dashboards, KPI owners, data fields, data health checks, data consumers, and not everything is connected. They’re flowing in an ocean, and no one understands their meaning or how they’re related,” Otavio explained.

But with a well-defined governance program, these dots are connected, expressing meaningful links between Contentsquare’s data assets, technology, and people, avoiding inefficiencies across their data operations.

The first step toward launching a governance program was to better define ownership and responsibility using a Data Product Tree, and to ruthlessly prioritize by defining the most important metrics to their business.

Underpinning the concept of data products was carefully defined domain ownership, and the recognition that treating dashboards, analysis, and reports as products required orchestration across technical functions. “The concept is quite simple. You have data products linked to use cases. Use cases are linked to some underlying data. And you must perform some data quality monitoring on each data field,” Otavio explained.

Defining a Data Product Tree

Underpinning the concept of data products was carefully defined domain ownership, and the recognition that treating dashboards, analysis, and reports as products required orchestration across technical functions. “The concept is quite simple. You have data products linked to use cases. Use cases are linked to some underlying data. And you must perform some data quality monitoring on each data field,” Otavio explained.

Prioritizing Metrics

With a framework for cross-functional ownership of data products defined, it was crucial for the team to focus on rolling out products where they would have the highest impact. For Contentsquare, that meant defining the metrics that their business demanded, defining their priority, and executing from most-to-least important.

“Our top priority was our Top KPIs at the company, which measure the health of the business around the globe for any department in the company,” Otavio shared. “That’s where we focused 60–80% of our effort, until now. Our second priority was Strategic KPIs, important at a department level. Maybe these KPIs are only important to the Sales, Marketing, or Finance departments. And the last priority was team-level metrics, maybe measuring the performance of a project in a small team in the company, these are Operational KPIs.”

Executing Data Governance and Data Quality

With technology and ownership defined, and a prioritization framework identified, Contentsquare then started a 5-step process to put their data governance and quality function into practice.

Step 1: Data Prioritization

First was formalizing data prioritization, identifying the most important metrics and use cases for enrichment. Storing this information in a profile for each critical asset in Atlan, Contentsquare’s team documented descriptions, business rules, how metrics were calculated, formulas, and formal owners.

Step 2: Data Cataloging and Data Lineage

Next, with data assets properly enriched, the team utilized Atlan’s automated lineage to map how each data asset comes into being.

“We know where data is coming from, we know where data is going to. Is this data going to new tables, new dashboards, new extractions on Tableau, for example? That’s what we can know using Atlan,” Otavio shared. “And what is very nice is that Atlan will automatically sync to our databases and to our data visualization tool. For all of our mapping, we didn’t do anything manually.”

Step 3: Data Protection

With data assets properly defined and contextualized, and with their full lineage understood, the team undertook a classification exercise, defining the sensitivity of data and specifying who would have access to it.

“What’s very interesting is that Atlan will automatically propagate these classifications to the underlying data assets,” Otavio explained. “If you have a metric, and anyone in the company can access that metric, then the data behind that asset can also be accessed by anyone. It just keeps coherence between our metrics and underlying data.”

Step 4: Data Quality

Turning to Monte Carlo, the team built out monitoring and data quality triggers. In the event of a data issue, alerts would be sent directly to Slack, where data owners could click into incidents, triage, and correct them, as needed. Further, using Atlan’s native integration with Monte Carlo, each alert creates an announcement within all affected data asset profiles in Atlan.

Step 5: Data Policies and Processes

Finally, having defined best practices, implemented a glossary and documentations, and built a framework for data quality, the team moved to create data policies in processes, starting the process of training users, and iterating to ensure their hard work would result in a persistent, improved way of working.

Building Personas

“Atlan facilitated the centralization of all critical assets and business definitions, so we could then fulfill the day-to-day challenges of different people at Contentsquare,” Kenza shared. With a diverse array of potential users across different departments, with different operational needs, the team moved their focus to improving data value streams. Referencing the Atlan Value Stream Library, a collection of best practices available to Atlan customers, Kenza and her team started the process by defining personas.

First was Sego, a Business Analyst responsible for delivering reports and dashboards that help Contentsquare’s business teams understand their performance. “She might say ‘Why is it so hard to understand what I can use and who I should reach out to?’ She needs a go-to space where data cataloging, data lineage, and dashboards are just a click away,” Kenza explained.

Next was Hiroto, a Data Engineer responsible for maintaining business intelligence data pipelines and securing data flows. The team learned that Contentsquare’s Data Engineers would benefit from metadata to help them prioritize migrations, optimize data processes, and improve their data warehouse, and automated lineage to understand the downstream impacts of their work.

Jill, a business user from Contentsquare’s Customer Success Operations team, was less technically savvy, but was data-driven in her business decisions. These business users would benefit from more context to better conceptualize the data available to them as they planned the quarters and years ahead for their departments. “All of these departments look forward to a clear way of digesting the data generated by all of our sources,” Kenza shared.

The final persona was Martin, Contentsquare’s CIO, who required full visibility on their data environment, and a view of all of their data, sourced and protected in a single location.

“As you can see, all of our personas are important to consider. But most share common difficulties. It’s why it’s important to listen to data experts and data consumers, who can build a strong data governance framework with you,” Kenza explained.

Overcoming Rollout Challenges

Even with a strategy as well-defined as Contentsquare’s, the team faced early challenges, and suggested taking time to align stakeholders, ensure strong ownership, and thoughtfully address user needs to increase adoption.

Challenge 1: Alignment

The first challenge faced by the Contentsquare team was one of alignment, referring to a single, shared sense of truth around each of their KPIs and BI deliverables, available in a single, simple-to-access location. “We went from three definitions and formulas for the same KPI, which was quite a lot, to a single definition and formula. We wanted to make sure that everyone was aligned, on the same page, on Atlan,” Kenza shared.

Challenge 2: Ownership

Second was ensuring that for each data asset and business-critical KPI, that a single owner would be identified. “Identifying the right person reduced stress, of course, but also time. It would ease adoption, and address correctly, in a timely manner, all data-related issues,” Kenza explained.

Where Contentsquare’s assets once had multiple owners, driving confusion from end users, and extra effort for domain experts, there’s now a single, empowered owner for each data asset and KPI.

Challenge 3: User Engagement

Finally, the team took a thoughtful approach to user engagement, maintaining strong individual relationships and a high standard of service, while also scaling their reach through a community approach.

“It was clear that training and building a strong community around data cataloging, helped us go from low adoption to growing adoption,” Kenza explained. To ensure that the users they needed to engage understood the importance of participating, Kenza and Otavio also enlisted the support of Contentsquare’s leadership team.

Contentsquare’s Governance Strategy in Action

With the right technology in place, a data governance and quality strategy successfully launched, and personas that ensured solutioning was valuable to their end users, Otavio shared an example of what’s possible.

Using the integration between Atlan and Monte Carlo, Contentsquare’s team successfully launched a use case to communicate data quality alerts to an array of downstream stakeholders. When an incident is identified on Monte Carlo, information is pushed into Atlan, where an alert appears in each affected data asset’s profile.

“This script will automatically detect issues, and put an announcement in Atlan telling the user to contact the data owner,” Otavio explained. “Anyone navigating to Atlan and trying to find the right dataset will know if they can use a dataset. Here, we have a complete propagation and link between the business world and the data world. We’re sending alerts to datasets on Atlan, but we’re also sending alerts to any metrics that are using a dataset as a reference to be calculated.”

For more technical users, like data owners or engineers, a simple search for a data asset in Atlan will yield a link directly into Monte Carlo, where further analysis can be conducted, like viewing past incidents, freshness, volume evolution, lineage, and schema changes.

When to Start Data Governance

Crucial to Contentsquare’s success was starting Data Governance and an opportune time. Otavio began by considering, then measuring, the size of his team and the velocity of their output, establishing a baseline.

With a small team of one to two data analysts, Contentsquare found that it would take one month to build a production dashboard. Over time, their data team grew to support the growing needs of their business.

“More people means fewer parallel tasks. So you can deliver value faster. You recruit more people, and instead of spending one month building a dashboard, maybe you spend two weeks and the dashboard will be live.”

The obvious assumption for data teams is that as headcount and resources grow, time-to-value will continue to diminish, but Contentsquare found that this was not the case. As the number of analysts grew, their solutions increased in technical depth, and a growing number of technical assets like scripts would go without updates. Their data lake grew, making it more difficult to understand what data assets were available. Processes changed at a high pace, and with new teams across the globe, new ways of working, including how metrics were calculated or data quality was measured, became misaligned.

Despite a growing headcount, and better technology, the pace of delivery slowed.

“So when is the good time to start data governance? It’s when you start realizing that you’re recruiting more data analysts and maybe even more data engineers, and it’s not solving your problems. You’re actually spending more and more time to deliver a data project, because you have so many blocks, so many pitfalls, and so many traps along the way.”

To identify the perfect time to begin a data governance program, the Contentsquare team emphasized the importance of measurement. Using a project management tool like Jira, data teams should measure the time it takes for projects to complete, from defining specifications until delivery. When time to value slows, the opportunity for data governance to accelerate outcomes is at its highest.

Key Takeaways

Looking back at the successful launch of Data Governance at Contentsquare, Kenza considers four key pieces of advice for data leaders considering a similar journey.

1: Centralize Less, Orchestrate More

“Feedback is very, very important. You need to invest time and effort into a successful framework without taking full responsibility and action. Think about engaging different stakeholders, centralizing less and orchestrating more. Define what good looks like for different personas, people, data teams, and business consumers, so we can all leverage collaboration and embrace change.”

2: Treat Deliverables as Data Products

“Reach out to different stakeholders, collect all the logical links to your data assets, and you’ll definitely optimize costs and generate value.”

3: Ensure Clear Ownership

“Establishing key stakeholders in the business is a key takeaway. They can help you build a strong, and long-term data quality and governance function.”

4: Break Down your Data Governance Program

“Data governance is new for a lot of people. You must take your time. You have to break it up into different trainings. You need to ensure what you pot in place is good for everyone.”

Subscribe to Metadata Weekly to stay updated about upcoming stories, case studies, and best practices from leading data teams.

I just need to get through this busy period, then I can focus on the important stuff… Haven’t we all thought this, only to find ourselves still mired in ad-hoc work months later? With one urgent P0 after another, it can be easy for data leaders and teams to focus on short-term deliverables and deadlines — all while missing out on the bigger picture.

From our work with hundreds of amazing data teams and as a data team ourselves, we’ve learned that defining who you are as a leader and what you want your team to work toward is a key component to success — just as important as hitting that next P0 deliverable.

We recently spoke with Gordon Wong, a consultant and founder of Wong Decision Intelligence who was formerly a senior leader of business intelligence at both Hubspot and Fitbit. He’s a hands-on, collaborative analytics leader with deep experience in building great data teams while helping businesses grow.

One of the big takeaways from our chat was the importance of forming a clear mission and vision for your team. “That seems way too simple,” he says. “But turns out, it’s really hard.” Here are some methods Gordon used to set up team missions and visions — both personal and professional — and how they helped his team perform at their best.

(This interview is from an ebook we recently published, titled The Secrets of a Modern Data Leader, featuring insights from Gordon and other amazing data leaders. Learn more and download here.)

Establish your team’s mission and vision

Let’s start with the basics — what is the difference between a team mission and vision? Your vision should contain your data team’s why, and your mission should focus on the what and how. The former should inspire, and the latter should drive action.

Whether you’re new to your role or looking to improve the layout of your existing team, getting solid, authentic definitions of your mission and vision requires lots of questions and deep listening.

“The most important trait of a leader is to establish the vision. You have to start somewhere.”

—Gordon Wong

Gordon starts by speaking with stakeholders across the organization to get a solid picture of what everyone needs. He focuses most of his early time on asking questions and adjusting based on what he learns.

While some of the questions might feel basic, they’re all about making sure everyone is on the same page. This is necessary for a functional mission. “Enterprise business and data teams are busy, and our product tends to be so abstract,” he says. Making the abstract more concrete by gathering context will help you more easily set goals.

Some questions he’s found useful at the beginning:

• What’s the common understanding of the problem?
• Who are our customers?
• What do they care about?
• How are we addressing their needs?

Once you identify the problems and the path is clear, you can form your mission and vision, and the KPIs required to make them successful.

“Because that kind of success is hard to measure, we don’t measure it,” says Gordon. “Every small shop owner, every small restaurant, they measure the business. They know the receipts for the day, they know their inventory. We should do the same.”

Define the contract

After years of experience in the data industry, something equally important to Gordon as a mission and vision is a well-defined contract. This is not the kind that outlines your salary and role expectations. It’s the contract you and your team enter into with your superiors and stakeholders.

“This is a new part for me,” says Gordon. “It’s all about deciding together what winning looks like.” Showing your boss a well-thought-out mission and vision for your team is great — getting your boss to agree to the terms of success is crucial.

• What is our agreed-upon definition of success?
• What resources are you willing to give me to support this mission?
• Do you agree to these set goalposts?

Most leaders don’t move goalposts maliciously, but because they’re ambitious. This doesn’t change the fact that quick pivots can strain your team and lower the quality of your work.

“When you have a mission, vision, and contract, you’ve incidentally outlined how to deal with disagreements.”

—Gordon Wong

Contract example:

If you give me these resources, I’ll deliver this work in this time frame.

If you don’t give me these resources, I can no longer guarantee the timeline.

“If you have a good relationship with your boss, you should be able to have that conversation,” says Gordon. “Do you acknowledge that if you change your scope or cut my resources, we’re putting quality and delivery at risk?” Risky as it might feel to set clear boundaries, remember that your role as a leader is to speak not just for yourself, but for your team.

“We’re all adults,” says Gordon. “Get this conversation out there up front.”

Determine where you are on Maslow’s pyramid

Maslow’s hierarchy of needs is a theory of motivation — everyone is motivated by something, and figuring out where you and your employees fall in the pyramid can help you best connect them to your larger goals.

“Your employees are internal stakeholders. Work with them to establish why they’re here.”

—Gordon Wong

Some common motivations Gordon has seen over the years include money, education and mentorship opportunities, community and camaraderie, and an alignment with the company’s values.

No matter your employees’ reasons for showing up every day, once they’re settled you can begin to build trust with them. “I’m not going to criticize you if you’re here to feed your family,” says Gordon. “I’ve got to jump right in — I have to feed my family as well. So I try to establish trust with you by saying, ‘Hey, I think I believe that if I am your coach, and I help us sell, and we drive business, we can help you meet those goals.’ That’s where we get that alignment.”

Communicate your personal mission and vision

Something that has gone a long way for Gordon when building relationships with new teams is being upfront about his own mission and vision, his own reasons for being there, and telling his employees the contract from day one.

He even wrote up a manual called “How to Gordon.” He shares the guide with people he works with so there is no gray area about what he promises and what drives his decisions.

“I’m a big believer in psychological safety, so really, this has nothing to do with data, but rather has to do with just leadership in general,” says Gordon. “I need to establish trust with my team, so my first decision is I start communicating about myself and my motivations: I need to get paid, I need to be learning, I need to be good for employees, for the company, for the world.”

Much like with a team’s mission and vision, a clear outline creates space for open and honest conversations. “It’s been really helpful for me,” he says.

“I try to give people a reason to trust me. Missions and visions — my own and team-based — say, ‘here’s how I think you will benefit if you are willing to put some trust in me.’”

—Gordon Wong

Gordon recently did a live Q&A about his learnings and lessons as a modern data leader. Watch the recording here.

Found this helpful? Read our latest free ebook — Secrets of a Modern Data Leader: The First 365 Days Inside a Data Team.

It’s chock-full of insights from innovative data leaders like Erica Louie, Stephen Bailey (Whatnot, Immuta), Taylor Murphy (Meltano, GitLab), and Gordon Wong (Wong Decision Intelligence, Hubspot, Fitbit).

This article was co-written with Christine Garcia (Director of Content at Atlan).

In my annual tradition, I spent some downtime at the end of 2022 reflecting on what happened in the data world. As you can tell by the length of this article, my co-author Christine and I had a lot of thoughts.

For the last few years, data has been in hypergrowth mode. The community has been stirring up controversy with hot takes, debating the latest tech, raising important conversations, and duking it out on Twitter with Friday fights. The data world seemed infinite, and everyone was just trying to keep up as it exploded.

Now data is entering a different world. 98% of CEOs expect a recession for the next 12–18 months. Companies are preparing for war by amping up the pressure, laying people off, cutting budgets, and shifting from growth mode to efficiency mode.

So what does this mean for the data world? And, more importantly, for data leaders and practitioners? This article breaks down the 10 big trends for 2023 that anyone in the data space should know about—4 emerging trends that will be a big deal, and 6 existing trends that are poised to grow even further.

With the recent economic downswing, the tech world is looking into 2023 with a new focus on efficiency and cost-cutting. This will lead to four new trends related to how modern data stack companies and data teams operate.

Storage has always been one of the biggest costs for data teams. For example, Netflix spent $9.6 million per month on AWS data storage. As companies tighten their budgets, they’ll need to take a hard look at these bills.

Snowflake and Databricks have already been investing in product optimization. They’ll likely introduce even more improvements to help customers cut costs this year.

For example, in its June conference, Snowflake highlighted product improvements to speed up queries, reduce compute time, and cut costs. It announced 10% average faster compute on AWS, 10–40% faster performance for write-heavy DML workloads, and 7–10% lower storage costs from better compression.

At its June conference, Databricks also devoted part of its keynote to cost-saving product improvements, such as the launches of Enzyme (an automatic optimizer for ETL pipelines) and Photon (a query engine with up to 12x better price to performance).

Later in the year, both Snowflake and Databricks doubled down by investing further in cost optimization features, and more are sure to come next year. Snowflake even highlighted cost-cutting as one of its top data trends for 2023 and affirmed its commitment to minimizing cost while increasing performance.

In 2023, we’ll also see the growth of tooling from independent companies and storage partners to further reduce data costs.

Dark data, or data that never actually gets used, is a serious problem for data teams. Up to 68% of data goes unused, even though companies are still paying to store it.

This year will include the growth of cost-management tools like Bluesky, CloudZero, and Slingshot designed to work with specific data storage systems like Snowflake and Databricks.

Modern data stack partners will also likely introduce compatible optimization features, like dbt’s incremental models and packages. dbt Labs and Snowflake even wrote an entire white paper together on optimizing your data with dbt and Snowflake.

Metadata also has a big role to play here. With a modern metadata platform, data teams can use popularity metrics to find unused data assets, column-level lineage to see when assets aren’t connected to pipelines, redundancy features to delete duplicate data, and more. Much of this can even be automated with active metadata, like automatically optimizing data processing or purging stale data assets.

For example, a data team we work with reduced their monthly storage costs by $50,000 just by finding and removing an unused BigQuery table. Another team deprecated 30,000 unused assets (or two-thirds of their data estate) by finding tables, views, and schemas that weren’t used upstream.

Jake Thomas on Twitter: "Data engineers should not write ETL.Instead, they should spend ~80% of their day keeping airflow running, ~10% cramming dbt manifest.json into dags, and ~10% figuring out how to not get yelled at by finance for the Snowflake bill. / Twitter"

Data engineers should not write ETL.Instead, they should spend ~80% of their day keeping airflow running, ~10% cramming dbt manifest.json into dags, and ~10% figuring out how to not get yelled at by finance for the Snowflake bill.

In the past few years, data teams have been able to run free with less regulation and oversight.

Companies have so much belief in the power and value of data that data teams haven’t always been required to prove that value. Instead, they’ve chugged along, balancing daily data work with forward-looking tech, process, and culture experiments. Optimizing how data people work has always been part of the data discussion, but it’s often relegated to more pressing concerns like building a super cool tech stack.

Next year, this will no longer cut it. As budgets tighten, data teams and their stacks will get more attention and scrutiny. How much do they cost, and how much value are they providing? Data teams will need to focus on performance and efficiency.

In 2023, companies will get more serious about measuring data ROI, and data team metrics will start becoming mainstream.

It’s not easy to measure ROI for a function as fundamental as data, but it’s more important than ever that we figure it out.

This year, data teams will start developing proxy metrics to measure their value. This may include usage metrics like data usage (e.g. DAU, WAU, MAU, and QUA), page views or time spent on data assets, and data product adoption; satisfaction metrics like a d-NPS score for data consumers; and trust metrics like data downtime and data quality scores.

Sarah Catanzaro on Twitter: "Tell me how you're USING data. I'm sick of hearing about how you're producing data or building data stacks. As one great data scientist once said, the only stacks that matter are those of benjamins. pic.twitter.com/c0Ht5mPHQb / Twitter"

Tell me how you're USING data. I'm sick of hearing about how you're producing data or building data stacks. As one great data scientist once said, the only stacks that matter are those of benjamins. pic.twitter.com/c0Ht5mPHQb

For years, the modern data stack has been growing. And growing. And growing some more.

As VCs pumped in millions of dollars in funding, new tools and categories popped up every day. But now, with the economic downturn, this growth phase is over. VC money has already been drying up — just look at the decrease in funding announcements over the last six months.

We’ll see fewer data companies and tools launching next year and slower expansion for existing companies. Ultimately, this is probably good for buyers and the modern data stack as a whole.

Yes, hypergrowth mode is fun and exciting, but it’s also chaotic. We used to joke that it would suck to be a data buyer right now, with everyone claiming to do everything. The result is some truly wild stack diagrams.

This lack of capital will force today’s data companies to focus on what matters and ignore the rest. That means fewer “nice to have” features. Fewer splashy pivots. Fewer acquisitions that make us wonder “Why did they do that?”

With limited funds, companies will have to focus on what they do best and partner with other companies for everything else, rather than trying to tackle every data problem in one platform. This will lead to the creation of the “best-in-class modern data stack” in 2023.

As the chaos calms down and data companies focus on their core USP, the winners of each category will start to become clear.

These tools will also focus on working even better with each other. They’ll act as launch partners, aligning behind common standards and pushing the modern data stack forward. (A couple of examples from last year are Fivetran’s Metadata API and dbt’s Semantic Layer, where close partners like us built integrations in advance and celebrated the launch as much as Fivetran and dbt Labs.)

These partnerships and consolidation will make it easier for buyers to choose tools and get started quickly, a welcome change from how things have been.

Seth Rosen on Twitter: "Previously: the simple "modern data stack" when first discovered by data teams was *magic*. It made it absurdly easy to quickly centralize, transform, and present data for analyticsNow: the MDS is a landscape of hundreds of logos across all of "data"RIP MDS (magic data stack) / Twitter"

Previously: the simple "modern data stack" when first discovered by data teams was *magic*. It made it absurdly easy to quickly centralize, transform, and present data for analyticsNow: the MDS is a landscape of hundreds of logos across all of "data"RIP MDS (magic data stack)

Tech companies are facing new pressure to cut costs and increase revenue in 2023. One way to do this is by focusing on their core functions, as mentioned above. Another way is seeking out new customers.

Guess what the largest untapped source of data customers is today? Enterprise companies with legacy, on-premise data systems. To serve these new customers, modern data stack companies will have to start supporting legacy tools.

In 2023, the modern data stack will start to integrate with Oracle and SAP, the two enterprise data behemoths.

This may sound controversial, but it’s already begun. The modern data stack started reaching into the on-prem, enterprise data world over a year ago.

In October 2021, Fivetran acquired HVR, an enterprise data replication tool. Fivetran said that this would allow it to “address the massive market for modernizing analytics for operational data associated with ERP systems, Oracle databases, and more”. This was the first major move from a modern data stack company into the enterprise market.

Matthew Mullins on Twitter: "Every modern data stack diagram has one dwh at the center, but every enterprise I've ever talked to has at least three. The one that's current, the one they're moving to, and the one they got in an acquisition. / Twitter"

Every modern data stack diagram has one dwh at the center, but every enterprise I've ever talked to has at least three. The one that's current, the one they're moving to, and the one they got in an acquisition.

These are six of the big ideas that blew up in the data world last year and only promise to get bigger in 2023.

This was one of the big trends from last year’s article, so it’s not surprising that it’s still a hot topic in the data world. What was surprising, though, was how fast the ideas of active metadata and third-generation data catalogs continued to grow.

In a major shift from 2021, when these ideas were new and few people were talking about them, many companies are now competing to claim the category.

Metadata is seen as one of the big gaps in the data world, so even as VC funding started to dry up, there were some big raises in the cataloging space last year. These included Alation’s $123M Series E, Data.world’s $50M Series C, our $50M Series B, and Castor’s $23.5M Series A.

The other cause for this growth was analysts, who embraced and amplified the ideas of active metadata and modern data catalogs in 2022. For example, Gartner went all in on active metadata at its annual conference and G2 released a new “Active Metadata” category.

Our prediction is that active metadata platforms will replace the “data catalog” category in 2023.

This actually started last year when Forrester renamed its Wave report on “Machine Learning Data Catalogs” and reversed its rankings. It moved the 2021 Leaders (Alation, IBM, and Collibra) to the bottom and middle tiers of its 2022 Wave report, and replaced them with a new set of companies (us, Data.world, and Informatica).

The “data catalog” is just a single use case of metadata: helping users understand their data assets. But that barely scratches the surface of what metadata can do.

Activating metadata holds the key to dozens of use cases like observability, cost management, remediation, quality, security, programmatic governance, optimized pipelines, and more — all of which are already being actively debated in the data world. Here are a few real examples:

• Eventbridge event-based actions: Allows data teams to create production-grade, event-driven metadata automations, like alerts when ownership changes or auto-tagging classifications.
• Trident AI: Uses the power of GPT-3 to automatically create descriptions and READMEs for new data assets, based on metadata from earlier assets.
• GitHub integration: Automatically creates a list of affected data assets during each GitHub pull request.

This started in August with Chad Sanderson’s newsletter on “The Rise of Data Contracts”. He later followed this up with a two-part technical guide to data contracts with Adrian Kreuziger. He then spoke about data contracts on the Analytics Engineering Podcast — with us! (Shoutout to Chad, Tristan Handy, and Julia Schottenstein for a great chat.)

The core driver of data contracts is that engineers have no incentive to create high-quality data.

Because of the modern data stack, the people who create data have been separated from the people who consume it. As a result, companies end up with GIGO data systems — garbage in, garbage out.

The data contract aims to solve this by creating an agreement between data producers and consumers. Data producers commit to producing data that adheres to certain rules — e.g. a set data schema, SLAs around accuracy or completeness, and policies on how the data can be used and changed. After agreeing on the contract, data consumers can create downstream applications with this data, assured that engineers won’t unexpectedly change the data and break live data assets.

After Chad Sanderson’s newsletter went live, this conversation blew up. It spread across Twitter and Substack, where the data community argued whether data contracts were an important conversation, frustratingly vague or self-evident, not actually a tech problem, doomed to fail, or obviously a good idea. People hosted Twitter fights, created epic threads, and watched battle royales from a safe distance, popcorn in hand.

While data contracts are an important issue in their own right, they’re part of a larger conversation about how to ensure data quality.

It’s no secret that data is often outdated or incomplete or incorrect — the data community has been talking about how to fix it for years. First, people said that metadata documentation was the solution, then it was data product shipping standards. Now the buzzword is data contracts.

This is not to dismiss data contracts, which may be the solution we’ve been waiting for. But it seems more likely that data contracts will be subsumed in a larger trend around data governance.

In 2023, data governance will start shifting “left”, and data standards will become a first-class citizen in orchestration tools.

For decades, data governance has been an afterthought. It’s often handled by data stewards, not data producers, who create documentation long after data is created.

However, we’ve recently seen a shift to move data governance “left”, or closer to data producers. This means that whoever creates the data (usually a developer or engineer) must create documentation and check the data against pre-defined standards before it can go live.

Major tools have recently made changes that support this idea, and we expect to see even more in the coming year:

• dbt’s yaml files and Semantic Layer, where analytics engineers can create READMEs and define metrics while creating a dbt model
• Airflow’s Open Lineage, which tracks metadata about jobs and datasets as DAGs execute
• Fivetran’s Metadata API, which provides metadata for data synced by Fivetran connectors
• Atlan’s GitHub extension, which creates a list of downstream assets that will be affected by a pull request

Alex Dean on Twitter: "It's technically not a Data Contract unless it comes from the Côte d'Ata region of France. Otherwise it's just a sparkling schematization. / Twitter"

It's technically not a Data Contract unless it comes from the Côte d'Ata region of France. Otherwise it's just a sparkling schematization.

Also called a “metrics layer” or “business layer”, the semantic layer is an idea that’s been floating around the data world for decades.

The semantic layer is a literal term — it’s the “layer” in a data architecture that uses “semantics” (words) that the business user will understand. Instead of raw tables with column names like “A000_CUST_ID_PROD”, data teams build a semantic layer and rename that column “Customer”. Semantic layers hide complex code from business users while keeping it well-documented and accessible for data teams.

In October 2022, dbt Labs made a big splash at its annual conference by announcing the new Semantic Layer.

The core concept behind dbt’s Semantic Layer: define things once, use them anywhere. Data producers can now define metrics in dbt, then data consumers can query those consistent metrics in downstream tools. Regardless of which BI tool they use, analysts and business users can look up a stat in the middle of a meeting, confident that their answer will be correct.

Making metrics part of data transformation intuitively makes sense. Making them part of dbt — the dominant transformation tool, which is already well-integrated with the modern data stack — is exactly what the semantic layer needed to go from idea to reality.

Since dbt’s Semantic Layer launched, progress has been fairly measured — in part because this happened less than three months ago, and in part because changing the way that people write metrics will take time.

In 2023, the first set of Semantic Layer implementations will go live.

Many data teams have spent the last couple of months exploring the impact of this new technology — experimenting with the Semantic Layer and thinking through how to change their metrics frameworks.

This process gets easier as more tools in the modern data stack integrate with the Semantic Layer. Seven tools were Semantic Layer–ready at its launch (including us, Hex, Mode, and Thoughtspot). Eight more tools were Metrics Layer–ready, an intermediate step to integrating with the Semantic Layer.

Robert Yi 🐳 on Twitter: "Okay let's do this:Who's going to win the battle for the semantic layer?(comment if someone else) / Twitter"

Okay let's do this:Who's going to win the battle for the semantic layer?(comment if someone else)

In 2022, some of the main players in reverse ETL (one of last year’s big trends) sought to redefine their category as “data activation”, a new take on the “customer data platform”.

A CDP combines data from all customer touchpoints (e.g. website, email, social media, help center, etc). A company can then segment or analyze that data, build customer profiles, and power personalized marketing. For example, they can create an automated email with a discount code if someone abandons their cart, or advertise to people who have visited a specific page on the website and used the company’s live chat.

CDPs are designed around using data, rather than simply aggregating and storing it. This is where data activation comes in — “activating” data from the warehouse to handle CDP functions.

Data activation in various forms has been around for a few years. However, this idea of data activation as the new CDP took off in 2022.

For example, Arpit Choudhury analyzed the space in April, Sarah Krasnik broke down the debate in July, Priyanka Somrah included it as a data category in August, and Luke Lin called out data activation in his 2023 data predictions last month.

In part, this trend was caused by marketing from former reverse ETL companies, who now brand themselves as data activation products. For example, Hightouch rebranded itself with a big splash in April, dropping three blogs on data activation in five days:

• Data Activation: The Next Step After Analytics by Pedram Navid
• Hightouch: The Data Activation Platform by Kashish Gupta
• What is Data Activation? by Luke Kline

In part, this can also be traced to the larger debate around driving data use cases and value, rather than focusing on data infrastructure or stacks. As Benn Stancil put it, “Why has data technology advanced so much further than value a data team provides?”

In part, this was also an inevitable result of the modern data stack. Stacks like Snowflake + Hightouch have the same data and functionality as a CDP, but they can be used across a company rather than for only one function.

CDPs made sense in the past. When it was difficult to stand up a data platform, having an out-of-the-box, perfectly customized customer data platform for business users was a big win.

Now, though, the world has changed, and companies can set up a data platform in under 30 minutes — one that not only has customer data, but also all other important company data (e.g. finance, product/users, partners, etc).

At the same time, data work has been consolidating around the modern data stack. Salesforce once tried to handle its own analytics (called Einstein Analytics). Now it has partnered with Snowflake, and Salesforce data can be piped into Snowflake just like any other data source.

The same thing has happened for most SaaS products. While internal analytics was once their upsell, they are now realizing that it makes more sense to move their data into the existing modern data ecosystem. Instead, their upsell is now syncing data to warehouses via APIs.

In this new world, data activation becomes very powerful. The modern data warehouse plus data activation will replace not only the CDP, but also all pre-built, specialized SaaS data platforms.

With the modern data stack, data is now created in specialized SaaS products and piped into storage systems like Snowflake, where it is combined with other data and transformed in the API layer. Data activation is then crucial for piping insights back into the source SaaS systems where business users do their daily work.

For example, Snowflake acquired Streamlit, which allows people to create pre-built templates and templates on top of Snowflake. Rather than developing their own analytics or relying on CDPs, tools like Salesforce can now let their customers sync data to Snowflake and use a pre-built Salesforce app to analyze the data or do custom actions (like cleaning a lead list with Clearbit) with one click. The result is the customization and user-friendliness of a CDP, combined with the power of modern cloud compute.

Jessica Laughlin on Twitter: "looking for funding for my new startup idea: Reverse Data Activation https://t.co/VLbUhYx8Yp / Twitter"

looking for funding for my new startup idea: Reverse Data Activation https://t.co/VLbUhYx8Yp

This idea came from Zhamak Dehghani — first with two blogs in 2019, and then with her O’Reilly book in 2022.

Here’s the TL;DR from Data Mesh Learning Community: “The shortest summary: treat data as a product, not a by-product. By driving data product thinking and applying domain driven design to data, you can unlock significant value from your data. Data needs to be owned by those who know it best.”

The data mesh was everywhere in 2021. In 2022, it started to move from abstract idea to reality.

The data mesh conversation has shifted from “What is it?” to “How can we implement it?” as real user stories grew and the ideas behind the data mesh became less abstract and more actionable.

Meanwhile, companies started to brand themselves around the data mesh. So far, we’ve seen this with Starburst, Databricks, Oracle, Google Cloud, Dremio, Confluent, Denodo, Soda, lakeFS, and K2 View, among others.

Four years after it was created, we’re still in the early phases of the data mesh.

In 2023, the first wave of data mesh “implementations” will go live, with the “data as a product” concept front and center.

This year, we’ll start seeing more and more real data mesh architectures — not the aspirational diagrams that have been floating around data blogs for years, but real architectures from real companies.

The data world will also start to converge on a best-in-class reference architecture and implementation strategy for the data mesh. This will include the following core components:

• Metadata platform that can integrate into developer workflows (e.g. Atlan’s APIs and GitHub integration)
• Data quality and testing (e.g. Great Expectations, Monte Carlo)
• Git-like process for data producers to incorporate testing, metadata management, documentation, etc. (e.g. dbt)
• All built around the same central data warehouse/lakehouse layer (e.g. Snowflake, Databricks)

John Cutler on Twitter: "the opposite of data mesh isdata mehproducers do whatever they want, and some poor soul needs to make sense of it all so it can be consumed / Twitter"

the opposite of data mesh isdata mehproducers do whatever they want, and some poor soul needs to make sense of it all so it can be consumed

One of our big trends from last year, data observability held its own and continued to grow alongside adjacent ideas like data quality and reliability.

Existing companies became even bigger (e.g. Databand’s acquisition by IBM in July 2022, Monte Carlo’s $135M Series B), new companies went mainstream (e.g. Kensu and Monte Carlo jumped into thought leadership), and new tools launched every month (e.g. Bigeye’s Metadata Metrics).

In a notable change, this space also saw significant open-source growth in 2022. Datafold launched an open-source diff tool, Acceldata open-sourced its data platform and data observability libraries, and Soda launched both its open-source Soda Core and enterprise Soda Cloud platforms.

One of our open questions in last year’s report was where data observability was heading — towards its own category, or merging with another category like data reliability or active metadata.

Data observability and quality will converge in a larger “data reliability” category centered around ensuring high-quality data.

This may seem like a big change, but many of the companies in these categories have changed names multiple times over the years, such as Datafold going from data diffs to a “data reliability platform”.

As these companies compete to define and own the category, we’ll continue to see more confusion in the short term. However, there are early signs that this will start to settle down into one category in the near future.

Sarah Catanzaro on Twitter: "Are we turning a page? I sense a growing recognition among data Twitter that perhaps it's the data and not the tools, people, or titles that's the source of our collective grief? / Twitter"

Are we turning a page? I sense a growing recognition among data Twitter that perhaps it's the data and not the tools, people, or titles that's the source of our collective grief?

It feels interesting to welcome 2023 as data practitioners. While there’s a lot of uncertainty looming in the air (uncertainty is the new certainty!), we’re also a bit relieved.

2021 and 2022 were absurd years in the history of the data stack.

The hype was crazy, new tools were launching every day, data people were constantly being poached by data startups, and VCs were throwing money at every data practitioner who even hinted at building something. The “modern data stack” was finally cool, and the data world had all the money and support and acknowledgment it needed.

At Atlan, we started as a data team ourselves. As people who have been in data for over a decade, this was a wild time. Progress is generally made in decades, not years. But in the last three years, the modern data stack has grown and matured as much as in the decade before.

It was exciting… yet we ended up asking ourselves existential questions more than once. Is this modern data stack thing real, or is it just hype fueled by VC money? Are we living in an echo chamber? Where are the data practitioners in this whole thing?

While this hype and frenzy led to great tooling, it was ultimately bad for the data world.

Confronted by a sea of buzzwords and products, data buyers often ended up confused and could spend more time trying to get the right stack than actually using it.

Let’s be clear — the goal of the data space is ultimately to help companies leverage data. Tools are important for this. But they’re ultimately an enabler, not the goal.

As this hype starts to die down and the modern data stack starts to stabilize, we have the chance to take the tooling progress we’ve made and translate it into real business value.

We’re at a point where data teams aren’t fighting to set up the right infrastructure. With the modern data stack, setting up a data ecosystem is quicker and easier than ever. Instead, data teams are fighting to prove their worth and get more results out of less time and resources.

Now that companies can’t throw money around, their decisions need to be targeted and data-driven. This means that data is more important than ever, and data teams are in a unique position to provide real business value. But to make this happen, data teams need to finally figure out this “value” question.

Now that we’ve got the modern data stack down, it’s time to figure out the modern data culture stack. What does a great data team look like? How should it work with business? How can it drive the most impact in the least time?

These are tough questions, and there won’t be any quick fixes. But if the data world can crack the secrets to a better data culture, we can finally create dream data teams — ones that will not just help their companies survive during the next 12–18 months, but propel them to new heights in the coming decades.

Ready for spicy takes and expert insights on these trends? We assembled a panel of superstars (Bob Muglia, Barr Moses, Benn Stancil, Douglas Laney, and Tristan Handy) for the first Great Data Debate of 2023. Watch the recording here.

The Future of the Modern Data Stack in 2023 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

At the beginning of this year, I made some bold predictions about the future of the modern data stack in 2022.

Instead of just kicking off 2023 with a new set of predictions — which, let’s be real, I’m still going to do — I wanted to pause and look back at the last year in data. What did we get right? What didn’t quite go as expected? What did we completely miss?

This time of year, as social media is flooded with lofty predictions, it’s easy to think that the people behind them are all-knowing experts. But really, we’re just people. People who have been buried neck-deep in the data world for years, yes, but still fallible.

That’s why this year, instead of just doing this exercise internally, I’m opening it up to the public.

Here are my reflections on six major trends from 2022 — what I got right and where I went completely wrong.

The verdict: Mostly true ✅ but progressing slower than expected ❌

TL;DR: We did see a lot of market consolidation around the “data mesh platform”, but implementation practices and tooling stack are farther behind the hype than we expected. Data mesh is still on my radar, though, and will stay as a key trend for 2023.

Where we started

Here’s what I said at the beginning of this year:

“In 2022, I think we’ll see a ton of platforms rebrand and offer their services as the ‘ultimate data mesh platform’. But the thing is, the data mesh isn’t a platform or a service that you can buy off the shelf. It’s a design concept with some wonderful concepts like distributed ownership, domain-based design, data discoverability, and data product shipping standards — all of which are worth trying to operationalize in your organization.

So here’s my advice: As data leaders, it is important to stick to the first principles at a conceptual level, rather than buy into the hype that you’ll inevitably see in the market soon.

I wouldn’t be surprised if some teams (especially smaller ones) can achieve the data mesh architecture through a fully centralized data platform built on Snowflake and dbt, whereas others will leverage the same principles to consolidate their ‘data mesh’ across complex multi-cloud environments.”

(All snippets are from the Future of the Modern Data Stack in 2022 Report.)

Where we are now

My prediction that companies would brand themselves around the data mesh absolutely happened. We saw this with Starburst, Databricks, Oracle, Google Cloud, Dremio, Confluent, Denodo, Soda, lakeFS, and K2 View, among others.

There has also been progress in the data mesh’s shift from idea to reality. Zhamak Dehghani published a book with O’Reilly about the data mesh, and real user stories are growing on the Data Mesh Learning Community.

The result is two increasingly popular theories of how to implement the data mesh:

• Via team structures: Distributed domain-based data teams that are responsible for publishing data products, supported by a central data platforms team that provides tools for the distributed teams
• Via “data as a product”: Data teams that are responsible for creating data products — i.e. pushing data governance to the “left”, closer to the data producers rather than consumers.

While this progress is notable, it ultimately didn’t move the needle far enough, and the data mesh is about as vague as a year ago. Data people are still craving clarity and specificity. For example, in Starburst’s conference on the data mesh, the most common question in the chat was “How can we actually implement the data mesh?”

While I expected that, this year, we as a community would move closer to the “how to implement the data mesh” discussion, we’re still about where we were last year. We’re still in the early phases as teams figure out what implementing the data mesh really means. Though more people have now bought into the concept, there’s a real lack of real operational guidance about how to achieve a data mesh in operation.

This is only compounded by the fact that the mesh tooling stack is still premature. While there’s been a lot of rebranding, we still don’t have a best-in-class reference architecture of how a data mesh can be achieved.

The verdict: Mostly true ✅ but slower than expected ❌

TL;DR: dbt Labs’ Semantics Layer launched as expected. This was a massive step forward for the metrics layer, but we’re still waiting to see the full impact on the way that data teams work with metrics. The metrics layer promises to remain a significant trend going into 2023.

Where we started

Here’s what I said at the beginning of this year:

“I am extremely excited about the metrics layer finally becoming a thing. A few months ago, George Fraser from Fivetran had an unpopular opinion that all metrics stores will evolve into BI tools. While I don’t fully agree, I do believe that a metrics layer that isn’t tightly integrated with BI is unlikely to ever become commonplace.

However, existing BI tools aren’t really incentivized to integrate an external metrics layer into their tools… which makes this a chicken and egg problem. Standalone metrics layers will struggle to encourage BI tools to adopt their frameworks, and will be forced to build BI like Looker was forced to many years ago.

This is why I’m really excited about dbt announcing their foray into the metrics layer. dbt already has enough distribution to encourage at least the modern BI tools (e.g. Preset, Mode, Thoughtspot) to integrate deeply into the dbt metrics API, which may create competitive pressure for the larger BI players.

I also think that metrics layers are so deeply intertwined with the transformation process that intuitively this makes sense. My prediction is that we’ll see metrics become a first-class citizen in more transformation tools in 2022.”

Where we are now

I put my money on dbt Labs, rather than BI tools, as the leader of the metrics layer — and that turned out to be right.

dbt Labs’ Semantic Layer launched (in public preview) as promised, along with integrations across the modern data stack from companies like Hex, Mode, Thoughtspot, and Atlan (us!). This was a huge step forward for the modern data stack, and it is definitely paving the way for metrics to become a first-class citizen.

What we didn’t get right was what came next. We thought that along with dbt’s Semantic Layer, the metrics layer would be rocket-launched into everyday data life. In reality, though, progress has been more measured, and the metrics layer has gained less traction than expected.

In part, this is because the foundational technology took longer than I expected to launch. After all, the Semantic Layer was just released in October at dbt Coalesce.

It’s also because changing the way that people write metrics is hard. Companies can’t just flip a switch and move to a metric/semantic layer overnight. The change management process is massive, and it’s more likely that the switch to the metrics layer will take years, rather than months.

The verdict: Mostly true ✅ but also starting to head in a new direction ❌

TL;DR: As expected, this space is starting to consolidate with ETL and data ingestion. At the same time, however, reverse ETL is now attempting to rebrand itself and expand its category.

Where we started

Here’s what I said at the beginning of this year:

“I’m pretty excited about everything that’s solving the ‘last mile’ problem in the modern data stack. We’re now talking more about how to use data in daily operations than how to warehouse it — that’s an incredible sign of how mature the fundamental building blocks of the data stack (warehousing, transformation, etc) have become!

What I’m not so sure about is whether reverse ETL should be its own space or just be combined with a data ingestion tool, given how similar the fundamental capabilities of piping data in and out are. Players like Hevo Data have already started offering both ingestion and reverse ETL services in the same product, and I believe that we might see more consolidation (or deeper go-to-market partnerships) in the space soon.”

Where we are now

My big prediction was that we’d see more consolidation in this space, and that definitely happened as expected. Most notably, the data ingestion company Airbyte acquired Grouparoo, an open-source reverse ETL product.

Meanwhile, other companies cemented their foothold in reverse ETL with launches like Hevo Data’s Hevo Activate (which added reverse ETL to the company’s existing ETL capabilities) and Rudderstack’s Reverse ETL (a rebranded version of its earlier Warehouse Actions product line).

However, rather than trending toward consolidation, some of the main players in reverse ETL have focused on redefining and expanding their own category this year. The latest buzzword is “data activation”, a new take on the “customer data platform” (CDP) category, driven by companies like Hightouch and Rudderstack.

Here’s their broad argument — in a world where data is stored in a central data platform, why do we need standalone CDPs? Instead, we could just “activate” data from the warehouse to handle traditional CDP functions like sending personalized emails.

In short, they’ve shifted from talking about “pushing data” to actually driving customer use cases with data. These companies still talk about reverse ETL, but it’s now a feature within their larger data activation platform, rather than their main descriptor. (Notably, Census has resisted this trend, sticking with the reverse ETL category across its site.)

The verdict: Mostly true ✅

TL;DR: This category continued to explode with buy-in from analysts and companies alike. While there’s not one dominant winner yet, the space is starting to draw a clear line between traditional data catalogs and modern catalogs (e.g. active metadata platforms, data catalogs for DataOps, etc).

Where we started

Here’s what we said at the beginning of this year:

“The data world will always be diverse, and that diversity of people and tools will always lead to chaos. I’m probably biased, given that I’ve dedicated my life to building a company in the metadata space. But I truly believe that the key to bringing order to the chaos that is the modern data stack lies in how we can use and leverage metadata to create the modern data experience.

Gartner summarized the future of this category in a single sentence: ‘The stand-alone metadata management platform will be refocused from augmented data catalogs to a metadata ‘anywhere’ orchestration platform.’

Where data catalogs in the 2.0 generation were passive and siloed, the 3.0 generation is built on the principle that context needs to be available wherever and whenever users need it. Instead of forcing users to go to a separate tool, third-gen catalogs will leverage metadata to improve existing tools like Looker, dbt, and Slack, finally making the dream of an intelligent data management system a reality.

While there’s been a ton of activity and funding in the space in 2021, I’m quite sure we’ll see the rise of a dominant and truly third-gen data catalog (aka an active metadata platform) in 2022.”

Where we are now

Given that this is my space, I’m not surprised that this prediction was fairly accurate. What I was surprised by, though, was how this space outperformed even my wildest expectations.

Active metadata and third-gen catalogs blew up even faster than I anticipated. In a massive shift from last year, when only a few people were talking about it, tons of companies from across the data ecosystem are now competing to claim this category. (Take, for example, Hevo Data and Castor’s adoption of the “Data Catalog 3.0” language.) A few have the tech to back up their talk. But like the early days of the data mesh, when experts and newbies alike appeared equally knowledgable in a space that was still being defined, others don’t.

Part of what made the space explode this year is how analysts latched onto and amplified this idea of modern metadata and data catalogs.

After its new Market Guide for Active Metadata in 2021, Gartner seems to have gone all in active metadata. At its conference this year, active metadata popped up as one of the key themes in Gartner’s keynotes, as well as in what seemed like half of the week’s talks across different topics and categories.

G2 released a new “Active Metadata Management” category in the middle of the year, marking a “new generation of metadata”. They even called this the “third phase of…data catalogs”, in keeping with this new “third-generation” language.

Similarly, Forrester scrapped its Wave report on “Machine Learning Data Catalogs” to make way for “Enterprise Data Catalogs for DataOps”, marking a major shift in their idea of what a successful data catalog should look like. As part of this, Forrester upended their Wave rankings, moving all of the previous Leaders to the bottom or middle tiers — a major sign that the market is starting to separate modern catalogs (e.g. active metadata platforms, data catalogs for DataOps, etc.) from traditional data catalogs.

The verdict: Didn’t come true ❌

TL;DR: As much as I wish this had come true, we made far less progress on this trend than I expected. Twelve months later, we’re pretty much where we started.

Where we started

Here’s what we said at the beginning of this year:

“Of all the hyped trends in 2021, this is the one I’m most bullish on. I believe that in the next decade, data teams will emerge as one of the most important teams in the organization fabric, powering the modern, data-driven companies at the forefront of the economy.

However, the reality is that data teams today are stuck in a service trap, and only 27% of their data projects are successful. I believe the key to fixing this lies in the concept of the ‘data product’ mindset, where data teams focus on building reusable, reproducible assets for the rest of the team. This will mean investing in user research, scalability, data product shipping standards, documentation, and more.”

Where we are now

Looking back on this one hurts. Of all my predictions, this one not coming true (yet? 🤞) makes me incredibly sad.

Despite the talk, we’re still so far from the reality of data teams operating as product teams. While data tech has matured a lot this year, we haven’t progressed much farther than we were last year on the human side of data. There just hasn’t been much progress on how data teams fundamentally operate — their culture, processes, etc.

The verdict: Mostly true ✅

TL;DR: As predicted, this space continued to expand and fragment itself this year. Where it will go next year, though, and whether it will merge with adjacent categories is still an open question.

Where we started

Here’s what we said at the beginning of this year:

“I believe that in the past two years, data teams have realized that tooling to improve productivity is not a good-to-have but a must-have. After all, data professionals are one of the most sought-after hires you will ever make, so they shouldn’t be wasting their time on troubleshooting pipelines.

So will data observability be a key part of the modern data stack in the future? Absolutely. But will data observability continue to exist as its own category or will it be merged into a broader category (like active metadata or data reliability)? This is what I’m not so sure about.

Ideally, if you have all your metadata in one open platform, you should be able to leverage it for a variety of use cases (like data cataloging, observability, lineage and more). I wrote about that idea last year in my article on the metadata lake.

That being said, today, there’s a ton of innovation that these spaces need independently. My sense is that we’ll continue to see fragmentation in 2022 before we see consolidation in the years to come.”

Where we are now

The big prediction was that this space would continue to grow, but in a fragmented rather than consolidated fashion — and that certainly happened.

Data observability has held its own and continued to grow in 2022. The number of players in this space has just continued to grow, with existing companies getting bigger, new companies becoming mainstream, and new tools launching every month.

For example, in company news, there were some major Series Ds (Monte Carlo with $135M, Unravel with $50M) and Series Bs (Edge Delta with $63M, and Manta with $35M) in this space.

As for tooling, Acceldata open-sourced its platform, Kensu launched a data observability solution, AWS introduced observability features into Amazon Glue 4.0, and Entanglement spun out another company focused on observability.

And in the thought leadership arena, both Monte Carlo and Kensu published major books with O’Reilly about data observability.

To make things more complicated, many industry-adjacent or early-stage companies have also been expanding and cement their role in this space. For example, after starting in the data quality space, Soda is now a major player in data observability. Similarly, Acceldata started in logs observability but now brands itself as “Data Observability for the Modern Data Stack”. Metaplane and Bigeye have also been growing in prominence since their launch and Series B, respectively, in 2021.

Like last year, I’m still not sure where data observability is heading—towards independence or a merge with data reliability, active metadata, or some other category. But at a high level, it seems that it’s moving closer to data quality, with a focus on ensuring high-quality data, rather than active metadata.

As we close out December 2022, it’s amazing to see how much the data world has changed.

It was just nine months ago in March that Data Council happened, where we debated the heck out of the data world. We put out all the hot takes on our tech, community, vibe, and future — because we could. We were in growth mode, looking for the next new thing and vying for a chunk of the seemingly infinite data pie.

Now we’re in a different world, one of recession and layoffs and budget cuts. We’re shifting from growth mode to efficiency mode.

Don’t get me wrong — we’re still in the golden age of data. Just a few weeks ago, Snowflake announced record revenue and 67% year-over-year growth.

But as data leaders, we’re facing new challenges in this golden age of data. As most companies start talking about efficiency, how can we think about using data to leverage the most efficiency in our work? What can data teams do to become the most valuable resource in their organizations?

I’m still trying to puzzle out how this will affect the modern data stack, and I can’t wait to share my thoughts soon. But the one thing I’m sure about is that 2023 will be a year to remember in the data world.

The 2023 Future of the Modern Data Stack Report is out! Read it here or download the PDF.

Ready for spicy takes and expert insights on these trends? We assembled a panel of superstars (Bob Muglia, Barr Moses, Benn Stancil, Douglas Laney, and Tristan Handy) for the first Great Data Debate of 2023. Watch the recording here.

What I Got Wrong: Looking Back at My 2022 Predictions for the Modern Data Stack was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.