<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by AstroBee on Medium]]></title>
        <description><![CDATA[Stories by AstroBee on Medium]]></description>
        <link>https://medium.com/@AstroBeeai?source=rss-21f2c43adacd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*krOosEL26tL_gDowHTbJ1A.png</url>
            <title>Stories by AstroBee on Medium</title>
            <link>https://medium.com/@AstroBeeai?source=rss-21f2c43adacd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 03:04:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AstroBeeai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From ETL to ELT and Back Again: Transformations in AI Token Space]]></title>
            <link>https://medium.com/astrobee-ai/from-etl-to-elt-and-back-again-transformations-in-ai-token-space-1074843b2b14?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1074843b2b14</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ontology]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 20:21:33 GMT</pubDate>
            <atom:updated>2025-10-28T20:58:50.744Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuYiOwHMn6gO2IzuF_xTbA.png" /></figure><p>Everyone and their grandmother nowadays knows about ETL (extract-transform-load). Whoever ran that category creation push did a great job, because it’s become the household name for “any process that puts data into the data warehouse”. There’s even a concept of Reverse ETL, which has nothing to do with “load-transform-extract” but instead just means “taking data out of the warehouse and putting it back into your software systems”.</p><p>The problem is, ETL is mostly dead nowadays. People don’t talk about it that way, but it’s true. In reality, most people practice ELT (extract-load-transform) as a best practice. Gone are the days where transformations in-route from source systems to the data warehouse were essential for high-performance analytics. Compute and storage are cheaper than ever, and modern data warehouses can more or less take all the transformations you want to throw at them without breaking the bank.</p><p>I think etymology and semantics matter. They teach important things about history and industry. It might seem like ETL and ELT are close enough that the distinction doesn’t matter, but the switch between the two can teach us a lot about designing effective systems within the constraints of our time.</p><p>Another semantic oddity: “Fivetran” is short for “five transformations”, the original idea being that you would only ever need five transformations to do anything you want with your data: filter, join, aggregate, project, and sort. In 2012, the “Five Transformations” company was pulling data from source systems and shoving it into warehouses that were expensive and limited, like Teradata. They entered a market where true “ETL” was standard, you transform your data <em>before</em> loading it into a data warehouse. Those pre-loading transformations were so important that the founders named their whole company after them.</p><p>Nowadays, most people view Fivetran as a standard for moving data from point A to point B, with minimal transformation. It’s mostly a data integration product, not a data transformation product. Their name is a vestigial structure from a time long gone, when running transformations entirely in data warehouses was prohibitively expensive. Back then, it made no sense to dump raw data as-is into Teradata and run the rest of your pipelines in the warehouse. It would have been silly.</p><p>I think there’s a lot to learn from this story. We can apply these lessons the newest and grooviest technology hitting modern data stacks: AI applications with MCP connectors.</p><p>Everyone is providing an MCP connector for their software systems. There’s MCP connectors for HubSpot, Intercom, Stripe… and soon for everything you can imagine. Looking at Anthropic’s interface for connecting tools, you might think you can just go to your favorite model provider, connect directly to your systems via MCP, and be able to answer any question you’d want to answer. This is version 0 of AI enablement. With good, focused prompt engineering, it can work decently when your workflows are concerned within only one system.</p><p>The problems arise when you must work with questions that span multiple systems. If you achieve AI adoption with MCP on top of your operational systems, then <em>any</em> cross-system workflows rely on doing data integration and transformation entirely in token-space. There is no centralized storage and computation layer for the underlying data, so you’re relying on a sequence of tool calls strung together by token generation to figure out how to pull data together.</p><p>The real issue is that this is terribly inefficient and that’s not going to change soon. <strong>AI token generation of 2025 is the Teradata computation of 2012.</strong> Powerful, but prohibitively expensive for enterprise-scale integration and transformation work. It is significantly cheaper, and honestly more effective, to run your integrations and transformations in a central data warehouse, form an ontology or other semantic layer, and then expose that to your AI workloads. If the 1980s to early 2000s were the ETL era for data warehousing (as in transforming before loading to the data warehouse), then we are currently in the ETL era of AI (transform before loading data into AI).</p><blockquote>If the 80s to early 2000s were the ETL era for data warehousing, then we are currently in the ETL era of AI.</blockquote><p>A lot of people don’t want to hear this because it requires either piling another stack of work on top of an already overworked data team, or it requires getting a data team together for the first time when you’ve been able to scrape by without one in the past. These folks want to jump ahead directly to the ELT era (load raw data directly into AI and let it figure it out). AI makes it easy to think this will work, because it works on small-scale, narrowly scoped use cases. However, this tendency is a big reason for the failure of many enterprise AI projects to return real ROI… poor performance for real business use cases, and expensive.</p><p>Our vision at AstroBee is to make the process of transforming data before it gets into your AI as efficient and affordable from a human labor standpoint as possible. AstroBee is a platform that builds the centralized, integrated source of truth for you (the <a href="https://blog.astrobee.ai/what-the-heck-is-an-ontology-22f08d655175">ontology</a>). That way, with minimal investment, you can feed your AI projects the data they deserve. Out of the box, AstroBee comes with an analytics agent that works for you as your enterprise data analyst, but you can plug your data into any other AI workflows you’re working on. If you’re interested in working with an early version of AstroBee just let us know at <a href="/service/mailto:hello@astrobee.ai">hello@astrobee.ai</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1074843b2b14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/astrobee-ai/from-etl-to-elt-and-back-again-transformations-in-ai-token-space-1074843b2b14">From ETL to ELT and Back Again: Transformations in AI Token Space</a> was originally published in <a href="https://medium.com/astrobee-ai">AstroBee AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where MCP falls short: data integration in the AI world]]></title>
            <link>https://medium.com/@AstroBeeai/where-mcp-falls-short-data-integration-in-the-ai-world-ddb29fce965f?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddb29fce965f</guid>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 08 Aug 2025 18:58:02 GMT</pubDate>
            <atom:updated>2025-08-08T18:58:02.924Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*078I0qV0dXXxM9f3gTZ_Tg.png" /></figure><p>Connecting to disparate source systems is a solved problem. Tools like FiveTran, AirByte, and <a href="http://merge.dev/">merge.dev</a> sink thousands of hours of engineering work into supporting connectors to every software system on the planet, and most teams who supporting 1,000s of connectors can use one of these tools under the hood to get there.</p><p>Once data is connected, you still have the problem of data representation. Each of your source systems (like HubSpot and Intercom, or your billing system and product database), will represent data in their own way. That’s what you get when you connect to APIs, you must speak the language of the API provider. It’s what you get when you connect to MCP servers, too. It’s the same structure, and presents data the same way.</p><p>If you connect an LLM to 5 different MCP servers and then ask for information regarding your customer, say Acme Incorporated , your answer <em>will</em> be degraded by the fact that each source system might represent customers differently. Maybe in HubSpot, companies are represented by their primary domain, <a href="http://getacme.com/">getacme.com</a> . Maybe in your billing system, they’re represented by the “Doing-business-as” name; Acme. And then maybe in your product, there are three subsidiaries that have engaged and are submitted support requests in Intercom under the names of those subsidiaries.</p><p>For those who are dead-set on AI enablement, there are two options available to solving this problem:</p><ol><li><strong>Resolve Entities on Inference:</strong> Trust the LLM to expend enough tokens in “thought” to scan through <em>all</em> of the data available in each source system via its MCP connections to exhaustively search around the Acme Incorporated concept to figure out where data related to Acme lives in your systems, and then report on it</li><li><strong>Resolve Entities Ahead of Time:</strong> Use traditional data processing techniques to reveal gaps in entity resolution <em>prior</em> to inference, and then use inputs from business stakeholders and data experts to resolve entities in an integrated “source of truth” layer. LLMs refer to this layer on inference, instead of raw source systems.</li></ol><p>If you just upload small amounts of data to Chat-GPT or Claude, it seems like the first option will work great. LLMs are very good at this semantic search and conceptual construction work. Your CSVs fit entirely into the context window of the LLM, and every token emitted is constructively contributing to grouping like elements together.</p><p>The moment you start working on larger datasets, the first option falls apart. For large scale search-and-compute, LLM-driven crawling is painfully slow compared to traditional data processing techniques. <em>Semantic search</em> works great at larger scales, for what it does, but is different altogether from reasoning occurring within one context window.</p><p>You could view this as an issue of context window limitations, but I think this is short-sighted. The context window is what it is. In the medium term, at least, it seems like it’ll always be a factor we need to design around, in the same way that RAM is still a factor to design around no matter how much memory we manage to get on our laptops. I think the context window is just the wrong tool for the job. Use token-based compute for what its good at, and use classical compute for what it’s good at: large scale data processing.</p><p>At AstroBee, we don’t foresee a world where MCP alone is the primary way to get strong AI capabilities across your enterprise. We see MCP working great for specific tool calling, and faltering when data needs to be reconciled <em>across</em> tools in order to make good decisions.</p><p>AstroBee integrates your data for you, so that entity resolution across your different systems happens before inference, and with the right tools: a mixture of SQL-driven integration, semantic search, and a touch of LLM-driven reasoning. Then, when AI agents work over your enterprise data, they work with pre-integrated data, delivering significantly better results and making significantly better decisions than before.</p><p>To see an alpha research preview of AstroBee, visit us at <a href="https://astrobee.ai/">https://astrobee.ai</a>. If you’d like to discuss our approach and give feedback or special requests, contact me directly at <a href="/service/mailto:galen@astrobee.ai">galen@astrobee.ai</a></p><p><em>Author: Galen Marchetti, CEO</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddb29fce965f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How ontologies drive proactive insight discovery]]></title>
            <link>https://medium.com/@AstroBeeai/how-ontologies-drive-proactive-insight-discovery-2e16d333fa02?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e16d333fa02</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 27 Jun 2025 18:51:24 GMT</pubDate>
            <atom:updated>2025-06-27T18:51:24.796Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*btnb1UdsGtt1IFIwSj_wMg.png" /></figure><p>Most data teams need to balance two opposing priorities in the tools they provide business stakeholders: accuracy of insights and breadth of data availability. Typically, you can not maximize one without sacrificing the other. The holy grail of the “text2sql” and “chat with your data” space is to break the dichotomy and allow full data democratization with full confidence in accuracy of insights. With the latest advances in foundation models, I believe we’re getting close to being able to do this. I also believe that the ability to “chat with your data” is only the beginning of the AI revolution in the data space. The next frontier is proactive insight discovery; the ability to answer questions you didn’t know you wanted to ask.</p><p>I’ll illustrate what I mean by starting with a business intelligence asset that’s existed since the 90s: the KPI dashboard.</p><h3>KPI Dashboard: Answer one question well</h3><p>Picture your typical KPI dashboard. Many key decisions hinge on accurate reporting of KPIs, and presenting the KPI in a central and trusted place is much more important than enabling ad-hoc analyses around it. It has a very high accuracy requirement, and a very narrow data availability requirement. When users view the KPIs, they might have a bunch of followup questions: Why did the number go down this week? How does this quarter compare to the last 4 quarters?</p><p>You send these requests to the data team, for them to prioritize and tackle with a days-to-weeks timeline. You don’t satisfy them within the KPI dashboard. In the KPI dashboard, flexibility is traded for clarity to the end user. They might have a hundred other questions they want to ask about the KPI. Curiosity is a virtue! But the KPI dashboard is designed to answer the KPI well and just the KPI, not to satisfy curiosity.</p><p>Realistically, unless there really is a burning curiosity in the heart of the user, many of those questions disappear into the void. Hopefully someone else at some point catches those lost whispers of curiosity and pursues them with more vigor. You don’t know what you don’t know and these questions sometimes unveil critical information relevant to making better decisions.</p><h3>Chat with your Data: Answer any question well</h3><p>Contrast the KPI dashboard with the emerging “chat with your data” experiences that AI-forward data teams are starting to provide. The “chat with your data” experience matches the users curiosity where it is**.** The user does the creative work and has a set of questions they want to ask, so they go to a chat interface to ask questions and get answers.</p><p>Whatever their heart desires, the magical AI fulfills. Providing a reliable “chat with your data” experience is worlds better than providing your team with a set of static dashboards. It enables them to move at the pace of their curiosity and imagination. To draw an analogy to the consumer content space, this is a similar experience to Google Videos or the classic YouTube experience. The primary interface to get the videos you want is the search bar, where you come with your own curiosity about what videos you want to see, and you’re satisfied once you complete a search and find a cool video.</p><h3>Ontology Building: Answer questions you didn’t know you had</h3><p>There’s an emerging category of ontology building tools, which enable teams to automatically build business-relevant structures on top of their sprawling raw data. The ontology building experience is one which stokes the end users curiosity. It answers questions they didn’t they even had.</p><p>When you build an ontology, you model raw data in terms of the objects and events that users actually think about. You map data to their real world. With every object and event in the real world as the rooting structure, an ontology generation AI like <a href="https://astrobee.ai/">AstroBee</a> populates your data model with useful properties and statistics that are within your raw data, and are sensibly relevant. These insights are useful, verifiable by data experts, but are not direct answers to questions any of your employees have ever asked.</p><p>Think of ontology building as hiring a team of senior data scientist who are experts in your particular industry and have complete visibility into your data. Such an employee would come to your team knowing what questions to ask, and would munge through your data to figure out the answers to those questions. They would then provide those insights to you on a platter.</p><p>To complete the analogy to the consumer content space, this is similar to modern TikTok-esque feeds. The user is presented with content they didn’t know existed and they didn’t know they wanted to see, because the underlying algorithm understands what is trending and what the user specifically is interested in seeing. Generated ontologies offer a chance to do the same for insights on top of enterprise data. By understanding the underlying data, and understanding the objects and events relevant to the users real life workflows, tools like AstroBee can discover and push important insights to the people who need them.</p><p><em>AstroBee is in early beta. We’re gearing up to launch in the coming weeks. If you’re interested in assisting in pre-launch testing and gaining early access discounts on launch, reach out to us at </em><a href="/service/mailto:galen@astrobee.ai"><em>galen@astrobee.ai</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e16d333fa02" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kimball Star Schema vs Palantir’s Ontology]]></title>
            <link>https://medium.com/@AstroBeeai/kimball-star-schema-vs-palantirs-ontology-ab12728ff129?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab12728ff129</guid>
            <category><![CDATA[star-schema]]></category>
            <category><![CDATA[inmon-vs-kimball]]></category>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 15:03:59 GMT</pubDate>
            <atom:updated>2025-06-20T15:03:59.039Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*axiqjwEunsVWM-f2j1ruNA.png" /></figure><p><em>Author: Galen Marchetti, CEO.</em></p><p>Here’s a joke I thought of:</p><p>Q: What does the <a href="https://en.wikipedia.org/wiki/Bill_Inmon">father of data warehousing</a> have in common with a $300B analytics powerhouse named after a Lord of the Rings spying device?</p><p>A: Both of them defy the star schema paradigm and encourage you to model your data top-down, in a highly normalized format.</p><p>…maybe you didn’t think that was funny, but I did. Not funny in a “ha-ha” way, but in a glitch-in-the-matrix kind of way. Like “huh, that’s funny”.</p><p>Pretty much everyone in data today uses some variation of the star schema paradigm popularized by <a href="https://en.wikipedia.org/wiki/Ralph_Kimball">Ralph Kimball</a>. If you’re familiar with data warehousing or business intelligence, you’ve probably seen “facts” and “dimensions” tables floating around in your tools. Those are Kimball’s work and they really are the industry standard for modeling data for BI purposes. They’re everywhere.</p><p>Palantir, a $300B data analytics company, makes a ton of money selling services that model enterprise data in a way that runs directly counter to the Kimball industry standard. They center their offering around the <strong>ontology,</strong> a highly normalized data model that represents the “objects” a business user would think about. These objects have links between them characterized as 1-to-1, 1-to-many, or many-to-many. It’s a highly intuitive structure.</p><p>The weird thing is that most of the industry doesn’t do that. Palantir’s roaring success in deploying teams of forward deployed engineers to build out ontologies in their platform is a testament to the power of this modeling technique. But usually there’s nothing new under the sun. So why isn’t anyone else doing this?</p><p>To me, as a former Palantir forward deployed engineer, it felt like a glitch in the matrix. It didn’t quite make sense.</p><p>I would come across customer data warehouses which were highly denormalized and structured in “facts” and “dimensions”. I asked myself why on earth they adopted <em>this</em> strategy, instead of the more intuitive, straightforward object-property approach. Following this train of thought, I uncovered a fascinating story of a passionate but respectful technological debate between titans of the data industry, <a href="https://en.wikipedia.org/wiki/Bill_Inmon">Bill Inmon</a> and <a href="https://en.wikipedia.org/wiki/Ralph_Kimball">Ralph Kimball</a>.</p><p>Bill Inmon, from the beginning, advocated for a top-down approach to data modeling. He felt enterprises should establish what they need from their data warehouse in a single source of truth, and then do the integration work upfront to map raw data from source systems into the source of truth in a highly consistent way. He advocated for a <strong>normalized</strong> layer (many tables, not pre-joined), pushing joins until the very end of the analytics process. This maximizes data availability, ensuring information is not aggregated away or dropped in “join” steps before it reaches the end user.</p><p>Ralph Kimball, on the other hand, is largely responsible for the “dimensional modeling” approach that most people use today. You may know this as the “star schema”. The approach has a lot of great features that I love: it’s incremental, decentralized, simpler up-front, and returns user-facing results quickly. It’s a “bottoms up” methodology; you can incrementally deliver analytics from each source system in your organization before you create a centralized, consistent source of truth. As a result of these really great properties, Kimball’s approach has achieved full ideological capture of the data warehousing community.</p><p>So what’s going on with Palantir? Why did they go so off-track of the consensus?</p><p>I don’t think they’re contrarian for the sake of being contrarian. Palantir sells contracts on business results and the price tag is high. High enough to be able to deploy specialized, highly skilled resources with the sole goal of developing a data model that achieves 7 or 8 figure business outcomes. In this environment, Palantir is able to pay the heavy up-front human cost of the Inmon, top-down approach: integrate all the data you can, pay down the effort to achieve strong consistency across disparate data sources, and deliver a high-level model that fully delivers on data availability in a human-understandable way (the ontology).</p><p>Most data teams don’t have the luxury of the budget, focus, and firepower of Palantir to build and maintain an ontology. Kimball’s approach is far easier, but comes with the downside of a lack of data visibility and coverage for the business user. Usually people close the gap with fancy dashboards and on-call schedules for analysts who respond to ad-hoc requests from business users.</p><p>AI is rapidly changing what people can do with a limited amount of budget and human firepower. In my opinion, now is the time to invest in technology that cuts the expense of ontology curation by 90%. That’s what we’re working on at AstroBee. We’re using LLM-based agents to do the upfront integration and data consistency work required to bring a full-coverage ontology to your team at an affordable price point. You could say that AstroBee brings the firepower of Palantir to the little guy. We’re gearing up to launch soon, but if you’d like to take part in early pre-launch user testing, please reach out to <a href="/service/mailto:galen@astrobee.ai">galen@astrobee.ai</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab12728ff129" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Gap in the AI-Native Data Stack]]></title>
            <link>https://medium.com/@AstroBeeai/the-gap-in-the-ai-native-data-stack-6cafe81f2a57?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cafe81f2a57</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 15:17:48 GMT</pubDate>
            <atom:updated>2025-06-12T16:35:16.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoiNp18MhGKiGzIBQUWDnQ.png" /></figure><p>When people talk about applying LLMs to the modern data stack, they usually focus on the analytics layer.</p><p>That makes sense. The analytics layer is the part most people are familiar with, where tools like Power BI, Tableau, and Looker live. These tools sit on top of data that’s been cleaned and modeled by a data team, so nontechnical users like customer success managers, product teams, sales, and marketing analysts can find insights from their data.</p><p>Given advancements in foundational models, the analytics layer is the lowest-hanging fruit of AI transformation. It’s relatively trivial to get a foundational model to take clean table headers from a CSV or a warehouse and generate SQL, Python, or R that answers a simple business question. This does however, assume your data is well-structured, correct, and aligned with your organization’s definitions of things like “active users” or “revenue,”. Provided that’s the case, natural language interfaces can go a long way in helping people find answers.</p><p>Several highly competent products have been built around this idea. They let users upload data and ask questions in plain English. A model generates a SQL query or Python code and spits out an answer. These tools are fairly popular for good reason. They’ve captured the lowest-hanging fruit of AI data transformation, but I fear their position is fragile. It’s easy to imagine OpenAI or another major provider adding upload-and-analyze features directly, just as they added search to regular prompting. This raises questions on the longevity of specialized tools like Perplexity, built around search.</p><p>Meanwhile, the analytics has been a crowded space for years. Legacy vendors like Tableau and Power BI have deep roots in the enterprise and have already locked in most large customers. With the rapid proliferation of text-to-SQL tools, one can make the argument that there isn’t much left to innovate on in this layer.</p><h3>The Opportunity Under the Analytics Layer</h3><p>That’s where the real problem lives: how do you create clean, trustworthy, and well-defined data in the first place? This is the hard part every data team deals with. It’s not just technical. The data has to be structured in a way that makes sense to people across the company who don’t live in the data. Most business users don’t have the time or context to interpret things like schema design, column logic, or how data was joined.</p><p>To make things manageable, data teams usually lock down the surface area. They present just a handful of metrics in a very controlled way. You’ve seen this before. It’s the dashboard with one chart and one number that everyone agrees matters. Green is good. Up and to the right means growth.</p><p>But as soon as someone wants to go beyond that, things get slower. They send an ad hoc request to the data team. The data team has to figure out what’s being asked, how to model it, and how to answer it. Even the “one chart” that goes on the dashboard can take weeks to get right. Since time is limited, there’s usually a prioritization process. Most questions never get asked, and many that are asked don’t get answered. Adding an LLM on top of your existing data doesn’t really change this. The limit isn’t the interface. It’s the number of reliable, well-defined datasets the business has access to. That’s the bottleneck.</p><h3>That’s what we’re working on at Astrobee.</h3><p>We’re not building an incremental improvement in a BI interface. We’re focused on helping data teams automatically build new sources of truth: datasets that are clear, consistent, and usable by everyone in the organization. This involves everything from figuring out what a business user actually means (disambiguation of business terms), to resolving entity mismatches across systems (deduplication and entity resolution), to mapping raw data into forms that reflect how the business actually thinks (contextual semantic layer).</p><p>Solving this problem calls for a foundational different kind of tool. One that collects nuance from both technical and non-technical users. One that checks if a model’s output matches not just a query, but the intent behind it. One that might use different models for different tasks throughout the process.</p><p>All of these factors make this a very hard, but very fascinating problem to solve. If you’re interested in following along as we tackle this problem, please follow our socials on LinkedIn and X and check out our product as we progress at <a href="http://astrobee.ai/">astrobee.ai</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cafe81f2a57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Drift: How to Build Institutional Memory into your Data Stack]]></title>
            <link>https://medium.com/@AstroBeeai/semantic-drift-how-to-build-institutional-memory-into-your-data-stack-7b22bf2bff78?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b22bf2bff78</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Wed, 28 May 2025 19:02:41 GMT</pubDate>
            <atom:updated>2025-05-28T19:02:41.954Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RWDpiV0Q8MQAtYkXXILknQ.png" /></figure><p>It always starts the same way.</p><p>A dashboard looks off. Maybe the churn rate spiked unexpectedly. Or the LTV curve flattened when it shouldn’t have. An analyst is tasked with figuring out what happened. They retrace the query. They check the join. They pull the raw tables. Nothing is obviously broken. And yet, the numbers don’t make sense.</p><p>After an hour (or several), they find the culprit: someone defined “active customer” differently for a campaign report six months ago, and that logic has quietly propagated into multiple dashboards and downstream metrics ever since.</p><p>No one was trying to mislead anyone. There was just no canonical definition to begin with, or if there was, it wasn’t discoverable.</p><p>This kind of silent logic drift happens everywhere:</p><ul><li>Revenue numbers that include or exclude refunds, depending on the team</li><li>Subscription metrics that split differently between self-serve and enterprise</li><li>Retention curves skewed by inconsistent event filters</li></ul><p>It’s not that people aren’t asking the right questions. It’s that they’re using subtly different definitions every time. AstroBee is built to stop that cycle.</p><h3>What Is Semantic Drift?</h3><p>Semantic drift is what happens when the reality of your business changes, but your data logic doesn’t.</p><p>Consider a subscription-based SaaS company. Initially, the company only had one core product and two straightforward subscription plans: monthly and annual. The analytics team built a metric called active_subscriptions, defined simply as:</p><pre>SELECT COUNT(DISTINCT subscription_id)<br>FROM subscriptions<br>WHERE status = &#39;active&#39;</pre><p>This simple definition worked perfectly at launch and was embedded across dashboards, quarterly investor presentations, financial reports, and growth analyses. It quickly became a simple, single source of truth for how the team understood their subscriber base.</p><p>A year passes, and the company grows rapidly, introducing several new product tiers, pricing options, and free-trial plans. Product and growth teams add complexity: now, subscriptions can have statuses like active, trialing, paused, pending_cancel, and past_due. Some plans have grace periods or trial extensions, creating ambiguity around when exactly a subscription counts as truly “active.”</p><p>One analyst revises the core metric for a critical executive dashboard, updating it to:</p><pre>SELECT COUNT(DISTINCT subscription_id)<br>FROM subscriptions<br>WHERE status IN (&#39;active&#39;, &#39;trialing&#39;, &#39;past_due&#39;)<br>AND end_date &gt; CURRENT_DATE</pre><p>This new logic was subtly different, intended to reflect a more realistic measure of engagement. However, they didn’t document it thoroughly or propagate it clearly across other analytics assets.</p><p>Meanwhile, the finance team continued using the old, simpler definition in their reporting, which feeds into monthly revenue forecasts. Marketing built email campaigns and conversion funnels using yet another variation, excluding ‘trialing’ and ‘past_due’ entirely:</p><pre>SELECT COUNT(DISTINCT subscription_id)<br>FROM subscriptions<br>WHERE status = &#39;active&#39;<br>AND trial_end_date IS NULL</pre><p>Months later, discrepancies begin to appear. Executives begin to notice that the active subscriber numbers from the product team’s dashboards consistently exceed those from finance’s reports. Marketing’s conversion rates look worse because their denominator (active subscriptions) is significantly smaller. Each team defends their definition as “correct” given their respective contexts, yet no definition matches another.</p><p>Eventually, after weeks of confusion and lost productivity, someone discovers the root cause: a seemingly minor difference in how each team defined a basic metric months earlier. This subtle drift of logic, this semantic drift, ****caused compounded losses in trust, decision-making quality, and countless hours of wasted labor.</p><p>And the consequences are subtle but compounding: decisions based on outdated logic, conflicting reports in exec reviews, and wasted hours “just trying to get the numbers to match.”</p><h3>Why does it happen?</h3><p>Most teams treat logic as a side effect of the query, not a first-class asset. Current BI and analytics stacks like Looker, Tableau, Power BI, Mode, prioritize speed and ease of querying over consistency and governance. Analysts typically pull raw or lightly transformed data and build logic directly into dashboards or ad-hoc queries. Here’s what that looks like in practice:</p><ul><li><strong>Copy/paste culture:</strong> Someone grabs a snippet of SQL from an old dashboard, tweaks it, and ships it without realizing the original had context-specific assumptions.</li><li><strong>No review layer:</strong> Metric logic is often written solo, without peer review or approval from the domain experts who actually own the concept.</li><li><strong>No version control:</strong> Logic changes over time, but there’s no audit trail or changelog to show <em>what</em> changed, <em>why</em> it changed, or <em>who</em> changed it.</li><li><strong>Org churn:</strong> The analyst who originally wrote the query leaves. The logic stays. But no one knows if it’s still right.</li><li><strong>Business evolution:</strong> The business changes faster than the queries. New product lines, pricing models, or customer types make old logic incomplete or misleading.</li></ul><h3>Business Logic Should Be Responsive</h3><p>Can we do better? Imagine if your most-used entities and metrics like CAC, LTV, active users, or MRR weren’t redefined every time they were queried. What if your semantic laye<strong>r a</strong>ctively detected <strong>w</strong>hen business changes required updating logic and alerted domain experts instantly?</p><p>In our SaaS subscription scenario, AstroBee would have actively recognized the introduction of new subscription statuses (trialing, past_due) as soon as they appeared in your data warehouse. Instead of waiting months for teams to notice discrepancies, AstroBee immediately surfaces the change:</p><ul><li><strong>Detect:</strong> AstroBee notices new statuses like ‘trialing’ and ‘past_due’ appearing in the subscriptions table.</li><li><strong>Propose:</strong> AstroBee sends a proactive notification to the subscription metric owner (finance or analytics teams) with a clear message: “<em>Detected new statuses ‘trialing’ and ‘past_due’ in subscriptions data. Should these be included under active subscriptions?”</em></li><li><strong>Validate &amp; Govern:</strong> The responsible stakeholder confirms, ensuring the change aligns with relevant rules and business context</li><li><strong>Deploy &amp; Enforce:</strong> AstroBee updates the semantic layer immediately-automatically propagating the new definition across all dashboards, analyses, and queries that reference the active_subscriptions metric, and alerts stakeholders about the change.</li></ul><p>With AstroBee, semantic drift is resolved proactively. Now, when your business evolves, your data logic evolves right alongside it, without waiting days or weeks for someone to notice a problem.</p><p>Each decision adds to institutional memory. What used to live in tribal knowledge or scattered files becomes searchable, governed, and centralized.</p><h3>Analytics That Compounds Like Interest</h3><p>The current state of analytics is wasteful:</p><ul><li>Same joins rewritten 20 times</li><li>Same metrics revalidated every quarter</li><li>Same bugs rediscovered by new hires</li></ul><p>Astrobee flips the script. It builds a shared memory layer for your business, one where logic gets better over time. Questions stop being one-offs and instead become contributions.</p><p>The result is not just faster analysis, but better strategy. Less time debugging, more time deciding. Astrobee doesn’t just help you ask questions. It helps your organization <strong>remember</strong> the answers.</p><p>Because if your data stack forgets what it’s already done, you’re always starting from zero.</p><p>If your team is interested in trying out true end-to-end self-serve analytics, we’d love to hear from you. Follow our progress on socials (<a href="https://www.linkedin.com/company/astrobee/">LinkedIn</a> and <a href="https://x.com/astrobeeai">X</a>) and reach out to <a href="/service/mailto:hello@astrobee.ai">hello@astrobee.ai</a> to say hi.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b22bf2bff78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is AstroBee a Business Intelligence Tool?]]></title>
            <link>https://medium.com/@AstroBeeai/is-astrobee-a-business-intelligence-tool-eee46618ec53?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/eee46618ec53</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Wed, 21 May 2025 19:19:03 GMT</pubDate>
            <atom:updated>2025-05-21T19:42:44.905Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQmFRxwAgweccNnYVFWoEA.png" /></figure><p>The term “<em>Business Intelligence</em>” was first used in 1865 by a writer named Richard Millar Devens to describe how a 19th‑century banker used data to gain a competitive advantage.</p><p>Etymology is funny. When someone asks us if AstroBee is a BI tool, I think Richard Millar Devens would say <em>yes</em>. AstroBee is a tool for using data to gain a competitive advantage. It combines agentic AI with human business context, drawing on both the data itself and the people who know how to operationalize it.</p><p>However, when a venture analyst or someone at Gartner asks us if AstroBee is a BI tool, I think I have to say no. Sadly, in the 90s, when Gartner and others started defining a concrete market category called Business Intelligence, those words lost their imaginative power. Modern BI looks like Tableau, Looker, Power BI, and Mode. Snowflake is not a “business intelligence” tool in the analyst’s eyes, even though Snowflake has become a critical part of using data to gain a competitive advantage. Over time, analysts narrowed “business intelligence” to <strong>only</strong> the layer that connects to <strong>clean,</strong> useful data assets in order to surface insights directly to users.</p><h3>Two Huge Steps Before “BI”</h3><p>Before dashboards can even begin serving useful information, there’s a ton of prep work to transform enterprise data into a workable form.</p><ol><li><strong>Data Integration:</strong> First, all data from useful source systems, including event data, CRM records, payment history, and everything in between, is pulled into one place. This means data scientists toiling to gather an understanding of their data architecture, then write ETL pipelines, schedule batch jobs, and monitor for any issues.</li><li><strong>Data Transformation:</strong> Taking that raw data and working it into tables ready for analysis involves endless battles with join logic, unit conversions, inconsistent measures, and the eternal debate over what exactly “active customer” means. Transformation is where business logic becomes code, and where the smallest mistakes silently poison every downstream metric.</li></ol><p>Data teams routinely spend huge chunks of their time here, wrangling data into workable form.</p><h3>Where AstroBee Lives</h3><p>AstroBee does its magic in these two unsung steps.</p><ul><li><strong>Integration on Autopilot:</strong> AstroBee connects to your data warehouse where you’ve pooled raw data. Our AI agents take a scan of metadata, including column types, schemas, and labels. They can then infer relationships between various tables, identify keys, and write joins where appropriate.</li><li><strong>Transformation to Encode Business Logic:</strong> Instead of ad-hoc SQL scripts, AstroBee learns the mapping between “orders,” “subscriptions,” and “MRR” by observing data patterns and asking lightweight validation questions. It identifies the delta between its understanding of raw source data and the business logic needed to answer common BI queries. It then closes that delta and dilutes that logic into an ontology, or an auto-governed semantic layer that BI tools or AstroBee itself can query directly.</li></ul><p>In other words, AstroBee uses a team of AI agents to generate clean, useful data assets that conventional BI tools consume. It plays the role that data engineers and analysts have played for decades, often begrudgingly, so they can focus on predictive modeling, experimentation, and optimization.</p><h3>A Category that Doesn’t Exist…Yet</h3><p>Would you call such a tool a “business intelligence” tool? Someone with the creative capacity of Richard Devens in 1865 certainly would, as AstroBee enhances a business’s capability to use data to gain a competitive advantage.</p><p>An analyst in 2025, fluent in the many data products that proliferate the market, would certainly say no. They’d need to splice together key pieces of Data Integration, Data Transformation, and Business Intelligence to fully describe AstroBee’s end‑to‑end workflow. The old boxes don’t fit the new shape.</p><p>So, how do we answer the question, “Is AstroBee a Business Intelligence Tool?”</p><p><em>Kinda,</em> but also no. We’ll figure out what the right label for it is, and in the meantime, if your team is interested in trying out true end-to-end self-serve analytics, we’d love to hear from you. Follow our progress on socials (<a href="https://www.linkedin.com/company/astrobee/">LinkedIn</a> and <a href="https://x.com/astrobeeai">X</a>) and reach out to <a href="/service/mailto:hello@astrobee.ai">hello@astrobee.ai</a> to say hi.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eee46618ec53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What the heck is an Ontology?]]></title>
            <link>https://medium.com/@AstroBeeai/what-the-heck-is-an-ontology-22f08d655175?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/22f08d655175</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 16 May 2025 20:33:39 GMT</pubDate>
            <atom:updated>2025-05-16T20:34:43.153Z</atom:updated>
            <content:encoded><![CDATA[<p>Is Palantir onto something, or are they just blowing hot air?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6xvQYom7XFYfKEAcUHCWyw.png" /><figcaption>e</figcaption></figure><p>You may have heard the term Ontology, and why leading businesses rely on them to build trustworthy data models. Recent advances in LLMs have made it possible for startups and enterprises alike to build flexible, reliable semantic layers faster than ever before.</p><h3>What do we mean by “ontology”?</h3><p>The ontology has its roots in philosophy, where it represents a formal catalog of objects, the properties that describe them, and the relationships that connect them. Here’s a simple example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5Semtb_ObZdE0FpPUoS6w.png" /><figcaption>Example Simple Coffee Ontology</figcaption></figure><p>You might notice that this begins to resemble a data model. The Ontology has become popularized by Palantir as a semantic layer solution, which, to be reductive, is essentially a relational database with a semantic structure (different from the web 2.0 semantic ontology). This Ontology acts as an abstraction between raw data and the business concepts they represent. There are many implementations of semantic layers under a variety of names, but the bottom line is they’re working to make data represent our real world as we understand it.</p><h3>Why do people want semantic layers anyway?</h3><p>Every term you’ve heard — <em>source of truth</em>, <em>golden tables</em>, <em>semantic layer </em>— is chasing the same promise: let anyone ask and receive answers that faithfully reflect how the business works. An analyst should be able to ask, “How many premium users bought espresso last month?” and trust the answer. Well-implemented Ontologies deliver on that promise by encoding business logic and enforcing queries adhere to that model.</p><h3>Ontologies in the Real World</h3><p>As you might guess, this isn’t a new idea.</p><p>The star schema, introduced in the 1990s, puts a wide fact table at its center and fan‑outs to dimension tables on the edges. It celebrates speed and simplicity; query planners can predict access patterns, and BI tools understand the layout instinctively. This modeling approach forces data into either “dimensions” or “facts”. Facts generally have many, many rows with a few columns. The idea is to represent that which is <strong>large</strong> and highly repeatable row-over-row.</p><p>However, this hard divide between facts (immutable events) and dimensions (reference data) falls apart as the business evolves-say, a new pricing plan blurs the line between a transaction and a subscription, those early modeling choices become concrete shoes.</p><p>Snowflake schemas attempt to tame redundancy by further normalizing dimension tables. Storage usage drops, but the price is paid in join complexity. What once took a single hop now requires a small expedition through intermediary tables, and each extra join is another surface for error or performance regressions.</p><p>Both designs inherit the fact‑dimension mindset. It has served analytics well for decades, but it assumes the world stays still long enough for engineers to carve data into neat shapes. Modern businesses rarely stay that sessile.</p><h3>Palantir’s Ontology: turning the model into a graph</h3><p>Palantir threw out the fact‑dimension wall and treated every table as a node in a directed graph. Links between nodes come with explicit types, one‑to‑one, one‑to‑many, and so forth, so analysts can traverse the graph without guessing how joins should behave. The model is also iterative. If tomorrow’s reality demands a new object or a new relationship, you add a node or an edge and keep moving. Query optimizers still have plenty to chew on, but modelers are no longer forced to contort reality to fit a star pattern decided months earlier when the business looked dramatically different.</p><p>Palantir’s Ontology reflects current semantic logic while remaining fluid enough to reflect the changing reality of the business.</p><h3>Why volatility hurts everyone-enterprises and startups alike</h3><p>Unfortunately, it is incredibly challenging to build a solid, trustworthy source of truth that is also flexible enough to reflect a fast-moving environment.</p><p>Consider an enterprise that suddenly faces a new regulatory regime or an unexpected cybersecurity threat. Core schemas that once changed quarterly may now shift weekly as compliance teams add tables, columns, or entire data sources. Meanwhile, a fast‑growing startup is rewriting its own playbook-pivoting products, experimenting with pricing, or integrating a surge of user‑generated events. In both cases, pipelines that were handcrafted for yesterday break under the weight of today’s questions. The cost is measured in stale dashboards, duplicate metrics, and lost decisions.</p><p>Palantir gets around this problem by using the forward-deployed engineer model; embedded analysts who work to deeply understand how a business operates across industries, then task themselves with manually, and often painstakingly, maintaining an Ontology for a Fortune 500 company. Meanwhile, startups simply accept analytical debt as the price of speed.</p><h3>LLMs have changed the game: automatic ontology generation</h3><p>Large‑language models have changed the economics of generating and maintaining Ontologies. Point them at your data warehouses and they can take a massive snapshot of the current state of your data. They can then:</p><ul><li>Scan thousands of objects, notice common naming conventions, and summarize as a data model</li><li>Spot tables that share primary keys or semantic similarities in values, and propose relationships</li><li>Propose relationships between objects with cardinality</li></ul><p>What previously required weeks of an expensive embedded analyst working to understand every nook and cranny of your data warehouse, an LLM can gain a version of that understanding in moments. The challenge is that LLMs, unlike a human analyst, don’t innately know that your “customer” excludes free‑trial users or the specific way your company calculates key metrics. Without guidance, they hallucinate, generalize, or miss edge cases. That’s where human context and Astrobee come in.</p><h3>Astrobee, the collaboration layer</h3><p>Astrobee is the collaboration layer between subject‑matter experts and the LLM runtime.</p><ul><li><strong>Extract</strong> -We ingest your warehouse and scrape lineage to give the LLM its raw material.</li><li><strong>Draft</strong> — The model proposes objects, links, tests, plus the SQL and pipeline code to populate them.</li><li><strong>Review</strong> — Stakeholders approve, comment, or override in a Git‑style diff. Astrobee records every decision.</li><li><strong>Democratize Data</strong> — Anyone across your business can query your data, and rest assured they’re all referencing a single source of truth, Ontology</li></ul><p>As more questions flow through Astrobee, it spots patterns and optimizes itself. Repeated joins become reusable generated pipelines. Expensive ad‑hoc queries trigger recommendations to use specific measures org wide.</p><p>The result is compound leverage: each inquiry refines the ontology, and consequently, business insights, for subsequent users. Enterprises manage schema drift without large data teams, and startups achieve enterprise-grade modeling agility at seed-stage budgets.</p><p>If this sounds interesting to you, we’d love to chat. Drop us a line at hi@astrobee.ai</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22f08d655175" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fast Data with Entropy or Clean Data with Compromise?]]></title>
            <link>https://medium.com/@AstroBeeai/the-endless-entropy-in-your-data-stack-f5b364396e58?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5b364396e58</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 20:29:35 GMT</pubDate>
            <atom:updated>2025-03-27T14:41:55.442Z</atom:updated>
            <content:encoded><![CDATA[<p>Let&#39;s start by defining what entropy means in your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SPMx5cfFT9DUGmCmh82d6w.png" /></figure><p>In the fast-paced world of modern organizations, data is both a goldmine and a puzzle. Every team wants answers, and they want them yesterday. Yet as demands for analytics grow, so does the complexity of managing data. Systems fragment, definitions diverge, and chaos ensues, we’ll call that <strong>entropy</strong> in your data stack.</p><p>Entropy is the increasing randomness and inconsistency that can creep into an organization’s data systems over time. Left unchecked, it leads to inefficiency, inaccuracy, and frustration. The challenge for data teams? Balancing two seemingly opposing priorities:</p><ol><li>Delivering fast, actionable insights to stakeholders.</li><li>Maintaining a clean, consistent, and reliable data foundation.</li></ol><p>The trade-off is clear: centralization ensures consistency but creates bottlenecks, while decentralization enables speed but increases the risk of entropy. But does it have to be a choice? Can we reduce entropy without fully sacrificing either speed or consistency?</p><h3>The Role of Data Modeling in Analytics</h3><p>Data modeling is the unsung hero of analytics. It’s how we transform raw source data — often fragmented across systems — into tables that reflect business logic. Without it, even the most powerful BI tools or AI systems can’t deliver reliable insights.</p><p>But data modeling is no easy task. A great data model presents business logic in a way that reflects business users’ mental model of how the business is structured and operates. It requires expertise, context, and constant iteration to keep up with the evolving needs of an organization. Even with dedicated data teams, there’s always more demand for analytics than resources to meet it.</p><p>This tension — between servicing analytics needs and maintaining a clean data stack — is the root of the trade-offs we face. To understand these trade-offs, let’s take a deeper look at the two common approaches: centralization and decentralization.</p><h3>The Trade-Offs of Centralized vs. Decentralized Approaches</h3><p><strong>Centralized</strong></p><p>In a centralized model, the data team is the gatekeeper. They own the data stack, handle all requests, and ensure every report adheres to consistent definitions and logic.</p><p><strong>The Pros:</strong></p><ul><li><strong>Accuracy and consistency:</strong> With data experts at the helm, you can trust the results.</li><li><strong>Systemic reliability:</strong> Centralized ownership ensures that data definitions don’t diverge across teams.</li></ul><p><strong>The Cons:</strong></p><ul><li><strong>Bottlenecks:</strong> Every ad hoc request requires back-and-forth with the data team, slowing the process.</li><li><strong>Inefficiency:</strong> Loosely scoped requests can lead to time-consuming rework and delays.</li></ul><p><strong>Decentralized</strong></p><p>In a decentralized model, the data team builds foundational systems, but individual teams handle their own analytics. This often involves tools like semantic layers or more recently AI-based assistants.</p><p><strong>The Pros:</strong></p><ul><li><strong>Speed:</strong> Teams can directly query data without waiting for the data team’s bandwidth.</li><li><strong>Empowerment:</strong> Users leverage their unique understanding of their business to ask better questions.</li></ul><p><strong>The Cons:</strong></p><ul><li><strong>Inconsistency:</strong> Different teams interpret the same data differently, leading to conflicting results.</li><li><strong>Inaccuracy:</strong> Varying levels of expertise can result in errors and flawed insights.</li></ul><p>So, do you choose consistency or speed? Why not both?</p><p><strong>Why Not Both?</strong></p><p>The trade-off between centralization and decentralization is a false choice. The key is to democratize access to data while <strong>reducing entropy.</strong> This way, organizations can achieve the best of both worlds: the consistency of a centralized system with the speed of a decentralized approach.</p><p>Reducing entropy doesn’t mean eliminating divergence altogether. It means actively managing it — detecting where and why divergence happens and designing systems to reconcile differences. This requires a hybrid approach that prioritizes both staying consistent and meeting the changing needs of your teams.</p><h3>How Astrobee Tackles This Problem</h3><p>At Astrobee, our approach to reducing entropy in data systems isn’t about choosing between centralization and decentralization — it’s about designing a system that actively manages and reduces divergence over time. While every organization’s data stack is different, we believe that entropy can be controlled through a structured, adaptive process.</p><h3>Step 1: Identifying what we can learn from source data</h3><p>Before we can bring consistency to an organization’s data stack, we first need to understand what’s in the source tables and how different datasets relate to each other. Astrobee does this by using machine learning and semantic analysis to detect:</p><ul><li>Which tables are related to one another through semantic search</li><li>What columns can be joined or will require more transformation</li><li>How compatible schemas are across different source systems</li></ul><p>However, the effectiveness of this step depends on whether the necessary information actually exists in the source data.</p><p><strong>When the information exists in the data itself</strong></p><p>If the source data contains enough structure — shared IDs, email addresses, timestamps, or even natural language column names — we can automatically infer relationships between tables using:</p><ul><li><strong>Semantic vector models</strong> to detect similarities in column contents.</li><li><strong>Existing ETL code</strong> to analyze how data is currently being transformed.</li><li><strong>Pattern-based heuristics</strong> to suggest potential joins based on historical queries.</li></ul><p><strong>When the information does NOT exist in the data</strong></p><p>If the necessary relationships aren’t present in the data itself, we need to look externally. This means:</p><ul><li><strong>Polling users who work with the data daily</strong> to understand how they manually join datasets.</li><li><strong>Asking domain experts</strong> (often in Slack, through documentation, or in data team office hours) for clarification.</li><li><strong>Inferring relationships from business logic</strong> — for example, if two systems store customer data differently, we may need to map them using external rules.</li></ul><p>This step is crucial because entropy starts here. If different teams are using different heuristics for stitching together data, inconsistency becomes inevitable. By explicitly surfacing relationships at this stage, we create a shared foundation for the rest of the process.</p><h3>Step 2: Layering Business Logic on Top of Raw Data</h3><p>Once we understand the structure of the data, the next challenge is <strong>applying business logic</strong> — that is, translating raw data into a representation that aligns with how the business thinks about its operations.</p><p>At Astrobee, we don’t treat business logic as something that needs to be fully defined upfront. Instead, we surface the ability for business users to define and refine business logic iteratively, on a per-query basis.</p><p><strong>Why this approach?</strong></p><ul><li>Business logic is <strong>constantly evolving.</strong> Definitions of revenue, customer engagement, or churn vary between teams and change over time.</li><li>No data model is perfect from day one. A flexible, iterative approach allows organizations to gradually refine their logic without creating bottlenecks.</li></ul><p><strong>How Astrobee surfaces business logic iteratively</strong></p><ul><li>When a user runs a query, Astrobee suggests relevant transformations based on historical queries and metadata.</li><li>If discrepancies exist (e.g., two teams define “active customers” differently), Astrobee surfaces those differences and allows users to align on a single definition or maintain separate ones with documented reasoning.</li><li>Users can <strong>refine business logic as they go,</strong> rather than needing a fully locked-in data model before they get value.</li></ul><p>This approach helps balance <strong>speed and consistency.</strong> Business users get fast insights without waiting for a central data team to build out a perfect model, but the system still captures and enforces logical consistency over time.</p><h3>Step 3: Meeting Analytics Needs While Controlling Entropy</h3><p>The final step is ensuring that, as analytics needs evolve, the system <strong>actively reduces entropy rather than increasing it.</strong></p><p>Most organizations experience data entropy because users:</p><ol><li>Don’t know what’s already available.</li><li>Genuinely need something different but have no structured way of capturing that difference.</li></ol><p>At Astrobee, we tackle this by detecting when data models are diverging and making that divergence explicit:</p><ul><li>When two teams use different definitions for a metric, Astrobee notifies users and provides context on the discrepancy.</li><li>If a new data transformation is created that mirrors an existing one, Astrobee surfaces that overlap to prevent redundant work.</li></ul><p>Over time, as edge cases arise, the system <strong>logs and learns,</strong> capturing most scenarios so that teams don’t have to reinvent the wheel every time.</p><p>This means that instead of enforcing rigid centralization or allowing unchecked decentralization, we let divergence happen, but make it transparent, explainable, and ultimately resolvable.</p><h3>A Living, Adaptive Data Model</h3><p>Reducing entropy isn’t about locking data models in place — it’s about creating a flexible system that evolves with the needs of your organization and users.</p><p>At Astrobee, we:</p><ul><li>Identify what exists in source data using machine learning and human input.</li><li>Allow business users to iteratively define logic rather than forcing rigid, pre-defined models.</li><li>Proactively surface divergence so that inconsistencies are addressed before they become systemic problems.</li></ul><p>This approach ensures that analytics can be both <strong>fast and reliable</strong> — without forcing teams into an artificial trade-off between centralization and decentralization.</p><p>Every organization’s data stack is unique, and there are multiple ways to achieve the same goal. But at the core, the key to reducing entropy is <strong>building systems that make divergence explicit and reconcilable rather than letting it grow unchecked.</strong></p><p>Want to learn more about how we’re tackling this problem? We’d love to chat — drop us a line at <a href="/service/mailto:hi@astrobee.ai">hello@astrobee.ai</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5b364396e58" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How much SQL can your Text-to-SQL SQL?]]></title>
            <link>https://medium.com/@AstroBeeai/how-much-sql-can-your-text-to-sql-sql-6c821b59a49d?source=rss-21f2c43adacd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c821b59a49d</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[AstroBee]]></dc:creator>
            <pubDate>Fri, 17 Jan 2025 20:40:20 GMT</pubDate>
            <atom:updated>2025-02-11T22:41:09.998Z</atom:updated>
            <content:encoded><![CDATA[<p>Lets take a look at what democratizing data really means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qh14Lr81GGRavH7UnspBg.png" /></figure><p>The idea of “democratizing data” in enterprise settings has existed for a while. It’s taken different forms over the years such as self-service BI tools, better access to data teams, and more recently, text-to-SQL AI assistants. Still, the core concept has remained consistent: “Your business needs data to operate, and more people in your organization should be able to use it effectively.”</p><p>This push for making data useful has driven a lot of improvements:</p><ul><li>BI tools are excellent for analyzing clean, well-structured data.</li><li>A well-maintained data stack keeps data well structured — until scaling outpaces your team’s ability to manage that data.</li><li>There are any number of AI-based tools that claim to democratize data access, which for what it&#39;s worth, sometimes do a decent job</li></ul><p>But what happens when you need something more than a simple query? <strong>For most teams working with raw or incomplete data, getting to a point where tools like BI dashboards and text-to-SQL can shine still requires significant data manipulation and modeling.</strong></p><p>This gap — between messy source data and usable, actionable insights — is what we’ll call the <em>presentation layer</em> of your data warehouse. It’s the polished, business-ready version of your data.</p><p>The challenge is that most “chat with your data” tools assume this layer already exists. What they’re really offering is “chat with your clean data.” But what we really need is the ability to chat with <strong>all data</strong> — wherever it is and in whatever shape it’s in.</p><p>Let&#39;s see what this looks like in practice:</p><h3>Case Study:</h3><p>Let&#39;s say you’re a PM building an enterprise event planning SaaS. You just got out of a meeting with your CEO, where he made a few observations:</p><ul><li>It seems our support ticket volume has ramped up across a few areas of our product</li><li>These might represent friction points that are increasing churn</li><li>We need to take a look at how where these support ticket requests are spiking, how they’re related to feature usage, and how this might affect important accounts.</li></ul><p>These are good flags from the boss, but it’s up to us to take a deeper look at the finer details. Let&#39;s take a look at the data available to us. How easy this is will depend on the state of our data, lets take a look at four different scenarios.</p><h3><strong>Level 1: The Promised Land</strong></h3><p>Welcome to data nirvana. Your data team has done the heavy lifting: clean joins, well-defined metrics, and a pristine presentation layer make your life a breeze. They were able to pull together the exact table you need that neatly includes your accounts, what features they use, and number of support tickets per feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6lcxKJblYHEJOtZx7JSng.png" /><figcaption>That’s convenient!</figcaption></figure><p>In this world, we can just add one helper column to calculate the number of tickets per feature used, that way we know which features are being used most often while still requiring a lot of support. Here, with relatively clean data, a text-to-SQL tool shines, easily generating the below.</p><pre>SELECT<br>    feature_name,<br>    SUM(num_tickets) AS total_tickets,<br>    SUM(usage) AS total_usage,<br>    CASE <br>        WHEN SUM(usage) &gt; 0 THEN CAST(SUM(num_tickets) AS FLOAT) / SUM(usage)<br>        ELSE NULL<br>    END AS tickets_per_usage<br>FROM<br>    projection<br>GROUP BY<br>    feature_name<br>ORDER BY<br>    tickets_per_usage DESC;</pre><p>If you have that table handy, you’ll get your answer quite easily! If you haven’t run this query before, we’ll have to do a little more work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKOM_aM6Mgh4_GKSLAlMHQ.png" /><figcaption>We should definitely take a look at Cross-Org Party Planning</figcaption></figure><h3><strong>Level 2: It’s not bad, but…</strong></h3><p>Things aren’t perfect, but they’re manageable. Your team has put effort into getting data into a warehouse and doing basic transformations, so you’re not starting from scratch. However, joins aren’t seamless and your business logic isn’t fully obvious. Shared keys like account_id might not match across all the tables you need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*60arIFxBIlz002uYyPRNIg.png" /><figcaption>Looking through these is going to take a minute…</figcaption></figure><p>Well first let&#39;s start by identifying relevant tables and creating the right joins:</p><ul><li><strong>Support tickets</strong> for issues by account and feature</li><li><strong>Product events</strong> for feature usage metrics</li><li><strong>Accounts table</strong> to link everything together</li></ul><p>Let&#39;s go ahead and write a query to join these datasets. We can try to get a text-to-SQL tool to help, but it will likely need some cajoling to select the correct columns and identify the right joins. Hopefully, we find something like this.</p><pre>WITH<br>-- Subtable for usage calculation<br>usage_table AS (<br>    SELECT<br>        a.account_name,<br>        product_events.feature_name,<br>        COUNT(product_events.user_id) AS usage<br>    FROM<br>        product_events<br>    JOIN<br>        analytics_to_support_user_mapping ON product_events.user_id = analytics_to_support_user_mapping.user_id<br>    JOIN<br>        customers c ON analytics_to_support_user_mapping.customer_id = c.customer_id<br>    JOIN<br>        accounts a ON c.account_id = a.account_id<br>    GROUP BY<br>        a.account_name, product_events.feature_name<br>),<br><br>-- Subtable for num_tickets calculation<br>tickets_table AS (<br>    SELECT<br>        a.account_name,<br>        st.feature_name,<br>        COUNT(st.ticket_id) AS num_tickets<br>    FROM<br>        support_tickets st<br>    JOIN<br>        customers c ON st.customer_id = c.customer_id<br>    JOIN<br>        accounts a ON c.account_id = a.account_id<br>    GROUP BY<br>        a.account_name, st.feature_name<br>)<br><br>-- Combine usage and tickets subtables<br>SELECT<br>    u.account_name,<br>    u.feature_name,<br>    u.usage,<br>    t.num_tickets<br>FROM<br>    usage_table u<br>LEFT JOIN<br>    tickets_table t<br>ON<br>    u.account_name = t.account_name AND u.feature_name = t.feature_name<br><br>UNION ALL<br><br>SELECT<br>    t.account_name,<br>    t.feature_name,<br>    NULL AS usage,<br>    t.num_tickets<br>FROM<br>    tickets_table t<br>LEFT JOIN<br>    usage_table u<br>ON<br>    t.account_name = u.account_name AND t.feature_name = u.feature_name<br>WHERE<br>    u.feature_name IS NULL<br>ORDER BY<br>    account_name, feature_name;</pre><p>This query is a little more involved, but it results in the clean table in <strong>Level 1</strong> that we can easily query and find the answers we need. Still, this situation assumes our data was clean enough to find shared keys and write some clean joins.</p><h3><strong>Level 3: The struggle is real</strong></h3><p>Here, you’re working with <strong>raw source data</strong> from various systems — product analytics, CRMs, and support ticket systems. While you’ve taken basic steps like consolidating data into a warehouse and resolving a few obvious issues, the data is far from analytics-ready. Transforming this messy, siloed data into a usable state requires a lot of work, collaboration, and technical expertise.</p><p>We won’t walk through every step as these would vary depending on where your data is coming from but we’ll call out a few consistent challenges.</p><p>First, data arrives in different formats, for example:</p><ul><li><strong>Product Analytics</strong>: Event-based JSON logs with fields like user_id, event_name, and timestamp.</li><li><strong>CRM</strong>: CSV exports with accounts, users, and customer metadata.</li><li><strong>Support Tickets</strong>: Semi-structured data, often with free-text fields for ticket descriptions.</li></ul><p>We’ll have to then clean and standardize data across systems:</p><ul><li><strong>Deduplicate Records</strong>: Remove duplicate rows in CRMs and event logs.</li><li><strong>Standardize Columns</strong>: Rename mismatched fields like acct_id vs. account_id.</li></ul><p>We can then start aggregating data into meaningful metrics:</p><ul><li><strong>Create Lookup Tables</strong>: Manually map relationships between user IDs and accounts when direct links don’t exist.</li><li><strong>Usage</strong>: Group product analytics data by account_id and feature_name to calculate usage totals.</li><li><strong>Tickets</strong>: Count support tickets by account_id and feature_name for issue tracking.</li></ul><p>This is still a relatively limited list but gives you an idea of some of the steps required to get your data in shape even to begin finding the insights you need. Text-to-SQL unfortunately won’t be able to help with many of these.</p><h3>Does it have to be this way?</h3><p>At its core, the challenge of democratizing data boils down to two simple ideas:</p><ol><li>Bring your data together, whatever state it’s in</li><li>Enable everyone to find answers from that source of truth data</li></ol><p>Whether you’re working with pristine, analytics-ready data or raw, messy source files, the goal should be to empower teams to access and use data without endless preprocessing.</p><p>By bridging the gap between raw data and actionable insights, businesses can truly democratize data and make it a tool for everyone — not just for those with the time and skills to wrestle with it. We’re working on Astrobee to make that vision possible by:</p><ul><li><strong>Dividing the data pipeline into manageable chunks</strong></li><li><strong>Automating each chunk with AI agents</strong></li><li><strong>Enabling your domain experts to provide nuanced context every step along the way, ensuring a shared and accurate source of truth</strong></li></ul><p>If that sounds interesting to you, we’d love to chat. Drop us a line at hello@astrobee.ai !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c821b59a49d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>