<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roberto Coluccio on Medium]]></title>
        <description><![CDATA[Stories by Roberto Coluccio on Medium]]></description>
        <link>https://medium.com/@robertocoluccio?source=rss-f2cfad448222------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j0zR61I0PDlQ6gDk-c_jgA.jpeg</url>
            <title>Stories by Roberto Coluccio on Medium</title>
            <link>https://medium.com/@robertocoluccio?source=rss-f2cfad448222------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:01:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@robertocoluccio/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An example of AI coworker for people who have never queried data]]></title>
            <link>https://medium.com/@robertocoluccio/an-example-of-ai-coworker-for-people-who-have-never-queried-data-694ccda05271?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/694ccda05271</guid>
            <category><![CDATA[data-democratization]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 05:01:01 GMT</pubDate>
            <atom:updated>2026-11-06T14:35:23.145Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Talking to your data doesn’t mean your data understands you. AI knows data and query languages, but it must also learn the specific jargon people use on the job to interpret what they really mean.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfS0GsvnkIhfix41FJqgIQ.jpeg" /><figcaption>Gloves on! Databricks can support Field operators … and understand what they say, even if the data itself wouldn’t.</figcaption></figure><p>An <strong>AI coworker</strong> can work for people who have never queried data, as long as it “speaks” their language before it speaks SQL. Databricks Genie gives you the mechanism to teach it that language, but collecting, persisting, and validating the words is still your job. The accuracy, latency, token efficiency, and user experience can be totally different if you do.</p><p>In the worldwide movement of “<strong>democratizing data and AI</strong>”, data teams connected conversational agents to tables and, sometimes, when lucky, to their metadata, then handed them over to business users who are not familiar with query languages, table structures, and technicalities around joins or window functions. However, this attempt often failed because operators, technicians, and people in the field still ask questions in the language of their job, and those words live in no table. It’s a linguistic gap, because the AI agent knows how to write SQL queries, but it doesn’t necessarily write the right ones when the question doesn’t point straight to the context it needs to answer correctly. This sometimes creates subtle issues, e.g. when these conversational agents reply with absolute confidence reporting numbers and results calculated from the right data … and the wrong assumptions.</p><p>For example, if someone asks about “ARR” is that previous day revenue* 365 or last month * 12? A “anomaly pattern” is something wrong in the “patterns” table, or is a recurring series of similar occurrences in a signal? Which signal? Is a “ revenue “ to be considered gross or net? The space of the possible interpretations is incredibly vast.</p><p>In this article, I would like to discuss how to create an AI coworker for people who have never queried data using Databricks Genie.</p><h3>A practical use case</h3><p>An AI coworker is an AI assistant that works on your team’s data and knowledge, in your team’s language, and with each person’s permissions. In the Databricks Genie world, there are multiple ways of creating an AI coworker to address different requirements:</p><p><strong>Genie One</strong> is Databricks’ AI coworker for business users, and it comes with:</p><ul><li>a chat interface via the <a href="https://docs.databricks.com/aws/en/genie-one/chat">standard Databricks workspace UI</a></li><li>a <a href="https://docs.databricks.com/aws/en/genie-one/desktop">native desktop app</a>, for Mac OS and Windows</li><li>A <a href="https://docs.databricks.com/aws/en/genie-one/mobile">mobile native app</a>, for iOS and Android</li><li>MCP Servers integrations, Skills, Scheduled tasks, and more</li></ul><p><strong>Genie Agents</strong> (formerly known as Genie Spaces) are curated conversational agents built for one specific business area, with their own data boundaries and instructions.</p><p>Both rely on the <strong>Genie Ontology</strong> to augment their available context and better understand our questions and how to answer.</p><p>Now, let’s delve into a practical use case for which I’ve developed, using <strong>Genie Code</strong>, a pipeline generating synthetic data and everything else you are about to read below.</p><h3>5:40 AM — Production Plant</h3><p>It’s the end of the night shift, and the operator opens the plant dashboard, a Databricks App also accessible from the Genie One mobile app.</p><p>🚨 Line 3 is red again! Why? A dashboard can count the stops, but it can’t explain the root cause of anomalies.</p><figure><img alt="A databricks App, with mobile iOS-like look-and-feel, accessed from an iPhone via the Genie One mobile app." src="https://cdn-images-1.medium.com/max/1024/1*inqQNpC6d10W5BXgy3ONMA.png" /><figcaption>The plant dashboard and its embedded Genie Agent, on the phone.</figcaption></figure><p>Gloves on, he switches to the app’s conversational agent and dictates a question:</p><p><em>“Why does the old filler keep starving on nights since we started the magnums?”</em></p><p>In no-jargon human language, this could be rephrased as <em>“Why does machine FIL-03 log many stops between 10 PM and 6 AM since we started to fill in the 1.5L bottles?” </em>… nobody would realistically ask a question like that! I mean, the engineers creating the AI agent would love that, but in the real world it just doesn’t happen.</p><p>That chat above is a Genie Agent <a href="https://docs.databricks.com/aws/en/dev-tools/databricks-apps/genie">embedded in the app</a>, loaded with the crew’s jargon, resulting from a few days spent onsite by the tech/analyst team to collect actual conversations, doing interviews with field workers, to extract snippets of dialect, of factory-floor language, of jargon to then integrate in the Genie Agent instructions and configurations.</p><p>For test purposes, I’ve created two versions of the same Genie Agent: one instructed with the field’s jargon and specific terminology that’s normally not available in the table’s metadata, and one without.</p><p>Here is the same question, asked to both of them:</p><figure><img alt="side by side comparison" src="https://cdn-images-1.medium.com/max/1024/1*gHcKjmKUMkno-fgBnS_lNQ.png" /><figcaption>The difference in the answer returned by the same technology (Genie Agent embedded in a Databricks App), reading the same data, with and without instructing the agent with the field jargon.</figcaption></figure><p>The machine logs its stops as cryptic controller codes, and the crew doesn’t necessarily know which code means a jam. With the crew’s definitions, the agent answered that the old filler is <em>“mostly jamming, not starving”</em> and pointed at its own infeed on the big bottles. Without them, an agent over the same tables blamed bottles not arriving and sent the fix upstream — same data, same technology, different agent instructions, <strong>different impact on the business</strong>.</p><p>No spare parts are available to repair the lane; the operator must submit a work order request. Luckily, he can do it directly in-app, also reusing the conversation context with the agent as the root cause. The app saves it, and the shift can end.</p><figure><img alt="Not just a prettier dashboard: an App can be interactive and provide operators immediate actionability of the insights." src="https://cdn-images-1.medium.com/max/1024/1*-s1QdY6Awk-qUVfDgBJiwg.png" /><figcaption>The operator submits a work order from within the Databricks App.</figcaption></figure><h3>8:00 AM — Maintenance Office</h3><p>The maintenance planner gets her morning brief, a daily task she scheduled in Genie One. It tells her what went wrong overnight, which tickets are open, including the one the operator raised two hours earlier, and which are still waiting for parts. On her iPhone she gets a notification, and in her mailbox the report with attached the PDF version of it.</p><figure><img alt="The 08:00 maintenance brief in Genie One: what went wrong overnight on line 3, the open jobs, and which ones the planning sheet on Google Drive says can be done today." src="https://cdn-images-1.medium.com/max/1024/1*VJYBcnD_ZdXoJHluqhhTOA.png" /><figcaption>The planner’s 08:00 brief, from plant data and a sheet on Google Drive.</figcaption></figure><figure><img alt="The 08:00 maintenance brief in Genie One: what went wrong overnight on line 3, the open jobs, and which ones the planning sheet on Google Drive says can be done today. Here as email report." src="https://cdn-images-1.medium.com/max/1024/1*QTaR8VNjsPJBszBWkkko5Q.png" /><figcaption>Genie One scheduled tasks also send e-mail reports.</figcaption></figure><p>The brief also combines the task planning sheet, taken from Google Drive via the Genie One <a href="https://docs.databricks.com/aws/en/genie-one/external-sources">MCP connector</a>: a reconditioned spare part is on the shelf, two fitters are free, and Line 3 has a planned stop late in the morning, great! She books the repair and, by lunchtime, the ticket is closed.</p><h3>The day after</h3><p>The plant manager asks Genie One desktop app:</p><p><em>“Is L3 back on plan for the magnums since the fix, and what did it do to throughput?”</em></p><figure><img alt="Genie One routing the plant manager’s question, “Is L3 back on plan for the magnums since the fix, and what did it do to throughput?”, to the plant agent, which answers with L3’s plan attainment and cases per running hour before and after the fix." src="https://cdn-images-1.medium.com/max/1024/1*XnSGYIo8KeLtz_RV16ucgg.png" /><figcaption>Same data, a question in the plant manager’s words</figcaption></figure><p>Line 3 went from 84% of plan to 100% the night after the repair, with almost a fifth more cases an hour. Here, Genie One got it right because management terms like plan and throughput already appear in the column names, plus a Unity Catalog Page (more explained below) has helped in translating L3 into proper business semantics.</p><p>Same data, different words: the operator’s, the planner’s, and the manager’s. Sometimes, these people also speak different languages, and that makes it even trickier.</p><h3>What is field jargon, and why can’t a large language model learn it by itself?</h3><p><em>Field jargon </em>is the vocabulary people use on the job: nicknames for machines and places, local abbreviations, shift slang, and shorthand for business rules. It’s specific not just to an industry, but to a company, a region, a plant, a group of people, a culture…the variations can be numerous. This can also include abbreviations, different meanings of the same word depending on the context, sarcastic questions, and much more.</p><p>I split field jargon into three types:</p><ul><li><strong>Aliases</strong> hide an entity or a value, like “L3”. Complementary to Synonyms.</li><li><strong>Rules</strong> hide a filter or a metric, like “micro-stop”.</li><li><strong>Scopes</strong> hide a time window, like “last night”.</li></ul><p>Rules and scopes are business logic disguised as words, so a synonym list won’t catch them entirely.</p><p>Here follows a simple list of examples, split by industry and jargon type according to the above definitions, to raise awareness about how far these misinterpretations could go.</p><p><strong>Healthcare</strong></p><figure><img alt="Jargon examples for Healthcare" src="https://cdn-images-1.medium.com/max/1024/1*2uRK07JO3m_y5Zy5CVAxfA.png" /><figcaption>An example by type of Jargon in the Healthcare industry</figcaption></figure><p><strong>Banking &amp; Finance</strong></p><figure><img alt="An example of Jargon in the Banking &amp; Finance industry" src="https://cdn-images-1.medium.com/max/1024/1*NjHarf5GBc-6TueluMPmzw.png" /><figcaption>An example by type of Jargon in the Banking &amp; Finance industry</figcaption></figure><p><strong>Logistics &amp; Supply Chain</strong></p><figure><img alt="An example of Jargon in the Logistics &amp; Supply Chain industry" src="https://cdn-images-1.medium.com/max/1024/1*1ynOfct0zA57oyVNdW-FAg.png" /><figcaption>An example by type of Jargon in the Logistics &amp; Supply Chain industry</figcaption></figure><p>Going back to the use case reported in this article, Databricks Genie or any other conversational agent can’t know that “the old filler” is the 2009 machine on line 3 because nobody wrote that down in a blog post, on StackOverflow or Reddit, on a website, or even in the metadata of the table containing the machine telemetry.</p><h3>How to teach domain jargon to a Genie Agent then?</h3><p>Collecting comes first: the field explains, and the data team turns it into governed definitions, <a href="https://learn.microsoft.com/en-gb/azure/databricks/genie-agents/tune-quality">fine-tuning</a> the agent quality:</p><ul><li><strong>Spend some shifts in the field or factory-floor.</strong> Domain experts should collect from the field and those actually being the target of the “talk-to-your-data” application to phrase the 20 most common questions, and write them down verbatim.</li><li><strong>Assign an owner to every term from the field.</strong> Metadata and enterprise knowledge are just as important as a data product, they deserve long-term and proper ownership.</li><li><strong>Evaluate accuracy using ground-truth experiments. </strong>Create a set of questions and the expected resulting queries as ground truth, then run benchmarks (over time, as data, instructions, or the jargon itself evolves) to measure the agent’s accuracy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3OIdMzRByRVv2ytVFezbA.png" /><figcaption>Genie Agent experiment, to benchmark accuracy against a ground truth.</figcaption></figure><ul><li><strong>Mine the failures and collect feedback.</strong> A Genie Agent’s <a href="https://docs.databricks.com/aws/en/genie-agents/monitor">Monitoring</a> tab shows what people asked and how they voted. Downvoted answers and quickly rephrased questions are your jargon backlog.</li></ul><figure><img alt="Collect feedback and monitor the Genie agent to identify improvement opportunities." src="https://cdn-images-1.medium.com/max/1024/1*Cacqqd1SX8okj_pCo6aOnA.png" /><figcaption>Genie Agent monitor, where feedback can become actionable to fine-tune the agent quality.</figcaption></figure><p>Persisting comes second, we call it “<a href="https://docs.databricks.com/aws/en/genie-agents/tune-quality#build-a-knowledge-store">Knowledge Store</a>”. I’d give each type its own home:</p><ul><li><strong>Aliases → Unity Catalog Pages (accessible by Genie One only, as of today).</strong> A <a href="https://docs.databricks.com/aws/en/uc-semantics/pages">Page</a> is a governed definition of a term, an entity or an acronym, with an owner and synonyms: “L3”, “line 3” and “L three” can all point to line PKG-L03. It can include multi-language jargon. Genie One prefers a Page’s definition over inferred context and cites the Page. As mentioned, they are the explicit/provided knowledge pieces of <a href="https://docs.databricks.com/aws/en/genie/genie-ontology">Genie Ontology</a>. This shared business-context layer combines what you govern in Unity Catalog with the context it infers from dashboards, queries, and also Genie Agents. Inside a Genie Agent, aliases are column synonyms or filters with synonyms.</li></ul><figure><img alt="A Unity Catalog Page for packaging line 3 that lists L3, line 3, L three and linea 3 as synonyms" src="https://cdn-images-1.medium.com/max/1024/1*AvHYYpLVqSFvhbHcvIuAlg.png" /><figcaption>A Unity Catalog Page for Line 3, with its aliases as synonyms.</figcaption></figure><ul><li><strong>Rules → metric views (Genie One and Genie Agents).</strong> Define <em>“micro-stop”</em> once, as a governed measure in a <a href="https://docs.databricks.com/aws/en/uc-semantics/metric-views/">Unity Catalog metric view</a>. Genie One then finds it through the Genie Ontology, and a Genie Agent can use the metric view as a data source.</li></ul><pre>version: 1.1<br><br>source: &lt;my-catalog&gt;.&lt;my-schema&gt;.stop_events<br><br>comment: &quot;Stops on the packaging lines, with the plant rules for micro-stops and downtime&quot;<br><br>dimensions:<br>  - name: line_code<br>    expr: line_code<br>    display_name: Line<br>    synonyms:<br>      - line<br>      - packaging line<br><br>  - name: machine_id<br>    expr: machine_id<br>    display_name: Machine<br><br>  - name: stop_start<br>    expr: start_ts<br>    display_name: Stop start<br><br>measures:<br>  - name: micro_stops<br>    expr: COUNT(1) FILTER (WHERE stop_category = &#39;UNPLANNED&#39; AND duration_sec &lt; 300)<br>    comment: Unplanned stops shorter than 300 seconds<br>    display_name: Micro-stops<br>    synonyms:<br>      - micro-stop<br>      - micro stops<br>      - short stops<br>      - microfermate<br><br>  - name: downtime_minutes<br>    expr: SUM(CASE WHEN stop_category = &#39;UNPLANNED&#39; THEN duration_sec ELSE 0 END)<br>      / 60.0<br>    comment: &quot;Minutes of unplanned downtime; changeovers, cleaning and maintenance\<br>      \ are planned stops&quot;<br>    display_name: Downtime minutes<br>    synonyms:<br>      - downtime<br>      - minutes down<br>      - down time</pre><ul><li><strong>Scopes → the Genie Agent semantic layer (Genie Agents).</strong> Shift patterns vary by plant, so <em>“last night”</em> could belong to each plant’s agent: a filter under Configure &gt; Examples, with its synonyms and instructions for defining that “translation layer” specific to the agent’s domain.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*0SM5lWGxo5y03WpteQ9czQ.png" /><figcaption>“Last night” as a filter, with synonyms and instructions, in the plant agent.</figcaption></figure><figure><img alt="Instructions for the Genie Agent." src="https://cdn-images-1.medium.com/max/1024/1*i74nYzrehQpmTS-4kxZL0g.png" /><figcaption>Instructions for the Genie Agent provide the translation layer for most of the field’s jargon.</figcaption></figure><h3>Does Databricks Genie understand field jargon in other languages?</h3><p><a href="https://docs.databricks.com/aws/en/genie-agents/concepts#language-support">Yes</a>, with a catch. Genie Agents work in languages such as Portuguese and French, but the agent framework wraps prompts in English.</p><p>I asked the plant agent, in Italian, <em>why “la vecchia riempitrice” keeps starving at night</em>. It answered in Italian and found the jams, but quoted my English definitions word for word in the middle of the Italian answer. Databricks recommends writing metadata in your users’ language, so include local terms.</p><h3>What mode should I use to query an agent?</h3><p>As a final piece of advice, remember to choose the right “<a href="https://docs.databricks.com/aws/en/genie-agents/concepts#choose-the-right-mode"><strong>mode</strong></a>” (Agent/Chat) BEFORE starting a conversation with an agent to get the right balance between cost, latency, and the depth of the response/analysis.</p><p>While the “Chat” mode is for questions about structured data that Genie can answer quickly with a single SQL query, the “Agent” mode is for longer and more complex, exploratory questions, such as root cause analysis, deep dives, etc., and returns a detailed report with citations, visualizations, and supporting tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZPHSk46nBTwBDMoTpRSMA.png" /><figcaption>Agent and Chat mode: choose the right one, balancing speed, cost, and depth of analysis, at the beginning of a conversation with Genie</figcaption></figure><h3>Wrapping up</h3><p>Before an agent goes live on a plant floor or with a field team, ask one question: <em>“Who has spoken with the target users and collected their jargon?”</em> If people start looking at each other with doubtful faces or the answer is <em>“We will do it after the rollout”</em>, pause the release for a day and let the engineers read this article first.</p><p>What examples of jargon in your field would definitely be misunderstood by an agent without context? Please, share that in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=694ccda05271" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Source Doesn’t Mean You’re Free. Managed Service/Closed Source Doesn’t Mean You’re Locked-in]]></title>
            <link>https://medium.com/@robertocoluccio/open-source-doesnt-mean-you-re-free-managed-service-closed-source-doesn-t-mean-you-re-locked-in-7eee088c8be0?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/7eee088c8be0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[data-strategy]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 12:53:32 GMT</pubDate>
            <atom:updated>2026-07-24T12:53:32.697Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Does your “open source” architecture actually deliver </em><strong><em>freedom</em></strong><em>? For an enterprise, lock-in isn’t a binary choice; it is a spectrum of exit costs that must be weighed against total TCO — specifically the hidden weight of OPEX. This piece aims to challenge the common narrative, exploring how open source can generate significant operational gravity while certain commercial stacks remain surprisingly portable.</em></p><blockquote>[<strong>Disclaimer</strong>: the opinions expressed in this article are my own and do not represent any official statement from my employer]</blockquote><p>I have been doing this for more than a decade as a Consultant on big data, Cloud, and Platform architectures, migrations, governance, and now AI … the whole catalog: every other one or two quarters, someone in a meeting opens a slide with two columns. “Open source” on the left, green checkmarks ✅. “Proprietary” on the right, red crosses ❌. The conclusion is supposed to be obvious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1XCxdnqQT5ehANsnDrkAhw.png" /></figure><p><strong>It isn’t.</strong> The slide is wrong, or at least <em>it is the wrong slide</em>.</p><p>Having spent a lot of time in my career advocating for “lock-in-free architectures” across diverse industries, teams, tech stacks, and cloud environments, I now possess a comprehensive view of this dynamic. Joining Databricks has now also provided me with a full product and portfolio lens. Consequently, I can clearly see both perspectives: the business side, which battles the fear of lock-in while balancing operational costs, talent shortages, immediate time-to-market, and ROI; and the vendor side, which concentrates on integrating an expanding catalog of products into efficient, seamless, and rapidly deployable production frameworks for the enterprise.</p><p>The purpose of this article is to clarify, with examples and references, what I personally believe is a misconception, as per the title:</p><p><strong><em>Open Source Doesn’t (Necessarily) Mean You’re Free. Closed Source Doesn’t (Necessarily) Mean You’re Locked-in.</em></strong></p><p>I also want to thank my network of present and former colleagues for the intense and insightful conversations we have more and more often on this topic, which inspired me to write this piece.</p><p>—</p><p>So, we were saying that Open Source doesn’t automatically mean <strong>freedom</strong>, just like Closed Source doesn’t automatically mean a <strong>cage</strong>. What actually decides where you end up are the things nobody puts on the slide:</p><ol><li><strong>how much does it cost</strong> <strong>to integrate</strong> this tool into the rest of your stack today, and</li><li><strong>how much does it cost to leave when the time comes</strong>? Because the time comes. <em>Always</em>.</li></ol><p>Let’s elaborate together on these two points.</p><h3>A quick word on definitions</h3><p>The <strong>Open Source</strong> Initiative has a <a href="https://opensource.org/osd">very specific definition of open source</a>. The license must give you the source code, plus the right to modify and redistribute it for any purpose, including commercial use. Important and useful. But notice what the definition itself does <em>not</em> hand you: a healthy community, a sane upstream roadmap, more than one significant maintainer, <em>or any of the integrations you actually need</em>. The license tells you what you are <em>allowed</em> to do. It says nothing about what you can <em>afford</em> to do.</p><p><strong>Lock-in</strong> has its own <a href="https://en.wikipedia.org/wiki/Vendor_lock-in">clean definition</a>: <em>a situation in which switching to another vendor incurs substantial switching costs</em>. That word “<strong>substantial</strong>” is doing a lot of work and the costs you need to consider come from a long list of sources: proprietary data formats, APIs, and tooling. But it can also include the integrations your team built around the product, the skills your engineers have acquired, and the downstream pipelines and consumers that depend on the current behavior. Only some of that has anything to do with whether the source code is on GitHub.</p><h3>I have watched open source lock people in</h3><p>You might already have seen this happen without noticing: open source can also lock people and companies in. The most frequent ways for this to unfold are, in my experience:</p><ol><li><strong>Relicensing</strong>: the project starts with an open license, gets traction, and then transitions into a commercial one</li><li><strong>Operational Gravity</strong>: an open source tool or framework still needs to be installed, operated, and maintained by somebody in your organization</li><li><strong>Proprietary surfaces inside (and outside) open source</strong>: when the surrounding container is open source, but some internals are proprietary</li><li><strong>A combination of (open source) tools</strong>: the compounding effect of operational gravity multiplied by the number of tools to combine and integrate together</li></ol><p>Let’s analyze these points one by one in detail with examples.</p><h3>Relicensing</h3><p>You pick a once-permissive project precisely because it is open, then watch the license change under your feet a few years later. All the examples below sit firmly in the data, streaming, or AI territory:</p><ul><li><strong>MongoDB</strong> moved its Community Server from AGPL to the SSPL in <a href="https://www.mongodb.com/company/newsroom/press-releases/mongodb-issues-new-server-side-public-license-for-mongodb-community-server">October 2018</a>. The OSI never recognized SSPL as an open source license, and MongoDB withdrew its submission for approval in 2019.</li><li><strong>Confluent</strong> relicensed <a href="https://www.confluent.io/blog/license-changes-confluent-platform/">KSQL, Schema Registry, REST Proxy, and several Confluent Platform connectors</a> from Apache 2.0 to the Confluent Community License in December 2018. Apache Kafka itself stayed under the Apache 2.0 license at the ASF. Still, anyone who had standardized on the Confluent distribution discovered that parts of what they relied on now had explicit hosting restrictions.</li></ul><p>Just to cite a few, but there are stories about <a href="https://www.elastic.co/pricing/faq/licensing">Elasticsearch and Kibana</a>, <a href="https://redis.io/blog/redis-adopts-dual-source-available-licensing/">Redis</a>, <a href="https://www.hashicorp.com/en/license-faq#who-is-impacted">Terraform</a>, and many others, all following the same pattern: in each of these cases, customers who had picked those projects to escape vendor concentration ended up with exactly the problem they were trying to dodge. The OSS community sometimes reacted to these relicensing operations with OSS forks (e.g., <a href="https://opentofu.org/">OpenTofu</a> fork of Terraform), thus theoretically “solving” the licensing problem, but… solving the lock-in one? Are community-driven forks of mainstream projects really worth being the foundation of your next Enterprise Data &amp; AI strategy?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yzBYx1L-JC4gV72le9X_LA.png" /></figure><h3>Operational gravity</h3><p>Self-hosting Kafka, Airflow, Spark, Iceberg, dbt, and friends is legitimate, but has a drawback: it implies having also a multi-engineer platform team, a release-coordination calendar, and a thick pile of internal documentation that exists nowhere else in the world — because it’s your custom-tailored and integrated stack. <strong>The licenses are free; the engineers, runbooks, upgrade choreography, security hardening, and on-call rotations are not</strong>. The dominant “live” costs of an open source stack live in:</p><ul><li>Integration</li><li>Customization</li><li>Maintenance (bug fixes — are you sure you can fix yourself, or wait until it’s fixed and released in OSS? Upgrades, especially major ones — they may require a significant amount of work)</li><li>On-call</li></ul><p><strong>Vendor invoices become payroll</strong>.</p><p>I have seen this so many times: the OPEX costs for the platform team were 5 to 10x the infra costs, delivery was not fast enough, and fear of change was high due to re-wiring needs. Then, with people turnover and legacy knowledge fading, what was a no-lock-in quickly became a legacy-knowledge-no-innovation-slow-delivery lock-in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ItzA5dVur7aeOJ9VtGA9Q.png" /></figure><h3>Proprietary surfaces inside open source projects</h3><p>This one is more subtle. Take ClickHouse: it is Apache 2.0, so the license checkbox is firmly green. But the on-disk format used by the MergeTree engine is <a href="https://altinity.com/blog/the-future-has-arrived-parquet-on-iceberg-finally-outperforms-mergetree">understood only by ClickHouse itself</a>: not Parquet, not Iceberg, not Delta. ClickHouse can read and write Parquet at the edges. Still, its native storage format is not an open standard, which means the day you <em>choose</em> to query the same data from Spark, Trino, DuckDB, or any cloud warehouse, you are not flipping a config: you are running an ETL job to convert (read: duplicate somewhere else) every byte you ever wrote.</p><p>On top of that, fine-grained access control, row and column security, lineage, audit, data quality, and data catalog are not first-class enterprise concerns of the project, so you bolt on Apache Ranger or Apache Polaris or roll your own — more glue, more on-call, <strong>more lock-in to your own glue</strong>.</p><p>The license is open. The data and the surrounding control plane are not free to leave.</p><p>Another example could be <a href="https://docs.datastax.com/en/dse/6.9/get-started/get-started-dse.html#AdvFunctionality">Apache Cassandra vs. the commercial offering of DataStax Enterprise (DSE)</a>: while the former provides a basic open source core, you are forced to rely on the commercial counterpart if you want enterprise-grade features like advanced security, replication, indexed search, and other proprietary-only extensions. Not to mention the enterprise support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMDSKvO1G3nFLhMvt-9mFg.png" /></figure><h3>“Talk-to-your-data” lock-in</h3><p>Let’s take the “Talk-to-your-data” use case as an example, as it’s the freshest, and the one I see eating the most calendar lately. The brief sounds innocent: “<em>Let business users ask questions in natural language and get answers from our data</em>.” Half a sentence on a slide. Six months of engineering.</p><p>You wire up an open source text-to-SQL agent stack. A vector store for schema documentation, sample queries, and the business glossary. The embedding model of the quarter. An orchestration framework. A schema-linking (semantic) layer that maps natural-language entities to columns. A reranker. Prompt templates you tune over weeks of trial and error. A handful of MCP servers wrapping your warehouse(s) and metric layer. A SQL-validation pass. A hallucination guard. An evaluation harness on top of MLflow or LangFuse. A homegrown agent loop holding the whole thing together. Every component is open source. Every component is, on paper, replaceable.</p><p>Then the fun begins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1otziNJe6rNNefMvrgPB8g.png" /></figure><p>Change the embedding model, and every chunk of schema documentation, every certified query, and every glossary entry has to be re-embedded and re-indexed; embedding spaces are model-specific and not interoperable across providers.</p><p>Change the orchestration framework and your chains, retrievers, callbacks, and tool definitions all change shape; your code does not port over unmodified.</p><p>Change the vector store and ingestion, the indexing strategy, and the latency budget; all get redesigned.</p><p>Change the underlying LLM and prompts; evals and tool-calling formats also need retuning. None of these projects is closed.</p><p><strong>The lock-in lives in the <em>combination</em></strong>, in the embedding you committed to, in the schema-linking heuristics you wrote, and in the bespoke glue between every layer. Your team owns all of it. <em>Forever</em>.</p><p>And after those six months and the work of 4 people, you have a “talk to your data” agent that probably works on twenty tables for the first 3 months. The business has thousands. Hundreds more every other quarter.</p><h3>Managed Service/Closed source can be surprisingly portable</h3><p>Now flip it. A commercial managed product that stores its data in open formats, exposes open APIs, releases parts of its products as open source to foster an ecosystem of possible native integrations, and integrates cleanly with what you already run via open standards/protocols is, in practice, easy to leave. It’s not always the same for every commercial product, of course, that’s why you must choose wisely.</p><p>Your data sits in Delta Lake or Iceberg for OLAP, and in Postgres for OLTP. Your queries are plain SQL. The accumulated engineering expertise is not necessarily locked to the specific vendor. I will be honest, Databricks is a commercial platform and has its own proprietary surfaces, like Photon, runtime optimization on top of Spark, some unique Unity Catalog or Open Sharing capabilities, and its own BI layer.</p><p>The point of this article is not “Databricks is open and therefore free of lock-in”.</p><p>The point is that the data lives in open formats (Delta, Iceberg, Postgres), the query language is SQL, and the engineering skills (Spark, Python, MLflow, Unity Catalog) can be transferred to other platforms.</p><p>Those are the surfaces that determine the cost of leaving. Proprietary runtime optimizations are a tradeoff customers can knowingly accept in exchange for performance and operational simplicity, but they should accept them knowing what they are.</p><p>The day you decide to change it, you change it. Painful? Yes. But the data is already in an open format another engine can read, the queries are already SQL, another engine can run, and the skills your team built, e.g., SQL, Python, Spark, MLflow, transfer to whatever comes next. The switching cost is real and unavoidable (by the way, regardless of your stack!), but it lies in the migration project, not in the data or people layers. That’s a different cost category than rebuilding a hand-assembled OSS stack from scratch.</p><p>The two-column slide hides where actually <strong>lock-in</strong> lives in:</p><ul><li>storage (formats, location, accessibility)</li><li>integration capabilities / available APIs</li><li>accessibility to tech AND business users</li><li>slowness of innovation</li><li>high OPEX</li><li>lack of talent availability</li><li>legacy knowledge</li><li>and “isolated open silos”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7NshAKqBmWQdHGufIrxuA.png" /></figure><h3>Migrations happen on a clock</h3><p>Here is the bit architects know and PowerPoint forgets: enterprises migrate. Constantly.</p><p>Hardware refresh cycles run <a href="https://horizontechnology.com/news/data-center-hardware-refresh-cycles/">3 to 5 years for compute and storage, 5 to 7 for networking</a>. Application portfolios churn faster than those under the joint pressure of cloud, AI, and regulation. New trends and opportunities show up, new tools are released every quarter, and a new integration need is at the door in another month. It’s said that enterprise software applications become legacy the day after they are released.</p><p>The takeaway is the same regardless of which study you trust: <strong>you are never <em>not</em> migrating</strong>. There will always be a “legacy platform” being decommissioned, a “current one” being optimized and integrated with newer stuff, and a “future one” being evaluated. Which means that when you make an architecture decision today, you are not picking the tool you will use forever: you are picking the tool whose cost of leaving will land on somebody’s desk in three to seven years.</p><p><strong>YES</strong>, that somebody might be you. The same one today has to answer the business why it takes so long to get access to the data, to query it with natural language, or to integrate a new data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hj9SfiK12rprqBQjXFZwig.png" /></figure><p>So an important strategic question (at least, when drafting an enterprise Data &amp; AI strategy/architecture) should be, in my opinion:</p><p><em>Over the next five to seven years, how much of my team’s calendar does this choice consume, first to integrate, then to operate, and eventually to migrate?</em></p><p>BONUS: <em>And what will my competitors do in the meantime?</em></p><h3>The full-stack tax</h3><p>Also, please, stop thinking about one tool. Picture instead the actual shopping list for a modern enterprise data and AI platform, and decide for each layer whether you build it, buy it, or assemble it from open source projects.</p><p>The list is not short:</p><ul><li>Object storage and table format</li><li>Distributed computing for batch <em>and</em> streaming. ETL and ELT</li><li>A SQL query engine</li><li>Orchestration</li><li>Catalog with lineage tracking and auditing</li><li>Security layer for fine-grained access control</li><li>Data quality</li><li>Identity management and SSO</li><li>Machine Learning models training, experiment tracking, model registry, model serving, model monitoring.</li></ul><p>On top of all that, the new generative-AI surface:</p><ul><li>Agents (and their memory, guardrails, orchestration)</li><li>LLMs (and their usage, observability, monitoring, cost management)</li><li>RAG</li><li>Vector Search</li><li>“Talk to your data” interfaces</li><li>Applications</li><li>Skills, MCP servers, and the prompt registry around them.</li><li>A BI/Reporting layer</li><li>Private connectivity all over</li></ul><p>And of course, observability and monitoring for every one of those layers, plus cost attribution (FinOps), plus disaster recovery (sometimes Cross-Cloud).</p><p>You can build every single one of those layers from open source projects.</p><p>The names are well-known: Iceberg, Delta, Spark (with on top — you can’t hide it — your own custom YAML-based ingestion framework), Flink, Kafka, Airflow, dbt, Trino, ClickHouse, MLflow, LangFuse, Ray, KServe, vLLM, Apache Polaris, Apache Ranger, OpenMetadata, OpenLineage, Marquez, Prometheus, Grafana, Keycloak, OPA, LangGraph, MCP servers, plus whichever “natural-language-to-SQL” library is in fashion this quarter. None of these is a bad project. Some are <strong>excellent</strong>.</p><p><strong>The question is what happens when you put all of them in the same room and tell them to behave like one platform</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZxCuQmIihcozRLC1GPlVw.png" /><figcaption>Herding cats</figcaption></figure><p>Three things, all of which appear as line items in the budget the year <em>after</em> the architecture decision was approved.</p><p>First, <strong>integration code</strong>. Each project is internally coherent and externally indifferent. Identity models do not match. Permission models do not match. Metadata models do not match. Lineage stops at the boundary of each tool. Quality rules are expressed in three different ways. Audit events live in three different sinks. To make any of this look like one platform, somebody has to write and own the bridges, and that somebody is your team.</p><p><em>Forever</em>.</p><p>Second, <strong>release coordination</strong>. Every project ships on its own clock. Compatibility matrices across a dozen projects are a small combinatorial nightmare. A non-trivial fraction of platform engineering on an OSS-only stack is making sure that next quarter’s Spark, Iceberg, Polaris, Ranger, Airflow, dbt, and MLflow versions still play nicely together. None of that work is visible to the business, but “maintenance service interruption windows” to upgrade these tools.</p><p><em>And you also do this forever.</em></p><p>Third, <strong>expertise scarcity</strong>. The single Spark expert. The single Airflow expert. The single MLflow operator. The one engineer who keeps the Polaris-and-Ranger glue alive. Each of these people is a single point of failure for a layer of your stack, and the labor market for replacing any of them is brutal. The software is free. The expertise is not, and please don’t fall into the trap of thinking “AI will do it for me” — at least not yet.</p><p><em>This is a forever concern as well.</em></p><p>Add the three together, and what you actually get is a <strong>longer time-to-market</strong>, because every business request now has to pass through that integration layer before it can ship, and an <strong>overall</strong> <strong>higher Total Cost of Ownership (TCO)</strong>, because the integration layer never stops needing care and feeding. Both costs are paid out of the same budget that was supposed to fund “creating business value”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0_ctjGiTjDNojyUzj4U_jQ.png" /></figure><h3>Leaving open source is its own migration</h3><p>Here is the part that catches people out. The instinct says, “<em>If I ever want to move, I have the source code, I can move”.</em> In practice, leaving a hand-assembled open source stack can be as painful as leaving a managed service or Microsoft Excel, sometimes more so, for three concrete reasons:</p><ol><li><strong>The data formats are not always portable</strong>. ClickHouse MergeTree, untouched Hive directory layouts, MLflow runs persisted into project-specific schemas: all of those need conversion before any successor platform can read a byte. Having the source code does not change the bytes on disk.</li><li><strong>The integration code is bespoke</strong>. The bridges between Airflow, Ranger, Polaris, dbt, OpenMetadata, and your custom RBAC layer are documented exactly as well as your team had time to document them, which is to say, partially and — let’s be honest — not always up to date. Whoever inherits the migration has to reverse-engineer that code from scratch, and they get to do it without the people who originally wrote it.</li><li><strong>The skills are stack-specific</strong>. The engineer who is fluent in Kafka, your Spark physical plan, your Airflow DAGs, and your custom authentication layer cannot just transfer that fluency to a different stack. Retraining time is a real cost, and it is paid in calendar quarters, not weeks.</li></ol><p>I know what you’re thinking… YES, it would be similar in the case of migrating from one vendor to another… so we do agree on the title of this section, right?<strong> A product can do the integration work for you during the build years</strong> and <strong>keep your data and skills portable during the leave years</strong>. Those are the two axes that matter.</p><h3>A better slide</h3><p>If I had to redraw the comparison, I would retire “open vs. closed” entirely and put two questions on screen instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Rb1yDleTKPohyPpnFKaCA.png" /></figure><p>A tool can be fully open source and score badly on both questions. A commercial product can score well on both. The license or cost of the managed service is one input among many. An integrated portfolio can dramatically lower integration cost during the build years, and that’s a real economic benefit worth paying for. But that is not the same thing as being free to leave. <strong>What determines whether you’re locked in is whether your data, formats, and skills travel with you when you <em>choose</em> to migrate</strong>. <em>Choose</em> vendors that score well on both axes; integration cost is low, exit cost is low, and be honest that those are two separate questions.</p><p>This section could be extended far longer, as taking architectural decisions is an art, refined over decades of experience, but also supported by tools and frameworks, better if quantitative (i.e., I know well it’s not just an open vs. closed comparison, but this article’s focus was on the demystification of this narrative).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eS69XlK7M_wy6TlpUrPfaQ.png" /></figure><p>For example, a more fair and robust approach I’ve sometimes used is the <strong>Weighted Decision Matrix</strong> framework or, especially when a vendor we have already adopted as part of our stack provides what we might need, the <strong>Pugh Matrix</strong> one to set the current platform/vendor as the benchmark baseline, then define the decision pillars that apply to that specific decision, i.e., aligned to the specific use case requirements.</p><p>Finally, quantify (absolute or relative quantities), optionally assign each pillar a weight, and obtain an overall score to drive the decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GNUF8WpFcm9Ybv07RbNX-Q.png" /></figure><p>But I don’t want to go off-topic here, so I’ll leave it to the reader to dive deeper as a follow-up if necessary.</p><h3>In the end</h3><p>Open source matters. I have spent a decade using, deploying, and operating it, and I am not stopping. <strong>Databricks</strong> itself <a href="https://www.databricks.com/product/open-source">is built on open source</a>, and it has a dedicated principle in its “<a href="https://docs.databricks.com/aws/en/lakehouse-architecture/interoperability-and-usability/">Well Architected Framework</a>” specific to interoperability and usability, which I believe are the real foundation of an “<em>open (read: not locked-in) architecture</em>”.</p><p>Furthermore, in the recent 1st day keynote of the 2026 Databricks Data + AI Summit (check out the <a href="https://www.databricks.com/dataaisummit/session/tuesday-keynote">recording</a>, if you haven’t watched it), the Databricks CEO, Ali Ghodsi, presented a panel that resonated a lot with this article’s scope, I think:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEIql4lSYkUOYptBNfjOcw.png" /></figure><p>I think the pillar “<strong><em>Choice</em></strong>” there on the right is what I’ve tried to unfold in this article, besides the more obvious, but not to underestimate — “<strong><em>Cost</em>”</strong> one: no lock-in means having choices for interoperability, integration, migration, and evolvability. Today more than ever, you don’t want to be locked-in with <em>one</em> AI, one foundation model, one LLM vendor, one API, one vertical — I think we can agree on this at this stage — <strong>BUT</strong> the point is: the best alternative is not necessarily and only the full open source way.</p><p>“<strong><em>Control</em></strong>” is tightly coupled, because a messy integration among open source “free” tools lacks the enterprise-grade control and governance you are looking for (not necessarily when experimenting, but — trust me — BEFORE going into production with an Agentic system). “<strong><em>Context</em></strong>” is the glue that makes “n-tools-integrated-together” a real AI-ready, enterprise-grade platform, but this topic may be a deeper analysis for another article.</p><p>I’d like to wrap up recommending fellow architects and tech leaders to drop the <em>religion</em>, as the narrative “Open license equals freedom, closed license equals a cage” has been costing more team-years than almost any other myth in our industry. The next time the two-column slide shows up in a meeting, ask one question: over the next migration cycle, where exactly will the effort and the lock-in actually live?</p><p>The version of you running the migration five years from now will be grateful. The business users of today will be as well.</p><p>—</p><p><em>Roberto Coluccio is a Delivery Solutions Architect at Databricks. The views in this post are his own.</em></p><p><em>Thank you João “Jota” Martins, Rouzbeh Rahimi, and Alex Ott for the valuable review and comments.</em></p><p><em>Originally posted in the </em><a href="https://community.databricks.com/t5/technical-blog/open-source-doesn-t-mean-you-re-free-managed-service-closed/ba-p/156800"><em>Databricks Technical Community Blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7eee088c8be0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake World Tour Berlin 2025: 10 Honest Takeaways You Won’t Find in the Slides]]></title>
            <link>https://medium.com/@robertocoluccio/snowflake-world-tour-berlin-2025-10-honest-takeaways-you-wont-find-in-the-slides-0fffe47d557e?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fffe47d557e</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-as-a-product]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 21:00:45 GMT</pubDate>
            <atom:updated>2025-10-02T10:13:51.021Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GDK5irqSnVNdcC-vHAINw.jpeg" /></figure><p>Today, I attended the Snowflake World Tour — Berlin, 2025 edition.</p><p>I wasn’t sure what to expect — I’ve never really worked hands-on with Snowflake before. But the event turned out to be insightful, giving me both confirmations and new perspectives.</p><p>Here are the 10 takeaways that stood out the most:</p><p><strong>1. Everyone Defines “Data Product” Differently — And That’s Okay</strong></p><p>Every company has its own definition of a <em>data product</em>. That’s fine — as long as it’s shared, agreed upon, and understood across the organization. What matters is whether it delivers value to the “customers” of those products. Minimum viable product features should be part of governance and engineering review, not just an afterthought.</p><p><strong>2. Ownership of Source-Aligned Data Products Is Still Messy</strong></p><p>Too often, data platform teams end up owning source-aligned data products because source-system domains aren’t motivated or incentivized. Analytical spaces are still seen as IT cost centers, while business domains “mind their own business.”</p><p>This isn’t a tooling problem — it’s cultural and organizational. Companies that succeed invest in people, platform foundations, and literacy.</p><p><strong>3. Autonomy Without Standards = Hidden Risk</strong></p><p>Autonomy can backfire if guidelines are ignored or standards aren’t enforced. Best practices today are scattered across docs, multiple Git repos, and different tools/platforms, making them hard to follow or maintain consistent.</p><p>Central platform teams may try to help, e.g. a speaker mentioned their presence as enabling team of Subject Matter Experts during data products design/development, but success often still depends on the human factor, which doesn’t scale.</p><p><strong>Governance, guardrails, and best practices must be built-in and by-design, enforced computationally by the platform</strong>. Snowflake’s integrated approach reduces some complexity, but the risk remains unless governance is end 2 end and native to the platform.</p><p><strong>4. Snowflake and Databricks Look More and More Alike</strong></p><p>Snowflake and Databricks are converging feature-wise. It’s clear that both platforms are increasingly positioning themselves as universal data intelligence hubs, not just warehouses or analytics tools. This blurs vendor differentiation for enterprises and raises questions about real openness versus integrated experience (see next 2 points as follow-up).</p><p><strong>5. The Word of the Day: “Easy”</strong></p><p>“Easy” was pitched in almost every presentation. Snowflake’s all-in-one approach reduces entry friction, avoids the complexity of maintaining and tuning open-source tools, and provides features like lineage, AI-augmented governance, and metadata-powered catalogs out of the box.</p><p>Enterprises seem willing to trade potential long-term lock-in for immediate ease and integration, which explains why “easy” adoption resonates so strongly.</p><p><strong>6. From Databases to Data Lakes… and Back Again</strong></p><p>Database → Data warehouse → Data lake → Cloud warehouse → Cloud lakehouse → Lakebase → … than what?</p><p>OLAP and OLTP are both needed for AI-driven analytics, and the HTAP (Hybrid Transactional/Analytical Processing) concept is an increasingly important pattern aiming to integrate these workloads. Many attempts in the past years of implementing this unification within a single database failed, now we see platforms like Snowflake and Databricks offering more and more both capabilities with a virtualized seamless integration experience on top.</p><p>Traditionally, analytical platforms were not considered first-class citizens in the CIO agenda — they were seen as supporting reporting and non-critical IT tasks. With the rise of AI-driven applications powered by these platforms, that perception must change: <strong>analytical platforms now provide critical business services</strong>, not just reporting.</p><p>This also means that support processes, change management, and implementation of end-to-end observability must become central topics.</p><p><strong>7. Data Quality Still Checked After the Fact</strong></p><p>Data quality controls remain largely reactive: dashboards and automated checks catch issues after <em>datastrophes </em>(as a dear friend of mine called them during a conference talk on this topic) occur. Resistance to <a href="https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/"><strong>Write-Audit-Publish</strong></a> (WAP) patterns persists, leaving organizations vulnerable. The reason can be the lack of a seamless integration with git-like storage versioning (e.g. <a href="https://lakefs.io/">https://lakefs.io/</a>).</p><p>Other platforms, like Palantir Foundry, offer a “Branching” feature that I see as extremely valuable. In production, each ETL job could output data as a branched version of the main dataset (<strong>Write</strong>), then undergo <strong>Audit </strong>checks for all data quality rules, and only once validated, merge the branch into the main dataset (<strong>Publish</strong>). This approach prevents consumers from ever reading data that doesn’t meet quality standards, and the same concept can be applied during development and testing — even in production — making WAP fully practical.</p><p>I remember <a href="https://www.dremio.com/newsroom/polaris-catalog-to-be-merged-with-nessie-now-available-on-github/">news</a> about Snowflake integrating <a href="https://projectnessie.org/">Nessie</a> into Polaris, but I haven’t heard a single word today on it and even talking about it with a Snowflake engineer lead to a dead end.</p><p><strong>8. Notebooks, Scripts, and SQL Everywhere</strong></p><p>Notebooks, ad hoc scripts, and SQL statements triggering Python code that calls LLMs — productionizing these workflows is still messy. Testing, governance, and reliability remain significant challenges, which are usually glossed over in presentations.</p><p>I’ve seen a SQL statement calling out Python code prompting out an LLM: seriously guys, how do you test and productionize this stuff?</p><p><strong>9. Data Mesh: Still Niche</strong></p><p>I was surprised how few people knew what Data Mesh is. Compared to the nearly universal buzz around data products, this shows that some emerging concepts are still aspirational rather than widely understood or adopted.</p><p>Many talks mentioned the decentralized ownership pillar (although point 2 considerations), really a few self-service platform embedding the computational governance.</p><p><strong>10. Ontology / Semantic Layer Absence</strong></p><p>Interestingly, none of the talks today mentioned the use of ontology models or semantic layers to empower the platform. This is a major differentiator with platforms like Palantir Foundry, where semantic layers and ontologies are central.</p><p>Snowflake is moving in this direction with its Semantic Modeling Language (SML), which powers Cortex AI (<a href="https://www.snowflake.com/en/blog/open-semantic-interchange-ai-standard/">see more here</a>). While still early, this shows potential to bridge the gap between raw data and semantically rich, AI-ready datasets.</p><p><strong>Final Notes</strong></p><p>Overall, the event was well organized, with balanced sessions across six tracks. Most sessions were high-level business/strategic, so hardcore techies might have wanted more.</p><p>Lowlight: one presenter literally read through their notes the entire session. The audience quickly lost interest, no one engaged with the content, and many left. Reflection: even the best-marketed product loses its impact if delivered poorly — was the effort of presenting that session worth it?</p><p>Attending was still worthwhile. I walked away with sharper perspectives on ownership, culture, platform convergence, and how integrated AI and apps are reshaping analytical platforms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fffe47d557e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boost Data Governance with Governance Decision Record]]></title>
            <link>https://medium.com/agile-lab-engineering/boost-data-governance-with-governance-decision-record-ca3e4bc70f3a?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca3e4bc70f3a</guid>
            <category><![CDATA[elite-data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[decision-record]]></category>
            <category><![CDATA[computational-policies]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 09:08:20 GMT</pubDate>
            <atom:updated>2023-05-03T12:06:59.115Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5X5hTVATzOqNuHBplBUSWQ.jpeg" /></figure><p>In this article I will present the open-source <strong>Governance Decision Record</strong> (GDR) framework and how it could boost the efficiency and efficacy of Data Governance frameworks.</p><p>We care a lot about the word <em>practice</em> in Agile Lab, as a foundational piece of our <a href="https://www.agilelab.it/blog/elite-data-engineering">Elite Data Engineering manifesto</a>.</p><p>There’s a clear reference and a perfect fit of the Governance Decision Record into the Data Mesh scenario in the form of a Federated Computational practice for Data Governance goals. Still, the idea of structuring an operating model for dealing with the <em>decisions</em> to be taken as part of the enterprise Data Governance accountabilities can very easily be adopted even in many data management contexts and architectures, like the Data Lakehouse.</p><p>Let’s take a look!</p><h3>Table of contents</h3><ul><li><strong>Introduction</strong></li><li><strong>2023 State of Federated Computational Governance</strong></li><li><strong>What is a Data Governance Decision Record?</strong></li></ul><p>a. Policy Lifecycle State</p><p>b. Policy History State</p><p>c. Consequences and Accepted Tradeoffs</p><p>d. Implementation Steward</p><p>e. Where the Policy Becomes Computational</p><p>f. Automation Needs Metadata</p><p>g. Governance Decision Record Example</p><p>h. Federated Decisions</p><ul><li><strong>Wrapping Up</strong></li></ul><h3>2023 State of Federated Computational Governance</h3><p>First, let’s understand very briefly what the modern view of federated computational data governance is. To do so, let’s go over the status quo first.</p><p>Yeah, <strong><em>late</em></strong>.</p><p>In one word, that’s what Data Governance Frameworks have been most of the time in the majority of the enterprises I was able to put my eyes on (I’m Staff Data Architect at an Elite Data Engineering solutions firm). Centralized Data Governance teams usually:</p><ul><li>Performed “offline” (late!) periodic analysis of data to verify compliance policies.</li><li>Used to tag schemas with business terms on data catalogs while data was already there (usable/consumed by clients).</li><li>Went through ad-hoc (per use-case) design decisions for data infrastructure and applications.</li><li>Had to think about data quality metrics for every domain of data while also being responsible for its guarantee.</li><li>Had sometimes to be part of the access management process.</li><li>Had to define the privacy constraints for every domain of data, and much more.</li></ul><p>A long list, which always made the centralized practice inefficient and not scalable.</p><p>Another word that represents typical Data Governance initiative is <strong><em>manual</em></strong>.</p><blockquote>Humans don’t scale out. We sleep, and we make mistakes. Data don’t sleep, so mistakes mean risks and losses.</blockquote><p>High-quality analytical data must be provided with clear ownership and timeliness. They must be interoperable across domains, from both a semantic and technical point of view. They need to be safely but securely accessed and need to be discovered, understood, and especially used with trust and reliability.</p><p>Ok, I’ve brutally summarized several concepts of the Data Mesh paradigm (excuse me Zhamak- but I wanted to get straight to the chase).</p><p>To get all the features above, we need to answer the following questions:</p><ul><li><strong>What</strong> are the standards for interoperability, secure access and privacy, self-serve provisioning, immutability, change management, breaking changes, schema evolution, polysemes, cloud costs management, data cataloging, compliance, etc.? The list goes on.</li><li><strong>Why</strong> are these standards required? We could trade off agility for safety, reliability for compliance, efficacy for efficiency, etc.</li><li><strong>Who</strong> should take the decisions about these standards, or with whom? Who will take ownership of its maintenance along the way?</li><li><strong>Where</strong> and <strong>How</strong> can we apply these standards to our data architecture, design, and engineering practice? How do we monitor and verify they are applied?</li></ul><p>Furthermore, one last question:</p><p>How are effective data governance decisions cataloged, maintained, and shared with the whole organization?</p><p>The answer to this last question is the whole purpose of the <strong>Governance Decision Record (GDR)</strong>. It also provides a framework/tool to answer the others above.</p><p>Make sure to star/fork the GitHub repo:</p><p><a href="https://github.com/agile-lab-dev/governance-decision-record">GitHub - agile-lab-dev/governance-decision-record: The Governance Decision Record (GDR) is a specification model for (computational) data governance policies inspired from the ADR (Architectural Decision Record).</a></p><h3>What is a Data Governance Decision Record?</h3><p>Every decision leads to a series of policies. They need clear ownership but can be discussed with different organizational models. Eventually, decisions can change, evolve, and improve. Such decisions also should lead to automation. These are architectural decisions.</p><p>Automation is what brings us from late to on time. From to be verified to compliant by design. From arbitrary to structural.</p><p>The Governance Decision Record is basically an evolution of the <a href="https://adr.github.io/">Architectural Decision Record</a>. It’s a broadly adopted framework to consolidate architectural decisions, with 2 major improvements dedicated to computational governance:</p><ol><li>It has a specific section to report “where and how the decision becomes a <strong>computational policy</strong>”. It provides clarity on how we automate it, in which piece of the platform, the architecture, the infrastructure the decision is implemented as automation, and who is going to steward such implementation.</li><li>It suggests a <strong>policy-as-code</strong> approach. It adopts a version-controlled repository as a container for both decisions (documents) and the associated policies-as-code (pieces of testable software that implement the decision logic, through a platform of some sort — that’s up to the user)</li></ol><p>A Governance Decision Record contains:</p><ul><li>a policy <strong>lifecycle state</strong></li><li>a policy <strong>history state</strong></li><li>the policy <strong>title</strong></li><li>the <strong>context</strong></li><li>the <strong>decision</strong></li><li>the <strong>consequences</strong> and accepted trade-offs.</li><li>an <strong>implementation steward</strong></li><li>where the policy becomes <strong>computational</strong></li></ul><p>Also, policies can have different scopes:</p><ul><li><strong>LOCAL</strong>: the policy is applied and run within the local scope of the specific data asset (or data product)</li><li><strong>GLOBAL</strong>: where the policy is applied with global scope, run at the platform level for all data use cases (or data products)</li></ul><p>More details in the following sections.</p><p>A <a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/gdr_template.md">Markdown Governance Decision Record template is provided</a>, as a version-controlled document. I believe a Git repo is way more searchable and maintainable (in terms of releases, evolutions, concurrent contribution, and enabler for automation).</p><p>Let’s dive into each section.</p><h4>Policy Lifecycle State</h4><p>This can be as simple as a label tracking down the <strong>lifecycle</strong> state of a policy. Common values are:</p><ul><li><strong>Draft </strong>— when a policy is being developed and still needs to be formally approved or has been submitted for approval.</li><li><strong>Approved</strong> — when a policy has been formally approved. This makes it actionable and a reference for the overall governance.</li><li><strong>Rejected </strong>— when a policy has been formally rejected after the approval process.</li></ul><p>In the <a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/gdr_template.md">Governance Decision Record template</a> file, some pre-compiled web-rendered labels are provided.</p><p>Let’s go over the composing parts.</p><h4>Policy History State</h4><p>This can be as simple as a label tracking down the <strong>history</strong> state of a policy. Common statuses are:</p><ul><li><strong>New</strong> - when a policy is created for the first time, it doesn&#39;t amend or supersede an existing one.</li><li><strong>Amends</strong> or <strong>Amended</strong> - when an approved policy amends or is amended by another existing policy.</li><li><strong>Supersedes </strong>or <strong>Superseded</strong> - when a policy supersedes or is superseded by another existing policy.</li><li><strong>Deprecated </strong>- when a policy ceases to be valid/applied and no other one amends or supersedes it.</li></ul><p><strong>NOTE:</strong> in the case of amend* and supersede* the related policy should be linked.</p><p>In the <a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/gdr_template.md">GDR template</a> file, some pre-compiled web-rendered labels are provided.</p><figure><img alt="here is how the labels look when rendered. Cool, huh ?" src="https://cdn-images-1.medium.com/max/1024/1*5LH3lDdKAcPha-_AFMCplw.png" /><figcaption>The rendered labels. Cool huh?</figcaption></figure><h4>Context</h4><p>This section describes what is the context to which the policy applies (and why).</p><h4>Decision</h4><p>The decision the policy aims to apply. Below, an example will clarify the scope of a possible governance decision.</p><h4>Lifecycle</h4><p>Declare what changes to the metadata (or anything else) would be considered <strong>breaking</strong> and what <strong>not breaking</strong>. This is important to implement automation at the platform level and create a robust change management process based on trust.</p><h4>Consequences and accepted trade-offs</h4><p>What we accept to happen while the policy is applied, including pros (improvements) and cons (impacts, rework, new accountabilities, or requirements).</p><p>Since there’s no “universally optimal decision”, the policy should also report the trade-offs the organization is going to accept with this policy, which could mean in some scenarios making the accumulated tech debt explicit.</p><p>A note on tech debt: it’s usually hidden and hard to track. When making it explicit, it is easier to measure/keep track of the overall tech debt, system quality in terms of architecture and behavior, etc.</p><h4>Implementation Steward</h4><p>The person responsible for taking care of the implementation. We talk about implementation since the policy is supposed to become as “computational” as possible, thus leading to automating the data management practice, probably with the help of a backing platform.</p><p>It can also be the role of accountability to follow the application of such policy.</p><h4>Where the policy becomes computational</h4><p>These are the specific points in the architecture, the platform, the system, the context, etc., where this policy and its checks, if any, are implemented to become <em>automation</em> (thus becoming <em>“computational”</em>).</p><p>This is split into <strong>LOCAL</strong> and <strong>GLOBAL</strong> policy to highlight: while the former assesses the context of a policy locally (i.e., to the context of a specific Data Product) implemented/applied/verified, the latter is for policies globally applied (i.e., at platform level).</p><p>Both can be applied at runtime and deploy time.</p><p>If using a descriptive modelling language, a metadata validation policy-as-code file can be provided (it will probably be integrated into the platform, e.g., using CUE lang for YAML).</p><h4>Automation needs metadata</h4><p>In the example provided in the repo, we assume that a policy becomes <strong>computational</strong> thanks to an enabling platform, driven by machine-readable metadata to model the policy content.</p><p>In the repo, examples leveraging the <a href="https://github.com/agile-lab-dev/Data-Product-Specification">Data Product Specification</a> are presented but, again, the framework is agnostic to whether we are adopting the Data Mesh paradigm or not. We are decentralizing ownership towards domains of the central Data Team that is managing the whole data platform. We are creating Data Products or “just” data assets.</p><h4>Governance Decision Record Example (decision-related to a Data Product’s File output port)</h4><p>A pretty exhaustive example policy and related metadata + policy-as-code validation files are provided in the <a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/example/data-mesh/data-product/output-port/files">example</a> folder of the repo. In this example, the specific GDR is provided to describe how an <em>Output Port</em> of type “FILES” should be defined, provisioned, configured, described, and validated. The folder contains 3 files:</p><ul><li><a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/example/data-mesh/data-product/output-port/files/0001-data-product-output-port-files.md"><strong>0001-data-product-output-port-files.md</strong></a> containing the descriptive Governance Decision Record (as an implementation of the <a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/gdr_template.md">template</a>). In this example, we took inspiration from the Financial Services world (in terms of constraints);</li><li><a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/example/data-mesh/data-product/output-port/files/0001-data-product-output-port-files.cue"><strong>0001-data-product-output-port-files.cue</strong></a> containing the <a href="https://cuelang.org/">CUE lang</a> policy-as-code validation file, that supposedly will be integrated into the supporting platform;</li><li><a href="https://github.com/agile-lab-dev/governance-decision-record/blob/main/example/data-mesh/data-product/output-port/files/0001-data-product-output-port-files-example.yaml"><strong>0001-data-product-output-port-files-example.yaml</strong></a> containing an example of metadata specification with real-world values.</li></ul><p>The Governance Decision Record versioning assumes this is the first policy created to address this governance topic.</p><p>The policy metadata can be validated with the policy-as-code file using the CUE CLI (if installed):</p><pre>cue vet example/data-mesh/data-product/output-port/files/0001-data-product-output-port-files-example.yaml example/data-mesh/data-product/output-port/files/0001-data-product-output-port-files.cue</pre><p>Many more details can be found in the <a href="https://github.com/agile-lab-dev/governance-decision-record">official GitHub repo</a>.</p><h3>Federated decisions</h3><p>This article won’t dive into operating models for federated decision-making, however, the <strong>GDR</strong> with its status labels is ready to be adopted in any Agile-like workflow. Please let me know if you are interested in a follow-up article on this topic.</p><h3>Wrapping Up and Next Steps</h3><p>How do you persist data governance decisions (both as policies and as code)? What specification does such a policy follow?</p><p>The <strong>Governance Decision Record</strong> is a handy open-source option for that, but alone it’s not enough:</p><p>Next step 1 — An organizational and operating model needs to be structured to take <em>federated</em> decisions.</p><p>Next Step 2 — A platform is where policies become <em>computational</em>. Here, we also might have a <a href="https://www.agilelab.it/witboost-computational-governance-platform">solution</a>.</p><p><strong>Thank you for reading. </strong>Please, follow me if you think this article was interesting. It’ll also help you to be notified about new content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca3e4bc70f3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/boost-data-governance-with-governance-decision-record-ca3e4bc70f3a">Boost Data Governance with Governance Decision Record</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Product Specification]]></title>
            <link>https://medium.com/agile-lab-engineering/data-product-specification-50610de8c152?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/50610de8c152</guid>
            <category><![CDATA[product-data]]></category>
            <category><![CDATA[open-metadata]]></category>
            <category><![CDATA[data-product-descriptor]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 12:39:44 GMT</pubDate>
            <atom:updated>2023-04-11T07:33:42.115Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automate your Data Mesh with Data Product Specification</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjSPKoGiRPgvtvRFPJuYsg.jpeg" /><figcaption>In order to fit in a broad scenario, Data Products should comply with well-defined and declarative standards. This is a floor captured in Bamberg (Germany), by Roberto Coluccio — all rights reserved.</figcaption></figure><p>Just like every informative content in the <strong>Data Mesh</strong> area should do, I’d like to start quoting the <a href="https://martinfowler.com/articles/data-mesh-principles.html#CorePrinciplesAndLogicalArchitectureOfDataMesh">reference article</a> on the topic, by <strong><em>Zhamak Dehghani</em></strong>:</p><blockquote>[…] For data to be usable there is <strong>an associated set of metadata</strong> including data computational documentation, semantic and syntax declaration, quality metrics, etc; metadata that is intrinsic to the data e.g. its semantic definition, and metadata that communicates the traits used by computational governance to implement the expected behavior e.g. access control policies.</blockquote><p>A “<strong>set of metadata</strong>” is hereby to be associated with our <strong>Data Products</strong>. I’ve read a lot of articles (I’m gonna report some references along the way in the article) about why it’s important and what theoretically can be achieved by leveraging them… they all make a lot of sense. But — as frequently happens when someone talks about Data Mesh related aspects — there’s often a lack of “SO WHAT”.</p><p>For this reason, I’d like to share, after a brief introduction, <strong>a practical approach</strong> we are proposing at <a href="http://www.agilelab.it/">AgileLab</a>. It’s open-source, it’s absolutely not perfect for every use case, but it’s evolving along with our real-world experience of driving enterprise customers’ Data Mesh journeys. Most importantly, <strong>it’s currently used — in production</strong>, not just on articles — at some important customers of ours to enable the automation needed by their platform supporting their Data Mesh.</p><h3>WHY METADATA ARE SO IMPORTANT</h3><p>I believe that <strong>automation </strong>is the only way to survive in a jungle of domains willing to create Data Products. Automation with respect to:</p><ol><li>easy and fast creation of Data Products leveraging pre-built reusable templates that guarantee out-of-the-box compliance to the Data-as-a-Product principle;</li><li>implementation and execution of the Federated Governance policies (which therefore become “<em>computational</em>”);</li><li>enablement of Self-Service Infrastructure, as a fundamental pillar to provide Decentralized Domains the autonomy and ownership they need.</li></ol><p>The pillars together are all mandatory to reach the <strong>scale</strong>. They are complex to set up and spread, both as cultural and technical requirements, but they are foundational if we want to build a Data Mesh and harvest that so-widely promised value from your data.</p><blockquote>But <strong>how can we get to this level of automation?</strong></blockquote><h4>By leveraging a metadata model representing/associated with a Data Product.</h4><p>Such a model must be general, <strong>technology agnostic</strong>, and imagined as the key enabler for the automation of a Data Mesh platform (to be intended as an ecosystem of components and tools taking actions based on specific sections of this metadata specification, which must be standardized to allow interoperability across the platform’s components and services that need to interact with each other).</p><h3>State of the art</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKSJkiu3l3RLAoRnstQYUw.jpeg" /><figcaption>No clear sight due to an Atlantic perturbation — 2014, Madeira Island. Photo by Roberto Coluccio — all rights reserved.</figcaption></figure><p>In order to provide a broader view on the topic, I think it’s important to report some references to understand what brought us to do what we did (OK, now I’m creating hype on purpose 😁).</p><ul><li><strong>Feed the Control Ports: </strong>Arne Roßmann in <a href="https://medium.com/@arne.rossmann/shaking-the-kinder-egg-or-metadata-of-data-products-211c42925910">this article</a> describes how control ports of Data Products should serve foundational APIs by leveraging metadata collected at different levels.</li><li><strong>Active Metadata</strong>: Prukalpa Sankar in <a href="https://towardsdatascience.com/the-metadata-foundation-that-your-data-mesh-needs-9a1be28c5da6?source=email-f2cfad448222-1650759949552-digest.reader-7f60cf5620c9-9a1be28c5da6----2-96------------------fe6e14dd_0ca9_4d3f_8314_8dbd25e7fea6-1-">this article</a> mentions a commercial product tackling the active metadata space as a way to autonomously collect precious pieces of information about several things from SQL plans to understand data lineage, to monitor Data Products update rate, and other very cool stuff.</li><li><strong>Metadata types</strong>: Zeena in <a href="https://zeenea.com/the-3-types-of-metadata-to-master-to-be-a-data-centric-enterprise/">this post</a> introduces their Data Catalog solution after reporting a very interesting differentiation of metadata into DESCRIPTIVE, STRUCTURAL, ADMINISTRATIVE.</li><li><strong>Standards for Metadata in general</strong>: <a href="https://docs.open-metadata.org/overview/readme">OpenMetadata</a> is doing excellent work in creating standards and a framework for metadata management, including specific vocabulary and syntax (even if in some cases it becomes quite complex IMHO).</li><li><strong>General Data Product definition</strong>: what gets closer to our vision is <a href="http://opendataproducts.org/#open-data-product-specification">OpenDataProducts</a>, a good work to factor out some important information around the general Data Product concept (e.g. Billing and SLA). However, generalizing too much is risky and, in this case, I believe they got a little too far from the way Data Mesh requires a Data Product to be described and built. For example, there’s no mention of Domains, Schema, Output Ports, or Observability.</li></ul><p>According to these references, there’s still no clear metadata-based representation of a Data Product addressing — specifically — the Data Mesh and its automation requirements.</p><h3>HOW WE INTEND A DATA PRODUCT SPECIFICATION</h3><p>We believe in an <strong>Infrastructure-As-Code</strong> declarative, idempotent, and versioned approach. The goal is to have a standardized yet customizable specification addressing this holistic representation of the Data Mesh’s architectural quantum that is the Data Product.</p><p>Principles involved:</p><ul><li>Data Product <strong>components </strong>(input ports, output ports, workloads, staging areas)</li><li><strong>self-documenting</strong> Data Product (from schema to dependencies, to description and access patterns)</li><li>clear <strong>ownership </strong>(from Domain to Data Product team)</li><li><strong>semantic linking</strong> with other Data Products (like polysemes to help improve interoperability and references to build a semantic layer or Knowledge Graph on top of the Data Mesh)</li><li>data <strong>observability</strong></li><li><strong>SLA/SLO and Data Contracts</strong> (expectations, agreements, objectives)</li><li><strong>Access and usage control</strong></li></ul><p>The Data Product Specification aims to gather together crucial pieces of information related to all these aspects, under the strict ownership of the Data Product Owner.</p><h3>WHAT WE DID SO FAR</h3><p>In order to fill up this empty space, we tried to create a <strong>Data Product Specification</strong> by starting this open-source initiative:</p><p><a href="https://github.com/agile-lab-dev/Data-Product-Specification">GitHub - agile-lab-dev/Data-Product-Specification: An open specification for data products in Data Mesh</a></p><p>The repo contains a detailed documentation field-by-field; however, I’d like to point out here some features I believe to be important:</p><ul><li>One point of extension and — at the same time — standardization is the expectation that some of the fields (like tags) are expressed in the <a href="https://docs.open-metadata.org/openmetadata/schemas/entities/tagcategory"><strong>OpenMetadata</strong> </a>specification.</li><li>Depending on the context, several of those fields can be considered <strong>optional</strong>: their valorization is supposed to be checked by specific federated governance policies.</li><li>Although there’s an aim for standardization, the model is <strong>expandable </strong>for future add-ons: your Data Mesh (and platform) will probably keep evolving for a long time and the Data Product Specification is supposed to do the same, along the way. You can <strong>start simple</strong>, then add pieces by pieces (and related governance policies).</li><li>being in <strong>YAML </strong>format it’s easy to parse and validate, e.g. via a JSON parsing library like <a href="https://circe.github.io/circe/">circe</a>.</li></ul><p>The Data Product Specification itself covers the main components of a Data Product:</p><ul><li>general bounded context and domain metadata the Data Product belongs to (to support descriptive API on control ports)</li><li>clear ownership references</li><li>input ports, workloads, internal storage/staging areas, output ports, and logical dependencies between them</li><li>data observability</li><li>data documentation, semantic linking info, schema, usage guidelines (from serialization format and access patterns to billing models)</li></ul><p><strong>NOTE</strong>: what is here presented as a “conceptual” quantum can be (and it is, in some of our real-world implementations) split into its main componing parts that are then git-version controlled under their own repositories (belonging to a common group, which is the Data Product one).</p><h4>KEY BENEFITS</h4><p>The Data Product Specification is intended to be heavily exploited by a central Data Mesh-enabling platform (as a set of components and tools) for:</p><ul><li><strong>Data Products creation boost</strong>, since templates can be created out of this specification (for the single components and for the whole Data Products). They can then be evolved, also hiding some complexity from end-users.</li><li><strong>Federated Governance policies execution (</strong>e.g. a policy statement could be “the DP description must be at least 300 chars long, the owner must be a valid ActiveDirectory user/group, there must be at least 1 endpoint for Data Observability, the region for the S3 bucket must be US-East-1”, etc…), thus making “approved templates” to be compliant by-design with the Data Mesh principles and policies.</li><li><strong>Change Management improvement</strong>, since specifications are version controlled and well structured, thus allowing to detect breaking changes between different releases.</li><li><strong>Self-service Infrastructure enablement</strong>, since everything a DP needs to be provisioned and deployed for is included in a specific and well-known version of the DP specification, thus allowing a platform to break it down and trigger the automated provisioning and deployment of the necessary storage/compute/other resources.</li></ul><p>As I said, this specification is evolving and can be surely improved. Being open-source, every contribution is welcome.</p><h3>Wrapping up</h3><p>How can a Data Mesh journey reach the scale? With <strong>automation </strong>to guarantee out-of-the-box compliance of Data Products to the Data Mesh principles.</p><p>If you made it this far and you’re interested in other articles about Data Mesh, give a look at <a href="https://www.agilelab.it/ideas-knowledge-base/">our blog</a>. Also, get in touch if you’d like Agile Lab to help with your Data Mesh journey or if you’re curious about how we set up the aforementioned automation layer.</p><p><strong>Thank you for reading. </strong>Please, follow me if you think this article was interesting. It’ll also help you to be notified about new content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50610de8c152" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/data-product-specification-50610de8c152">Data Product Specification</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh adoption: 10 practical tips to remove roadblocks and improve domains’ engagement]]></title>
            <link>https://medium.com/agile-lab-engineering/10-practical-tips-to-reduce-data-meshs-adoption-roadblocks-and-improve-domains-engagement-d97459cddddb?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/d97459cddddb</guid>
            <category><![CDATA[data-product-creation]]></category>
            <category><![CDATA[ddd]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[data-strategy]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Tue, 22 Feb 2022 21:09:07 GMT</pubDate>
            <atom:updated>2022-04-06T14:53:45.663Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G330pPnx7w72WCudGzFXVQ.jpeg" /><figcaption>I thought this shot of mine could have been cool to use here because the anti-mosquito net with hexagonal holes just reminds me of a mesh full of Data Products. Another interpretation could be “Trap the value of your data with a Data Mesh” — All rights reserved, but I’d love to read the title you’d give to this photo (share it in the comments).</figcaption></figure><p>Hi! If you got here is probably because:</p><p><strong>A)</strong> you know what <strong>Data Mesh</strong> is;</p><p><strong>B)</strong> you are about to start (or already started) a journey towards Data Mesh;</p><p><strong>C)</strong> you and your managers are sure that Data Mesh is the way to sky-rocket the data-driven value for your business/company; but</p><p><strong>D)</strong> you are among the few who have bought the thing in and around you there are skepticism and worries.</p><p>The good news is: <strong>you’re DEFINITELY not alone!</strong></p><p>At <a href="https://www.agilelab.it/data-mesh-in-action/">Agile Lab</a> we’re driving Data Mesh journeys for several customers of ours and, every time a new project kicks off, this roadblock is always one of the first to tackle.</p><p>Hoping to help out anyone just approaching the problem and/or looking for some tips, here are 10 ways we found out being somehow successful in real Data Mesh projects. We’re gonna group them by “area of worry”.</p><h3>BUDGET 💸</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtCyizVfzu6xDEcsbk1Kzg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/it-it/@karolina-grabowska?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Karolina Grabowska</strong></a> from <a href="https://www.pexels.com/it-it/foto/seguire-gomma-ricco-pila-4386476/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>Well, you know, in the IT world every decision is usually based on the price VS benefit trade-off.</p><p>Domains are expected to start developing Data Products as soon as possible and we need them aboard, but usually “there’s no budget for new developments”, or “there’s no budget to enlarge the cluster size”. There you go:</p><h4>1️⃣ Compute resources to consume data designed as part of consumers’ architecture</h4><p>This is a general architectural decision rather than a tip: you should design your output port (and their consumption model) so to provide shared governed access to potential consumers: with simple ACLs over your resources, you can allow secure read-only access to your Data Products’ output ports so that no compute resources are necessary at the (source) domain/Data Product level in order to consume that data, all the power is required (AND BILLED) at the consumers’ sides. Obviously, this is only possible if you adopt an architecture with decoupled storage and compute. For example, this is not (hardly) possible if you use an RDBMS as output port.</p><p>Ok, so let’s say we figured out a way to avoid (reduce?) TCO for our Data Products. But what about the budget to develop them?</p><h4>2️⃣ Design a roadmap of data products creation that makes use of available operational systems maintenance budget streams</h4><p>In complex organizations there are usually several budget streams flowing for operational systems maintenance: leverage them to create Source Aligned Data Products (which are flawlessly part of the domain so nobody will argue)!</p><p>Very well, we’re probably now getting some more traction, but why should we maintain long-term ownership over this stuff?</p><h4>3️⃣ Pay-as-you-consume billing model</h4><p>I know, this is bold and, so far, still in the utopia sphere. But there’s a key point to catch: if you figure out a way to “give something back” to Data Products’ creators/owners/maintainers, being that extra budget for every 5 consumers of an output port, or participation by the consumers to the maintenance budget … anything can pull up the spirit and diffuse the “it’s not a give-only thing this one” mood. If you get it, you win.</p><p>But let’s say I’m the coordinator of the Data Mesh journey and I received a budget just for a PoC… well, then:</p><h4>4️⃣ Start small, with the most impactful yet less “operative business-critical” consumer-driven data opportunity</h4><p>No big bang, don’t make too much loud (yet), pick the right PoC use case and make it your success story to promote the adoption company-wise, without risking too much.</p><p>This will probably make the C-level managers happy. The same ones who are repeatedly asking for the budget to migrate (at least) the analytical workloads to the Cloud, without receiving it “because we’re still amortizing the expense for that on-prem system”. Well, here’s the catch: if they want you to build a Data Mesh, you’re gonna need an underline stack that allows you to develop the “Self-Serve Infrastructure-as-a-Platform”. That’s it:</p><h4>5️⃣ Leverage the Data Mesh journey as an opportunity to migrate to the Cloud</h4><p>Your domain teams will probably be happy to fast forward to the present days technology-wise and work on Cloud-based solutions.</p><p>This last tip brings to light the other “area of worry”:</p><h3>THE UNKNOWN ⚠</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2wOKl214yWXP1lLhNMTS7A.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/it-it/@jens-johnsson-14223?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Jens Johnsson</strong></a> from <a href="https://www.pexels.com/it-it/foto/freccia-in-legno-marrone-firmata-66100/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>Data Mesh is an <strong>organizational shift</strong> that involves processes and, most of all, <strong>people</strong>.</p><p>People are usually scared of what they don’t know, especially in the work environment where they might be “THE reference” to everybody’s eyes for system XY, while they’d feel losing power and control by “simply becoming part of a decentralized domain’s Data Product Team”.</p><p>First of all, you’re gonna need to:</p><h4>6️⃣ Train people</h4><p>Training and mentoring of key (influencer?) people across the domains is a successful-proven way of making people UNDERSTAND WHY the company decided to embrace this journey. Data Mesh Experts should mentor the focal point so that they can, in their turn, spread the positive word within their teams.</p><p>But teams are probably scared too of being forced to learn new stuff, so:</p><h4>7️⃣ Make people with “too legacy technical background” understand this is the next big wave in the data world</h4><p>Do they really wanna miss the train? Changes like this one might come once every 10 or 20 years in big organizations: don’t waste the opportunity and get out of that comfort zone!</p><p>Nice story, so are you telling me we have organizational inefficiencies and we don’t make smart use of our data to drive or automate business decisions? Well, my dear, it’s not true: MY DOMAIN publishes data that I’ve been told (by a business request) should be used somewhere.</p><p>I see you, but understand that your view might be limited. There are plenty of others complaining about change management issues, dependencies issues, slowness in pushing evolutions and innovations, lack of ownership and quality, etc.. etc.. etc …</p><h4>8️⃣ Organize collaborative workshops where business people meet technical people and share pain points from their perspectives</h4><p>You will be surprised by the amount of “hidden technical debt” that will come out, and people will start empathizing with each other: this will start transferring the “My data can bring REAL VALUE” (because real people are complaining in front of them) to the organization.</p><p>Does this end once the Data Products have been released? Absolutely not! We talked about «long-term ownership» so we need to provide «long-term motivation»:</p><h4>9️⃣ Collect clear insights about the value produced with data and make it accessible to everybody within the organization (through the platform)</h4><p>This can start as simple as «Number of consumers of a Data Product» (what if it’s zero?) and evolve in more complex metrics like the time-to-market (it’s expected to be reduced, it’s the time to bring into production a new Data Product), the network effect (the more the interconnections across Domains, the more value is probably added afterward), and other metrics more specific and valuable to your corporation.</p><p>You convinced me. I will develop Data Products. Where do I start?</p><h4>🔟 Organize collaborative workshops to identify data opportunities</h4><p>Following the DDD (Domain Driven Design) practice the first pillar of Data Mesh is based on, Domain-oriented Decentralized Ownership, and adding it up to the Data-as-a-Product one, we get a “business decisions”-driven design model to identify data opportunities (i.e. Data Products to be created in order to support/automate data-driven business decisions).</p><p>On this topic, you might be interested in the “<strong><em>Data Product Flow</em></strong>” we designed at Agile Lab, check it out here: <a href="https://medium.com/agile-lab-engineering/how-to-identify-data-products-welcome-data-product-flow-76d7d85d23af">https://medium.com/agile-lab-engineering/how-to-identify-data-products-welcome-data-product-flow-76d7d85d23af</a></p><h3>Wrapping up</h3><p>We saw the major roadblocks impacting people&#39;s engagement and their agility/motivation to embrace the Data Mesh paradigm. Most of them are due to budget issues or fear of the unknown, but there are several actions we can take to overcome these concerns and build a solid and engaged group of Data Mesh enthusiasts.</p><p>If you made it this far and you’re interested in other articles about Data Mesh, follow me and give a look at <a href="https://www.agilelab.it/ideas-knowledge-base/">our blog</a>. Also, get in touch if you’d like Agile Lab to help with your Data Mesh journey.</p><p>Please, follow me if you think this article was interesting. It’ll also help you to be notified about new content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d97459cddddb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/10-practical-tips-to-reduce-data-meshs-adoption-roadblocks-and-improve-domains-engagement-d97459cddddb">Data Mesh adoption: 10 practical tips to remove roadblocks and improve domains’ engagement</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer 360 and Data Mesh: friends or enemies?]]></title>
            <link>https://medium.com/agile-lab-engineering/customer-360-and-data-mesh-friends-or-enemies-d7c5e583d9b1?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7c5e583d9b1</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[ddd]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[mdm]]></category>
            <category><![CDATA[customer-360]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Mon, 07 Feb 2022 17:04:57 GMT</pubDate>
            <atom:updated>2022-04-06T14:53:25.843Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFk8IIDbLMw52O0ssw-weA.png" /></figure><p>Raise a hand who saw, was asked to design, tried to implement, struggled with the “Customer 360” view/concept in the last 5+ years…</p><p>Come on, don’t be shy …</p><p>🙋‍♀️🙋‍♂️🙋‍♀️🙋‍♂️🙋‍♀️🙋‍♂️🙋‍♀️🙋‍♂️🙋‍♀️🙋‍♂️🙋‍♀️🙋‍♂️</p><p>How many <strong>Customer 360</strong> stories did you see succeeding? Me? Just … a few, let’s say. This made me ask: why? Why put so much effort into creating a monolithic 360 view thus creating a new maintenance-evolution-ownership nightmare silo? Some answers might fit here, in the context of centralized architectures, but recently a new antagonist to fight against came in town: <strong>Data Mesh</strong>.</p><blockquote>“Oh no, another article about the Data Mesh pillars!”</blockquote><p>No, I’m not gonna spam the web with yet-another-article about the Data Mesh principles and pillars, there’s plenty out there, like <a href="https://martinfowler.com/articles/data-mesh-principles.html#CorePrinciplesAndLogicalArchitectureOfDataMesh">this</a>, <a href="https://www.agilelab.it/data-mesh-explanation/">this</a>, or <a href="https://www.google.com/search?q=data+mesh">those</a>, and if you got here is because you might already know what we’re talking about.</p><p>The question that arises is:</p><blockquote><strong>How does the (so struggling or never really achieved) Customer 360 view fit into the Data Mesh paradigm?</strong></blockquote><p>Well, in this article I’ll try to come up with some options, coming from some real Agile Lab’s customers approaching the “Data Mesh journey”.</p><p>The Data Mesh pillar I want to focus on is <strong>Domain-oriented Distributed Data decomposition and ownership</strong>.</p><p>This fascinating principle, inherited (in terms of effectiveness) from the microservices world, brings to light the necessity to keep together tech and business knowledge, within specific bounded contexts, so as to improve autonomy and velocity of data products lifecycle along with a smoother change management. Quoting <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html#DataAndDistributedDomainDrivenArchitectureConvergence">Zhamak Dehghani</a>:</p><blockquote>Eric Evans’s book <a href="https://domainlanguage.com/ddd/">Domain-Driven Design</a> has deeply influenced modern architectural thinking, and consequently the organizational modeling. It has influenced the microservices architecture by decomposing the systems into distributed services built around business domain capabilities. It has fundamentally changed how the teams form, so that <strong>a team can independently and autonomously own a domain capability</strong>.</blockquote><p>Examples of well-known domains are Marketing, Sales, Accounting, Product X, or Sub-company Y (with subdomains, probably). All of these domains have probably something in common, right? And here we connect with the prologue: <strong>the customer</strong>. We all agree on the importance of this <em>view</em>, but eventually do we really need this to be materialized as a monolithic entity of some sort, on a centralized system?</p><p>I won’t answer this question, but I’ll make another one:</p><blockquote>Which domain would a customer 360 view be part of?</blockquote><p>Remember: decentralization and domain-driven design imply having clear <strong>ownership </strong>which, in the Data Mesh context, means:</p><ul><li>decoupling from the change management process of other domains</li><li>owning the persistence (hence: storage) of the data</li><li>provide valuable-on-their-own views at the Data Products’ output ports (<strong>is a holistic materialized 360 customer view really valuable-on-its-own?</strong>) — this is part of the “Data as a Product” pillar, to be precise</li><li>guarantee no breaking changes to domains that depend on the data we own (i.e. own the full data lifecycle)</li><li>(last but not least) having solid knowledge (and ownership) of the business logic orbiting around the owned data. Examples are: knowing the mapping logic between primary and foreign keys (in relational terms), knowing the meaning of every bit of information part of that data, knowing what are the processes that might influence such data lifecycle</li></ul><p>If you can’t now answer the above question DON’T WORRY: you’re not alone! It just means you’re starting as well to feel the friction between the decentralized ownership model and the <em>centralized</em> customer 360 view.</p><p><strong>IMHO, they are irreconcilable</strong>. Here’s why, with respect to the previous points:</p><ul><li>centralized ownership over all the possible customer-related data would <strong>couple the change management processes of all the domains</strong>: a 360 view would be the only point-of-access for customer data, thus requiring strong alignment across the data sources feeding the customer 360 (which usually would end up slooooowing down every innovation or evolution initiative). Legit question: <em>“So, this means we should never join together data coming from different domains?”</em> — Tough but realistic answer: <em>“You should do that only if it makes sense, if it produces clear value, if it doesn&#39;t require further interpretation or effort on the consumers’ side to grasp such value”.</em></li><li>If you bind together data coming from different domains, it must be to create a valuable-on-its-own data set. Just the fact that you read, store and expose data would make you the owner of it, that’s why a customer 360 persisted view would make sense only actually owning the whole dataset, not just a piece of it. Furthermore, Data Products require immutable and bi-temporal data at their output ports, can you imagine the effort of delivering such features over such a huge data asset?</li><li>valuable-on-its-own Data Products, well, I hope the previous 2 points already clarified this point 😊</li><li>If we sort out all the previous issues and actually become the owner of a customer 360 persisted materialized view, while still being compliant to the Data Mesh core concepts, <strong>we should never push breaking changes towards our data consumers</strong>. If we need to do make a breaking change, we must guarantee sustainable migration plans. Data versioning (like <a href="https://lakefs.io/">LakeFS</a> or <a href="https://projectnessie.org/">Project Nessie</a>) could come in handy in this case, but there might be several organizational and/or technical ways to do so. Having many dependencies (like we’d have owning a customer 360 view) would make it VERY hard to always have a smooth data lifecycle and avoid breaking changes since we could find ourselves just overwhelmed by breaking changes made at the source operational systems side.</li><li>The whole Data Mesh idea emerged as a decentralization need, after seeing so many “central data engineering / IT teams” fall under the pressure of all the organization which was relying on them to have quality data. It was (is) a bottleneck in many cases worsened by the fact that these centralized tech teams couldn’t just have sufficiently <strong>strong business knowledge of every possible domain</strong> (while unfortunately they were developing and managing all the ETLs/ELTs of the central data platform). For this very same reason, the Customer 360 team would be required to master the whole business logic around such data, i.e. being THE acknowledged experts of basically every domain (having customer-related data) — just impossible.</li></ul><figure><img alt="a customer360 logical view, with domains owning only a slice of it" src="https://cdn-images-1.medium.com/max/1024/1*m1eKIOrmJ_C5o71r0C9ZDg.png" /><figcaption>a customer360 logical view, with domains owning only a slice of it</figcaption></figure><p>OK, I’m done with the bad news 😇 Let’s start with the good ones!</p><h3>Customer 360 view and Data Mesh can be friends</h3><p><strong>As long as it remains a holistic logical/business concept</strong>. Decentralized domains should own the data related to their bounded contexts, even if referring to the customer. Domains should own a slice of the 360 view and slices should be correlated together to create valuable-on-their-own Data Products (e.g. just a join between sales and clickstreams doesn’t provide any added value, but a behavioral pattern on the website with the number of purchases per customer with specific browsing-behaviors do).</p><p>In order to achieve that, domains and - more in general - data consumers must be capable of <strong>correlating customer-related data across different domains with ease</strong>. They must be facilitated on the joining logic level, which means they should NOT be required to know the mapping between domain-specific keys, surrogate keys, and customer-related keys. I’ll try to expand this last element.</p><h4>Globally Unique Identifiable Customer Key</h4><p>According to the Data Mesh literature, data consumers shouldn’t always copy (ingest in the first place) data in order to do something with it, since storing data means having ownership over it. As a technical step justified by volumes or other constraints for a single internal ETL step, that’s ok, but <em>Consumer Aligned Data Products</em> shouldn’t just pull data from another Data Product’s output port, perform a join or append another column to the original dataset, and publish at their output port an enhanced projection of some other domain’s data, for many reasons, but I won’t digress on this (again, it’s part of the Data as a Product pillar). <strong>What data consumers need is a well-documented, globally identifiable, and unique key/reference to the customer, in order to perform all the possible correlations across different domains’ data</strong>. The literature would also call them <em>polysemes</em>.</p><p><strong>Note</strong>: <em>well-documented</em> should become a computational policy (do you remember the <em>Federated Governance</em> Data Mesh’s pillar?) requiring for example that, in the Data Product’s descriptor (want to contribute to <a href="https://github.com/agile-lab-dev/Data-Product-Specification">our standardization proposal</a>?), a specific set of metadata must describe where a certain field containing such key comes from, which domain generated it, what business concepts it points to. Maybe that could be done also leveraging a syntax or language that will facilitate the automated creation (thank you <em>Self-Serve Infrastructure-as-a-Platform</em>) of a Knowledge Graph afterward.</p><p>OK, the concepts of polysemes and the globally unique identifiable customer-related key are not new to you, and they shouldn’t, but what I want to put the lights on is how to conjugate it with the Data Mesh paradigm, especially because several architectural patterns are technically possible but just a few (one?) of them should guarantee long term scalability and compliance with the Data Mesh pillars.</p><h4>An MDM is still a good alley</h4><p>The question you might have asked yourself at this point is:</p><blockquote>Who has the ownership of issuing such a key?</blockquote><p>The answer is probably in the good-old-friend the <strong>Master Data Management</strong> (MDM) system, where the <strong>customer’s <em>golden record</em> data</strong> can be generated. There’s a lot of literature (<a href="https://blog.metamirror.io/abandon-the-sirens-call-of-the-golden-record-1c49e1f3cd47">example</a>) on that concept, also because it’s definitely not new in the industry (and that’s why a lot of companies approaching Data Mesh are struggling to understand how to make it fit in the picture since it can just be thrown away after all the effort spent on building it up).</p><p><strong>Note</strong>: eventually, such a transactional/operational system might also lead to developing a related <em>Source Aligned Data Product</em>, but it should just be considered as a source of the customers’ registry information.</p><blockquote>OK, but who should be then responsible of performing the reverse lookup, to map a domain-specific record (key) to the related customer (golden record’s) key?</blockquote><p>We narrow down the spectrum into 3 possible approaches. To facilitate the reading of what follows, I’d like to point out a few things first:</p><ul><li>SADP = Source Aligned Data Product</li><li>CADP = Consumer Aligned Data Product</li><li>CID = Customer MDM unique globally identifiable key</li><li>K = example of a domain-specific primary key</li></ul><h4>Approach 1: The Ugly</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YVEmsV3teL06Sx6ZGoGwg.png" /></figure><h4>Approach 2: The Bad</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1q2wsiANsyVT9JFpDiv9g.png" /></figure><h4>Approach 3: The Good</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDrO2JlnBtTJ-Hc6Qcppbg.png" /></figure><p>The latter approach is what could make a customer 360 view shine again! This way, domains — and in particular the operational systems’ teams- preserve the ownership of the mapping logic between the operational system’s data key and the CID, while in the analytical plane no other interactions with the operational one are required to create <em>Source Aligned Data Products</em> other than pulling data from the source systems, and at the <em>Consumer Aligned Data Products</em> sides no particular effort is spent to correlate customer-related data coming from different domains.</p><p><strong>Disclaimer 1</strong>: approach 3 requires strong Near-Real-Time alignment between the Customer MDM system and the other operational systems since a misalignment could imply publishing data from the various domains not attached to a brand new available CID (many MDM systems operate their match-and-merge logic in batch, or many operational systems might think to update-set the CID field with batch reverse lookups quite after domain data is generated).</p><p><strong>Disclaimer 2</strong>: a fourth approach could be possible, i.e. having microservices issuing in NRT global IDs which become master key in all the domains having customer-related data. This is usually achievable in custom implementations only.</p><h3>Wrapping up</h3><p>In the Data Mesh world, domain-driven decentralized ownership over data is a must. The centralized Customer 360 monolithic approach doesn’t fit the picture, so a shift is required in order to maintain what actually matters the most: the business principle of customer-centric insights, derived from the correlation of data taken from different domains (owners) via a well documented and defined globally unique identifier, probably generated at Customer MDM level and integrated “as left as possible” into the operational systems, so to reduce the burden at the Data Mesh consumers’ side of knowing and applying the mapping logic between domain-specific key and Customer MDM key.</p><p>What has been presented has been discussed with several customers and it seemed to be the best option so far. We hope the hype around Data Mesh will bring some more options in the very next years and will keep our eyes open but, if you already put in place a different approach, I’ll be glad to know more.</p><p>If you made it this far and you’re interested in other articles on the Data Mesh topics, you can find them on <a href="https://www.agilelab.it/ideas-knowledge-base/">our blog</a>. Also, get in touch if you’d like Agile Lab to help with your Data Mesh journey.</p><p>Please, follow me if you think this article was interesting. It’ll also help you to be notified about new content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7c5e583d9b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/customer-360-and-data-mesh-friends-or-enemies-d7c5e583d9b1">Customer 360 and Data Mesh: friends or enemies?</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secondary Indexing on HBase]]></title>
            <link>https://medium.com/agile-lab-engineering/secondary-indexing-on-hbase-8a3acea394f0?source=rss-f2cfad448222------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a3acea394f0</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[hbase]]></category>
            <dc:creator><![CDATA[Roberto Coluccio]]></dc:creator>
            <pubDate>Fri, 12 Jul 2019 13:29:56 GMT</pubDate>
            <atom:updated>2022-04-06T14:54:08.087Z</atom:updated>
            <content:encoded><![CDATA[<p>So, they told you that <strong>NoSQL </strong>is the place where to store your de-normalized ex-relational (big) data, right? And you believed them, didn’t you? Well, that could not have been your worst decision, as long as you have thought about a plan B to apply when the first users — coming right after your data migration has been completed — will ask you:</p><blockquote>“I don’t have the key to access the key-value store. Is that OK if I perform a (full?) SCAN to search for the data I need?”</blockquote><h4>No, it’s NOT OK.</h4><figure><img alt="Searching through secondary indexing on HBase is not like looking up for the take away pizza number on the agenda" src="https://cdn-images-1.medium.com/max/960/0*tjQ2H8yIR-xKufnx.jpg" /></figure><p>If your data system must provide massive alongside with punctual access to small piece of data in a lake of potentially (b)millions of records, sequential flows like file systems (even if distributed) are not the right choice: that’s why key-value stores, backed by a distributed and scalable architecture, gained so much success in the last few years, especially when companies facing re-platforms of their relational data systems approached the Big Data world.</p><p>Unfortunately, the lack of <em>efficient</em> access patterns (a metric which can obviously be different use case by use case) different from the key-value one usually brought to light a sad piece of news: there’re <strong>no secondary indexes</strong>! WHAT?! Well, I’m afraid so. At least, not usually built-in in most of the broad-adopted technologies like <strong>Apache HBase</strong>.</p><h4>So here’s your plan B.</h4><p>In most of the projects we worked on at <strong>AgileLab</strong>, we leveraged Apache HBase to tackle the problem of having a consistent, distributed, fault tolerant and PetaByte-scalable data base where to store unstructured or relational-denormalized data in key-value fashion. In fact, the NoSQL datastore HBase provides CP [1] capabilities on top of an Hadoop cluster, leveraging Zookeeper for state and configuration management and synchronization, and the <strong>Hadoop Distributed File System</strong> (HDFS) as scalable, replicated, fault tolerant persistence layer.</p><p>Although HBase provides a “table” abstraction, it doesn’t allow users (as of HBase 1.x) to declare any “relational” schema-based structured, besides a “column family” based physical and row-oriented logical data model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/0*EvisjcJKZ4XZcSoU.png" /><figcaption><em>HBase Table structure (source: </em><a href="https://www.oreilly.com/library/view/hbase-the-definitive/9781449314682/httpatomoreillycomsourceoreillyimages889234.png)"><em>https://www.oreilly.com/library/view/hbase-the-definitive/9781449314682/httpatomoreillycomsourceoreillyimages889234.png)</em></a></figcaption></figure><p>Data on HBase is organized in rows (each identified by a unique “row key”), which are physically persisted in different files on HDFS (“HFile”) according to the column family a given piece of data belongs (HFiles can be multiple, depending on the regions splits, but this is out of the scope of this post). Within column families, data is organized in lexicographically-ordered “column qualifiers”, which uniquely identify a “cell” that will contain nothing but a bare array of bytes. We can than state that, given a unique key made of &lt;row key + column family + column qualifier&gt; (a.k.a. “cell qualifier”), data access is offered from HBase in a key -&gt; value pattern being the key the “cell qualifier” modeled above, and the value the array of bytes contained in the cell.</p><h4>Don’t panic, let’s make an example</h4><p>Let’s imagine a possible use case. MyOnlineShopCompany has millions of customers, each identified by a unique id (customer_id). Every customer places orders and buys stuff from the online shop. The company’s IT decided to store such orders on HBase, because they can be “a lot” and they can be accessed from a company’s “My Orders” web page one by one and customer by customer. Each customer, during the “active account” period, can place up to x000 (thousands) orders, so the order of magnitude of just an hypothetical “orders” table would be in the billions of records. The company must provide online access to such orders for at least 5 years, but it could need to make historical older data available to its data science team which will perform prediction, analytics, and other cool stuff. Orders business objects are modeled with dozens of fields.</p><p>OK, accepting the fact that we are gonna leverage HBase to store these orders (we could discuss about business/costs/tradeoffs-wise arguments all night long), how could we design our table?</p><p>A possible physical <strong>data model</strong> could be:</p><p>— <strong>ROW KEY</strong>: the customer_id &gt; this allows us to reference a single row to access all the orders of a customers with a GET operation; also it implies our table doesn’t grow “vertically” thus allowing an upfront-optimized HBase’s regions keys splitting management.</p><p>— <strong>COLUMN FAMILY</strong>: the year the order was placed &gt; this allows us to precisely access a single family (read: avoid opening files that don’t contain orders of a specific requested year); also it allows hussle-free data historicization (just grab the 6th year old CF and do your historical data management read/write/delete operations without affecting other data).</p><p>— <strong>COLUMN QUALIFIER</strong>: the day and minutes the order was placed, represented as the number of “days left in the year” (1 to 366) concatenated to the “minutes left in the day” &gt; this allows us to have the “first” order in the array of column qualifiers for a single row being the most recent (of a certain year), which could speed up reads by a “most-recent” access pattern (with bissextile and timezones shifts management). Again, there are several optimizations (what if a customer places more than one order within a minute? I know, I know) that we could imagine on this model: let’s move on and accept this for now, but we can discuss it further in another place.</p><p>And the content? Avro, of course. The application business object serialized with Avro <strong>Single Object Encoding</strong>, to be precise (just a byte array from an HBase point of view).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCd4O5dyryPKaDjXdPCAOw.png" /><figcaption>Physical data model of the “orders” HBase table</figcaption></figure><p>A very common use case, that we at AgileLab leveraged in several projects, is to put complex business objects inside those cells, serializing them with <strong>Apache Avro</strong> (it requires a schemas management framework and we developed, leveraged and open sourced <strong>Darwin</strong> [2] to address that). Nonetheless this mechanism provides users the powerful possibility to have multiple schema-less data models within a single table, it forces them to know upfront cell qualifiers to access the single record in “GET” access pattern (which can be pushed down up to milliseconds latency!). HBase also provides “SCAN” access patterns, but these are not as much efficient, nor are the use case this technology has been designed for.</p><h4>So how do we access our huge table by a pattern different from the key -&gt; value one?</h4><p>Let’s say our orders have, besides the other fields, the “category”, “price”, and “description” one. How would you fetch the orders, for a specific customer or maybe (worse) across all the customers, having “XY” <em>category</em>, <em>price </em>range between 10 and 100, and <em>description </em>containing the word “USB” (case insensitive)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/0*LeYAOV7ognzRqD-R.jpg" /><figcaption>Would you really perform (FULL) SCANs? Please, don’t! (Image source: <a href="https://erickbio.files.wordpress.com/2012/05/tebel.jpg">https://erickbio.files.wordpress.com/2012/05/tebel.jpg</a>)</figcaption></figure><p>OK, so maybe now I got your full attention.</p><p>Well, as of HBase documentation [3] a reference to “<strong>Secondary Indexing</strong>” suggests users to develop a custom “<em>Coprocessor</em>” that deserializes our “PUT” mutations, extracts secondary index keys and writes them to another “shadow” HBase index table, which … has the same limitation of our source table. Data is opaque, there’s no high-level schema or column value type to help in accessing such data, besides the problem of keeping the two tables consistent among each other.</p><p>Also, have you ever developed (Java)/debugged (on clusters)/deployed (as jar)/maintained a custom Coprocessor? We did, and it wasn’t a pleasant journey. Coprocessors can be seen as an intermediate server-side entry-point API which is triggered before a <em>Mutation </em>(a PUT/DELETE request from a client) is handled by a Region Server (the slave servers of the master-slave HBase architecture).</p><p>We found out this approach is worthwhile of the effort only when our workload is beyond TeraByte-scale. When not in such a scenario, another architecture provides extremely easier and user friendlier approach.</p><p>While working on Cloudera clusters, as in most of our customers scenarios, we leveraged the <strong>Lily Indexer + SolR (Cloud) </strong>stack to tackle the secondary indexing problem [4].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QgPZ6jxhfq86LTnI.png" /><figcaption><em>Cloudera Search architecture (source: </em><a href="http://techkites.blogspot.com/2015/02/near-line-search-indexing-using-hbase.html)"><em>http://techkites.blogspot.com/2015/02/near-line-search-indexing-using-hbase.html)</em></a></figcaption></figure><p><strong>Lily HBase Indexer</strong> service embeds the <strong>NG-Data</strong> <strong>Indexer </strong>[5] to provide a <em>Near-Real-Time</em> (NRT) resilient automated configuration-driven mechanism to trigger <strong>Morphline</strong> (<strong>Kite SDK</strong>) [6] parsers over an HBase mutation so to extract from a deserialized cell value our significant secondary index fields and publish them as documents on a pre-determined SolR collection (a.k.a. <strong>Solr Search</strong> in Cloudera environment, given that index data can be easily accessed via Hue Web UI for browsing and dashboarding). PUT and DELETE mutations are automatically handled updating the SolR collections.</p><p>From an application development point of view, indexes on secondary fields provide a millisecond-grade multi-modal access pattern on SolR (either accessed from batch jobs or web services), which would allow us to solve our initial data problem and retrieve HBase’s cell qualifiers in order to access our original full records.</p><p>In our example (UPDATE: based on technologies available as of CDH 5.x), after having configured our SolR collection with a schema like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e44d59ce3cb13df699b395eff2ef3788/href">https://medium.com/media/e44d59ce3cb13df699b395eff2ef3788/href</a></iframe><p>We would issue client-side the SolR query:</p><blockquote>category:”XY” AND price:[10 TO 100] AND description:”USB”</blockquote><p>This would return the documents containing the “row_key”, “column_family”, “qualifier” stored fields that, on the application side, we could leverage to issue the related GET operations on HBase and efficiently retrieve our records within, probably, milliseconds.</p><h4>I like it! What do I need to set it up?</h4><p>On a Cloudera cluster with Lily HBase indexer service properly installed and configured, the applicative setup is pretty easy, we just need to:</p><p>1. enable replication on the target HBase table (if you don’t have cluster-wide replication already enabled);</p><p>2. register an indexer for each table we are interested in indexing through a shell command (and a Morphline configuration file that contains the parsing logic);</p><p>3. properly create/setup the related SolR collections;</p><p>4. enjoy accessing the indexes, maybe through <strong>Cloudera Hue</strong> thanks to which we could also create dynamic interactive dashboards (what about NRT KPI?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AErk65plSLmUIiXb.png" /><figcaption><em>Interactive Dashboarding with Hue (source:</em> <a href="https://blog.cloudera.com/blog/2014/07/new-in-cdh-5-1-hues-improved-search-app/"><em>https://blog.cloudera.com/blog/2014/07/new-in-cdh-5-1-hues-improved-search-app/</em></a><em> )</em></figcaption></figure><p>The resiliency and at-least-once data handling semantic of the indexers is given by the fact that those register themselves as HBase replication peers, thus exploiting HBase’s built-in replication mechanism. Also, the Lily HBase indexer supports <strong>Kerberos </strong>for authentication, and <strong>Apache Sentry</strong> for authorization. Scalability of SolR Cloud is provided by its sharding capabilities.</p><p>Last but not least, what about a batch distributed <strong>MapReduce </strong>job to re-build the indexes in case of data-migration or data corruption, schema change or new environment setup? Well, it’s already available in Cloudera environment, and you just need to launch it by passing it the very same <em>morphline.conf</em> configuration file passed to the Lily HBase Indexer creation shell command.</p><h4>OK the <strong>pros</strong>, but what about the <strong>cons</strong>?</h4><p>Well, there are a few. You tell us which one, in your experience, could be the worst to face among:</p><p>a) DDLs are not propagated (e.g. ‘<em>truncate</em>’ commands) to SolR (but none of you guys trigger truncates in production environment, right?).</p><p>b) SolR collections are indexed with the <strong>Lucene Inverted Indexes</strong> [7], so pay attention to the range of values that your secondary index keys can have: in fact, inverted indexes are known not to outperform when keys are mostly condensed around a very little subset of values.</p><p>c) As of the latest CDH 5.x release, Lily Indexers registered as HBase replication peers will always receive all the mutations of all the tables configured with a “<em>REPLICATION_SCOPE</em>” greater than 0 (they are just table-agnostic replication end-points to send mutations to from an HBase perspective). Indexers discard all the mutations not related to their pre-configured target table.</p><h4>Conclusions</h4><p>At <strong>AgileLab </strong>we leveraged this architecture to provide a “<strong>light-OLTP</strong>” access pattern to our customer’s HBase-based Big Data use cases, handling thousands of updates per minute across hundreds of tables and related indexes (per single use-case).</p><p>Get in touch with us to know more about this architecture. We also developed an <strong>OLAP </strong>access layer (another “anti-pattern” to access <em>key -&gt; value</em> data stores) on top of HBase that maybe will be a matter of another blog post in the future.</p><p>Please, follow me if you think this article was interesting. It’ll also help you to be notified about new content.</p><p><strong>References</strong>:</p><p><em>[1] CAP theorem: </em><a href="https://en.wikipedia.org/wiki/CAP_theorem"><em>https://en.wikipedia.org/wiki/CAP_theorem</em></a></p><p><em>[2] AgileLab Darwin Avro Schema Evolution manager: </em><a href="https://github.com/agile-lab-dev/darwin"><em>https://github.com/agile-lab-dev/darwin</em></a></p><p><em>[3] HBase documentation, secondary indexes: </em><a href="https://hbase.apache.org/book.html#secondary.indexes"><em>https://hbase.apache.org/book.html#secondary.indexes</em></a></p><p><em>[4] Cloudera Lily Indexer + SolR: </em><a href="https://www.cloudera.com/documentation/enterprise/5-14-x/topics/search_config_hbase_indexer_for_search.html"><em>https://www.cloudera.com/documentation/enterprise/5-14-x/topics/search_config_hbase_indexer_for_search.html</em></a></p><p><em>[5] NG-Data indexer: </em><a href="https://ngdata.github.io/hbase-indexer/"><em>https://ngdata.github.io/hbase-indexer/</em></a></p><p><em>[6] Morphline parser:</em> <a href="http://kitesdk.org/docs/1.1.0/morphlines/"><em>http://kitesdk.org/docs/1.1.0/morphlines/</em></a></p><p><em>[7] Lucene Inverted Index: </em><a href="https://lucene.apache.org/core/"><em>https://lucene.apache.org/core/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a3acea394f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/secondary-indexing-on-hbase-8a3acea394f0">Secondary Indexing on HBase</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>