<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carlos Costa on Medium]]></title>
        <description><![CDATA[Stories by Carlos Costa on Medium]]></description>
        <link>https://medium.com/@epilif1017a?source=rss-156ac7d3a898------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*srImSEkBs0meTGde3zusBQ.jpeg</url>
            <title>Stories by Carlos Costa on Medium</title>
            <link>https://medium.com/@epilif1017a?source=rss-156ac7d3a898------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:43:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@epilif1017a/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Re-Explaining Your Lakehouse to Every AI Assistant]]></title>
            <link>https://medium.com/@epilif1017a/stop-re-explaining-your-lakehouse-to-every-ai-assistant-5b63475cfdf1?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b63475cfdf1</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 15:32:05 GMT</pubDate>
            <atom:updated>2026-07-14T15:32:05.753Z</atom:updated>
            <content:encoded><![CDATA[<p>Meet themis-lake — one context manager that teaches every AI agent how <em>your</em> team builds data products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qC5JgutSx0CmXn6yz9iMqw.png" /><figcaption>themis-lake representative image. Generated by AI.</figcaption></figure><p>If you build on a Lakehouse, you already know the tax.</p><p>You open Claude Code, Copilot, Cursor, opencode, or a Databricks Genie session, ask it to “add a bronze ingestion for orders.csv,” and out comes confident, plausible, <em>wrong</em> code. Wrong framework. Wrong layering. Naming that violates the convention three teams agreed on last quarter. Data-quality checks bolted on the way the model learned from some blog post — not the way your platform actually does them.</p><p>So you correct it. Again. Tomorrow you correct the next engineer’s assistant the same way. And the new hire’s. The knowledge that should make you fast — your naming rules, your bronze/silver/gold contracts, your golden-path templates, your “we use <em>this</em> engine, not that one” decisions — lives in a wiki nobody reads, a Confluence page nobody found, and the heads of two senior engineers who are always in meetings.</p><p>→ <a href="https://github.com/epilif1017a/themis-lake"><strong>themis-lake</strong></a><strong> exists to end that loop.</strong></p><h3>The idea in one sentence</h3><blockquote><em>Package your organization’s lakehouse “way of doing things” once — and let every AI assistant plug into the same source of truth.</em></blockquote><p>Named after the Greek titaness of law and right conduct, themis-lake is an AI-assisted SDLC <em>governor</em> for Data &amp; AI. It doesn’t enforce rules blindly. It’s the companion that helps engineers <strong>rightfully</strong> create data products on the Lakehouse — with your standards riding shotgun in every session, no matter which AI tool they happen to prefer.</p><h3>Why this shape works</h3><p>Under the hood, themis-lake leans on the <a href="https://modelcontextprotocol.io/">Model Context Protocol (MCP)</a> as the integration layer, plus a small set of file-based agent assets. That combination gives you four things that copy-pasted system prompts never will:</p><ul><li><strong>One source of truth.</strong> Update a standard, a skill, or a rule in one repo. Every assistant picks it up on the next session — no prompt fragments drifting across teams and getting stale in six different places.</li><li><strong>Consistent guardrails.</strong> A data engineer in Claude Code, an analytics engineer in Copilot, and a platform engineer in opencode all hit the <em>same</em> conventions and scaffolds. Governance stops depending on who’s driving.</li><li><strong>Composable.</strong> themis-lake is just one MCP server among many. Run it next to a Git server, a Jira server, a catalog server — the agent picks the right tool for each step.</li><li><strong>Self-contained delivery.</strong> The server ships as a Docker image and fetches your wiki from GitHub at startup. Nothing is baked in, so updating a standard is <em>just a git commit</em> — no image rebuild, no redeploy.</li></ul><h3>What actually ships</h3><p>themis-lake draws a clean line between <strong>knowledge</strong> and <strong>runtime</strong>, and that distinction is the whole trick.</p><p><strong>The knowledge (installed into your AI clients):</strong></p><ul><li><strong>Skills</strong> — think of these as <em>procedures the assistant can look up mid-task</em>. A skill is a Markdown file describing, step by step, how your team performs one recurring job — create-pipeline, create-metric, create-table, review-ml-model, plus a vendored databricks-* family. When an engineer says &quot;add a bronze ingestion,&quot; the assistant matches the request to the relevant skill and follows <em>your</em> golden path instead of improvising one. Skills load on demand, so they cost nothing until the task actually calls for them.</li><li><strong>Agents</strong> — <em>personas that keep the assistant in its lane</em>. An agent definition pins the assistant to a role (data engineer, analytics engineer, platform engineer), tells it which tools it may reach for, sets its guardrails, and points it at the specific wiki sections that role should defer to. Instead of one generalist that knows a little about everything, you get a focused collaborator that behaves consistently for the job at hand.</li><li><strong>Instructions</strong> — <em>the always-on rules that ride in every single session</em>. These are short, high-priority directives appended to the host’s config file (CLAUDE.md / AGENTS.md) at install time. They encode precedence (&quot;the wiki wins over generic patterns&quot;), safety (&quot;never drop or overwrite without confirmation&quot;), and the non-negotiables the assistant must honor before anything else. Where skills and agents are pulled in situationally, instructions are always present.</li><li><strong>Wiki</strong> — <em>the single source of truth everything else points back to</em>. Your lakehouse standards written as plain Markdown, versioned in git: naming rules, medallion layering (bronze/silver/gold), data contracts, quality expectations, governance. Humans read it directly; the assistant reads it through MCP. Change a standard here and it propagates everywhere on the next session — no prompt-copying, no drift.</li></ul><p><strong>The runtime (wired in as MCP servers):</strong></p><p>MCP — the <a href="https://modelcontextprotocol.io/">Model Context Protocol</a> — is the open standard that lets an AI tool call out to external systems in a uniform way. An MCP <em>server</em> exposes capabilities (data to read, actions to take); the AI <em>client</em> discovers and calls them. themis-lake ships two:</p><ul><li><strong>themis-lake</strong> — the <em>wiki brain</em>. A fast Rust MCP server that surfaces your wiki as MCP resources and exposes tools like answer_from_wiki, so the assistant pulls authoritative context on demand instead of guessing. This is what turns a static Markdown repo into something an AI can actively query.</li><li><strong>ai-dev-kit</strong> — the <em>Databricks hands</em>. A thin wrapper around the Databricks AI Dev Kit for Unity Catalog lookups, SQL warehouse queries, and workspace/job/vector-search operations. Where themis-lake supplies knowledge, this one supplies reach into a live workspace. Optional — add it when your agent actually needs to touch real infrastructure.</li></ul><p>A single themis CLI stitches it together: themis install drops the file assets into each AI tool&#39;s config directories, and themis mcp configure wires the MCP servers into the host&#39;s config in whatever dialect it expects (Claude, Copilot, opencode). Same server, same answers, everywhere your engineers work — including uploads into a Databricks workspace so the standards show up inside <strong>Genie Code Agent</strong> mode.</p><h3>Five minutes to a governed assistant</h3><p><a href="https://opencode.ai/">opencode</a> is a terminal AI agent that speaks MCP. Point it at a local <a href="https://ollama.ai/">Ollama</a> model and you get a zero-cost feedback loop for the whole stack — server, skills, and agents — no cloud spend, no API keys.</p><p>You’ll need curl + tar, a Docker-compatible daemon (Docker Desktop, Rancher Desktop, colima), and Ollama installed. Then walk through five short steps.</p><p><strong>1. Pull any tool-capable Ollama model</strong> (qwen3.5 works well):</p><pre>ollama pull qwen3.5:4b</pre><p><strong>2. Install the </strong><strong>themis CLI</strong> — the installer drops a binary into ~/.themis/bin:</p><pre>curl -fsSL https://raw.githubusercontent.com/epilif1017a/themis-lake/master/cli/install.sh | bash</pre><blockquote><em>[!WARNING] </em><strong><em>Put the CLI on your </em></strong><strong><em>PATH before continuing.</em></strong><em> The installer drops the </em><em>themis binary into </em><em>~/.themis/bin, which usually isn&#39;t on your </em><em>PATH yet — so the very next step (</em><em>themis config set …) fails with </em><em>command not found: themis. Add this line to your shell rc (</em><em>~/.bashrc, </em><em>~/.zshrc, </em><em>~/.config/fish/config.fish, …) and </em><strong><em>open a new terminal</em></strong><em> before continuing:</em></blockquote><blockquote><em>export PATH=&quot;$HOME/.themis/bin:$PATH&quot;</em></blockquote><p><strong>3. Point it at the repo</strong> (or your fork) plus a GitHub PAT for the wiki:</p><pre>themis config set --repo epilif1017a/themis-lake --branch master \<br>  --themis-git-token $(gh auth token)</pre><blockquote><em>[!NOTE] The </em><em>--themis-git-token is </em><strong><em>optional when you point at the public </em></strong><strong><em>epilif1017a/themis-lake repo</em></strong><em> — an unauthenticated fetch works, and a token only lifts the GitHub API rate limit from 60 to 5,000 requests/hour. A token </em>is<em> required when your wiki lives in a </em><strong><em>private fork</em></strong><em>. If you don&#39;t have the </em><a href="https://cli.github.com/"><em>GitHub CLI</em></a><em>, just drop the </em><em>--themis-git-token flag:</em></blockquote><blockquote><em>themis config set --repo epilif1017a/themis-lake --branch master</em></blockquote><p><strong>4. Install the file assets</strong> — skills, agents, and instructions:</p><pre>themis install all --target opencode</pre><p><strong>5. Wire in the wiki-brain MCP server:</strong></p><pre>themis mcp configure --target opencode --transport stdio --include themis-lake</pre><p>Working on Databricks? Add the <strong>ai-dev-kit</strong> server for live Unity Catalog, SQL, and workspace access — it mints a fresh OAuth token from your ~/.databrickscfg profile each session, so nothing sensitive lands in your config:</p><pre>themis config set --databricks-profile my-profile<br>themis mcp configure --target opencode --transport stdio --include ai-dev-kit</pre><p>Now launch opencode against Ollama from your repo root:</p><pre>ollama launch opencode</pre><p>That’s it. On first launch the MCP image pulls from GHCR (~30s); after that it’s near-instant. themis-lake’s tools and skills are live in your session — and the assistant answers <em>from your standards</em>, not from the average of the internet.</p><h3>But does the assistant actually obey?</h3><p>Here’s the part most “context for your AI” projects skip: a perfect wiki is worthless if the model ignores it.</p><p>themis-lake ships an <strong>evals harness</strong> that treats that as a first-class problem. It drives <em>real</em> opencode + Ollama + MCP sessions end-to-end and asserts on what the assistant actually <em>does</em> — does it call answer_from_wiki before answering? Does it route pipeline work to the right engine? Does it refuse to invent a convention the wiki doesn&#39;t document? Does it resist a prompt-injection telling it to ignore the wiki? Deterministic substring checks plus LLM-as-judge grading, all running on a local model. It&#39;s a regression net for governance itself — so you find out when a model, wiki edit, or tool change quietly breaks obedience.</p><h3>The most important part: fork it</h3><p><strong>themis-lake is a scaffold, not a finished product.</strong> The CLI, the MCP servers, and the four asset conventions (skills, agents, instructions, wiki) are the <em>product</em>. The wiki and skills shipped in the repo are an <strong>example</strong> — AI-generated lakehouse standards, useful as a starting point, absolutely <em>not</em> meant to be used verbatim.</p><p>The design assumption is that every organization forks it and runs their own private source of truth. Because the wiki is fetched from GitHub at runtime rather than baked into the image, that fork is mostly a <strong>content swap</strong>:</p><ol><li>Replace wiki/ with <em>your</em> standards.</li><li>Point agents/ at your wiki paths.</li><li>Edit instructions/AGENTS.md with your always-on rules.</li><li>Keep, rewrite, or drop the skills based on your platform.</li><li>Set your fork coordinates in ~/.themis/config.</li></ol><p>Change those, and the same battle-tested machinery now reads from your knowledge base instead of the example one. Updating a standard forever after is a pull request — reviewable, versioned, and instantly live for every engineer’s assistant.</p><h3>Contribute back</h3><p>The example content will always be exactly that — an example. But the <strong>engine</strong> improves for everyone:</p><ul><li>New install targets and MCP host dialects.</li><li>More (and sharper) evals for measuring assistant obedience.</li><li>Additional MCP servers under mcp-servers/.</li><li>Support for the environments not covered yet (GitHub Enterprise Server, airgap).</li><li>Better skills and agent patterns that generalize across platforms.</li></ul><p>If you fork themis-lake and sand down a rough edge in the CLI, the server, or the evals, send it upstream. Every team that governs its lakehouse AI a little better makes the scaffold better for the next one.</p><p>Bug reports, feature requests, and PRs are all welcome. Start with the repo, read CONTRIBUTING.md, and open an issue with what you&#39;re trying to do.</p><p><strong>Stop re-explaining your lakehouse.</strong> Fork themis-lake, load your standards once, and let every AI assistant your team touches finally build data products <em>your</em> way.</p><p>→ <a href="https://github.com/epilif1017a/themis-lake">github.com/epilif1017a/themis-lake</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b63475cfdf1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structuring a Data & AI Organization as an Octopus]]></title>
            <link>https://medium.com/@epilif1017a/structuring-a-data-ai-organization-as-an-octopus-adfc7ab45b92?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/adfc7ab45b92</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[organization-design]]></category>
            <category><![CDATA[organizational-change]]></category>
            <category><![CDATA[organizational-culture]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Thu, 23 Oct 2025 14:14:51 GMT</pubDate>
            <atom:updated>2025-10-23T14:55:19.004Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why the old models are broken and how we can structure a federated intelligence model that is built for the AI era</h4><blockquote>Disclaimer: <a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy">AI-assistive technology</a> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</blockquote><p>Data &amp; Analytics departments have long struggled with a fundamental organizational dilemma. On one side, there’s the <strong>centralized “Ivory Tower”.</strong> This model provides strong governance and a single source of truth, but it inevitably becomes a bottleneck, disconnected from real-world business needs. On the other, there’s the <strong>decentralized “Wild West”.</strong> Here, engineers and scientists are embedded in business units. This model offers speed and alignment but descends into chaos: redundant data sets and software, spiralling costs, and a patchwork of non-compliant and unscalable solutions.</p><blockquote>Not a Medium Member? <a href="https://medium.com/@epilif1017a/structuring-a-data-ai-organization-as-an-octopus-adfc7ab45b92?sk=9f536e11893fe792fec07adbbfb186de">Click here to read the full article</a> 🫂</blockquote><p>In the age of generative AI, neither model is sufficient. The pace of change is too fast, and the cost of organizational friction is too high. Therefore, a new blueprint is required. One powerful concept is the <a href="https://hbr.org/2025/11/become-an-octopus-organization"><strong>“Octopus Organization”</strong></a>, a model built for the new realities of AI.</p><blockquote>The Octopus Organization model combines central coordination with decentralized autonomous action.</blockquote><p>In this article, we explore a possible way to apply the concept of “Octopus Organization” to a Data &amp; AI department, exploring potentially related concepts like <a href="https://martinfowler.com/articles/data-mesh-principles.html">Data Mesh</a> and <a href="https://medium.com/another-integration-blog/centre-4-enablement-vs-centre-of-excellence-3aed656993b7">Center for Enablement</a>.</p><h3>What is an Octopus Organization?</h3><p>The octopus can be a metaphor for a modern organization. The octopus is an incredibly intelligent creature, but its intelligence is not centralized. While it has a central brain (the “head”), over two-thirds of its neurons are in its 8 arms (the “tentacles”).</p><p>This creates a system of <strong>distributed intelligence</strong>. Each tentacle can independently sense, think, and act. It can solve a problem with one arm while another is busy with a different task, all while the central head provides strategic coordination and intent.</p><blockquote><em>The octopus is a system built for adaptability, real-time sensing, and decentralized decision-making. </em>Rigid hierarchies, in contrast, are like the ammonite — a creature with a perfect and rigid shell that was wiped out by the asteroid to which the octopus survived.</blockquote><p>For a Data &amp; AI organization, this model can be a way of resolving the “centralized vs. decentralized” debate by being <strong>both, simultaneously</strong>.</p><h3>The Data &amp; AI Octopus Org Blueprint</h3><p>Here is what this model can look like when applying principles of the octopus’ system to a Data &amp; AI department.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2M4zIhdsuXaGLpXkHlwXVQ.png" /></figure><h3>1. The “Head”: Self-Serve Data &amp; AI Platform</h3><p>The “head” is <em>not</em> a command-and-control center. Instead, it’s a <strong>central actor</strong> that empowers the rest of the organization. It provides a <strong>Self-serve Data &amp; AI platform</strong> that enables the “tentacles” to act with speed and autonomy.</p><p><strong>Core Responsibilities:</strong></p><ul><li><strong>Data Platform Component:</strong> Building and maintaining the core data infrastructure, tools &amp; frameworks. This means a <strong>Data Lakehouse</strong> architecture built on the separation of storage and compute. This core component handles the storage and processing of all data types and velocities (e.g., structured, unstructured, and streaming), and provides the foundation for all analytical workloads.</li><li><strong>AI/ML Platform Component:</strong> Providing shared tools &amp; frameworks for the (Generative) AI era. This includes robust <strong>MLOps/LLMOps tooling and framework </strong>for model deployment, a central <strong>model hub</strong> for approved models, and managed services for capabilities like <strong>vector stores, model inference </strong>and<strong> web app hosting</strong>.</li><li><strong>Governance &amp; Security Platform Component:</strong> Setting the “non-negotiable” things. The central team provides the tools for federated governance — like access control, data quality, and data classification — that are embedded as code within the platform.</li></ul><blockquote>Want to know more about building a Data &amp; AI Lakehouse Platform? <a href="https://medium.com/@epilif1017a/structuring-a-data-ai-organization-as-an-octopus-adfc7ab45b92">Check my article series</a>.</blockquote><h3>2. The “Tentacles”: Data &amp; AI Pods as part of an End-to-End Domain</h3><p>A “tentacle” is an autonomous cross-functional organizational domain embedding the respective Data &amp; AI pod, Tech team, and Business team for a specific business area — Sales, Finance, Product, Supply Chain, etc. A Data &amp; AI pod will be the contact point between the Data &amp; AI octopus’ “head” and one of its “tentacles”. This maps directly to the <a href="https://martinfowler.com/articles/data-mesh-principles.html">data mesh</a> principle of <strong>domain-oriented decentralized data ownership</strong>.</p><p><strong>Core Characteristics:</strong></p><ul><li><strong>Composition:</strong> Each pod is a self-contained team of data scientists, AI specialists, data engineers, and product managers, all sitting with their tech counterparts (i.e., the team building the transactional systems) and business stakeholders (i.e., the team actually using the analytical and transactional systems for process execution and decision-making).</li><li><strong>Data as a Product:</strong> The Data &amp; AI pod’s primary job is to “sense” the business environment and build, publish, and own <strong>Data Products</strong>. They don’t just build pipelines; they are accountable for the quality, reliability, and freshness of their domain’s Data &amp; AI artefacts (e.g., the “Sales &amp; Marketing” pod <em>owns</em> the customer_ltv data product, including the database, the pipelines, the metrics/KPIs, the AI models, the Dashboards, etc..).</li><li><strong>Autonomous Model Development:</strong> The tentacles use the central MLOps/LLMOps platform to build and train their <em>own</em> domain-specific models. This also enables federated learning, where models can be trained on data distributed across different domains without the sensitive data ever leaving the tentacle’s control.</li></ul><h3>3. The “Nervous System”: Center for Enablement (C4E)</h3><p>The “nervous system” might be the most critical component in this blueprint. How to provide autonomy without creating chaos? The answer is <strong>federated governance</strong> — a system of trust through the automation of Data &amp; AI compliance checks, best practices, blueprints and observability, connecting the department’s head and tentacles. In this type of organizational blueprint, building a <strong>Data &amp; AI </strong><a href="https://medium.com/another-integration-blog/centre-4-enablement-vs-centre-of-excellence-3aed656993b7"><strong>Center for Enablement</strong></a> is a must.</p><p><strong>Core Components:</strong></p><ul><li><strong>Customization &amp; Configuration of the Unified Data Catalog:</strong> This is the “neural network”. All data products published and used by the “tentacles” should be discoverable in the central catalog provided by the platform team. But in here, the C4E team is responsible to configure and customize that catalog to make sure it contains data that is discoverable, addressable, trustworthy, secure, interoperable and self-described.</li><li><strong>Data Contracts:</strong> To ensure reliability, data products adhere to <strong>data contracts</strong>. This is a versioned, machine-readable agreement on schema, data quality metrics, and service-level agreements (SLAs). For example, this is how the “Finance” tentacle can confidently consume data from the “Sales” tentacle.</li><li><strong>Automated Compliance Checks:</strong> Policies for security, privacy, and quality are automated and embedded in the self-serve data platform. A pod can’t publish a data product that doesn’t meet the organization’s compliance standards.</li><li><strong>Maturity Framework:</strong> a framework to automatically assess and track the adoption of Data &amp; AI guidelines and best practices (e.g., cost-efficient usage of cloud storage and compute, performance, versioning, etc..).</li><li><strong>Data &amp; AI Blueprints: </strong>real projects that implement all the Data &amp; AI platform tooling &amp; frameworks to simultaneously serve business needs and act as an example for all the Data &amp; AI pods.</li></ul><h3>Why the Octopus Data &amp; AI Org is Built for the AI Era?</h3><p>The old Data &amp; AI org models can be too slow for the AI-first world. On paper, this federated octopus model is designed to accommodate for this new reality:</p><ol><li><strong>Speed &amp; Scalability:</strong> AI is not one big project; it’s thousands of small ones. This model allows for dozens of parallel experiments to run simultaneously across the business.</li><li><strong>Business Rich &amp; Reliable AI:</strong> AI is only as good as the data it’s trained on. By making Data &amp; AI pods as a part of an end-to-end organizational domain (AKA a “tentacle”), the entire tentacle is responsible and accountable for their data products and their usage in business, dramatically increasing the context, quality, and reliability of data &amp; models.</li><li><strong>A True AI Experience Layer:</strong> This organizational architecture allows Generative AI to become a natural language “skin” on top of the entire data ecosystem. A user can ask, “How good were our recent products sales given our marketing initiatives in the northeast?” and the AI can query the trusted data products from Product and Marketing &amp; Sales “tentacles” to get a reliable answer, without them being developed by “one single team” that tries to take care of all the data products for the company.</li><li><strong>Talent:</strong> Top Data &amp; AI talent wants impact. This model gives them the autonomy to build and deploy with a very tangible impact within each business domain (“tentacle”), making the organization a magnet for talent.</li></ol><p>The era of the rigid top-down data hierarchy might be over. The challenge for Data &amp; AI leaders is no longer to be the central “brain”, but rather to build an organic system for a smarter, faster, and more adaptable organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adfc7ab45b92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aeropress Soup Espresso Shots]]></title>
            <link>https://medium.com/@epilif1017a/aeropress-soup-espresso-shots-4f2146bfdba2?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f2146bfdba2</guid>
            <category><![CDATA[espresso]]></category>
            <category><![CDATA[coffee-roasters]]></category>
            <category><![CDATA[coffee]]></category>
            <category><![CDATA[coffee-culture]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 22:31:34 GMT</pubDate>
            <atom:updated>2025-09-15T17:18:52.234Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently the soup method for espresso has been gaining some traction amongst coffee enthusiasts. These shots can be summarised as espresso style shots pulled with a very gentle soak and a &lt;1 bar, almost no pressure, extraction.</p><blockquote>Not a Medium Member? <a href="https://medium.com/@epilif1017a/aeropress-soup-espresso-shots-4f2146bfdba2?sk=1ad20e0c3ce6a2395f0fe41729738d63">Read the full article here</a> 🫂</blockquote><p>There are some options when it comes to coffee gear that you can use for soup shots. Lance Hedrick has a <a href="https://youtu.be/_GeFEwNfLZg?si=XeERtTKAAUHYMobY">great video on that</a>. On his video, Lance addresses that soup shots on the Aeropress, although possible, might be cumbersome or prone to some problems (e.g., leakage from the bottom sides or difficulties related to the piston gasket).</p><p>I’m a long Aeropress user, I love it, and I’m also a very conscious buyer, so I don’t tend to go buy new gear very often. If you have an Aeropress, or were thinking about buying it for other brew methods, and are also interested in soup shots, stick around ;)</p><h3>What will you Need?</h3><ol><li>Obviously, an Aeropress.</li><li>A 3D printed <a href="https://www.thingiverse.com/thing:4931178">Aeropress tamper like this</a> (or anything that fits the Aeropress chamber and can tamp the coffee bed). I have 3D printer, so for me it was easy. But if you don’t have one, try to ask a friend to print it, or maybe find a 3D printing service nearby.</li><li>58mm <a href="https://amzn.eu/d/9tfcwxS">paper filters like this</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHr13jCAb2BaZAHR8r-RRA.jpeg" /></figure><h3>My Aeropress Soup Method</h3><ul><li>Grind 20g of your favourite coffee some clicks above your top espresso range. For me, on a Mazzer Philos, this is between 40–60 clicks (240–360 microns), depending on the coffee variety and roast level. Your mileage may vary, but use the bitter/sour compass to adjust grind size by taste. Moreover, use resistance as a measure to grind coarser or finer. You should find some resistance when pressing in the Aeropress, but it should not feel like you need all your body weight to get the water through the coffee bed.</li><li>(optional) Inside your catch cup, or Blind Shaker if you have one, shake the grounds. Shaking the grounds has been correlated to a higher extraction and some arguable flavour improvements.</li><li>Boil water to 96°c.</li><li>Put an Aeropress paper filter in the Aeropress screen; optionally, rinse it; and attach it to the Aeropress chamber.</li><li>Dump the coffee in the chamber and shake it to level the bed.</li><li>Tamp the coffee bed using the tamper. Tamp it until you feel like the bed is super compact. Remove the tamper from the Aeropress.</li><li>Rinse a 58mm paper filter and “glue” it to the bottom of the 3D printed tamper.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUcCsxTBs4TJZVKAFfaRQw.jpeg" /></figure><ul><li>Tamp firmly once again, but this time with the “glued” paper filter on the tamper. You will notice the 58mm paper filter will be &quot;glued” to the coffee bed. If not, before lifting the tamper, gently rotate it a few times, and the paper filter should “unglue”.</li><li>At this moment, you can place the Aeropress on top of a cup.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ts2p75758shIN-D4oZdqDQ.jpeg" /></figure><ul><li>Using a scale, add 80–100g of the 96°c water to the Aeropress chamber. You will notice the water stays perfectly on top of the coffee bed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyVmMxXsT2-AHutktmFEYw.jpeg" /></figure><ul><li>Remove it from the scale and start pressing gently to soak the puck and to have the coffee start dripping into the cup. When you feel like all holes have started dripping, press as hard and fast as you can till you hear the hissing sound.</li><li>Stop at the hissing sound to avoid astringency.</li><li>Clean the Aeropress and admire the perfect puck that comes out of it 😉</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_rktWaIZ3F6xGaxXlCKczw.jpeg" /></figure><h3>Results</h3><p>Good news! I found no problems with this Aeropress soup method. The resulting cup, depending on the coffee beans and roast level, should be a concentrated, clean, bright, balanced, and sweet soup espresso without crema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-A57Q-cV5E35owlfzIVSvg.jpeg" /></figure><p>I can only say that as a medium light 9 bar espresso lover… an Aeropress ~200ml brew lover… I’m now an Aeropress soup shot lover!</p><p>If you’re curious, give it a shot… literally 😂 I hope you enjoyed the article and that it helps you brew a tasty shot!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f2146bfdba2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Secret to Guilt-Free Dessert: A 5-Minute Protein Cheesecake You Can’t Resist]]></title>
            <link>https://medium.com/@epilif1017a/the-secret-to-guilt-free-dessert-a-5-minute-protein-cheesecake-you-cant-resist-6feb8d710e5f?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/6feb8d710e5f</guid>
            <category><![CDATA[recipe]]></category>
            <category><![CDATA[fitness]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[food]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 19:14:09 GMT</pubDate>
            <atom:updated>2025-09-09T06:15:04.687Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Secret to Guilt-Free Dessert</h3><h4>A 5-Minute Protein Cheesecake You Can’t Resist</h4><blockquote><em>Disclaimer: </em><a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy"><em>AI-assistive technology</em></a><em> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</em></blockquote><p>Ever find yourself craving something rich and creamy but don’t want to derail your healthy eating goals? I’ve been there. The struggle between wanting a delicious treat and staying on track is real. That’s why I started experimenting, and let me tell you, this recipe is a game-changer.</p><blockquote>Not a Medium Member? <a href="https://medium.com/@epilif1017a/the-secret-to-guilt-free-dessert-a-5-minute-protein-cheesecake-you-cant-resist-6feb8d710e5f?sk=7475aa348e126cb5acc500d0fbbd1dad">Click here to read the full article</a> 🫂</blockquote><p>This isn’t your grandma’s dense, hours-in-the-oven cheesecake. This is a 5-minute, no-bake protein cheesecake that tastes just as good, if not better, and is packed with protein. It’s perfect for a quick breakfast, a satisfying post-workout snack, or a late-night treat that won’t leave you with a sugar crash.</p><p>The secret? A combination of light cottage cheese and light cream cheese for that authentic cheesecake texture, sweetened with erythritol and a punch of flavor from lime. And of course, the protein powder makes it extra creamy while adding that muscle-building boost.</p><p>Ready to make your new favorite dessert? Let’s get to it.</p><h3>Ingredients</h3><p>This recipe is simple, with three main components: the base, the cream, and the topping.</p><h4>For the Base:</h4><ul><li>20g oats</li><li>5g sugar-free maple syrup</li><li>5g light butter</li></ul><h4>For the Cream:</h4><ul><li>100g light cottage cheese</li><li>100g light cream cheese</li><li>30g vanilla or white chocolate protein powder</li><li>Juice of 1 lime</li><li>Zest of 1 lime</li><li>30g erythritol</li><li>3g xanthan gum</li></ul><h4>For the Topping:</h4><ul><li>15g grated white chocolate OR</li><li>15g raw melted cacao butter mixed with 10g milk powder</li></ul><h3>Instructions</h3><ol><li>Prep the Base: In a small bowl, combine the oats, sugar-free maple syrup, and light butter. Mix until it forms a crumbly paste. Press this mixture firmly into the bottom of a small glass or ramekin. This will be your crunchy, delicious base.</li><li>Make the Cream: This is where the magic happens. Add the light cottage cheese, light cream cheese, vanilla or white chocolate protein powder, lime juice and zest, erythritol, and xanthan gum to a blender or food processor. Blend on high until the mixture is completely smooth and all the cottage cheese lumps are gone. This should only take about a minute. The xanthan gum will help thicken the mixture to that perfect cheesecake consistency.</li><li>Assemble and Top: Pour the smooth cream mixture over your prepared oat base. For the topping, you have two equally delicious options: either sprinkle the grated white chocolate over the top or, for a more intense flavor, drizzle the melted cacao butter and milk powder mixture.</li><li>Chill and Enjoy: This is the most important step for a proper cheesecake! Place your dessert in the refrigerator and let it chill for at least 6 hours, or preferably overnight. This allows the xanthan gum to work its magic and the flavors to meld, resulting in a firm and truly satisfying texture.</li></ol><p>This cheesecake is so simple to make, yet it feels incredibly indulgent. It’s rich, tangy, and sweet, hitting all the right notes without the added sugar and calories of a traditional recipe.</p><p>So next time you’re craving something sweet, skip the store-bought junk and whip up this protein-packed masterpiece instead. Your taste buds—and your body—will thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6feb8d710e5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building the Data Lakehouse Infra with Infrastructure as Code (IaC)]]></title>
            <link>https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-eb50d53bbd27?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb50d53bbd27</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Fri, 15 Aug 2025 16:00:15 GMT</pubDate>
            <atom:updated>2025-09-09T06:17:26.169Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2.1 of “The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era”</h4><blockquote>Disclaimer: <a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy">AI-assistive technology</a> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</blockquote><p>In the first module of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>, we introduced the concepts and laid out a possible open-source architecture for our lakehouse. In <strong><em>Module 2</em> — Building the Foundation with Infrastructure as Code, </strong>we will start by getting our hands dirty with Infrastructure as Code (IaC) to automate the provisioning and management of our data lakehouse infrastructure. This will ensure a repeatable, version-controlled, and scalable environment.</p><blockquote>Not a Medium member? <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-eb50d53bbd27?sk=0a6245f2a9b29711f497e666050810f2">Read the full article here</a> 🫂</blockquote><p>We’re building this platform for a fictitious specialty coffee shop and roastery here in Portugal, fictitiously named <em>“BeanInPT”</em>. Think of it as creating the digital backbone for everything from tracking bean origins to predicting the demand for our signature <em>pastel de nata</em> lattes.</p><p><a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f">Our blueprint</a>, as you’ll recall, features a powerful stack including Apache Spark, Delta Lake, MLflow, S3, Unity Catalog, among others, designed to turn raw data into actionable insights and intelligent applications.</p><p>Now, it’s time to roll up our sleeves and start building. But before we pour any data, we need to build the house itself. In the world of modern data platforms, we don’t lay bricks by hand. We write code. This brings us to the core of this module: <strong>Infrastructure as Code (IaC)</strong>.</p><p>This article will cover three foundational lessons:</p><ul><li><strong>Lesson 1:</strong> Why IaC is Essential for Modern Data Platforms.</li><li><strong>Lesson 2:</strong> An Introduction to Terraform: Core Concepts and Syntax.</li><li><strong>Lesson 3:</strong> Setting Up the Foundational Infrastructure Stack.</li></ul><p>All of the code is available in the <a href="https://github.com/epilif1017a/data-lakehouse-101">data-lakehouse-101 repo</a>, and you can check the README.md file there for troubleshooting.</p><h3>Why IaC is Essential for Modern Data Platforms?</h3><p>Imagine designing the perfect layout for our coffee shop. You’d want a precise, repeatable blueprint to ensure every outlet has the same efficient workflow and customer experience. Manually setting up servers, storage, and networks for our data platform is like building each coffee shop from memory — it’s slow, inconsistent, and prone to costly errors.</p><p><strong>Infrastructure as Code (IaC)</strong> is the practice of managing and provisioning infrastructure through code and software development techniques, such as version control and continuous integration. Instead of clicking through a cloud provider’s console, we define our entire infrastructure in configuration files.</p><p>For our data lakehouse, this approach is not just a nice-to-have; it’s essential for several reasons:</p><ul><li><strong>Automation &amp; Speed:</strong> We can spin up, modify, or replicate our entire platform — from the S3 buckets for our Bronze, Silver, and Gold data layers to the clusters running Spark — in minutes.</li><li><strong>Consistency:</strong> IaC eliminates configuration drift between development, testing, and production environments. This ensures that what we test locally is exactly what gets deployed.</li><li><strong>Version Control:</strong> By storing our infrastructure definitions in a Git repository, we get a full history of every change. Need to roll back to a previous version? It’s as simple as git revert.</li><li><strong>Scalability:</strong> As our coffee empire grows from one shop in Guimarães, Portugal to a nationwide chain, our data needs will explode. IaC allows us to scale our platform effortlessly by simply changing a few lines of code.</li></ul><p>Adopting IaC means we treat our infrastructure with the same rigor as our application code, leading to a more robust, secure, and cost-effective data platform.</p><h3>Introduction to Terraform: Core Concepts and Syntax</h3><p>Our chosen tool for implementing IaC is <strong>HashiCorp Terraform</strong>, an open-source standard for defining and provisioning cloud infrastructure. It uses a declarative syntax, meaning you describe the <em>desired state</em> of your infrastructure, and Terraform figures out how to get there.</p><p>Let’s break down Terraform’s core concepts:</p><ul><li><strong>Providers:</strong> These are plugins that allow Terraform to interact with a specific API, whether it’s a cloud provider like AWS, Azure, or GCP, or other platforms. For our architecture, we’ll primarily use the AWS provider implemented through <a href="https://www.localstack.cloud/">Localstack</a>.</li><li><strong>Resources:</strong> A resource is a single infrastructure object, such as a virtual machine, a database, or an S3 bucket. Each resource has a type (e.g., aws_s3_bucket) and a set of arguments that define its configuration.</li><li><strong>Modules:</strong> This is where the real power of IaC comes in. A module is a container for multiple resources that are used together. Think of them as reusable blueprints for common infrastructure patterns. For instance, we can create a module for a standard S3 bucket that includes specific tagging, versioning, and security policies.</li><li><strong>Stacks:</strong> A stack is a collection of modules and resources that are deployed and managed together as a single unit. A stack can represent a complete <strong>data product</strong> (e.g., a “sales data” stack) or a piece of <strong>foundational infrastructure</strong>. Our first stack, the lakh-nucleus-infra, is the latter—it provides the core networking, storage and metadata management services that all other data products will later depend on.</li><li><strong>State:</strong> Terraform keeps a record of the infrastructure it manages in a state file. This file is crucial as it maps the resources defined in your code to the real-world objects that have been created. In this guide, for educational purposes we are storing it locally (default). For production setups, it is <a href="https://aws.amazon.com/blogs/devops/best-practices-for-managing-terraform-state-files-in-aws-ci-cd-pipeline/">generally recommended</a> that this state file is stored in a secure and concurrency-controlled storage system (e.g., S3).</li><li><strong>Execution Plan:</strong> When you run terraform plan, Terraform compares your desired state (your code) with the current state (the state file) and presents a summary of the changes it will make (create, modify, or destroy).</li></ul><p>By combining resources and modules into stacks, we can create a clean, organized, and highly scalable data platform architecture.</p><h3>Assembling the Foundational Stack</h3><p>Theory is great, but let’s get our hands dirty. Our goal is to create a complete local development environment that mirrors <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f">our target architecture</a>.</p><p>To build and test our lakehouse without running up a cloud bill, we’ll create a local development environment using <strong>Docker</strong> and <strong>LocalStack</strong>.</p><ul><li><strong>LocalStack</strong> will simulate AWS cloud services on our local machine. This is a game-changer for development, allowing us to provision “cloud” resources like a VPC and S3 buckets locally.</li><li><strong>Docker Engine </strong>(or Podman) will help us run the platform’s core technologies in isolated containers. In a real production environment, this is likely not required, as our needed technologies are likely to be part of a Platform as a Service (PaaS) offering, such as Databricks, and/or installed on a Public Cloud (e.g., AWS).</li><li><strong>Terraform</strong> will assist us in managing and provisioning all the required infrastructure.</li></ul><h4>Prerequisites</h4><ol><li><a href="https://docs.docker.com/engine/install/">Docker Engine</a> (for Podman, check the <a href="https://github.com/epilif1017a/data-lakehouse-101">data-lakehouse-101 README.md</a>).</li><li><a href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli">Terraform</a>.</li><li><a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">AWS cli</a>.</li></ol><h4><strong>Step 1: Create the Project Structure</strong></h4><p>This structure separates our reusable modules inlakh-nucleus-iac-modules, from the foundational lakh-nucleus-infra stack.</p><p>Create a parent folder named, for example, `data-lakehouse-101` (optionally, you can also checkout the <a href="https://github.com/epilif1017a/data-lakehouse-101">data-lakehouse-101 repo</a>).</p><pre>.<br>├── lakh-nucleus-iac-modules/<br>│   ├── s3-bucket/<br>│   │   ├── main.tf<br>|   │   ├── providers.tf<br>│   │   └── variables.tf<br>│   └── vpc/<br>│       ├── main.tf<br>|       ├── providers.tf<br>│       └── variables.tf<br>└── lakh-nucleus-infra/<br>    ├── data-catalog.tf<br>    ├── local-cloud.tf<br>    └── providers.tf</pre><p>Our first stack, the lakh-nucleus-infra, will establish the core, shared infrastructure:</p><ol><li>A local S3 bucket to store everything related to lakehouse management (e.g., platform metadata, platform configurations, etc..).</li><li>A Data Catalog, our Metadata Management and Data Governance system based on <a href="https://www.unitycatalog.io/">Unity Catalog</a>.</li><li>A very simple local <strong>Virtual Private Cloud (VPC) </strong>to emulate an AWS VPC. We will not dive deep into a production-grade VPC setup, as this local emulation will not be required throughout our guide, since we will be using local containers. Therefore, replicating an adequate production-ready VPC would be an unrewarding complex task. If you want to understand how to setup a custom VPC suitable for a lakehouse you can refer to <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-getting-started.html">AWS docs</a> and <a href="https://docs.databricks.com/aws/en/security/network/classic/customer-managed-vpc">Databricks docs</a>, for example, and use them to understand the required fundamentals for networking.</li></ol><h4><strong>Step 2: Create Reusable Terraform Modules</strong></h4><p><strong>VPC Module:</strong> This module creates a basic network environment.</p><p>File: lakh-nucleus-iac-modules/vpc/main.tf</p><pre>resource &quot;aws_vpc&quot; &quot;main&quot; {<br>  cidr_block = var.vpc_cidr_block<br>  tags = {<br>    Name = &quot;local-vpc&quot;<br>  }<br>}</pre><p>File: lakh-nucleus-iac-modules/vpc/providers.tf</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}</pre><p>File: lakh-nucleus-iac-modules/vpc/variables.tf</p><pre>variable &quot;vpc_cidr_block&quot; {<br>  description = &quot;CIDR block for the VPC&quot;<br>  type        = string<br>  default     = &quot;10.0.0.0/16&quot;<br>}</pre><p><strong>S3 Bucket Module:</strong> This module creates a standardized S3 bucket.</p><p>File: lakh-nucleus-iac-modules/s3-bucket/main.tf</p><pre>resource &quot;aws_s3_bucket&quot; &quot;this&quot; {<br>  bucket = var.bucket_name<br>  tags = {<br>    Name = &quot;Data Lakehouse Bucket&quot;<br>  }<br>}</pre><p>File: lakh-nucleus-iac-modules/s3-bucket/providers.tf</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}</pre><p>File: lakh-nucleus-iac-modules/s3-bucket/variables.tf</p><pre>variable &quot;bucket_name&quot; {<br>  description = &quot;The name of the S3 bucket&quot;<br>  type        = string<br>}</pre><h4><strong>Step 3: Define the </strong><strong>lakh-nucleus-infra Stack</strong></h4><p>This stack will consume our modules to build the core platform components.</p><p><strong>File</strong>: lakh-nucleus-infra/local-cloud.tf</p><pre>resource &quot;docker_image&quot; &quot;localstack&quot; {<br>  name         = &quot;localstack/localstack:latest&quot;<br>  keep_locally = true<br>}<br><br>resource &quot;docker_network&quot; &quot;lakh_net&quot; {<br>  name = &quot;lakh-net&quot;<br>}<br><br>resource &quot;docker_volume&quot; &quot;localstack_volume&quot; {<br>  name = &quot;localstack&quot;<br>}<br><br># LocalStack container, equivalent to the &#39;localstack&#39; service.<br>resource &quot;docker_container&quot; &quot;localstack&quot; {<br>  name  = &quot;localstack&quot;<br>  image = docker_image.localstack.image_id<br><br>  ports {<br>    internal = 4566<br>    external = 4566<br>  }<br><br>  # Dynamically creates port mappings for the 4510-4559 range.<br>  dynamic &quot;ports&quot; {<br>    for_each = range(4510, 4560) # Note: range() is exclusive of the end value.<br>    content {<br>      internal = ports.value<br>      external = ports.value<br>    }<br>  }<br><br>  env = [<br>    &quot;SERVICES=s3,ec2&quot;,<br>    &quot;DEBUG=1&quot;,<br>    &quot;GATEWAY_LISTEN=0.0.0.0:4566&quot;,<br>    &quot;MAIN_CONTAINER_NAME=localstack&quot;<br>  ]<br><br>  volumes {<br>    volume_name    = docker_volume.localstack_volume.name<br>    container_path = &quot;/var/lib/localstack&quot;<br>  }<br>  volumes {<br>    host_path      = &quot;/var/run/docker.sock&quot;<br>    container_path = &quot;/var/run/docker.sock&quot;<br>  }<br><br>  networks_advanced {<br>    name = docker_network.lakh_net.name<br>  }<br>}<br><br>resource &quot;time_sleep&quot; &quot;wait_30_seconds&quot; {<br>  # make sure that localstack container is ready<br>  create_duration = &quot;30s&quot;<br><br>  depends_on = [<br>    docker_container.localstack<br>  ]<br>}<br><br>module &quot;vpc&quot; {<br>  source = &quot;../lakh-nucleus-iac-modules/vpc&quot;<br><br>  depends_on = [<br>    time_sleep.wait_30_seconds<br>  ]<br>}<br><br>module &quot;bucket&quot; {<br>  source      = &quot;../lakh-nucleus-iac-modules/s3-bucket&quot;<br>  bucket_name = &quot;beaninpt-lakh-nucleus&quot;<br><br>  depends_on = [<br>    module.vpc<br>  ]<br>}</pre><p>This core configuration focuses on our local emulation of the AWS cloud, using a local container in Docker, and required to set the ground for the storage (S3) and networking (AWS VPC) needs of the lakehouse.</p><p><strong>Terraform Provider Configuration:</strong> This tells Terraform how to connect to our local AWS simulation. File: lakh-nucleus-infra/providers.tf</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br><br>    docker = {<br>      source  = &quot;kreuzwerker/docker&quot;<br>      version = &quot;~&gt; 3.0.2&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region                      = &quot;us-east-1&quot;<br>  access_key                  = &quot;test&quot;<br>  secret_key                  = &quot;test&quot;<br>  skip_credentials_validation = true<br>  skip_metadata_api_check     = true<br>  skip_requesting_account_id  = true<br>  endpoints {<br>    s3  = &quot;http://s3.localhost.localstack.cloud:4566&quot;<br>    ec2 = &quot;http://localhost.localstack.cloud:4566&quot;<br>  }<br>}<br><br>provider &quot;docker&quot; {}</pre><p><strong>Data Catalog: </strong>This is our Data Catalog infrastructure, which is also part of our core, as we cannot offer a Lakehouse without any system to manage metadata (e.g., schemas, tables, columns).</p><p>File: lakh-nucleus-infra/data-catalog.tf</p><pre>resource &quot;docker_volume&quot; &quot;unitycatalog_conf&quot; {<br>  name = &quot;unitycatalog_conf&quot;<br>}<br><br>resource &quot;docker_volume&quot; &quot;unitycatalog_data&quot; {<br>  name = &quot;unitycatalog_data&quot;<br>}<br><br>resource &quot;docker_image&quot; &quot;uc_server&quot; {<br>  name         = &quot;unitycatalog/unitycatalog:latest&quot;<br>  keep_locally = true<br>}<br><br>resource &quot;docker_image&quot; &quot;uc_ui&quot; {<br>  name         = &quot;unitycatalog/unitycatalog-ui:main&quot;<br>  keep_locally = true<br>}<br><br>resource &quot;docker_container&quot; &quot;uc_server&quot; {<br>  name  = &quot;uc-server&quot; # do not change as unity catalog ui as it configured as PROXY_HOST build arg<br>  image = docker_image.uc_server.image_id<br><br>  ports {<br>    internal = 8080<br>    external = 8080<br>  }<br><br>  volumes {<br>    volume_name    = docker_volume.unitycatalog_conf.name<br>    container_path = &quot;/opt/unitycatalog/etc/conf&quot;<br>  }<br>  volumes {<br>    volume_name    = docker_volume.unitycatalog_data.name<br>    container_path = &quot;/opt/unitycatalog/etc/data&quot;<br>  }<br><br>  # All containers are added to the same network for inter-service communication.<br>  # The alias is crucial for the Spark container to resolve &#39;unity-catalog-server&#39;.<br>  networks_advanced {<br>    name    = docker_network.lakh_net.name<br>    aliases = [&quot;server&quot;, &quot;unity-catalog-server&quot;]<br>  }<br>}<br><br>resource &quot;docker_container&quot; &quot;uc_ui&quot; {<br>  name  = &quot;uc-ui&quot;<br>  image = docker_image.uc_ui.image_id<br><br>  ports {<br>    internal = 3000<br>    external = 3000<br>  }<br><br>  networks_advanced {<br>    name = docker_network.lakh_net.name<br>  }<br><br>  depends_on = [<br>    docker_container.uc_server<br>  ]<br>}</pre><h4>Step 4: Provision the Nucleus Infra</h4><p>With all our configuration files in place, launching the environment is a 2-step process.</p><ul><li>Using your terminal, navigate into the data-lakehouse-101/lakh-nucleus-infra directory:</li></ul><pre>$ cd data-lakehouse-101/lakh-nucleus-infra</pre><ul><li>Provision the lakh-nucleus-infra infrastructure using Terraform.</li></ul><pre># Initialize Terraform<br>$ terraform init</pre><pre># Plan and apply the changes<br>$ terraform plan<br><br>$ terraform apply<br><br># To destroy it, you can run `terraform destroy`.</pre><h4>Step 5: Test the Nucleus Infra</h4><ul><li>List the S3 buckets in the lakehouse.</li></ul><pre>$ export AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566<br><br>$ aws s3 ls          <br>2025-08-15 12:26:30 beaninpt-lakh-nucleus</pre><ul><li>Browse through the <a href="http://localhost:3000/">Unity Catalog</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yFxIVke6SBvhiBJJpc8oEQ.png" /></figure><p>Congratulations! You have just used a modular IaC approach to build and deploy the foundational infrastructure of our lakehouse. With our foundational lakh-nucleus-infra, we’ve created a version-controlled, repeatable, and stable base upon which we can now start building our actual data products!</p><h3>Where to go from here?</h3><p>Once available, check the <strong>next article </strong>in<strong> Module 2 </strong>of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-from-lakehouse-infrastructure-to-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb50d53bbd27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Platform Nucleus, Data Products & Example Open Source Architecture]]></title>
            <link>https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d61c962424f</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 21:16:06 GMT</pubDate>
            <atom:updated>2025-09-09T06:18:10.988Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1.2 of “The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era”</h4><blockquote>Disclaimer: <a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy">AI-assistive technology</a> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</blockquote><p>I’ve seen organizations go from “data chaos” to “data wizardry” all thanks to a solid data platform. If you’ve ever felt overwhelmed by the sheer volume of information out there, or wondered how companies turn raw data into brilliant insights, you’re in the right place! This article it’s the <strong>2nd</strong> article of <strong>Module 1</strong> of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-from-lakehouse-infrastructure-to-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>. If you haven’t check the <strong>1st article</strong> of <strong>Module 1</strong>, where I introduce different data platforms and focus a bit on the data lakehouse, you can <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-625debae325b">check it here.</a></p><blockquote>Not a Medium member? <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f?sk=46a80e2cde5c5f07d26e5f2baad4cbca">Read the full article here</a> 🫂</blockquote><p>In this article, we’re going to unpack three super cool concepts: the mysterious Data Platform Nucleus, the ever-so-useful Data Products, and a peek into an awesome open source architecture that brings it all to life.</p><h3>The Data Platform Nucleus: The Brains of the Operation! 🧠</h3><p>Imagine your data platform as a highly sophisticated, super-powered cell. At its very core, the “Data Platform Nucleus” is the brain, the control center, the beating heart of your entire data ecosystem. It’s what keeps everything running smoothly and smartly.</p><p>This nucleus isn’t just one thing; it’s a dynamic duo of layers working in perfect harmony:</p><ul><li><strong>The Physical Layer: The Nucleus Infra</strong> — This is like the skeleton and muscles of our data cell. It’s the foundational infrastructure, the “hardware”, and the underlying services that power everything. Think servers, storage, networks — the heavy lifting stuff!</li><li><strong>The Logical Layer: The Nucleus Framework</strong> — Now, this is where the magic truly happens! This layer is all about the operational and governance mechanisms that define how the infrastructure is created and managed and how the data is stored, cataloged, moved, processed, and secured.</li></ul><p>Within this Logical Layer, we have specialized “Nucleons” — like tiny, super-efficient organs, each with a crucial job:</p><ul><li><strong>Infra as Code (IaC) Nucleon:</strong> Ever wished your infrastructure could just… build itself? That’s what IaC does! This nucleon automates the provisioning and management of all the tech for both the Nucleus and your awesome Data Products. It’s like having a robot architect!</li><li><strong>Data I/O &amp; Transformation Nucleon:</strong> This is your data’s personal stylist and translator! It handles all sorts of data formats, performs mind-bending transformations, and manages those clever ingestion tools that suck data in from everywhere.</li></ul><blockquote>My favourite <strong>Data I/O &amp; Transformation Nucleon for a lakehouse is for sure the </strong><a href="https://github.com/adidas/lakehouse-engine"><strong>Lakehouse Engine</strong></a>, check it out!</blockquote><ul><li><strong>MLOps Nucleon:</strong> For all you AI enthusiasts, this one’s for you! The MLOps Nucleon is the maestro behind managing the entire lifecycle of your machine learning models, from development to deployment and monitoring. A simple path for AI app development!</li><li><strong>Data Visualization Nucleon:</strong> Because who wants to stare at endless spreadsheets? This nucleon makes sure your data can be presented beautifully and interactively, turning numbers into compelling stories.</li><li><strong>Data Governance Nucleon:</strong> This is the wise elder of the group, ensuring everything is fair, square, and secure! It’s absolutely critical for data quality, privacy, and compliance. This includes things like strict access controls, PII (Personally Identifiable Information) rules, data quality rules, among others.</li></ul><p>The ultimate mission of this incredible Nucleus? To be “rigorously governed, easily extensible”. Meaning, it’s super secure and compliant, but also flexible enough to grow and adapt with your wildest data dreams!</p><h3>Data Products: Your Data, Packaged for Awesomeness! 📦✨</h3><p>Alright, let’s talk about something truly revolutionary: “Data Products”. Forget just raw data sitting around! Imagine data not as a messy pile, but as a perfectly wrapped gift, ready to be used. A data product is literally the “smallest indivisible unit of Data” that’s ready to be consumed. It’s like a LEGO brick, but for data — self-contained and ready to snap into place. This perfectly aligns with the principles of the Data Mesh, a modern approach to data architecture. In the <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-625debae325b">first article of this guide</a>, you can find a brief introduction to the concept of Data Product and how it relates to the <a href="https://martinfowler.com/articles/data-mesh-principles.html">data mesh paradigm</a>.</p><p>What makes a Data Product so special?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ETAuQhQ0fKfoto03" /><figcaption>Source: generated by AI.</figcaption></figure><ul><li><strong>All-Inclusive:</strong> Each data product isn’t just data; it’s a complete package! It includes its own infrastructure and software for storage, processing, <em>and</em> governance throughout its entire life cycle. It’s like getting a pre-assembled, fully functional mini-data center!</li><li><strong>Nucleus Approved:</strong> These cool kids are offered by the Nucleus Framework, meaning they leverage all the awesome capabilities and governance rules provided by our Data Platform Nucleus.</li><li><strong>Compute Versatility:</strong> Data products are flexible! They can use various compute environments, whether it’s SQL Purpose Clusters for your standard database queries, Generic Purpose Clusters for more complex tasks, or even Serverless options for ultimate scalability without managing servers.</li><li><strong>Real-World Examples:</strong> A data product can be built using an AWS S3 Bucket for storage, a Database to store tabular metadata on top of the data stored on S3, which can then be processed with beefy Spark Clusters for heavy-duty ETL and analytics. The possibilities are endless!</li></ul><h3>Medallion Architecture: Your Data’s VIP Journey! 🏆</h3><p>Now, how do we organize all this amazing data so it makes sense and is trustworthy? Enter the “Medallion Architecture”! Think of it like a meticulous purification process, turning raw ore into gleaming gold. It’s a three-tiered system, and each layer has a very specific job and a golden rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PX77iGSiaFTcciRr" /><figcaption>Source: generated by AI.</figcaption></figure><ul><li><strong>Bronze Layer: The Raw Gemstone!</strong> — This is where all your “raw data” first lands, directly from its source, no questions asked. The golden rule here is simple: “extract as is, load as is, do not share”. It’s your immutable, untouched historical record — your data’s original blueprint.</li><li><strong>Silver Layer: The Polished Stone!</strong> — Here, the data gets a good scrub and a little polish! We clean it up, deduplicate it, and enrich it. The rule for the Silver layer: “Extract once, limitedly transform once, and share”. This layer gives you a clean, consistent, and reliable view of your data.</li><li><strong>Gold Layer: The Crown Jewel!</strong> — This is where the magic happens for your business users! The Gold layer serves up “aggregated and consolidated data” that’s perfectly tailored for consumption. The rule: “Extract as much as needed, transform and share freely as much as needed”. This data is perfect for “Dashboards, Reports, AI, Downstream Data Products, Downstream Apps”. It’s the sparkling, ready-to-use insights!</li></ul><h3>A Glimpse into an Open Source Lakehouse: Building Your Data Dream Home! 🏠</h3><p>Want to see how all these pieces fit into a real-world blueprint? Let’s check out a possible lakehouse open source architecture that uses popular open source tools. Imagine building a beautiful data lakehouse, complete with different rooms for different purposes!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDFpXmT7qVna2pCJMUa3hw.png" /><figcaption>Example Open Source Data Lakehouse Architecture.</figcaption></figure><p>Here’s a simplified tour:</p><ul><li><strong>The Entrance (Source &amp; Extract):</strong> Your data starts its journey from various Sources. Then, Apache Spark often acts as the powerful extractor, pulling data into our lakehouse.</li><li><strong>The Lakehouse Itself (Medallion Layers):</strong></li><li><strong>Bronze Room:</strong> This is where the raw, untouched data first lands. It’s like the mudroom of your data house! We can use Delta Lake UniForm here, which makes our data open and accessible. Storage often happens on AWS S3, a super scalable cloud storage.</li><li><strong>Silver Parlor:</strong> Data gets a makeover here! Apache Spark cleans and transforms it, making it shiny and consistent, still leveraging Delta Lake UniForm to store data.</li><li><strong>Gold Ballroom:</strong> This is the grand finale! Here, data is aggregated and refined into easy-to-use tables/views, ready for anyone to enjoy. Again, Delta Lake UniForm keeps things organized.</li></ul><blockquote>What data modelling strategy to choose for the data lakehouse? I previously shared an <a href="https://medium.com/@epilif1017a/yet-another-data-modelling-approach-for-the-data-lakehouse-7b532b60c0d2">article with an approach to model data in a lakehouse</a>. In summary, it’s a performance-aware hybrid approach based on Flat Tables and Star Schemas, and I highly recommend it for the tables stored in the Silver and Gold layers.</blockquote><ul><li><strong>The Brain of the House (Unity Catalog):</strong> Unity Catalog acts like the central library and security guard, providing unified governance and metadata management across all your data assets.</li><li><strong>The Fun Rooms (Downstream Platforms — sometimes integrated into the same Lakehouse provider — e.g., Databricks): DS/ML Platform:</strong> For our data scientists and machine learning engineers, this room is equipped with tools like Apache Spark Jupyter notebooks, LangChain for Gen AI apps, Keras for deep learning, and MLflow for managing the ML models lifecycle; <strong>DA/BI Platform:</strong> For the business analysts and decision-makers, this room has powerful tools like Apache Superset, an open source marvel for building dashboards and reports.</li><li><strong>The Speed Booster:</strong> For extra performance, Gluten can be used to accelerate query processing — making Apache Spark run even faster for analytical purposes! Alternatives, using other SQL engines, include, Presto or Trino, for example.</li></ul><p>This open source blueprint, to which there can be many technological alternatives, shows you that building a cutting-edge data platform doesn’t require reinventing the wheel or breaking the bank. By combining these powerful concepts and tools, you can create a data ecosystem that’s not just functional, but truly transformative.</p><p>This example architecture will be used throughout the articles in this guide, but it’s obviously more suitable for educational purposes rather than for production-ready scenarios in organizations, as maintaining a completely open source stack can be a daunting task. Realistically, you are likely to sleep better by combining a mature lakehouse provider like Databricks on AWS/AZURE/GCP, which either will offer you commercial versions of the open source technologies mentioned above, or can be extended with a few of them to support your concrete Lakehouse use cases.</p><p>So, there you have it — a whirlwind tour of the foundational concepts to build a data lakehouse! Understanding these core concepts is your first step towards becoming a true data maestro. You are one step closer to go forth and build amazing things with data!</p><h3>Where to go from here?</h3><p>Once available, check the <strong>next article </strong>in<strong> Module 1 </strong>of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-from-lakehouse-infrastructure-to-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d61c962424f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Data Platforms and Lakehouse]]></title>
            <link>https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-625debae325b?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/625debae325b</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 21:36:15 GMT</pubDate>
            <atom:updated>2025-09-09T06:18:56.114Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1.1 of “The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era”</h4><blockquote>Disclaimer: <a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy">AI-assistive technology</a> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</blockquote><p>The world is awash in data, and the rise of Generative AI has only intensified the need for robust, scalable, and flexible data platforms. But building one from scratch can be a daunting task. Where do you even begin? This article it’s part of <strong>Module 1</strong> of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-from-lakehouse-infrastructure-to-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>.</p><blockquote>Not a Medium member? <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-625debae325b?sk=b64cc90b68d960a1ff61645395598349">Read the full article here</a> 🫂</blockquote><p>This post will walk you through the essential first principles, explore the evolution of data architectures, and help you decide where your data products should ultimately live.</p><h3>First Things First: Align with Your Organization</h3><p>Before laying a single digital brick, it’s crucial to ensure your data platform strategy is deeply integrated with your company’s DNA. A platform doesn’t exist in a vacuum; it’s a living part of the organization it serves.</p><p>Consider these key questions:</p><ul><li><strong>Company Strategy:</strong> Is your organization’s data strategy centralized, or is it federated across different business units? The platform’s architecture needs to reflect this.</li><li><strong>Key Processes:</strong> What are the most critical business processes? The platform should be built to support these priorities first.</li><li><strong>Ways of Working:</strong> Does your company lean more towards a DevOps culture or a more traditional, ITIL-based approach?</li><li><strong>Value for the Organization’s Ecosystem:</strong> the platform must integrate well with the organizational workflows, so design its perimeter for secure ways to generate value.</li></ul><h3>Core Design Principles for a Modern Data Platform</h3><p>Once aligned with the organization, it’s time to think about the design. Modern platforms are not monolithic giants; they are flexible, user-centric, and built for evolution.</p><p>Here are the guiding principles:</p><ul><li><strong>Minimalist Nucleus:</strong> Start with a lean, core platform that meets the most immediate needs. Build it in a modular fashion so you can easily expand it with new features and capabilities in the future.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SoXqmzKo1as0XKh3" /><figcaption>The platform nucleus and the data products. Source: generated by AI.</figcaption></figure><ul><li><strong>The Data Product:</strong> Think of a “Data Product” as the smallest indivisible unit of your data architecture. The architectural quantum. It encapsulates its own storage and compute, abstracting away organizational complexity.</li></ul><blockquote>Data Product is a fundamental concept of the <a href="https://martinfowler.com/articles/data-mesh-principles.html">data mesh paradigm</a>. I will not delve much into the data mesh paradigm, but it’s important that throughout this guide you acknowledge a data product as an isolated set of data assets that have clear ownership, are domain-specific, and are discoverable, addressable, trustworthy, self-describing, interoperable, and secure. This will be fundamental to build a decentralized data platform without thinking too much if it is a lakehouse, a mesh, a factory, a hub, whatever…</blockquote><ul><li><strong>Data Products</strong> are built with <strong>plug-n-play</strong> elements from the platform Nucleus, and contribute with new features to expand it (inner source).</li><li><strong>Freedom within Boundaries:</strong> Empower your data product teams with one (flexible) Framework (e.g., <a href="https://github.com/adidas/lakehouse-engine">lakehouse engine</a> with custom injection of sources, transformers and destinations) and adopt a “shift-left” mindset where teams can bring their own programming habits and own their data access policies, all within the governance boundaries set by the platform.</li><li><strong>Design for Personas:</strong> Different users have different needs. Design specific user journeys for data engineers, data scientists, and data analysts.</li><li><strong>Govern, Don’t Gatekeep:</strong> Establish clear, “just enough” governance rules for data access (push and pull interfaces), authorization, and privacy. Make data discovery and exploration a first-class citizen, not an afterthought.</li></ul><h3>If We Were Data, Where Would We Live?</h3><p>The central question in data architecture is where the data itself will reside. Over the years, we’ve seen three main contenders: the Data Warehouse, the Data Lake, and the modern Data Lakehouse.</p><h4>The Data Warehouse: Tidy but Rigid</h4><p>The traditional Data Warehouse is excellent for structured data and business intelligence. It’s fast, easy to query using SQL, and provides a single source of truth for reporting. However, it struggles with the demands of the modern data world. It has difficulty handling unstructured data, offers limited support for streaming, and its proprietary formats can create connectivity challenges for data science and machine learning platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvD-u82FxVWhVgJSjmvwnQ.png" /><figcaption><em>Acknowledgment: this image uses icons made by Freepik from </em><a href="http://flaticon.com/"><em>flaticon</em></a><em>.</em></figcaption></figure><h4>The Data Lake: Flexible but Wild</h4><p>The Data Lake emerged to solve the warehouse’s rigidity and scalability problem. It can store vast amounts of raw data in any format — structured, semi-structured, and unstructured. This makes it ideal for data science and machine learning use cases. But this flexibility comes at a cost. Data Lakes often suffer from performance issues, a lack of ACID transactions (which guarantee data integrity), and poor schema management and governance. It can quickly turn into a “data swamp” without rigorous discipline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RC-qLwEuNZ_PSHXEpZ9eWA.png" /><figcaption><em>Acknowledgment: this image uses icons made by Freepik from </em><a href="http://flaticon.com/"><em>flaticon</em></a><em>.</em></figcaption></figure><h4>The Data Lakehouse: The Best of Both Worlds</h4><p>So, if the warehouse is too rigid and the lake is too wild, where do we go? Enter the Data Lakehouse. The Lakehouse combines the performance and governance features of a data warehouse with the low-cost, flexible and scalable storage of a data lake. It’s built on open data formats (e.g., Delta, Iceberg, Hudi) and provides a unified platform for virtually all data workloads, from BI and analytics to data science and generative AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGmngBuJBToik2LpGc8ztg.png" /><figcaption><em>Acknowledgment: this image uses icons made by Freepik from </em><a href="http://flaticon.com/"><em>flaticon</em></a><em>.</em></figcaption></figure><h3>So, Where Should Your Data Products Live?</h3><p>If a Data Warehouse is a tidy but strict home and a Data Lake is like swimming freely in the wild, the <strong>Data Lakehouse is like having a clean, comfortable house right on the shore.</strong></p><p>It offers the best of both worlds:</p><ul><li><strong>An open and diverse ecosystem.</strong></li><li><strong>Reliable ACID transactions.</strong></li><li><strong>Strong governance and metadata management.</strong></li><li><strong>High-speed performance for BI.</strong></li><li><strong>Features like “Time Travel” to track data changes.</strong></li><li>Among many other innovations…</li></ul><p>For any organization looking to build a future-proof data platform, with a decentralized strategy, that can handle the diverse and demanding needs of modern applications — especially Generative AI — the Lakehouse architecture, coupled with the concept of a data product, as explained previously, presents the most compelling and powerful path forward, in my modest opinion.</p><h3>Who Can Help You to Build the Lakehouse?</h3><p>So, you’re sold on the data lakehouse architecture, but who are the key architects you can partner with? The landscape is dominated by a few major players, each offering a unique blueprint.</p><blockquote>While it is possible to build a fully open-source lakehouse on-premises or in the cloud (e.g., AWS, GCP, Azure), there are some key providers of integrated lakehouse solutions, often including fully fledged Data Analytics (DA), BI, and AI platforms also.</blockquote><p><strong>Databricks</strong> is often seen as the pioneer, building its platform on open-source foundations like Apache Spark and Delta Lake to unify data engineering and AI. Then there’s <strong>Snowflake</strong>, a cloud data giant, which extends its powerful data warehousing capabilities to embrace lakehouse workloads, allowing you to query unstructured data right where it lives.</p><p>Of course, the big three cloud providers are all in. <strong>AWS</strong> offers a powerful toolkit with Amazon S3, Lake Formation, Athena, Redshift, and others. <strong>Google Cloud</strong> centers its approach around the powerhouse duo of Google Cloud Storage and the serverless magic of BigQuery, combined with others like Dataproc and Dataflow, for example. Meanwhile, <strong>Microsoft Azure</strong> integrates everything seamlessly with Azure Synapse Analytics and Azure Data Lake Storage, and, more recently, Microsoft Fabric.</p><h3>Where to go from here?</h3><p>Check the <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f"><strong>next article</strong></a><strong> </strong>in<strong> Module 1 </strong>of <a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-from-lakehouse-infrastructure-to-4cee34c94881"><strong>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=625debae325b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Complete Guide for Building a Modern Federated Data Platform for the Generative AI Era]]></title>
            <link>https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-federated-data-platform-for-the-generative-ai-era-4cee34c94881?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cee34c94881</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 21:20:28 GMT</pubDate>
            <atom:updated>2025-08-29T06:01:54.762Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Complete Guide for Building a Modern Decentralized Data Platform for the Generative AI Era</h3><h4>Introduction &amp; Table of Contents</h4><blockquote><em>Disclaimer: </em><a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy"><em>AI-assistive technology</em></a><em> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</em></blockquote><p>I’m excited to introduce you to a guide composed of series of articles focusing on a topic that’s very close to my heart, i.e., how to build a modern data platform from scratch… from infrastructure to data products for GenAI. This isn’t just about throwing a bunch of tech together. It’s about aligning your data strategy with your organization’s unique needs and building something truly impactful. Think of it as “Data Platform 101”.</p><p>This is the introductory post to the complete guide for building a modern decentralized data platform from the ground up, specifically a data lakehouse. In here, besides a small introduction to the guide, you will find all of its modules indexed in the <em>“What’s in the Guide?” </em>section<em> </em>below. These modules and the respective articles will be added as I write them, so the guide will be also be built on a modular fashion! Stay tuned for the release of new articles!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O6HKuS_nrKr5a1Cb" /><figcaption>Source: generated by AI.</figcaption></figure><h3>Why This Guide?</h3><p>In today’s world, data is gold, and building a robust data platform is essential for any organization looking to leverage its data assets effectively. This comprehensive, hands-on guide will mentor you through the end-to-end process of doing just that. You’ll learn to leverage Infrastructure as Code (IaC) to create a robust and scalable data platform, and then crucially, utilize this platform to power cutting-edge Generative AI applications.</p><h3>Who Is This Guide For?</h3><p>Whether you are an aspiring data engineer, a software developer looking to specialise in data, a data analyst eager to understand the underlying infrastructure, or a data manager/executive, this guide will provide you with the practical skills needed to design, build, and manage a complete data lakehouse ecosystem. While we’ll cover everything step-by-step, it’s helpful if you have some foundational understanding of cloud computing concepts, data &amp; analytics concepts, and basic programming skills in Python.</p><h3>What’s in the Guide?</h3><p>Throughout this guide, we’ll work towards several key learning outcomes:</p><p><strong><em>Module 1 </em>— Understand Data Platform Fundamentals</strong>: You’ll grasp how to design a data platform that scales for organisations of any size.</p><ol><li><a href="https://medium.com/@epilif1017a/module-1-625debae325b">Introduction to Data Platforms and Lakehouse</a>.</li><li><a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-1d61c962424f">Platform Nucleus, Data Products &amp; Example Open-Source Architecture</a><em>.</em></li></ol><p><strong><em>Module 2</em>— Building the Foundation with Infrastructure as Code: </strong>you will get your hands dirty with Infrastructure as Code (IaC) to automate the provisioning and management of your data lakehouse infrastructure. This will ensure a repeatable, version-controlled, and scalable environment.</p><ol><li><a href="https://medium.com/@epilif1017a/the-complete-guide-for-building-a-modern-decentralized-data-platform-for-the-generative-ai-era-eb50d53bbd27">Building the Data Lakehouse Infra with Infrastructure as Code (IaC)</a>.</li></ol><blockquote>… new modules and articles will be released on a continuous basis. Stay tuned!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cee34c94881" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Lighter Side of a Portuguese Icon]]></title>
            <link>https://medium.com/@epilif1017a/the-lighter-side-of-a-portuguese-icon-a-francesinha-you-can-feel-good-about-66e6b47f83a7?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/66e6b47f83a7</guid>
            <category><![CDATA[food]]></category>
            <category><![CDATA[recipe]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[nutrition]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Sat, 05 Jul 2025 06:15:59 GMT</pubDate>
            <atom:updated>2025-09-09T06:19:40.320Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Francesinha You Can Feel Good About!</h4><blockquote><em>Disclaimer: </em><a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy"><em>AI-assistive technology</em></a><em> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDe9J40R8KmPQ6oVFA-VYQ.jpeg" /><figcaption>The Light Francesinha that is certainly not light in flavor :)</figcaption></figure><p>Ah, the <strong>Francesinha</strong>! Just the name conjures images of a glorious, gravy-laden, cheese-smothered sandwich. It’s a true culinary icon from Portugal, a dish synonymous with hearty, comforting food.</p><blockquote>Not a Medium member? <a href="https://medium.com/@epilif1017a/the-lighter-side-of-a-portuguese-icon-a-francesinha-you-can-feel-good-about-66e6b47f83a7?sk=9cb92d3c2f75b94424493e8b3ef0a538">Read the full article here</a> 🫂</blockquote><p>But let’s be honest, the traditional Francesinha isn’t exactly known for being a light meal. That’s where this recipe comes in! I’ve taken the essence of this beloved dish — the incredibly rich sauce, the delightful layered meats, and the gooey, melty cheese — and given it a lighter, fresher spin. The result? All the incredible flavor you crave, with significantly less guilt. This is your new go-to for a satisfying, flavor-packed meal that won’t leave you feeling like you need a nap for the rest of the day.</p><p>Ready to dive into this lighter take on a Portuguese classic? Let’s get cooking!</p><h3>Lightened-Up Francesinha</h3><p><strong>Serves:</strong> 6 <strong>Prep &amp; Cook Time:</strong> Approximately 1 hour</p><h3>The Star of the Show: Our Light &amp; Luscious Francesinha Sauce</h3><p>This sauce is the heart and soul of any good Francesinha. We’re building deep layers of flavor with aromatic vegetables, a touch of beer and (optional) Port wine, and a clever use of <strong>evaporated milk, light butter </strong>and <strong>cornstarch</strong> for that irresistible creamy, shiny finish — all without the usual heavy cream and butter overload.</p><blockquote>Side note: the traditional Porto Francesinha sauce is less creamy than the version of my home district (Braga, Portugal). I love both, but this recipe is more about like the creamy ones :)</blockquote><h4>Sauce Ingredients:</h4><ul><li>2 medium <strong>onions</strong>, roughly chopped</li><li>3 <strong>garlic</strong> cloves, minced</li><li>2 <strong>bay leaves</strong></li><li>1 tablespoon <strong>olive oil</strong></li><li>50g <strong>Matinal 85% less fat butter</strong> (or equivalent light butter)</li><li>250ml <strong>tomato purée</strong></li><li>1 <strong>Knorr meat bouillon cube </strong>(or other brands)</li><li>0.5 <strong>Knorr seafood bouillon cube</strong> (or other brands)</li><li>500ml <strong>water</strong></li><li>2x 330ml <strong>beer</strong> (non-alcoholic works perfectly!)</li><li>(Optional but highly advised) 30ml <strong>Port wine</strong></li><li>200ml <strong>evaporated milk</strong></li><li>1.5 tablespoons <strong>cornstarch</strong></li><li>(Optional) 0.5 teaspoon <strong>xanthan gum</strong> (for extra smoothness and thickness)</li><li>1 tablespoon <strong>soy sauce</strong> OR <strong>Worcestershire sauce </strong>(more caloric)</li><li>1 tablespoon <strong>erythritol</strong> (or sugar)</li><li>2 tablespoons <strong>mustard</strong></li><li>3 teaspoons <strong>salt</strong> (adjust to taste)</li><li>Just a hint of <strong>black pepper</strong></li><li>1 teaspoon <strong>smoked paprika</strong></li><li>1 teaspoon <strong>chili sauce</strong> of choice</li></ul><h4>Sauce Instructions:</h4><ol><li><strong>Sauté the Aromatics:</strong> In a large pot or deep pan, heat the <strong>olive oil</strong> over medium heat. Add the <strong>onions</strong>, <strong>garlic</strong>, <strong>bay leaves</strong>, and 1 teaspoon of <strong>salt</strong>. Sauté until the onions are lightly golden and you see a nice brown crust forming at the bottom of the pan — this is where the flavor magic truly begins!</li><li><strong>Melt the Butter &amp; Deglaze:</strong> Add the <strong>light butter</strong> to the pan. Let it melt, then use it to scrape up all those delicious brown bits from the bottom of the pan.</li><li><strong>Add the Liquids &amp; Bouillon:</strong> Once everything is incorporated, pour in the <strong>beer</strong>, the <strong>beef stock cube</strong>, <strong>seafood stock cube</strong>, and, if you’re using it, the <strong>Port wine</strong>.</li><li><strong>Simmer &amp; Reduce:</strong> Bring the mixture to a simmer and let it cook uncovered for 10 minutes. This helps evaporate any alcohol (if you’re using alcoholic beer and Port wine) and allows the flavors to meld beautifully.</li><li><strong>Tomato &amp; Spice Infusion:</strong> Stir in the <strong>tomato purée</strong>, a hint of <strong>black pepper</strong>, your chosen <strong>chili sauce</strong>, <strong>smoked paprika</strong>, the <strong>water</strong>, another teaspoon of <strong>salt</strong>, and the <strong>erythritol</strong> (or sugar).</li><li><strong>Second Simmer:</strong> Let it simmer uncovered for another 20 minutes.</li><li><strong>Final Flavor Boost:</strong> Add the <strong>mustard</strong>, <strong>soy sauce</strong> (or Worcestershire sauce), <strong>evaporated milk</strong>, and the final teaspoon of <strong>salt</strong>. Now’s a good time to taste and adjust the seasoning if needed!</li><li><strong>Remove Bay Leaves:</strong> Carefully fish out and discard the <strong>bay leaves</strong>.</li><li><strong>Thicken &amp; Blend:</strong> Get your <strong>electric immersion blender</strong> ready! In a small bowl, dissolve the <strong>cornstarch</strong> (and the optional <strong>xanthan gum</strong>) in 50ml of boiling water. Pour this mixture into the simmering sauce. Immediately use your immersion blender to blend everything for about 30 seconds until the sauce is smooth and takes on a beautiful, shiny sheen. The xanthan gum really helps with that luxurious mouthfeel!</li><li><strong>Final Simmer:</strong> Let the sauce simmer uncovered for a final 10 minutes.</li><li><strong>Ready to Serve:</strong> Turn off the stove. You can reheat the sauce just before assembling your Francesinhas if needed.</li></ol><h3>Building Your Lighter Francesinha Sandwich</h3><p>This is where the magic comes together! We’re using lighter meats and cheese without compromising on the iconic layers and rich flavor.</p><h4>Ingredients &amp; Assembly <strong><em>(1x Francesinha sandwich)</em></strong>:</h4><ol><li><strong>Bread Base:</strong> Take half a slice of <strong>Rustik bread</strong> (or a similar, sturdy but not overly dense sliced bread). You can lightly toast it if you like, but I prefer the fluffier texture it brings to the Francesinha.</li><li><strong>Turkey/Chicken Ham:</strong> Lay 3 slices of <strong>turkey or chicken ham</strong> over the bread.</li><li><strong>Sausage “Sashimi”:</strong> Grill 1 or 2 <strong>turkey or chicken sausages</strong>. Once cooked, thinly slice them horizontally into three equal parts — they’ll look like delicious sausage “sashimi”! Arrange these on top of the ham.</li><li><strong>Turkey “Chorizo”:</strong> Add 4 slices of <strong>turkey cured loin</strong> that resembles turkey “chorizo.” (Mercadona often has a great option for this!).</li><li><strong>Steak:</strong> Grill a 1cm thick <strong>sirloin or picanha steak</strong> to your preferred doneness. Place it on top of the meat layers.</li><li><strong>Top Bread:</strong> Place the other half of your <strong>Rustik bread</strong> slice on top.</li><li><strong>Melted Cheese Crown:</strong> Arrange 4 slices of your preferred <strong>light cheese</strong> on top of the bread. My go-to is Paiva light cheese, but flamengo and limiano also work wonderfully. To melt the cheese, pop the Francesinha into the oven under the grill function until the cheese is bubbly and melted. <strong>Pro Tip:</strong> Avoid using the oven with air circulation, as it can overcook your steak.</li><li><strong>Runny Egg:</strong> The <em>pièce de résistance</em>! Fry an egg so the yolk is nice and runny, then gently place it on top of your cheese-covered Francesinha.</li></ol><h3>(Optional) Air-Fried French Fries</h3><p>No Francesinha is truly complete without a side of fries to soak up that incredible sauce!</p><p>I love using my air fryer for this. Simply cook your favorite frozen French fries according to the package directions. My current obsession is Mercadona’s (if you’re in Portugal or Spain) thin-strip fries in the blue bag, but any fries you enjoy will do the trick!</p><h3>Bringing It All Together</h3><p>Now for the best part: the serving! Carefully transfer your assembled, cheese-melted Francesinha to a deep plate or bowl. Generously ladle your warm, light Francesinha sauce over the entire sandwich, making sure it’s completely covered. Arrange your air-fried French fries around the edges, ready to dive into the savory sauce.</p><p>There you have it — a lighter, yet incredibly flavorful, <strong>Francesinha</strong>! It’s a taste of Portugal’s culinary heart, made with a little less indulgence but all the same love.</p><p>Enjoy every glorious bite!</p><p>What are your favorite lighter twists on classic comfort foods? Share your ideas in the comments below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66e6b47f83a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Framework for Software Engineering Management]]></title>
            <link>https://medium.com/@epilif1017a/a-framework-for-software-engineering-management-2e0ae758b9fb?source=rss-156ac7d3a898------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e0ae758b9fb</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Carlos Costa]]></dc:creator>
            <pubDate>Wed, 14 May 2025 16:44:15 GMT</pubDate>
            <atom:updated>2025-09-09T06:20:18.912Z</atom:updated>
            <content:encoded><![CDATA[<h4>A “<em>Keep It Simple”</em> Approach</h4><blockquote><em>Disclaimer: </em><a href="https://help.medium.com/hc/en-us/articles/22576852947223-Artificial-Intelligence-AI-content-policy"><em>AI-assistive technology</em></a><em> was used to help me write this story… I hope you don’t mind. I promise you that it does not change the fact that I care deeply for helping people with every article I write, personally! ❤</em></blockquote><p>Life as a software engineering manager or director can be complex by itself, why would we want to complicate it even more with highly theoretical and extensive planning methodologies and HR frameworks?</p><blockquote>Not a Medium member? <a href="https://medium.com/@epilif1017a/a-framework-for-software-engineering-management-2e0ae758b9fb?sk=fcd8be576fd209cdb8d58eafb652542b">Read the full article here</a> 🫂</blockquote><p>This post aims to provide to any engineering manager or director a simple framework for the day-to-day portfolio and people management activities they must carry. I apologize beforehand if most of this content feels common sense to you. Congrats if that is the case, you’re doing great! However, I just felt motivated to structure my thoughts on a post that might, or not, be useful for other people looking for some tips on the management of software engineering portfolios, products and people.</p><h3>It all Starts with the Area’s Purpose &amp; Portfolio</h3><p>Well, technically, not really. Goals of a product area are highly inter twined with the company’s vision and mission, but that’s only so much I can discuss before I bore you ;) So let’s start with the toolkit to plan and track an area’s Objectives and Key Results (OKRs).</p><h4>Step 1: Define the OKR Planning Cadence</h4><p>Planning cadence has a huge impact on the burden it puts on managers and directors to plan according to that cadence. It’s not an exact science. Some do it quarterly, others annually, others do something else. I tried quarterly before, but recently started liking the feeling of defining OKRs every semester with a pulse check in the middle of the semester to discuss if we are still heading onto the right direction. In my experience, this provides a nice balance between agility and management burden. It also can give engineering teams the feeling of having well-paced progress for medium-term increments that can be meaningful for users.</p><blockquote>Note: this does not mean you always need to plan due dates for the end of the semester. It just means that, at the beginning of the planning cycle, you plan the entire roadmap for the full semester.</blockquote><h4>Step 2: Define &amp; Prioritise Objectives</h4><p>Directors, involving senior managers, should decide on what will be the objectives for the next 6 months, based on stakeholder needs. Expect to spend around 2 to 4 weeks of the last month of the current semester defining and prioritising the objectives for the next one.</p><p>Different product directors in the area/department will bring different objectives to the table. Some will be more product specific, others more cross-product and therefore demanding collaboration between product teams.</p><p>Prioritisation can become tricky. I like to apply the <a href="https://www.microsoft.com/en-us/microsoft-365-life-hacks/organization/understanding-the-rice-model-and-its-framework">RICE Scoring lenses</a>. It’s simple and mostly effective. Just make sure the area is aligned on the units used in each of the RICE metrics.</p><blockquote>Tip: For defining objectives I like to skip the “E” estimation (effort). I’m just not a big believer of estimations at this point.</blockquote><blockquote>Example: “Enhance the efficiency and reliability of our data pipelines.”</blockquote><h4>Step 3: Define &amp; Prioritise Key Results (KRs)</h4><p>After the objectives are defined, you will define KRs for each objective. That will serve the purpose of make their fulfilment tangible. At this stage, I like to take a step back as a director and open even more space for the managers to drive the definition of KRs. Needless to say, make them <a href="https://en.wikipedia.org/wiki/SMART_criteria">SMART</a>. The RICE scoring model continues to find its place here for prioritisation of KRs.</p><blockquote>Example: <em>“Reduce data processing time by 30% within the next 3 months.”</em></blockquote><h4>Step 4: Setup the Tracking Mechanism</h4><p>As an area is likely composed of multiple product teams, with different backlogs and agile processes, you will need to establish some mechanism to track the delivery progress against the OKRs.</p><ol><li>Create a dynamic portfolio (e.g., JIRA Advanced Roadmaps) with all the OKRs and their concrete owners, target start, end date, etc.</li><li>Create one or more epics in the backlogs of the team(s) involved in each KR. This will allow to understand which teams are responsible for technically delivering that KR.</li><li>Enable that dynamic portfolio for your area that drills-down into the planned outcomes (Objectives -&gt; KRs -&gt; Epics -&gt; User Stories), by properly linking the different objects created in your planning tool (e.g., JIRA). Use <em>target end</em> <em>date</em> to track initial planned delivery date and <em>due date</em> to keep the latest tentative date for delivery if any change of plans occurs. This should be maintained with minimal effort on your agile planning tool of choice (e.g., JIRA Advanced Roadmaps).</li><li>Make that portfolio public to everyone, specially making it noticeable for your direct stakeholders. In the beginning of the semester also prepare a roadmap presentation for your stakeholders. It’s a nice engagement moment with them.</li><li>Meet with the desired periodicity to discuss key status updates and current blockers in your area.</li></ol><p>In a nutshell, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*hiL2YvhPuUjEWoVlfmDE-w.png" /></figure><h4>Step 5: Celebrate &amp; Retrospect</h4><p>With an adequate cadence, conduct area demos for your stakeholders, where you show of the latest and greatest from your products. Moreover, at the end of the semester, when all the work has been done, and hopefully the OKRs achieved, join all the teams and stakeholders for a final product demo looking back at the main features delivered by your area. Leave plenty of space for stakeholder interactions and Q&amp;A.</p><h3>It then continues with Great Team Delivery</h3><p>An area of many products is likely to be composed by many product teams. In this section, we focus on instruments to achieve a great team delivery culture so it can be applied on a per product team basis.</p><p>There are multiple facets to what composes a great team that is set for success. In this post, I’ll leave aside what in my opinion is the undoubtedly important thing for a long-term success team: build true people-driven relationships through empathy, transparency, positivity, honesty, respect, psychological safety, collaboration, among many other human facets. I’ll instead focus on how to organize the team delivery process for a well-paced high impact agile product engineering process.</p><h4>Agile, agile… to be or not to be… and how?</h4><p>This is not an agile lecture. Neither an “agile is dead” rant. There’s plenty of material for us to consume online, I’ll be straight to the point of what I believe it can work for a balanced delivery process. I tried a few things in my professional experience, from pure SCRUM to Kanban. Once again, there’s no silver bullet. And I feel a bit ashamed of reverting to the “the secret is in the middle” suggestion, but I recently started appreciating the Scrumban approach.</p><p>The picture below shows a delivery cycle of 2 weeks, with a proposal of how to structure the Scrumban board and some cerimonies (be honest, who does not start hitching when they hear this fancy word for a meeting :P).</p><blockquote>The secret to agile cerimonies is to have has few has possible, as short as possible.</blockquote><p>The sprint planning is key for OKR status check and next increment understanding and acceptance. Heartbeat (a less recurring concept of daily) is optional, but sometimes useful for unblocking the road for the team. The sprint demo is critical for knowledge sharing amongst the team and for direct product stakeholder feedback. The team retro (notice is not a sprint retro) is key in a team with an open continuous improvement mindset, but adjust the cadence to your team’s needs. I believe a retro each sprint is way too many meetings when you have a culture of open feedback. What’s really important is that you maintain a good retro logbook of topics to act upon to improve team dynamics (e.g., promote team workshops on specific improvement and learning topics; resolve organisational barriers for the team; optimise the planning framework…).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v8Oaxur-vH35acPRtlXS8Q.png" /><figcaption>Proposal of a Scrumban Board and Cerimonies.</figcaption></figure><h4>Make Team Performance a Data-Driven Show</h4><p>Remember I said that I consider the team retro, with the good enough cadence, to be super important for a continuous improvement mindset? Well, so is to have facts to fuel the discussion around continuous improvement towards top-notch performance.</p><p>I see a team delivery dashboard as a way to generate transparency about the “how we are doing in terms of delivery” and to give the team high-level insights that allow us to drill-down into them as needed. This can be very useful to fuel team retro discussions and for you the director, managers and the team to work something out together.</p><p>There’s reasons why your team may not be delivering as many story points or not closing many of the open stories. Use the dashboard as just a fact-based instrument to drive discussions and actions. Amongst those facts look for things like “number of stories closed over time”, “sprints success over time” (e.g., points/stories committed vs delivered or occurrences of scope change). Do not use the dashboard alone to take conclusions about the performance of the team or a team member, as you should not conclude on potential specific problems by just looking at the numbers in it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfDZoMTQYSyw20hjg6_bJA.jpeg" /><figcaption><em>Just an illustrative Dashboard about Team Delivery Performance. This image was generated by AI.</em></figcaption></figure><h3>And ultimately, it’s All About the Players in the Team</h3><p>To keep it simple, for the purpose of this framework only, I’ll narrow down people management to 3 key topics: player’s performance; player’s display of team and company values; player’s growth. Once again, I’ll be leaving aside the human aspect of the relationship between a manager and the players in the team for potential posts in the future, but can only recommend for you to browse through the extensive set of reading materials available in this area. Believe me, it costs me to not delve here on the human-side of being a director/manager, because I believe that without it, nothing works for a sustainable period of time, but for this framework I really wanted to navigate you through the more tangible, less personal, managerial aspects.</p><h4>Player’s Performance</h4><p>Previously, we have defined a framework capable of drilling down delivery items from Objectives -&gt; KRs -&gt; Epics -&gt; User Stories. Players will deliver Epics (collaborating between them most likely), and some of them will be Epic owners also (e.g., component leads or tech leads in the team). Therefore, just maintain a dynamic instrument that allows you to frame the performance discussion on your one-on-one meetings with the respective player. Here’s how it can look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZ7xeiHWjbjcVOb83sO_pA.png" /><figcaption>Player’s Performance Tracking Instrument.</figcaption></figure><h4>Player’s Values</h4><p>On top of performance, a team is only as healthy as the culture that is cultivated in it. Guess who is one of the main responsible (if not the main) for cultivating culture within a product team? Well, I can tell it’s most likely not the high leadership… it’s the players of the team, of course inspired by a great company culture and values. Here’s a crazy idea, what if you track team and company values as if they were skills?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QAMzpK_Rro8sgYvmIbdC9w.png" /><figcaption>Player’s Values Tracking Instrument.</figcaption></figure><h4>Player’s Growth</h4><p>Through time, a player most likely wants to grow. For that, I also like to keep it simple and focus on 2 dimensions of growth:</p><ol><li>The player grows skills expected from their current role and/or grade.</li><li>The player grows skills expected from their desired next role and/or grade.</li></ol><p>For both dimensions, you can use the same instrument to frame conversations and assessments, one for the current expectations and another for the next-level expectations. The skills should be coming from the job architecture framework of your organization (i.e., job roles, job grades, job descriptions…).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4jYCYohfeaMk6ptSPt4BqA.png" /><figcaption>Player’s Growth Tracking Instrument.</figcaption></figure><h3>Wrapping Up</h3><p>In this post, I have proposed a very simplistic framework for portfolio, product and people management in the field of software engineering. I hope you took something out of this framework — if you can even call it that :) — and that somehow it finds its way into your day-to-day managerial activities. Have fun!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e0ae758b9fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>