<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lucas Obadia on Medium]]></title>
        <description><![CDATA[Stories by Lucas Obadia on Medium]]></description>
        <link>https://medium.com/@lucas.simon.qwerty?source=rss-19b3bc305aa3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bAIzuXSmUlCygDvrRt1XXg@2x.jpeg</url>
            <title>Stories by Lucas Obadia on Medium</title>
            <link>https://medium.com/@lucas.simon.qwerty?source=rss-19b3bc305aa3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:43:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lucas.simon.qwerty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Data Platform @ Combo]]></title>
            <link>https://engineering.combohr.com/scaling-data-platform-combo-590d3bfc232b?source=rss-19b3bc305aa3------2</link>
            <guid isPermaLink="false">https://medium.com/p/590d3bfc232b</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Lucas Obadia]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 10:48:10 GMT</pubDate>
            <atom:updated>2026-02-06T10:48:10.755Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*71cUrllj8uJ6yEhugSff5g.png" /><figcaption>Building the invisible backbone of a data platform.</figcaption></figure><p>Data powers our product decisions, guides our marketing choices, and supports our financial analyses. At <a href="https://combohr.com/"><strong>Combo</strong></a>, it is the cornerstone of our mission to simplify HR and payroll.</p><p>Since 2016, the company has grown rapidly. But as we scaled, our data platform didn’t just evolve — it sprawled. By late 2024, we realized that adding more tools wasn’t solving our problems; it was creating them. A “Modern Data Stack” looks great on architecture slides, but daily operations told a different truth.</p><h4>The Situation: The High Cost of Fragmentation</h4><p>On paper, our stack checked all the boxes of a cutting-edge ecosystem. In reality, it was a fragmented collection of overlapping tools rather than a cohesive platform:</p><ul><li><strong>Ingestion Overload:</strong> We were juggling <strong>Airbyte, Fivetran, and Stitch</strong> simultaneously.</li><li><strong>BI Sprawl:</strong> Insights were scattered across Metabase, ThoughtSpot, Looker Studio, and <strong>countless Google Sheets</strong>.</li><li><strong>Disjointed Orchestration:</strong> Critical logic was buried in <strong>dbt</strong> <strong>Cloud</strong> or hard-to-maintain <strong>GitHub Actions</strong>.</li></ul><p><strong>This fragmentation was the root cause of our friction.</strong> Because we had multiple tools doing the same job, we lacked a single source of truth. The technical debt wasn’t just code; it was structural complexity that led directly to:</p><ul><li><strong>Orphaned Datasets:</strong> With no clear ownership, documentation was nonexistent.</li><li><strong>Rising Costs:</strong> Paying for redundant services without governance capabilities.</li><li><strong>Eroded Trust:</strong> Business teams struggled to navigate the maze of sources to find the right metric.</li></ul><p>The breaking point was a simple question from a Product Manager:</p><blockquote>“Why does this number in Looker not match the one in Finance?”</blockquote><p>This wasn’t just a bug. It was a symptom of a siloed architecture where each team defined their own truth using different tools. The lack of cohesion stalled decision-making and drained operational efficiency. We realized that technology alone wouldn’t fix our data — we needed a fundamental shift in mindset.</p><h3>A New Mindset: Data as a Product</h3><p>We adopted a guiding principle: <strong>treat data like a product</strong>.</p><p>That means it must be:</p><ul><li><strong>Reliable</strong> — with automated tests, versioned documentation, and clearly assigned owners.</li><li><strong>Readable</strong> — with clear definitions and easy-to-access lineage.</li><li><strong>Aligned with business needs</strong> — using metrics co-built and validated with stakeholders.</li><li><strong>Accessible</strong> — inspired by <strong>Data Mesh</strong> principles and supported by self-service tools.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*lutx6cBuAknzl030aizxmw.png" /></figure><h3>Every transformation starts with tough decisions.</h3><h4>Regaining Control of the Stack</h4><p>To gain control, flexibility, and cost efficiency, we began a strategic shift:</p><p>Reducing reliance on managed services, and internalizing components like <a href="https://datahub.com/"><strong>DataHub</strong></a> and <a href="https://dagster.io/"><strong>Dagster</strong></a>, deployed on Kubernetes (<a href="https://cloud.google.com/kubernetes-engine?hl=en">GKE</a>).</p><h4>Bringing Data Closer to Engineering</h4><p>The Data team, historically under Operations team with Revops, moved into the Platform &amp; DevOps team.</p><p>This new setup lets us benefit from our <a href="https://combohr.com/">Combo</a> Engineering Principles and also share <strong>Infrastructure as Code</strong>, <strong>CI/CD</strong>, and <strong>observability</strong> practices, while aligning data work with the <a href="https://combohr.com/">Combo</a> Business vision.</p><h4>A Complementary, Cross-Functional Team</h4><p>Our strength comes from the mix of skills.</p><p>Data Engineers, Analytics Engineers, and DevOps work side by side to build a platform that is reliable, relevant, and scalable.</p><p>Together, we cover the full spectrum — from robust pipelines to business-accurate models — while ensuring automation, monitoring, and infrastructure resilience.</p><p>This cross-functional approach helps us design a Data platform that serves business needs and meets engineering standards.</p><h3>Change becomes real when people start to feel it.</h3><h4>Early Wins</h4><ul><li><a href="https://datahub.com/"><strong>DataHub</strong></a><strong> as our control tower:</strong> Deployed on GKE, connected to production Postgres, <a href="https://cloud.google.com/bigquery?hl=en">BigQuery</a>, <a href="https://www.getdbt.com/">dbt</a>, and <a href="https://www.metabase.com/">Metabase</a>, it provides end-to-end traceability and makes previously hidden information visible.</li><li><strong>Major cleanup:</strong> 42% of obsolete datasets removed (515 out of 1,222), a clarified three-layer architecture (staging → intermediate → marts), and close monitoring of usage.</li><li><strong>Better developer experience:</strong> CI tests, mandatory owner reviews, no more SELECT *, and other measures to make deployments more reliable and boost trust in the data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*6uRiWjDCZnBV11gG6hj6yw.png" /></figure><p><strong><em>The best data platforms are built not just with tools, but with lessons.</em></strong></p><h4><strong>From confusion to clarity</strong></h4><p>Remember that question: <em>“Why does this number in Looker not match the one in Finance?”</em></p><p>If a Product Manager asks that today, the answer is no longer a week-long investigation. It’s a 15-second check in <a href="https://datahub.com/"><strong>DataHub</strong></a>.</p><p>Thanks to our new architecture, we now have:</p><ul><li><strong>A single source of truth:</strong> One validated model in our ‘Marts’ layer, used by all company.</li><li><strong>Clear Ownership:</strong> A designated owner who guarantees the logic behind the metric.</li><li><strong>Transparency:</strong> Anyone can see the lineage — from the raw <a href="https://airbyte.com/">Airbyte</a> ingestion to the final <a href="https://www.metabase.com/">Metabase</a> dashboard.</li></ul><p>We haven’t just fixed a technical bug; we’ve fixed a trust issue.</p><h3>What We’ve Learned So Far</h3><ol><li><strong>Done is better than perfect:</strong> Making <a href="https://www.getdbt.com/">dbt</a> documentation visible before <a href="https://datahub.com/">DataHub</a>’s launch had an immediate impact. A useful imperfect version today is better than an unreachable ideal tomorrow.</li><li><strong>Build governance through usage:</strong> Documenting, cleaning, and engaging regularly with teams allows rules to emerge naturally.</li><li><strong>Bridge the gap with the business:</strong> Great dashboards come from great questions, not complex queries. Regular check-ins with Finance, Product, Marketing, and RevOps teams keep us aligned.</li></ol><p><strong><em>Transformation is never a one-time project — it’s a continuous evolution.</em></strong></p><h3>What’s Next</h3><ul><li><strong>Orchestration:</strong> Full deployment of <a href="https://dagster.io/">Dagster</a> to orchestrate our pipelines.</li><li><strong>Infrastructure Isolation:</strong> Creating a dedicated Kubernetes cluster to host our data stack (<a href="https://www.metabase.com/">Metabase</a>, <a href="https://dagster.io/">Dagster</a>, <a href="https://datahub.com/">DataHub</a>).</li><li><strong>Resilience:</strong> Implementing a complete staging environment to test infrastructure changes and secure deployments.</li><li><strong>Delivery:</strong> Adopting an agile release system for predictable iterations.</li></ul><h3>A Collective and Lasting Evolution</h3><p>This transformation goes beyond technology.</p><p>It’s a new way of thinking about data: <strong>closer to engineering, rooted in business needs, and open to the entire company</strong>.</p><p>By structuring our platform, sharing responsibilities, and promoting transparency, we’re creating the foundation for data that is reliable, accessible, and valuable to everyone.</p><p>The journey continues — this is just the beginning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590d3bfc232b" width="1" height="1" alt=""><hr><p><a href="https://engineering.combohr.com/scaling-data-platform-combo-590d3bfc232b">Scaling Data Platform @ Combo</a> was originally published in <a href="https://engineering.combohr.com/">Combo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>