<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Corey Cheung on Medium]]></title>
        <description><![CDATA[Stories by Corey Cheung on Medium]]></description>
        <link>https://medium.com/@coreycheung?source=rss-284cbec8ac38------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nMsTaiDpbvQx5Sv0kSKoPg.jpeg</url>
            <title>Stories by Corey Cheung on Medium</title>
            <link>https://medium.com/@coreycheung?source=rss-284cbec8ac38------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:47:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@coreycheung/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We built a data lakehouse to help dogs live longer]]></title>
            <link>https://medium.com/@coreycheung/we-built-a-data-lakehouse-to-help-sell-dog-food-a94f6ea9c648?source=rss-284cbec8ac38------2</link>
            <guid isPermaLink="false">https://medium.com/p/a94f6ea9c648</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Corey Cheung]]></dc:creator>
            <pubDate>Mon, 17 Mar 2025 05:03:20 GMT</pubDate>
            <atom:updated>2025-03-26T08:39:03.683Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here’s what the data platform looks like at a Pet Food start-up</h4><blockquote>By <a href="https://www.linkedin.com/in/blake-im/">Blake Im</a> and <a href="https://www.linkedin.com/in/coreycheung/">Corey Cheung</a>.</blockquote><h3>Table of Contents</h3><p>· <a href="#86ef">Introduction</a><br>· <a href="#1513">The Old World</a><br>· <a href="#bf68">The New World</a><br>· <a href="#dcbf">Migration</a><br> ∘ <a href="#767f">Storage layer</a><br> ∘ <a href="#d960">Processing layer</a><br> ∘ <a href="#ad5e">Usage Layer &amp; Change management</a><br>· <a href="#e4b9">Conclusion</a></p><h3>Introduction</h3><p>In Q4 of 2024, the data analytics team at Lyka migrated from a Google BigQuery data warehouse to a lakehouse architecture hosted in AWS. <strong>Our new stack features Iceberg tables stored in S3, Glue as the catalog, dbt for transformation, Snowflake &amp; Athena as the query engines, and Airflow for orchestration.</strong> This article will highlight how we migrated ~400 dbt models and ~90 Omni dashboards while implementing all these new technologies.</p><p><em>But first some context…</em></p><p><a href="https://lyka.com.au/">Lyka</a> is a D2C e-commerce brand selling human-grade dog food. We manufacture in our Sydney and Melbourne facilities and deliver right to your doorstep. The Analytics team at Lyka aims to establish a data-driven culture by embedding ourselves in business domains. This allows us to build data products, deliver proactive insights and manage key metrics effectively. To do this well, the data platform is the heart of our operations.</p><h3><strong>The Old World</strong></h3><p>Our previous data platform was a fairly standard implementation of the ‘<a href="https://www.getdbt.com/blog/future-of-the-modern-data-stack">modern data stack</a>’ — an ELT pattern that has been battle-tested by many data teams in the past decade or so. We used tools like <strong><em>Fivetran</em></strong> and <strong><em>Segment</em></strong> to ingest data into our <strong><em>BigQuery</em></strong> warehouse, centralising disparate data sources. Next, <strong><em>dbt</em></strong> would transform the raw data into our ‘exposure’ layer (think gold in medallion architecture). From there <strong><em>Omni</em></strong> and <strong><em>Hightouch</em></strong><em> </em>will leverage this data for BI reporting and reverse ETL respectively — standard stuff.</p><p>Our old data use cases mostly involved running daily batch jobs for reporting, data activation in HubSpot and CustomerIO, and ad hoc exploratory data analysis. This stack served us perfectly well for these straightforward processes — it will always have a special place in our heart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5IhFIxGrohadT_L34XNJw.jpeg" /><figcaption>Lyka’s (old) modern data stack</figcaption></figure><h3><strong>The New World</strong></h3><p>So why <em>did </em>we build a data lakehouse?</p><p>Firstly, we made the important decision to consolidate our cloud providers. Since our engineering team operates on AWS, and the data team on GCP, aligning our tech stack unlocks closer collaboration. Additionally, our internal initiatives to push Manufacturing &amp; Supply Chain operations towards Industry 4.0 require handling low latency, semi-structured IoT data. Enhancements to our Martech stack further demand low latency data enrichment in downstream tools. While we believe a lakehouse architecture can address our diverse requirements, I am sure it’s not the only option — after all there are many ways to skin a data stack.</p><p>A lakehouse combines features of a data lake and a data warehouse into a unified platform — <em>thanks, Sherlock</em>. In this architecture, we can store raw structured, semi-structured and unstructured data in our own managed storage, like a data lake. With our processing engine of choice, we can leverage robust query performance and ACID transactions for analytical workloads, like a data warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*rxds0t9l-uvbJKdBQQorNA.png" /><figcaption>Photo from Databricks</figcaption></figure><p><strong>Our new data lakehouse</strong></p><p>Let’s outline the parts of the platform that are staying the same. Our ingestion and rETL tools remain unchanged — however, we are soon looking to replace Fivetran with a low-latency ingestion option for our transactional databases. Omni and dbt core will still handle BI and transformation.</p><p>So what’s different? Our new lakehouse utilises <strong><em>S3</em></strong> as the data lake, with data stored as Parquet files in the <strong><em>Iceberg</em></strong> table format. <strong><em>Snowflake</em></strong> is the primary query engine, leveraging <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg">external iceberg tables</a> to read from S3 directly. In scenarios where this gets expensive, <strong><em>Athena</em></strong><em> </em>also queries the data lake directly (currently for our lower latency workflows). Everything is orchestrated using <strong><em>Airflow</em></strong> on MWAA while <strong><em>Terraform</em></strong> manages Snowflake infrastructure.</p><p>Let’s dive into how we migrated everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uE-IQPaQs7P0hVpXr_PnDQ.jpeg" /><figcaption>Lyka’s new lakehouse (yes its name is Jeff)</figcaption></figure><h3><strong>Migration</strong></h3><h4><strong>Storage layer</strong></h4><p>To align our cloud provider with our engineering team, AWS was our new far-away-computers of choice, and storing our data in S3 was a no-brainer. Since Fivetran continues to be our primary ingestion tool, migrating our data sources was mostly about chasing credentials from around the business. Once we finished the game of cat and mouse, we were ready to move some data.</p><p><strong>File formats, table formats, and catalogs</strong></p><p>There are inherent limitations when data is stored in parquet files without a table format. For example, parquet files are immutable, meaning they do not support <em>update</em> or <em>delete</em> operations out of the box. Additionally, there is no native way to index or partition the data, which may lead to slow queries. Athena external tables built on top of Parquet files can address some of these issues by enabling partitioning and schema management, however, they don’t solve all the problems. There has to be a better way.</p><p>If you work in the data space, you’ve heard the words <a href="https://iceberg.apache.org/"><em>Apache</em> <em>Iceberg</em></a> shoved in your ear holes. Iceberg is an open-source table format, which is just a fancy way to say it turns your parquet files into a table by adding a metadata layer. This metadata is then governed and organised by a data catalogue (we use the <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/serverless-etl-aws-glue/aws-glue-data-catalog.html">Glue Catalog</a>). A few key Iceberg benefits are:</p><ul><li>Schema evolution — a mutable data structure means we can add, drop and rename columns.</li><li>ACID transactions — provide atomicity, consistency, isolation, and durability (essentially it ensures data integrity).</li><li>Partitioning and time travel — improve query performance and the ability to query your data as it existed at a specific point-in-time.</li><li>Scalability — it can handle petabytes of data efficiently.</li></ul><p>You might be thinking, <em>I get all of these things with Snowflake, BigQuery and Redshift, </em>and you would be correct. But perhaps the most important thing about Iceberg, and why you might want a lakehouse, is that it’s open-source and cross-engine compatible. This means you’re not locked into a vendor <strong><em>and</em></strong> you can bring your favourite query engine to process the data.</p><p>In saying this, Snowflake is our primary query engine for batch analytics workloads. For Snowflake to query our Iceberg tables, we first configure an <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-s3-compatible#create-an-external-volume-for-s3-compatible-storage">external volume</a> so it can talk to AWS. After that, we simply create the iceberg table in DDL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*upkPit6FpBmIdtJFXkzK8A.png" /></figure><p>Once this is done, we’re ready to query the external Iceberg table as if it were a native table in Snowflake (except it’s in our storage, and we can use any query engine we want)! In cases where we want lower latency and it’s expensive to keep a Snowflake warehouse always running, we use Athena as the query engine.</p><h4><strong>Processing layer</strong></h4><p>Let’s move down the DAG and talk about data transformation. With dbt, our transformation logic is somewhat platform agnostic, living independently in Github. This greatly simplifies the migration process — as long as the source tables have the same schema, the dbt project <em>should work</em> seamlessly across different vendors … as long as the SQL syntax is correct. In theory, the big task left was converting the code in dbt from BigQuery to Snowflake SQL. By this time, we already had over 400 dbt models (SQL files), pushing us to search for an automated way to do this.</p><p><strong>Using Claude 3.5 Sonnet to translate SQL</strong></p><p>To transpile all our dbt models, we tested both SQLGlot and prompting various LLMs. We found the most success with Claude 3.5 Sonnet and wrote a Python script to programmatically transpile our dbt models — this mostly worked well. But like many LLM use cases, it was right <strong><em>almost</em></strong> all the time, not quite <strong><em>all</em></strong> the time. The token limitations meant that bigger dbt models had to be broken up into chunks of 200–300 lines to be processed, which confused the model.</p><p>This was the point where we made the call that further efforts to automate would have diminishing returns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*zgW4vuIcWd-TOTru" /></figure><p>Given that this was a once-off migration with high stakes and there were <em>“</em><strong><em>only</em></strong><em>”</em> ~400 dbt models (grim but feasible to do manually), we used the script to assist us in the migration, not fully automate it. This meant we manually checked every output of the translation script, ensuring it produced reasonable results (more on testing later). During this period, we enforced dbt code freezes for the data team.</p><p><strong>Adjusting to different cost models and processing methods</strong></p><p>An unexpected challenge was the difference between how BigQuery and Snowflake scale for larger queries. In our BQ pricing model, the compute scales dynamically to meet the demand of the query, while we pay for the bytes processed. In Snowflake, you get more granular control over how much processing power to allocate to each query with its concept of compute warehouses.</p><p>When we first naively put everything on the smallest compute cluster, some of our bigger models would time out. We refactored some obvious low-hanging inefficiencies and introduced different cluster sizes for the bigger models, using a dbt pre-hook. Although the fixes were simple, the cost difference is worth understanding and considering up-front.</p><p><strong>Testing the dbt output</strong></p><p>To ensure the correctness of the dbt code translation, we combined automated and manual testing. We found that no single approach was enough to cover all grounds, but a combination of different testing methods gave us a good level of confidence:</p><ul><li>dbt tests: Primary keys, business logic, referential integrity tests already existed in our project. These caught the most obvious issues. While we hadn’t implemented the new <a href="https://docs.getdbt.com/docs/build/unit-tests">dbt unit tests</a>, they would have been useful here.</li><li>Row level tests: Matching on primary keys and testing row level diffs. We wrote a Python script to do this, but faced some challenges. Inconsistent refresh times meant data never matched 100%, lack of PK tagging meant manual inputs were required, and inconsistent data types between Snowflake and BigQuery gave false negatives. With these limitations, this served more as a high-level sense check.</li><li>Aggregate tests: We created an exhaustive list of the most important business-critical metrics and ensured the correct output (e.g. Monthly Active Customers for the past 24 months).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*7jgiRCLJepk-_v2uBHWNiQ.png" /></figure><p>In terms of key learnings, investing in the new dbt unit tests for important models and maintaining consistent primary key tagging and row-level versioning would have made the testing process more robust and automated.</p><h4><strong>Usage Layer &amp; Change management</strong></h4><p>The final step was exposing our migrated data to all the tools and humans that use data at Lyka — we began with our BI tool, <a href="https://omni.co/">Omni</a>. This was the true test of the migration, as it’s how most stakeholders interface with our data platform.</p><p>Migrating dashboards is a time-consuming endeavour, but it also presented an opportunity to clean up redundant reports that we stopped maintaining. We started by focusing only on core, company-wide dashboards, then collaborated with various business domains to identify other critical ones. Lastly, we culled unused dashboards, significantly reducing the migration effort, and giving us a clean start with improved governance.</p><p>The technical side of the dashboard migration had its unique challenges as Omni is a relatively new tool with limited migration functionalities. In short, we had to recreate the dashboards and point them to the Snowflake connection, using an internal API that the Omni team kindly exposed. Additionally, we repointed any Python scripts and rETL pipelines to our new data.</p><p><strong>Communicating the changes</strong></p><p>Once the new Snowflake dashboards were tested and ready, we made the leap to archive old BigQuery reports and changeover. From most stakeholders’ perspective, the migration is done. This is where any consequences of migration reach real business operations and financial implications. This was a delicate period where we had to stay very in tune with our stakeholders. Key things we learned were:</p><ul><li>There is no such thing as over-communicating — posting once in a random slack channel doesn’t mean you’ve communicated! Keeping the stakeholders on the same page through the migration process is a huge part of success.</li><li>No matter how well you communicate, there’s nothing like hard-controls to herd people the right way. Remove access to artifacts from the old platform ASAP — you’ll be surprised how many people find ways to keep using the old data.</li><li>Even after all the tests, some things definitely will go wrong. All stakeholders must understand this too and should stay a little skeptical to help raise any issues early.</li></ul><p>We kept both BigQuery and Snowflake instances running for a few weeks to provide redundancy measures for any urgent issues.</p><h3>Conclusion</h3><p>Just like that, we migrated our entire data stack in a quarter. From data storage, orchestration and processing, to testing and communicating the results, there were BIGGG learnings at every stage. The world of data moves so very quickly, and we can’t wait to see how our data platform evolves in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a94f6ea9c648" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>