<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jimmy Pang on Medium]]></title>
        <description><![CDATA[Stories by Jimmy Pang on Medium]]></description>
        <link>https://medium.com/@jimmypang?source=rss-b47cdbdc70c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*tErpgAz2LFogsY1iQJyWRA.png</url>
            <title>Stories by Jimmy Pang on Medium</title>
            <link>https://medium.com/@jimmypang?source=rss-b47cdbdc70c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:34:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jimmypang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Panda is now Data Biz — Join me on Substack!]]></title>
            <link>https://medium.com/data-panda/data-panda-is-now-data-biz-join-me-on-substack-b24c2d76c41c?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b24c2d76c41c</guid>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 20:04:03 GMT</pubDate>
            <atom:updated>2025-11-25T20:04:03.466Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Panda is now Data Biz — Join me on Substack!</h3><p>Hey there,</p><p>If you’re reading this, you were part of the <strong>Data Panda</strong> community on Medium — and I’m grateful you have been there since then.</p><p><strong>Here’s what’s changing:</strong></p><p>I’ve moved from Medium to Substack and rebranded to <a href="https://open.substack.com/pub/jimmypang/p/from-berlin-to-your-inbox-why-im?r=2i7jb6&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcomeOnShare=false"><strong>Data Biz</strong></a>. Why? Because I want to go deeper, publish more consistently, and build a direct relationship with readers like you — without Medium’s algorithmic noise getting in the way.</p><p><strong>What this means for you:</strong></p><ul><li><strong>Same insights, better delivery</strong> — You’ll still get data strategy, BI, analytics engineering, and leadership content, but now it lands straight in your inbox</li><li><strong>More depth, less fluff</strong> — Substack lets me write longer, more technical pieces without worrying about paywalls or platform politics</li><li><strong>Direct engagement</strong> — Hit reply anytime. I actually read and respond to emails here</li><li><strong>Business-first focus</strong> — The new name reflects the shift: data that drives measurable business outcomes, not just dashboards and pipelines for their own sake</li></ul><p><strong>What stays the same:</strong></p><ul><li>My voice, my experience (7+ years across Foodpanda, HelloFresh, Vestiaire Collective, and the Berlin data scene)</li><li>Practical, hands-on content grounded in real-world challenges</li><li>Focus on bridging technical excellence with business impact</li></ul><p><strong>What to expect going forward:</strong></p><ul><li>Thought leadership from the trenches — not theory, but tested strategies</li><li>Deep dives into trends reshaping data teams today</li><li>Practical frameworks to elevate your data strategy and BI capabilities</li><li>Case studies connecting technical work to business results</li><li>Content that respects your time and challenges you to think differently</li></ul><p>I’m excited to continue this journey with you on a platform that lets me serve you better. Reply anytime with your challenges, questions, or topics you want me to tackle.</p><p>Welcome to <a href="https://open.substack.com/pub/jimmypang/p/from-berlin-to-your-inbox-why-im?r=2i7jb6&amp;utm_campaign=post&amp;utm_medium=web&amp;showWelcomeOnShare=false"><strong>Data Biz</strong></a>.</p><p>Best regards,<br><strong>Jimmy Pang</strong><br>Data Leader | Data Strategy | BI &amp; Analytics | Data Engineering<br>LinkedIn: <a href="https://www.linkedin.com/in/jimmy-pang-hk603/"><strong>https://www.linkedin.com/in/jimmy-pang-hk603/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b24c2d76c41c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/data-panda-is-now-data-biz-join-me-on-substack-b24c2d76c41c">Data Panda is now Data Biz — Join me on Substack!</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing Your Substack: A Exit  Post for Data Panda Readers]]></title>
            <link>https://medium.com/data-panda/announcing-your-substack-a-exit-post-for-data-panda-readers-c553255cd94c?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c553255cd94c</guid>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 15:21:37 GMT</pubDate>
            <atom:updated>2025-11-25T15:23:55.364Z</atom:updated>
            <content:encoded><![CDATA[<h3>Announcing Your Substack: An Exit Post for Data Panda Readers</h3><h4>Why I’m Moving Data Panda to Substack (And Why You Should Follow)</h4><p>After years of sharing data insights with you on Medium, I’m making a strategic move to <a href="https://jimmypang.substack.com/">Substack</a> — and I want you to come with me. This isn’t just about changing platforms; it’s about building a stronger, more direct connection with you and creating better content.</p><h3>The Substack Advantage: Why This Matters for You</h3><p><strong>Direct Connection, Zero Algorithm</strong></p><p>Medium’s algorithm decides what you see and when. On Substack, when I publish something, it lands directly in your inbox. No middleman, no feed filtering, no missed insights. You’re in control of your reading experience.​</p><p><strong>Better Quality, More Focus</strong></p><p>Substack’s model aligns my incentives with yours. I’m not chasing algorithmic favor or competing in Medium’s content marketplace — I’m focused entirely on delivering value to subscribers who actively choose to read my work. This means deeper analysis, more thoughtful pieces, and content that serves your professional growth as data leaders.​</p><p><strong>Email Ownership and Community</strong></p><p>On Medium, I have 175 followers, but I don’t actually own that relationship. On Substack, our connection is direct and permanent. You’ll also gain access to Substack’s community features like Chat, livestreaming, and real-time discussions — creating opportunities for deeper engagement beyond just reading articles.​</p><p><strong>Multimedia Integration</strong></p><p>Substack now supports video, podcasts, and livestreaming (and with a lot more AI-powered features). This means I can share data concepts through multiple formats — whether it’s live coding sessions, dashboard walkthroughs, or visual explanations of complex BI strategies — all in one place.​</p><h3>What’s Happening to Medium Content?</h3><p>I’ll be gradually migrating all my Data Panda content from Medium to Substack. This won’t happen overnight, but over the coming months, my best articles on data strategy, BI implementation, analytics engineering, and leadership will find their permanent home on Substack.​</p><p>Why the move? Substack allows me to maintain better control over my content, organize it more effectively for long-term reference, and ensure you can always access the full archive in one centralized location. Think of it as consolidating the Data Panda knowledge base into a single, searchable, reader-friendly platform.​</p><p>For existing Medium readers: Subscribe now to ensure you don’t lose access to updated versions and new insights. While some content will remain accessible on Medium temporarily, <a href="https://jimmypang.substack.com/"><strong>Substack</strong></a><strong> will become the definitive source for all Data Panda content</strong>.</p><h3>What You’ll Get on Substack</h3><p>As a data leader myself, I understand what you need: actionable insights on data strategy, BI implementation, analytics engineering best practices, and leadership guidance. My Substack will focus on business-oriented content that helps you navigate the intersection of data and business decisions.​</p><p>Expect:</p><ul><li>Weekly insights on data leadership and strategy</li><li>Deep dives into modern data stack implementations</li><li>Case studies from my work in eCommerce, logistics, and luxury fashion</li><li>Practical frameworks for stakeholder management and team building</li><li>Updated and enhanced versions of popular Medium articles</li><li>Early access to resources and tools I develop</li></ul><h3>Making the Move Is Easy</h3><p>Simply subscribe at <a href="https://jimmypang.substack.com/">Data Biz</a>, and you’ll receive my content directly in your inbox. The newsletter will remain free, with premium deep-dives and exclusive content available for those who want to go deeper.</p><p>Why Now?</p><p>The data landscape is evolving rapidly, and so should how we share knowledge about it. Substack gives me the independence to write what matters most to you — without platform constraints, algorithmic pressures, or content farm dynamics that have increasingly characterized Medium.​</p><p>I’ve spent years building expertise in BI, analytics engineering, and data leadership across companies like Vestiaire Collective, HelloFresh, and Delivery Hero. It’s time to share that knowledge in a way that truly serves the data community — and that means owning the relationship with my readers.</p><h3>Join Me</h3><p>I’ll continue cross-posting select pieces to Medium for discoverability during the transition, but my primary home — and where all content will ultimately live — is now on Substack.</p><p>Subscribe today at <a href="https://jimmypang.substack.com/">Data Biz</a> and let’s build a stronger data leadership community together. You’ll get the same insights, analysis, and practical wisdom you’ve come to expect, delivered directly to you with zero friction — plus access to the complete, continually updated Data Panda archive.</p><p>See you in your inbox,<br>Jimmy Pang<br><em>Data Leader | BI &amp; Analytics Engineering Specialist</em></p><p><em>P.S. Already subscribed? Share this with fellow data leaders who could benefit from joining our community. The more voices we have in the conversation, the richer our collective learning becomes.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c553255cd94c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/announcing-your-substack-a-exit-post-for-data-panda-readers-c553255cd94c">Announcing Your Substack: A Exit  Post for Data Panda Readers</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[L & D — Learning & Development Resources for Data people in 2024]]></title>
            <link>https://medium.com/data-panda/l-d-learning-development-resources-for-data-people-in-2024-d4f1cebf4e57?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4f1cebf4e57</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[leadership-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sun, 11 Feb 2024 21:48:53 GMT</pubDate>
            <atom:updated>2025-12-17T15:48:52.146Z</atom:updated>
            <content:encoded><![CDATA[<h3>L &amp; D — Learning &amp; Development Resources for Data people in 2024</h3><p><em>📌 December 2025 Update</em></p><p><em>I’ve moved my writing to Substack where I publish deeper insights on data leadership, analytics engineering, and business strategy.</em></p><p><a href="https://jimmypang.substack.com/"><em>👉 Read my latest articles on Data Biz</em></a></p><p><em>I’m now covering topics like:</em></p><ul><li><em>Defining P0 SLAs with real value-stream examples from Vestiaire Collective</em></li><li><em>BI-agnostic data architecture design</em></li><li><em>Data team leadership and stakeholder management</em></li><li><em>dbt best practices and modern data marts</em></li><li><em>Prioritization frameworks for data leaders</em></li></ul><p><a href="https://jimmypang.substack.com/"><em>Subscribe for free →</em></a><em> to get weekly insights from building data teams in Berlin’s startup ecosystem.</em></p><h3>Motivation</h3><p>Learning has been the utmost vital topic for everyone work in the data and tech industry. As a data practitioner and a lead myself, I have been consuming and collecting all sort of learning materials here and there over the years.</p><p>At work, the ask of L &amp; D emerges and the team feels the need for staying relevant is getting more and more urgent. Hence, I figure it might be helpful to share this curated list of L &amp; D resources to the boarder data and tech community over the globe as well.</p><h3>Table of Content</h3><ul><li>Type of Learning Style</li><li>Individual Contributor Track</li><li>Manager Track</li><li>Meetups &amp; Conferences</li></ul><h3>Type of Learning Style</h3><p>Before jumping to the L &amp; D resources, it is important to understand one’s preference on Learning Style.</p><p>I have came across a nice categorization of 4 kind of learning styles and tips for them: <strong>VARK.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2y3chU7A0ZUYthyoXRG0bw.png" /><figcaption><a href="https://www.verywellmind.com/vark-learning-styles-2795156">4 VARK learning styles</a></figcaption></figure><p>Before looking into the L &amp; D resources, it is suggested to have a quick review of one’s preferred learning style. It directly impacts the efficiency of learning:</p><ul><li><strong>Visual</strong>: YouTube videos, Udemy courses, etc.</li><li><strong>Read/Write</strong>: Articles on Medium/Substack, documentation from open source tools, etc.</li><li><strong>Auditory</strong>: Podcast (e.g. Spotify)</li><li><strong>Kinesthetic</strong>: New projects, Side projects, DataCamp, Kaggle, etc.</li></ul><p>It is suggested to play to the strength — spend more effort &amp; time on preferred materials &amp; style to maximize the effect.</p><h3>Individual Contributor Track</h3><h4><a href="https://substack.timodechau.com/">Hipster Data Stack</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*GiI45-lejh2lGOnE" /></figure><p>It is a blog about Data &amp; Product Analytics from <a href="https://www.linkedin.com/in/timo-dechau/">Timo Dechau</a>. It has plenty of good quality content about tracking and data modelling. Strongly recommended to anyone in the field of Product Analytics and Analytics Engineering.</p><h4>Seattle Data Guy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lq4xRJ148QmWAD92" /><figcaption>Logo from Seattle Data Guy (<a href="https://www.theseattledataguy.com/">https://www.theseattledataguy.com/</a>)</figcaption></figure><p>Seattle Data Guy is a rather famous data content creator and also a consultancy firm by <a href="https://www.linkedin.com/in/benjaminrogojan/">Benjamin Rogojan</a>.</p><p>Benjamin creates interesting data engineering &amp; consultant content on <a href="https://www.linkedin.com/company/seattle-data-guy/">Linkedin</a>, <a href="https://seattledataguy.substack.com/">Substack</a>, <a href="https://medium.com/coriers">Medium</a>, and <a href="https://www.youtube.com/@SeattleDataGuy">YouTube</a>. Pretty omnipresence in the online space.</p><h4><a href="https://towardsdatascience.com/">Towards Data Science</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*59xp6sUl8BKQJKnaveTbFA.png" /><figcaption>A screenshot of the TDS blog.</figcaption></figure><p>I have been a reader of the TDS blog for years, and it never ceases to amaze with the high quality of data content.</p><p>The TDS blog contain curated Data Science article, along with a couple of good read in data engineering and analytical engineering. It is basically and must-know for anyone work in data.</p><h3><strong>dbt Learn</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_yOp7tv97BGIfFcqdu6Sw.png" /><figcaption><a href="https://www.getdbt.com/dbt-learn">dbt Learn</a> page.</figcaption></figure><p>dbt Labs provides <strong>free of charge</strong> learning content to help anyone begins their journey of analytical engineering with dbt Core and dbt Cloud. One could simple create an account there and start browsing the courses.</p><p>In addition, dbt Labs also offer <a href="https://www.getdbt.com/dbt-certification">Certification Exam</a> like Tableau does:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xrs3Rf2HTitnXiUzyDyIew.png" /></figure><p>Currently, it offers 2 exams: <a href="https://www.getdbt.com/certifications/analytics-engineer-certification-exam">dbt Analytics Engineering Certification Exam</a> and <a href="https://www.getdbt.com/certifications/dbt-cloud-administrator-certification-exam">dbt Cloud Administrator Certification Exam</a>. Both last for 2 hours, 65 questions, and cost US$ 200 (as of Jan 2024). If your company offer a education budget, this might be a good place to spend it.</p><h3>Tableau</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*toXEjmj6CBSDSS3Ub-lKrg.png" /><figcaption>Tableau Learn (<a href="https://www.tableau.com/learn">https://www.tableau.com/learn</a>)</figcaption></figure><p>As part of the Modern Data Stack, Tableau does offer robust learning resources. Highlighted ones are:</p><ol><li><a href="https://www.tableau.com/learn/training">Free Training Videos</a> (including Creator, Explorer, Viewer, and Admin )</li><li><a href="https://www.tableau.com/learn/certification"><strong>Tableau Certification Exams</strong></a></li></ol><p>There are 5 differnt Tableau Certification Exams at the moment:</p><ul><li><a href="https://www.tableau.com/learn/certification/desktop-specialist">Tableau Desktop Specialist</a> (Level: Associate)</li><li><a href="https://www.tableau.com/learn/certification/certified-data-analyst">Tableau Certified Data Analyst</a> (Level: Professional)</li><li><a href="https://www.tableau.com/learn/certification/server-certified-associate">Tableau Server Certified Associate</a> (Level: Professional)</li><li><a href="https://www.tableau.com/learn/certification/certified-consultant">Tableau Certified Consultant</a> (Level: Professional)</li><li><a href="https://www.tableau.com/learn/certification/certified-architect">Tableau Certified Architect</a> (Level: Professional)</li></ul><p>Each of them cost from US$100 or US$250. Some of them also pairs with Tableau sponsered exam prep courses, but last time I checked they costed more than US$1,000 — I personally feel it is more like a cash grab instead of an honest offer. Nevertheless, there are everything for everyone, probably just not for me.</p><h3>DataCamp</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M2K0fs4YoSsFkZr_" /></figure><p><a href="https://www.datacamp.com/">DataCamp</a> is fairly famous for interactive data online courses. It covers wide range of data topics — Data Science, Data Visualization, Data Engineering, and more (e.g. Data Goverance &amp; Data Quality). DataCamp also offers preset tracks for Analysts, Data Engineer, and Data Scientists with selected course to guide learners.</p><p>It would require subscription fee to access the full content of the courses. For individual learners, DataCamp often offer more than 60% off discount from time to time. If you haven’t have subscription yet, it may be a good idea to wait a bit.</p><h3>Kaggle Learn</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*bBzKIBjssky8aTeo" /></figure><p>Kaggle has been a rather famous data science community. They have been known for hosting data science competitions.</p><p>On top of those, they also offer <a href="https://www.kaggle.com/learn">Kaggle Learn</a> for free &amp; interactive courses of basic Python, Intro to Machine Learning &amp; Deep Learning, Feature Engineering etc.</p><h3>DataTalks.Club</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ENmyznjhF-Oc2Imt" /><figcaption>DataTalks.club logo</figcaption></figure><p><a href="https://datatalks.club/">DataTalks.Club</a> is a global online community of data enthusiasts covering general data topics — Data Science, Data Engineering, and Data Leadership as well. It is present in various space: <a href="https://datatalks.club/podcast.html">Podcast</a> (e.g. Spotify), and <a href="https://datatalks.club/slack.html">Slack</a>. It is founded by <a href="https://www.linkedin.com/in/agrigorev/">Alexey Grigorev</a>, and the story behind can also be found on <a href="https://youtu.be/IxTyq96juVE?si=ybRYOTkZ_QUxTQOS">YouTube</a>.</p><p>They also offer good quality free online courses, e.g. <a href="https://github.com/DataTalksClub/machine-learning-zoomcamp">Machine Learning Zoomcamp</a>, <a href="https://github.com/DataTalksClub/data-engineering-zoomcamp">Data Engineering Zoomcamp</a> etc.. They are backed by a vital Slack community with certifcation as well.</p><h3><a href="https://www.youtube.com/@CodeAesthetic">CodeAesthetic</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aW6_w3cdd8I7XdhyTTz7-g.png" /></figure><p><a href="https://www.youtube.com/@CodeAesthetic">CodeAesthetic</a> is a small YouTube channel about having good code behind the applications, along with the code organization and optimization skills. It doesn’t have tons of videos per se, but all of them are fairly high quality. Recommened for Software Engineers, Data Engineers, and Analytics Engineers.</p><h3>Refactoring Guru</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*2q3J2-OKJDv64qf2" /><figcaption>Logo of Refactoring Guru</figcaption></figure><p><a href="https://refactoring.guru/">Refactoring Guru</a> is an Ukrainian website about Refactoring techiques and Design Patterns in software engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnPB8bGyXOTcz-aKOLim2g.png" /></figure><p>It comes with good explanation and code examples for the key concepts, which are worth checking out.</p><h3>Fireship</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*ce_VK8k79n7gGKt413ovQQ.png" /><figcaption>The banner from the YouTube channel: <a href="https://www.youtube.com/@Fireship">https://www.youtube.com/@Fireship</a></figcaption></figure><p>Strictly speaking, Fireship isn’t exactly a typical L&amp;D resource — but more like a light-hearted way to stay in touch with the latest trend in tech &amp; data with memes.</p><h3>Manager Track</h3><p>The followings give all the relevant resources I refer to as data lead learning resources. Compared to Indivudal Contributor track, they are more high-level and “abstract” to some people — yet they are essential.</p><p>They range from podcasts, books, pulication etc. and would evolve with time. Some of them are more futureproof and some are for staying close with the latest trend, and I see them being complmentary for my learning needs.</p><h4>The Data Chief</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYfCD2OBDer2W0nGMHzESA.png" /><figcaption>The Data Cheif website from Thoughtspot</figcaption></figure><p><a href="https://www.thoughtspot.com/data-chief/podcast">The Data Chief</a> is a podcast from Thoughspot, which acquired Mode previsously. Every episode, the host would invite a data leader in the industry for an interview. It is very valuable for data leads to stay on top of the trends.</p><h4>Forbes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rw1XW_8j5raZsLli" /></figure><p>Forbes is a well-known media covering business, inventment, technology, entrepreneurship, leadership, and lifestyle.</p><p>My personal favorite is their <a href="https://www.forbes.com/cio-network/">CIO Network</a>. Plenty of curated content for CIO (Cheif Information Officer) are avialable there, it gives a good flavour of how data leadership are being perceived in the business world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZgX7Fzdge8-owHq4kV4Krw.png" /><figcaption>The CIO Network in Forbes</figcaption></figure><h4>Good to Great: Why Some Companies Make the Leap… and Others Don’t</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*0_f0BlseISpLth4p" /></figure><p>The book <a href="https://www.goodreads.com/book/show/76865.Good_to_Great">Good to Great</a> is basically the business management bible, a must-read for business executives.</p><p>It introduces plenty of useful techiques and concepts, the Flywheel and the Doom Loop, <a href="https://www.jimcollins.com/media_topics/ThinkingAboutYourEconomic.html">Economic Denominator</a>, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*4TgexQ-KrMi0ndwJ" /><figcaption>The flywheel from Good to Great</figcaption></figure><p>My personal favourate is the concept of Economic Denominator. In the context of data analytics, <strong>it could easily be tranlsated into North Star KPIs (and hence develop into KPI tree)</strong>. In data/analytical engineering, it could also be translated into Entities if Dimensional Modelling (aka. Kimball Model) is employed in the organiztion.</p><p>As a side note, this topic might worth a delicate article for it — I could write an article about that if this topic proves popluar.</p><h4>Principles by Ray Dalio</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2_GepMWv9w_SlEzN" /><figcaption>The book of “Principles” by Ray Dalio.</figcaption></figure><p><a href="https://www.principles.com/">Principles</a> by Ray Dalio has been rather famous among the executives and investment world for a couple of years already. The Principles do not solely exist just as <a href="https://www.goodreads.com/book/show/34536488-principles">books</a>, Ray Dalio also share the key summary generously on the offical website — <a href="https://www.principles.com/principles/418eaeca-564f-41e8-bd90-8e0d042f28d5/#table-of-contents">Principles on the Web</a>.</p><p>Ray also build a whole suites around the Principles. He made Principles in Action available as Mobile apps (both <a href="https://apps.apple.com/us/app/principles-in-action/id1211294305">iOS</a> &amp; <a href="https://play.google.com/store/apps/details?id=com.principles.pia">Android</a>). <a href="https://principlesyou.com/">PrinciplesYou</a> for individual career aptitude test assessment, <a href="https://principlesus.com/">PrinciplesUS</a> for corprate.</p><p>In addition, Ray also has a <a href="https://www.youtube.com/@principlesbyraydalio">YouTube channel</a> for Principles and other related topics. There are so many valuable content, such as <a href="https://youtube.com/playlist?list=PLykIL_1_MFWkWDDgvdZ6L7rsvKCKl-39j&amp;si=If5npEqSmM1-RgFM">Principles For Success in 8 Episodes</a> to summarize the key points in 8 videos, <a href="https://youtu.be/PHe0bXAIuk0?si=4fdvCfJZdZuIXvyB">How The Economic Machine Works</a> to briefly explain how global Ecnomy in documentary style, <a href="https://youtu.be/xguam0TKMw8?si=3JdsozmJhqL9gpiY">Principles for Dealing with the Changing World Order</a> for dealing with international and intranational political dynamics. All these are invaluable and yet free-of-charge resources for dealling with career and investment.</p><h4>The Art of War (孫子兵法)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aRkNdXID7CVhdwAL" /></figure><p>The Art of War (written as “孫子兵法” in Traditional Chinese) has been a all-time classic from ancient China. It was authored by Sun Tzu (孫子 or 孫武), a legnedary milltary strategist. The book has 13 chatpers, topics range from diplomacy, siege tactic, legion formation, marching, landscape, and espionage operations. The book has been known to have plenty implication in business management.</p><p>Here comes my favorite quote from the book:</p><blockquote><em>夫未戰而廟算勝者，得算多也；未戰而廟算不勝者，得算少也。</em><strong><em>多算勝，少算不勝</em></strong><em>，而況於無算乎？吾以此觀之，勝負見矣。」 — </em><strong>孫武《孫子兵法》始計篇</strong></blockquote><blockquote>“Now the general who wins a battle makes many calculations in his temple where the battle is fought. The general who loses a battle makes but few calculations beforehand. <strong>Thus do many calculations lead to victory, and few calculations to defeat</strong>: how much more no calculation at all! It is by attention to this point that I can foresee who is likely to win or lose.” — Sun “<a href="https://en.wikipedia.org/wiki/The_Art_of_War">The Art of War</a>”/Section I — Laying Plans</blockquote><p>Let it be the reminder for why we do data.</p><h4>Radical Candor: Be a Kickass Boss Without Losing Your Humanity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*moBek8Nu7JDuJfV3" /></figure><p>The <a href="https://www.goodreads.com/book/show/29939161-radical-candor">Radical Candor</a> is a book from Kim Scott about the very principle of being — well, a kickass boss without losing your humanity. <a href="https://www.linkedin.com/in/kimm4/">Kim Scott</a> has been leader in Apple and in Google leading AdSense, YouTube, and DoubleClick teams.</p><p>The book covers the following:</p><ul><li>Management Philosophy — building radically candid relationships, guidance, motivation per individuals, how to drive results collaboratively</li><li>Tools &amp; Techiques — Relationship with direct reports, Guidance with Praise &amp; Criticism, techniques for avoiding boredom &amp; burnout, getting results faster</li></ul><p>It has been a great lighthouse book for my leadership style as well. I would strongly recommend this to any other leaders and executives.</p><h4>How Google Works</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/0*BYIE0rT8tozfTrUF" /></figure><p><a href="https://www.goodreads.com/book/show/23158207-how-google-works">How Google works</a> is a good book by Eric Schmidt and Jonathan Rosenberg. Both Eric and Jonathan have been senior executive in Google and there are a couple of popluar management techques are popluarized, e.g. OKR (Objectives &amp; Key Results), Rules of delegation.</p><h4>Harvard Business Review</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SZC7xegsThoh9qCE" /></figure><p>The <a href="https://hbr.org/">HBR</a> is a high quality source of insights for business management, strategy, innovation and leadership. It is not specificly for data people per se, but it has fair amount of great content for readers leadership.</p><h4>Udemy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*1Eijs9CCQtOZXRSR" /></figure><p><a href="https://www.udemy.com/">Udemy</a> is also a famous online learning platform offering courses. It is exceptional useful for soft skills like business management, project management, leadership etc. . As for handson and techical topics, it still lacks some interactivity.</p><p>Udemy offers both individual and enterprise subscrption, or each courses can be bought on individual basis so it is still rather flexible.</p><h3>Meetups &amp; Conferences</h3><p>Thanks to COVID, many data and tech conferences are avilalbe online nowadays and usually do not cost anything for online attendance. Listing a couple of these below:</p><h4>Tableau Conferences</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdVL5BIhTX-8jo3jhjGGYA.png" /></figure><p>A annual event organized by Tableau for data visualization. The sentiment to me usually strikes me as many sales pitches in the day 1 and sometimes even many sessions in day 2 and even day 3. In my perspective, it is good enough to just join online and check the agenda ahead, then pick the interesting sessions to join.</p><h4>Coalesce</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*Yqbl_ijO9QKVzpR-hIN0Eg.png" /></figure><p><a href="https://coalesce.getdbt.com/">Coalesce</a> is a annual conference hosted by dbt Lab, usually US based. It also offers free online attendance with plenty of good quality content in Analytical Engineering. I find it fairly educational every single year. Strongly recommend anyone in Analytical Engineering or traditional BI to at least check it out once.</p><h4>PyData</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*pfAPXdxjOL4Oy_Sx" /></figure><p><a href="https://pydata.org/upcoming-events/">PyData</a> is a global community of Python developers and users of open source data tools.</p><h4>Meetup.com</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N7Hsx1Oe6eAQj_wK" /></figure><p>Last but not least, the <a href="https://www.meetup.com/">Meetup.com</a> is always a good place to find local meetups with other tech and data devs! Plenty of startups and tech companies also make use of it to host events to maintain trations.</p><h3>Wrap up</h3><p>There are plenty of Learning &amp; Development resources in data and tech. What listed above are merely a few selected one in the 3 categories:</p><ul><li>Individual Contributor Track</li><li>Manager Track</li><li>Meetups &amp; Conferences</li></ul><p>The <strong>VARK framework </strong>(<strong>V</strong>isual, <strong>A</strong>uditory, <strong>R</strong>ead/Write, <strong>K</strong>inesthetic) was also referred and suggested to the readers to factor in while they are carving their own path of learning.</p><p>With the industry evolves in high velocity, I wouldn’t be surpised that there are someone somewhere building a new platform or new community to foster knowledge sharing.</p><p>Let us all live and learn!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4f1cebf4e57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/l-d-learning-development-resources-for-data-people-in-2024-d4f1cebf4e57">L &amp; D — Learning &amp; Development Resources for Data people in 2024</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt Tests]]></title>
            <link>https://medium.com/data-panda/dbt-tests-813f0aeacac8?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/813f0aeacac8</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 01:53:51 GMT</pubDate>
            <atom:updated>2025-12-17T15:49:18.195Z</atom:updated>
            <content:encoded><![CDATA[<p>A necessary assertion to ensure the reliablity of the building blocks</p><p><em>📌 December 2025 Update</em></p><p><em>I’ve moved my writing to Substack where I publish deeper insights on data leadership, analytics engineering, and business strategy.</em></p><p><a href="https://jimmypang.substack.com/"><em>👉 Read my latest articles on Data Biz</em></a></p><p><em>I’m now covering topics like:</em></p><ul><li><em>Defining P0 SLAs with real value-stream examples from Vestiaire Collective</em></li><li><em>BI-agnostic data architecture design</em></li><li><em>Data team leadership and stakeholder management</em></li><li><em>dbt best practices and modern data marts</em></li><li><em>Prioritization frameworks for data leaders</em></li></ul><p><a href="https://jimmypang.substack.com/"><em>Subscribe for free →</em></a><em> to get weekly insights from building data teams in Berlin’s startup ecosystem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iIV_Uoob4M35vDGA.png" /><figcaption>The dbt logo.</figcaption></figure><h3>Table of Content</h3><ul><li>Overview</li><li>Types of Tests in dbt — Generic Tests &amp; Singular Tests</li><li>Additional Advance Tests</li><li>How to run tests</li><li>Test “severity”</li><li>Test Implementation Tactics</li><li>Appendix</li></ul><h3>Overview</h3><p>From dbt official documentation (<a href="https://docs.getdbt.com/docs/build/tests">Add tests to your DAG | dbt Developer Hub</a> ):</p><blockquote>Tests are assertions you make about your models and other resources in your dbt project (e.g. sources, seeds and snapshots). When you run dbt test, dbt will tell you if each test in your project passes or fails.</blockquote><p>In short, there are 2 types of in dbt tests:</p><ul><li><a href="https://docs.getdbt.com/docs/build/tests#generic-tests">Generic tests</a>, which are set up by macros, and some of them are already available out of the box.</li><li><a href="https://docs.getdbt.com/docs/build/tests#singular-tests">Singular tests</a>, which are .sql scripts that you expect to return zero rows, otherwise it failed.</li></ul><h4>Generic Tests</h4><p>Generic Tests are put in place by calling a macro. They are defined in a .yaml file within the folder where the model is.</p><p>There are <a href="https://docs.getdbt.com/reference/resource-properties/tests#out-of-the-box-tests">4 Out-of-the-box tests</a> provided by dbt:</p><ol><li><a href="https://docs.getdbt.com/reference/resource-properties/tests#not_null">not_null</a> : check a column to validate there are no null values.</li><li><a href="https://docs.getdbt.com/reference/resource-properties/tests#unique">unique</a>: check that values in a column don’t contain duplicates.</li><li><a href="https://docs.getdbt.com/reference/resource-properties/tests#accepted_values">accepted_values</a>: define a list of expected values a column can have.</li><li><a href="https://docs.getdbt.com/reference/resource-properties/tests#relationships">relationships</a>: defines a column on the current table that references another column in another table, i.e. asserting <a href="https://database.guide/what-is-referential-integrity/">Referential Integrity</a>.</li></ol><p>Under the hood, dbt constructs a select query for each test, using the parameterized query from the generic test block. These queries return the rows where your assertion is <em>not</em> true; if the test returns zero rows, your assertion passes.</p><p><a href="https://docs.getdbt.com/reference/resource-properties/tests#not_null"><strong>not_null</strong></a></p><p>Asserting if there is any NULL value under the selected column.</p><p>By default, this test will fail even if just 1 NULL record is found. This behavior can be adjusted in <a href="https://docs.getdbt.com/reference/resource-configs/severity">Severity</a>.</p><p><a href="https://docs.getdbt.com/reference/resource-properties/tests#unique"><strong>unique</strong></a></p><p>Asserting if the column is unique across the whole table.</p><p><a href="https://docs.getdbt.com/reference/resource-properties/tests#accepted_values"><strong>accepted_values</strong></a></p><p>Asserting if the value in the column is within the expected values.</p><pre>version: 2<br><br>models:<br>  - name: orders<br>    columns:<br>      - name: status<br>        tests:<br>          - accepted_values:<br>              values: [&#39;placed&#39;, &#39;shipped&#39;, &#39;completed&#39;, &#39;returned&#39;]<br><br>      - name: status_id<br>        tests:<br>          - accepted_values:<br>              values: [1, 2, 3, 4]<br>              quote: false</pre><p><a href="https://docs.getdbt.com/reference/resource-properties/tests#relationships"><strong>relationships</strong></a></p><p>It asserts the Relationship between the 2 models. The Relationship is based on the <a href="https://www.w3schools.in/dbms/er-model">Entity Relationship (ER) Diagram</a>. A typical example would be the id of the Buyer in the Orders table should always be found in the Customer dimension table.</p><p>The following example tests that every order’s customer_id maps back to a valid customer:</p><pre>version: 2<br><br>models:<br>  - name: orders<br>    columns:<br>      - name: customer_id<br>        tests:<br>          - relationships:<br>              to: ref(&#39;customers&#39;)<br>              field: id</pre><h4>Singular Tests</h4><p><a href="https://docs.getdbt.com/docs/build/tests#singular-tests">Singular Tests</a> are custom sql scripts we can build to test specific values.</p><ul><li>You can think of it as a query from which you expect zero results: if there are returned rows, then the test will fail.</li><li>These are defined in .sql files, typically in your tests directory.</li><li>You can use <a href="https://docs.getdbt.com/docs/build/jinja-macros">Jinja</a> (including ref and source) in the test definition, just like you can when creating models.</li><li>Each .sql file contains one select statement, and it defines one test.</li><li>For reference, you can check the <a href="https://gitlab.vestiairecollective.com/bi-team/dbt/-/blob/dev/tests/">tests</a> folder in the main root of the project. And the folder structure should be mirroring the model folder as well</li></ul><h3>Additional Advance Tests</h3><p>More additional Advance Tests are available after implementing <a href="https://github.com/dbt-labs/dbt-utils">dbt-utils</a> and <a href="https://github.com/elementary-data/elementary">Elementary</a>, more advance Tests are available.</p><h4>dbt-utils</h4><p>The following tests come from the dbt-utils package.</p><ul><li><a href="https://github.com/dbt-labs/dbt-utils#recency-source">recency</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#expression_is_true-source">expression_is_true</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#expression_is_true-source">accepted_range</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#not_constant-source">not_constant</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#not_empty_string-source">not_empty_string</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#not_null_proportion-source">not_null_proportion</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#not_accepted_values-source">not_accepted_values</a></li><li><a href="https://github.com/dbt-labs/dbt-utils#relationships_where-source">relationships_where</a> — The advanced version of the relationship.</li></ul><h4>Elementary — Data Anomaly Detection</h4><p>Elementary also adds extra <a href="https://docs.elementary-data.com/guides/data-anomaly-detection">dbt Tests for anomaly detection</a>, and it comes in 3 different levels:</p><ul><li>Table level monitors</li><li>Column level monitors</li><li>Dimension monitors</li></ul><h3>How to run tests</h3><p>in dbt, tests are run via the dbt tests command (<a href="https://docs.getdbt.com/reference/commands/test">About dbt test command | dbt Developer Hub</a> ). Examples:</p><p>Run tests on one specific model:</p><pre>dbt test --select hourly_tmp_dim_vc_usr_customer</pre><p>Run tests on a model and all its dependencies:</p><pre>dbt test --select hourly_tmp_dim_vc_usr_customer+</pre><p>Run tests on a model and all upstream models:</p><pre>dbt test --models +hourly_fct_vc_usr_client_stats</pre><p>Run tests on a list of models (separated by whitespace only):</p><pre>dbt test --models hourly_fct_vc_usr_client_stats hourly_fct_vc_usr_client_stats hourly_dim_vc_staging_product_staging</pre><p>When it comes to model select, the choices in dbt is actually very flexible — please see <a href="https://docs.getdbt.com/reference/node-selection/methods">Methods | dbt Developer Hub</a> under Node selection section.</p><h3>Test “severity”</h3><p><a href="https://docs.getdbt.com/reference/resource-configs/severity">Configuring test `severity` | dbt Developer Hub</a></p><p>dbt support 2 kinds of severity for failed dbt test: error and warn.</p><p>The relevant configs are:</p><ul><li>severity: error or warn (default: error)</li><li>error_if: conditional expression (default: !=0)</li><li>warn_if: conditional expression (default: !=0)</li></ul><p>For example,</p><pre>version: 2<br><br>models:<br>  - name: large_table<br>    columns:<br>      - name: slightly_unreliable_column<br>        tests:<br>          - unique:<br>              config:<br>                severity: error<br>                error_if: &quot;&gt;1000&quot;<br>                warn_if: &quot;&gt;10&quot;</pre><p>This yml file is saying a couple of things:</p><ol><li>for the test of the column slightly_unreliable_column, make sure there is no duplication (unique test)</li><li>the severity of this test is error (default value), BUT:</li><li>It would start generating warn messages in the logs (CLI) if there are more than 10duplicated records; AND,</li><li>The test will fail if there are more than 1000 duplicated records found, which makes the run of dbt test failed</li></ol><p>Severity could also be configured in various locations in the dbt repo, please see here (<a href="https://docs.getdbt.com/reference/resource-configs/severity">Configuring test `severity` | dbt Developer Hub</a> ) for details.</p><p><strong>Note</strong></p><ul><li>Only use Error for business-critical bases, as data quality issue like duplication of those would have big impact of a lot of tables downstream</li><li>Most issues are minor and shouldn’t break a whole pipeline, so Warning is more than enough</li><li>It would also be nice if there are different layers of Warning to help assessing the impact</li></ul><h3>Test Implementation Tactics</h3><h4>Primary Key</h4><p>Even though Snowflake and the other DWH solutions do not enforce Primary Key, it is still very vital to set the Primary Key to all tables.</p><p>To assert the Primary Key of the table, it is suggested to use both of the following Test:</p><ul><li><a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/4459288/dbt+Test#not_null">not_null</a></li><li><a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/4459288/dbt+Test#unique">unique</a></li></ul><h4>Surrogate Key</h4><p><a href="https://vertabelo.com/blog/what-is-surrogate-key/">What Is a Surrogate Key?</a></p><p>The ingested tables do not always come with a clear key. By working with stakeholders at Product, Tech, and Business, Surrogate Key can be generated with the relevant domain knowledge (perfect example <a href="https://vestiairecollective.atlassian.net/browse/DP-8763?focusedCommentId=623920">here</a>).</p><p>The assertion of Surrogate Key is similar to Primary Key:</p><ul><li><a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/4459288/dbt+Test#not_null">not_null</a></li><li><a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/4459288/dbt+Test#unique">unique</a></li></ul><p>It is also recommended to <a href="https://github.com/dbt-labs/dbt-utils#generate_surrogate_key-source">generate the Surrogate Key by using Macro from dbt-utils</a> for the sake of standardization. BUT note that it is very critical to conduct enough investigation to find out the hidden granularity in the table before the implementation.</p><h4>NULLs sometimes</h4><p>During data development, there is a lot of times that the BI team doesn’t have in-depth domain knowledge yet.</p><p>It is always useful to start with this one for new column:</p><ul><li><a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/4459288/dbt+Test#not_null">not_null</a></li></ul><p>If the not_null test doesn’t pass, we could fall back to make use of <a href="https://docs.getdbt.com/reference/resource-configs/where">where</a> config to make a selective test, or use not_null_proportion. The rule of thumb is always to start very strictly, then fall back step by step.</p><p>Example:</p><pre>version: 2<br><br>models:<br>  - name: large_table<br>    columns:<br>      - name: sometimes_null_column<br>        tests:<br>          - not_null:<br>              config:<br>                where: &quot;category = &#39;category that we know they shouldn&#39;t be NULL&#39;</pre><h4>Dimension columns with categorical values</h4><p>Suggested to use these 2 together:</p><ul><li>not_null</li><li>accepted_value</li></ul><h4>Metrics columns with continuous values</h4><p>Suggested to use:</p><ul><li>not_null</li><li>accepted_range</li></ul><h4>Future TIMESTAMPs</h4><p>In general, future TIMESTAMPs are not a very critical data quality issue.</p><p>Suggested to use:</p><ul><li>not_null</li><li>accepted_range OR expression_is_true</li></ul><p>Note:</p><ul><li>Also while comparing TIMESTAMPs, it is very vital to keep in mind the timezone of different TIMESTAMP values.</li></ul><h4>Referential Integrity issues</h4><p><a href="https://database.guide/what-is-referential-integrity/">What is Referential Integrity?</a></p><p>A typical example of a referential integrity issue is the customer_id in the Sales table cannot be found as customer_id in the Customer table table.</p><p>Suggested to use:</p><ul><li>relationship</li><li>(If the previous one fails, fall back to) relationships_where</li></ul><h3>Appendix</h3><h4>Compile</h4><p>There is an additional check that can be run In dbt: <a href="https://docs.getdbt.com/reference/commands/compile/">Compile</a></p><pre>dbt compile</pre><p>According to dbt, it would:</p><blockquote><em>dbt compile generates executable SQL from source model, test, and analysis files. You can find these compiled SQL files in the target/ directory of your dbt project.</em></blockquote><p>If there are files in the repository that cannot be generated into executable SQL, this test would fail.</p><h4>Further Reads</h4><ul><li><a href="https://medium.com/data-panda/dbt-models-marts-layer-i-e-data-marts-1220abda4471">dbt Models — Marts layer (i.e. Data Marts)</a></li><li><a href="https://medium.com/data-panda/dbt-models-staging-layer-55f0f2ddc5e4">dbt Models — Staging layer</a></li><li><a href="https://medium.com/data-panda/dbt-the-supernova-in-data-analytics-engineering-79d0ceaee450">dbt — An overview of the supernova in data &amp; analytics engineering</a></li><li><a href="https://docs.getdbt.com/docs/build/tests">dbt Lab — Add tests to your DAG</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=813f0aeacac8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/dbt-tests-813f0aeacac8">dbt Tests</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt Models — Marts layer (i.e. Data Marts)]]></title>
            <link>https://medium.com/data-panda/dbt-models-marts-layer-i-e-data-marts-1220abda4471?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1220abda4471</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 01:22:43 GMT</pubDate>
            <atom:updated>2025-12-17T15:32:31.593Z</atom:updated>
            <content:encoded><![CDATA[<h3>dbt Models — Marts layer (i.e. Data Marts)</h3><h4>Business facing entities, ready for reporting, ad-hoc analysis, Machine Learning and Reverse ETLs</h4><p><em>📌 December 2025 Update: This article has been refreshed and republished on my Substack newsletter with updated insights from my work scaling analytics engineering teams at foodpanda and Vestiaire Collective.</em></p><p><a href="https://jimmypang.substack.com/p/understanding-data-marts-in-modern"><em>👉 Read the updated version: Understanding Data Marts in Modern Analytics Engineering</em></a></p><p><em>I’ve moved all my writing to </em><a href="https://jimmypang.substack.com/"><em>Data Biz on Substack</em></a><em>, where I publish weekly insights on data leadership, analytics engineering best practices, and real-world case studies. </em><a href="https://jimmypang.substack.com/"><em>Subscribe free →</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MrGdMhG1eubCbiQK.png" /><figcaption>The dbt logo.</figcaption></figure><h3>Table of Contents</h3><ul><li>What are Mart models in dbt?</li><li>Types of Data Marts — Dimension, Fact, Summary, Snapshots, and Manual tables</li><li>Data Modelling for Marts &amp; Naming Conventions</li><li>What is an Entity, and Entity Relationship (ER)?</li><li>Where is the correct place for Mart models in dbt repo?</li><li>How are the Mart models being run?</li><li>DOs &amp; DON’Ts</li><li>Further considerations</li><li>Appendix</li></ul><h3>What are Mart models in dbt?</h3><p>From <a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/4-marts">the dbt Lab</a>, a very informative description was put here:</p><blockquote><em>This is the layer where everything comes together and we start to arrange all of our atoms (staging models) and molecules (intermediate models) into full-fledged cells that have identity and purpose.</em></blockquote><blockquote><em>We sometimes like to call this the entity layer or concept layer, to emphasize that all our marts are meant to represent a specific entity or concept at its unique grain. For instance, an order, a customer, a territory, a click event, a payment — each of these would be represented with a distinct mart, and each row would represent a discrete instance of these concepts.</em></blockquote><blockquote><em>Unlike in a traditional Kimball star schema though, in modern data warehousing — where storage is cheap and compute is expensive — we’ll happily borrow and add any and all data from other concepts that are relevant to answering questions about the mart’s core entity.</em></blockquote><blockquote><em>Building the same data in multiple places, as we do with </em><em>orders in our </em><em>customers mart example below, is more efficient in this paradigm than repeatedly rejoining these concepts (this is a basic definition of denormalization in this context).</em></blockquote><ul><li>The <a href="https://en.wikipedia.org/wiki/Dimensional_modeling">Kimball model (aka Dimensional Modelling)</a> is more for traditional <a href="https://en.wikipedia.org/wiki/Relational_database#RDBMS">RDBMS(Relational Database Management System)</a>, e.g. MySQL, MariaDB (which is just a fork of MySQL), PostgreSQL, AWS Redshift (i.e. a fork from PostgreSQL 7.0.1) etc.</li><li>Modern<a href="https://en.wikipedia.org/wiki/Online_analytical_processing"> OLAP(Online Analytical Processing Platform)</a>/Data Warehouse solutions like Snowflake and BigQuery are <a href="https://www.mongodb.com/databases/non-relational"><strong>non-relational</strong></a>, hence the best practices and optimization directions are completely different</li></ul><h3>Types of Data Marts</h3><p>Important Note: To maximize the performance &amp; querying from stakeholders side, and also the the user experience of the Data Products, <strong>de-normalized Fact table building should always be prioritized over building Dimension tables</strong>.</p><p>In the other words, it is always a better experience for analysts while they don’t need to do many JOINs at their own queries if the needed columns are always in the Fact table, e.g. the Sales table should already have ISO Week, Year Number, Country Codes, etc. So the analysts don’t need to JOIN with Date &amp; Country dimensions all the time.</p><p>The most common types of Data Marts are Dimension and Fact tables. As time evolves, there are more types of Data Marts being introduced and might makes sense to keep:</p><ol><li>Dimension (dim)</li><li>Fact (fct)</li><li>Summary (sum)</li><li>Snapshots (snp)</li><li>Manual (mnl)</li></ol><h4>Dimension tables</h4><p>From <a href="https://en.wikipedia.org/wiki/Dimension_(data_warehouse)">Wiki</a>:</p><blockquote><em>A </em><strong><em>dimension</em></strong><em> is a structure that categorizes </em><a href="https://en.wikipedia.org/wiki/Fact_(data_warehouse)"><em>facts</em></a><em> and </em><a href="https://en.wikipedia.org/wiki/Measure_(data_warehouse)"><em>measures</em></a><em> in order to enable users to answer business questions. Commonly used dimensions are people, products, place and time.</em><a href="https://en.wikipedia.org/wiki/Dimension_(data_warehouse)#cite_note-1"><em>[1]</em></a><a href="https://en.wikipedia.org/wiki/Dimension_(data_warehouse)#cite_note-2"><em>[2]</em></a><em>.</em></blockquote><p>Good Examples would be:</p><ul><li>dim_countries</li><li>dim_customers</li><li>dim_shops</li><li>dim_locations</li></ul><h4>Fact tables</h4><p>From <a href="https://en.wikipedia.org/wiki/Fact_table">Wiki</a>:</p><blockquote><em>In </em><a href="https://en.wikipedia.org/wiki/Data_warehousing"><em>data warehousing</em></a><em>, a </em><strong><em>fact table</em></strong><em> consists of the measurements, metrics or </em><a href="https://en.wikipedia.org/wiki/Fact_(data_warehouse)"><em>facts</em></a><em> of a </em><a href="https://en.wikipedia.org/wiki/Business_process"><em>business process</em></a><em>. It is located at the center of a </em><a href="https://en.wikipedia.org/wiki/Star_schema"><em>star schema</em></a><em> or a </em><a href="https://en.wikipedia.org/wiki/Snowflake_schema"><em>snowflake schema</em></a><em> surrounded by </em><a href="https://en.wikipedia.org/wiki/Dimension_table"><em>dimension tables</em></a><em>.</em></blockquote><p>Good Examples would be:</p><ul><li>fct_orders</li><li>fct_order_status_transitions</li><li>fct_shipments</li></ul><h4>Summary tables</h4><p>Summary tables are pre-computed &amp; pre-aggregated Data Marts. Summary tables would contain pre-defined dimensions (e.g. Dates, Country, etc.) and agreed KPIs (e.g. number of orders, GMV, etc.)</p><p>Good Examples would be:</p><ul><li>sum_performance_kpi</li><li>sum_customer_metrics</li></ul><p><strong>Notes</strong></p><ul><li>To build a new summary table, it is always a good idea to look at the reporting done previously already</li><li>In a conceptual sense, Summary tables are actually similar to Metrics Layer (aka <a href="https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl">Semantic Layer</a>)— the key benefits are centralized logic and much better query performance during consumption</li></ul><h4>Snapshots tables</h4><p>Snapshots table is a type of Data Mart that would merely capture the state of an existing table, and store the data as a separate table. The best use case of it would be SCD (Slowly Changing Dimensions).</p><p>The <a href="https://docs.getdbt.com/docs/build/snapshots">dbt Snapshot</a> is the recommended way to do so.</p><p>Examples in Vestiarie would be:</p><ul><li>snp_country</li><li>snp_order_status</li></ul><h4>Manual tables</h4><p>Manual table is a type of Data Mart that is manually generated by any means (e.g. 1 off copy of csv file from S3, Google Sheet import, static SQL query, dbt seeds etc.)</p><p>Unless there is a strong business reason (e.g. the manual data is expected to change frequently), <a href="https://docs.getdbt.com/reference/commands/seed">dbt Seeds</a> would be the recommended way for Manual tables.</p><p>Examples in Vestiarie would be:</p><ul><li>mnl_shipment_status_mapping</li></ul><h3>Data Modelling for Marts</h3><p>Different organizations have different way to structure the Data Marts. Some would only have 2 schemas:</p><ul><li>dimension stores all Dimension tables</li><li>fact stores all Fact tables</li></ul><p>Following this pattern, there should also be dedicated schemas for Summary, Snapshot, and Manual tables.</p><p>There is also another way to structure so: by domain, e.g.</p><ul><li>dwh_ops: All data marts related to Operations</li><li>dwh_mkt: All data marts related to Marketing</li><li>dwh_fin: All data marts related to Finance</li></ul><p>There is a catch here though — If there is a personnel change, re-org, or scope changes, the ownership of the tables could become messy if there is no proper Data Catalog solution in place.</p><h4>Data Marts naming conventions</h4><p>For Data Marts, it would be a great idea to follow this naming convention:<br>&lt;dim/fct/sum/mnl&gt;_&lt;BU&gt;_&lt;domain&gt;_&lt;entity&gt;</p><h4><strong>dim/fct/sum/mnl/snp</strong></h4><p>It indicates table type, where:</p><ul><li>dim: Dimension table</li><li>fct: Fact table</li><li>sum: A summary table that only reports aggerated &amp; pre-computed metrics</li><li>mnl: Manual table, usually generated from Google Sheet imports</li><li>snp: Snapshot table</li></ul><h4><strong>BU (i.e. Business Unit)</strong></h4><p>Mostly for organization having multiple brands, and the naming convention of the BUs really varies a lot from organizations to organization. The rule of thumb is maintain consistence</p><h4><strong>Domain</strong></h4><p>Using predefined domain name 3 letter abbreviations, e.g. ops for Operations, fin for Finance etc.</p><h4><strong>Entity</strong></h4><p>The Entity stands for the “Entity” in the ER diagram, details goes as the below.</p><h3>What is an Entity, and Entity Relationship (ER)?</h3><p>According to <a href="https://en.wikipedia.org/wiki/Entity">Wikipedia</a>, an entity is:</p><blockquote>In <a href="https://en.wikipedia.org/wiki/Computer_science">computer science</a>, an entity is an object that has an <a href="https://en.wikipedia.org/wiki/Identifier">identity</a>, which is independent of the changes of its <a href="https://en.wikipedia.org/wiki/Attribute_(computing)">attributes</a>. It represents long-lived information relevant for the users and is usually stored in a <a href="https://en.wikipedia.org/wiki/Database">database</a>.</blockquote><p>Even in the modern days, most of the popular OLAPs (e.g. Snowflake, BigQuery) are no longer relational. Yet, the concept of the <a href="https://www.w3schools.in/dbms/er-model">Entity Relationship (ER) Model</a> is still very vital in terms of Data Marts design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*ENJtbBdNWmM2zT5P" /><figcaption>Entities example from <a href="https://www.w3schools.in/dbms/er-model">W3schools</a></figcaption></figure><p>ER modeling is a top-down structure to database design that begins with identifying the important data called entities and relationships in combination with the data that must be characterized in the model. Then, database model designers can add more details such as the information they want to hold about the entities and relationships, which are the attributes, and any constraints on the entities, relationships, and attributes. ER modeling is an important technique for any database designer to master and forms the basis of the methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*ZZwcvUdmvXQZwMiR" /><figcaption>Entitiy Relationship example from <a href="https://www.w3schools.in/dbms/er-model">W3schools</a></figcaption></figure><p>A relationship type is a set of associations between one or more participating entity types. Each relationship type is given a name that describes its function.</p><h3>Where is the correct place for Mart models in dbt repo?</h3><p>In <a href="https://github.com/dbt-labs/jaffle_shop/tree/main/models">the Jaffle Shop example</a> from dbt Lab, the Mart models are directly stored under the models folders:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*FeFu9DsMq3LlkQRjRAPPOw.png" /></figure><p>It is not a bad idea ot start with. But as time goes, it might becomes hard to manage when the number of Mart models grow. Hence, it may be a better idea to use this structure instead:</p><ul><li>models/marts/{{mart_schema_name}}</li></ul><p>Or, if the idea of Data Mesh or even more complex architure is planned ahead, it may be something like:</p><ul><li>models/marts/core/{{mart_schema_name}}</li><li>models/marts/ops/{{mart_schema_name}}</li><li>models/marts/fin/{{mart_schema_name}}</li></ul><h3>How are the Mart models being run?</h3><p>Much like <a href="https://medium.com/data-panda/dbt-models-staging-layer-55f0f2ddc5e4">another article</a> mentioned, the commend dbt run supports various <a href="https://docs.getdbt.com/reference/node-selection/methods">Node Selection Methods</a>. The recommended way remains as <a href="https://docs.getdbt.com/reference/node-selection/methods#the-tag-method">the “tag” method</a> (e.g. dbt run --select tag:staging).</p><p><a href="https://docs.getdbt.com/reference/node-selection/methods#the-path-method">The “Path” method</a> (e.g. Models in the folder daily_01h00) is also a most commonly used Method (e.g. dbt run — select path:models/staging).</p><p>The reason why “tag” method is better is that it frees the data team from locking all .sql files in certain directories, and allows the team to organize the files in a more logical way.</p><h3>DOs &amp; DON’Ts</h3><p>The followings comes from <a href="https://docs.getdbt.com/best-practices/how-we-structure/4-marts">the dbt Labs</a>:</p><h4>DOs</h4><ul><li>✅ <strong>Group by department or area of concern:</strong> If you have fewer than 10 or so marts you may not have much need for subfolders, so as with the intermediate layer, don’t over-optimize too early. If you do find yourself needing to insert more structure and grouping though, use useful business concepts here.</li><li>✅ <strong>Name by entity.</strong> Use plain English to name the file based on the concept that forms the grain of the mart customers, orders. Note that for pure marts, there should not be a time dimension (orders_per_day) here, which is typically best captured via <a href="https://docs.getdbt.com/docs/build/metrics">metrics</a>.</li><li>✅ <strong>Materialized as tables or incremental models:</strong> Once we reach the marts layer, it’s time to start building not just our logic into the warehouse, but the data itself. This gives end users much faster performance for these later models that are actually designed for their use, and saves us costs recomputing these entire chains of models every time somebody refreshes a dashboard or runs a regression in python. As always, start simple and only add complexity as necessary. The models with the most data and compute-intensive transformations should absolutely take advantage of dbt’s excellent incremental materialization options, but rushing to make all your marts models incremental by default will introduce superfluous difficulty.</li><li>✅ <strong>Wide and denormalized.</strong> Unlike old school warehousing, in the modern data stack storage is cheap and it’s compute that is expensive and must be prioritized as such, packing these into very wide denormalized concepts that can provide everything somebody needs about a concept as a goal.</li><li>✅ <strong>Build on separate marts thoughtfully.</strong> While we strive to preserve a narrowing DAG up to the marts layer, once here things may start to get a little less strict. A common example is passing information between marts at different grains, as we saw above, where we bring our orders mart into our customers marts to aggregate critical order data into a customer grain. Now that we’re really ‘spending’ compute and storage by actually building the data in our outputs, it’s sensible to leverage previously built resources to speed up and save costs on outputs that require similar data, versus recomputing the same views and CTEs from scratch. The right approach here is heavily dependent on your unique DAG, models, and goals — it’s just important to note that using a mart in building another, later mart is okay, but requires careful consideration to avoid wasted resources or circular dependencies.</li></ul><h4>Don’ts</h4><ul><li>❌ <strong>Build the same concept differently for different teams:</strong> finance_orders and marketing_orders is typically considered an anti-pattern. There are, as always, exceptions — a common pattern we see is that, finance may have specific needs, for example reporting revenue to the government in a way that diverges from how the company as a whole measures revenue day-to-day. Just make sure that these are clearly designed and understandable as <em>separate</em> concepts, not departmental views on the same concept: tax_revenue and revenue not finance_revenue and marketing_revenue.</li><li>❌ <strong>Make the Mart model specific to a service, or a segment of a business unit/ country</strong> (e.g. dim_vc_ops_zendesk_users, DIM_VC_PRD_PIM_PRODUCT_COLOR_KR)</li><li>❌ <strong>Too many joins in one table:</strong> more complicated a table is, higher the chance to have issues (e.g. duplications)</li></ul><h3>Further considerations</h3><ul><li><strong>Troubleshoot via tables.</strong> While stacking views and ephemeral models up until our marts — only building data into the warehouse at the end of a chain when we have the models we really want end users to work with — is ideal in production, it can present some difficulties in development. Particularly, certain errors may seem to be surfacing in our later models that actually stem from much earlier dependencies in our model chain (ancestor models in our DAG that are built before the model throwing the errors). If you’re having trouble pinning down where or what a database error is telling you, it can be helpful to temporarily build a specific chain of models as tables so that the warehouse will throw the error where it’s actually occurring.</li><li><strong>After marts: the activation layer.</strong> In the same way that our staging models are building blocks for our marts, that also offer us direct views into specific source data, our marts are building blocks for our final outputs that also offer direct views into specific ideas. You can use marts directly, but they are equally important as components for building models in the <em>activation layer</em> after marts.</li></ul><h3>Appendix</h3><ul><li><a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/4-marts">Marts: Business-defined entities | dbt Developer Hub</a></li><li><a href="https://en.wikipedia.org/wiki/Ralph_Kimball">Ralph Kimball</a> — The father of data warehousing</li><li><a href="https://en.wikipedia.org/wiki/Dimensional_modeling">Dimensional modeling</a></li><li><a href="https://www.w3schools.in/dbms/er-model">Entity Relationship(ER) Model</a></li><li><a href="https://medium.com/data-panda/dbt-tests-813f0aeacac8">dbt Tests</a></li><li><a href="https://medium.com/data-panda/dbt-models-staging-layer-55f0f2ddc5e4">dbt Models — Staging layer</a></li></ul><h4>Data Modelling — Star schema</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sEDV-m86btDk9mIL.png" /><figcaption>Kimball’s Dimensional Data Modelling, from <a href="https://www.holistics.io/books/setup-analytics/kimball-s-dimensional-data-modeling/">holistics.io</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1220abda4471" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/dbt-models-marts-layer-i-e-data-marts-1220abda4471">dbt Models — Marts layer (i.e. Data Marts)</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt Models — Staging layer]]></title>
            <link>https://medium.com/data-panda/dbt-models-staging-layer-55f0f2ddc5e4?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/55f0f2ddc5e4</guid>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 00:22:02 GMT</pubDate>
            <atom:updated>2025-12-17T15:49:43.295Z</atom:updated>
            <content:encoded><![CDATA[<h3>dbt Models — Staging layer</h3><h4>The basic building blocks of a dbt project.</h4><p><em>📌 December 2025 Update</em></p><p><em>I’ve moved my writing to Substack where I publish deeper insights on data leadership, analytics engineering, and business strategy.</em></p><p><a href="https://jimmypang.substack.com/"><em>👉 Read my latest articles on Data Biz</em></a></p><p><em>I’m now covering topics like:</em></p><ul><li><em>Defining P0 SLAs with real value-stream examples from Vestiaire Collective</em></li><li><em>BI-agnostic data architecture design</em></li><li><em>Data team leadership and stakeholder management</em></li><li><em>dbt best practices and modern data marts</em></li><li><em>Prioritization frameworks for data leaders</em></li></ul><p><a href="https://jimmypang.substack.com/"><em>Subscribe for free →</em></a><em> to get weekly insights from building data teams in Berlin’s startup ecosystem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oedfw9RTQey1l8EHmILN-Q.png" /><figcaption>The dbt logo.</figcaption></figure><h3>Table of Content</h3><ul><li>What are Staging models in dbt?</li><li>Where is the correct place for Staging models in dbt?</li><li>How are the staging models being run?</li><li>Staging Models Materialization &amp; Implementation Tactics</li><li>Naming Convention</li><li>DOs &amp; DON’Ts, debatable cases, and good examples</li><li>Appendix</li></ul><h3>What are Staging models in dbt?</h3><p>From dbt Lab (<a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/2-staging">Staging: Preparing our atomic building blocks | dbt Developer Hub</a> ), a very nice description were put here:</p><blockquote><em>The staging layer is where our journey begins. This is the foundation of our project, where we bring all the individual components we’re going to use to build our more complex and useful models into the project.</em></blockquote><p>In dbt Lab’s analogy, Staging models are atomic — the very basic building blocks to build up the whole dbt project. It is also the second layer in the dbt project for ingested data (after ODS, i.e. <a href="https://en.wikipedia.org/wiki/Operational_data_store">Operational Data Store</a>).</p><p>Staging models act as the very first contact point for any Intermediate Model or Marts model. As there are very basic operations or data transformations that have to be done as early in the data transformation pipeline as possible, e.g. translating French columns into English, timezone conversion, etc. If it is done in Intermediate Models or even Mart Models, it is usually too late and would lead to code duplication.</p><p>Another important reason for having Staging models is <strong>having Data Quality Check (i.e. dbt test) as upstream as possible</strong>. Data Quality Checks indeed needs to be implemented to Mart Models, but it would be even better if the checks are directly implemented in the Staging models as well. If that is put in place, <strong>the AE team could immediately catch data quality issues coming from upstream</strong> <strong>with high confidence to know which ingested table failed</strong>.</p><p>It is also worth to note that the different teams have multiple understanding regarding “staging” previously, e.g.</p><ol><li>Staging<strong> schema in the Data Warehouse </strong>; OR,</li><li>The <strong>“Staging area”</strong> of data ingestion, i.e. the first place in Snowflake. This should be named as <a href="https://vestiairecollective.atlassian.net/l/cp/ctu0Mjh5">ODS</a> instead; OR,</li><li>“<strong>Staging Enviorment</strong>”, as against “Production Environment” and ‘Pre-production Environment”.</li></ol><p>And in this article, “Staging Models” is something different from any of above and it is strictly only referring to the first layer of dbt models depending on dbt Sources.</p><h3>Where is the correct place for Staging models in dbt?</h3><p>A very neat folder structure would be all proper staging models can be found in the path models/staging (dbt Lab Official example <a href="https://github.com/dbt-labs/jaffle_shop/tree/main/models/staging">here</a>)</p><ul><li>1 folder for 1 application</li><li>1 source yml file per application (e.g. Assuming primary Database being in PostgreSQL, src_postgre.yml for all tables listed out in the ODS s_postgresql ), please here (<a href="https://vestiairecollective.atlassian.net/wiki/spaces/DATA/pages/40894840">dbt Sources</a> ) for details</li><li>All files staging Models could be found under this folder</li></ul><p><strong>IMPORTANT NOTE</strong></p><p>Ideally speaking, there should be a delicate schema in the Data Warehouse solely for storing tables or views from staging Models. Anything else should be moved away (e.g. Intermedate Models should be using the schema intermediate instead, Google Sheet import should be landing in the schema s_gsheet instead.)</p><h3>How are the staging models being run?</h3><p>For the commend dbt run, it actually supports various Node Selection Methods (<a href="https://docs.getdbt.com/reference/node-selection/methods">Methods | dbt Developer Hub</a> ). The recommended way is actually <a href="https://docs.getdbt.com/reference/node-selection/methods#the-tag-method">the “tag” method</a> (e.g. dbt run --select tag:staging). Another commonly used method is <a href="https://docs.getdbt.com/reference/node-selection/methods#the-path-method">the “Path” method</a> (e.g. Models in the folder staging, goes as dbt run — select path:models/staging ).</p><p>The reason why the “tag” method is better is that it frees the data team from locking all .sql files in certain directories, and allows the team to organize the files in a more logical way.</p><h3>Staging Models Materialization</h3><p>There are 3 suggested <a href="https://docs.getdbt.com/docs/build/materializations">Materializations</a> for Staging models:</p><ol><li><a href="https://docs.getdbt.com/docs/build/materializations#view">View</a></li><li><a href="https://docs.getdbt.com/docs/build/materializations#table">Table</a></li><li><a href="https://docs.getdbt.com/docs/build/materializations#incremental">Incremental</a></li></ol><h4>Implementation Tactics</h4><p><strong>When to use View materialization?</strong></p><ul><li>If the usage of the Staging Model is not 100% clear yet, it is better to use View as default</li></ul><p><strong>When to use Table materialization?</strong></p><ul><li>If it is known for sure that the Staging Model will have many downstream dependency (e.g. 3–4+), Table materialization will be a good choice as it avoid the data warehouse from materializing the same table twice or even more times</li></ul><p><strong>When to use Incremental materialization?</strong></p><p>When the table in ODS has:</p><ul><li>Very clear Primary Key (e.g. id from order table); <strong>AND</strong>,</li><li>A timestamp indicating when this row of the record is being updated (e.g. updatedat timestamp etc.)</li></ul><p>Both of the above 2 conditions have to be met to use Incremental instead of Table materialization, or else the Incremental models could have weird behavior (e.g. duplication)</p><ul><li>(Advance Technique) The only exception for this case would be any event-based data source (e.g. Kafka, Snowplow/Google Analytics tracking events) as the behavior of those data is inserted only — as long as there is a reliable updatedat column, the table could still be using Incremental materialization even without a Primary Key</li></ul><h3>Naming Convention</h3><p>For Staging Models, the file name should always follow this pattern:</p><p>staging__stg_{{application_name}}_{{table_name}}</p><p>Examples:</p><ul><li>staging__stg_postgresql_customers</li><li>staging__stg_postgresql_orders</li><li>staging__stg_postgresql_products</li></ul><h3>DOs &amp; DON’Ts</h3><p>From dbt Lab, there are a couple of Dos &amp; Don’ts in Staging models</p><h4>Dos</h4><ul><li>✅ <strong>Renaming</strong>: e.g. Translating French column name to English, adding prefix datetime_ to TIMESTAMP columns, etc.</li><li>✅ <strong>Type casting</strong>: e.g. casting id_product (STRING) to id_product (INTEGER)</li><li>✅ <strong>Basic computations</strong>: e.g. cents to dollars, gram to kilogram</li><li>✅ <strong>Categorizing</strong>: using conditional logic to group values into buckets or booleans, such as in the CASE WHEN statements above</li><li>✅ <strong>Fetching fields from </strong><a href="https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html#semi-structured-data-types"><strong>VARIANT</strong></a><strong>, </strong>e.g. for Kafka topics, Zendesk API, etc.</li></ul><h3>Don’ts</h3><ul><li>❌ <strong>Joins</strong> — <em>the goal of staging models is to clean and prepare individual source conformed concepts for downstream usage. We’re creating the most useful version of a source system table, which we can use as a new modular component for our project. In our experience, joins are almost always a bad idea here — they create immediate duplicated computation and confusing relationships that ripple downstream — there are occasionally exceptions though (see </em><a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/2-staging#staging-other-considerations"><em>base models</em></a><em> below).</em></li><li>❌ <strong>Aggregations</strong> — aggregations entail grouping, and we’re not doing that at this stage. Remember — staging models are your place to create the building blocks you’ll use all throughout the rest of your project — if we start changing the grain of our tables by grouping in this layer, we’ll lose access to source data that we’ll likely need at some point. We just want to get our individual concepts cleaned and ready for use, and will handle aggregating values downstream.</li></ul><h3>Debatable cases, which need to treated case by case</h3><p>Flattening (e.g. JSON) data — It is up to debate if the Flattening of nested data should be done in Staging Model or Intermediate Model, as arguments at both sides are valid:</p><ul><li>Pro Flattening in Staging Models: Most of the use cases of Zendesk tickets, for example, are accessing the ARRAYs inside. As all downstream intermediate Models would need to flatten it anyway, might as well do it in Staging Model</li><li>Pro Flattening in Intermediate Models: Flattening in Staging model would directly destroy the Primary Key of the ODS table (i.e. the id column would no longer in unique after flattening), it would generate extra difficulty to spot data quality issue from ODS right away</li></ul><h3>Good Examples</h3><h3>stg_customers</h3><p><a href="https://github.com/dbt-labs/jaffle_shop/blob/main/models/staging/stg_customers.sql">https://github.com/dbt-labs/jaffle_shop/blob/main/models/staging/stg_customers.sql</a></p><p>This staging model was from the dbt Labs Jaffle Shop repo:</p><pre>with source as (<br><br>    {#-<br>    Normally we would select from the table here, but we are using seeds to load<br>    our data in this project<br>    #}<br>    select * from {{ ref(&#39;raw_customers&#39;) }}<br><br>),<br><br>renamed as (<br><br>    select<br>        id as customer_id,<br>        first_name,<br>        last_name<br><br>    from source<br><br>)<br><br>select * from renamed</pre><p>There are a couple of good things done right in this Staging Model:</p><ul><li>✅ It is an Incremental model, with id as the <a href="https://docs.getdbt.com/reference/resource-configs/unique_key">unique_key</a></li><li>✅ It starts with an “Import” CTE source</li><li>✅ Only used columns are selected (CTE renamed) - <strong>NEVER do SELECT * from other models!</strong></li></ul><h3>stg_orders</h3><p><a href="https://github.com/dbt-labs/jaffle_shop/blob/main/models/staging/stg_orders.sql">https://github.com/dbt-labs/jaffle_shop/blob/main/models/staging/stg_orders.sql</a></p><pre>with source as (<br><br>    {#-<br>    Normally we would select from the table here, but we are using seeds to load<br>    our data in this project<br>    #}<br>    select * from {{ ref(&#39;raw_orders&#39;) }}<br><br>),<br><br>renamed as (<br><br>    select<br>        id as order_id,<br>        user_id as customer_id,<br>        order_date,<br>        status<br><br>    from source<br><br>)<br><br>select * from renamed</pre><p>Basically it follows the same pattern as the stg_customers example, and consistency is a severly undervalued qualify of code.</p><h3>Appendix</h3><ul><li><a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/2-staging">Staging: Preparing our atomic building blocks | dbt Developer Hub</a></li><li><a href="https://medium.com/data-panda/dbt-tests-813f0aeacac8">dbt Tests</a></li><li><a href="https://medium.com/data-panda/dbt-models-marts-layer-i-e-data-marts-1220abda4471">dbt Models — Marts layer (i.e. Data Marts)</a></li><li><a href="https://medium.com/data-panda/dbt-the-supernova-in-data-analytics-engineering-79d0ceaee450">dbt — The supernova in data &amp; analytics engineering</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55f0f2ddc5e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/dbt-models-staging-layer-55f0f2ddc5e4">dbt Models — Staging layer</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Agile could work in data teams — Scrum & Kanban]]></title>
            <link>https://medium.com/data-panda/how-agile-could-work-in-data-teams-scrum-kanban-28e73a7fc10e?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/28e73a7fc10e</guid>
            <category><![CDATA[kanban]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sun, 03 Sep 2023 16:23:53 GMT</pubDate>
            <atom:updated>2025-12-17T15:49:59.572Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Agile could work in data teams — Scrum &amp; Kanban</h3><h4>One of the possible ways to organize data team workflow</h4><p><em>📌 December 2025 Update</em></p><p><em>I’ve moved my writing to Substack where I publish deeper insights on data leadership, analytics engineering, and business strategy.</em></p><p><a href="https://jimmypang.substack.com/"><em>👉 Read my latest articles on Data Biz</em></a></p><p><em>I’m now covering topics like:</em></p><ul><li><em>Defining P0 SLAs with real value-stream examples from Vestiaire Collective</em></li><li><em>BI-agnostic data architecture design</em></li><li><em>Data team leadership and stakeholder management</em></li><li><em>dbt best practices and modern data marts</em></li><li><em>Prioritization frameworks for data leaders</em></li></ul><p><a href="https://jimmypang.substack.com/"><em>Subscribe for free →</em></a><em> to get weekly insights from building data teams in Berlin’s startup ecosystem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*wGCTS7o9CT2eQnZN" /><figcaption>A simple illustration of the steps of Agile</figcaption></figure><h3>Overview</h3><p>As it kept being repeated among the data leaders: data is a team sport. To ensure team collaboration and stakeholder expectation management purpose, some sort of workflow management has to be put in place. At this aspect, Tech &amp; Product have already employed the Agile methodology for software engineering purpose for a fairly long time. As the development of the data industry leans towards software engineering more and more these years, the Agile methodology is one of the adopted workflow.</p><p>Nevertheless, the dynamic between software engineering and working in data (regardless Business Intelligence, Analytics Engineering, Data Engineering, or Data Science) is still different. In general, Data as a function is much closer to business, hence more volatile, and more unpredictable. In various organization, the interpretation of Agile could vary pretty widely. Yet, adoption in Data is usually more simplistic than in Tech.</p><p>This article is aimed at providing one of the interpretation of Agile in analytics teams, merely serving as an inspiration — every reading practitioner ought to understand the very essence of it, and adopt whatever component deemed useful and relevant in their context.</p><h3>What is Agile?</h3><p>If one spends enough time in the tech and data industry, he or she is bound to hear the keyword of <strong>Agile</strong>.</p><p>To simply put, Agile is a methodology and …</p><blockquote>a project management approach that involves breaking the project into phases and emphasizes continuous collaboration and improvement. Teams follow a cycle of <strong>planning</strong>, <strong>executing</strong>, and <strong>evaluating</strong>. — <a href="https://www.atlassian.com/agile">Atlassian</a></blockquote><p>When it comes to execution, there are 2 major subtypes of it in a board sense: <strong>Kanban</strong>, and <strong>Scrum</strong>.</p><h3>Kanban vs Scrum</h3><p>This comparison table provides a rather good overview of the key differences of the 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*oCVC8d2OrXY43wgj6_JI2Q.png" /><figcaption>From <a href="https://www.atlassian.com/agile/kanban/kanban-vs-scrum">Atlassian: Kanban vs Scrum</a></figcaption></figure><p>The highlights here are:</p><ul><li>Ideology: Scrum has the elements of regular learning &amp; reflection, which enable long-term growth of the team; Kanban here is merely just visualize the latest state of Work-In-Progress for collaboration and expectation management purposes</li><li>Cadence: The existence of Sprints at Scrum helps the team to stay focus on the prioritized tasks (i.e. deflect ad-hoc requests from stakeholders); Business stakeholders may be in favor for the Continuous Flow in Kanban, but it usually results in constant interruption to the team — which usually leads to frustration</li><li>Practices: Scrum typically have a couple of Ceremonies (e.g. Daily Standups, Grooming, Planning, and Retrospective session per Sprint); As for Kanban, the same set of Ceremonies could be involved but usually they are optional. Instead, pratices like limiting the number of Work-In-Progress would be more commonly used</li><li>Roles: This is in my opinion the biggest difference between Scrum and Kanban — At Scrum, the role of Product Owner, Scrum Master, and Dev team have to be present and conduct their duties in the Ceremonies (e.g. Planning, and Grooming sessions etc.); On the other hand, Kanban has no required roles</li></ul><h4>Considerations of moving from Kanban to Scrum</h4><p>While the team is small (e.g. just 1–2 people team), it may make sense to let things be dynamic and even sometimes chaotic — in this context, the approach of “move fast, break things” could still make sense as long as fast delivery can be maintained.</p><p>Yet, while the business and the team scale up, the collaboration within the development team becomes more and more complex and the cost of breaking things (e.g. data downtime, poor data quality) becomes greater than just quickly delivering 1 small ticket. While the organization reaches to a certain size, it would be better to ensure the reliability &amp; transparency of the data assets instead of just quickly building new features.</p><p>Compared to Kanban, Scrum is more suited to development environments that require more predictability &amp; stability rather than allowing changes to come and impact the planned work. It is essential to ensure the team could focus on certain planned tasks in Sprints instead of allowing random new requests jumping in and cost the team’s focus.</p><h4>Kanban</h4><p><strong>Kanban</strong> (看板, signboard in Japanese &amp; Chinese) is a basically a scheduling system for lean manufacturing originated from Toyota back in 1950s-1960. Toyota as a automobile manufacturer introduced the concept of Just-In-Time manufacturing (or, JIT in short) to improve efficiency and reduce waste during the process.</p><p>As software engineering has its origin of hardware manufacturing (e.g. IBM), this sort of process management is adopted during the early days of it. As of today, it is still one of the easiest to adopt process in many engineering &amp; analytics teams as it doesn’t have the threshold of Scrum expertise and roles of Product Owner, Scrum Master, and Dev team. And, the visual element of Kanban also make it rather appealing to the stakeholders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tLelWZ1oEVozR-1v" /><figcaption>A classic Kanban board from <a href="https://www.atlassian.com/agile/kanban">Atlassian</a></figcaption></figure><h4>Scrum</h4><p>Scrum on the other hand, has a higher threshold to adapt. Nevertheless, when it is done right, the benefits are also tremendous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*xUcbK_DdgJ6dBjA4" /><figcaption><a href="https://www.atlassian.com/agile/scrum">Sprint Cycle</a></figcaption></figure><p>An important thing to bear in mind is <strong>the implementation of Scrum is not entirely rigid</strong>, it can be tailored to better suit the needs of different organizations. Still, there are 2 must-haves for Scrum: Roles and certain Ceremonies.</p><p>The required roles in Scrums are:</p><ul><li>Product Owner — The champion who understand the business, the customers, and the product inside out</li><li>Scrum Master — The champion of Scrum practices, understands how works are done by the Dev team, and constantly coaching the Product Owner and the Dev team for better Scrum pratices</li><li>Dev team — The people who actually get their hands dirty and get the job done. A mature Dev team could actually self-organize and self-sustain for a period of time without much input from Product Owner and Scrum Master</li></ul><p>Respectively, these elements need to be addressed:</p><ul><li>Backlogs (e.g. User Stories, Requests in the form of Tasks, Bugs, etc.)</li><li>Hosting Agile Ceremonies (e.g. Daily Standups, Grooming &amp; Planning sessions etc.)</li></ul><h3>Agile Ceremonies</h3><p>There are typically 4 Agile Ceremonies being employed in data teams:</p><ol><li>Daily Standup</li><li>Grooming session</li><li>Planning session</li><li>Retrospective</li></ol><h4>Daily Standup</h4><p>Daily Standup is a short burst touch-point with the team, in order to discover these as quickly as possible:</p><ul><li>What was done in the previous working day</li><li>What would be the focus of today</li><li>If there is any blocker to the team or any support or question for the team</li><li>What action can be taken to unblock the team</li></ul><h4>Grooming session</h4><p>A grooming session is a team process that focuses on:</p><ul><li>Review newly created tickets</li><li>Refine the spec &amp; context of those new tickets</li><li>Estimated the effort required from the ticket</li><li>A certain level of re-calibration is needed, due to the different speeds of individual teammates.</li></ul><p>It should be done in <strong>with Domain experts</strong> to ensure collaboration synergy, and the team should support each other with the information obtained from this session.</p><h4>Story Points estimation</h4><p>Story Points is a commonly used measurement regarding the effort required in order to finish a task (ticket) in the Agile framework.</p><p>The measurement of Story Points estimation really depends on the context. Here is just one of the examples can be adapted for the data team context:</p><ul><li>Story Points: <strong>0</strong>/No ticket ← Estimated Effort Time: 0–30 mins</li><li>Story Points: <strong>1</strong> ← Estimated Effort: 30 mins — 2 hours</li><li>Story Points: <strong>2</strong> ← Estimated Effort: 2 — 4 hours</li><li>Story Points: <strong>3</strong> ← Estimated Effort: 4 hours — 1 day</li><li>Story Points: <strong>5</strong> ← Estimated Effort: 1–2 days</li><li>Story Points: <strong>8 </strong>← Estimated Effort: 2–3 days</li><li>Story Points: <strong>13 </strong>← Estimated Effort: 3–5 days</li></ul><p>In addition, a couple of important notes here:</p><ol><li>Any ticket being estimated to be more than 13 Story Points should be broken down to smaller pieces (i.e. split it into different smaller tickets and link them together with relationships). A nice proportion should be seeing many tickets in 3–5 Story Points, a few on 0–2.</li><li>The Effort means “time working on the ticket” <strong>only</strong>. In the other words, the Review time should NOT be taken into consideration during story points estimation</li><li>Small requests like “stakeholders requesting Tableau access” (i.e. adding the User to a specific User Group in Tableau Server) probably don’t need a ticket created for it.</li></ol><h4>Planning session</h4><p>The Planning session aims at planning tickets in the upcoming Sprint(s). As a rule of thumb, the capacity of 1 individual for 1 Sprint is around <strong>20–24 Story Points</strong> (it is open for adjustments in the future iteration).</p><h4><strong>Best Practices in the Planning session</strong></h4><ul><li>In the data team context (especially for analyst teams being very close to business), it might make sense to make it domain-specific (i.e. planning the upcoming Sprint with domain stakeholders)</li><li>It is always a good idea to spare buffer (e.g. 10–20%) in case of unexpected situations (e.g. HOTFIXs, Bugs, sick leaves of team members etc.) in the upcoming Sprint</li><li>It is always better to under-commit &amp; over-deliver than another way around — so it is better to commit to fewer tickets in the upcoming Sprint and only bring tickets from the next Sprint to the current Sprint</li><li>ALWAYS confirm with the team regarding vacations &amp; holidays in the upcoming Sprint — it could potentially have quite some impact on the estimation</li><li>ALWAYS remember — the whole point of Scrum is to help the team to focus on tasks in the upcoming Sprints in order to have a better quality of work. So, intruder tickets (i.e. tickets not committed but put in the current Sprint on the fly) should be avoided — the only exceptions are:</li><li>All planned tickets in the current Sprint are blocked: If it happens, put that blocking ticket in the current sprint and mark the Priority of that blocking ticket as “BLOCKER”</li><li>Bug tickets: Stakeholders should be able to use Data Assets with 100% confidence. While the quality of Data Assets is compromised, the focus should be on keeping stakeholders’ confidence on a high level instead of building new features.</li><li>Unexpected urgency — primary stakeholders explicitly request with valid reason. The team should make have proper judgments given the context to decide if that request should be allowed as an intruder ticket or not.</li></ul><h4>During the Sprint — Things to do</h4><ul><li>Respect Planning: Avoid intruder tickets if possible, the goal is to protect the focus of the team during the Sprint to ensure the quality of the delivery</li><li>Help each other: If an individual finishes the planned tickets ahead, the individual ought to help out other teammates first (e.g. Review MRs, reply to stakeholders inquiries on Slack or email) instead of moving new tickets to the existing Sprint.</li><li>If there is no help required from the team internally, it is always a good practice to review existing documentation and make sure all TODOs have well-written tickets. By doing so, the team would move even faster in long run and it also benefits the tech &amp; product team with well-written tickets</li><li>Only when the above are all fulfilled, the team could consider moving tickets from the next Sprint to the current Sprint to have early delivery for the stakeholders</li></ul><h4>Retrospective</h4><p>The goal of Retrospective is to have a continuous feedback loop within the team to constantly improve effectiveness, efficiency, and team satisfaction level.</p><p><strong>Important things to keep in mind</strong></p><p>In order to deliver honest &amp; constructive feedback regarding the completed Sprint, these conditions MUST be in place:</p><ul><li>A <strong>strong base of trust in each other in the team</strong></li><li>An open attitude for constructive feedback — the goal is to improve the quality of work, hence</li><li>Be honest: feedbacks and discussions are vital to bringing the team to the next level, in both of professional &amp; personal level</li></ul><p><strong>Common steps in Retrospective </strong>(Suggested tooling: <a href="https://easyretrospectives.com/">Easy Retrospective</a>)</p><p>A retrospective session usually takes around 1 hour, right after the end of a Sprint. It usually contains these elements:</p><p>1. <strong>Write (Should be Done BEFORE the session!)</strong></p><ul><li>What was Good in the previous Sprint?</li><li>What was Bad in the previous Sprint?</li><li>Iterate ALL cards written, starting from Good. Then move on to the Bad side</li><li>The teamwork together to group cards at Bad side together</li></ul><p>2. <strong>Vote</strong></p><ul><li>Give 5 minutes for the team to vote the grouped cards for the sake of deciding which issues to tackle first</li></ul><p>3. <strong>Solve</strong></p><ul><li>Starting with grouped cards with the highest votes, the team propose and discuss solutions</li></ul><p>4. <strong>Summary</strong></p><ul><li>Document the summary in Confluence (location to be decided)</li><li>It is vital to keep it documented as Issues should be addressed &amp; acted on after discussion</li></ul><h3>Further Readings</h3><ul><li><a href="https://www.atlassian.com/agile">What is Agile methodology?</a></li><li><a href="https://www.atlassian.com/agile/kanban/kanban-vs-scrum">Kanban vs. scrum: which agile are you?</a></li><li><a href="https://www.atlassian.com/agile/kanban">What is kanban?</a></li><li><a href="https://www.atlassian.com/agile/scrum">What is scrum?</a></li><li><a href="https://en.wikipedia.org/wiki/Kanban#cite_note-3">Kanban (Wiki)</a></li><li><a href="https://en.wikipedia.org/wiki/Lean_manufacturing#Key_principles_and_waste">Lean manufacturing (Wiki)</a></li><li><a href="https://agilemanifesto.org/">Agile Manifesto</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28e73a7fc10e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/how-agile-could-work-in-data-teams-scrum-kanban-28e73a7fc10e">How Agile could work in data teams — Scrum &amp; Kanban</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt — The supernova in data & analytics engineering]]></title>
            <link>https://medium.com/data-panda/dbt-the-supernova-in-data-analytics-engineering-79d0ceaee450?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/79d0ceaee450</guid>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Fri, 01 Sep 2023 20:27:22 GMT</pubDate>
            <atom:updated>2025-12-17T15:34:30.519Z</atom:updated>
            <content:encoded><![CDATA[<h3>dbt — An overview of the supernova in data &amp; analytics engineering</h3><p><em>📌 December 2025 Update: I’ve published a fresh perspective on dbt’s evolution and where it’s headed in 2025 on my Substack newsletter.</em></p><p><a href="https://jimmypang.substack.com/p/dbt-in-2025"><em>👉 Read: dbt in 2025 — The Next Chapter</em></a></p><p><em>This 2023 overview still holds up, but the ecosystem has changed dramatically. In my new article, I explore AI-powered analytics engineering, dbt’s expanding role, and what data teams should prepare for. </em><a href="https://jimmypang.substack.com/"><em>Subscribe to Data Biz</em></a><em> for weekly insights on analytics engineering and data leadership.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QdhM2lmpYcDzy1Xn" /><figcaption>The dbt logo.</figcaption></figure><h3>What is dbt (Data Build Tool)?</h3><p>From <a href="https://docs.getdbt.com/docs/introduction">dbt Lab</a>:</p><blockquote>dbt is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility.</blockquote><p>In the modern ELT (<strong>E</strong>xtract, <strong>L</strong>oad, <strong>T</strong>ransform) approach for data engineering, dbt takes care of the <strong>T</strong> (<a href="https://en.wikipedia.org/wiki/Data_transformation_(computing)">Data Transformation</a>) as an open-source Data Transformation tool. The team behind dbt is called dbt Lab (old name: <a href="https://www.getdbt.com/blog/welcome-to-fishtown-analytics/">Fishtown Analytics</a>).</p><h3>Commonly used components</h3><h4>Models</h4><h3>Recommended architecture from dbt Labs</h3><p>In <a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/1-guide-overview#guide-structure-overview">dbt Lab’s official documentation</a>, this structure is proposed as the best practice:</p><ol><li>Staging</li><li>Intermediate</li><li>Marts</li></ol><p>Where, <a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/2-staging">Staging Models</a> are:</p><blockquote>The staging layer is where our journey begins. This is the foundation of our project, where we bring all the individual components we’re going to use to build our more complex and useful models into the project.</blockquote><p>And, <a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/3-intermediate">Intermediate Models</a> are:</p><blockquote>The intermediate layer is where these molecules live, creating varied forms with specific purposes on the way towards the more complex proteins and cells we’ll use to breathe life into our data products.</blockquote><p>Last but not least, <a href="https://docs.getdbt.com/guides/best-practices/how-we-structure/4-marts">Marts Models</a> are:</p><blockquote>This is the layer where everything comes together and we start to arrange all of our atoms (staging models) and molecules (intermediate models) into full-fledged cells that have identity and purpose. We sometimes like to call this the <em>entity</em> <em>layer</em> or <em>concept layer</em>, to emphasize that all our marts are meant to represent a specific entity or concept at its unique grain.</blockquote><h3>Tests</h3><p><a href="https://docs.getdbt.com/docs/build/tests">dbt Tests</a> is:</p><blockquote>Tests are assertions you make about your models and other resources in your dbt project (e.g. sources, seeds and snapshots). When you run dbt test, dbt will tell you if each test in your project passes or fails.</blockquote><p>There are 4 types of Tests coming with dbt-core natively:</p><ul><li>not_null</li><li>unique</li><li>accepted_values</li><li>relationships</li></ul><p>The most typical use cases of test implementation would be Primary Key, which usually are applied with both the not_null (<a href="https://docs.getdbt.com/reference/resource-properties/tests#not_null">https://docs.getdbt.com/reference/resource-properties/tests#not_null</a>) and unique (<a href="https://docs.getdbt.com/reference/resource-properties/tests#unique">https://docs.getdbt.com/reference/resource-properties/tests#unique</a>) test.</p><p>For business critical columns, it is also suggested to start with not_null test to have a sense of how good the data quality is of those columns. Once more confidence in data quality of the column is gained, accepted_values (<a href="https://docs.getdbt.com/reference/resource-properties/tests#accepted_values">https://docs.getdbt.com/reference/resource-properties/tests#accepted_values</a>, for categorical columns) and relationships (for <a href="https://en.wikipedia.org/wiki/Referential_integrity">Referential integrity</a>).</p><p>For more sophisticated data quality tests, tests from <a href="https://github.com/dbt-labs/dbt-utils#generic-tests">dbt-utils</a> and <a href="https://hub.getdbt.com/calogica/dbt_expectations/latest/">dbt_expectations</a> can also be considered.</p><h3>Macros</h3><p><a href="https://docs.getdbt.com/docs/build/jinja-macros#macros">dbt Macros</a> is:</p><blockquote><a href="https://docs.getdbt.com/docs/build/jinja-macros#macros">Macros</a> in Jinja are pieces of code that can be reused multiple times — they are analogous to “functions” in other programming languages, and are extremely useful if you find yourself repeating code across multiple models. Macros are defined in .sql files, typically in your macros directory (<a href="https://docs.getdbt.com/reference/project-configs/macro-paths">docs</a>).</blockquote><h3>Packages</h3><p><a href="https://docs.getdbt.com/docs/build/packages">dbt Packages</a> are the additional library that could enhance the functionality of dbt.</p><blockquote>Software engineers frequently modularize code into libraries. These libraries help programmers operate with leverage: they can spend more time focusing on their unique business logic, and less time implementing code that someone else has already spent the time perfecting.</blockquote><p>dbt also hosts a hub of all supported packages on dbt here: <a href="https://hub.getdbt.com/">dbt — Package hub</a> .</p><p>Certain popular packages like <a href="https://hub.getdbt.com/dbt-labs/dbt_utils/latest/">dbt-utils</a> and <a href="https://hub.getdbt.com/elementary-data/elementary/latest/">Elementary</a> are widely implemented. Additional packages like <a href="https://hub.getdbt.com/dbt-labs/snowplow/latest/">Snowplow</a> is also in the progress of implementation as of today.</p><h3>Seeds</h3><p><a href="https://docs.getdbt.com/docs/build/seeds">Seeds</a> in dbt is:</p><blockquote>Seeds are CSV files in your dbt project (typically in your seeds directory), that dbt can load into your <a href="https://docs.getdbt.com/terms/data-warehouse">data warehouse</a> using the dbt seed command.</blockquote><p>In general, if there are business-defined tables (usually coming in the form of a Google Sheet import request), it is better to cover these requests with Seeds instead of building another Google Sheet import job if It is not expected to have very frequent changes of that table.</p><p>Here are a couple of benefits of using Seeds instead of Google Sheet import:</p><ol><li>Version Control</li><li>Everything in 1 place: the CSV files are directly in the repo</li><li>Enabling with ref functions, that improve the dbt Lineage &amp; dependency management</li></ol><p>Note:</p><ul><li>It is vital to make sure the relevant Airflow DAG has a task to run the dbt seed command (<a href="https://docs.getdbt.com/reference/commands/seed">About dbt seed command</a>) before dbt run, or else the table won’t be generated</li></ul><h3>Snapshots</h3><p><a href="https://docs.getdbt.com/docs/build/snapshots">Snapshots</a> in dbt is:</p><blockquote>Snapshots implement <a href="https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row">type-2 Slowly Changing Dimensions</a> over mutable source tables. These Slowly Changing Dimensions (or SCDs) identify how a row in a table changes over time. Imagine you have an orders table where the status field can be overwritten as the order is processed.</blockquote><h3>Analyses</h3><p><a href="https://docs.getdbt.com/docs/build/analyses">Analyses</a> in dbt is:</p><blockquote>dbt’s notion of models makes it easy for data teams to version control and collaborate on data transformations. Sometimes though, a certain sql statement doesn&#39;t quite fit into the mold of a dbt model. These more &quot;analytical&quot; sql files can be versioned inside of your dbt project using the analysis functionality of dbt.</blockquote><p>Some best use cases of Analyses are:</p><ul><li>Data Auditing</li><li>Onboarding &amp; Training purposes</li><li>1 off analysis</li></ul><h3>Further Readings</h3><p>More content to be added &amp; enriched in the future here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79d0ceaee450" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/dbt-the-supernova-in-data-analytics-engineering-79d0ceaee450">dbt — The supernova in data &amp; analytics engineering</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Tips To Improve Collaboration Between Data and Business Teams]]></title>
            <link>https://medium.com/vestiaire-connected/5-tips-to-improve-collaboration-between-data-and-business-teams-601b36ce00b7?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/601b36ce00b7</guid>
            <category><![CDATA[communication-skills]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[stakeholder]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Thu, 16 Feb 2023 07:02:40 GMT</pubDate>
            <atom:updated>2025-12-17T15:50:19.881Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to nurture healthy stakeholder relationships in an international scale-up organization.</h4><p><em>📌 December 2025 Update</em></p><p><em>I’ve moved my writing to Substack where I publish deeper insights on data leadership, analytics engineering, and business strategy.</em></p><p><a href="https://jimmypang.substack.com/"><em>👉 Read my latest articles on Data Biz</em></a></p><p><em>I’m now covering topics like:</em></p><ul><li><em>Defining P0 SLAs with real value-stream examples from Vestiaire Collective</em></li><li><em>BI-agnostic data architecture design</em></li><li><em>Data team leadership and stakeholder management</em></li><li><em>dbt best practices and modern data marts</em></li><li><em>Prioritization frameworks for data leaders</em></li></ul><p><a href="https://jimmypang.substack.com/"><em>Subscribe for free →</em></a><em> to get weekly insights from building data teams in Berlin’s startup ecosystem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*9avdIJ4iMZnP1IyO.jpeg" /></figure><h3>Motivation</h3><p>Since <a href="https://www.atlassian.com/agile/agile-at-scale/spotify">the Spotify model</a> was introduced in 2012, the tech industry has been adapting the ideas of Squads, Tribes, Chapters, Trios, and Alliances.</p><p>I have seen different adaptations of the Spotify model during my time in various Tech corporations. The first company pretty much followed through with the structure of Squads &amp; Tribes for the official reporting line. Guilds were a bizarre bottom-up movement. All Guilds were formulated by driven individuals and operated on a voluntary basis. Take the Data Guild for example, it started with around 20–30 data talents meeting in a glass room irregularly. By the time I left, it had boosted up to a community having two internal meetups per month with 50–150 participants — and the Guild channel on Slack has around 500+ active members today.</p><p>The second company took a slightly different adaptation strategy. They used Squads &amp; Tribes, however, they also used Alliances which were collections of more than one Tribe in the organization’s array. In addition, Chapters were also created to serve as a platform for data talents to share knowledge &amp; best practices within their own function (e.g. Analytic Chapter, Data Engineering Chapter, Data Science Chapter, etc.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LTXd-Hhpoo6mc88U" /><figcaption>Spotify model | <a href="https://www.atlassian.com/agile/agile-at-scale/spotify">Atlassian</a></figcaption></figure><p>While listening to those professional Data talents, the same set of struggles often come up from time to time: empty tickets created by stakeholders, requests without any specific details, ad-hoc requests coming in at the last minute, scope creeps, etc.</p><p>The collaboration between Data teams and their stakeholders (Business, Product, and Tech) is more an art than hard science. I figured that writing a tech blog post may be a beneficial way to bring in new angles for various Data teams and even stakeholders to have a more smooth collaboration — hence this article.</p><h3>1. Have empathy</h3><blockquote><em>Empathy is the capacity to understand or feel what another person is experiencing from within their frame of reference, that is, the capacity to place oneself in another’s position.</em></blockquote><blockquote><em>- Bellet PS, Maloney MJ (October 1991). “The importance of empathy as an interviewing skill in medicine”</em></blockquote><p>In plain language, empathy stands for the capacity to “put ourselves in another person’s shoes”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/0*9-Ch11rAoLEfIVeN" /><figcaption>The Culture Map | By Erin Meyer</figcaption></figure><p>The book <a href="https://erinmeyer.com/books/the-culture-map/">The Culture Map</a>, written by Erin Meyer, was quite a hit some years ago. It wasn’t a coincidence — instead, it caught people’s attention as it is getting more relevant than ever.</p><p>Being a France-based company, 60–70% of Vestiaire Collective’s employees are French. Nevertheless, it has already expanded to 70+ countries and has offices across Europe, APAC, and the Americas. In other words, our colleagues come from all around the world.​​</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*pygoCnCtAnpVzkfd" /><figcaption>Cultural differences between French, German, Chinese, and Japanese working culture across various dimensions — an interesting illustration from the book. | Erin Meyer</figcaption></figure><p>On top of different cultural backgrounds, the teams in Vestiaire Collective also have diverse expertise: Data, Product, Tech, etc. Even under the big umbrella of “Business”, there are also different domain functions: Marketing, Finance, Operations, Business Performance, and others.</p><p>Occupation drives certain ways of thinking. For example, Data and Tech teams always appreciate these elements whenever receiving requests:</p><ul><li>Clear Use Cases</li><li>Specific details of the request</li><li>Expected timeline</li><li>Relevant documentations</li></ul><p>On the other hand, Business teams are standing on the very frontline performing their duties and seeing a fairly different perspective. Their job is not always predictable due to the nature of business itself — sometimes life just happens! For them, the preference is always:</p><ul><li>Ad Hoc requests can be submitted whenever.</li><li>Requests being delivered ASAP — it should have been done yesterday!</li><li>Just tap the Data people’s shoulder and ask for it — it should be an easy task, 5 minutes tops!</li></ul><p>To collaborate smoothly, both sides need to understand each other’s thinking and rationale. <strong>Data and Business teams ought to be partners.</strong> A partnership is a relationship — and a relationship takes commitment and sometimes compromise to keep it running healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*_EbujD3ghh1ZZa9Y" /><figcaption>Steven Covey’s Maturity Continuum</figcaption></figure><p>The 5th habit from the book <a href="https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People">The 7 Habits of Highly Effective People</a> says it well: “<em>Seek first to understand then to be understood</em>”. We as human beings are given 2 ears and only 1 mouth for a good reason.</p><h3>2. Plan ahead</h3><blockquote><em>If you fail to plan, you plan to fail.</em></blockquote><blockquote><em>- Martin Lassen</em></blockquote><p>When it comes to collaboration between Data and Business teams, it is always a good idea to plan ahead.</p><p>Planning is essential for Business teams, sometimes even more so than for Data teams. The Business plan sets the tone, which helps the Data teams better understand where the value is and hence plan their tasks effectively. In other words,<strong> there is a dependency from Data teams on Business teams’ planning.</strong></p><p>Likewise, the Data teams’ planning is vital for the Business teams as it gives them a better sense of expectation for the deliverables. Whether for quarterly roadmaps or sprint prioritization, planning is a necessary action to decide on priorities ahead of time. Why is it important? Because it saves time to think “if this task is important” and allows the teams to fully focus on delivery.</p><h3>3. Document everything</h3><p>It is pretty self-explanatory: putting information down in black and white helps iron out many potential issues.</p><p>First, documentation helps the teams double-check if they genuinely understand the business concepts, data concepts, or what is expected. They often think they comprehend everything before unexpected issues pop up once they actually get their hands on it.</p><p>Secondly, it provides a good source of information for future reference — which aids in planning and framing communications. Proper documentation also eases the difficulties of onboarding new joiners and non-specialist contributors. On top of that, it further enhances collaboration at both Business and Data teams levels.</p><p>The team I stayed with in the second company mentioned in the introduction went so far as to stand by the following principle: “If it was not documented, it didn’t happen.”. It might sound radical at first glance, but this policy worked reasonably well and the documentation there was well organized and linked to other internal and external resources. The team used the traffic going through their documentation as a KPI.</p><h3>4. Communication</h3><p>The book <em>Talking to Each Other (</em>German<em>: </em><a href="https://www.amazon.com/Miteinander-reden-Vol-Psychologie-Kommunikation/dp/3499174898">Miteinander Reden</a>) by Friedemann Schulz von Thun introduced this very interesting model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/0*2_LeYsK12vQIZ8pv" /></figure><p>Graphic of the four-sides model | <a href="https://en.wikipedia.org/wiki/Four-sides_model">Wikipedia</a></p><p>Communication is merely an information exchange between the Sender and the Receiver. And Schulz v. Thun also highlighted that there are four components in the message delivery flow:</p><ol><li>Factual Information</li><li>Self-Revelation</li><li>Relationship</li><li>Appeal</li></ol><p>Factual Information is merely sharing information. E.g., “<em>Here is the ticket. It contains the content of what I need</em>”.</p><p>Self-Revelation is the message the Sender wishes to express about themselves. E.g., “<em>Hey, I know how to use JIRA. I know tech and data!</em>”.</p><p>Relationship implies how the Sender perceives the Receiver and vice-versa, as well as how the Sender perceives the interaction. E.g., “<em>I am your stakeholder in this ticket.</em>”.</p><p>The Appeal is the request the Sender wishes to communicate to the Receiver. E.g., “<em>Please work on this ticket ASAP!</em>”.</p><p>When communicating, individuals also share a fair amount of information non-verbally. This is especially true for in-person conversations, as our bodies are message transmitters! Source: <em>What Every BODY is Saying</em> by Joe Navarro and Marvin Karlins.</p><p>Depending on their cultural background, people convey and receive messages differently. Using the graph from point <em>1 — Have empathy,</em> we can see fairly significant differences in communication between the four countries in the graph, which is just a small sample of reality. For example, Germany has a low-context culture, while France has a high-context one. It implies that people with French backgrounds tend to expect the receiving end to decrypt the conveyed message from the context, while people with German backgrounds are expecting to get the message in a more explicit manner.</p><p>As a best practice, it is better to over-communicate and be specific and explicit rather than leaving too much room for interpretation — which is usually the cause of misunderstandings.</p><p>In addition, the relationship between the Sender and Receiver also plays an important role. As mentioned above, Data and Business teams are partners and should be equal. The communication should also express this relationship between both parties.</p><h3>5. Recognize the inputs from each party</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VBx7F26iFmBGph5f" /><figcaption>Pyramid Representation of Maslow’s Hierarchy of Needs | <a href="https://www.researchgate.net/publication/353316750_Linking_Motivation_and_Employee_Engagement_through_Gamification_in_Remote_Working">Researchgate</a></figcaption></figure><p>The famous physiologist Abraham Maslow has proposed the concept of human needs on 5 levels, namely “<a href="https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs">Maslow’s Hierarchy of Needs</a>”.</p><p>In this theory, humans would start pursuing needs on the next level while their basic needs are satisfied. Recognition of one’s achievement falls in the 4th level of the pyramid: Esteem. Successful delivery of projects is seldom only one person’s or team’s effort. Instead, they are mostly the outcome of effective collaborations across multiple teams (Data and Business teams included). While the Data teams ought to understand the value brought to the table by the Business teams, the Business teams should also recognize the contribution of the Data team.</p><p>Recognition is a two-way street. There is a famous Chinese expression in Hong Kong: “<em>一個巴掌拍不響</em>”. It could be literally translated into “a person cannot clap with only one hand”. It takes only one of them, either the Business team or the Data team, to start recognizing the other side’s contribution. The momentum would start building and eventually lead to a virtuous circle of motivation — a vibe everyone likes and is happy to be part of.</p><h3>Conclusion</h3><p>Out of the 5 tips, the first one about empathy is the very foundation of all other pieces of advice. Having empathy, the Data and Business teams will:</p><ul><li>Plan ahead to let each other know what to expect.</li><li>Build proper documentation for each other’s references.</li><li>Make sure they understand each other.</li><li>Recognize the contributions from different parties and celebrate success together.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CYgAY7cA-JSW9ag9" /><figcaption>We hear you | Jon Tyson via Unsplash</figcaption></figure><h3>Further Readings</h3><ul><li><a href="https://blog.crisp.se/2015/06/07/henrikkniberg/no-i-didnt-invent-the-spotify-model">No, I didn’t invent the Spotify model</a></li><li><a href="https://towardsdatascience.com/to-improve-your-relationships-with-ds-stakeholders-offer-them-a-menu-dfc15f3006c6">To Improve Your Relationships with DS Stakeholders, Offer Them a Menu</a></li><li><a href="https://erinmeyer.com/books/the-culture-map/">The Culture Map</a></li><li><a href="https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People">The 7 Habits of Highly Effective People</a></li><li><a href="https://www.goodreads.com/book/show/1173576.What_Every_Body_is_Saying?from_search=true&amp;from_srp=true&amp;qid=0hLT8fNhdw&amp;rank=1"><em>What Every BODY is Saying</em></a></li><li><a href="https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs">Maslow’s Hierarchy of Needs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=601b36ce00b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/vestiaire-connected/5-tips-to-improve-collaboration-between-data-and-business-teams-601b36ce00b7">5 Tips To Improve Collaboration Between Data and Business Teams</a> was originally published in <a href="https://medium.com/vestiaire-connected">Vestiaire Connected</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dashboarding SOP (Standard Operating Procedure) — How to build a dashboard, properly]]></title>
            <link>https://medium.com/data-panda/dashboarding-sop-standard-operating-procedure-how-to-build-a-dashboard-properly-78a0be930ef0?source=rss-b47cdbdc70c------2</link>
            <guid isPermaLink="false">https://medium.com/p/78a0be930ef0</guid>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Jimmy Pang]]></dc:creator>
            <pubDate>Sun, 08 Jan 2023 22:14:55 GMT</pubDate>
            <atom:updated>2025-12-17T15:50:37.889Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dashboarding SOP (Standard Operating Procedure) — How to build a dashboard, properly</h3><p><em>📌 December 2025 Update: I’ve republished and expanded this Dashboarding SOP on my new Substack newsletter, </em><a href="https://jimmypang.substack.com/p/dashboarding-sop-how-to-build-a-dashboard"><em>Data Biz</em></a><em>, where I now publish all my data leadership and analytics engineering content.</em></p><p><a href="https://jimmypang.substack.com/p/dashboarding-sop-how-to-build-a-dashboard"><em>Read the updated version here →</em></a></p><p><em>On Data Biz, I share deeper insights on data strategy, team leadership, and business-first analytics — topics like defining P0 SLAs with real value-stream examples, BI-agnostic architecture design, and navigating C-suite conversations as a data leader. </em><a href="https://jimmypang.substack.com/"><em>Subscribe free</em></a><em> to join the community.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FBvzr_w3sTlFG_2O" /><figcaption>Photo by <a href="https://unsplash.com/es/@pathdigital?utm_source=medium&amp;utm_medium=referral">path digital</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Dashboards — what it is and why it is still relevant</h3><p>Dashboards are the go-to 1 stop shop for business decision-makers to gauge the high-level performance of the business units. The whole objective of dashboarding is to provide <strong>actionable insights to the decision-makers, as fast as possible</strong>.</p><p>That being said, there should only be a few “source of truth” dashboards like the Business Performance Cockpit providing the highest granularity of the most important KPIs as part of the dashboarding best practices.</p><h3>Suggested steps to build a dashboard (i.e. TL;DR)</h3><ol><li>Discovery</li><li>Mockup</li><li>Implementation</li><li>Review</li><li>Iteration</li></ol><h3>Step 1 — Discovery</h3><p>The objective of the Discovery phase is to figure out what the stakeholders really need instead of taking their requests literally. The data team as a business unit must act as a data consultant during this phase, to guide the stakeholder to determine the real business value to be created.</p><p>Deliverables:</p><ul><li>Documented analytical questions (just 1–3 would do, e.g. “How effective were the marketing campaigns on the overall orders?”, “How is the new store performing compared to the existing store(s)?”</li><li>Agreed KPIs/Metrics to measure the answers to the analytical questions</li><li>Agreed commonly used Dimensions (e.g. t-dimension like Daily &amp; Weekly &amp; Monthly &amp; Quarterly, Country, City, Postal Code, Marketing Channel, Stores, etc.)</li></ul><p>In addition, it would be very nice if there are specific existing ad-hoc dashboards or analyses done previously to get some notion about what KPIs &amp; Dimensions were used. It would also be a great help for the Implementation phase for data quality checks later.</p><p>Usually, just a simple Task ticket with 2–3 story points (i.e. half to a full working day) would cover the Discovery phase.</p><h3>Step 2 — Mockup</h3><p>Based on what was found during the Discovery Phase, the data team should build a Mockup to showcase what kind of data visualization to expect from the stakeholders.</p><p>The primary objective is to make sure what is in the data team’s mind matches stakeholders’ expectations, and going visual is the fastest and most effective way to do it. It might sound counterintuitive, but it is <strong>strongly suggested to avoid directly jumping into dashboard building on Tableau, Looker, or whatever data visualization tool is being employed</strong>. it is always faster to make sure the spec is right before actually building something.</p><p>Deliverables:</p><ul><li>A mockup to demonstrate a rough idea of how the dashboard would look like</li></ul><p>A dashboard mockup could be drawn on paper like a sketch, or a proper Mock-up tool like <a href="https://moqups.com/">Moqups</a> used by designers. The key point of this stage is to give the stakeholders a more visual notion to know what to expect early on.</p><h3>Step 3 - Implementation</h3><p>The Implementation phase covers 2 components: building the data source backing up the dashboard, and building the dashboard by using the BI tool (e.g. Tableau).</p><h4><strong>3a. Data Source building with SQL</strong></h4><p>The best practice for dashboarding with Data Warehouse is usually incorporating Version Control System (e.g. Git) and SQL, with <strong>a particular schema solely for reporting purposes</strong>. I’ve seen companies naming it as the “reporting layer”, the key is to pre-compute and centralize the dashboarding logic.</p><p>Deliverables:</p><ul><li>A Table or a View for sole reporting purpose</li></ul><p>Please note that <strong>reporting tables are NOT Summary tables</strong>. Summary tables are pre-aggregated tables based on 1 or many <a href="https://en.wikipedia.org/wiki/Fact_table">Fact tables</a> on an agreed granularity (usually commonly used Dimensions, e.g. Daily + country, etc.). Reporting tables would sometimes have “weird” columns for visualization purposes, and if those tables are used as Summary tables would confuse users of the table without enough domain &amp; visualization knowledge.</p><h4><strong>3b. Data Visualization building with BI tool</strong></h4><p>This part is only focusing on the visual element being presented in the dashboard. As prior agreed during the Mockup phase, the data team would actually get their hands dirty and publish their dashboard in a “staging”/”QA” folder in the BI tool (Tableau) for stakeholders to review if the delivered dashboard meets the expectation as the v1.0.</p><p>Deliverables:</p><ul><li>A Tableau Workbook (or any equivalent) published in a “staging”/”QA” folder, which is clearly not in production yet but still visible to the stakeholders</li></ul><h3>Step 4 — Review</h3><p>Depending on the number of stakeholders, there are various ways for feedback collection.</p><p>For a small size of stakeholder group, it is okay to just do so with a quick meeting; If there are many stakeholders, it is suggested to have an organized way to collect feedback (e.g. Google Sheets). The duration Review phase varies, depending on the number of stakeholders: for 1–2 stakeholders, it usually takes only a few working days; for 3+ stakeholders, it is better to spare 1–2 weeks.</p><p>The objective of the Review phase is to iron out what is actually doable with the BI tool, what would take extra effort, and what are acceptable trade-offs with the stakeholders.</p><p>Deliverable:</p><ul><li>Fine-tuning the dashboard, if the feedback from the stakeholders are low-hanging fruit</li><li>Alignment with the primary stakeholder(s) about expectations</li><li>Follow up Tickets (Task or Bug) for the next phase, and/or Bug tickets if there are issues found from the dependent Fact, Dimension, or Summary tables.</li></ul><h3>Step 5 — Iteration</h3><p>At this phase, the v1.0 of the dashboard should be mostly done. Stakeholders would be spending time playing around with the dashboard and actually using it for business decision-making. If they are engaged enough, there will be questions coming in regarding the dashboard.</p><p>Depending on the context and the stakeholder engagement level, the dashboard builder(s) could consider sending out a public email explaining what is the primary objective of the dashboard, hosting a presentation session of showcasing how to use the dashboard to generate actionable insights quickly, what are the priorities, etc. Documentations regarding the dashboard ought to be done (Castor in this context) as well.</p><h4><strong>Potential followups</strong></h4><ul><li>Study the use cases of the primary stakeholders of the dashboard, gain inspiration for ideas for building Summary tables</li><li>Analyze the traffic to the dashboard (e.g. Tableau allows owners of the dashboard to do so), understand which tab is popular and which isn’t, and try to figure out why for further refinement of the dashboard</li><li>If there are demands on analytical visibility on a certain sub-domain (e.g. Warehouse Operations under the big umbrella of Operation), it would be a great idea to launch another specific dashboard solely for that sub-domain</li></ul><h3>Final Remarks</h3><p>There are too many dashboards being built by analysts or business stakeholders without careful consideration, due to a lack of process. A lot of KPIs and dimensions are being reported to executives, yet the reliability and trustworthiness are suspicious since there wasn’t any prep work done in prior dashboard building. What happens very often next is the dashboard builder would go back and investigate what is missing, usually after being called out by another executive in a management meeting. On some occasions, the dashboard builder would start the blame game and start pawning it as “data quality issues” and let the data team be the easy scapegoat.</p><p>Hence, regardless of the time pressure or other constraint, analysts or any dashboard builder ought to follow processes and best practices when it comes to dashboarding. Only by doing so, potential issues can be spotted early on and allow enough buffer for proper solutions.</p><h3>Further Readings</h3><ul><li><a href="https://www.goodreads.com/book/show/26535513-storytelling-with-data">Storytelling with Data: A Data Visualization Guide for Business Professionals</a></li><li><a href="https://www.goodreads.com/book/show/31396339-the-big-book-of-dashboards">The Big Book of Dashboards: Visualizing Your Data Using Real-World Business Scenarios</a></li><li><a href="https://www.goodreads.com/book/show/22213262-key-performance-indicators-for-dummies">Key Performance Indicators For Dummies</a></li><li><a href="https://www.goodreads.com/book/show/27038049-good-charts">Good Charts: The HBR Guide to Making Smarter, More Persuasive Data Visualizations</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78a0be930ef0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-panda/dashboarding-sop-standard-operating-procedure-how-to-build-a-dashboard-properly-78a0be930ef0">Dashboarding SOP (Standard Operating Procedure) — How to build a dashboard, properly</a> was originally published in <a href="https://medium.com/data-panda">Data Panda</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>