<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kelly Burdine on Medium]]></title>
        <description><![CDATA[Stories by Kelly Burdine on Medium]]></description>
        <link>https://medium.com/@kelly.burdine?source=rss-a9bd2a68266c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xcmcV7tzyr7Ky85c</url>
            <title>Stories by Kelly Burdine on Medium</title>
            <link>https://medium.com/@kelly.burdine?source=rss-a9bd2a68266c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:46:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kelly.burdine/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The 80/20 rule in data (and why it doesn’t have to be that way)]]></title>
            <link>https://medium.com/ublend-co/the-80-20-rule-in-data-and-why-it-doesnt-have-to-be-that-way-7659f8456d8f?source=rss-a9bd2a68266c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7659f8456d8f</guid>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[data-storage]]></category>
            <category><![CDATA[data-infrastructure]]></category>
            <category><![CDATA[fivetran]]></category>
            <dc:creator><![CDATA[Kelly Burdine]]></dc:creator>
            <pubDate>Mon, 08 Mar 2021 17:27:38 GMT</pubDate>
            <atom:updated>2021-03-08T17:27:38.499Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have ever worked as a data scientist/analyst, you have probably heard the 80/20 rule: you spend 80% of the time cleaning and prepping data and 20% of your time actually doing analysis and delivering insights.</p><p>For many years I lived in this world and just took the rule as a fact. Data was messy, schema changes break everything, and reporting that was accurate today might be wrong tomorrow. Although my team and I continually found ways to make incremental improvements through people and process change, it still felt like we were stuck in a hole and couldn’t dig ourselves out.</p><p>This is when I realized <strong>the 80/20 rule was actually just an excuse that covered up a problem</strong>. This way of doing analytics wasn’t efficient and data wasn’t an asset: it was a cost to the organization.</p><p>I researched our problems and talked to others in the industry, and discovered they weren’t unique to me and my team, and luckily there were a lot of people and organizations trying to solve them. The root cause of our issues was the data infrastructure we had built was not suited for modern analytics:we built too many custom solutions for problems that weren’t unique and didn’t require custom solutions. These required constant maintenance and the costs “saved” by not buying an out of the box solution quickly dissolved.</p><p><strong>Hiring more people or creating more process changes were just temporary band-aid solutions</strong>. We started implementing more modern tools like dbt and Looker (we were already using Redshift). Things were starting to improve, but the transition felt slow. After talking to the wonderful team at <a href="https://www.fishtownanalytics.com/">Fishtown Analytics</a>, I realized we were continuing to make a critical error when implementing those tools: we carried over too much of our existing “custom” infrastructure and design into our new tools. Just like putting lipstick on a pig doesn’t change the fact it is still a pig; integrating bad infrastructure, process, and design into new tools won’t change the outcome. You will still have data problems and the 80/20 rules still applies.</p><p>Around the same time I had this revelation I also had the opportunity to join <a href="https://www.aula.education/">Aula</a> as their first data hire. I decided to make the move, and that’s where things started changing.</p><h3><strong>Building out data infrastructure as a team of one</strong></h3><p>My first task was building out data infrastructure that could effectively and efficiently support analytics within <a href="https://www.aula.education/how-it-works">the Aula product</a> and for internal decision making.</p><p>I consider myself to have a high technical aptitude, but I don’t come from an engineering background. The task of building out infrastructure by myself seemed a bit daunting; however, it ended up being a blessing in disguise. I relied heavily on modern data tools and followed industry best practices for setup and design.</p><p>Despite the fact I already had a hunch this was the way to go, I didn’t realize the scale of the impact. The results were surprising even to me.</p><p><strong>In</strong> <strong>just 3 months, my mighty data team of one </strong>was able to:</p><ul><li>Stand up a data warehouse in Snowflake with separate databases for raw data and production data including the automation of branch schemas for testing</li><li>Create 50+ separate ingestion pipelines using Stitch and Fivetran</li><li>Implement dbt and created 600+ models, 2000+ tests, and 50+ snapshots</li><li>Implement Metabase using AWS ECS and Docker</li></ul><p>I ended up with a mostly automated data ingestion pipeline that I rarely have to think about. I have a transformation workflow that has a ton of test coverage and follows CI/CD so I don’t hesitate pushing changes to production. I can quickly and easily upgrade all my tools when new releases come out, because I didn’t build a custom solution that conflicts with newer versions. My company can visualize and analyze data coming from more than 50+ sources using a point-and-click tool.</p><p>The cost? It varies between $1500-$2000 per month (note: I don’t have petabytes of data at this point).</p><h3><strong>The data stack at Aula</strong></h3><p>This is what my data stack looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*l_zr5WKzmLDWJWZ8" /></figure><p>I use a combination of Fivetran and Stitch to load all of our source data into Snowflake. Although I would prefer to have one load tool, neither option came with connector options for all of our sources. With that being said, both are really easy to set up and maintain (hit me up if you ever want to discuss the differences).</p><p>I load all of the data into “raw” databases based on the loader. Then I use dbt to clean and transform the data and dump it into a production ready database. From there we use Metabase, a very lightweight point-and-click BI tool, to visualize and analyze our data. In addition, we can easily connect to the production analytics database from a Jupyter Notebook for more advanced analytics.</p><p>This is a pretty common modern data stack, and although there are areas that I can improve, it has been very easy to maintain and build upon. <strong>This gives me the time to focus on actually using data to make an impact at Aula</strong>. I’m no longer spending 80% (or more) of my time just prepping and cleaning data.</p><h3><strong>Take a step back and re-evaluate your data infrastructure</strong></h3><p>Just like leveraging credit can allow you to build wealth faster, leveraging modern data tools can allow you to derive insights from data much faster, because you aren’t spending all your time fixing and cleaning. I highly encourage anyone who is dealing with constant data problems such as constantly changing source schemas, messy or incomplete data, transformation changes that cause downstream reporting inaccuracies, stale data, etc to take a step back and evaluate your data infrastructure.</p><p>Here are some questions to help you evaluate:</p><ul><li><em>Are you using tools to solve problems they were meant to solve or are you stretching their limitations? (Don’t be the person who has a hammer so everything looks like a nail)</em></li><li><em>Do you have a particular tool or process that is your Achilles’ heel and always seems to be a source of problems?</em></li><li><em>Are you taking advantage of the latest technology in data? (If someone already created a solution why recreate the wheel?)</em></li><li><em>Are there any tasks you are doing over and over? Is there a tool that already exists that can help you easily automate without requiring tons of maintenance?</em></li><li><em>If you suddenly had 10x more data tomorrow would your infrastructure easily scale?</em></li></ul><p>I want to point out that I had some distinct advantages when I built out a modern data stack at Aula.</p><p>First, I have an awesome leadership team that gave me the <strong>autonomy to make these decisions and even make mistakes</strong> in order to learn, iterate, and build something great.</p><p>Second, I was <strong>starting from scratch</strong>. There was almost no data infrastructure to begin with, so even though I did need to provide some reporting throughout the process, it didn’t feel like changing out the engines of a plane flying at 30,000ft (I really believe replacing infrastructure faster and upfront is worth the investment, but that is a topic for another post).</p><p>Third, <strong>this wasn’t my first rodeo</strong>. I was the first data analytics hire at my last company too, and I made enough mistakes to know how to do things better the second time around. (<em>Pro tip: hire someone who has built a data stack before if you want to upgrade your data infrastructure!</em>)</p><p>Here are some links to blogs and articles that helped me along the way:</p><ul><li><a href="https://docs.getdbt.com/docs/guides/best-practices">Dbt Best practices</a></li><li><a href="https://discourse.getdbt.com/t/how-we-structure-our-dbt-projects/355">How we structure our dbt projects</a></li><li><a href="https://blog.getdbt.com/how-we-configure-snowflake/">How we configure Snowflake</a></li><li><a href="https://blog.getdbt.com/is-dbt-the-right-tool-for-my-data-transformations/">Is dbt the right tool for my data transformations?</a></li><li><a href="https://retina.ai/blog/dataops-principles">DataOps Principles: How startups do data the right way</a></li><li><a href="https://locallyoptimistic.com/post/building-a-data-practice/">Building a data practice from scratch</a></li><li><a href="https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/">Engineers shouldn’t write ETL</a></li></ul><h3>👋 Join us</h3><p>If you want to join a data team with a modern data stack and supportive leadership team, please view my open roles <a href="https://aulaeducation.recruitee.com/">here</a> or hit me up on <a href="https://www.linkedin.com/in/kellyburdine/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7659f8456d8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ublend-co/the-80-20-rule-in-data-and-why-it-doesnt-have-to-be-that-way-7659f8456d8f">The 80/20 rule in data (and why it doesn’t have to be that way)</a> was originally published in <a href="https://medium.com/ublend-co">Aula</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>