<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Laeticia Dos Santos (letdos) on Medium]]></title>
        <description><![CDATA[Stories by Laeticia Dos Santos (letdos) on Medium]]></description>
        <link>https://medium.com/@laeticia2santos?source=rss-6f71113a9af3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NYbpqU-rbLKwf2qAdd6S2w@2x.jpeg</url>
            <title>Stories by Laeticia Dos Santos (letdos) on Medium</title>
            <link>https://medium.com/@laeticia2santos?source=rss-6f71113a9af3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 23:28:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@laeticia2santos/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ 10 Metrics Every Data Pipeline Must Track — From Hard Lessons in Production]]></title>
            <link>https://medium.com/@laeticia2santos/10-metrics-every-data-pipeline-must-track-from-hard-lessons-in-production-627db70070eb?source=rss-6f71113a9af3------2</link>
            <guid isPermaLink="false">https://medium.com/p/627db70070eb</guid>
            <dc:creator><![CDATA[Laeticia Dos Santos (letdos)]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 12:45:38 GMT</pubDate>
            <atom:updated>2025-06-24T12:45:38.850Z</atom:updated>
            <content:encoded><![CDATA[<h3>📌 10 Metrics Every Data Pipeline Must Track — From Hard Lessons in Production</h3><h3>A practical guide for junior data engineers, analytics engineers, and DataOps teams</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ur4BktFdQDnXUV_piz7bJg.png" /></figure><p>A few months ago, I got a call at 3 AM.</p><p>Our main data pipeline had been silently failing for 6 hours, and nobody noticed — until the VP of Sales asked why his dashboard was showing yesterday’s data.</p><p>It was a brutal wake-up call: <strong>no matter how elegant your data architecture is, if you don’t know when something breaks, it’s worthless</strong>.</p><p>After several years working with data pipelines (and making plenty of mistakes), I’ve learned that the difference between a junior and senior data engineer isn’t just the code you write — <strong>it’s the metrics you monitor</strong>.</p><p>Here are the 10 most important metrics you should implement from day one.</p><h3>✅ 1. Data Availability</h3><p><strong>What it tracks:</strong> The percentage of time your data is available when users need it.</p><p>This sounds obvious, but you’d be surprised how many teams don’t measure it properly. It’s not just about jobs finishing without errors — it’s about data being accessible and delivered <strong>on time</strong>.</p><p><strong>Target:</strong> ≥ 99% availability<br> <em>(When I started, I celebrated hitting 95%)</em></p><p><strong>How to measure it:</strong></p><ul><li>Log every successful execution with timestamps</li><li>Set alerts when a job is delayed more than X minutes</li><li>In Airflow, leverage native SLA monitoring</li></ul><h3>✅ 2. End-to-End Latency</h3><p><strong>What it measures:</strong> How long it takes from data generation to data availability.</p><p>This metric has saved me more than once. We once had a “working” pipeline that took 8 hours to process data users needed in 30 minutes. The problem wasn’t data quality — it was performance.</p><p><strong>Targets that work:</strong></p><ul><li>Streaming: &lt; 15 minutes</li><li>Batch: &lt; 6 hours <em>(business-dependent)</em></li></ul><p><strong>My trick:</strong> Add timestamps at every critical point and store them in a monitoring table. Build dashboards to find bottlenecks quickly.</p><h3>✅ 3. Job Success Rate</h3><p><strong>What it tracks:</strong> The percentage of job executions that finish successfully.</p><p>Some teams think 90% is fine “because something always fails.” That’s a mistake. If your pipeline fails more than 2% of the time, you have an architecture problem — not a bad-luck problem.</p><p><strong>Target:</strong> ≥ 98% success rate</p><p><strong>Pro tip:</strong> Distinguish between <strong>data errors</strong> (which may be expected) and <strong>technical errors</strong> (which shouldn’t happen).</p><h3>✅ 4. Data Catalog Coverage</h3><p><strong>What it tracks:</strong> The percentage of tables that are documented and discoverable.</p><p>I learned this the hard way when I joined a company and spent 3 days trying to find the “active users” table. If the pipeline builder can’t find the data, imagine the analysts.</p><p><strong>Goal:</strong> ≥ 90% documentation for critical tables</p><p><strong>Tools that work:</strong></p><ul><li><a href="https://datahubproject.io/">DataHub</a></li><li>dbt docs</li><li>A good README.md (small teams)</li></ul><h3>✅ 5. Data Quality Score</h3><p><strong>What it measures:</strong> Percentage of records that pass validation (correct types, no nulls, no duplicates).</p><p>We once had a “successful” pipeline running for weeks, but 40% of the data had null fields after a silent API change. Nobody noticed until reporting went sideways.</p><p><strong>Target:</strong> ≥ 95% data quality</p><p><strong>How to get started:</strong></p><ul><li>Use dbt tests or <a href="https://greatexpectations.io/">Great Expectations</a></li><li>Start small: NOT NULL, UNIQUE, and type checks on key fields</li></ul><h3>✅ 6. Data Volume by Domain</h3><p><strong>What it measures:</strong> How much data is processed per business domain (e.g., sales, product, marketing).</p><p>This metric helps you understand <strong>growth trends</strong>, justify <strong>costs</strong>, and <strong>prioritize optimization</strong>. Plus, managers love it.</p><p><strong>Frequency:</strong> At least monthly</p><p><strong>How:</strong> Use row counts, storage usage, or ingestion metrics per domain.</p><h3>✅ 7. Data Lineage &amp; Traceability</h3><p><strong>What it measures:</strong> Whether you can trace where data came from and where it goes.</p><p>The first time someone asked me “what breaks if we change this API?” — and I had no idea — I realized how critical lineage is.</p><p><strong>Goal:</strong> 100% visibility for critical flows</p><p><strong>Where to start:</strong></p><ul><li>Document the top 10 flows manually</li><li>Use tools like <a href="https://openlineage.io/">OpenLineage</a>, dbt, or Microsoft Purview as you scale</li></ul><h3>✅ 8. Cost per Pipeline</h3><p><strong>What it measures:</strong> How much each pipeline costs (compute, storage, licensing).</p><p>This is what makes you a <strong>strategic</strong> data engineer. It’s not enough to make things work — you need to make them efficient.</p><p><strong>Real example:</strong> We reduced a pipeline’s run frequency from hourly to every 4 hours, saving 75% in compute.</p><p><strong>Track costs by:</strong></p><ul><li>Adding cloud tags per pipeline/domain</li><li>Using <a href="https://aws.amazon.com/aws-cost-management/">AWS Cost Explorer</a>, <a href="https://learn.microsoft.com/en-us/azure/cost-management-billing/">Azure Cost Management</a>, or equivalent</li></ul><h3>✅ 9. Mean Time to Resolution (MTTR)</h3><p><strong>What it tracks:</strong> How long it takes to detect and fix problems.</p><p><strong>Personal target:</strong> &lt; 2 hours for critical workflows</p><p><strong>Trick that works:</strong> Keep a <strong>playbook</strong> of common errors and fixes. Your future self (and your 3 AM self) will thank you.</p><h3>✅ 10. Number of Active Consumers</h3><p><strong>What it measures:</strong> Who is actually using your data.</p><p>This one hurts. I’ve seen massive tables taking hours to process — and no one had queried them in months.</p><p><strong>My rule:</strong> If a critical table doesn’t have <strong>at least 1 verified consumer per week</strong>, it’s time to investigate.</p><p><strong>Bonus tip:</strong> Use query logs from BigQuery, Snowflake, or your BI tool to track usage.</p><h3>🎯 Final Thoughts</h3><p>If you’re starting out, don’t try to implement all 10 metrics immediately.</p><p>Start with:</p><ul><li>✅ Data Availability</li><li>✅ Job Success Rate</li><li>✅ Data Quality</li></ul><p>If you’re leading a team, <strong>make these metrics part of your data culture</strong>. They’re not “nice to have” — they’re essential for observability, reliability, and trust.</p><blockquote>Metrics don’t just keep your pipeline healthy — they keep your team sane.</blockquote><h3>🧰 Resources I Actually Use</h3><ul><li><a href="https://docs.getdbt.com/docs/build/data-tests">dbt Testing Guide</a></li><li><a href="https://greatexpectations.io/">Great Expectations</a></li><li><a href="https://openlineage.io/">OpenLineage</a></li></ul><p>💬 <strong>What other metrics do you monitor in your pipelines?</strong> I’d love to hear your experience in the comments.</p><p>🧠 <strong>Follow me</strong> for more practical articles on data engineering, observability, and platform architecture.</p><p>#DataEngineering #Observability #DataOps #PipelineMonitoring #BigData #dbt #CloudData</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=627db70070eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>