<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saqib Khan on Medium]]></title>
        <description><![CDATA[Stories by Saqib Khan on Medium]]></description>
        <link>https://medium.com/@saqibk0510?source=rss-76ff414ceb69------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ektTdSPGR90QAeiRx-qy4g.jpeg</url>
            <title>Stories by Saqib Khan on Medium</title>
            <link>https://medium.com/@saqibk0510?source=rss-76ff414ceb69------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:59:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saqibk0510/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating data quality checks in your Data Integration workflow]]></title>
            <link>https://blog.dataengineerthings.org/automating-data-quality-checks-in-your-data-integration-workflow-4ed389c03e46?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ed389c03e46</guid>
            <category><![CDATA[great-expectations]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 02:20:02 GMT</pubDate>
            <atom:updated>2026-09-22T02:20:02.207Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Great Expectations to Add Automated Data Quality Checks to ETL Pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u7Su2xZRhGnX9PAV.jpeg" /></figure><p>We’ve all been there. You log into Airflow, scroll to your pipeline, and breathe a sigh of relief. Green checkmarks across the board. The source extraction finished, the load query ran, the transformations wrapped up cleanly, and the orchestration tool is practically high-fiving you.</p><p>Everything succeeded. Except for one small detail: the data inside is completely wrong.</p><p>This is the central paradox of modern data engineering: <strong>a pipeline run can be 100% successful and still load total garbage.</strong></p><p>Suppose your source system usually pushes five million records every morning. Today, an upstream API silently throttles, sending only 800,000. Nothing crashes. The extraction loop runs, the Snowflake load succeeds, and the target table updates. Technically, every single query executed without an error. But functionally, you just loaded a massive data outage straight into production.</p><p>The green checkmark didn’t lie about the pipeline; it lied about the <em>data</em>. And unless you’re actively testing the payload itself, your downstream stakeholders will be the ones to break the news to you.</p><h3>The Slow Death of Manual SQL Validation</h3><p>In the early days of a data platform, validation is deceptively easy. An analyst or engineer runs a handful of manual SQL checks after a load:</p><pre>SELECT COUNT(*) FROM target_table WHERE customer_id IS NULL;<br>SELECT MAX(transaction_date) FROM target_table;</pre><p>When you have three pipelines and five core tables, this feels fine. But manual validation doesn’t scale linearly — it breaks exponentially. Once you’re managing dozens of DAGs and hundreds of tables updating multiple times a day, nobody is sitting around manually running spot-checks.</p><p>Worse, manual checks happen <em>after</em> the pipeline has already finished. That means bad data is already sitting in front of executive dashboards, feature stores, and reverse-ETL triggers before anyone notices.</p><p>Consider the classic failure loop:</p><blockquote>Source → ETL Load → Target Table → Power BI</blockquote><p>If only 20% of your transactional volume lands in the warehouse, Power BI will refresh without throwing an error. Hours later, a VP opens a dashboard, notices revenue cratered 80% overnight, and fires off a panicky Slack message. Now, three different teams drop everything to run forensic tracebacks.</p><p>The fix isn’t “better monitoring after the load.” The fix is treating data validation as a core pipeline gate, not an afterthought.</p><h3>What’s Actually Worth Testing?</h3><p>When engineers start thinking about data quality, they often overcomplicate it. In practice, 90% of your production incidents can be caught with shockingly basic checks.</p><p>They generally fall into two categories:</p><p><strong>1. Content Expectations (The Data Itself)</strong></p><ul><li>Primary keys must never be NULL.</li><li>Unique IDs must actually be unique.</li><li>Numeric values (like price or age) shouldn’t fall outside logical boundaries.</li><li>Categorical fields should strictly match allowed enums (e.g., [&#39;PENDING&#39;, &#39;APPROVED&#39;, &#39;REJECTED&#39;]).</li></ul><p><strong>2. Context Expectations (The Load Metadata)</strong></p><ul><li>Total record count for the batch shouldn’t drop by more than $X\%$ compared to the 7-day rolling average.</li><li>Source count must match target count.</li><li>The max timestamp in the batch must align with the current processing window.</li></ul><p>None of these checks are ground-breaking. But if you enforce them programmatically, you eliminate the vast majority of silent data bugs before they contaminate downstream systems.</p><h3>Hand-Rolled SQL vs. Purpose-Built Frameworks</h3><p>You can write all of these checks in raw SQL using custom scripts or dbt tests. In fact, for small projects, hand-rolled SQL is totally fine.</p><p>The pain shows up six months down the road. Without a shared framework, every engineer writes validation scripts slightly differently. You end up babysitting a scattered museum of bespoke SQL queries, custom wrapper scripts, and ad-hoc alerting logic.</p><p>This maintenance nightmare is why open-source data quality frameworks like <strong>Great Expectations (GX)</strong> exist.</p><p>Created in 2017, Great Expectations borrows a foundational concept from software development: unit testing. Just as application software engineers write unit tests to confirm code behavior, GX lets you write declarative tests (called <strong>Expectations</strong>) to confirm data behavior.</p><p>Instead of hunting for NULL values with raw SQL, you declare your intent explicitly in Python:</p><pre>gx.expectations.ExpectColumnValuesToNotBeNull(<br>    column=&quot;TRANSACTION_ID&quot;<br>)</pre><p>Or, to make sure a transaction amount is never negative:</p><pre>gx.expectations.ExpectColumnValuesToBeBetween(<br>    column=&quot;TRANSACTION_AMOUNT&quot;,<br>    min_value=0<br>)</pre><p>The real superpower here isn’t that GX performs magic you couldn’t write in SQL. The superpower is <strong>software engineering rigor</strong>. Your data quality rules live as version-controlled code. They can be code-reviewed in pull requests, tested locally, and deployed cleanly through your standard CI/CD pipeline.</p><h3>Building a Production Quality Gate with GX</h3><p>Here is how you actually wire this into a real workflow.</p><h4>1. Define the Suite</h4><p>In GX, individual rules are bundled into an <strong>Expectation Suite</strong>.</p><pre>import great_expectations as gx<br><br>context = gx.get_context()<br>suite = context.suites.add(<br>    gx.ExpectationSuite(name=&quot;customer_transaction_quality&quot;)<br>)<br><br># Add assertion rules<br>suite.add_expectation(<br>    gx.expectations.ExpectColumnValuesToNotBeNull(column=&quot;TRANSACTION_ID&quot;)<br>)<br>suite.add_expectation(<br>    gx.expectations.ExpectColumnValuesToBeUnique(column=&quot;TRANSACTION_ID&quot;)<br>)<br>suite.add_expectation(<br>    gx.expectations.ExpectColumnValuesToBeBetween(column=&quot;TRANSACTION_AMOUNT&quot;, min_value=0)<br>)<br>suite.add_expectation(<br>    gx.expectations.ExpectTableRowCountToBeBetween(min_value=1)<br>)</pre><h4>2. Bind It to an Execution Checkpoint</h4><p>GX uses a <strong>Checkpoint</strong> to evaluate the data against your suite:</p><pre>validation_definition = context.validation_definitions.add(<br>    gx.ValidationDefinition(<br>        name=&quot;customer_transaction_validation&quot;,<br>        data=batch_definition,<br>        suite=suite<br>    )<br>)<br><br>checkpoint = context.checkpoints.add(<br>    gx.Checkpoint(<br>        name=&quot;customer_transaction_checkpoint&quot;,<br>        validation_definitions=[validation_definition]<br>    )<br>)<br><br># Run the validation<br>result = checkpoint.run()<br><br>if not result.success:<br>    raise ValueError(&quot;Data validation failed! Halting pipeline execution.&quot;)</pre><h4>3. Enforce the Gate in Airflow</h4><p>Instead of running validation after the entire DAG finishes, place it directly between your transformation and publication tasks:</p><blockquote>extract_data → load_snowflake → transform_data → validate_data → publish_data</blockquote><p>In an Airflow DAG using the TaskFlow API:</p><pre>from airflow.decorators import task<br>import great_expectations as gx<br><br>@task<br>def validate_transformed_data():<br>    context = gx.get_context()<br>    checkpoint = context.checkpoints.get(&quot;customer_transaction_checkpoint&quot;)<br>    result = checkpoint.run()<br>    <br>    if not result.success:<br>        # Failing the task prevents downstream dependencies from running<br>        raise ValueError(&quot;Quality check failed. Publishing aborted.&quot;)<br><br># DAG Dependencies<br>transform_data() &gt;&gt; validate_transformed_data() &gt;&gt; publish_to_bi()</pre><p>If upstream data drops from 5 million records to 600,000, your volume Expectation triggers a failure. Airflow halts the DAG <em>before</em> publish_to_bi ever executes. The pipeline fails loudly, on-call engineers get alerted, and the bad data stays safely isolated.</p><h3>The Reality Check: Where Frameworks Fail</h3><p>Great Expectations is fantastic, but it’s not a silver bullet. If you deploy it blindly, you’ll trade one set of problems for another:</p><ul><li><strong>Expectation Sprawl:</strong> Someone has to write and maintain these rules. If you have 500 tables with 10 expectations each, that’s 5,000 rules to maintain. When business logic changes, stale expectations will cause false-positive pipeline alerts that destroy team trust.</li><li><strong>Warehouse Compute Costs:</strong> Running heavy validation suites over massive tables (e.g., full table scans for uniqueness on multi-billion row tables) will skyrocket your Snowflake or BigQuery bill. <strong>Rule of thumb:</strong> Only run heavy validations on the incremental partition or current batch, not the historical dataset.</li><li><strong>Known Unknowns vs. Unknown Unknowns:</strong> GX requires you to know what to test for in advance. It’s great for <em>“customer ID shouldn’t be NULL.”</em> It’s useless for unexpected, subtle anomalies you didn’t anticipate. For fuzzy, baseline-driven anomaly detection, pair GX with a dedicated data observability tool (like Monte Carlo or Databand).</li></ul><h3>Stop Trusting Green Checkmarks</h3><p>Data integration isn’t just about moving bytes from Point A to Point B — it’s about protecting the integrity of the decisions made using those bytes.</p><p>The shift is simple:</p><blockquote>Source → Transform → Load → Success</blockquote><p>…becomes:</p><blockquote>Source → Transform → Load → Validate → Pass? → Continue / Stop</blockquote><p>The next time your orchestration tool turns green, don’t celebrate just yet. Ask the question that actually matters: <strong>Did we load what we thought we loaded?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ed389c03e46" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/automating-data-quality-checks-in-your-data-integration-workflow-4ed389c03e46">Automating data quality checks in your Data Integration workflow</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Light Weight Self -Service Data Lineage for Data Engineers]]></title>
            <link>https://blog.dataengineerthings.org/light-weight-self-service-data-lineage-for-data-engineers-5102a919f1c7?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/5102a919f1c7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Thu, 07 May 2026 19:14:22 GMT</pubDate>
            <atom:updated>2026-05-07T19:14:22.529Z</atom:updated>
            <content:encoded><![CDATA[<h4>When platform frameworks take months, engineers need answers today!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rBY9mLj1-DNh2bHkw3yuVw.png" /></figure><p>In any form of data platform implementations, most of the times, production issues usually starts with something small.</p><p>A user drops a message saying the numbers don’t look right. Nothing is broken. The pipeline didn’t fail. Dashboards are loading fine. But something feels off enough that it needs attention.</p><p>You open the dashboard, compare it with yesterday’s numbers, and now you see it too. Slight deviation. Not dramatic, but definitely wrong.</p><p>So, you do what any data engineer would do. You trace it back to the source.</p><p>Within a few minutes, you land on the table:</p><pre>FINANCE.FACT_REVENUE</pre><p>And there it is. Bad data.</p><p>At this point, the situation still feels under control. Fix the problem, rerun the job, and move on.</p><p>Like any sane data engineer, you quickly check downstream impact and that’s when you realize… <strong>this is going to take longer than expected :(</strong></p><h3>The real problem begins here</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*OakOqzZ7GE64iFgF6kjNcw.jpeg" /></figure><p>By the time this issue was discovered, the data loads already moved on.</p><p>The fact table has loaded. Aggregations have completed. Downstream extracts are generated. Dashboards are already refreshed.</p><p>In short, the bad data didn’t fail. It propagated.</p><p>Now the question in front of you is not “what went wrong.”</p><p>It’s this:</p><blockquote><em>If you fix the source, what exactly do you rerun next?</em></blockquote><p>And more importantly:</p><blockquote><em>What can you safely skip?</em></blockquote><h3>What usually happens next</h3><p>You open your workload scheduler.</p><p>The workflow looks familiar, but tracing exact dependencies in your head is not as straightforward as it seemed when you built it. Some paths are obvious. Others are not.</p><p>You switch to backend and look at query history.</p><pre>SELECT *<br>FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY<br>WHERE QUERY_TEXT ILIKE &#39;%FACT_REVENUE%&#39;<br>ORDER BY START_TIME DESC;</pre><p>It gives you activity, but not clarity.</p><p>You open your data integration application next, scrolling through transformations and trying to understand which tables were actually used versus what was just referenced along the way. The logic isn’t always obvious, and the more you trace it, the more it feels like you’re reconstructing the pipeline from memory rather than reading something definitive. Around this point, someone suggests rerunning everything. It’s a safe option, and in the middle of a production issue, that’s usually enough to convince everyone but you also know it means pulling in jobs that may not be related, increasing the chances of side effects, and turning a targeted fix into a much larger recovery effort.</p><h3>Pause here for a second</h3><p>Before jumping into solutions, it helps to pause for a moment and simplify what you’re actually trying to solve. In situations like this, it’s easy to start thinking about full lineage, column-level tracing, or even building a proper data catalog, because those are the “complete” solutions everyone talks about. But none of that is what you need right now. In the middle of a production issue, you’re not trying to model the entire data platform — you’re just trying to answer one very specific question:</p><blockquote>Given a broken dataset, which downstream datasets and jobs were impacted in this batch?</blockquote><p>That’s it.</p><h3>Now think about your pipelines</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3GGpijEE3LBRN-k9llSFw.gif" /></figure><p>Every job already knows something important:</p><ul><li>what it reads</li><li>what it writes</li></ul><p>That relationship is a good enough lineage to start with.</p><p>The only issue is that it’s not captured anywhere in a structured way.</p><p>So instead of waiting for a platform solution, you can start capturing it yourself.</p><h3>Step 1: create a simple lineage table</h3><p>Start with the bare minimum. No need to overdesign this.</p><pre>CREATE TABLE ops_lineage_event (<br>    event_time TIMESTAMP,<br>    job_name STRING,<br>    input_dataset STRING,<br>    output_dataset STRING,<br>    batch_date DATE,<br>    status STRING<br>);</pre><p>You just need enough to answer dependency questions.</p><h3>Step 2: log lineage from pipelines</h3><p>Now, add a small logging step in your pipelines.</p><p>In Airflow, after a task finishes:</p><pre>def log_lineage(job, input_ds, output_ds, batch_date):<br>    sql = f&quot;&quot;&quot;<br>    INSERT INTO ops_lineage_event<br>    VALUES (<br>        CURRENT_TIMESTAMP,<br>        &#39;{job}&#39;,<br>        &#39;{input_ds}&#39;,<br>        &#39;{output_ds}&#39;,<br>        &#39;{batch_date}&#39;,<br>        &#39;SUCCESS&#39;<br>    )<br>    &quot;&quot;&quot;<br>    execute_sql(sql)</pre><p>Use it right after your transformation:</p><pre>log_lineage(<br>    job=&quot;fact_revenue_load&quot;,<br>    input_ds=&quot;stg_revenue&quot;,<br>    output_ds=&quot;fact_revenue&quot;,<br>    batch_date=&quot;2025-01-05&quot;<br>)</pre><p>In Databricks, do the same at the end of a notebook:</p><pre>spark.sql(&quot;&quot;&quot;<br>INSERT INTO ops_lineage_event<br>SELECT current_timestamp(),<br>       &#39;agg_revenue_job&#39;,<br>       &#39;fact_revenue&#39;,<br>       &#39;agg_revenue_daily&#39;,<br>       &#39;2025-01-05&#39;,<br>       &#39;SUCCESS&#39;<br>&quot;&quot;&quot;)</pre><p>This takes minutes to add, but it changes how you debug issues.</p><h3>Step 3: use it during an incident</h3><p>Now replay the same scenario.</p><p>You detect bad data in: <strong>fact_revenue</strong></p><p>Instead of digging through multiple tools, start with simple backend query like this:</p><pre>SELECT output_dataset<br>FROM ops_lineage_event<br>WHERE input_dataset = &#39;fact_revenue&#39;<br>  AND batch_date = &#39;2025-01-05&#39;;</pre><p>This gives you immediate downstream datasets:</p><pre>agg_revenue_daily<br>pbi_revenue_summary<br>cfo_extract</pre><p>Then <strong>repeat the process to expand further</strong>.</p><p>Within seconds, you have the full impact chain.</p><h3>Step 4: apply a simple traversal</h3><p>At this point, the problem becomes systematic.</p><p>You can think of it like this:</p><pre>INPUT: broken_dataset<br><br>1. start with the broken dataset<br>2. find all datasets that depend on it<br>3. repeat until no new datasets are found<br>4. map datasets back to jobs<br>5. rerun only those jobs</pre><p>Pseudo code:</p><pre>queue = [broken_dataset]<br>impacted = []<br><br>while queue:<br>    current = queue.pop()<br>    downstream = get_outputs(current)<br><br>    for ds in downstream:<br>        if ds not in impacted:<br>            impacted.append(ds)<br>            queue.append(ds)</pre><h3>What this changes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*i3IhIIkFAlZx1lF85k20kw.jpeg" /></figure><p>Instead of guessing, you now have a clear path forward. You fix the source issue, rerun only the part of the pipeline that was actually impacted, and leave everything else untouched. There’s no second-guessing, no unnecessary reruns, and no feeling that you might have missed something along the way.</p><h3>What this is (and isn’t)</h3><p>This is not enterprise lineage.</p><p>It won’t give you column-level traceability. It won’t generate pretty graphs. It won’t satisfy governance teams.</p><p>But it solves the problem you actually face during production issues.</p><p>And it does it without waiting for platform teams, approvals, or new tools.</p><h3>Final thought</h3><p>Most teams wait for the “right” solution.</p><p>In the meantime, every production issue turns into a manual investigation.</p><p>A simple lineage table and a few logging steps won’t make your platform perfect.</p><p>But the next time someone says the numbers look off, you won’t be opening five different tools trying to piece things together.</p><p>You’ll run one query, see the impact, and move forward with confidence.</p><p>And that alone is worth it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5102a919f1c7" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/light-weight-self-service-data-lineage-for-data-engineers-5102a919f1c7">Light Weight Self -Service Data Lineage for Data Engineers</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Day I Realized Corporate Success Is a Trap]]></title>
            <link>https://medium.com/agileinsider/the-day-i-realized-corporate-success-is-a-trap-16bcd8c53107?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/16bcd8c53107</guid>
            <category><![CDATA[life]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[self-awareness]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[personal-growth]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 05:48:55 GMT</pubDate>
            <atom:updated>2026-03-23T12:04:37.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BpOHK7r41NoKTw20T76CKQ.jpeg" /><figcaption>The corporate success ladder</figcaption></figure><p>Not long time ago, I stumbled upon this quote that didn’t stay with me at the time, but somehow found its way back when I needed it the most:</p><blockquote><em>“</em><strong><em>The trouble is, you think you have time</em></strong><em>”</em> — Jack Kornfield</blockquote><p>Back then, it felt distant, unrelatable and almost philosophical in a way that didn’t apply to my everyday life. It mostly hit me as if it belonged to people who had already lived through something I hadn’t yet experienced.</p><p>But <strong>some ideas don’t leave you</strong>; they just wait quietly until you are ready to see them clearly.</p><p>The day this realization hit me wasn’t in any way dramatic or disruptive, and may be that’s why felt so real.</p><p>It was just another weekday, the kind that passes without leaving any visible mark, where everything feels routine and predictable, and yet something underneath begins to shift.</p><p>I remember logging in that morning and looking at my calendar, completely filled with meetings, each one carrying a sense of urgency that I had long stopped questioning.</p><p>By late morning, I hopped onto another call, and within minutes, it felt familiar in a way that was hard to ignore. The same structured conversation, the same phrases repeated with some variations, the same polite agreement without real clarity.</p><p>And while all of this was happening, I noticed my attention drifting, not out of disinterest, but because a part of me seemed to step back and observe the whole thing from a distance.</p><p>That is when the thought appeared, quietly but firmly:</p><blockquote><strong>If I keep doing this for the next ten years, what exactly changes?</strong></blockquote><p>That question stayed with me longer than I expected, and over the next few weeks, it started connecting with moments I had previously ignored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fA3kOwSA_15UvK9Qc6P-w.jpeg" /></figure><p>Another instance that I could recall is from a late evening, well past regular working hours, and I was still at my desk trying to resolve a data issue in production. A critical pipeline had failed, downstream reports were blocked, and suddenly something that no one had noticed during the day had become urgent enough to demand immediate attention.</p><p>There were messages coming in from different directions — business teams asking for timelines, leadership wanting status updates, data team trying to trace the root cause and in the middle of all this chaos, I was switching between logs, queries, and dashboards, trying to make sense of what went wrong, <em>WHY </em>and how quickly it could be fixed, even if that meant putting a temporary patch to stabilize things for time being.</p><p>After a couple of hours of digging, we identified the issue — a minor change that the upstream had quietly pushed through and passed our validations but broke a key assumption in the pipeline.</p><p>We fixed it, reran the jobs, validated the data, and by the time everything stabilized, it was late in the night.</p><p>I remember sitting there for a moment after it was done, staring at the screen, not feeling relieved or accomplished, just…….. <em>EMPTY…..</em></p><p>Deep down I knew what would happen next.</p><ul><li>The system would move on</li><li>The reports would refresh</li><li>People would stop asking questions</li></ul><p>And within a few days, no one would remember this incident at all, not that I want to or expect others to do so. Except maybe the people who had to stay back and fix it. That moment stayed with me, because it revealed something I hadn’t fully acknowledged before.</p><p>Not that the work wasn’t important — it was.</p><p>Not that the problem didn’t need immediate attention — it did.</p><p><strong>But the <em>cycle</em> itself never changed</strong>.</p><p>There was always another issue, another deadline, another escalation waiting just around the corner. And the sense of importance that came with each situation faded almost as quickly as it appeared.</p><p>A few days later, I found myself walking past a senior leader’s office, someone who had reached the level that many people, including me at one point, would consider <strong><em>the destination</em></strong><em> </em>(from a corporate lens).</p><p>He was sitting there, looking at his screen, completely still. No urgency, no visible pressure. Just a quiet, settled kind of exhaustion. And for some reason, that image stayed with me longer than anything else. Because it didn’t feel like an exception. It felt like a glimpse into where this path naturally leads if you continue walking it without questioning it.</p><p><strong>That’s when all the dots started to connect.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*YUod3N6BoZQEzLnoNv5eWQ.jpeg" /></figure><p>The meetings, the late nights, the constant sense of urgency, the temporary nature of most problems, and the endless cycle of chasing the next milestone — they weren’t isolated experiences. They were part of a system that keeps you moving continuously, while <strong>rarely giving you the space to ask whether the direction itself makes sense</strong>.</p><p>Corporate life is very effective at creating the feeling of progress. However, what <strong>most people don’t realize is that they are slowly getting used to a life they never consciously chose</strong>.</p><p>A new title, a higher salary, a larger scope, each step signals that you are moving forward. But if you look closely, the underlying experience often remains the same, just with more complexity and higher expectations layered on top of it. And without realizing, you soon start to associate this motion with meaning.</p><p>What unsettled me the most was how naturally I had accepted this pattern, without ever seriously questioning it. There was always this quiet assumption that things would feel more complete at the next level, that clarity and satisfaction were just one step ahead. But the more I observed, the more I realized that this sense of arrival keeps shifting forward, never fully materializing.</p><p>I didn’t walk away from my job after this realization, and nothing dramatic changed on the outside. I still work, still take responsibility, still show up the same way. But internally, something has shifted in a way that is difficult to reverse. I no longer see corporate success as something that defines the quality of my life.</p><p><strong>I see it as one part of it — important, but limited</strong>.</p><p>And this subtle change in mindset has made a big difference, not in what I do on the outside, but in how I experience it. The same work feels lighter, the same pressures feel less personal, and for the first time in a long while, I don’t feel like everything in my life depends on how things go at work.</p><p>I strongly believe that the trap is not the corporate world itself. It is how quietly it takes over your life, until one day you’re looking at something that resembles success on the outside, but doesn’t feel like yours on the inside.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16bcd8c53107" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agileinsider/the-day-i-realized-corporate-success-is-a-trap-16bcd8c53107">The Day I Realized Corporate Success Is a Trap</a> was originally published in <a href="https://medium.com/agileinsider">Agile Insider</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Your Snowflake Warehouse Becomes a Moving Target]]></title>
            <link>https://blog.dataengineerthings.org/when-your-snowflake-warehouse-becomes-a-moving-target-2923abd89789?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/2923abd89789</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 02:39:03 GMT</pubDate>
            <atom:updated>2026-02-21T02:39:03.564Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0xf8HxOWZ3_cxVtGLQMpg.jpeg" /></figure><p>A few months ago, a friend of mine who works on a data platform team told me about a strange production issue they had run into.</p><p>Their dashboards started failing without any code change and for no obvious reason. As usual, it took some firefighting just to rule things out and narrow down the actual root cause.</p><p>Here is what their initial debugging checklist looked like:</p><ul><li>No deployments</li><li>No schema changes</li><li>No reporting infrastructure updates</li></ul><p>Just red refresh icons in Power BI and confused business users asking,<br> <em>“This was working fine yesterday… what changed?”</em></p><p>The problem turned out to be far less visible but surprisingly common in modern data platforms.</p><p>In their case, the immediate cause was a suspended Snowflake warehouse.<br>But as they dug deeper, they realized that wasn’t the only way dashboards were breaking.</p><p>In other incidents, access to the warehouse had been revoked from reporting roles. In few cases, a new warehouse had been created and the old one quietly retired.</p><p>Different technical actions with same business impact — reports stopped refreshing.</p><p>That conversation got me thinking about how compute governance is handled in Snowflake environments, and how easily small operational decisions can ripple into production incidents. In this post, I want to share some lessons and best practices around warehouse lifecycle management that might help teams avoid similar surprises.</p><h3>The Invisible Dependency Most Teams Miss</h3><p>As they described what happened, it became clear the issue wasn’t really about one warehouse or one configuration change. It was about how compute was being treated overall.</p><p>From the platform team’s perspective, warehouses were simply infrastructure objects. They could be resized, replaced, retired, or reorganized as needed to control cost and maintain hygiene.</p><p>From the reporting team’s perspective, those same warehouses behaved more like service endpoints. Dashboards, refresh schedules, and user queries depended on them being stable and available.</p><p>That mismatch created a silent dependency.</p><p>And when that dependency broke, it showed up not as a platform change, but as a business outage.</p><h3>Why This Happens So Often</h3><p>Snowflake makes compute extremely easy to manage. You can spin warehouses up, suspend them, resize them, and reorganize them with just a few commands.</p><p>That flexibility is powerful but it also hides the fact that warehouses become embedded into workflows over time.</p><ul><li>BI tools cache connections</li><li>Refresh jobs expect predictable endpoints</li><li>Analysts reuse saved sessions</li></ul><p>None of these behaviors are visible when looking at warehouses purely as infrastructure.</p><p>So when a warehouse is suspended, renamed, or replaced, the change feels harmless at the platform level but disruptive everywhere else.</p><h3>The Pattern Behind the Incidents</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*pIZAwKhVlCpg5rZgbySd2A.jpeg" /></figure><p>When they looked back at previous outages, a pattern emerged.</p><p>Sometimes a warehouse had been suspended to save cost.<br>Sometimes access had been revoked during role cleanup.<br>Sometimes compute had been reorganized to align with new data structures.</p><p>Each action made sense individually.</p><p>But none of them accounted for the workloads depending on that compute.</p><p>The real issue wasn’t the action , it was the missing contract.</p><h3>Warehouses Are Contracts</h3><p>Once the team reframed the problem, the real issue became obvious.</p><p>Warehouses weren’t just compute clusters. They were service contracts between platform teams and workload owners.</p><p>Each warehouse implicitly promised something:</p><ul><li>predictable performance</li><li>consistent availability</li><li>stable connection behavior</li><li>known cost characteristics</li></ul><p>When that promise changed unexpectedly, workloads started breaking even when the underlying data itself hadn’t changed.</p><p>The solution wasn’t to lock warehouses down or stop optimizing them. It was to define them based on <strong><em>workload intent </em></strong>and not schema ownership.</p><p>Instead of creating warehouses per database, schema or team, they started organizing compute around how the system was actually used.</p><h3>ETL and Batch Processing</h3><p>Large ingestion jobs and transformations tend to run on schedules and process high volumes of data. These workloads benefit from predictable throughput rather than low latency.</p><p>For this type of processing, the best practice is to provision a dedicated warehouse tier sized for batch loads. It can be configured with aggressive auto-suspend settings and scaled up temporarily during heavy processing windows.</p><p>This keeps ingestion jobs isolated from user queries and prevents reporting workloads from being slowed down by large data movements.</p><h3>BI Refresh Workloads</h3><p>Scheduled dashboard refreshes behave very differently from ETL jobs. They often fire at fixed intervals, run many medium-sized queries in bursts, and need predictable completion times to meet business SLAs.</p><p>Because of this, mature Snowflake environments typically assign BI refreshes to their own warehouse tier. This warehouse is sized for concurrency rather than raw processing power and uses slightly longer auto-suspend windows to avoid constant start–stop cycles.</p><p>Separating refresh compute from ingestion prevents refresh failures during heavy data loads and gives reporting teams a stable endpoint to depend on.</p><h3>Interactive Analytics</h3><p>Analysts running ad-hoc queries have yet another usage pattern. Their queries are unpredictable, often exploratory, and sensitive to latency.</p><p>For this workload, a medium-sized interactive warehouse works best. It can auto-suspend quickly to control cost while still providing responsive performance when users connect.</p><p>This prevents ad-hoc exploration from competing with scheduled pipelines or dashboard refreshes.</p><h3>Data Science and Experimentation</h3><p>Model training, experimentation, and sandbox exploration tend to be bursty and resource-intensive. These workloads benefit from flexible, scalable compute that can grow temporarily and then shut down.</p><p>Providing a separate warehouse tier for data science keeps experimental workloads from accidentally consuming resources intended for production pipelines or dashboards.</p><h3>Why This Separation Matters</h3><p>Organizing warehouses this way does more than improving performance. It clarifies responsibility.</p><p>When a warehouse represents a workload class rather than a schema, everyone understands its purpose.</p><p>Platform teams can optimize size, suspend behavior, and cost knowing what depends on it.<br>BI teams know which compute endpoint their dashboards should use.<br>Data engineers know where pipelines belong.</p><p>Most importantly, changes become predictable.</p><p>Retiring a warehouse no longer feels like removing a random piece of infrastructure. It feels like deprecating a service that naturally triggers communication and monitoring.</p><p>That shift from infrastructure thinking to service thinking is what turns warehouses from fragile resources into reliable contracts.</p><p>And in modern data platforms, those contracts are often the difference between stable reporting and unexplained outages.</p><h3>What Changes After This Realization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IiDxRyHW4TpqQKGyUFcbpw.jpeg" /></figure><p>Once warehouses are treated as workload contracts rather than infrastructure objects, a few things start to change.</p><p>Warehouses get aligned to workload classes — ingestion, transformation, BI refresh, interactive analysis — instead of being tied to schemas, databases, or teams. This makes their purpose clear and predictable.</p><p>BI teams are given a stable compute tier designed specifically for reporting workloads, so dashboards have a consistent endpoint to depend on.</p><p>Platform teams still retain the flexibility to resize, optimize, and tune compute behind the scenes. The difference is that these changes now happen within a defined structure instead of through ad-hoc adjustments.</p><p>Most importantly, warehouse changes begin to follow a lifecycle process rather than happening abruptly.</p><p>Before retiring compute, usage is monitored.<br>Dependencies are identified.<br>Stakeholders are notified.</p><p>What once felt like random outages gradually turns into controlled transitions.</p><h3>The Broader Lesson</h3><p>Most data platform failures aren’t caused by broken pipelines or bad SQL.</p><p>They are caused by invisible contracts between teams.</p><p>Compute resources sit at the center of those contracts and yet they are often managed with the least coordination.</p><p>If your dashboards sometimes feel like they break “for no reason” the issue might not be in your data models at all.</p><p>It might be in how the compute layer is governed.</p><p>Hopefully this perspective provides some practical guidance on how to think about Snowflake warehouse governance and lifecycle management. Many of these lessons only surface after something breaks, and most teams end up learning them the hard way. If you have run into similar issues or solved them differently I would genuinely be interested in hearing how you approached compute governance. Feel free to share your experiences or suggestions.</p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2923abd89789" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/when-your-snowflake-warehouse-becomes-a-moving-target-2923abd89789">When Your Snowflake Warehouse Becomes a Moving Target</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Impact Analysis in IBM DataStage: A Simple Python-Based Approach]]></title>
            <link>https://medium.com/@saqibk0510/impact-analysis-in-ibm-datastage-a-simple-python-based-approach-e0e7f86f6f4c?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0e7f86f6f4c</guid>
            <category><![CDATA[utilities]]></category>
            <category><![CDATA[data-lineage]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[datastage]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 08:12:59 GMT</pubDate>
            <atom:updated>2026-02-10T08:12:59.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NuYoZH8hz3RY4pOL" /><figcaption>Photo by <a href="https://unsplash.com/@uxindo?utm_source=medium&amp;utm_medium=referral">UX Indonesia</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a Data Engineer, while working on IBM DataStage projects, I am pretty sure that you would have likely run into this situation when;</p><ul><li>A table changes OR</li><li>A file layout is updated OR</li><li>A parameter value needs to be modified OR</li><li>Need to figure out which job sent out email with specific subject or filename to users OR</li></ul><p>Could be anything else for which you need to quickly find out list of jobs/objects that contain a specific string, for impact analysis.</p><p>In real enterprise environments, answering this question is rarely straightforward. Projects often contain hundreds or thousands of jobs, routines, and parameter sets. Documentation is outdated, lineage tools are incomplete (or at times broken for your particular use case) and manual inspection is both time consuming and error prone.</p><h3>The practical reality</h3><p>In many teams, impact analysis still comes down to:</p><ul><li>Manually opening jobs in Designer</li><li>Searching keywords in exported DSX files</li><li>Relying on tribal knowledge</li></ul><p>What teams usually need is not perfect lineage, but a <strong>fast, reliable way to narrow down the impact list</strong> of a change.</p><h3>Treating DSX as structured text</h3><p>DataStage already gives us something useful: <strong>DSX exports</strong>.</p><p>DSX files contain the full definitions of DataStage jobs, parameter sets, and routines in a structured text format. By scanning these files directly, we can answer a very practical question:</p><blockquote>Which DataStage objects reference this table, file, or string?</blockquote><p>This doesn’t require connecting to internal repositories or building complex metadata models. It’s simple string-based impact analysis that has to be applied consistently.</p><h3>A lightweight Python utility</h3><p>To make this easier, I put together a basic but effective Python script that:</p><ul><li>Takes an exported DSX file as input</li><li>Searches for a given string (table name, file name, parameter, etc.)</li><li>Identifies DataStage objects where the string appears</li><li>Writes the results to a clean, readable output file</li></ul><p>The script is intentionally minimal:</p><ul><li>No external dependencies</li><li>No proprietary repository access</li><li>Easy to read and modify</li></ul><p>It mirrors how impact analysis is actually performed in many real-world DataStage environments.</p><h3>How it works — high level understanding</h3><p>The script follows a very simple flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3fdQSSlFX74UnHNA3XyCXw.png" /></figure><p>There is no attempt to infer lineage or semantics. The goal is purely practical: <strong>quick, repeatable impact analysis</strong>.</p><h3>Usage:</h3><pre>python ds_impact_analyzer.py &lt;dsx_file&gt; &lt;search_string&gt;</pre><p>For example:</p><pre>python ds_impact_analyzer.py project_export.dsx CUSTOMER_TABLE</pre><p>This produces a text file in the same directory as the script, containing:</p><ul><li>The search string</li><li>The input DSX file name</li><li>A count of impacted objects</li><li>A list of object names where the string was found</li></ul><p>It is meant to be something you can run quickly, get an answer, and move on.</p><h3>Example output</h3><p>The script generates a simple text report like this:</p><pre>datetime: 2026-02-10 15:01:33<br>search string = CUSTOMER_TABLE<br>input file: project_export.dsx<br>total match count: 4<br>------------------------<br>below is the list of objects where the search string was found<br>1. object name: LOAD_CUSTOMER_DIM<br>2. object name: UPDATE_CUSTOMER_FACT<br>3. object name: PS_CUSTOMER_TABLES<br>4. object name: ROUTINE_SQL_HELPER</pre><p>You can find the code here:</p><p><strong>Github Repo</strong>: <a href="https://github.com/LogicBuilder45/ibm-datastage-utility/tree/main/find-where-used">ibm-datastage-utility/find-where-used at main · LogicBuilder45/ibm-datastage-utility</a></p><p>I hope this small, practical script helps save valuable time during impact analysis. If you run into any issues or have ideas to enhance it, feel free to reach out. I’m happy to discuss and collaborate further.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0e7f86f6f4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Data Architecture — Best Practice]]></title>
            <link>https://blog.dataengineerthings.org/cloud-data-architecture-best-practice-0de01015d8eb?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/0de01015d8eb</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Fri, 30 Jan 2026 16:12:12 GMT</pubDate>
            <atom:updated>2026-01-30T16:12:12.288Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cloud Data Architecture — Best Practice</h3><h4>With more and more cloud adoption and tech modernization, here is a clear and crisp (non-nonsense) direction for designing a robust cloud platform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12nPEhLoL5G875hqusrtGQ.png" /><figcaption>Cloud Data Architecture</figcaption></figure><h3>Cloud Data Architecture: Getting the Basics Right Before You Scale</h3><p>Over the last decade, cloud adoption has gone from optional to unavoidable. Most organizations today are either already on the cloud or actively trying to get there. According to multiple industry reports, over 90 percent of enterprises now run at least part of their workloads in the cloud, and data platforms are usually among the first systems to move.</p><p>This shift is not just about cost or infrastructure. It is driven by the need to modernize legacy systems, support faster analytics, integrate new data sources, and respond to business questions in near real time. Old architectures were not built for this pace.</p><p>And that is exactly why you cannot afford to fall behind in this cloud race.</p><h3>Why Cloud Familiarity Is No Longer Optional</h3><p>Cloud platforms are no longer niche skills. They are becoming table stakes.</p><p>Even if your role is not deeply technical, you are expected to understand how data flows, where it lives, and how decisions are made using cloud-based systems. For engineers and architects, the expectation is even higher. You are expected to design systems that scale, stay cost-efficient, and do not fall apart under real workloads.</p><p>Staying relevant today means understanding how cloud data platforms are designed, not just which buttons to click.</p><h3>Cloud Vendor Landscape: Big Names and New Players</h3><p>Most people think of cloud in terms of the big three. AWS, Azure, and Google Cloud dominate the market and power a large share of enterprise workloads. Alongside them, data-focused platforms like Snowflake, Databricks, BigQuery, and Redshift play a central role in analytics and warehousing.</p><p>At the same time, new players continue to emerge. Some focus on real-time analytics, some on AI-native platforms, and others on simplifying operations. The tooling will keep changing.</p><p>What matters more than the vendor is understanding the underlying concepts. If you get those right, switching platforms becomes far easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*5TchngydekfjV8Ik2TAoFw.png" /><figcaption>Cloud concepts</figcaption></figure><h3>Cloud Concepts You Should Understand Regardless of Vendor</h3><p><strong>Cloud deployment models</strong></p><ul><li>Public cloud uses shared infrastructure managed by a provider</li><li>Private cloud is dedicated and often chosen for control or compliance</li><li>Hybrid cloud combines on-prem systems with cloud services</li><li>Multi-cloud means using more than one cloud provider</li></ul><p><strong>Service models</strong></p><ul><li>Infrastructure as a Service provides compute, storage, and networking</li><li>Platform as a Service abstracts infrastructure so teams focus on building</li><li>Software as a Service delivers ready-to-use applications</li></ul><p><strong>Execution and pricing models</strong></p><ul><li>Serverless computing removes server management and scales automatically</li><li>Virtual machines is the closest model to traditional infrastructure, where you provision a VM and manage OS, and scaling yourself.</li><li>Managed VM is where the cloud provider manages some aspects like patching and availability.</li><li>Resource-based pricing vs usage-based pricing:<br>Some services charge for provisioned capacity. Others charge per query, execution, or request.</li></ul><p><strong>Application architecture fundamentals</strong></p><ul><li>Monolithic systems bundle all functionality together</li><li>Microservices split systems into smaller, independent components</li><li>Event-driven<strong> s</strong>ystems react to events instead of running on schedules.</li><li>Containerization using Docker packages applications consistently</li><li>Orchestration platforms like Kubernetes manage containers at scale</li></ul><p><strong>Core building blocks</strong></p><ul><li>Compute runs workloads</li><li>Storage holds data</li><li>Networking connects everything securely and efficiently</li><li>Identity and access management (IAM) controls who can access what.</li><li>Observability deals with logging, monitoring, and alerting.</li></ul><h3>The Designing Phase: Spending time where it actually matters</h3><p>The biggest mistake teams make with cloud data platforms is rushing into implementation. Tools get selected quickly. Pipelines get built quickly. Architecture decisions get locked in even faster. And then teams spend years working around those decisions.</p><p>Good design starts before any service is provisioned.</p><p>The first thing to clarify is scope. What problems are you trying to solve right now, and what problems are explicitly out of scope? Not every dataset needs to be onboarded in the first phase. Not every use case needs real-time data. Being clear about this upfront helps avoid unnecessary complexity and cost.</p><p>If you are modernizing an existing system, spend time understanding how it works today. Look beyond documentation. Identify where it breaks, where it slows down, and where manual intervention is common. Long-running batch jobs, fragile dependencies, and unclear ownership are signals that should influence your cloud design. Cloud should simplify these areas, not replicate them.</p><p>For greenfield platforms, design should start from the business view of data, not from schemas or tables. Identify high-level subject areas such as customers, transactions, products, or usage. This becomes your conceptual data model. It helps align technical teams and business stakeholders before any physical design begins.</p><p>Another key decision is how data will arrive and leave the platform. Batch files, APIs, streaming events, and third-party feeds all have different characteristics. Even if you do not implement streaming on day one, the architecture should not block it later. The same applies to outbound data sharing, whether for BI tools, downstream systems, or external partners.</p><p>Layering also needs deliberate thought. Patterns like bronze, silver, and gold help separate raw data from refined and consumption-ready data. But the goal is not to follow a pattern blindly. The goal is to create clear boundaries so ingestion stays simple, transformations stay traceable, and consumption stays fast.</p><p>Finally, a good design defines what success looks like. Cost should be predictable. Data should be easy to find and trust. New pipelines should be quick to add. Security and lineage should be built in, not bolted on. Scalability should come from design choices, not emergency fixes.</p><p>Time spent here often feels slow. In reality, it saves months of rework later.</p><h3><strong>Design Checklist Before You Build</strong></h3><p>Before provisioning services or writing pipelines, pause and validate the basics.</p><ul><li>Have the business use cases been clearly defined and prioritized</li><li>Is the scope for phase one explicitly agreed upon</li><li>Do you understand how the current system behaves, not just how it is documented</li><li>Are high-level subject areas identified and agreed on with stakeholders</li><li>Is there clarity on how data will be ingested, both batch and near real-time</li><li>Have outbound data consumers been identified, including BI tools and partner systems</li><li>Is there a clear layering strategy for raw, refined, and consumption-ready data</li><li>Are security, access control, and data ownership defined upfront</li><li>Is cost behavior understood for the chosen services</li><li>Can the architecture scale without major redesign</li></ul><h3>Additional Design Considerations</h3><p>Modern data platforms rarely deal only with batch data. Streaming data, APIs, and event-based ingestion are increasingly common. Your architecture should account for this early, even if you do not implement it immediately.</p><p>Outbound data sharing is another important consideration. Data does not just flow inward anymore. It flows to BI tools, partner systems, and sometimes external consumers.</p><p>A good architecture also has measurable qualities. Cost efficiency. Reasonable turnaround time for new data. Support for analytical reporting. Clear data lineage. Strong security controls. Modular design that allows change without rewrites. Scalability without re-architecture.</p><p>If these are not considered upfront, they become painful retrofits later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*7CuIoTDOvxC71o0XvZ9hiQ.jpeg" /><figcaption>Tips and Tricks</figcaption></figure><h3>Pro Tips from the Field</h3><p>Performance issues almost always show up at the consumption layer first. I have seen platforms where the data pipelines were technically sound, but dashboards still felt slow. In many cases, the problem was not the data warehouse itself, but where users were accessing it from. Keeping services closer to end users, choosing the right region, and avoiding unnecessary cross-region traffic made a noticeable difference in query response times without changing any code.</p><p>Cost control is another area where small decisions add up quickly. One common example is leaving default settings untouched. Teams often spin up warehouses, clusters, or compute pools sized for peak usage and forget about them. In reality, workloads are rarely constant. Committing to reserved capacity for predictable jobs and scaling down aggressively for ad-hoc or development workloads can reduce costs significantly without impacting performance.</p><p>Monitoring also matters more than most teams expect. I have seen platforms where pipelines ran fine for weeks, then slowly degraded because of data growth or upstream changes. Without basic alerts on runtime, data volume, or failure patterns, issues were noticed only after users complained. Simple monitoring on job duration, row counts, and cost trends helped teams catch problems early and avoid firefighting.</p><p>Operational discipline is another differentiator. In one setup, every pipeline failure triggered a generic alert. Over time, people stopped paying attention. When alerts were tuned to reflect severity and ownership, response times improved immediately. Not every failure needs escalation, but every failure needs clarity on who owns it.</p><p>Security and lineage often feel like things to “add later.” That usually backfires. I have seen teams struggle to answer simple questions like where a number came from or who can see a sensitive column. Platforms that defined access patterns, data ownership, and lineage expectations early avoided these conversations altogether. Trust in data improved because answers were easy to find.</p><h3>Common Pitfalls to Avoid</h3><ul><li><strong>Lifting and shifting legacy designs without questioning assumptions</strong><br>Many teams move existing on-prem architectures to the cloud as-is. This often carries over rigid scheduling, over-provisioned resources, and tight coupling that the cloud does not require. The result is higher cost and limited flexibility.</li><li><strong>Over-engineering too early</strong><br>Designing for every possible future use case leads to unnecessary complexity. Features like real-time processing, complex orchestration, or multi-region setups should be introduced when there is a clear need, not upfront.</li><li><strong>Ignoring cost behavior during design</strong><br>Cloud platforms make scaling easy, but cost grows just as easily. Failing to understand how compute, storage, and data movement are priced often leads to surprises once usage increases.</li><li><strong>Treating security and governance as an afterthought</strong><br>Access control, data masking, and ownership should be defined early. Retrofitting security later is harder and usually more disruptive.</li><li><strong>Not planning for data growth</strong><br> Systems that perform well on day one can degrade quietly as data volume increases. Ignoring partitioning, clustering, or retention strategies often leads to slow queries and rising costs.</li><li><strong>Lack of clear ownership and accountability</strong><br> When everyone can access the data, but no one owns it, issues linger. Each dataset and pipeline should have a clear owner responsible for quality and communication.</li></ul><h3>Final Thoughts</h3><p>Cloud data architecture is not about picking the right cloud service or following a popular diagram. It is about making a series of small, deliberate decisions that hold up once real data, real users, and real pressure show up.</p><p>Most problems I have seen in cloud platforms were not caused by technology limits. They were caused by rushed design, unclear ownership, or assumptions carried over from legacy systems without being questioned. The cloud simply makes those problems visible faster.</p><p>If you get the fundamentals right, cloud platforms give you a lot of room to grow. If you get them wrong, you end up spending more time fixing issues than delivering value. Spending time on design, scope, and trade-offs early may feel slow, but it pays off quickly once the platform is in use.</p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0de01015d8eb" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/cloud-data-architecture-best-practice-0de01015d8eb">Cloud Data Architecture — Best Practice</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unix Shell Scripting essentials]]></title>
            <link>https://blog.dataengineerthings.org/unix-shell-scripting-essentials-d412a02d5a0e?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/d412a02d5a0e</guid>
            <category><![CDATA[unix]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[shell-script]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 19:25:11 GMT</pubDate>
            <atom:updated>2026-02-09T14:50:44.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*WE2vJf6T9IGRK3UsQgx0dA.jpeg" /></figure><p>Yes, I decided to write about Unix shell scripting in the age of cloud and AI. Not because I am trying to prove it is still relevant, or because I am nostalgic about old school tech. I am writing this because it is still very much alive and actively used in production environments at major organizations, quietly running behind the scenes.</p><p>Even today, with all the modern tooling around us like Kubernetes, managed cloud services, AI platforms, and orchestration layers, a lot of real work still comes down to a few lines in a script. Moving files, triggering jobs, parsing logs, stitching systems together, automating the boring but critical stuff.</p><p>As I understand things on the data engineering side of it, cloud didn’t kill shell scripting but it scaled the need for it. And speaking of AI, it as well didn’t replace it either but made it easier for people to write scripts without understanding what they’re doing, kind of vibe coding, which is exactly why understanding still matters if you want to be more than just a coder.</p><h3>How I Got Into It (Early 2000s)</h3><p>I started learning Unix shell scripting back in the early 2000s when you didn’t have detailed YouTube tutorials walking you through every command. Stack Overflow wasn’t the first page on your browser. It was mostly trial and error with a lot of debugging.</p><p>I still remember sitting with a terminal open, trying to understand why a command worked one day and failed the next. I didn’t even know the right words to search. I would read small pieces of documentation, try something, break something else, and then try again. Half the learning was just figuring out what the problem even was. But to be frank, once it clicked, it felt powerful and fulfilling.</p><p>To give you some backstory on how I got hooked into this — It was in one of my early projects where we had a lot of manual, repetitive tasks that everyone treated as normal day-to-day task. It involved tasks like moving files between servers, searching logs for error codes, cleaning up directories etc. Team members were doing it manually, step-by-step and were content with how things were getting done. I was new and had the zeal to showcase that I can meaningfully contribute to the team so took up the task of automating these manual steps. I wrote, what I think of it now as a not-so-sophisticated shell scripts to reduce the amount of manual work involved.</p><p>Those scripts did something for me beyond just “automation”, it gave me confidence and attention from senior members on the team. I got appreciation and for someone early in their career, that kind of recognition is a spark to do more and that was my kick to go deeper into advanced concepts of shell scripting.</p><h3>Why I am Writing This</h3><p>Over the time, I noticed that a lot of people especially data engineers in early stages of their career are curious about shell scripting but they hesitate. Some think it is too “old school” , some think it is only for Linux admins while others get overwhelmed by the syntax.</p><p>So I want to share this in a way that is simple, practical, and not intimidating. Just the essentials that actually help you in real work even if your day job is into cloud, data, AI, or anything else.</p><h3>Shell Scripting Explained</h3><p>A shell script is basically a <strong>text file</strong> <strong>that contains commands . T</strong>he same commands you type in the terminal but written in the order or execution, so the computer can run them for you.</p><p>Let’s do some hands-on to make things clear.</p><h4>Your First Script</h4><p>Open a text editor and create a file named “hello.sh”</p><p>Inside it, type:</p><pre>#!/bin/bash<br>echo &quot;Hello from shell scripting&quot;</pre><p>Now run:</p><pre>chmod +x hello.sh<br>./hello.sh</pre><p>You just executed a program you wrote.</p><p>That first line (#!/bin/bash) tells the system to run the commands using bash command line interpreter.</p><p>“chmod +x” is to set our script as executable.</p><h4>Variables</h4><p>Variables are placeholders for values that you want to store and then retrieve later in your program execution.</p><pre>#!/bin/bash<br>name=&quot;Aman&quot;<br>echo &quot;Hello, $name&quot;</pre><p>Run it and you’ll see your message — “Hello, Aman”</p><h4>Taking User Input</h4><pre>#!/bin/bash<br>echo &quot;What is your name?&quot;<br>read name<br>echo &quot;Nice to meet you, $name&quot;</pre><p>We just made our script interactive.</p><h4>Decision Making (If Else)</h4><pre>#!/bin/bash<br>echo &quot;Enter a number:&quot;<br>read n</pre><pre>if [ $n -gt 10 ]; then<br>  echo &quot;Number entered is greater than 10&quot;<br>else<br>  echo &quot;Number entered is smaller or equal to 10&quot;<br>fi</pre><h4>Looping</h4><pre>#!/bin/bash<br>for i in 1 2 3 4 5<br>do<br>  echo &quot;Count: $i&quot;<br>done</pre><p>Loops along with if-then-else blocks are powerful tools in scripting that will help you implement complex tasks in a very structured and easy to follow symantec.</p><h4>Let’s build something real</h4><p>A simple script that scans a directory and reports what it finds. This script takes one input parameter (folder/directory path)</p><p>Let’s create and name our script as “folder_report.sh”</p><pre>#!/bin/bash<br># This script generates a quick report about a folder:<br># - Total number of files<br># - Total number of directories<br># - Top 5 largest files inside it<br><br># The folder path is taken from the first argument<br>folder=$1<br><br># Check if the user provided a folder path. If not, provide usage details and exit<br>if [ -z &quot;$folder&quot; ]; then<br>  echo &quot;Usage: ./folder_report.sh &lt;folder_path&gt;&quot;<br>  exit 1<br>fi<br><br># Check if the given path actually exists and is a directory<br>if [ ! -d &quot;$folder&quot; ]; then<br>  echo &quot;Error: &#39;$folder&#39; is not a valid directory&quot;<br>  exit 1<br>fi<br><br># Print the folder being scanned<br>echo &quot;Scanning folder: $folder&quot;<br>echo &quot;------------------------&quot;<br><br># Count total files inside the folder (including subfolders)<br># Pipe &#39;|&#39; is used to redirect output of one command as inpurt to the next one<br>file_count=$(find &quot;$folder&quot; -type f | wc -l)<br><br># Count total directories inside the folder (including subfolders)<br>dir_count=$(find &quot;$folder&quot; -type d | wc -l)<br><br># Display summary report<br>echo &quot;Total files: $file_count&quot;<br>echo &quot;Total directories: $dir_count&quot;<br><br>echo &quot;&quot;<br><br># Show the largest files in the folder<br>echo &quot;Top 5 largest files:&quot;<br><br># 1. find all files in given folder<br># 2. list them with file sizes (ls -lh)<br># 3. sort them by size (largest first)<br># 4. display only the top 5 results<br>find &quot;$folder&quot; -type f -exec ls -lh {} \; | sort -k 5 -hr | head -n 5</pre><p>Go ahead and run it like:</p><pre>chmod +x folder_report.sh<br>./folder_report.sh .</pre><p>I hope this was helpful is someway in getting you introduced to UNIX shell scripting.</p><p>If you enjoyed it so far and would like to dig deeper to learn more high frequency commands, I would suggest to take a look at this page. Happy learning!</p><p><a href="https://www.geeksforgeeks.org/linux-unix/basic-shell-commands-in-linux/">Basic Shell Commands in Linux - GeeksforGeeks</a></p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d412a02d5a0e" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/unix-shell-scripting-essentials-d412a02d5a0e">Unix Shell Scripting essentials</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Data Sharing]]></title>
            <link>https://medium.com/agileinsider/snowflake-data-sharing-4f2cb76fed49?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f2cb76fed49</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 12:43:34 GMT</pubDate>
            <atom:updated>2026-01-19T12:43:34.075Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Snowflake enables secure, zero-copy data access across teams</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yuduaNgoyW1F_7WH" /><figcaption>Photo by <a href="https://unsplash.com/@kkalerry?utm_source=medium&amp;utm_medium=referral">Klara Kulikova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Snowflake data sharing is often described as a simple feature. In reality, it changes how teams think about data access, ownership, and duplication. Many teams still move data around because that is how things worked in the past, especially with the traditional ETL data copies to downstream systems. Snowflake, with its data share concept, changes the way this is now being done.</p><p>This article walks through Snowflake data sharing from a practical angle. What it is. How it works. Where it helps. And where teams usually get it wrong.</p><h3>What is data sharing in Snowflake and why it matters</h3><p>Snowflake data sharing allows one Snowflake account to share data within same account or with another account without actually copying or moving it. The data stays where it is. There is no export, no replication. Only a virtual copy of data is shared with the consumers, which is like a pointer to the original data store in provider’s account.</p><p>This share is between two parties — provider and consumer.</p><ul><li>Providers are the ones who own and store the data as a product.</li><li>Consumers on the other hand, are the recipient of the product (data in this case) to utilize</li></ul><p>The consuming account queries the shared data directly. What they see looks like a normal database. But the ownership never changes.</p><p>This matters because data duplication creates problems. Storage costs increase. Security becomes harder to manage. With data sharing, there is one version of the data, and everyone reads from it.</p><h3>How Data Sharing Works</h3><p>Under the hood, Snowflake does not move data when you share it. It shares metadata pointers.</p><p>The provider account owns the data and defines what is exposed. The consumer account creates a database from that share and queries it using its own compute.</p><p>This separation is important — Storage always stays with the provider. Compute is always paid for by the consumer. This design makes data sharing scalable without penalizing the data owner when usage grows.</p><h3>Components involved in a data share</h3><p>At the center is the share object. This is where the provider lists which databases, schemas, tables, or views are exposed.</p><p>The provider account controls access. The consumer account receives the share and creates a database from it. From there, it behaves like any other read-only database.</p><p>Privileges apply at the object level. Consumers cannot modify data. They cannot add objects. They can only query what is explicitly shared.</p><h3>What can be shared and what cannot</h3><p>Snowflake data sharing focuses on objects that can be queried. Tables and views are the most common. Secure views are widely used because they allow tighter control and hide implementation details.</p><p>Operational objects are not part of sharing. You cannot share stages, files, tasks, streams, or temporary tables. Data sharing is meant for consumption, not for orchestration or processing logic.</p><p>You can share the following Snowflake objects:</p><ul><li>Databases</li><li>Tables</li><li>Dynamic tables</li><li>External tables</li><li>Regular views (not recommended. underlying tables need to granted access as well, view definition is exposed, security policies might not work as expected)</li><li>Secure views (recommended over regular views)</li><li>Secure materialized views</li><li>Semantic views</li><li>User-defined functions (UDFs) (secure and non-secure)</li></ul><h3>Types of data sharing</h3><p>The most common pattern is <strong>account-to-account sharing</strong>. One Snowflake account shares data with another. This is typical between internal business units.</p><p><strong>Reader accounts</strong> are used when the consumer does not already have Snowflake. The provider creates and manages the account, but the data still lives in the provider environment.</p><p><strong>Marketplace sharing</strong> is a variation of the same concept, applied at scale. Data is published for external consumption, sometimes commercially.</p><p>The underlying mechanics are the same. Only the access model changes.</p><h3>Sharing data between business units</h3><p>A common scenario is a central data team owning curated datasets. Sales, finance, and marketing all need access to the same numbers.</p><p>Without data sharing, teams often copy tables into separate schemas or accounts. Over time, those copies drift. Someone applies a fix in one place and forgets the others.</p><p>With data sharing, the data team exposes curated views. Each business unit consumes the same data through its own database. Everyone sees the same numbers. Governance stays centralized and maintenance effort drops.</p><h3>Sharing data with BI tools</h3><p>BI tools usually connect through read-only service accounts. Instead of granting direct access to production schemas, teams expose a share built specifically for analytics.</p><p>Only required columns are exposed. Sensitive fields can be masked. Secure views prevent BI users from seeing underlying table logic.</p><p>This approach is cleaner than managing dozens of direct grants and much easier to audit.</p><h3>Cost implications: Who pays</h3><p>Snowflake’s cost model for data sharing is straightforward.</p><p>The provider pays for storage. The consumer pays for compute.</p><p>If multiple teams query the same shared data, storage is not duplicated. If a consumer runs heavy queries, the provider’s warehouses are unaffected.</p><p>This makes data sharing especially attractive for central platform teams. They can publish data broadly without absorbing downstream query costs.</p><h3>Data privacy and security</h3><p>Data sharing respects Snowflake’s security model and the policies still apply.</p><p>Row access policies continue to filter data. Masking policies continue to protect sensitive columns. Secure views can enforce business rules and hide complexity.</p><p>Access can be revoked instantly. There is no need to clean up copied data. This is one of the strongest arguments for data sharing in regulated environments.</p><h3>Common pitfalls</h3><p>One common mistake is sharing raw tables instead of curated views. This exposes internal structures and makes downstream users dependent on implementation details that may change.</p><p>Another mistake is assuming data sharing replaces all ETL. It does not. Sharing works best for read-only consumption. Transformations still belong in the provider account.</p><p>Teams often assume that once data is shared, access is done. In reality, sharing is just the start. You still need to be deliberate about what is exposed and to whom.</p><p>Finally, some teams forget to plan for change communications. When multiple consumers rely on a shared dataset, changes need communication. Data sharing makes access easier, but accountability still matters.</p><h3>Final thoughts and considerations</h3><p>Snowflake data sharing is not just a feature. It is a different way to think about data access which is very different from traditional data share models.</p><p>It works best when one team owns the data and others consume it. It works best when consistency matters more than duplication and added maintenance. And it works best when security and governance are treated as first-class concerns.</p><p>If your organization is copying the same data across accounts or schemas, data sharing is worth serious consideration. It simplifies architecture, reduces cost, and keeps everyone aligned on the same source of truth.</p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f2cb76fed49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agileinsider/snowflake-data-sharing-4f2cb76fed49">Snowflake Data Sharing</a> was originally published in <a href="https://medium.com/agileinsider">Agile Insider</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating AI assisted data analysis with Microsoft Fabric]]></title>
            <link>https://medium.com/@saqibk0510/evaluating-ai-assisted-data-analysis-with-microsoft-fabric-288b5dc4350b?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/288b5dc4350b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 01:21:26 GMT</pubDate>
            <atom:updated>2026-01-19T01:21:26.813Z</atom:updated>
            <content:encoded><![CDATA[<h4>Testing how Fabric Copilot performs in SQL queries, notebooks, and Power BI-based analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZI2Yy1tVS0-Zzp4ORjJbhA.jpeg" /><figcaption>courtesy: Transformer-Technology</figcaption></figure><p>Integrated AI assistants are rapidly becoming part of modern analytics platforms. Microsoft Fabric is one of the ecosystems where AI support is now embedded across SQL, notebooks, and visualization workflows, promising faster insight discovery and smoother exploration.</p><p>But how well do these capabilities actually work in practice?</p><p>Rather than testing small toy datasets or simple metrics, I wanted to evaluate Fabric’s AI-assisted analysis on a realistic business scenario — the kind that analysts, data engineers, and product teams encounter on a day-to-day basis.</p><p>In this article, I walk through a hands-on evaluation of Fabric Copilot and AI-supported features across:</p><p>• SQL exploration in Lakehouse<br>• AI-assisted data summaries in notebooks<br>• narrative explanations in Power BI</p><p>The goal is to understand:</p><p>• where AI assistance is genuinely useful<br>• where assumptions become risky<br>• its behavior with real-world analytical questions</p><h3>Dataset &amp; Scenario</h3><p>To keep the analysis realistic, I worked with a synthetic SaaS subscription dataset representing:</p><p>• customers<br> • subscription plans<br> • billing cycles<br> • churn events<br> • usage activity</p><p>The scenario focused on:</p><ul><li>churn behavior (rate at which customers stopped using services)</li><li>retention patterns</li><li>downgrade → churn transitions</li><li>cohort differences</li></ul><h3>Conceptual Data Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hk9w-5Yk9IubwIZoUZEgDA.png" /></figure><p>This conceptual data model represents the lifecycle of a customer relationship. A customer can own one or more subscriptions. Each subscription generates usage activity over time, which is used to understand engagement and behavior. Eventually, a subscription may end in a churn event, representing the point at which the customer cancels or stops renewing.</p><h3>Test 1: Using Copilot for SQL Exploration in Lakehouse</h3><p>I started by testing how well Copilot handled analytical questions requiring grouping, time windows, and event dependency logic.</p><p>Example question or prompt:</p><blockquote>“Calculate churn rate by monthly cohort over the last six months.”</blockquote><p>Based on this, the SQL that was generated was syntactically correct and runnable.</p><pre>SELECT <br>  DATE_TRUNC(&#39;month&#39;, churn_date) AS churn_month,<br>  COUNT(DISTINCT customer_id) AS churned_customers<br>FROM churn_events<br>WHERE churn_date &gt;= DATEADD(month, -6, CURRENT_TIMESTAMP())<br>GROUP BY churn_month;</pre><p>However, if you noticed it implicitly considered “churn cohort” as <em>month of churn </em>and<em> </em>not <em>month of acquisition</em> — which changes the interpretation from business standpoint.</p><p>To validate, I wrote an equivalent analytical query manually:</p><pre>SELECT <br>  DATE_TRUNC(&#39;month&#39;, subscription_start_date) AS cohort_month,<br>  COUNT(DISTINCT CASE WHEN churn_date IS NOT NULL THEN customer_id END) <br>    / COUNT(DISTINCT customer_id) AS churn_rate<br>FROM subscriptions s<br>LEFT JOIN churn_events ce ON ce.customer_id = s.customer_id<br> AND ce.churn_date &gt;= s.subscription_start_date<br> AND (ce.churn_date &lt;= s.subscription_end_date<br>      OR s.subscription_end_date IS NULL)<br>GROUP BY cohort_month;</pre><p>The difference is subtle but significant.</p><h3>Observation</h3><p>Copilot was very helpful for:</p><p>• generating a runnable exploration query<br>• structuring grouping &amp; filtering<br>• accelerating iteration</p><p>But it lacked awareness of business-specific cohort meaning.</p><p>It produced a <em>technically viable code </em>but without the deeper understanding of underlying data.</p><h3>Test 2: AI-Assisted Summaries in Notebooks</h3><p>Next, I explored how AI-supported summaries performed in notebooks — especially when data distributions and behavioral differences were involved.</p><p>Example question:</p><blockquote>“Describe behavioral differences between monthly and annual plan for customers before churn.”</blockquote><p>The assistant produced useful descriptive observations, including:</p><ul><li>higher churn ratio among monthly plans</li><li>longer tenure before churn among annual plans</li><li>higher usage variance in monthly subscription plans</li></ul><p>I then validated using explicit Python queries:</p><pre>df_summary = (<br>  subscriptions<br>  .groupby(&quot;plan_type&quot;)<br>  .agg({<br>      &quot;tenure_days&quot;: &quot;mean&quot;,<br>      &quot;sessions_last_30d&quot;: &quot;mean&quot;,<br>      &quot;downgraded_before_churn&quot;: &quot;mean&quot;<br>  })<br>)<br>display(df_summary)</pre><p>The numerical values from dataframe output were encouraging and aligned with the narrative given by AI assistant.</p><h3>Where It Was Strong</h3><p>• good at descriptive interpretation<br>• highlighted trends<br>• reduced time to first-pass insights</p><h3>Where Caution Was Needed</h3><ul><li>at times it felt like inferring conclusion — like it mentioned that the churn increased in Q3 due to lower engagement and pricing sensitivity even though these parameters were not even part of the dataset.</li><li>minor variances were amplified</li></ul><h3>Test 3: Narrative Explanations in Power BI</h3><p>Finally, I tested AI-supported explanations in a visualization workflow.</p><p>Example request:</p><blockquote>“Explain why churn increased in Q3.”</blockquote><p>The output included:</p><ul><li>concise description of increasing trend</li><li>identification of higher churn among monthly plans</li></ul><p>These narratives were in the right direction, but it attributed the changes to — <em>user behavior shifts and pricing sensitivity</em></p><p>Neither of which were part of the dataset.</p><h3>Observation</h3><p>AI did well at:</p><p>• summarizing trend direction<br>• highlighting categorical contributors<br>• interpreting business language</p><p>But tends to:</p><ul><li>express analytical confidence without evidence</li><li>amplify minor variances</li><li>define unknown business concepts with assumptions</li></ul><p>This is where human review or Human in the loop (HITL) becomes essential.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gb_DfxdkcAvGP98Ic86uw.png" /></figure><h3>Final Thoughts</h3><p>Integrated AI capabilities inside analytical platforms are not “plug and play” systems. They do not eliminate the need for modeling clarity or analytical reasoning.</p><p>But they <em>do</em> meaningfully improve:</p><p>• the speed of exploration and<br>• the ease of iteration</p><p>Used thoughtfully, it becomes another tool in the analytical toolkit — <br> especially powerful in early-stage discovery, provided that human judgment remains central in interpretation.</p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=288b5dc4350b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is Data Sharing in Snowflake and Why It Matters]]></title>
            <link>https://blog.dataengineerthings.org/what-is-data-sharing-in-snowflake-and-why-it-matters-2020bce21511?source=rss-76ff414ceb69------2</link>
            <guid isPermaLink="false">https://medium.com/p/2020bce21511</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Saqib Khan]]></dc:creator>
            <pubDate>Tue, 13 Jan 2026 01:10:15 GMT</pubDate>
            <atom:updated>2026-01-13T01:10:15.863Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Snowflake enables secure, zero-copy data access across teams</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*JPuSebYFjbmXIih9j61sEw.png" /></figure><p>Most data platforms I have worked on started with good intentions. <strong>A single source of truth, clearly owned datasets and controlled access.</strong> Over time, that clarity faded. New teams came in, access requests piled up and the quickest way to unblock people was to <strong>create another copy</strong>. Before long, the same data existed in multiple places, and <strong>every number needed a footnote</strong>.</p><p>This approach did not come from bad design. It came from <strong>the constraints of the time</strong>. On-prem systems were isolated, access was limited and <strong>moving data was often the only practical option</strong>. ETL pipelines became the default solution to what was really <strong>an access problem</strong>.</p><p>What is interesting is how little this pattern changed when platforms moved to the cloud. <strong>Storage became cheaper, compute more flexible</strong>, but the habit of copying data stayed. Pipelines multiplied, <strong>sync jobs became fragile,</strong> and ownership became harder to reason about.</p><p><strong>Snowflake’s data sharing model</strong> fits into this story not as an optimization, but as <strong>a change in how access is handled</strong>. Instead of pushing data into more places, it allows teams to <strong>expose data where it already lives</strong>, while keeping <strong>ownership and governance intact</strong>.</p><h3>What snowflake data sharing actually does</h3><p>Most data platforms I have worked on started with good intentions. <strong>A single source of truth, clearly owned datasets and controlled access.</strong> Over time, that clarity faded. New teams came in, access requests piled up and the quickest way to unblock people was to create another copy. Before long, the same data existed in multiple places, and <strong>every number needed a footnote</strong>.</p><p>This approach did not come from bad design. It came from <strong>the constraints of the time</strong>. On-prem systems were isolated, access was limited and moving data was often the only practical option. ETL pipelines became the default solution to what was really <strong>an access problem</strong>.</p><p>What is interesting is how little this pattern changed when platforms moved to the cloud. <strong>Storage became cheaper, compute more flexible</strong>, but the habit of copying data stayed. Pipelines multiplied, sync jobs became fragile, and ownership became harder to reason about.</p><p>This is where <strong>Snowflake data sharing</strong> fits in. Not as an optimization, but as a change in how access is handled. Instead of pushing data into more places, it allows teams to expose data where it already lives, while keeping <strong>ownership and governance intact</strong>.</p><h3>How data sharing works in practice</h3><p>Copying data hides a lot of decisions. Ownership, cost and access control are spread across pipelines and jobs, so no one has to think about them too hard.</p><p>Sharing data forces those decisions into the open. <strong>Someone has to own the data, decide what is exposed and accept that other teams will query it directly.</strong> That is the model Snowflake’s data sharing is built around.</p><p>Under the hood, Snowflake does not share data files. <strong>It shares metadata pointers.</strong> The data stays with the provider account. Consumers query it using their own compute.</p><h3>Provider, consumer, and the basic sharing model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrTeyQ2ep-0T9oyfLEok3A.png" /><figcaption>Provider-Consumer data share model</figcaption></figure><p>Before we dive deep into the code and practical implementation, I would like to discuss a little about some concepts involved.</p><p>In Snowflake data sharing, there are always two sides. <strong>A provider and a consumer.</strong> Everything else builds on that idea.</p><p>The <strong>provider</strong> is the account that owns the data. This is where data is stored, transformed, validated, and governed. The provider decides what can be shared, at what level, and under which controls. <strong>Ownership never leaves the provider account.</strong></p><p>The <strong>consumer</strong> is the account that receives access to the data. The consumer does not get a copy. Instead, it queries the provider’s data using its own warehouse. The data looks local, but <strong>it is always read-only</strong>.</p><p>This split is intentional. <strong>Storage and governance stay with the provider. Compute usage stays with the consumer.</strong> That separation is what allows data sharing to scale without creating cost or ownership conflicts.</p><p>There is one more concept that matters. The <strong>share object</strong>.</p><p>A share is not data by itself. It is a definition. It lists exactly which databases, schemas, tables, or views the provider is willing to expose. Until a share is granted to a consumer, nothing is visible.</p><p>Once these tiny tiny things are clear, the SQL starts to make sense. Creating a share is simply the provider declaring what can be accessed and who is allowed to access it.</p><h3>Creating a share from the provider account</h3><p>Everything starts in the provider account.</p><p>The provider creates a share object, also referred as <strong>data products</strong>. Think of this object like a <strong>window into the data</strong>. The data does not move. The provider decides how wide the window is and what can be seen through it.</p><pre>CREATE SHARE sales_data_share;</pre><p>Next, access is granted at the database and schema level.</p><pre>GRANT USAGE ON DATABASE analytics_db TO SHARE sales_data_share;<br>GRANT USAGE ON SCHEMA analytics_db.curated TO SHARE sales_data_share;</pre><p>Then the actual objects that needs to be shared are added.</p><pre>GRANT SELECT ON ALL TABLES IN SCHEMA analytics_db.curated<br>TO SHARE sales_data_share;</pre><p>At this point, <strong>nothing is visible to any consumer yet</strong>. The share only defines what can be accessed.</p><p>Finally, the share is granted to a consumer account.</p><pre>ALTER SHARE sales_data_share<br>ADD ACCOUNTS = &#39;CONSUMER_ACCOUNT_ID&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dxW1joaqV3BJ2HP3y1xiOw.png" /><figcaption>Shared object management screen</figcaption></figure><h3>Consuming shared data</h3><p>On the consumer side, the workflow is intentionally simple.</p><p>The consumer creates a database from the share.</p><pre>CREATE DATABASE sales_shared_db<br>FROM SHARE provider_account.sales_data_share;</pre><p>Once created, the database behaves like a normal database with one important difference. <strong>It is read-only.</strong></p><pre>SELECT<br>  region,<br>  SUM(total_sales) AS sales<br>FROM sales_shared_db.curated.daily_sales<br>GROUP BY region;</pre><p>The consumer uses their own warehouse to run queries. <strong>All compute costs stay with the consumer.</strong> The provider’s warehouses are never touched.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GB0h1SPLjQKciiyi_INVdw.png" /><figcaption>Read-only database access on consumer side</figcaption></figure><p>This is usually the moment where teams realize that the data feels local, but ownership has not changed.</p><h3>Why teams usually share secure views</h3><p>In real-world setups, most teams do not expose raw tables. Raw tables change frequently and expose internal structure that downstream consumers should not rely on<strong>.</strong></p><p>They expose <strong>secure views</strong> that apply business logic, filter rows, or mask sensitive columns.</p><pre>CREATE SECURE VIEW curated.v_sales_summary AS<br>SELECT<br>  region,<br>  product,<br>  total_sales<br>FROM raw.sales<br>WHERE is_active = true;</pre><p>Secure views matter because:</p><ul><li>underlying table definitions are hidden</li><li>masking and row access policies still apply</li><li>consumers cannot reverse-engineer logic</li></ul><p><strong>The provider stays in control even after the data is shared.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKPEuQBSPjG7qMGUPGzX8A.png" /><figcaption>Secure view creation</figcaption></figure><h3>Sharing data between business units</h3><p>A common pattern is a central data team owning curated datasets, while sales, finance and marketing all rely on the same numbers.</p><p><strong>Without data sharing</strong>, teams often copy tables into separate schemas or accounts. Over time, those copies drift. Fixes are applied in one place and missed in others. <strong>Trust in the data slowly erodes.</strong></p><p><strong>With data sharing</strong>, the central team exposes curated views. Each business unit consumes the same data through its own database. <strong>Everyone sees the same numbers</strong>, while governance remains centralized.</p><h3>Sharing data with BI tools</h3><p>BI tools typically connect using read-only service accounts.</p><p>Instead of granting direct access to production schemas, teams expose a dedicated share designed for analytics. Only required columns are exposed. Sensitive fields are masked. Secure views hide implementation details.</p><p><strong>Access becomes intentional instead of accidental.</strong></p><p>This approach is far easier to audit than managing dozens of direct grants.</p><h3>Cost model and why it matters</h3><p>The cost model for data sharing is deliberately simple.</p><p><strong>The provider pays for storage.</strong><br><strong>The consumer pays for compute.</strong></p><p>Data is not duplicated as more consumers are added. Heavy query workloads on the consumer side do not affect the provider environment. <strong>Cost follows usage, not access.</strong></p><p>This is why central platform teams prefer sharing over copying at scale.</p><h3>Common pitfalls teams run into</h3><p>One common mistake is <strong>sharing raw tables instead of curated views</strong>. This exposes internal structures and makes consumers dependent on details that will eventually change.</p><p>Another mistake is assuming data sharing replaces all ETL. It does not. <strong>Sharing works best for read-only consumption.</strong> Transformations still belong in the provider account.</p><p>Teams also underestimate coordination. When multiple consumers depend on the same dataset, schema changes require communication. <strong>Sharing simplifies access, not accountability.</strong></p><h3>Final thoughts</h3><p>Snowflake data sharing reflects <strong>a different way of thinking about data access</strong>.</p><p>It works best when one team owns the data and others consume it, when consistency matters more than duplication and when governance is treated as part of the design.</p><p>If your organization is still copying the same data across schemas or accounts just to satisfy access requests, data sharing is worth serious consideration. <strong>It reduces operational overhead and keeps everyone aligned on a single source of truth.</strong></p><p>Thanks for stopping by! If this article added value or sparked ideas, consider following me here on <a href="https://medium.com/@saqibk0510">Medium</a> or connecting on <a href="https://www.linkedin.com/in/saqib-khan-888a491a/">LinkedIn</a>. Your thoughts and feedback are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2020bce21511" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/what-is-data-sharing-in-snowflake-and-why-it-matters-2020bce21511">What Is Data Sharing in Snowflake and Why It Matters</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>