<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mili Tripathi on Medium]]></title>
        <description><![CDATA[Stories by Mili Tripathi on Medium]]></description>
        <link>https://medium.com/@milinangalia?source=rss-be961034efd4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*TEQLkjcIlt3pi6hNW1i8eA.jpeg</url>
            <title>Stories by Mili Tripathi on Medium</title>
            <link>https://medium.com/@milinangalia?source=rss-be961034efd4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:04:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@milinangalia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[2025 Was the Year of Agents. 2026 Is the Year of the Harness.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/2025-was-the-year-of-agents-2026-is-the-year-of-the-harness-18963356fcb9?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*UrHlWF1gQT-cBluVB1YJHA.png" width="1536"></a></p><p class="medium-feed-snippet">The model is a rented black box. The harness is what you own and it can cut token spend by half without swapping models.</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/2025-was-the-year-of-agents-2026-is-the-year-of-the-harness-18963356fcb9?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/2025-was-the-year-of-agents-2026-is-the-year-of-the-harness-18963356fcb9?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/18963356fcb9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 09:56:57 GMT</pubDate>
            <atom:updated>2026-09-21T21:01:32.323Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Karpenter + YuniKorn: Giving Kubernetes a Fair Queue and Room to Grow]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/karpenter-yunikorn-giving-kubernetes-a-fair-queue-and-room-to-grow-425a7518f4d7?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*V92NKjVde5RySxv2Y1jyKA.png" width="1536"></a></p><p class="medium-feed-snippet">How workload scheduling and elastic infrastructure work together &#x2014; and why adding nodes doesn&#x2019;t solve every pending pod.</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/karpenter-yunikorn-giving-kubernetes-a-fair-queue-and-room-to-grow-425a7518f4d7?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/karpenter-yunikorn-giving-kubernetes-a-fair-queue-and-room-to-grow-425a7518f4d7?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/425a7518f4d7</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 15:35:16 GMT</pubDate>
            <atom:updated>2026-09-20T05:23:51.551Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Your Data Platform Needs a Crew, Not Another Giant Agent]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/your-data-platform-needs-a-crew-not-another-giant-agent-c75315cb3eb2?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1672/1*msy_SB_hISur24sW6CLBYA.png" width="1672"></a></p><p class="medium-feed-snippet">A practical multi-agent architecture for a data platform that can investigate, propose, and recover</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/your-data-platform-needs-a-crew-not-another-giant-agent-c75315cb3eb2?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/your-data-platform-needs-a-crew-not-another-giant-agent-c75315cb3eb2?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c75315cb3eb2</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 06:12:34 GMT</pubDate>
            <atom:updated>2026-09-20T05:32:08.934Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Batch vs Streaming Ingestion: The Real Decision Guide (Cost, Latency, Incremental Volume & Iceberg…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/batch-vs-streaming-ingestion-the-real-decision-guide-cost-latency-incremental-volume-iceberg-2c9bdc768de6?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*HRYHJ-Xe27fsdd7AjriHgg.png" width="1536"></a></p><p class="medium-feed-snippet">How to choose ingestion mode from SLA, cost, and operational maturity not from platform hype</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/batch-vs-streaming-ingestion-the-real-decision-guide-cost-latency-incremental-volume-iceberg-2c9bdc768de6?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/batch-vs-streaming-ingestion-the-real-decision-guide-cost-latency-incremental-volume-iceberg-2c9bdc768de6?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c9bdc768de6</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Sun, 25 Jan 2026 11:50:01 GMT</pubDate>
            <atom:updated>2026-09-21T11:26:50.774Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Rise of Temporal: How Netflix and Leading Tech Companies Are Revolutionizing Workflow…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@milinangalia/the-rise-of-temporal-how-netflix-and-leading-tech-companies-are-revolutionizing-workflow-822fbcc736e6?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*MOMbJvFMRB42Gp2b37oBMw.png" width="1536"></a></p><p class="medium-feed-snippet">Non member link</p><p class="medium-feed-link"><a href="https://medium.com/@milinangalia/the-rise-of-temporal-how-netflix-and-leading-tech-companies-are-revolutionizing-workflow-822fbcc736e6?source=rss-be961034efd4------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@milinangalia/the-rise-of-temporal-how-netflix-and-leading-tech-companies-are-revolutionizing-workflow-822fbcc736e6?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/822fbcc736e6</guid>
            <category><![CDATA[cloud-operations]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[temporal]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 07:40:48 GMT</pubDate>
            <atom:updated>2026-09-20T15:16:58.370Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Fivetran-dbt Labs Merger: A Turning Point for Data Platform Teams]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/the-fivetran-dbt-labs-merger-a-turning-point-for-data-platform-teams-ce12a1e3f7a2?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*21riz01HQnfbzE0ac-9fpQ.png" width="1536"></a></p><p class="medium-feed-snippet">A Data Platform Leader&#x2019;s Perspective on What This Consolidation Really Means</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/the-fivetran-dbt-labs-merger-a-turning-point-for-data-platform-teams-ce12a1e3f7a2?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/the-fivetran-dbt-labs-merger-a-turning-point-for-data-platform-teams-ce12a1e3f7a2?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce12a1e3f7a2</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[fivetran]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Thu, 25 Dec 2025 07:35:14 GMT</pubDate>
            <atom:updated>2026-09-20T05:42:03.085Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ Don’t Break the Database: Zero-Impact CDC Ingestion]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/dont-break-the-database-zero-impact-cdc-ingestion-6890c1a3c49f?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1536/1*0-n2PtlY5_4aR1_X3_A_yQ.png" width="1536"></a></p><p class="medium-feed-snippet">Non Member link</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/dont-break-the-database-zero-impact-cdc-ingestion-6890c1a3c49f?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/dont-break-the-database-zero-impact-cdc-ingestion-6890c1a3c49f?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6890c1a3c49f</guid>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[modern-data-platform]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Thu, 25 Dec 2025 05:41:50 GMT</pubDate>
            <atom:updated>2026-09-20T06:23:44.535Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Data Platforms Are Becoming Operating Systems]]></title>
            <link>https://medium.com/towards-data-engineering/data-platforms-are-becoming-operating-systems-6e2f25c2755a?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e2f25c2755a</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 10:10:52 GMT</pubDate>
            <atom:updated>2025-12-23T11:34:04.230Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aThjkpcL0laALsV09D440A.png" /></figure><h3>Not Tools. Not Pipelines. An OS for the Organization.</h3><blockquote>“If your data platform feels like a collection of tools, you’re still in the pre-OS era.”</blockquote><p>For years, we’ve described data platforms as:</p><ul><li>Stacks</li><li>Toolchains</li><li>Lakehouses</li><li>Ecosystems</li></ul><p>But none of these metaphors fully explain what modern data platforms have become.</p><p>Here’s the more accurate truth:</p><blockquote><strong><em>A modern data platform behaves like an operating system for the organization.</em></strong></blockquote><p>It schedules work, allocates resources, enforces security, exposes APIs, isolates tenants, and enables applications to run safely at scale.</p><p>Once you see it this way, <strong>architecture decisions suddenly make sense</strong>.</p><h3>Act I: The Tool Era (Where Most Platforms Begin)</h3><p>Early data platforms look like toolboxes.</p><p>You assemble:</p><ul><li>An ingestion tool</li><li>A processing engine</li><li>A warehouse or lake</li><li>A scheduler</li><li>Some dashboards</li></ul><h3>🧰 Tool-Oriented Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlCLdYHIhpWqjO5EplaA_A.png" /></figure><p>This works — until scale hits.</p><p>Problems emerge:</p><ul><li>Each tool has its own configs, permissions, and failure modes</li><li>No single place enforces policy</li><li>Cost visibility is fragmented</li><li>Engineers become glue code specialists</li></ul><p>This is equivalent to running applications <strong>without an OS</strong> — each program managing memory, security, and scheduling on its own.</p><h3>Act II: The OS Analogy (Why It Fits Perfectly)</h3><p>Let’s compare.</p><h3>Traditional Operating System Responsibilities</h3><ul><li>Process scheduling</li><li>Memory management</li><li>Security &amp; permissions</li><li>Resource isolation</li><li>Stable APIs for applications</li></ul><h3>Modern Data Platform Responsibilities</h3><ul><li>Job orchestration</li><li>Compute &amp; cost management</li><li>Data access control</li><li>Multi-tenant isolation</li><li>APIs for ingestion, access, and governance</li></ul><p>This isn’t a coincidence.</p><p><strong>Data platforms have crossed the threshold from tooling to infrastructure abstraction.</strong></p><h3>Act III: Anatomy of a Data Platform OS</h3><h3>🧠 Think in OS Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHFM9Y0Xed3MUjxVAUoOQA.png" /></figure><p>This framing clarifies <strong>where responsibilities belong</strong>.</p><h3>Act IV: The Kernel — Where Most Platforms Are Weak</h3><p>In operating systems, the kernel is sacred.<br>In data platforms, it’s often accidental.</p><h3>The Data Platform Kernel Includes:</h3><h4>🕒 1. Scheduling &amp; Orchestration</h4><ul><li>Deterministic execution</li><li>Retries, backfills, dependencies</li><li>Isolation across tenants</li></ul><p>This is why platforms evolve beyond cron-like schedulers.</p><h4>🔐 2. Security &amp; Isolation</h4><ul><li>Dataset-level access</li><li>Runtime identity</li><li>Least-privilege execution</li></ul><p>Security should be <strong>enforced by the platform</strong>, not reimplemented in every job.</p><h4>💸 3. Resource &amp; Cost Management</h4><ul><li>Who can consume how much?</li><li>What is the cost per dataset?</li><li>When should workloads be throttled?</li></ul><p>In OS terms: <strong>memory and CPU limits</strong>.<br>In data platforms: <strong>cost and compute quotas</strong>.</p><h4>🧬 4. Metadata as the Control Plane</h4><ul><li>Ownership</li><li>Lineage</li><li>Schema evolution</li><li>Compliance state</li></ul><p>Without metadata, the OS is blind.</p><h3>Act V: APIs Are the Syscalls of Data Platforms</h3><p>Applications never talk directly to hardware.</p><p>They make <strong>system calls</strong>.</p><p>Likewise, users should not:<br>❌ Write raw Spark jobs<br>❌ Hardcode IAM logic<br>❌ Manually manage schemas</p><p>Instead, platforms expose <strong>opinionated APIs</strong>.</p><h3>Example: Ingestion as a System Call</h3><pre>POST /datasets<br>{<br>  &quot;source&quot;: &quot;postgres&quot;,<br>  &quot;schema&quot;: &quot;orders_v1&quot;,<br>  &quot;sla&quot;: &quot;daily&quot;,<br>  &quot;retention&quot;: &quot;3y&quot;,<br>  &quot;compliance&quot;: [&quot;gdpr&quot;]<br>}</pre><p>The platform decides:</p><ul><li>Where data lands</li><li>How it’s processed</li><li>How access is enforced</li><li>How costs are tracked</li></ul><p>This is <strong>abstraction done right</strong>.</p><h3>Act VI: User Space — Where Innovation Should Happen</h3><p>When the OS is strong, applications flourish.</p><p>User space includes:</p><ul><li>Analytics</li><li>ML pipelines</li><li>Feature stores</li><li>GenAI applications</li></ul><p>Critically:</p><blockquote><strong><em>Failures in user space should not crash the kernel.</em></strong></blockquote><p>This is why:</p><ul><li>Isolation matters</li><li>Guardrails matter</li><li>Standard paths matter</li></ul><p>Strong platforms enable experimentation <strong>without instability</strong>.</p><h3>Act VII: Why GenAI Makes the OS Model Inevitable</h3><p>GenAI accelerates this shift dramatically.</p><p>LLMs:</p><ul><li>Generate queries</li><li>Trigger pipelines</li><li>Explore datasets autonomously</li></ul><p>Without:</p><ul><li>Strong metadata</li><li>Policy enforcement</li><li>Cost boundaries</li></ul><p>GenAI becomes a <strong>runaway process</strong>.</p><p>In OS terms:</p><blockquote>GenAI is untrusted user space code.</blockquote><p>Platforms must act as the kernel that keeps it safe.</p><h3>Act VIII: Leadership Implications (This Is the Big Shift)</h3><p>Thinking in OS terms changes how leaders operate.</p><h3>❌ Old Leadership Focus</h3><ul><li>“Which tool should we buy?”</li><li>“How fast can we build this pipeline?”</li></ul><h3>✅ New Leadership Focus</h3><ul><li>“What belongs in the kernel?”</li><li>“What APIs do teams need?”</li><li>“How do we reduce cognitive load?”</li><li>“How do we scale trust, not just data?”</li></ul><h3>Act IX: A Real-World Data Platform OS</h3><h3>Iceberg + EMR + Snowflake (OS-Style Architecture)</h3><p>Let’s move from metaphor to reality.</p><p>Here’s how a <strong>production-grade data platform</strong> maps cleanly to the <strong>Operating System model</strong>, using technologies many enterprises actually run today.</p><h3>🧠 Mapping the OS Layers to Real Data Tech</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kv-eFjAuQX_ympuMI0cyUQ.png" /></figure><h3>🧱 OS-Style Data Platform Architecture (Real Stack)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*roz0mDCOK4FBZbEsVh1WxA.png" /></figure><h3>🔍 What Makes This an “Operating System” (Not Just a Stack)</h3><h3>1. Iceberg = The Platform File System</h3><p>Iceberg acts like a <strong>distributed, transactional file system</strong>:</p><ul><li>ACID guarantees</li><li>Schema evolution</li><li>Time travel</li><li>Snapshot isolation</li></ul><p>Just like an OS filesystem:</p><ul><li>Applications don’t manage files directly</li><li>They rely on stable abstractions</li></ul><p>Iceberg becomes the <strong>source of truth</strong>, not Snowflake or Spark.</p><h3>2. EMR on EKS = The Execution Kernel</h3><p>EMR on EKS plays the role of the <strong>kernel scheduler</strong>:</p><ul><li>Allocates compute per workload</li><li>Enforces runtime IAM roles</li><li>Isolates tenants via namespaces</li><li>Scales elastically</li></ul><p>Spark jobs are <strong>processes</strong>, not pets.</p><p>If one crashes, the system survives.</p><h3>3. Orchestration = Process Scheduler</h3><p>Temporal / Airflow function like the OS scheduler:</p><ul><li>Deterministic execution</li><li>Retries &amp; backoff</li><li>Dependency resolution</li><li>Backfills without chaos</li></ul><p>Importantly:</p><blockquote><em>Pipelines don’t control execution — the platform does.</em></blockquote><h3>4. Metadata = The Control Plane (Kernel Brain)</h3><p>Metadata is not documentation — it’s <strong>active control</strong>:</p><ul><li>Ownership</li><li>Lineage</li><li>Schema contracts</li><li>GDPR delete propagation</li><li>Cost attribution</li></ul><p>Without metadata, the OS cannot enforce policy.</p><h3>5. Cost Engine = Memory &amp; CPU Limits</h3><p>Cost is treated as a <strong>first-class resource</strong>:</p><ul><li>Per-dataset quotas</li><li>Per-tenant limits</li><li>Visibility at ingestion time</li></ul><p>This prevents the classic “runaway Spark job” problem.</p><h3>6. Snowflake = User-Space Application (Not the OS)</h3><p>This is the most important mindset shift.</p><p>Snowflake becomes:</p><ul><li>A <strong>consumer</strong></li><li>A <strong>query engine</strong></li><li>A <strong>presentation layer</strong></li></ul><p>Not:<br>❌ The system of record<br>❌ The governance engine</p><p>Iceberg remains the canonical data store.</p><p>This decoupling is what gives you:</p><ul><li>Cost leverage</li><li>Portability</li><li>Multi-engine freedom</li></ul><h3>🤖 GenAI in This OS Model</h3><p>GenAI agents live in <strong>user space</strong>.</p><p>They:</p><ul><li>Discover datasets via metadata</li><li>Query Iceberg via governed APIs</li><li>Trigger workflows through orchestration</li></ul><p>They <strong>never</strong> bypass:</p><ul><li>Security</li><li>Cost controls</li><li>Contracts</li></ul><p>Just like untrusted applications on an OS.</p><h3>Why This Architecture Scales (Technically and Organizationally)</h3><p>✅ New teams onboard via APIs, not tickets<br>✅ Compliance is enforced centrally<br>✅ Cost is predictable and visible<br>✅ Failures are isolated<br>✅ Innovation happens without destabilizing the platform</p><p>Most importantly:</p><blockquote><strong><em>The platform scales teams, not just data volume.</em></strong></blockquote><h3>Final Thought</h3><p>Pipelines move data.<br>Tools solve problems.</p><p>But <strong>operating systems enable ecosystems</strong>.</p><p>If your data platform:</p><ul><li>Enforces policy centrally</li><li>Exposes stable APIs</li><li>Scales teams safely</li><li>Enables innovation without chaos</li></ul><p>Then congratulations — you’re no longer building pipelines.</p><blockquote><strong><em>You’re building an operating system for data.</em></strong></blockquote><p>And that is the future of data platforms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e2f25c2755a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/data-platforms-are-becoming-operating-systems-6e2f25c2755a">Data Platforms Are Becoming Operating Systems</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Stop Forcing DBT to Lift Heavy: Why Your Data Ingestion & Processing Belong to PySpark (Not…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/stop-forcing-dbt-to-lift-heavy-why-your-data-ingestion-processing-belong-to-pyspark-not-4120ca3fa3e7?source=rss-be961034efd4------2"><img src="https://cdn-images-1.medium.com/max/1024/1*Wxrko93yth0ON0px_zixKg.png" width="1024"></a></p><p class="medium-feed-snippet">If your pipeline is sweating, it&#x2019;s probably running nested SQL compiled by DBT. Let&#x2019;s fix that.</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/stop-forcing-dbt-to-lift-heavy-why-your-data-ingestion-processing-belong-to-pyspark-not-4120ca3fa3e7?source=rss-be961034efd4------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/stop-forcing-dbt-to-lift-heavy-why-your-data-ingestion-processing-belong-to-pyspark-not-4120ca3fa3e7?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4120ca3fa3e7</guid>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 07:18:39 GMT</pubDate>
            <atom:updated>2026-09-20T15:18:29.282Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ The Application Layer Is Collapsing Into Agents: A Developer’s Guide to the Shift]]></title>
            <link>https://medium.com/@milinangalia/the-application-layer-is-collapsing-into-agents-a-developers-guide-to-the-shift-b1cd11f53609?source=rss-be961034efd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1cd11f53609</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[application]]></category>
            <dc:creator><![CDATA[Mili Tripathi]]></dc:creator>
            <pubDate>Sat, 11 Oct 2025 08:26:09 GMT</pubDate>
            <atom:updated>2025-10-18T04:21:33.227Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“The application layer is collapsing into agents.”<em><br> — </em><strong><em>Satya Nadella</em></strong><em>, CEO of Microsoft</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*675WWtp5OAM4yupOLF2BJQ.png" /><figcaption>Apps -&gt; Agents</figcaption></figure><p>In one sentence, Satya Nadella captured the silent revolution underway in software development. Applications as we know them — with screens, forms, and workflows — are transforming. Their logic, interaction models, and interfaces are being absorbed into <strong>autonomous agents</strong> that learn, reason, and act.</p><p>This blog will help developers understand what this shift means, how to adapt, and what to start building today.</p><h3>🚀 TL;DR: From Apps to Agents</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXgRh2AZXr4itOkAI2tfww.png" /></figure><h3>🏛️ Architecture Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOX_geHNpyvUKMl_eRScKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*SshMVG-R40iuMFbey72rhA.png" /></figure><p>This visual shows the <strong>collapse</strong> of multiple layers into one unified agent interface that <strong>interprets</strong>, <strong>reasons</strong>, and <strong>executes</strong>.</p><p>Here’s the <strong>final version of the creative, developer-focused blog</strong> on Satya Nadella’s insight:</p><h3>💡 Devs: Time to Change Your Approach</h3><h3>🚫 Old Way of Thinking:</h3><blockquote><em>“What screens do I need?”<br> “What buttons does the user click?”<br> “How do I paginate the table?”</em></blockquote><h3>✅ New Agent-First Thinking:</h3><blockquote><em>“What </em><strong><em>intent</em></strong><em> does the user express?”<br> “How do I fulfill it via tools or APIs?”<br> “How can the agent reason about this contextually?”</em></blockquote><h3>📊 Real-World Examples</h3><h3>🔹 GitHub Copilot</h3><p>Writes code using context from your editor. You don’t open a helper app. The agent is right inside your IDE.</p><h3>🔹 Microsoft Copilot</h3><p>Summarizes, drafts, and analyzes across Word, Excel, Outlook. The app still exists, but <strong>you rarely touch it directly</strong>.</p><h3>🔹 GPT with Plugins / Tools</h3><p>One interface, many actions: book flights, summarize PDFs, write SQL — with <strong>no separate app required</strong>.</p><h3>🧪 Example: Weather App — Old vs Agent Way</h3><h3>🔹 Traditional Flask App:</h3><pre>from flask import Flask, request<br>import requests</pre><pre>app = Flask(__name__)</pre><pre>@app.route(&#39;/weather&#39;)<br>def get_weather():<br>    city = request.args.get(&quot;city&quot;)<br>    response = requests.get(f&quot;http://api.weatherapi.com/v1/current.json?q={city}&quot;)<br>    return response.json()</pre><p>➡️ Requires: Route setup, HTML, query param management</p><h3>🔸 Agent-Driven Version:</h3><pre>from openai import OpenAI<br>import requests</pre><pre>def interpret_prompt(user_input):<br>    if &quot;weather&quot; in user_input:<br>        city = extract_city(user_input)<br>        return fetch_weather(city)</pre><pre>def extract_city(text):<br>    if &quot;Delhi&quot; in text:<br>        return &quot;Delhi&quot;<br>    return &quot;Unknown&quot;</pre><pre>def fetch_weather(city):<br>    response = requests.get(f&quot;http://api.weatherapi.com/v1/current.json?q={city}&quot;)<br>    return response.json()</pre><pre># Test<br>print(interpret_prompt(&quot;What&#39;s the weather like in Delhi today?&quot;))</pre><p>✅ This agent doesn’t require a UI.<br> ✅ It speaks the user’s language.<br> ✅ It abstracts intent → tool → result.</p><h3>🧱 What Should Developers Do Now?</h3><h3>1. Build Tools, Not Screens</h3><p>Design services (API endpoints, toolkits) that can be invoked by agents dynamically. Think like a service provider, not a UI developer.</p><pre># Example tool definition<br>def get_exchange_rate(base, target):<br>    # Pure tool function<br>    ...</pre><h3>2. Focus on Intents, Not Inputs</h3><p>Train your mental model to interpret goals:</p><ul><li>“Book me a flight next weekend”</li><li>“Summarize the last 3 emails from my boss”</li><li>“Generate a sales chart for Q3”</li></ul><p>Use LLMs to convert this into actions.</p><h3>3. Make Everything Observable + Orchestratable</h3><p>Agents will break if your tools don’t return traceable outputs. Design with:</p><ul><li>step logs</li><li>tool feedback</li><li>timeout safeguards</li></ul><pre># Add observability to a tool<br>def fetch_invoice_data(customer_id):<br>    logger.info(f&quot;Fetching invoice for {customer_id}&quot;)<br>    ...</pre><h3>4. Design Agent UX Like Conversation Trees</h3><p>Even if you don’t build the agent yourself, <strong>design for dialogue</strong>:</p><ul><li>What follow-up questions might the agent ask?</li><li>What if the user gives ambiguous input?</li><li>How will the agent know it succeeded?</li></ul><h3>🚀 Developer’s Action Plan</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J--YlLq4Hy_odiqS_j6C4g.png" /></figure><h3>🔄 Rethinking Your Job as a Developer</h3><blockquote><em>Before: “What app are you building?”<br> After: </em><strong><em>“What agent are you enabling?”</em></strong></blockquote><p>The lines are blurring between UI designer, backend engineer, and AI orchestrator. Developers must evolve from:</p><p>🧱 Builders of Interfaces → 🧠 Architects of Intelligence</p><p>👨‍💻 CRUD Engineers → 🤖 Intent Orchestrators</p><h3>🎤 Final Thought</h3><blockquote><em>💡 “In 5 years, you won’t be opening 5 apps.<br> You’ll have one agent who opens the right APIs for you.”<br> — Probably your future self</em></blockquote><p>The Application Layer is not dying. It’s being reborn — <strong>as agents</strong>.</p><p>And you? You’re the new Agent Whisperer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1cd11f53609" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>