<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dharmik Soni on Medium]]></title>
        <description><![CDATA[Stories by Dharmik Soni on Medium]]></description>
        <link>https://medium.com/@dhsoni2510?source=rss-12fc7e4bb9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RipkFoWsl3Kyed6JIae_Gw.png</url>
            <title>Stories by Dharmik Soni on Medium</title>
            <link>https://medium.com/@dhsoni2510?source=rss-12fc7e4bb9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:14:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhsoni2510/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS Just Split Lambda in Two. Here’s Which Half You Should Use.]]></title>
            <link>https://medium.com/@dhsoni2510/aws-just-split-lambda-in-two-heres-which-half-you-should-use-e0a018bd62fa?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0a018bd62fa</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 20:09:52 GMT</pubDate>
            <atom:updated>2026-10-17T20:09:52.290Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your Lambda API has been paying to wait. On October 12, AWS finally fixed that. Here’s what changed, in plain English, with pictures.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26m9YBacfjhlHx2eZOBGDQ.png" /></figure><p>Picture 500 users hitting your API at the same moment. Each request calls a database, then waits 150 ms for the answer.</p><p>On classic AWS Lambda, that means <strong>500 separate execution environments</strong>. Possibly 500 cold starts. And you pay for every millisecond each one sits there doing nothing but waiting.</p><p>That’s not a bug. It’s how Lambda has always worked: one request, one box.</p><p>On <strong>October 12, 2026</strong>, AWS changed that. It launched <strong>Lambda Web Functions</strong>, a new kind of Lambda where your code is just a normal web server. One environment can handle up to <strong>128 requests at once</strong>, and you pay for the CPU your code actually uses, not the time it spends waiting.</p><p>And quietly, the Lambda you already know got a new name: <strong>Lambda Event Functions</strong>. Nothing about it changes. No redeploys, no code changes. It’s just a name to tell the two apart.</p><p>If you build APIs, web apps, or AI endpoints on AWS, this is the biggest change to Lambda in years. By the end of this post you’ll know exactly how the two differ, what each one costs, and which one to pick.</p><p>Let’s go.</p><h3>First, how Lambda has always handled web traffic</h3><p>Classic Lambda only understands <strong>events</strong>. A JSON object comes in, your handler runs, a JSON object goes out.</p><p>A browser doesn’t speak events. It speaks HTTP. So something has to sit in front of your function and translate: API Gateway, an Application Load Balancer, or a Function URL. It turns the HTTP request into an event, and turns your return value back into an HTTP response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lo1HzXkSOxJFApkajUr09A.png" /></figure><p>This works. Millions of APIs run this way. But it comes with three costs most people stop noticing:</p><ol><li><strong>Your code is shaped around Lambda.</strong> You write a handler(event), not an Express app. To use a real framework, you add an adapter like Serverless Express or the AWS Lambda Web Adapter.</li><li><strong>Every concurrent request gets its own environment.</strong> 500 slow requests in flight = 500 environments, and potentially 500 cold starts.</li><li><strong>You pay for waiting.</strong> Billing is memory × wall-clock time. If your function waits 180 ms on a database and computes for 20 ms, you pay for all 200.</li></ol><h3>Meet Lambda Web Functions: just run your server</h3><p>A Web Function flips the model. Instead of a handler, you give Lambda a <strong>normal HTTP server that listens on a port</strong>. Express, Hono, Fastify, Next.js, plain Node.js. The same code you run on your laptop.</p><p>Lambda does the rest:</p><ul><li>Gives you an <strong>HTTPS endpoint</strong> out of the box. No API Gateway needed.</li><li><strong>Scales</strong> your server out with traffic, and back down to zero when it’s idle.</li><li>Lets each environment handle <strong>many requests at once</strong> (64 by default, up to 128).</li><li>Can <strong>replicate to several Regions</strong> behind one global URL, routing each user to the closest one.</li><li>Bills <strong>active CPU time</strong>, not wall-clock time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPqLC5Qi5qQec0mcnjATdQ.png" /></figure><p>Two new words to learn. A <strong>revision</strong> is an immutable snapshot of your code and config, like a published version. An <strong>endpoint</strong> is the HTTPS URL that routes traffic to one or two revisions, like an alias that also owns the URL, the auth, and the rate limit. Want a canary release? Send 90% of traffic to the old revision and 10% to the new one. One API call.</p><h3>The big difference: one request per box vs. many</h3><p>If you remember one thing from this post, make it this picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nKo1muFNH2LDVDa6BObpw.png" /></figure><p>When a burst of traffic arrives, Web Functions fill up warm environments first instead of spinning up a new one for every request. Fewer environments means fewer cold starts. Lambda also keeps some warm headroom ahead of demand, so small traffic bumps rarely hit a cold start.</p><p>Here’s the full side-by-side:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VE-PSoL0qw31FYqzR8aZ_A.png" /></figure><h3>The billing change (and why “cheaper” isn’t automatic)</h3><p>Most API requests spend their time <strong>waiting</strong>: on DynamoDB, on another service, and more and more on an LLM streaming tokens back. The two models treat that waiting very differently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S77fOXJe-EXGjbePPEMwfg.png" /></figure><p><strong>Event Functions</strong> bill requests + configured memory × wall-clock duration. Waiting costs the same as working.</p><p><strong>Web Functions</strong> bill four things, all at 1 ms granularity: requests, active vCPU time, the memory each environment holds for as long as it exists, and data transfer. Waiting on I/O uses almost no CPU, so it costs very little.</p><h3>A worked example</h3><p>AWS’s pricing page uses an I/O-heavy API: 10 million requests a month, each taking 200 ms end to end, with only 50 ms of real CPU work. US East (N. Virginia), Arm, before free tier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WK3aKVDz2g1OjNWVPCXcxw.png" /></figure><p>Surprised? So was I. <strong>Web Functions are not automatically cheaper.</strong> The request price is 50% higher, and you pay for memory while an environment sits idle between requests.</p><p>They win when requests mostly wait <strong>and</strong> you’d otherwise bump Event Function memory up just to get more CPU. They can lose on quiet or tiny endpoints. Model your real traffic and load test before you switch.</p><h3>Show me the code</h3><p><strong>Before: an Event Function behind API Gateway.</strong> Your logic lives inside Lambda’s event format.</p><pre>export const handler = async (event) =&gt; {<br>  const name = event.queryStringParameters?.name ?? &#39;world&#39;;<br>  return {<br>    statusCode: 200,<br>    headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>    body: JSON.stringify({ message: `Hello, ${name}` }),<br>  };<br>};</pre><p><strong>After: a Web Function.</strong> It’s just Express. The only Lambda-specific line reads the host and port from AWS_LAMBDA_HTTP_ENDPOINT.</p><pre>import express from &#39;express&#39;;</pre><pre>const app = express();<br>const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(&#39;:&#39;);</pre><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  res.json({ message: `Hello, ${req.query.name ?? &#39;world&#39;}` });<br>});</pre><pre>app.listen(port, host);</pre><p>Run it locally with AWS_LAMBDA_HTTP_ENDPOINT=127.0.0.1:3000 node index.js. Deploy it with one command:</p><pre>aws lambda-web deploy --name my-web-app --code ./my-app</pre><p>That bundles your code, creates the function, publishes a revision, attaches an endpoint, and hands back a live HTTPS URL. AWS says it takes under a minute.</p><p>Want it global? Add --regions us-east-1,eu-west-1,ap-northeast-1. Lambda copies your code to all three and routes each user to the fastest one through a single URL.</p><p>One catch: <strong>your code must now be safe for concurrent requests.</strong> Module-level variables that assume “one request at a time” will bite you. Audit those before you port anything.</p><h3>So which one should you use?</h3><p>Web Functions don’t replace Event Functions. They take over one job: <strong>answering HTTP requests directly.</strong> Everything else stays where it is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zI46CHQtLDKO5yHTv0kTmg.png" /></figure><p>AWS’s own picture of a modern app uses both: Web Functions for the user-facing API and server-side rendering, Event Functions for background work like queues and file processing, and Lambda MicroVMs (the third new form factor) for running untrusted code in a sandbox.</p><p>A simple pattern: the Web Function takes the request, validates it, and replies fast. Anything slow or multi-step gets handed off with SendMessage, PutEvents, or a Step Functions StartExecution, where Event Functions do the heavy lifting.</p><h3>Read this before you migrate</h3><p>This launched five days ago. Day-one services have rough edges. The ones that matter most:</p><ul><li><strong>Node.js only.</strong> Node.js 24 on Arm64, .zip packages. Python, Java, .NET or containers? Stick with Event Functions plus the Lambda Web Adapter for now.</li><li><strong>Five Regions.</strong> N. Virginia, Ohio, Oregon, Tokyo and Ireland.</li><li><strong>No knobs for size.</strong> Every environment is 2 GB and 2 vCPUs. You tune with max concurrency per environment instead.</li><li><strong>No provisioned concurrency.</strong> It always scales to zero, so the first request after a quiet period is a cold start. Keep startup fast.</li><li><strong>HTTP/1.1 only, no WebSockets.</strong> No custom domain or WAF on the endpoint directly. Put CloudFront in front; it now supports Web Functions as a private origin with OAC.</li><li><strong>HTTP-only triggers.</strong> A Web Function can’t be an SQS consumer, an S3 trigger, or a Step Functions Lambda task.</li><li><strong>“Active” doesn’t mean healthy.</strong> Lambda doesn’t check that your server actually starts at deploy time.</li></ul><p>Limits will change fast. Check the developer guide before you lean on any number here.</p><h3>The bottom line</h3><p>For ten years, Lambda asked web developers to bend their code to fit it. Web Functions flip that around: a server on a port, a framework you already know, and the same code on your laptop and in the cloud.</p><p>The trade? You unlearn “one request, one environment” and write concurrency-safe code.</p><p><strong>Try it this weekend.</strong> In a sandbox account, run:</p><pre>aws lambda-web deploy --name hello-world --hello-world</pre><p>Then pick one Node.js API that spends most of its time waiting on I/O, port it, load test it, and compare the bill.</p><p><em>If this helped you understand Lambda’s biggest change in years, give it a clap and follow for more hands-on AWS breakdowns. Are you moving anything to Web Functions? Tell me in the comments.</em></p><h3>Sources</h3><ul><li><a href="https://aws.amazon.com/blogs/aws/announcing-aws-lambda-web-functions-for-fast-consistent-web-app-performance-anywhere/">Announcing AWS Lambda Web Functions</a>, AWS News Blog</li><li><a href="https://aws.amazon.com/about-aws/whats-new/2026/10/aws-lambda-web-functions/">AWS Lambda announces Lambda Web Functions</a>, What’s New at AWS</li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/web-functions.html">Lambda Web Functions developer guide</a>, AWS Docs</li><li><a href="https://builder.aws.com/content/3KgdjNoXaCVsE5cpXwlOXiGModm/aws-lambda-web-functions-bring-your-http-server-leave-the-handler-behind">Bring Your HTTP Server, Leave the Handler Behind</a>, AWS Builder Center (pricing comparison and launch caveats)</li><li><a href="https://aws.amazon.com/lambda/pricing/">AWS Lambda pricing</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0a018bd62fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jev: The AI Model That Can’t Write a Single Sentence, and Why That’s the Point]]></title>
            <link>https://medium.com/@dhsoni2510/jev-the-ai-model-that-cant-write-a-single-sentence-and-why-that-s-the-point-c657d655c9e0?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c657d655c9e0</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[jevs]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 17:37:03 GMT</pubDate>
            <atom:updated>2026-09-22T17:37:03.914Z</atom:updated>
            <content:encoded><![CDATA[<h3>And that might be exactly why software developers will want to talk about it. A plain-English field guide to Jev vs. LLMs: decisions, language, and the new shape of machine intelligence.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9haJU670brr6HpF0xN9isw.png" /></figure><h3>Most AI writes. Jev decides.</h3><p>Ask a large language model to handle a support ticket and it may draft an empathetic reply.</p><p>Ask Jev, and it might return:</p><blockquote><strong><em>billing, 91% probability.</em></strong></blockquote><p>That difference sounds small until you picture the software wrapped around the model.</p><p>A chatbot produces language for a person to read. A business system usually needs something else: a branch it can execute. Route this ticket. Flag that transaction. Rank these search results. Decide whether a tool call looks risky.</p><p>Today, developers often ask a text-generating model to make that decision, force its answer into JSON, parse the JSON, validate the fields, handle malformed output, and then decide whether to trust it.</p><p>It works. It is also a little like hiring a novelist to operate a traffic light.</p><blockquote><strong><em>Jev’s pitch is simple: stop generating a paragraph when the application only needs a decision.</em></strong></blockquote><p>Jev is TypeSafe AI’s first <strong>System One model</strong>, the company’s name for models built to make fast, structured judgments that software can consume directly. It reads natural-language state, but it does not answer with prose. It returns typed values, probabilities, and, for some question types, a confidence signal. [1][2]</p><p>The name borrows from the idea of fast, intuitive “System 1” thinking. The product’s job is not to hold a conversation or write code.</p><p>Its job is to judge.</p><h3>The real difference is the shape of the answer</h3><p>An LLM and Jev can both receive natural language. The split happens at the output.</p><p><strong>Large language model: “Write the answer.”</strong> Generates an open-ended sequence of tokens: a reply, explanation, plan, code block, or structured object.</p><p><strong>Jev: “Judge these options.”</strong> Returns a typed choice, score, or probability inside an answer space defined before the call.</p><p><em>Same broad input medium. Radically different output contract.</em></p><p>Think of an LLM as a brilliant improviser. You give it context, and it creates the next useful thing. That flexibility is why it can summarize a contract, debug code, explain quantum physics, or invent a bedtime story.</p><p>Think of Jev as an intelligent function call. You give it a <strong>state</strong> (perhaps a customer message or a JSON record) plus typed questions. Jev evaluates those questions and sends back values that code can branch on, sort by, or compare with a threshold. TypeSafe says the questions are evaluated independently and in parallel against the same state. [1]</p><pre>state: &quot;My payment failed twice and I need access today.&quot;</pre><pre>question: Which team should own this?<br>options: [billing, technical, account]</pre><pre>result: billing<br>probabilities: {billing: 0.91, technical: 0.06, account: 0.03}</pre><p>The example is illustrative, but the contract is the point: the software receives an allowed value and a distribution, not a miniature essay that must be interpreted.</p><h3>Three primitives turn messy language into usable signals</h3><p>Jev currently exposes three question types. They are intentionally small, because small outputs are easier for software to trust and use. [1]</p><p><strong>1. Choice</strong> Select one option from a list. Use it for routing, categorization, or picking the best candidate.</p><p><strong>2. Score</strong> Place the state on an ordered rubric. Use it for urgency, quality, relevance, or severity.</p><p><strong>3. Noul</strong> Estimate whether a statement is true. It returns a 0-to-1 probability for a yes/no judgment.</p><p>The training idea behind Jev is called <strong>Reinforcement Learning for Calibrated Decisions (RLCD)</strong>. In plain English, the aim is not merely to pick the right answer, but to make the probability meaningful across many similar predictions. If a well-calibrated model repeatedly says “70%,” roughly seven out of ten of those cases should be correct.</p><p>That does not turn uncertainty into certainty. It turns uncertainty into a signal software can use. A team might auto-route cases above 0.95, ask an LLM to investigate cases from 0.70 to 0.95, and send the rest to a person.</p><ul><li><strong>Low risk → Act.</strong> High probability; known workflow.</li><li><strong>Unclear → Investigate.</strong> Let an LLM gather or explain.</li><li><strong>High stakes → Review.</strong> Keep a human in the loop.</li></ul><p>This is why Jev is interesting. <strong>It treats “I am not sure” as data, not as an embarrassing sentence to hide.</strong></p><h3>Jev is narrower than an LLM, on purpose</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BISaDkkPstp1dvfEALQ1Lw.png" /></figure><p>TypeSafe’s current model documentation lists Jev 1.13 with a 64k-token request context and text-only input. The company prices input at $0.042 per million tokens and says output is free. These are current product details, not timeless properties of the category. [3]</p><p>Speed and cost are part of the attraction. Jev avoids generating a long answer token by token and evaluates multiple typed questions in parallel. TypeSafe markets substantial efficiency gains, while early developer reports cited by TechCrunch describe faster classification workflows. Those claims are promising, but teams should still benchmark on their own data and network path. [4]</p><h3>“Can’t hallucinate” does not mean “can’t be wrong”</h3><p>TypeSafe says Jev cannot hallucinate because the developer defines the possible outputs in advance. If the only valid teams are billing, technical, and account, Jev will not invent a fourth department called <em>customer happiness engineering</em>.</p><p>That is a real advantage. It guarantees an answer that fits the schema. But it does not guarantee the correct answer. Jev can still send a billing problem to technical support. The model’s own documentation is explicit that calibration is measured across groups of predictions and does not make any single answer certain. [2]</p><blockquote><strong><em>The clean distinction.</em></strong><em> </em>Schema reliability:<em> the output has an allowed shape. </em>Decision accuracy:<em> the selected answer matches reality. Jev is designed to guarantee the first, not perfection on the second.</em></blockquote><p>Its documented “jaggedness” also matters. TypeSafe says the current model may struggle with literal phrasing, arithmetic, date comparisons, multiple hops of indirection, irrelevant context, contradictory criteria, and adversarial content. The recommended fixes are refreshingly unmagical: write precise criteria, keep arithmetic in code, reduce irrelevant state, test edge cases, and use a generative model when the task requires generation. [5]</p><p>This is not a fatal flaw. It is a boundary. In production AI, boundaries are valuable because they tell you where to place tests, thresholds, fallbacks, and human review.</p><blockquote><strong><em>A typed wrong answer is still wrong. The win is that it is easier to detect, measure, and route.</em></strong></blockquote><h3>Let an AI agent write the plan. Let Jev guard the button.</h3><p>Imagine a file-cleanup agent. The user says:</p><blockquote>“Remove duplicate downloads, but do not touch anything financial.”</blockquote><p>An LLM can interpret the request and propose a plan. The risky moment comes next: <strong>which actions should the system actually execute?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*luaA9k814ieGF0sGcijHqA.png" /></figure><p><em>This is an illustrative architecture, not a claim about a live Jev deployment. It shows how Jev’s typed judgments and probabilities can sit inside a safer agent loop.</em> [1][2]</p><p><strong>Step 1: The creative layer.</strong> The LLM proposes three cleanup actions. The system has already confirmed exact duplicates by file hash. The LLM turns the user’s words and file metadata into candidate actions.</p><p><strong>Step 2: Jev judges every proposed action.</strong> Same state, typed questions:</p><ul><li><strong>Choice:</strong> Does it match, look ambiguous, or violate the instruction?</li><li><strong>Score:</strong> Rate the action’s reversibility risk from 1 to 5.</li><li><strong>Noul:</strong> How likely is this file to count as financial?</li></ul><p><strong>Step 3: Code acts on the result.</strong></p><ul><li>🟢 <strong>ALLOW</strong> vacation-copy.mov: Move the verified duplicate to the recycle bin. <em>(match .99, risk 1)</em></li><li>🟡 <strong>ASK</strong> report-final(2).pdf: The meaning is unclear, so ask the user first. <em>(ambiguous .71, risk 3)</em></li><li>🔴 <strong>BLOCK</strong> Tax_2025/W2-copy.pdf: The proposal conflicts with the protected category. <em>(financial .99)</em></li></ul><p>The probabilities are illustrative. The durable pattern is: <strong>propose openly, judge inside a declared answer space, then let deterministic code enforce the policy.</strong></p><h3>Why Jev adds something</h3><p>The agent does not receive another paragraph to interpret. It gets allowed labels, a risk score, and probabilities that map cleanly to execute, ask, or block.</p><h3>Why Jev is not the whole system</h3><p>Code still proves which files are duplicates, applies thresholds, uses recoverable deletion, and logs every action. Low confidence goes to the user; Jev never gets unilateral power over high-stakes actions. [5]</p><h3>The most interesting system is Jev plus an LLM</h3><p>Jev is not a replacement for the model that writes your emails, explains a diagnosis, drafts code, or synthesizes five conflicting documents. It cannot do those things.</p><p>But many AI products contain dozens of tiny decisions around the big generative moment:</p><ul><li>Which model should receive this request?</li><li>Is the retrieved document relevant?</li><li>Does a proposed tool call match the user’s intent?</li><li>Is the answer safe enough to show?</li><li>Should the system retry, escalate, or stop?</li></ul><p>Those are Jev-shaped jobs. A capable architecture might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4bJqD0AUYFQZdgTMuoLtQ.png" /></figure><ol><li><strong>Judge: Jev routes.</strong> Choose path, score risk, expose uncertainty.</li><li><strong>Create: LLM works.</strong> Generate, reason, explain, or call tools.</li><li><strong>Check: Jev gates.</strong> Evaluate output before software acts.</li></ol><h3>Choose by asking one question</h3><p><strong>Use Jev when…</strong></p><ul><li>Every valid answer is known before the call.</li><li>Your code needs a branch, rank, score, or threshold.</li><li>You will repeat the judgment at high volume.</li><li>A probability is more useful than an explanation.</li></ul><p><strong>Use an LLM when…</strong></p><ul><li>The answer must be created, not selected.</li><li>The user needs prose, code, or an explanation.</li><li>The task needs open-ended reasoning or synthesis.</li><li>The output space cannot be listed in advance.</li></ul><blockquote><strong><em>LLMs are built to say something useful. Jev is built to return something your software can do something with.</em></strong></blockquote><p>The key idea survives the hype: not every intelligent system needs to speak.</p><p>Sometimes it only needs to decide.</p><h3>Sources and further reading</h3><ol><li>TypeSafe AI, “Introduction.” Product overview, question primitives, parallel evaluation, and output contract. <a href="https://docs.typesafe.ai/introduction">https://docs.typesafe.ai/introduction</a></li><li>TypeSafe AI, “System One.” Definition of System One models, comparison with LLMs, and the caveat that calibration does not guarantee an individual answer. <a href="https://docs.typesafe.ai/concepts/system-one">https://docs.typesafe.ai/concepts/system-one</a></li><li>TypeSafe AI, “Models.” Current Jev 1.13 context, input, pricing, and rate-limit details. <a href="https://docs.typesafe.ai/models">https://docs.typesafe.ai/models</a></li><li>TechCrunch, “A new kind of AI model from a ChatGPT inventor is thrilling developers.” Independent launch coverage and early developer reactions. September 18, 2026. <a href="https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/">https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/</a></li><li>TypeSafe AI, “Jev 1.13 jaggedness.” Documented limitations and mitigation guidance. <a href="https://docs.typesafe.ai/model-jaggedness/jev-1.13">https://docs.typesafe.ai/model-jaggedness/jev-1.13</a></li><li>TypeSafe AI, “Introducing System One Models &amp; Jev.” The company’s launch argument and description of RLCD and parallel sampling. <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">https://typesafe.ai/blog/introducing-system-one-models-and-jev</a></li></ol><p><em>Editorial note: “System One” and RLCD are TypeSafe AI’s own category and training labels. Public product documentation describes the behavior and goals; it does not fully disclose the model architecture or provide an independently reproducible training recipe. Product specifications and pricing can change.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c657d655c9e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Copilot Isn’t Broken. Your Harness Is.]]></title>
            <link>https://medium.com/@dhsoni2510/your-copilot-isnt-broken-your-harness-is-9b3cf0f3a6f0?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b3cf0f3a6f0</guid>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 19:42:11 GMT</pubDate>
            <atom:updated>2026-06-27T19:42:11.294Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How to turn VS Code Copilot from a confident guesser into a teammate that proves its work with a free, measurable course you can start in 15 minutes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fxFzIx2XUD77EaBpb7GnQ.png" /></figure><p>You open VS Code. You ask Copilot to add a feature. Twenty minutes later it cheerfully says <strong>“All done!”</strong></p><p>You run the tests. Red.<br>You check the diff. Three unrelated files changed.<br>Tomorrow’s chat? Zero memory of any of it.</p><p>If this sounds familiar, you’ve probably blamed the model.</p><p><strong>Try blaming the harness instead.</strong></p><h3>The uncomfortable truth</h3><p>A capable AI model without a <strong>harness</strong> is like a fast intern with no onboarding binder, no task list, and no one checking their work before they leave.</p><p><strong>Harness engineering</strong> is the discipline of building the system <em>around</em> the model:</p><ul><li>What it reads before it codes</li><li>How it remembers progress across sessions</li><li>What proof it must show before saying “done”</li><li>How much it’s allowed to do in one sitting</li><li>How every session starts and ends the same way</li></ul><p>This is <strong>not</strong> prompt engineering. You are not polishing a single message. You are designing a <strong>repeatable environment</strong> inside your repository.</p><h3>The Reliability Loop (the whole game in one picture)</h3><p>Most reliable Copilot sessions follow the same rhythm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8f8KWXczSop7AB_RMcO3XQ.png" /></figure><p>Miss any step and you get the chaos you already know: false “done,” scope creep, amnesia, environment fires.</p><h3>The five pillars (your diagnostic lens)</h3><p>When something fails, don’t swap models first. Ask which pillar broke:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qw1CZrbZJzWcGR_x3KYkqg.png" /></figure><p>I built a free course around this framework, <strong>Copilot-first</strong>, with copy-ready templates and labs where you <em>see</em> the before/after difference.</p><h3>What I built for you (free, open source)</h3><h3>Agent Harness Blueprint</h3><p>A measurable course for teams using <strong>VS Code Copilot</strong>.</p><p><strong>Live course:</strong> <a href="https://dharmik2510.github.io/agent-harness-blueprint/">https://dharmik2510.github.io/agent-harness-blueprint/</a><br><strong>GitHub:</strong> <a href="https://github.com/Dharmik2510/agent-harness-blueprint">https://github.com/Dharmik2510/agent-harness-blueprint</a></p><p>What makes it different from “yet another AI tips thread”:</p><ol><li><strong>You score your repo first, </strong>the Harness Scorecard (0–100) shows your weakest pillar</li><li><strong>You learn in short chapters, </strong>~8 minutes each, Copilot file paths included</li><li><strong>You practice on a real app, </strong>6 labs, including prompt-only vs harness side-by-side</li><li><strong>You re-score, </strong>the number going up is the proof you actually improved</li></ol><h3>Start here (pick your path)</h3><h3>Path A “Show me my gap in 2 minutes” (recommended)</h3><p>Take the interactive scorecard:</p><p><a href="https://dharmik2510.github.io/agent-harness-blueprint/diagnose"><strong>https://dharmik2510.github.io/agent-harness-blueprint/diagnose</strong></a></p><p>You’ll see which pillar is starving, Instructions, State, Verification, Scope, or Lifecycle — and what to fix first.</p><p>Or from terminal:</p><pre>git clone https://github.com/Dharmik2510/agent-harness-blueprint<br>cd agent-harness-blueprint &amp;&amp; npm install<br>npm run score -- /path/to/your/repo</pre><h3>Path B “Fix my repo today” (15 minutes)</h3><ol><li>Open your project in VS Code with Copilot Chat</li><li>Run /init to draft .github/copilot-instructions.md</li><li>Copy the <strong>template packs</strong> from the repo (templates/universal/ + templates/copilot/minimal/)</li><li>Run bash scripts/init.sh before every agent session</li><li>End every session by updating PROGRESS.md</li></ol><p><strong>Step-by-step guide:</strong> <a href="https://dharmik2510.github.io/agent-harness-blueprint/start-here/quick-start">https://dharmik2510.github.io/agent-harness-blueprint/start-here/quick-start</a></p><h3>Path C “Convince me with evidence” (45 minutes)</h3><p>Do <strong>Lab 01, Baseline vs harness</strong>: run the same feature task twice on a sample React app — once with prompts only, once with a minimal harness.</p><p><strong>Lab:</strong> <a href="https://dharmik2510.github.io/agent-harness-blueprint/labs/lab-01-baseline-vs-harness">https://dharmik2510.github.io/agent-harness-blueprint/labs/lab-01-baseline-vs-harness</a></p><p>Most people feel the difference in one session.</p><h3>The four-stage journey (how the course is organized)</h3><pre>Diagnose  ──►  Learn  ──►  Build  ──►  Prove<br>     │            │           │           │<br>  Scorecard    9 modules    6 labs    Re-score<br>  (2 min)     (~8 min ea)  (hands-on)  (number ↑)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwZNwsGylcaQHJi0oYYC7w.png" /></figure><h3>A Copilot session that actually works</h3><p>Paste this into Copilot Chat after you copy the templates:</p><pre>Follow AGENTS.md and .github/copilot-instructions.md.</pre><pre>1. Run bash scripts/init.sh<br>2. Read PROGRESS.md and feature_list.json<br>3. Pick exactly one feature with status &quot;open&quot;<br>4. Run all verification commands before saying done<br>5. Update PROGRESS.md with pass/fail evidence</pre><p>That’s the harness doing its job, not a longer prompt, a <strong>system</strong>.</p><h3>Who this is for</h3><ul><li><strong>Developers</strong> using GitHub Copilot in VS Code who are tired of cleaning up agent PRs</li><li><strong>Tech leads</strong> who want a shared vocabulary and rollout plan for the team</li><li><strong>Anyone</strong> who suspected the model was fine but the <em>workflow</em> wasn’t</li></ul><p>No prior “agent engineering” background required. If you can open a repo and run npm test, you&#39;re ready.</p><h3>What you’ll walk away with</h3><ul><li>A <strong>Harness Score</strong> you can re-run anytime</li><li>Copy-ready files: AGENTS.md, copilot-instructions.md, feature_list.json, init.sh</li><li>A <strong>Reliability Loop</strong> your team can repeat every session</li><li>Language to diagnose failures without saying “the model isn’t smart enough”</li></ul><h3>Your move</h3><ol><li><strong>Score your repo (2 min):</strong> <a href="https://dharmik2510.github.io/agent-harness-blueprint/diagnose">https://dharmik2510.github.io/agent-harness-blueprint/diagnose</a></li><li><strong>Start the course:</strong> <a href="https://dharmik2510.github.io/agent-harness-blueprint/start-here/quick-start">https://dharmik2510.github.io/agent-harness-blueprint/start-here/quick-start</a></li><li><strong>Star the repo</strong> if it helps: <a href="https://github.com/Dharmik2510/agent-harness-blueprint">https://github.com/Dharmik2510/agent-harness-blueprint</a></li></ol><p>The model was never the whole story. The harness was.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b3cf0f3a6f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Claude Code Setup Is a Solo Dev. Here’s How to Turn It Into a Team.]]></title>
            <link>https://medium.com/@dhsoni2510/your-claude-code-setup-is-a-solo-dev-heres-how-to-turn-it-into-a-team-75eddb67862d?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/75eddb67862d</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Thu, 21 May 2026 14:54:02 GMT</pubDate>
            <atom:updated>2026-05-21T14:54:02.685Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stop copy-pasting the same instructions into every chat. Five folders is all it takes to graduate Claude Code from a smart autocomplete into something that behaves like an entire engineering org. (And yes, the same blueprint works in VS Code Copilot,I’ll cover both.)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvdt2KtrnTgSUEXQkbA3OQ.png" /></figure><p>I’ll bet you’re using maybe a tenth of what Claude Code can actually do.</p><p>I was too. For months, my workflow was: open terminal, type a prompt, watch it work, close terminal. Every new session started from zero, re-explaining the codebase, re-pasting the style guide, re-reminding it not to touch the unnecessary files. It worked, but it felt like hiring a brilliant contractor who arrives every morning with full amnesia.</p><p>The fix isn’t a longer prompt or a more clever phrasing. It’s giving Claude Code an actual <em>environment</em> to live in. The tool ships with a layered configuration system that, if you set it up right, lets one assistant behave like a coordinated team, a junior dev who already knows your conventions, a code reviewer who only ever sees the diff, a CI gate that runs your linter before any code lands, an on-call bot that pings Slack when a deploy completes.</p><p>The whole system is <strong>five folders</strong>. That’s it.</p><p>I’ll walk through them in order, what each one does, when you actually need it, and what mistake usually makes people skip past it. Then I’ll show you how the same five-layer model maps almost one-to-one onto VS Code with GitHub Copilot, in case that’s where your team lives.</p><h3>A Quick Map of the Territory</h3><p>Before we go layer-by-layer, here’s the cheat sheet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*Ff0fvQYjTzSbQk4dgNKZpw.png" /></figure><p>Two things to notice already. First, the layers move from “passive memory” at the top to “active orchestration” at the bottom. Second, the <em>determinism</em> varies by layer on purpose. You want creativity in the skills layer and inflexibility in the hooks layer, they’re solving different problems.</p><p>The single most common configuration mistake is collapsing all five layers into CLAUDE.md (&quot;I&#39;ll just put everything in one file&quot;). That&#39;s how CLAUDE.md ends up at 2,000 lines, your context window catches fire, and Claude starts ignoring rules because they&#39;re buried under three procedure manuals. Keep the layers separate. Each one is cheap.</p><p>Let’s go.</p><h3>1. CLAUDE.md → The Repo&#39;s Standing Orders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eb3DFtgqSFgc_jVaMr-OVw.png" /></figure><p>Think of CLAUDE.md as the document you&#39;d hand a new hire on day one. Not a tutorial, a reference card. &quot;This is what we use, this is what we don&#39;t, here&#39;s how we name things, here&#39;s the one weird quirk about how auth works in this codebase.&quot;</p><p>You get two of them:</p><ul><li><strong>~/.claude/CLAUDE.md</strong> — your global file. Personal preferences that follow you across every project. Mine has things like <em>&quot;pnpm, never npm or yarn,&quot;</em> <em>&quot;if there&#39;s a </em><em>Makefile, prefer </em><em>make over remembering the actual command,&quot;</em> and <em>&quot;don&#39;t write comments that just restate the next line.&quot;</em></li><li><strong>./CLAUDE.md</strong>committed to the repo. Project-specific facts. The stack, the directory layout, the testing convention, the &quot;we tried that already and it broke prod&quot; warnings.</li></ul><p>The temptation, once you discover this file exists, is to dump everything into it. Resist. The rule that’s worked for me: <strong>if it’s a fact that’s true every day, it goes here. If it’s a procedure with steps, it doesn’t.</strong> Procedures belong one layer down, in skills/.</p><p>A healthy CLAUDE.md lives between 50 and 200 lines. When yours starts pushing 500, that&#39;s the universe telling you a skill needs to be born.</p><p>Here’s a slimmed-down version of one of mine:</p><pre># Project: payments-service</pre><pre>## Stack<br>- Node 22, TypeScript strict, pnpm<br>- Postgres via Drizzle, Redis via ioredis<br>- Tests: vitest + supertest</pre><pre>## Conventions<br>- Errors thrown as typed AppError subclasses, never bare throws<br>- All routes return Result&lt;T, AppError&gt;, never raw values<br>- Money handled in cents, never floats</pre><pre>## Don&#39;t<br>- Don&#39;t add new ORMs. Drizzle stays.<br>- Don&#39;t read .env directly — use `config/env.ts`</pre><p>Short. Declarative. Will still be true next Tuesday. That’s the bar.</p><h3>2. skills/ → Procedural Knowledge That Loads Itself</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1hHNsQdGlBhKHst_crN-Q.png" /></figure><p>Skills are the layer I wish I’d found first. Most people, including me, only discover them after they’ve built three messier workarounds.</p><p>Here’s the move. A skill is a folder at .claude/skills/&lt;name&gt;/ with a SKILL.md inside. The file has YAML frontmatter (a name, a description) and a body of instructions. When you ask Claude to do something whose description matches the skill&#39;s description field, <strong>Claude loads the skill on its own and follows it</strong>. No /skill-name command. No reminder. Pattern match, fetch, execute.</p><pre>---<br>name: pr-review<br>description: Use when reviewing pull requests, diffs, or when the user<br>  asks for code review. Provides structured feedback on correctness,<br>  style, security, and tests.<br>---</pre><pre># PR Review Workflow</pre><pre>1. Run `git diff origin/main` to scope the review.<br>2. Group findings into: Bugs, Style, Security, Missing Tests.<br>3. For each finding, cite file:line and propose a minimal fix.<br>4. Never approve if there are no tests and the diff touches src/.</pre><p>Three things this does that nothing else does:</p><ul><li><strong>It’s lazy.</strong> A skill costs nothing in your context window until it’s actually invoked. You can have 40 skills in a repo and the typical session loads two. Compare that to CLAUDE.md, which gets injected into every single conversation regardless of whether any of it is relevant.</li><li><strong>It’s self-routing.</strong> The description field is the matcher. Spend real effort on it, write it the way you&#39;d write a Stack Overflow title, focused on the <em>trigger</em> not the <em>output</em>. &quot;Use when reviewing PRs&quot; routes well. &quot;PR review skill&quot; doesn&#39;t.</li><li><strong>It scales.</strong> Skills can pull in supporting files (templates, checklists, example outputs) that only load when the skill itself fires. Your pr-review/ folder can have a 5-line SKILL.md and a 200-line examples.md and the second one only enters context when needed.</li></ul><p>The personal test I use: <em>if I’ve pasted the same paragraph of instructions into Claude more than twice this week, it’s a skill now.</em> Promote it.</p><h3>3. hooks/ → Walls Claude Cannot Walk Through</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3XAzTPhCKGpwqV5UJykUw.png" /></figure><p>Here’s a hard truth. Skills are instructions, and instructions get ignored sometimes. Even a great LLM will, on a bad day, decide that <em>this particular case</em> is the exception and reach for rm -rf anyway. If you&#39;ve been hand-holding the AI through &quot;no, please don&#39;t touch the migrations folder&quot; for the fifth time, you&#39;ve reached the limit of what prompting alone can do.</p><p>Hooks are the answer, and they’re refreshingly dumb. A hook is a shell script (or any command, really) that Claude Code runs at lifecycle events, <em>before</em> a tool call, <em>after</em> a tool call, on session start, on session end. The script returns an exit code. Non-zero kills the tool call cold. There’s no AI involved, no negotiation, no “I understand but I’ll try anyway.” Just bash and exit statuses.</p><p>You configure them in settings.json:</p><pre>{<br>  &quot;hooks&quot;: {<br>    &quot;PreToolUse&quot;: [{<br>      &quot;matcher&quot;: &quot;Bash&quot;,<br>      &quot;hooks&quot;: [{<br>        &quot;type&quot;: &quot;command&quot;,<br>        &quot;command&quot;: &quot;.claude/hooks/block-dangerous.sh&quot;<br>      }]<br>    }],<br>    &quot;PostToolUse&quot;: [{<br>      &quot;matcher&quot;: &quot;Write|Edit&quot;,<br>      &quot;hooks&quot;: [{<br>        &quot;type&quot;: &quot;command&quot;,<br>        &quot;command&quot;: &quot;pnpm lint --fix&quot;<br>      }]<br>    }]<br>  }<br>}</pre><p>The four things hooks actually earn their keep doing:</p><ul><li><strong>Killing dangerous commands.</strong> A PreToolUse hook on Bash that greps the incoming command for things like rm -rf, DROP TABLE, git push --force origin main, or any read of .env. Exit 1, command dies before it runs.</li><li><strong>Cleaning up after writes.</strong> A PostToolUse hook on Write or Edit that fires your formatter and linter. By the time Claude says &quot;done,&quot; the code is already formatted. You can&#39;t ship unformatted code because the hook fixed it before you saw it.</li><li><strong>Audit trails you can actually use later.</strong> Append every tool call to a log file with a timestamp, the command, and the exit code. Three weeks from now when someone asks “what did Claude do on Tuesday,” you have an answer.</li><li><strong>Notifications outside the loop.</strong> Hook fires on a deploy command, hook does curl to your Slack webhook. The team knows what shipped without anyone having to remember to post.</li></ul><p>The mental model swap here is important. With CLAUDE.md you write rules and hope. With hooks you write rules and they <em>are</em> the rule. The AI doesn&#39;t get a vote. That&#39;s why hooks are the right layer for anything you need to be true 100% of the time, not 99%.</p><h3>4. subagents/ → Cloning Yourself, On Purpose</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h78ERo3SdqDIeYG9IvVduQ.png" /></figure><p>Once your workflow gets serious, you hit a problem the first three layers don’t solve. Your main session is in the middle of a multi-file refactor. You ask for a code review of the in-flight diff. The reviewer reads 800 lines of diff, thinks out loud about each finding, prints a report. Now your main session’s context is half reviewer thoughts, and the next time you ask “where were we?” the answer is fuzzy because the buffer is polluted.</p><p>Subagents fix this. A subagent is a separate Claude instance, spawned from your main session, with <strong>its own context window, its own system prompt, and (this part is great) its own tool permissions</strong>. It does its work in isolation, hands back a clean summary, and then evaporates. Whatever mess it made in its own context goes with it.</p><p>Each subagent is just a markdown file in .claude/agents/:</p><pre>---<br>name: code-reviewer<br>description: Reviews a diff for bugs, style violations, and missing tests.<br>  Returns a structured review only.<br>tools: [Read, Grep, Bash]<br>---</pre><pre>You are a code reviewer. You see only the diff and the files it touches.<br>You do not edit code. You return findings in the format:<br>  - [SEVERITY] file:line — description — suggested fix</pre><p>Two payoffs make this layer worth the setup time:</p><p><strong>Context stays clean.</strong> Your main session keeps its working memory focused on what <em>you’re</em> doing. The reviewer’s chatter never enters it. When 40 turns later you ask “remind me what we were trying to do here,” the answer is sharp, not blurry.</p><p><strong>Permissions get scoped.</strong> Your main session has full write. Your test-runner subagent gets Bash access but only for npm test. Your reviewer is read-only and can&#39;t run anything. If a subagent goes off the rails, the damage is bounded to whatever you let it touch. This alone is reason to use them.</p><p>There’s also a third payoff that turns the “full dev team” framing literal: <strong>subagents run in parallel</strong>. Your main session can spawn three at once, one digs through the codebase for usages, another scans logs, a third pulls down the OpenAPI spec, all working concurrently, all reporting back to a single orchestrator. That’s not “AI assistant” anymore. That’s a fan-out.</p><p>A useful heuristic for when to reach for a subagent: the task is <em>bounded</em> (clear input, clear output), <em>noisy</em> (lots of intermediate thinking that won’t matter later), or <em>needs less power than the main session</em> (read-only, scoped tools). If any of those are true, push it into a subagent.</p><h3>5. plugins/ → Making the Whole Setup Shareable</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yFYD8bPhx4MqaZn62Q9BpA.png" /></figure><p>Now picture the situation you’ve created. You’ve spent a weekend perfecting your CLAUDE.md, you have eight skills, four hooks, three subagents, and you love your setup. Then you remember you have a team.</p><p>Without distribution, what happens next is messy. You DM your config to a teammate. They paste it in but skip the hook because it broke locally once. Two months later, your team’s five Claude Code setups have visibly diverged. Alice has the new PR-review skill, Bob doesn’t. Nobody is sure whose hooks are firing. The whole thing slowly desyncs.</p><p>Plugins are the version-controlled fix. A plugin is a single bundle that packages CLAUDE.md additions, skills, hooks, and subagents together. Your teammate runs one install command, restarts their session, and they have your full stack, same skills, same guardrails, same agents.</p><p>Inside, a plugin is structured roughly like this:</p><pre>my-team-plugin/<br>├── plugin.json<br>├── CLAUDE.md          # appended to user&#39;s CLAUDE.md<br>├── skills/<br>│   ├── pr-review/<br>│   └── release-notes/<br>├── hooks/<br>│   └── settings.json<br>└── agents/<br>    ├── code-reviewer.md<br>    └── test-runner.md</pre><p>The leverage here goes beyond your own team. There’s a growing community of public plugins , testing workflows, doc-generation kits, security-scanning stacks — that you can install into your own setup with a single command. Other people have spent their weekends building these. You get to skip ahead.</p><p>That’s also the upgrade path: build it for yourself first, then bundle it as a plugin when your team asks for it, then publish it when the community could use it.</p><h3>What This Looks Like, Drawn Out</h3><pre>~/.claude/CLAUDE.md             ← global memory<br>./CLAUDE.md                     ← repo memory<br>./.claude/<br>   ├── skills/                  ← auto-invoked workflows<br>   │   ├── pr-review/<br>   │   └── release-notes/<br>   ├── hooks/                   ← deterministic guardrails<br>   │   └── settings.json<br>   ├── agents/                  ← isolated workers<br>   │   ├── code-reviewer.md<br>   │   └── test-runner.md<br>   └── plugins/                 ← bundled for the team</pre><p>Five folders. That’s the entire Agent Development Kit.</p><p>The discipline, once it’s all set up, is keeping each layer doing only its own job:</p><ul><li><strong>A fact about your repo?</strong> That’s CLAUDE.md.</li><li><strong>A procedure you repeat?</strong> That’s skills/.</li><li><strong>A rule that cannot be broken?</strong> That’s hooks/.</li><li><strong>A side-task you want isolated?</strong> That’s agents/.</li><li><strong>A configuration you want your team to share?</strong> That’s plugins/.</li></ul><p>Get those boundaries right and the system runs itself. Get them wrong, and the classic wrong is “I’ll just put it in CLAUDE.md”, and you’ll be debugging why your AI ignores rules that are technically right there in the file. They’re right there, but they’re buried under 1,800 other lines that are also right there.</p><h3>“Cool, But Our Team Is on VS Code Copilot”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWU_frFyavvJdtIPLjJmJA.png" /></figure><p>Good news: every layer maps over. The file paths and extensions are different, the ecosystem is younger on a few of the layers, but the architecture is the same architecture.</p><h3>Layer 1: Memory → copilot-instructions.md (and friends)</h3><p>VS Code automatically reads .github/copilot-instructions.md from your repo root and applies it to every chat. Same idea as CLAUDE.md: standing facts, conventions, &quot;don&#39;t touch these files.&quot;</p><pre>.github/<br>└── copilot-instructions.md</pre><p>VS Code also supports AGENTS.md, which is the cross-tool standard a bunch of AI editors (Cursor, Aider, others) all read. And, useful migration detail, VS Code will <em>also</em> read a CLAUDE.md if it finds one. You don&#39;t have to throw your existing file away when switching contexts.</p><p>If you need different rules in different parts of a monorepo, .instructions.md files with glob patterns in the frontmatter do that.</p><h3>Layer 2: Knowledge → Agent Skills + Prompt Files</h3><p>This one’s better than you’d expect. VS Code now supports <strong>Agent Skills, </strong>a folder with a SKILL.md, frontmatter with name and description, auto-invoked by description match. It&#39;s the same open standard Claude Code uses. A skill written for one works in the other unchanged.</p><pre>.github/skills/pr-review/SKILL.md</pre><p>For the lighter “I want to type a slash command” case, there are also <strong>prompt files</strong> (.prompt.md). The split is: skills for things Copilot should just <em>know</em> to do when the task fits, prompt files for things you want to explicitly invoke.</p><h3>Layer 3: Guardrails → Hooks (with a fallback)</h3><p>VS Code added hooks to its Copilot configuration with the same shape as Claude Code’s lifecycle events, command matchers, deterministic enforcement. If you’ve ever set up Claude Code hooks, the VS Code equivalents will feel familiar.</p><p>For teams on older VS Code versions or environments where hooks aren’t wired up where you need them, the fallback is well-trodden: tasks.json for VS Code-side automation, plus husky or lefthook for git-level enforcement. Less elegant than first-party hooks, but you get the same &quot;this rule is non-negotiable&quot; property.</p><h3>Layer 4: Delegation → Custom Agents</h3><p>Copilot’s term for subagents is <strong>custom agents</strong> (which used to be called custom chat modes before the rename). They’re markdown files with .agent.md extension and frontmatter that declares the tools available, which models to try, and even handoff rules to other agents.</p><pre>---<br>description: Reviews a diff for bugs and missing tests.<br>name: code-reviewer<br>tools: [&#39;search/codebase&#39;, &#39;search/usages&#39;]<br>model: [&#39;Claude Opus 4.7&#39;, &#39;GPT-5.2&#39;]<br>---<br>You are a code reviewer. You see only the diff...</pre><p>They appear in the agent dropdown inside the Chat view. Pick one, the body gets prepended to your prompt, and you’re running in that agent’s persona. Handoffs let one agent kick off another, which is the same parallelism story subagents tell on the Claude Code side.</p><h3>Layer 5: Distribution → Repo-Committed + Org-Level Agents</h3><p>This is the layer where Copilot is still catching up. There’s no single-command “install a plugin” yet. But you can get most of the way with two practical tactics:</p><ol><li><strong>Commit the stack to the repo.</strong> .github/copilot-instructions.md, .github/skills/, .github/prompts/, .github/agents/,anyone who clones the repo gets the full setup. Not as clean as plugin install, but it works and your config is version-controlled.</li><li><strong>Org-level custom agents.</strong> Flip github.copilot.chat.organizationCustomAgents.enabled to true and any agents defined at the GitHub org level show up automatically in every member&#39;s agent dropdown. Centralized rollout without per-repo setup.</li></ol><p>The community side is growing too , the “Awesome Copilot” repo has reusable skill and agent collections you can lift wholesale.</p><h3>The Cheat Sheet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbHDJIocNNF21IOqPp98Ow.png" /></figure><h3>So, Where Do You Start?</h3><p>Don’t try to build all five layers at once. You’ll burn out and the result will feel artificial. Build them in the order you start to feel the pain:</p><ol><li><strong>You re-explain your stack every session.</strong> Time to write CLAUDE.md. One sitting, 80 lines, you&#39;re done.</li><li><strong>You paste the same instructions twice.</strong> Time for your first skill. Pick the workflow you do most — PR review, writing tests, generating release notes — and promote it.</li><li><strong>The AI does something you wish it physically couldn’t.</strong> Time for a hook. Start with one (block rm -rf is the classic) and grow from there.</li><li><strong>A side-task is flooding your main session.</strong> Time for a subagent. Wrap the side-task, scope its tools tight, let it run isolated.</li><li><strong>A teammate asks “how do I get your setup?”</strong> Time for a plugin. Bundle what you have. Ship it.</li></ol><p>The bigger shift, which applies on both Claude Code and Copilot, is the mental one: <strong>stop treating your AI assistant like a smarter chatbot, and start treating it like a programmable runtime.</strong> A chatbot you re-explain things to. A runtime you configure once and use forever.</p><p>Five folders. One stack. A team that shows up the same way every morning, knowing the rules, knowing the codebase, with the guardrails already up.</p><p>That’s the goal. Pick the first folder and go.</p><p>#ClaudeCode #GitHubCopilot #AIAgents #DeveloperTools #LLM #Programming #SoftwareEngineering #Anthropic #VSCode #AgenticAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75eddb67862d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HTML vs Markdown for LLMs: When Each One Wins (and What It Costs You)]]></title>
            <link>https://medium.com/@dhsoni2510/html-vs-markdown-for-llms-when-each-one-wins-and-what-it-costs-you-21c1c4020f75?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/21c1c4020f75</guid>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[markdown]]></category>
            <category><![CDATA[html]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Fri, 15 May 2026 13:27:57 GMT</pubDate>
            <atom:updated>2026-05-15T13:27:57.009Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>“HTML is the new Markdown.” a claim worth interrogating, not repeating.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Q2jPjp35DdHBKiEBrAEiw.png" /></figure><p>A recent take from Thariq Shihipar (lead engineer on Claude Code) argues that HTML is overtaking Markdown as the preferred format for LLM-generated documents. It’s a sharp claim. Let’s separate what’s actually true from what’s hype, and look at the <strong>real cost</strong> of choosing one over the other.</p><h3>The Short Answer</h3><ul><li><strong>Use Markdown</strong> when you want low token cost, portability, diff-friendly source, and “good enough” rendering everywhere.</li><li><strong>Use HTML</strong> when you need layout, styling, interactivity, embedded media, or pixel-level control, and you’re willing to pay for it in tokens.</li><li><strong>Markdown is more cost-effective.</strong> Period. HTML can match its expressiveness, but never its token economy.</li></ul><h3>Token Cost: The Part Nobody Wants to Quantify</h3><p>Tokens are what you actually pay for. Here’s the same content in both formats:</p><p><strong>Markdown (≈ 9 tokens)</strong></p><pre># Report<br>- Revenue: $1.2M<br>- Growth: 18%</pre><p><strong>HTML (≈ 28 tokens)</strong></p><pre>&lt;h1&gt;Report&lt;/h1&gt;<br>&lt;ul&gt;<br>  &lt;li&gt;Revenue: $1.2M&lt;/li&gt;<br>  &lt;li&gt;Growth: 18%&lt;/li&gt;<br>&lt;/ul&gt;</pre><p>Same information. <strong>~3× more tokens in HTML.</strong> Scale that across thousands of generated documents and the bill is not subtle.</p><h3>Visual: Token Overhead by Format</h3><pre>Plain text   │█                          │  baseline<br>Markdown     │██                         │  +10–20%<br>HTML (basic) │██████                     │  +150–300%<br>HTML + CSS   │████████████               │  +400–700%<br>HTML + JS    │██████████████████         │  +800%+</pre><p>The overhead comes from tags, attributes, closing elements, and boilerplate (&lt;!DOCTYPE&gt;, &lt;head&gt;, &lt;meta&gt;, etc.) none of which carry semantic information the model didn&#39;t already encode in plain prose.</p><h3>When HTML Genuinely Wins</h3><p>These are the cases where HTML is the right tool, not nostalgia:</p><ol><li><strong>Final-mile rendering.</strong> You need a self-contained artifact that opens in a browser and looks the same on every machine. Markdown depends on a renderer; HTML carries its own.</li><li><strong>Layout control.</strong> Multi-column dashboards, side-by-side comparisons, absolute positioning. Markdown can’t express this.</li><li><strong>Embedded interactivity.</strong> Collapsible sections (&lt;details&gt;), forms, charts via &lt;svg&gt; or &lt;canvas&gt;, sortable tables. Markdown&#39;s tables are static.</li><li><strong>Rich styling.</strong> Brand colors, custom fonts, dark mode toggles, print stylesheets.</li><li><strong>One-shot deliverables.</strong> Reports, invoices, dashboards, landing pages — anything where the output IS the product.</li><li><strong>Email and PDF pipelines.</strong> Both ultimately consume HTML.</li></ol><p>If your output is going to be <em>looked at</em> rather than <em>edited</em>, HTML earns its tokens.</p><h3>When HTML Is the Wrong Choice</h3><ol><li><strong>Source-of-truth documents.</strong> READMEs, docs, RFCs, ADRs. These live in git, get diffed, get reviewed. HTML diffs are noisy; Markdown diffs are clean.</li><li><strong>Pipelines that re-process the content.</strong> If another LLM call (or a human) is going to read it back, you’re paying the token tax twice.</li><li><strong>Notes, scratchpads, conversational outputs.</strong> Markdown renders fine in every chat UI, IDE, and terminal. HTML doesn’t.</li><li><strong>Cost-sensitive batch generation.</strong> Generating 10,000 summaries? Markdown can cut your bill by more than half.</li><li><strong>Anywhere portability matters.</strong> Markdown works in GitHub, Slack, Notion, Obsidian, VS Code, Jira, ChatGPT, Claude. HTML only fully renders in browsers.</li></ol><h3>The Tradeoff Table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*I_RfIioAIsswht5snMYJHw.png" /></figure><h3>So Is HTML “The New Markdown”?</h3><p>No. It’s <strong>a different tool for a different job.</strong></p><p>The framing that one format is replacing the other obscures the real decision: <em>what is this document for?</em></p><ul><li>If it’s <strong>consumed by humans visually, once</strong> → HTML can be worth it.</li><li>If it’s <strong>edited, versioned, re-read, or piped through more LLM calls</strong> → Markdown is the rational default.</li></ul><p>The argument that HTML is “winning” usually comes from people generating polished, standalone artifacts. That’s a real and growing use case, but it’s not most use cases.</p><h3>Decision Rule</h3><p>Ask three questions:</p><ol><li><strong>Who reads this?</strong> Human eyes once → HTML is on the table. Anything else → Markdown.</li><li><strong>Does it need layout or interactivity?</strong> Yes → HTML. No → Markdown.</li><li><strong>Will it be re-processed?</strong> Yes → Markdown, every time.</li></ol><p>If you can’t answer “yes, HTML” to question 1 <em>and</em> question 2, Markdown is the cheaper, faster, more portable choice.</p><h3>Bottom Line</h3><p><strong>Markdown is the most cost-effective format for LLM output.</strong> HTML is more expressive but pays for that expressiveness in tokens, portability, and review overhead. The smart move isn’t picking a winner, it’s matching the format to the artifact’s lifecycle.</p><p>Use HTML when the output is the destination. Use Markdown when the output is a step along the way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21c1c4020f75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Turned My Medium Article into a Voiced Explainer Video in an Afternoon. Here’s the Stack.]]></title>
            <link>https://medium.com/@dhsoni2510/i-turned-my-medium-article-into-a-voiced-explainer-video-in-an-afternoon-heres-the-stack-ebf24a2b22b5?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebf24a2b22b5</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Fri, 08 May 2026 18:28:12 GMT</pubDate>
            <atom:updated>2026-05-08T18:28:12.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEksWMU9h885wiLoma4HJA.png" /></figure><h3>React, Remotion, and macOS’s built-in TTS — no After Effects, no timeline scrubbing, no dead air.</h3><p>I’d just published a 9-minute Medium piece on Spark window functions and wanted a companion video. The instinct was to open After Effects, regret that decision, and ship nothing. Instead I tried <a href="https://www.remotion.dev/">Remotion</a> — a library that lets you build videos as React components — and shipped a 5:36 voiced explainer the same day.</p><p>This guide is the path I took, written for someone who has never opened Remotion and has no audio engineer on speed dial. We’ll go from blank folder to a rendered MP4 with synced voiceover. Every step is a copy-pasteable command.</p><blockquote><strong><em>What you’ll have at the end</em></strong><em>: a 1920×1080 video, 14 scenes, full voiceover, scene durations driven by audio length so there’s no dead air. Total cost: $0 if you stay on macOS’s built-in voices.</em></blockquote><h3>Why this works (and why it didn’t before)</h3><p>Three things made this possible in 2026 that weren’t in 2022:</p><ol><li><strong>Remotion 4</strong> — mature animation primitives, dynamic metadata, stable rendering.</li><li><strong>macOS Reed/Eddy/Sandy voices</strong> — Apple’s neural TTS voices are now good enough for technical narration. Not Eleven Labs good, but very acceptable.</li><li><strong>Mediabunny + </strong><strong>calculateMetadata</strong> — Remotion can read your audio files at composition-load time and size each scene to match. This is the trick that kills the &quot;dead silence at the end of every scene&quot; problem.</li></ol><p>If you’ve tried Remotion before and bounced off, this last piece is what’s new. You don’t have to guess scene timings anymore.</p><h3>Step 1 — Scaffold the project (2 minutes)</h3><p>You need Node 20+ and macOS (or Linux, with a different TTS later).</p><pre>npx create-video@latest --yes --blank dharmik-content-video<br>cd dharmik-content-video<br>npm install<br>npm run dev</pre><p>npm run dev opens <strong>Remotion Studio</strong> at http://localhost:3000. This is your timeline + preview. Leave it running — it hot-reloads as you edit code.</p><p>The starter ships with one empty composition in src/Composition.tsx. Ignore it. We&#39;re replacing the whole thing.</p><h3>Step 2 — Plan scenes before you write any code</h3><p>This is the step every tutorial skips and every first attempt regrets.</p><p>Open a notes file. List your scenes as [name, rough seconds, key visual]. Don&#39;t worry about exact lengths — voice will dictate them later.</p><p>For my Spark explainer:</p><p>#Scene~SecondsVisual1Hook10Title card2The old advice21Two SQL blocks side-by-side3Hidden bugs in max_by30Two warning cards4Why pre-3.4 was slow35Animated 3-step pipeline...14Closing10Big takeaway</p><p>Why this matters: Remotion is React. Each scene becomes a component. Naming them up front means your file structure and your script outline are the same thing.</p><h3>Step 3 — Build a single scene (the mental model)</h3><p>Create src/scenes/Hook.tsx. The whole library boils down to two hooks:</p><pre>import { useCurrentFrame, interpolate, AbsoluteFill, Easing } from &quot;remotion&quot;;</pre><pre>export const Hook = () =&gt; {<br>  const frame = useCurrentFrame(); // 0, 1, 2, ... per render</pre><pre>  // Fade in over 18 frames<br>  const opacity = interpolate(frame, [0, 18], [0, 1], {<br>    extrapolateRight: &quot;clamp&quot;,<br>    easing: Easing.bezier(0.16, 1, 0.3, 1),<br>  });</pre><pre>  return (<br>    &lt;AbsoluteFill style={{ background: &quot;#0B0F19&quot;, color: &quot;white&quot; }}&gt;<br>      &lt;h1 style={{ opacity, fontSize: 130 }}&gt;Hello, video.&lt;/h1&gt;<br>    &lt;/AbsoluteFill&gt;<br>  );<br>};</pre><p>That’s the entire mental model. <strong>There is no timeline.</strong> You declare what frame N looks like, and Remotion renders frame 0, 1, 2, … in order.</p><blockquote><strong><em>The gotcha that traps everyone</em></strong><em>: CSS transitions and Tailwind animation classes are forbidden. They don’t render correctly in Remotion’s frame-by-frame model. All animation must come from </em><em>useCurrentFrame() + </em><em>interpolate() (or </em><em>spring()). It feels weird at first; it stops feeling weird in about 20 minutes.</em></blockquote><h3>Step 4 — Sequence scenes with &lt;Series&gt;</h3><p>Once you have multiple scenes, compose them with &lt;Series&gt;:</p><pre>// src/Video.tsx<br>import { Series } from &quot;remotion&quot;;<br>import { Hook } from &quot;./scenes/Hook&quot;;<br>import { OldAdvice } from &quot;./scenes/OldAdvice&quot;;<br>// ...</pre><pre>export const SCENES = [<br>  { name: &quot;hook&quot;, durationInFrames: 240, component: Hook },<br>  { name: &quot;old-advice&quot;, durationInFrames: 600, component: OldAdvice },<br>  // ...<br>];</pre><pre>export const SparkVideo = () =&gt; (<br>  &lt;Series&gt;<br>    {SCENES.map((s) =&gt; (<br>      &lt;Series.Sequence key={s.name} durationInFrames={s.durationInFrames}&gt;<br>        &lt;s.component /&gt;<br>      &lt;/Series.Sequence&gt;<br>    ))}<br>  &lt;/Series&gt;<br>);</pre><p>useCurrentFrame() inside each scene component is <strong>scoped to that sequence</strong>, so frame 0 is the start of <em>that scene</em>, not the whole video. This is the single most important thing to internalize.</p><h3>Step 5 — Build reusable atoms, not one giant scene file</h3><p>After two scenes you’ll notice the same patterns: a section heading, a code block, a bullet list, a fade-up animation. Extract them into src/components/ as soon as you copy-paste once. Mine ended up as:</p><pre>src/components/<br>  Stage.tsx          # Background + padding wrapper every scene uses<br>  anim.ts            # fadeUp(), fade(), popIn() — animation primitives<br>  CodeBlock.tsx      # Monospace SQL/code with syntax tinting<br>  SectionHeading.tsx # Eyebrow + title + subtitle, all animated in<br>  Pill.tsx           # Small accent labels<br>  Bullet.tsx         # Icon + title + body row</pre><p>Each scene then becomes a thin layout file that composes these atoms. A 60-line scene instead of a 200-line one.</p><h3>Step 6 — Register the composition and set dimensions</h3><p>src/Root.tsx is where you tell Remotion how big and how long the video is.</p><pre>import { Composition } from &quot;remotion&quot;;<br>import { SparkVideo, SCENES } from &quot;./Video&quot;;</pre><pre>const TOTAL = SCENES.reduce((s, x) =&gt; s + x.durationInFrames, 0);</pre><pre>export const RemotionRoot = () =&gt; (<br>  &lt;&gt;<br>    &lt;Composition<br>      id=&quot;MyVideo&quot;<br>      component={SparkVideo}<br>      fps={30}<br>      width={1920}<br>      height={1080}<br>      durationInFrames={TOTAL}<br>    /&gt;<br>  &lt;/&gt;<br>);</pre><p>At this point you have a silent, static-duration video. Refresh the studio and watch your scenes flow. Fix any timing weirdness now — the next step makes durations dynamic.</p><h3>Step 7 — Generate voiceover with macOS say (the free path)</h3><p>This is where most tutorials say “use ElevenLabs”. You don’t have to. macOS ships with credible TTS voices — <strong>Reed (en-US)</strong>, <strong>Eddy</strong>, <strong>Sandy</strong> — that handle technical narration well. Cost: $0. Latency: instant.</p><p>First, sample voices: say -v &#39;?&#39; | grep en_US. Pick one and test:</p><pre>say -v Reed &quot;If you&#39;ve been writing Spark for a few years...&quot;</pre><p>Then write your scripts as JSON (one entry per scene id) and a generator script:</p><pre>// scripts/generate-voiceover.mjs<br>import { execFileSync } from &quot;node:child_process&quot;;<br>import { readFileSync, mkdirSync, existsSync } from &quot;node:fs&quot;;</pre><pre>const cfg = JSON.parse(readFileSync(&quot;scripts/voiceover.json&quot;, &quot;utf8&quot;));<br>const outDir = &quot;public/voiceover&quot;;<br>if (!existsSync(outDir)) mkdirSync(outDir, { recursive: true });</pre><pre>for (const scene of cfg.scenes) {<br>  const out = `${outDir}/${scene.id}.m4a`;<br>  execFileSync(&quot;say&quot;, [<br>    &quot;-v&quot;, cfg.voice,         // &quot;Reed&quot;<br>    &quot;-r&quot;, String(cfg.rate),  // 175 wpm — roomy. 200 = brisk.<br>    &quot;-o&quot;, out,<br>    &quot;--file-format=m4af&quot;,<br>    &quot;--data-format=aac&quot;,<br>    scene.text,<br>  ]);<br>  console.log(`✓ ${scene.id}`);<br>}</pre><p>Run with node scripts/generate-voiceover.mjs. You now have 14 .m4a files in public/voiceover/.</p><blockquote><strong><em>Two writing tips that made my VO sound good</em></strong><em>:</em></blockquote><ul><li>Spell out abbreviations: “ROW NUMBER” not “ROW_NUMBER”, “C plus plus” not “C++”. TTS reads punctuation literally.</li></ul><p>Keep sentences short. The voice doesn’t pause naturally on long compound clauses.</p><p>If you want premium quality later, swap say for an <a href="https://elevenlabs.io/docs/api-reference">ElevenLabs API call</a> in the generator. Same script structure, different one-line call. You can do this without touching the rest of your code.</p><h3>Step 8 — Sync scene durations to audio length (the killer trick)</h3><p>This is what makes the whole thing feel professional instead of student-project. Without it, you’re guessing scene durations and getting either dead air or cut-off VO. With it, every scene ends exactly when its narration ends.</p><p>Two pieces:</p><p><strong>Piece A: a helper to read audio duration</strong> (uses <a href="https://mediabunny.dev/">Mediabunny</a>, works in Node and browser):</p><pre>// src/get-audio-duration.ts<br>import { Input, ALL_FORMATS, UrlSource } from &quot;mediabunny&quot;;</pre><pre>export const getAudioDuration = async (src: string) =&gt; {<br>  const input = new Input({<br>    formats: ALL_FORMATS,<br>    source: new UrlSource(src, { getRetryDelay: () =&gt; null }),<br>  });<br>  return await input.computeDuration(); // seconds<br>};</pre><p>npm i mediabunny @remotion/media</p><p><strong>Piece B: </strong><strong>calculateMetadata</strong> — Remotion calls this <em>before</em> rendering, so you can compute durations from your audio files and pass them in as props:</p><pre>// src/Root.tsx<br>import { CalculateMetadataFunction, Composition, staticFile } from &quot;remotion&quot;;<br>import { getAudioDuration } from &quot;./get-audio-duration&quot;;<br>import { SCENES, SparkVideo, SparkVideoProps } from &quot;./Video&quot;;</pre><pre>const FPS = 30;<br>const TAIL_PAD_S = 0.4; // small breath after each VO clip</pre><pre>const calc: CalculateMetadataFunction&lt;SparkVideoProps&gt; = async () =&gt; {<br>  const seconds = await Promise.all(<br>    SCENES.map((s) =&gt; getAudioDuration(staticFile(s.audio))),<br>  );<br>  const sceneFrames = seconds.map((s) =&gt;<br>    Math.ceil((s + TAIL_PAD_S) * FPS),<br>  );<br>  return {<br>    durationInFrames: sceneFrames.reduce((a, b) =&gt; a + b, 0),<br>    props: { sceneFrames },<br>  };<br>};</pre><pre>export const RemotionRoot = () =&gt; (<br>  &lt;Composition<br>    id=&quot;MyVideo&quot;<br>    component={SparkVideo}<br>    fps={FPS}<br>    width={1920}<br>    height={1080}<br>    defaultProps={{ sceneFrames: SCENES.map(() =&gt; 240) }}<br>    calculateMetadata={calc}<br>  /&gt;<br>);</pre><p>And in your video component, use the prop instead of hardcoded durations:</p><pre>// src/Video.tsx<br>import { Audio } from &quot;@remotion/media&quot;;</pre><pre>export const SparkVideo = ({ sceneFrames }: { sceneFrames: number[] }) =&gt; (<br>  &lt;Series&gt;<br>    {SCENES.map((scene, idx) =&gt; (<br>      &lt;Series.Sequence<br>        key={scene.name}<br>        durationInFrames={sceneFrames[idx]}<br>      &gt;<br>        &lt;AbsoluteFill&gt;<br>          &lt;scene.component /&gt;<br>          &lt;Audio src={staticFile(scene.audio)} /&gt;<br>        &lt;/AbsoluteFill&gt;<br>      &lt;/Series.Sequence&gt;<br>    ))}<br>  &lt;/Series&gt;<br>);</pre><p>Now every scene is exactly the length of its VO + 0.4 seconds of breathing room. Re-record any line, regenerate audio, refresh the studio — durations recalculate automatically.</p><h3>Step 9 — Render to MP4</h3><pre>npx remotion render MyVideo out.mp4</pre><p>That’s it. Default codec is H.264, default quality is broadcast-fine. For a quick low-res preview while iterating: --scale=0.5.</p><p>For a single scene’s MP4 (great for sharing snippets while you’re still iterating): register each scene as its own composition in Root.tsx and render by id. I added per-scene compositions for exactly this reason.</p><h3>Things I wish I’d known on day one</h3><p><strong>1. The </strong><strong>* in </strong><strong>SELECT * will silently disappear from your code blocks</strong> if your tokenizer regex doesn&#39;t include it. Spent 20 minutes on this. The fix is one character: [*(),;=&lt;&gt;] instead of [(),;=].</p><p><strong>2. Composition IDs can only contain </strong><strong>[a-zA-Z0-9-].</strong> No dots, no underscores, no spaces. scene-3.4-pipeline will throw. scene-34-pipeline is fine.</p><p><strong>3. Don’t reach for tween libraries.</strong> interpolate(frame, [start, end], [from, to], { easing }) is enough for 90% of animations. spring() covers the other 10%. Adding Framer Motion fights the frame-by-frame model.</p><p><strong>4. Use Google Fonts via </strong><strong>@remotion/google-fonts</strong>, not @font-face. The CDN form fails to load mid-render about 5% of the time and leaves you debugging mysterious font fallbacks.</p><p><strong>5. Read the script aloud before generating audio.</strong> If you stumble, the TTS will too — except it can’t recover. Rewrite the line into something a robot wouldn’t choke on.</p><p><strong>6. The 0.4-second tail padding is opinionated.</strong> Adjust it. Tutorials feel less rushed at 0.6s. Reels need 0.0s. Test with your content.</p><h3>What to try next</h3><p>Once the basic loop works, the additions are cheap:</p><ul><li><strong>Captions</strong>: <a href="https://www.remotion.dev/docs/captions">@remotion/captions</a> — auto-generated from VO, professional accessibility for free</li><li><strong>Background music</strong>: drop another &lt;Audio&gt; with volume={0.15} under everything; remember to duck it during VO</li><li><strong>Scene transitions</strong>: &lt;TransitionSeries&gt; from @remotion/transitions — fade, slide, wipe between scenes</li><li><strong>Premium voice</strong>: swap macOS say for ElevenLabs in scripts/generate-voiceover.mjs. The rest of the pipeline doesn&#39;t change.</li></ul><h3>Closing</h3><p>I had three hours. I expected to spend them learning timeline shortcuts in After Effects. Instead I spent them writing JSX, which I already know. The stack — Remotion + macOS TTS + Mediabunny — is small, free at the entry point, and scales naturally to higher-quality voices when you outgrow say.</p><p>If you’ve been putting off making companion videos for your articles because the toolchain feels like a different career, this is the path. Article on Monday, video on Tuesday afternoon, both with the same author.</p><p>If you build something with this stack, drop a link in the comments — I want to see what other formats this opens up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebf24a2b22b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Stopped Reading.Claude Code and NotebookLM Read for Me.]]></title>
            <link>https://medium.com/@dhsoni2510/i-stopped-reading-claude-code-and-notebooklm-read-for-me-b9de7c94f2a9?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9de7c94f2a9</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[notebooklm]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Wed, 06 May 2026 19:59:02 GMT</pubDate>
            <atom:updated>2026-05-06T19:59:02.933Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Claude Code can now drive NotebookLM directly. Here’s the setup, the prompt, and what comes out the other side.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*plbdZZN9Lvs7abNHY7b68g.png" /></figure><p>I had forty-seven tabs open. Three of them were the same Stratechery post. Two were YouTube videos I’d “watch later” eleven months ago. The PDF in my downloads folder was titled final_v3_REAL_FINAL.pdf. I had a LinkedIn post due the next morning that was supposed to synthesize all of it.</p><p>So I stopped pretending I was going to read any of it. I gave the URLs to Claude Code, told it to use NotebookLM, and went to make tea. By the time I came back, I had a podcast, a slide deck, an infographic, a mind map, and a quiz — drawn from the actual sources, not hallucinated around them.</p><p>By the end of this post, you’ll have the same setup running locally and a single prompt that turns any reading list into shippable content.</p><blockquote><strong><em>TL;DR</em></strong></blockquote><ul><li>Claude Code becomes a NotebookLM controller via one skill install.</li><li>notebooklm-py is the unofficial CLI that exposes every Studio output.</li><li>First-time login uses Playwright; after that, cookies persist.</li><li>One prompt → podcast, video, slides, infographic, mind map, quiz.</li><li>You get formats the NotebookLM web UI doesn’t expose (PPTX, JSON mind maps, CSV tables).</li><li>Setup is roughly ninety seconds. The hard part is choosing what to read.</li></ul><h3>Why this workflow exists</h3><p>The constraint on knowledge work has flipped. Five years ago, the bottleneck was producing a coherent paragraph; today, large models will write you a usable paragraph in two seconds. The bottleneck now is <em>reading the right things first</em> — actually getting the source material into your head in a structured-enough form that you can have a real opinion about it. Most of us have a Pocket queue, a Readwise backlog, a folder of PDFs, and a YouTube “watch later” list that together represent maybe sixty hours of intake we’ll never do.</p><p>NotebookLM solved the comprehension half of this. You drop in sources, it grounds every answer in them, and the Studio panel turns them into podcasts and mind maps. What it didn’t solve was the <em>driving</em> — you still had to click through the UI for every notebook, every output, every download. This workflow closes that gap. Claude Code reads your prompt, the CLI talks to NotebookLM, and the artifacts land in your working directory while you’re doing something else.</p><h3>The 90-second setup</h3><p>Five steps. The first time you run it, budget about a minute and a half.</p><p><strong>1. Install uv if you don’t have it.</strong> uv is the Python tool runner we’ll use to install the CLI without polluting your global Python.</p><pre>curl -LsSf https://astral.sh/uv/install.sh | sh</pre><p><strong>2. Install the CLI with browser support.</strong> The [browser] extra pulls in Playwright, which we need for first-time Google login.</p><pre>uv tool install &quot;notebooklm-py[browser]&quot;<br>playwright install chromium</pre><p><strong>3. Log in once.</strong> This opens a Chromium window, you sign into Google normally, and the CLI captures the session cookie. After this, you won’t see a browser again.</p><pre>notebooklm login</pre><p><strong>4. Verify auth and check the supported commands.</strong></p><pre>notebooklm auth check --test</pre><p><strong>5. Install the agent skill into Claude Code.</strong> This drops a SKILL.md into ~/.claude/skills/notebooklm so Claude Code knows when and how to call the CLI without you spelling out every flag.</p><pre>notebooklm skill install</pre><p>That’s it. Open Claude Code in any directory and it now has NotebookLM as a tool.</p><h3>What’s actually happening under the hood</h3><p>Worth a paragraph because it explains the failure modes. NotebookLM has no public API. notebooklm-py is a community library that talks to the same internal endpoints the web app uses, authenticated with a cookie that Playwright captures during that one-time login. When Claude Code wants to add a source or generate a podcast, it shells out to notebooklm commands; the CLI makes the request; NotebookLM does the work; the artifact comes back. Two practical implications: (1) Google can change those endpoints any time and things will break until the maintainer pushes a fix, and (2) you&#39;re rate-limited to roughly what a heavy human user would hit. Don&#39;t try to process a thousand notebooks in a script. Do use it for the reading you&#39;d actually do anyway.</p><h3>The single prompt</h3><p>Here is the prompt I use. Drop it into Claude Code in a fresh directory after you’ve put your URLs and PDF paths into a file called sources.txt:</p><pre>Read sources.txt. Create a NotebookLM notebook called<br>&quot;$(date +%Y-%m-%d) intake&quot;. Add every line as a source —<br>URLs go in as URLs, .pdf paths go in as files, YouTube<br>links go in as YouTube. Wait for indexing.</pre><pre>Then generate, in parallel where possible:<br>  - an audio overview, deep-dive format, instructions:<br>    &quot;skeptical, no hype, name the disagreements&quot;<br>  - a slide deck, presenter format<br>  - an infographic, portrait orientation<br>  - a mind map<br>  - a 10-question quiz, hard difficulty<br>  - a data table comparing the key claims across sources</pre><pre>Download everything to ./out/ with sensible filenames.<br>When done, write me a 200-word brief in brief.md that<br>names the three strongest claims and the one I should<br>push back on.</pre><p>Claude Code will translate this into a sequence of notebooklm create, notebooklm source add, notebooklm generate ..., and notebooklm download ... calls. The whole thing takes about four minutes for a typical input of ten sources, most of which is NotebookLM doing the generation server-side. You can keep working in the meantime.</p><h3>What you actually get</h3><p>The ./out/ folder ends up looking like this:</p><pre>out/<br>├── podcast.mp3          # ~12 min, two hosts, grounded in your sources<br>├── slides.pdf           # presenter notes included<br>├── slides.pptx          # editable — the web UI won&#39;t give you this<br>├── infographic.png      # portrait, suitable for LinkedIn carousels<br>├── mindmap.json         # structured, importable into Obsidian/Excalidraw<br>├── quiz.md              # markdown — drop straight into Notion<br>├── data-table.csv       # claims × sources matrix<br>└── brief.md             # the 200-word synthesis Claude wrote</pre><p>The PPTX, the JSON mind map, and the CSV are the ones that change the math. NotebookLM’s web UI gives you a flat PDF for slides, a rendered mind map you can’t export, and quizzes locked inside an interactive viewer. The CLI exposes all of them as files you can pipe into your actual workflow, which is the whole point. A LinkedIn carousel is infographic.png plus three pulled slides. A team standup is brief.md plus the podcast link in Slack. A study guide is quiz.md plus the mind map.</p><h3>Things to know before you ship</h3><p>A few honest caveats:</p><p>The unofficial-API risk is real. The library has been stable for months at a stretch, but expect occasional days where a release breaks because Google shipped something. Pin your version in production scripts.</p><p>Quality scales with source quality. NotebookLM is grounded, which means it won’t make things up — but it also won’t fix bad inputs. A backlog of clickbait yields a podcast about clickbait. Curate.</p><p>The audio overviews are conversational by default, which means they have filler. The instructions parameter is the lever — &quot;skeptical,&quot; &quot;name disagreements,&quot; &quot;no hype&quot; all work. &quot;Make it sound like Tyler Cowen&quot; works surprisingly well too.</p><p>Don’t use this for anything you wouldn’t trust a smart, fast intern with. Grounded summaries of things you’ve curated: yes. Legal analysis, medical decisions, anything where being wrong is expensive: no.</p><h3>What to do now</h3><p>If you read this far, here’s the smallest possible next step. Pick five things from your reading backlog. Put them in a sources.txt. Run the setup, run the prompt. Twelve minutes from now, you&#39;ll have a podcast you can listen to on a walk and a brief you can decide whether to bother reading.</p><p>If the answer is “this was useful,” you’ve automated the hardest part of knowledge work. If the answer is “this missed the point of those sources,” good — that’s a signal to actually read them. Either way, you’ve gotten the backlog off your desk in an afternoon instead of carrying it for a year.</p><p>The forty-seven tabs are still there, by the way. I just don’t feel guilty about them anymore.</p><p><em>notebooklm-py is an unofficial community project. It’s not affiliated with Google or Anthropic. The repo is at github.com/teng-lin/notebooklm-py.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9de7c94f2a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Avoiding Window Functions in Spark, Spark 3.4 Just Changed the Rules]]></title>
            <link>https://medium.com/@dhsoni2510/stop-avoiding-window-functions-in-spark-spark-3-4-just-changed-the-rules-f83f09a4ab01?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f83f09a4ab01</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Sat, 25 Apr 2026 19:13:03 GMT</pubDate>
            <atom:updated>2026-04-25T19:13:03.258Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qvIqNKZGp-G2wB_UJqjGfQ.png" /></figure><h3>Why ROW_NUMBER() is now competitive with max_by, deterministic, and ready for multi-column logic</h3><blockquote><strong><em>Author’s note:</em></strong><em> A while back I wrote </em><a href="https://medium.com/@dhsoni2510/stop-using-row-number-for-deduplication-in-spark-use-max-by-instead-a69bc13c3dd0"><em>“Stop Using </em></a><a href="https://medium.com/@dhsoni2510/stop-using-row-number-for-deduplication-in-spark-use-max-by-instead-a69bc13c3dd0"><em>ROW_NUMBER() for Deduplication in Spark — Use </em></a><a href="https://medium.com/@dhsoni2510/stop-using-row-number-for-deduplication-in-spark-use-max-by-instead-a69bc13c3dd0"><em>max_by Instead&quot;</em></a><em>. That advice was correct for Spark &lt; 3.4 and a single ordering column. Spark 3.4 (and the Databricks Photon engine) shipped optimizer rules that change the math. This article is the follow-up-when the old advice still holds, when it doesn&#39;t, and what to use instead.</em></blockquote><p>If you’ve been writing Spark for more than a couple of years, you’ve probably internalized the rule: <em>“don’t use window functions for deduplication, they shuffle everything and sort everything, and they’ll blow up your executors.”</em> For a long time, that was correct. The standard trick was to rewrite this:</p><pre>-- The &quot;slow&quot; pattern<br>SELECT * FROM events<br>QUALIFY ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY event_ts DESC) = 1;</pre><p>…into this:</p><pre>-- The &quot;fast&quot; pattern<br>SELECT user_id, max_by(event_type, event_ts) AS latest_event<br>FROM events GROUP BY user_id;</pre><p>I wrote a whole article advocating for that swap. It worked. But that advice came with two quiet asterisks that most teams didn’t notice until they got bitten:</p><ol><li>max_by is <strong>non-deterministic</strong> when ordering values tie. That&#39;s a silent data-quality bug waiting to surface in an audit.</li><li>max_by only gives you one column back. If you actually want the <em>whole row, </em>which is what deduplication usually means, you have to pack everything into a struct or join back to the source table.</li></ol><p><strong>Spark 3.4 changed the equation.</strong> Three optimizer features in particular, WindowRankFilter (Top-K pruning), the QUALIFY clause, and replaceWindowsWithAggregatesclosed most of the performance gap that made max_by worth the trade-off. On Databricks, Photon&#39;s vectorized execution closes the rest. In modern environments, <strong>window functions are usually the right answer, </strong>for both performance <em>and</em> correctness.</p><p>Let’s walk through why.</p><h3>The historical bottleneck: shuffle-and-sort vs. incremental aggregation</h3><p>To understand what changed, you first have to understand what was actually slow.</p><p>A pre-3.4 window query like ROW_NUMBER() OVER (PARTITION BY key ORDER BY val) forced Spark into a fixed sequence:</p><ol><li><strong>Full shuffle</strong> of every input row, so all rows sharing a partition key land on the same executor.</li><li><strong>Full sort</strong> within each partition — O(N log N) — to assign row numbers.</li><li><strong>Filter</strong> down to rn = 1, throwing away ~99% of the work.</li></ol><p>When partitions exceeded executor memory, step 2 spilled to disk. This is the “double penalty” — shuffle every row, then sort every row.</p><p>max_by, introduced in Spark 3.0, sidestepped this entirely. It runs as a hash aggregate with <strong>map-side partial aggregation</strong>: each executor keeps only the running &quot;best&quot; row per key (O(1) state per group), and only those condensed results get shuffled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wFJuxdX_TPxSub4lixKCbA.png" /></figure><p>The performance gap was real:</p><p>Metric Window (pre-3.4) max_by Data movement Full shuffle (100% of rows) Partial agg → reduced shuffle Complexity O(N log N) (full sort) O(N) (incremental compare) Memory per group O(N) (spill risk) O(1) Determinism High (explicit tie-break) <strong>Low (arbitrary on ties)</strong> Row context preserved Yes No</p><p>That last column — determinism — is the part that got swept under the rug.</p><h3>What changed in Spark 3.4: WindowRankFilter (Top-K pruning)</h3><p>The single biggest change is an optimizer rule that recognizes a very common pattern:</p><pre>ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) = 1<br>-- or, more generally:<br>ROW_NUMBER() OVER (...) &lt;= K</pre><blockquote>Pre-3.4, Spark would faithfully sort the entire partition and then filter. The optimizer didn’t connect the dots that “you only want the top K, so you don’t actually need to sort everything below position K.”</blockquote><p>Spark 3.4’s Catalyst optimizer now does connect those dots. When it sees a ranking window followed by a &lt;= K filter (whether expressed via WHERE, QUALIFY, or a CTE), it replaces the Sort + Window operators with a specialized TakeOrderedAndProjectExec-style operator that maintains a <strong>bounded heap of size K</strong> per partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQWyZed6MXf-zxMzPPgudQ.png" /></figure><p>The complexity drops from O(N log N) to O(N log K). For deduplication where K=1, that&#39;s effectively O(N) — the same complexity as max_by, with none of the determinism trade-offs.</p><h3>The QUALIFY clause makes this easier to write</h3><p>Spark 3.4 also added the QUALIFY clause, which is the cleanest way to express &quot;filter on a window function&quot;:</p><pre>-- Old: nested subquery (works, but optimizer has more work to recognize the pattern)<br>SELECT * FROM (<br>  SELECT *, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ts DESC) AS rn<br>  FROM events<br>) WHERE rn = 1;</pre><pre>-- New (Spark 3.4+): QUALIFY<br>SELECT * FROM events<br>QUALIFY ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ts DESC) = 1;</pre><p>QUALIFY isn&#39;t just sugar — it makes the Top-K pattern more reliably detectable by the optimizer because the filter sits directly next to the window definition rather than being buried in a subquery.</p><h3>The determinism problem max_by never solved</h3><p>This is the part that doesn’t show up in benchmarks, and it’s the reason I’m walking back the original recommendation.</p><p>max_by(value, ordering_col) returns the value associated with the maximum ordering_col. When multiple rows tie on ordering_col, <strong>Spark picks one arbitrarily</strong>. There is no documented tie-breaking rule. The &quot;winner&quot; can change between runs, between cluster sizes, between Spark versions — anything that affects task scheduling can flip the result.</p><p>In high-volume event streams, ties on timestamps are <em>not</em> edge cases. Two events fired in the same millisecond, two records ingested in the same micro-batch, two CDC entries with identical commit times — these happen all the time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lt6flEp8Rgs-1cKxHZlMKQ.png" /></figure><p>Window functions let you specify multi-column tie-breakers explicitly:</p><pre>-- Deterministic: if event_ts ties, fall back to event_id<br>SELECT * FROM events<br>QUALIFY ROW_NUMBER() OVER (<br>  PARTITION BY user_id<br>  ORDER BY event_ts DESC, event_id DESC<br>) = 1;</pre><p>Trying to do this with max_by requires struct-packing every column you care about for tie-breaking into the ordering argument:</p><pre>-- The max_by equivalent — awkward, error-prone, hard to maintain<br>SELECT user_id,<br>       max_by(struct(event_type, payload, ...),<br>              struct(event_ts, event_id)) AS latest<br>FROM events GROUP BY user_id;</pre><p>…and you still only get one column back unless you packed <em>every</em> column of interest into the value struct. Which means more memory pressure, more serialization overhead, and SQL that nobody on your team wants to maintain.</p><p>For audit logs, regulatory reporting, ML feature engineering, or anywhere a downstream consumer expects the same input to produce the same output — <strong>non-determinism is a bug, not a quirk.</strong> The original article didn’t take that seriously enough. This one does.</p><h3>The other Spark 3.4 win: replaceWindowsWithAggregates</h3><p>There’s a second, quieter optimizer rule that landed in Spark 3.4 (and Databricks Runtime 15.3) called replaceWindowsWithAggregates. It catches the case where someone writes a window function but doesn&#39;t actually need windowing semantics — just a per-group total appended to each row.</p><pre>SELECT *,<br>       SUM(sales) OVER (PARTITION BY region) AS region_total<br>FROM sales_table;</pre><p>There’s no ORDER BY, no frame — this is &quot;really&quot; a group sum that happens to be expressed as a window. Catalyst now recognizes this and rewrites it into a hash-aggregate plan with map-side partial aggregation, the same shape max_by would produce.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w10af99gpmUY67qPBMudIw.png" /></figure><p>The point: <strong>you can write the more readable window syntax and still get the efficiency of a </strong><strong>groupBy.</strong> The optimizer takes care of the translation. This is the &quot;best of both worlds&quot; pattern Spark has been working toward for years.</p><h3>Photon: hardware-level acceleration on Databricks</h3><p>If you’re on Databricks, there’s a third lever: Photon. Photon is a vectorized C++ query engine that replaces parts of Spark’s JVM-based execution. For window functions specifically, it provides three benefits that the standard Spark engine doesn’t:</p><ol><li><strong>Vectorized (SIMD) execution</strong> — operations like LAG, LEAD, RANK, and arithmetic over window frames run on batches of values using AVX-512 instructions, instead of one row at a time.</li><li><strong>Off-heap memory management</strong> — partition buffers live outside the JVM heap, so window operations on huge partitions don’t trigger GC pauses or OOM the executor.</li><li><strong>Vectorized shuffle</strong> — when a shuffle is unavoidable, Photon’s native I/O path moves columnar batches at hardware bandwidth.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1vdd5xm3Z_zzFIXXhMySw.png" /></figure><p>The historical complaint that “window functions cause GC death spirals” largely doesn’t apply on Photon. The buffers that previously lived as Java objects on the heap now live as C++ arrays off-heap, invisible to the garbage collector.</p><h3>When window functions go beyond what max_by can do</h3><p>Deduplication is the use case where the two approaches overlap. Most window function use cases have no max_by equivalent at all:</p><pre>-- 7-day moving average — not expressible with max_by<br>SELECT date, sales,<br>       AVG(sales) OVER (<br>         PARTITION BY store_id<br>         ORDER BY date<br>         ROWS BETWEEN 6 PRECEDING AND CURRENT ROW<br>       ) AS rolling_7d_avg<br>FROM daily_sales;</pre><pre>-- Previous row&#39;s value (LAG) — not expressible with max_by<br>SELECT user_id, event_ts,<br>       LAG(event_ts, 1) OVER (PARTITION BY user_id ORDER BY event_ts)<br>         AS prev_event_ts<br>FROM events;</pre><pre>-- Top-3 (not top-1) per group — would need 3 separate max_by calls<br>SELECT * FROM products<br>QUALIFY ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) &lt;= 3;</pre><p>Spark 3.4 also added <strong>WindowSpec reuse</strong>: if multiple window functions in the same query share the same PARTITION BY / ORDER BY, Spark performs the shuffle and sort exactly once and reuses the result across all functions. This is a big deal for feature engineering pipelines that compute 10-20 windowed features over the same partitioning.</p><h3>The decision framework for Spark 3.4+</h3><p>So when do you use what? Here’s how I think about it now:</p><p><strong>Default to window functions when:</strong></p><ul><li>You need <strong>deterministic results</strong> (audit, regulatory, ML feature stores)</li><li>You need <strong>the full row</strong>, not just one column</li><li>You’re computing <strong>Top-N where N &gt; 1</strong>, ranks, percent-ranks, dense-ranks</li><li>You’re doing <strong>time-series work</strong> — moving averages, running totals, LAG/LEAD</li><li>You have <strong>multi-column tie-breakers</strong></li><li>You have <strong>multiple windowed expressions sharing the same partitioning</strong></li></ul><p><strong>Reach for </strong><strong>max_by only when:</strong></p><ul><li>You’re doing <strong>pure single-column deduplication</strong> at extreme scale where the network is genuinely the bottleneck</li><li>Non-determinism on ties is <strong>acceptable</strong> for the use case (e.g., approximate analytics, exploratory work)</li><li>You can verify that the partial aggregation actually reduces shuffle volume meaningfully (high-cardinality partition keys won’t reduce much)</li></ul><p><strong>Stop using </strong><strong>max_by when:</strong></p><ul><li>The downstream consumer expects reproducible results</li><li>You’d need to pack multiple columns into a struct just to get the row back</li><li>You’re on Spark 3.4+ and the runtime difference is in single-digit minutes — that’s not worth the data-quality risk</li></ul><h3>Migrating from the old max_by pattern</h3><p>If you have existing pipelines using the max_by pattern, the migration is mechanical. Here&#39;s the before/after:</p><pre>-- Before: max_by pattern (the recommendation in my old article)<br>SELECT user_id,<br>       max_by(struct(event_type, payload, session_id), event_ts) AS latest<br>FROM events<br>GROUP BY user_id;</pre><pre>-- After: window pattern with deterministic tie-breaking<br>SELECT user_id, event_type, payload, session_id, event_ts<br>FROM events<br>QUALIFY ROW_NUMBER() OVER (<br>  PARTITION BY user_id<br>  ORDER BY event_ts DESC, event_id DESC  -- explicit tie-break<br>) = 1;</pre><p>A few things to validate after the migration:</p><ol><li><strong>Confirm </strong><strong>QUALIFY is supported</strong> in your runtime. Spark 3.4+ and Databricks Runtime 13.3+ have it. Older runtimes need the subquery form.</li><li><strong>Check the physical plan</strong> for TakeOrderedAndProjectExec (or the equivalent Top-K operator). If you see SortExec followed by WindowExec followed by FilterExec, the optimizer didn&#39;t pick up the pattern — usually a sign the filter isn&#39;t expressed cleanly enough. Move it into QUALIFY or a single-level subquery.</li><li><strong>Add the explicit tie-breaker column</strong> to your ORDER BY. This is the entire point of moving away from max_by — don&#39;t skip it.</li></ol><h3>Closing</h3><p>The original advice — “use max_by instead of ROW_NUMBER() for deduplication&quot; — was right for the engine that existed at the time. Spark 3.4 isn&#39;t that engine anymore.</p><p>The combination of WindowRankFilter, QUALIFY, replaceWindowsWithAggregates, and (on Databricks) Photon&#39;s vectorized execution means window functions now give you almost all the performance of max_by <em>and</em> keep the things you actually wanted: determinism, full row context, multi-column ordering, and SQL that reads the way the business logic actually works.</p><p>If you only remember one thing: <strong>on Spark 3.4+, write the window function. Add an explicit tie-breaker. Move on.</strong></p><p><em>If you have benchmarks of your own from migrating off </em><em>max_by, I&#39;d love to see them in the comments — particularly cases where the gap is wider or narrower than what I&#39;ve described here.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f83f09a4ab01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using ROW_NUMBER() for Deduplication in Spark — Use max_by() Instead]]></title>
            <link>https://medium.com/@dhsoni2510/stop-using-row-number-for-deduplication-in-spark-use-max-by-instead-a69bc13c3dd0?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a69bc13c3dd0</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[performance-optimization]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 15:19:17 GMT</pubDate>
            <atom:updated>2026-03-28T15:23:43.962Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How a simple aggregate function can replace an expensive shuffle+sort window operation — and why its algebraic properties make it even more powerful than you think.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*cAeYpRz6fIPDx6z4PDlQIA.png" /></figure><p>If you’ve worked with Spark at any enterprise scale, you’ve written this pattern at least a hundred times: <em>“Give me the latest record per customer.”</em> And you’ve probably reached for the same weapon every time — ROW_NUMBER() with a window partition, filtered to rn = 1.</p><p>It works. But it’s one of the most expensive things you can ask Spark to do. In this article, I’m going to show you why max_by() is almost always the better choice, how it works under the hood, and why its <strong>commutative</strong>, <strong>associative</strong>, and <strong>cumulative</strong> properties make it a first-class citizen in distributed computation.</p><h3>The Classic Pattern: Window + ROW_NUMBER</h3><p>Let’s say you have an orders table with millions of rows, and you want the most recent order per customer. The textbook approach looks like this:</p><pre>-- ❌ The ROW_NUMBER() pattern: expensive shuffle + sort<br>SELECT *<br>FROM (<br>  SELECT *,<br>    ROW_NUMBER() OVER (<br>      PARTITION BY customer_id<br>      ORDER BY order_date DESC<br>    ) AS rn<br>  FROM orders<br>)<br>WHERE rn = 1</pre><p>This is correct. But let’s dissect what Spark actually has to do to execute it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*p5hYeDRvnu5kPdM12Vv_7A.png" /></figure><p><em>The ROW_NUMBER() approach forces a full shuffle + sort, processing every single row through 5 stages just to keep one per group. 99.6% of the work is wasted.</em></p><p>That’s <strong>five stages</strong>, two of which — shuffle and sort — are the most expensive operations in distributed computing. Every single row travels across the network, gets sorted, assigned a row number, and then… 99.6% of them get thrown away.</p><blockquote><strong><em>⚠️ The Hidden Cost of Data Skew</em></strong></blockquote><blockquote><em>If one customer has 10 million orders and most have only 5, the window function must sort all 10 million rows in a single partition. This creates a classic </em><strong><em>data skew</em></strong><em> problem — one executor does all the work while the rest sit idle. Spark cannot partially aggregate window functions.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*o3Rp51RJ9pzgp329_ZqwPQ.png" /></figure><p><em>Window functions cannot partially aggregate, so the entire skewed partition must be sorted on a single executor. This is the #1 cause of OOM in deduplication jobs.</em></p><h3>The Better Way: max_by()</h3><p>Spark 3.0+ introduced max_by(value, ordering) — an aggregate function that returns the value associated with the maximum ordering. Here&#39;s the same query, rewritten:</p><pre>-- ✅ The max_by() pattern: partial aggregation, no sort<br>SELECT<br>  customer_id,<br>  max_by(order_id,    order_date) AS order_id,<br>  max_by(product,     order_date) AS product,<br>  max_by(amount,      order_date) AS amount,<br>  max(order_date)                 AS order_date<br>FROM orders<br>GROUP BY customer_id</pre><p>That’s it. No subquery, no window, no filter. Let’s look at what Spark does now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*hEfBMRTJIEXNSFsEpVB0Yg.png" /></figure><p><em>max_by() as an aggregate function benefits from partial (map-side) aggregation, reducing 500M rows to ~6M before any data crosses the network. That’s a 27x reduction in shuffle I/O.</em></p><p>The key difference: because max_by() is an <strong>aggregate function</strong> (not a window function), Spark can apply <strong>partial aggregation</strong> on the map side. That means each executor reduces its local data down to one row per customer <em>before</em> any data crosses the network.</p><h3>How max_by() Processes Your Data — Step by Step</h3><p>Let’s trace through a concrete example. Suppose customer C42 has 6 orders scattered across the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1cJpx_sp5kIzGq7XI62zQ.png" /></figure><p>Side-by-Side: The Real Difference</p><p><strong>ROW_NUMBER() Window ❌</strong></p><ul><li>✗ Full shuffle of ALL rows</li><li>✗ Requires O(n log n) sort</li><li>✗ No partial aggregation</li><li>✗ Skew causes OOM errors</li><li>✗ 5 execution stages</li><li>✗ Cannot be combined</li></ul><p><strong>max_by() Aggregate ✅</strong></p><ul><li>✓ Shuffle only partial results</li><li>✓ O(n) — just track the max</li><li>✓ Partial (map-side) aggregation</li><li>✓ Handles skew gracefully</li><li>✓ 3 execution stages</li><li>✓ Commutative + associative</li></ul><h3>Benchmark: What Does “Faster” Actually Mean?</h3><p>Representative performance numbers from a Databricks cluster running on a 500M-row event table with ~2M unique customers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K11CzJEUJ8s1BOPIEiq3-g.png" /></figure><p>The shuffle reduction is the headline number. When you move from 48 GB of shuffle to under 2 GB, you’re not just faster — you’re freeing up cluster resources for other jobs, reducing cloud costs, and eliminating the primary cause of OOM failures in your pipeline.</p><h3>The Algebraic Properties That Make max_by() Special</h3><p>This is where it gets interesting. The reason max_by() can be partially aggregated and parallelized so effectively comes down to three mathematical properties. These aren&#39;t just academic — they directly determine how Spark&#39;s Catalyst optimizer can rearrange and optimize the execution plan.</p><h3>1. Commutativity — f(A, B) = f(B, A)</h3><p>The order in which you process records doesn’t affect the result. Whether you compare row A with row B first, or row B with row A, you get the same winner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*woY24O_Xd-CzgaJTVQoDiA.png" /></figure><p><strong>Why it matters:</strong> Spark can read partitions in any order without coordination between executors.</p><h3>2. Associativity — f(A∪B) = merge(f(A), f(B))</h3><p>You can group and sub-group the computation however you like. This is the property that enables <strong>partial aggregation</strong> — Spark can compute local winners on each executor, then merge winners across executors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*anWEGHfvavsmAPZlKB7SdQ.png" /></figure><p><em>Because max_by() is associative, Spark can compute local winners first, then only shuffle 3 rows instead of 8. At scale, this is the difference between shuffling 1 billion rows and 1 million.</em></p><p>In formal terms: max_by(A ∪ B, ord) = merge(max_by(A, ord), max_by(B, ord)). You can split the data any way you want, compute partial results independently, and merge them. This is exactly the pattern Spark uses for its <strong>map-side combine → shuffle → reduce</strong> pipeline.</p><h3>3. Cumulative Property (Incremental Computation) — f(old ∪ new) = merge(f(old), f(new))</h3><p>Here’s a property that becomes crucial in <strong>streaming</strong> and <strong>Delta Lake incremental pipelines</strong>: you can incrementally update a max_by() result without recomputing from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*w0vkK6BNtOXtfNZTy2-wwg.png" /></figure><p>When new data arrives, you don’t need to re-scan the entire table. You just compare the new batch’s max_by() result against the existing stored result and keep the winner. This makes max_by() a natural fit for <strong>Structured Streaming</strong> and <strong>Delta Live Tables</strong>.</p><blockquote><strong><em>💡 Why ROW_NUMBER() can’t do this</em></strong></blockquote><blockquote><em>Window functions are </em><strong><em>not associative</em></strong><em>. You cannot compute </em><em>ROW_NUMBER() on a subset and merge it with another subset — the numbering depends on the entire sorted partition being present. This means every incremental update requires a full recomputation of the entire window.</em></blockquote><h3>Full Practical Example on Databricks</h3><p>Let’s see both approaches on a realistic dataset. Suppose we have a customer_events table in Unity Catalog with event tracking data and we want the latest event per customer with all its details.</p><pre>-- ❌ The Old Way — Window (expensive shuffle + sort)<br>WITH ranked AS (<br>  SELECT *,<br>    ROW_NUMBER() OVER (<br>      PARTITION BY customer_id<br>      ORDER BY event_timestamp DESC<br>    ) AS rn<br>  FROM catalog.schema.customer_events<br>)<br>SELECT<br>  customer_id, event_type, event_timestamp,<br>  page_url, device, session_id, geo_region<br>FROM ranked<br>WHERE rn = 1</pre><pre>-- ✅ The New Way — max_by (partial aggregation, no sort)<br>SELECT<br>  customer_id,<br>  max_by(event_type,      event_timestamp) AS event_type,<br>  max(event_timestamp)                     AS event_timestamp,<br>  max_by(page_url,        event_timestamp) AS page_url,<br>  max_by(device,          event_timestamp) AS device,<br>  max_by(session_id,      event_timestamp) AS session_id,<br>  max_by(geo_region,      event_timestamp) AS geo_region<br>FROM catalog.schema.customer_events<br>GROUP BY customer_id</pre><h3>PySpark Equivalent</h3><pre>from pyspark.sql import functions as F</pre><pre>df = spark.table(&quot;catalog.schema.customer_events&quot;)</pre><pre>result = (<br>    df.groupBy(&quot;customer_id&quot;)<br>    .agg(<br>        F.max_by(&quot;event_type&quot;,  &quot;event_timestamp&quot;).alias(&quot;event_type&quot;),<br>        F.max(&quot;event_timestamp&quot;).alias(&quot;event_timestamp&quot;),<br>        F.max_by(&quot;page_url&quot;,    &quot;event_timestamp&quot;).alias(&quot;page_url&quot;),<br>        F.max_by(&quot;device&quot;,      &quot;event_timestamp&quot;).alias(&quot;device&quot;),<br>        F.max_by(&quot;session_id&quot;,  &quot;event_timestamp&quot;).alias(&quot;session_id&quot;),<br>        F.max_by(&quot;geo_region&quot;,  &quot;event_timestamp&quot;).alias(&quot;geo_region&quot;),<br>    )<br>)</pre><pre>result.display()</pre><h3>When You Still Need ROW_NUMBER()</h3><p>To be fair, there are cases where max_by() won&#39;t work:</p><p><strong>Top-N per group (N &gt; 1):</strong> If you need the 3 most recent orders per customer, max_by() only gives you the single best. You&#39;ll need a window function or LATERAL VIEW explode with slice(sort_array(collect_list(...))).</p><p><strong>Tie-breaking with secondary sorts:</strong> If your ordering column has ties and you need a deterministic tiebreaker (e.g., ORDER BY event_timestamp DESC, event_id DESC), you&#39;d need to construct a composite ordering — for instance, using struct(event_timestamp, event_id) as the ordering argument, since Spark compares structs field-by-field.</p><p><strong>Access to the rank itself:</strong> If downstream logic needs the actual row number value (not just the top row), a window function is the only option.</p><blockquote><strong><em>💡 The struct trick for tie-breaking</em></strong></blockquote><blockquote><em>You can handle multi-column ordering with </em><em>max_by() by passing a </em><em>struct as the ordering argument:</em></blockquote><blockquote><em>max_by(value_col, struct(timestamp_col, id_col))</em></blockquote><blockquote><em>Spark compares structs lexicographically, so this gives you </em><em>ORDER BY timestamp DESC, id DESC semantics without any sort operation.</em></blockquote><h3>TL;DR — Decision Framework</h3><p><em>In the vast majority of deduplication scenarios (N=1), max_by() is the correct choice. Reserve ROW_NUMBER() for Top-N and rank-access use cases.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*23hmkSdxBLkWwH3kqif3aA.png" /></figure><h3>Final Thoughts</h3><p>The ROW_NUMBER() WHERE rn = 1 pattern is one of the most common performance anti-patterns I see on Databricks. It&#39;s not wrong — it&#39;s just wildly overqualified for the job. You&#39;re deploying a full sort to find a maximum, like using a bulldozer to pick a flower.</p><p>max_by() is purpose-built for this exact problem. Its algebraic properties — commutativity, associativity, and incremental computability — aren&#39;t just theoretical niceties. They&#39;re the reason Spark can execute it with partial aggregation, minimal shuffling, and zero sorting. That translates directly into faster jobs, lower costs, and pipelines that don&#39;t fall over at scale.</p><p>Next time you write ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) = 1, stop. Reach for max_by() instead. Your cluster will thank you.</p><p><em>Apache Spark · Databricks · Performance · SQL · Data Engineering · max_by · Window Functions</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a69bc13c3dd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Trap of Delta Lake Time Travel: Why Your Structured Streaming Queries Keep Crashing]]></title>
            <link>https://medium.com/@dhsoni2510/the-trap-of-delta-lake-time-travel-why-your-structured-streaming-queries-keep-crashing-dd12f7013868?source=rss-12fc7e4bb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd12f7013868</guid>
            <category><![CDATA[structured-streaming]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[optimize]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Dharmik Soni]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 03:56:30 GMT</pubDate>
            <atom:updated>2026-03-06T03:56:30.637Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Hidden Trap of Delta Lake Time Travel: Why Your Structured Streaming Queries Keep Crashing</h3><p><strong>Understanding the real relationship between VACUUM, startingTimestamp, and Parquet files in Databricks.</strong></p><p>If you are building real-time data pipelines on Databricks with Spark Structured Streaming, you have likely encountered this baffling scenario.</p><p>You want to start a new stream from a historical point in your Delta table. You know your data retention is 7 days, but you decide to set your startingTimestamp to <strong>10 days ago</strong>.</p><p>Before launching, you do a quick sanity check:</p><pre>SELECT * FROM my_table WHERE date = &#39;10-days-ago&#39;</pre><p>The query runs perfectly! The data is right there. So you confidently launch your stream.</p><p>But instead of processing events, the cluster immediately crashes with a dreaded FileNotFoundException.</p><p><strong>How is this possible?</strong> If you can SELECT the data, why can’t Structured Streaming read it? The answer lies in how Delta Lake separates <strong>data</strong> from <strong>history</strong> — and how the “cleanup crew” (VACUUM) actually works.</p><h3>1. The Tale of Two Layers: Logs vs. Parquet</h3><p>Every Delta table you interact with is actually split into two distinct parts:</p><p><strong>The Transaction Log (_delta_log/)</strong> is the “Brain” of the table. It keeps a JSON record of every single change — every commit — that ever happened. By default, Delta retains these log entries for <strong>30 days</strong>.</p><p><strong>The Data Files (.parquet)</strong> are the “Muscle.” These are the physical data files sitting in your cloud storage bucket (S3/ADLS/GCS).</p><blockquote><strong><em>🔑 Key Insight:</em></strong><em> Structured Streaming relies </em>entirely<em> on the Transaction Log to figure out exactly which Parquet files it needs to read to recreate the past. It replays history commit by commit.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCoTSmvAwYhMHL3oKRF4Zw.png" /></figure><h3>2. Active Files vs. Tombstoned Files (The Garbage Bin)</h3><p>To understand why the stream crashes, we have to categorize Parquet files into two buckets: <strong>Active</strong> and <strong>Tombstoned</strong>.</p><p>Imagine you ingested records 10 days ago as tiny 10MB Parquet files.</p><ul><li><strong>Active Files:</strong> If those files were never updated, deleted, or optimized, they are still considered “Active.” The transaction log says they are part of the current table state.</li><li><strong>Tombstoned Files:</strong> If you run an OPTIMIZE command (which compacts small files into larger, efficient files), Delta creates brand new files. The original 10MB files from 10 days ago are marked as &quot;tombstoned&quot; — logically deleted from the current state, but kept in storage for time travel.</li></ul><blockquote><strong><em>💡 Golden Rule:</em></strong><em> OPTIMIZE does NOT delete data. It moves data from lots of small files into big files, and throws the original small files into the “Trash Bin” (Tombstoned state).</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4LcnkNffjHohhFVUmt8kw.png" /></figure><h3><strong>3. Enter VACUUM — The Cleanup Crew</strong></h3><p>If you tombstone files every night, your AWS/Azure storage bill will skyrocket. This is where VACUUM comes in.</p><p>VACUUM is purely a garbage collector. It looks at the Trash Bin (tombstoned files) and <strong>physically deletes</strong> any file that has been sitting there longer than your defined retention period (default 7 days).</p><blockquote><strong><em>⚠️ Crucial Distinction:</em></strong><em> VACUUM physically deletes the </em><strong><em>Data Files (.parquet)</em></strong><em>, but it does NOT delete the </em><strong><em>Transaction Log</em></strong><em> entries for that time period. VACUUM also never touches Active files, no matter how old they are.</em></blockquote><p>This explains why you can still SELECT 10-day-old data! As long as the data is safely sitting inside the optimized &quot;Large File C,&quot; VACUUM ignores it. It&#39;s an Active file — totally safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7XGxjVKecQKICK_jOovMxg.png" /></figure><h3>4. The Anatomy of the Crash</h3><p>So, if the data is safely sitting in “File_C,” why does my Stream crash when I set startingTimestamp back to Day 1?</p><p>When you give Spark a startingTimestamp, you are <strong>not</strong> just saying &quot;fetch the old data.&quot; You are telling Spark to operate like a <strong>DVR replaying history</strong>. It is trying to reconstruct exactly what the table looked like 10 days ago, and play every single commit forward from that exact moment.</p><p>Here is the failure chain:</p><ol><li>Streaming asks the Transaction Log: <em>“What files existed on Day 1?”</em></li><li>The Log (retained for 30 days) happily responds: <em>“Read File_A and File_B to reconstruct Day 1.”</em></li><li>Streaming reaches out to Cloud Storage (S3) to fetch File_A and File_B.</li><li><strong>But File_A and File_B were tombstoned by OPTIMIZE on Day 2… and physically deleted by VACUUM on Day 9.</strong></li><li>S3 returns: ❌ 404 FileNotFoundException</li><li><strong>Stream crashes.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4V47BexkZxbYftNEziUVHw.png" /></figure><p>Spark does not care that the data from File_A currently resides perfectly safely inside File_C.</p><p>Because Structured Streaming is strictly replaying the Day 1 commit, it <strong>absolutely requires</strong> the historical, physical File_A to process the “insert” event. Because VACUUM shredded that file 7 days after it was tombstoned, the stream cannot replay history.</p><h3>5. The Differing Dials: Why Default Settings Trick You</h3><p>This is the core of the trap. Two different retention settings work against each other:</p><ul><li><strong>Transaction Log retention:</strong> 30 days (default)</li><li><strong>VACUUM retention:</strong> 7 days (default)</li></ul><p>The log <em>tells</em> Structured Streaming that history exists for up to 30 days. But VACUUM has already <em>destroyed</em> the physical files that history refers to after just 7 days. That <strong>23-day gap</strong> between Day 7 and Day 30 is where your pipeline dies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0bwylOMoUivHfMtsL6jvcw.png" /></figure><blockquote><em>The Transaction Log is a map to buried treasure that has already been incinerated.</em></blockquote><h3>6. The Exception to the Rule</h3><p>There <em>is</em> technically one scenario where you can successfully read past the VACUUM retention marker with Structured Streaming:</p><p>If you have a fundamentally <strong>append-only</strong> table where you <strong>NEVER</strong> run OPTIMIZE, UPDATE, DELETE, or MERGE.</p><p>In this scenario, files are never tombstoned. Because no files ever enter the “Trash Bin,” VACUUM finds zero files to delete. You can time-travel all the way back to your Log Retention limit (30 days by default).</p><p>However, in the real world, almost all production tables have background OPTIMIZE jobs running to improve query performance. Relying on this exception is essentially playing Russian Roulette with your pipeline.</p><h3>7. Best Practices for Databricks Developers</h3><p><strong>Rule #1: The Golden Rule</strong> Never set a streaming startingTimestamp older than your VACUUM retention period on tables that use OPTIMIZE, DELETE, or UPDATE.</p><p><strong>Rule #2: Beware of Defaults</strong> Your stream log retention is 30 days, but your VACUUM retention is 7 days. These differing dials create the illusion that you can travel back further than you actually can.</p><p><strong>Rule #3: Use Checkpoints, Not Timestamps</strong> Do not rely on manual startingTimestamp management. Rely on native Structured Streaming checkpoints to track offsets. If a stream crashes and stays down for 10 days, checkpoints ensure safe resumption (as long as underlying files haven&#39;t been vacuumed).</p><p><strong>Rule #4: Extend Retention for Backfills</strong> If you know you need to routinely restart streams from 14 days ago, alter your table properties so VACUUM spares those files longer:</p><pre>ALTER TABLE db.my_table<br>SET TBLPROPERTIES (<br>  &#39;delta.deletedFileRetentionDuration&#39; = &#39;15 days&#39;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9kGYpcxjOfqJIABz7fKt7w.png" /></figure><h3>TL;DR</h3><p><strong>SELECT queries</strong> read the <em>current</em> table state (Active files). <strong>Structured Streaming</strong> replays <em>historical</em> commits (needs Tombstoned files). VACUUM destroys Tombstoned files after 7 days. The Log remembers for 30 days. That 23-day gap is where your pipeline dies.</p><blockquote>Time travel is a powerful feature in Databricks, but if the garbage collector has already destroyed your past, some timelines are better left unvisited.</blockquote><p><strong>Tags:</strong> Databricks · Apache Spark · Delta Lake · Structured Streaming · Data Engineering · VACUUM</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd12f7013868" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>