<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hansi on Medium]]></title>
        <description><![CDATA[Stories by Hansi on Medium]]></description>
        <link>https://medium.com/@dhulanjala?source=rss-43348a0591b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RhTc9S1Elt5h3P5umDikMQ.jpeg</url>
            <title>Stories by Hansi on Medium</title>
            <link>https://medium.com/@dhulanjala?source=rss-43348a0591b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:49:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhulanjala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Two Failure Models, One System: A Simple Question That Decides How You Handle Errors in Distributed…]]></title>
            <link>https://medium.com/@dhulanjala/two-failure-models-one-system-a-simple-question-that-decides-how-you-handle-errors-in-distributed-2d9cd3b198cc?source=rss-43348a0591b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d9cd3b198cc</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[awsserverless]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-athena]]></category>
            <dc:creator><![CDATA[Hansi]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 10:48:27 GMT</pubDate>
            <atom:updated>2026-09-05T10:48:27.045Z</atom:updated>
            <content:encoded><![CDATA[<h3>Two Failure Models, One System: A Simple Question That Decides How You Handle Errors in Distributed Architecture</h3><p>Why “is someone waiting for a response right now?” is the only question that matters when you’re deciding how a piece of your system should fail.</p><h3>The question I kept asking myself</h3><p>While building an event-driven data lake on AWS — an ingest API feeding Kinesis Firehose into S3, a scheduled Glue ETL job producing queryable Parquet, an Athena-backed query API on top — I kept running into the same decision, over and over, for every single component: <strong>when this fails, what should actually happen?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6wwTkkSczbdPelSCM-AVzA.png" /></figure><p>The obvious answer — “add a retry, add a dead-letter queue, log it somewhere” — is true often enough to be dangerous. Applied indiscriminately, it either adds infrastructure nobody needed or, worse, misses the one place where a caller is sitting there, waiting, with no idea anything went wrong.</p><p>Eventually I noticed the decision always came down to one question, asked before anything about retries or queues or logging: <strong>is someone waiting for a response right now, or not?</strong></p><p>Everything else follows from the answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9ACauZX7yvZSYDHiczbGQ.png" /></figure><h3>Case 1: someone’s waiting</h3><p>ingestEventFn is invoked synchronously through API Gateway. A client sent an HTTP request and is holding a connection open, waiting for a status code. If the call to Kinesis Firehose inside that handler fails — throttled, misconfigured, temporarily unreachable — the correct response isn&#39;t a queue, a retry policy, or a background alert. It&#39;s simple: <strong>tell the caller, right now, honestly.</strong></p><pre>try {<br>  await firehose.send(new PutRecordCommand({<br>    DeliveryStreamName: STREAM_NAME,<br>    Record: { Data: Buffer.from(JSON.stringify(record) + &quot;\n&quot;) },<br>  }));<br>} catch (err) {<br>  console.error(&quot;Firehose PutRecord failed&quot;, err);<br>  return {<br>    statusCode: 502,<br>    body: JSON.stringify({ error: &quot;Failed to accept event for processing. Please retry.&quot; }),<br>  };<br>}</pre><p>That’s the entire fault-tolerance story for this component. No dead-letter queue. No retry infrastructure. The caller is already positioned to receive the bad news the moment it happens and decide what to do about it — retry, back off, surface it to a user. Adding a DLQ here wouldn’t add safety. It would add a second place failures could hide, watched by nobody, duplicating information the caller already has.</p><p>The mistake I wanted to avoid was the version of this code with <em>no</em> try/catch — where an unhandled exception propagates out of the handler, Lambda marks the invocation failed, and API Gateway returns some generic, unstyled 502 that gives the caller no useful information at all. Technically &quot;the caller finds out,&quot; but in the ugliest, least actionable way possible. Synchronous failure handling isn&#39;t optional just because nobody needs a queue — it still needs to be a <em>clean, honest, typed response</em>, not silence dressed up as an accident.</p><h3>Case 2: nobody’s waiting</h3><p>triggerEtlFn is invoked by EventBridge on an hourly schedule. There is no caller. Nobody is holding a connection open. If this function throws, the only way anyone will ever find out is if <em>something else</em> is specifically watching for that failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KqzysbPEASAaeYp4suk2Qw.png" /></figure><p>This is the case that actually needs a dead-letter mechanism — not because dead-letter queues are generally good practice, but because this is precisely the failure mode they exist to solve: work that happens with no one present to receive a synchronous answer.</p><pre>const triggerEtlFn = new lambda.NodejsFunction(this, &quot;TriggerEtlFn&quot;, {<br>  entry: &quot;../src/lambdas/triggerEtl.ts&quot;,<br>  handler: &quot;handler&quot;,<br>  onFailure: new destinations.SqsDestination(etlFailureQueue),<br>});</pre><p>One line of CDK. But the reasoning behind reaching for it is the entire point — it’s not a default I apply everywhere. It’s the specific answer to a specific shape of problem: asynchronous invocation, no synchronous observer, failure that would otherwise vanish silently into CloudWatch logs nobody’s tailing in real time.</p><p>The same logic extended to two more components in the same system, each independently: a Glue ETL job failing has no caller either — so it gets a CloudWatch state-change rule watching for its FAILED status, feeding an alert. A Glue Crawler failing is the same shape again — another state-change rule, same pattern, different resource.</p><h3>The mistake this framework prevents</h3><p>I’d made a version of exactly this mistake before, on a different project — an Azure Durable Functions pipeline where a queue-triggered retry handler called startNew() on a failed orchestration and returned immediately, without waiting to see whether the retry actually succeeded. The handler was, in effect, treating an asynchronous operation as if someone were synchronously waiting on its outcome — reporting success the instant it <em>scheduled</em> work, not when that work actually finished. The queue&#39;s own retry mechanics, which depended on an honest failure signal, never fired, because the function never gave one.</p><p>That bug and this framework are the same insight, approached from opposite directions. There, I’d accidentally treated async work as if it had a synchronous observer. Here, the fix is to explicitly ask, for every component, which of the two situations it’s actually in — and let that answer, not habit or convenience, decide whether it needs a try/catch and a clean response, or a destination queue and a state-change alert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcheK2KVljEg-KPi-ir2-w.png" /></figure><h3>Why this is worth naming explicitly</h3><p>It would have been easy to build this data lake by pattern-matching: “AWS pipelines usually have DLQs, so I’ll add one everywhere,” or the opposite mistake, “this is a portfolio project, error handling can be an afterthought.” Both produce a system that <em>looks</em> fault-tolerant without actually being coherent about <em>why</em> each piece is handled the way it is — and coherence is what falls apart first under real failure, when you’re trying to reason about what actually happened at 2am.</p><p>The one-question framework doesn’t require memorizing which AWS service pattern to use where. It requires knowing, for any given piece of code you’re about to write, whether there’s a synchronous observer on the other end of it. Everything downstream — try/catch vs. DLQ, honest error response vs. state-change alert — falls out of that single fact.</p><h3>Key takeaways</h3><ul><li><strong>Ask “is someone waiting right now?” before deciding how a component should fail.</strong> The answer determines everything else — it’s a cause, not a checklist item alongside other choices.</li><li><strong>Synchronous failure still needs real handling, just not a queue.</strong> A try/catch returning a clean, typed error response is the whole fault-tolerance story for a component with a live caller.</li><li><strong>Asynchronous failure needs external observation, because nothing else will notice.</strong> Dead-letter queues, Lambda destinations, and state-change alerts all solve exactly this — and only this.</li><li><strong>Applying the wrong model to the right-shaped problem is a real, specific bug class</strong> — not a style preference. Treating async work as if someone’s synchronously waiting on it is exactly what silently broke a retry system I built on a different cloud, on a different platform, for a completely different reason.</li><li><strong>Consistency of reasoning matters more than consistency of pattern.</strong> A system where every component’s failure handling can be explained by the same one-sentence rule is more trustworthy than one where every component “does error handling,” each for a slightly different, unexamined reason.</li></ul><p>The full project — ingest API, Glue ETL pipeline, Athena query layer, and the fault-tolerance design discussed here —</p><p><a href="https://github.com/dhulanjala/aws-event-data-lake">https://github.com/dhulanjala/aws-event-data-lake</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d9cd3b198cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debugging a Silent Failure in Azure Durable Functions: Why startNew() Isn't Enough]]></title>
            <link>https://medium.com/@dhulanjala/debugging-a-silent-failure-in-azure-durable-functions-why-startnew-isnt-enough-0505cd13feca?source=rss-43348a0591b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/0505cd13feca</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Hansi]]></dc:creator>
            <pubDate>Wed, 22 Jul 2026 10:40:59 GMT</pubDate>
            <atom:updated>2026-07-22T10:40:59.080Z</atom:updated>
            <content:encoded><![CDATA[<p>How a 33-millisecond function duration exposed a fundamental misunderstanding about fire-and-forget in distributed retry systems.</p><h3>The setup</h3><p>I recently built a serverless document-processing pipeline on Azure — Blob Storage for uploads, Durable Functions for orchestration, Cosmos DB for indexing. Nothing exotic: a file lands in a container, an orchestration extracts metadata, computes a checksum, and indexes the result. The kind of thing you’d expect to work first try and mostly did.</p><p>The part that didn’t was dead-letter handling — what happens when processing fails. I wanted something more honest than “it probably works”: actual failures, actual retries, actual proof that a message could survive five failed attempts and land somewhere a human could inspect. That decision is what surfaced the bug this post is about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDbt28juM6EzUoRkY7uskQ.png" /></figure><h3>The design</h3><p>Azure Storage Queues already solve most of dead-letter handling natively. Every queue message carries a dequeueCount, incremented automatically every time it&#39;s read and not deleted. Set maxDequeueCount in host.json, and once a message exceeds it, the Functions runtime moves it to an automatically-created &lt;queue-name&gt;-poison queue — no code required for any of that.</p><p>So the design was simple: on failure, write one message to a processing-deadletter queue. A queueTrigger function, retryFailedDocument, picks it up and restarts the orchestration. If it fails again, don&#39;t delete the message — let the queue&#39;s own visibility timeout make it reappear, incrementing dequeueCount on its own. After five failures, it escalates to the poison queue automatically. My job was just to <em>not delete the message on failure</em> — everything else was Azure&#39;s problem.</p><p><strong>Here’s roughly what that looked like:</strong></p><pre>export async function retryFailedDocument(queueItem: unknown, ctx: InvocationContext) {<br>  const input = queueItem as OrchestrationInput;<br>  const client = df.getClient(ctx);<br><br>  const instanceId = await client.startNew(&quot;documentOrchestrator&quot;, {<br>    input: { ...input, isRetry: true },<br>  });<br>}</pre><p>Clean. Minimal. And completely broken in a way that took real digging to see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFWKhwESQR4aRrJj22FfBg.png" /></figure><h3>The symptom</h3><p>I forced a test failure and watched the logs. Every single attempt showed the same shape:</p><pre>Retrying document &lt;id&gt; (dequeue attempt 1)<br>Executed &#39;Functions.retryFailedDocument&#39; (Succeeded, Duration=33ms)</pre><p>dequeue attempt 1 — every time. Never 2, never 3. The message would vanish from the queue and, moments later, a <em>brand new</em> message would appear with a different MessageId. No matter how long I waited, no matter how many times I retested, the poison queue never appeared. It looked like the escalation logic simply didn&#39;t exist.</p><h3>The false leads</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l53U3eDd1PxRrezjZOHsOg.png" /></figure><p><strong>The first few hypotheses were all reasonable and all wrong:</strong></p><ul><li>Maybe the orchestrator wasn’t re-throwing after dead-lettering — checked, it was (after fixing an earlier bug where the throw was nested inside an if and only fired on the very first failure, not on retries).</li><li>Maybe FORCE_TEST_FAILURE wasn&#39;t actually set — checked directly via the Azure CLI, confirmed &quot;true&quot;.</li><li>Maybe there was a typo in an activity name — there <em>was</em> one (extractMetaData vs. extractMetadata), and fixing it changed the symptom but didn&#39;t resolve it.</li></ul><p>Each of these was a real bug, and fixing them was necessary. None of them explained the pattern. The tell was hiding in plain sight the whole time: <strong>Duration=33ms.</strong></p><h3>The actual bug</h3><p>Think about what retryFailedDocument is actually doing. client.startNew(...) doesn&#39;t run an orchestration — it <em>schedules</em> one and returns an instance ID almost immediately. The orchestration itself runs asynchronously, independently, on its own timeline. startNew resolving has nothing to do with whether the work it kicked off will eventually succeed or fail.</p><p>So the function was doing exactly this: schedule the retry, then return. No exception was ever thrown, because nothing in that function <em>could</em> throw — it never waited around long enough to find out what happened. From the queue trigger’s perspective, the function succeeded unconditionally, every single time, regardless of what the restarted orchestration went on to do five seconds later. The message got deleted immediately. Meanwhile, the <em>new</em> orchestration would fail on its own schedule, hit its own catch block, and dead-letter <em>again</em> — as a completely new message, with a fresh MessageId and a dequeueCount reset to zero.</p><p>33 milliseconds is roughly what it takes to call an API and get an instance ID back. It is not remotely enough time for an orchestration to run three internal retry attempts with exponential backoff and then fail. The duration itself was the evidence — I just hadn’t been looking at it as evidence.</p><h3>The fix</h3><p><strong>The retry handler needed to actually wait for the outcome it caused, and propagate failure honestly:</strong></p><pre>const instanceId = await client.startNew(&quot;documentOrchestrator&quot;, {<br>  input: { ...input, isRetry: true },<br>});<br><br>const start = Date.now();<br>while (Date.now() - start &lt; MAX_WAIT_MS) {<br>  const status = await client.getStatus(instanceId);<br><br>  if (status?.runtimeStatus === &quot;Completed&quot;) return;<br><br>  if (status?.runtimeStatus === &quot;Failed&quot; || status?.runtimeStatus === &quot;Terminated&quot;) {<br>    throw new Error(`Retry orchestration ${instanceId} ended as ${status.runtimeStatus}`);<br>  }<br><br>  await new Promise((resolve) =&gt; setTimeout(resolve, POLL_INTERVAL_MS));<br>}</pre><p>Now retryFailedDocument polls until the orchestration reaches a terminal state, and re-throws if it failed. That throw is what makes the queue&#39;s native machinery actually apply: the message stays in the queue, its visibility timeout expires, it reappears with dequeueCount genuinely incremented — 1, then 2, then 3 — against the <em>same</em> MessageId, until it lands in the poison queue exactly where it was always supposed to.</p><p>Duration jumped from 33 milliseconds to several seconds. That, too, was confirmation — not just that the bug was fixed, but that I now understood <em>why</em> the number mattered as a signal in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YII0ZzQef7ypinCUsSA3A.png" /></figure><p><strong>Why polling, specifically</strong></p><p>Durable Functions does offer a built-in “wait for completion” helper — waitForCompletionOrCreateCheckStatusResponse — but it&#39;s built for HTTP-triggered clients; it expects an HttpRequest to shape its response around. A queue trigger doesn&#39;t have one. So the fallback is to poll client.getStatus(instanceId) manually, on a simple timer:</p><pre>const POLL_INTERVAL_MS = 1000; // check once per second<br>const MAX_WAIT_MS = 60_000;    // give up after 60 seconds<br><br>const start = Date.now();<br>while (Date.now() - start &lt; MAX_WAIT_MS) {<br>  const status = await client.getStatus(instanceId);<br><br>  if (status?.runtimeStatus === &quot;Completed&quot;) return;<br>  if (status?.runtimeStatus === &quot;Failed&quot; || status?.runtimeStatus === &quot;Terminated&quot;) {<br>    throw new Error(`Retry orchestration ${instanceId} ended as ${status.runtimeStatus}`);<br>  }<br><br>  await new Promise((resolve) =&gt; setTimeout(resolve, POLL_INTERVAL_MS));<br>}<br>throw new Error(`Timed out waiting for retry orchestration ${instanceId}`);</pre><p>In plain terms: check once a second whether the orchestration has finished; the moment it reaches a terminal state, stop and report the result either way; if a full minute passes with no terminal state, treat that as a failure too rather than waiting forever. The two constants are a deliberate trade-off — too short a poll interval wastes status-check calls for no benefit, too long delays how quickly completion is noticed; too short a timeout risks giving up on legitimately slow work, too long lets a genuinely stuck execution burn time before anything notices. There’s no universally correct pair of numbers, only ones appropriate to how long the underlying work is expected to take.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mC2yqU4LyEuYEB3RsVUcJg.png" /></figure><h3>Key takeaways</h3><ul><li><strong>startNew() confirms scheduling, not success.</strong> It resolves as soon as an orchestration instance is created — long before that orchestration has actually run.</li><li><strong>A suspiciously fast duration is a real signal.</strong> 33ms for a function that was supposed to wait on multi-step, retried work was the clue the whole time — implausible speed usually means something didn’t happen that was supposed to.</li><li><strong>Queue-native retry only works if failure actually propagates.</strong> dequeueCount and poison-queue escalation are free — but only if the triggered function throws when the work it&#39;s responsible for genuinely fails.</li><li><strong>Where no built-in “await completion” exists, poll deliberately.</strong> A bounded loop with a sensible interval and timeout is a normal, honest pattern — not a hack — when the SDK’s helper doesn’t fit your trigger type.</li><li><strong>Fixing one bug can just reveal the next one.</strong> This bug only became visible after two earlier ones (a missing re-throw, a typo’d activity name) were fixed — each layer of debugging exposed the next, truer symptom underneath.</li></ul><p><a href="https://github.com/dhulanjala/azure-document-pipeline">The full project</a> — including the corrected retry logic, a Durable Functions orchestrator unit test suite, and Bicep infrastructure-as-code — is on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0505cd13feca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Callbacks to Async/Await: How Asynchronous JavaScript Actually Works]]></title>
            <link>https://medium.com/@dhulanjala/from-callbacks-to-async-await-how-asynchronous-javascript-actually-works-f9ae031e5c7a?source=rss-43348a0591b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9ae031e5c7a</guid>
            <category><![CDATA[async-await-javascript]]></category>
            <category><![CDATA[javascript-promise]]></category>
            <category><![CDATA[callbacks-promises]]></category>
            <category><![CDATA[javascript-callback-hell]]></category>
            <category><![CDATA[mastering-async-js]]></category>
            <dc:creator><![CDATA[Hansi]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 06:41:12 GMT</pubDate>
            <atom:updated>2026-06-13T06:41:12.536Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXTwHh8w-6GQPJLGgGRwQA.png" /></figure><p>Most of us learn async/await, Get our code working, and never look back. The syntax is clean, the mental model feels synchronous, and life is good — until a promise rejects somewhere with no catch, a Node process crashes in production, or someone asks you <em>whyit</em> try/catch doesn&#39;t work the way you expect inside a callback.</p><p>This article walks through the entire async story in JavaScript: callbacks, the error-first convention, promises, and async/await. Not as isolated syntax to memorize, but as one continuous evolution where each step exists to fix a real problem in the step before it. By the end, the behavior that used to feel arbitrary should feel inevitable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-tTZbXAYkDipLLRGzIxZw.png" /></figure><h3>Callbacks: where it all started</h3><p>A callback is just a function passed as an argument to another function, to be run later.</p><pre>const fs = require(&#39;fs&#39;);<br><br>fs.readFile(&#39;data.txt&#39;, (err, data) =&gt; {<br>  console.log(data);<br>});</pre><p>fs.readFile doesn&#39;t block while it reads from disk. It registers that arrow function and moves on. When the read finishes, Node calls your function back — hence the name. For every asynchronous event, Node lets you register a callback to handle the result.</p><p><strong>The error-first pattern</strong></p><p>If you look closely at the callback signature above, the first parameter is err, not data. That&#39;s not a coincidence — it&#39;s a convention baked into Node&#39;s core APIs:</p><pre>fs.readFile(&#39;file.txt&#39;, (err, data) =&gt; {<br>  if (err) return console.error(err);<br>  console.log(data);<br>});</pre><p>The rule is simple:</p><ul><li>err === null → success</li><li>err !== null → failure</li></ul><p><strong>Why error-first? Why not just </strong><strong>throw?</strong></p><p>This is the question that unlocks everything about async error handling.</p><p>Asynchronous operations run <em>later</em> — after the function that started them has already returned. By the time readFile actually fails, the surrounding synchronous code has long since finished executing. There is no try/catch left on the stack to catch anything. The error has nowhere to bubble up to.</p><p>So Node doesn’t throw. It passes the error to you <strong>manually</strong>, as the first argument:</p><pre>callback(error, result);</pre><p>Once you internalize this — that async errors can’t travel up the call stack the way synchronous ones do — the next problem becomes obvious.</p><p><strong>Why </strong><strong>try/catch fails with async callbacks?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_bJbwKy_-MJB9tmwNR1Og.png" /></figure><p>Beginners reach for the tool they know:</p><pre>try {<br>  fs.readFile(&#39;file.txt&#39;, () =&gt; {<br>    throw new Error(&#39;boom&#39;);<br>  });<br>} catch (error) {<br>  console.log(error); // this never runs<br>}</pre><p>That catch will never fire. Here&#39;s the sequence of events:</p><ol><li>The try block executes fs.readFile and <strong>finishes immediately</strong>.</li><li>The callback runs <em>later</em>, in a completely new execution stack.</li><li>By then, the original try/catch is gone.</li></ol><p>When the callback finally throwsboom, the catch That was supposed to protect it no longer exists on the stack. The error escapes and crashes the process.</p><p><strong>Where </strong><strong>try/catch <em>does</em> work</strong></p><p>try/catch isn&#39;t useless in async code — it just only works around <strong>synchronous</strong> logic. Two valid places:</p><ol><li><strong>Around synchronous code:</strong></li></ol><pre>try {<br>  JSON.parse(&#39;..invalid json..&#39;);<br>} catch (error) {<br>  console.error(error);<br>}</pre><p>2. Inside a callback, for the synchronous work happening within it:</p><pre>fs.readFile(&#39;file.txt&#39;, (err, data) =&gt; {<br>  try {<br>    JSON.parse(data); // synchronous — try/catch works here<br>  } catch (e) {<br>    console.error(e);<br>  }<br>});</pre><p><strong>Callback hell</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KnEK3j77yde8dfYl2Wnmag.png" /></figure><p>Callbacks work, but they don’t <em>compose</em>. The moment one async operation depends on the result of another, you start nesting:</p><pre>getUser(id, (err, user) =&gt; {<br>  getOrders(user.id, (err, orders) =&gt; {<br>    getOrderDetails(orders[0], (err, details) =&gt; {<br>      // ...and we keep drifting right<br>    });<br>  });<br>});</pre><p>This is the <strong>pyramid of doom</strong>: callbacks nested inside callbacks, many levels deep. It’s hard to read, hard to maintain, and genuinely difficult to debug. Error handling has to be repeated at every level. Something better was needed.</p><h3>Promises: a placeholder for a future value</h3><p>Instead of returning the final value immediately (impossible — it isn’t ready yet) or demanding a callback up front, an async operation hands you a <strong>placeholder object</strong>: a promise. You hold onto it, and it eventually settles into a result.</p><p>A promise is always in exactly one of three states:</p><ul><li><strong>pending</strong> — still working, no result yet</li><li><strong>fulfilled</strong> — finished successfully (via resolve())</li><li><strong>rejected</strong> — failed (via reject())</li></ul><p>Once a promise moves from pending to either fulfilled or rejected, it is <strong>settled</strong>. And here’s the crucial guarantee:</p><p><em>Once a promise is settled, it is </em><strong><em>immutable</em></strong><em>. It can never change its state again.</em></p><p>This is what makes promises trustworthy. A function can’t accidentally resolve your promise twice and run your success handler twice. <strong>Only the first </strong><strong>resolve or </strong><strong>reject counts — every call after that is ignored.</strong></p><p>Two handlers connect to the two outcomes:</p><ul><li>if resolve() was called → .then() runs</li><li>if reject() was called → .catch() runs</li></ul><p><strong>The detail most people miss: </strong><strong>.then() chaining</strong></p><p>This is where promises stop being “callbacks with nicer syntax” and become something genuinely more powerful.</p><p><em>.then() always returns a </em><strong><em>brand new promise object</em></strong><em>, immediately</em></p><p>And inside a .then(), JavaScript <strong>automatically unwraps</strong> any promise you return — it waits for that inner promise to settle, extracts the value, and passes it to the next .then() in the chain:</p><pre>getUser(id)<br>  .then(user =&gt; getOrders(user.id)) // returns a promise...<br>  .then(orders =&gt; console.log(orders)) // ...but JS unwraps it for us<br>  .catch(handleError);</pre><p>You never have to manually dig the value out of the promise returned by getOrders. JS unwraps it before calling the second .then(). And if you return a plain value inside .then(), the new promise simply resolves with it automatically.This unwrapping is the entire reason the pyramid of doom flattens into a clean vertical chain.</p><p><strong>Running promises in parallel: </strong><strong>Promise.all()</strong></p><p>Promise.all() lets you run multiple promises <strong>in parallel</strong> and wait for all of them to finish.</p><pre>const results = await Promise.all([promise1, promise2, promise3]);</pre><p>It returns a new promise that:</p><ul><li><strong>Resolves</strong> when <em>all</em> the promises resolve (you get an array of results)</li><li><strong>Rejects immediately</strong> the moment <em>any one</em> promise rejects</li></ul><p>When you want every result, even the failures: Promise.allSettled()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bN9oYOUTDhJ_kHK1E3NuGg.png" /></figure><p>async/await is syntactic sugar over promises — but understanding what it desugars to is what separates confident code from cargo-culting.</p><p>Every async function returns a promise</p><p><strong><em>Any function marked </em></strong><strong><em>async automatically returns a promise, wrapping whatever value you returned (or </em></strong><strong><em>undefined if you returned nothing).</em></strong></p><pre>async function test() {<br>  return &#39;hello&#39;;<br>}<br><br>test(); // Promise { &#39;hello&#39; }</pre><p><strong>What </strong><strong>await actually does</strong></p><p>await pauses your function and waits for a promise to settle:</p><ul><li>if the promise <strong>fulfills</strong> → it gives you back the value</li><li>if the promise <strong>rejects</strong> → it <em>throws</em> the error</li></ul><p>That second point is the magic. Because await turns a rejection into a regular thrown error, normal try/catch suddenly works again:</p><pre>async function readConfig() {<br>  try {<br>    const data = await readFile(&#39;file.txt&#39;);<br>    return JSON.parse(data);<br>  } catch (err) {<br>    // handle error here — and it actually fires<br>  }<br>}</pre><p>Step by step:</p><ol><li>readFile returns a promise.</li><li>The promise rejects.</li><li>await turns that rejection into a thrown error.</li><li>catch catches it.</li></ol><p>A useful mental model: when you await, you&#39;re telling the engine <em>&quot;pause this function, unwrap the promise in the background, and hand me the value directly.&quot;</em> Node unwraps the promise for you — exactly like .then() did, just without the callback.</p><p><strong>Does </strong><strong>await block Node? (No.)</strong></p><p><strong>await only pauses the function it&#39;s inside.</strong> The event loop keeps running, other tasks keep executing, other requests keep getting served. You&#39;ve paused one function, not the runtime.</p><p><strong>Error handling, properly</strong></p><p><strong>With callbacks</strong>, error handling is manual and repetitive. You check if (err) at every single level, and forgetting one silently swallows a failure.</p><p><strong>With promises and async/await</strong>, error handling is centralized:</p><pre>getUser(id)<br>  .then(user =&gt; getOrders(user.id))<br>  .then(orders =&gt; process(orders))<br>  .catch(handleError); // one catch for the entire chain</pre><p>One .catch() can handle every error in the chain. One try/catch can wrap an entire sequence of awaits. The error-handling logic lives in one place instead of being scattered across every step.</p><p><strong>Unhandled rejections — and why Node now crashes on them</strong></p><p>So what happens if a promise rejects and there’s <strong>no</strong> .catch() and <strong>no</strong> surrounding try/catch? It becomes an <strong>unhandled promise rejection</strong>, and Node&#39;s behavior here changed in an important way:</p><ul><li><strong>Old Node (before v15):</strong> Node just printed a warning and kept running.</li><li><strong>Modern Node (v15+):</strong> Node <strong>crashes the process</strong> — it throws and exits.</li></ul><p>That sounds harsh, but it’s the safer default. Why? Because <strong>unhandled rejections are dangerous</strong>:</p><ul><li>They can leave a <strong>database transaction half-finished</strong>.</li><li>They can <strong>leak memory</strong>.</li><li>They can leave your <strong>server in an inconsistent state</strong>.</li></ul><h3><strong>Putting it together</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNUH68ROQwUdgDWLejhXIg.png" /></figure><p>The progression isn’t three competing styles — it’s one problem being solved, refined, and solved again:</p><ul><li><strong>Callbacks</strong> gave us asynchrony, but cost us composition and control.</li><li><strong>Promises</strong> gave back control and flattened the pyramid, with automatic unwrapping and centralized error handling.</li><li><strong>Async/await</strong> wrapped promises in synchronous-looking syntax, making try/catch work again — without ever blocking the event loop.</li></ul><p>You can write modern JavaScript using only async/await and never type .then() again. But knowing what sits underneath — that an async function <em>is</em> a promise, that await <em>throws</em> on rejection, that unhandled rejections will <em>crash</em> your process — is the difference between code that works on your machine and code you can trust in production.</p><p>That’s the whole point of understanding the layers, not just the syntax on top.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9ae031e5c7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering MongoDB Schema Design: A Senior Engineer’s Playbook]]></title>
            <link>https://medium.com/@dhulanjala/mastering-mongodb-schema-design-a-senior-engineers-playbook-5b79bde70abe?source=rss-43348a0591b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b79bde70abe</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[schema-design]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[schema]]></category>
            <dc:creator><![CDATA[Hansi]]></dc:creator>
            <pubDate>Thu, 14 May 2026 10:59:48 GMT</pubDate>
            <atom:updated>2026-05-14T12:31:07.209Z</atom:updated>
            <content:encoded><![CDATA[<p>Beyond CRUD — Designing for Scalability, Performance, and the 16MB Frontier</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*66iWpRbmqEVFZjZw_C8N8g.png" /></figure><p>In the world of Relational Databases (RDBMS), we are taught to normalize until it hurts. We split tables, enforce rigid constraints, and rely on the magic of JOINs. But when you step into the world of MongoDB, the mindset flips: <strong>Data that needs to be together should live together.</strong></p><p>As a Senior Engineer, you aren’t just writing queries; you are architecting data flows. In this article, we’ll dive deep into the core schema design patterns you must master to build high-performance, scalable MongoDB applications.</p><h3>1. Embedding (Denormalization)</h3><p>Embedding is the bread and butter of MongoDB. It involves storing related data in a single document as nested objects or arrays</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KFJwh9NNB-u6bMSza2rkpg.png" /></figure><pre>{ &quot;_id&quot;: &quot;u1&quot;, <br>&quot;name&quot;: &quot;hansi&quot;, <br>&quot;email&quot;: &quot;dhulanjala@gmail.com&quot;, <br>&quot;settings&quot;: { <br>&quot;theme&quot;: &quot;dark&quot;, <br>&quot;language&quot;: &quot;en&quot;, <br>&quot;notifications&quot;: true <br>}}</pre><h4>The Benefits</h4><p><strong>Extreme Speed</strong>: MongoDB can fetch the entire record in one single read operation.</p><p><strong>No Joins</strong>: Eliminates the need for $lookup or application-side population. <strong>Atomicity</strong>: Updates to a single document are atomic.</p><h4>The Trap</h4><p>The primary constraint is the <strong>16MB BSON document size limit</strong>. You must never embed <strong>unbounded arrays</strong>. If you are storing comments on a post or event logs that grow indefinitely, embedding will eventually crash your system.</p><h3>2. Referencing (Normalization)</h3><p>Referencing is the “SQL-like” approach where you store only the ID of a related document. Use this when data grows independently or has a many-to-many relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RiRWpltOnRwKO7zyNx5RYg.png" /></figure><p><strong>The Trap</strong>: Since data is split across collections, you need multiple queries or heavy aggregation stages to fetch the full picture. This can lead to latency issues if overused.</p><h3>3. The Extended Reference Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mvj1h7GrSMkk2ZNTVpkiQ.png" /></figure><p>This is where senior-level design shines. You reference the full document, but partially embed the most frequently read fields to avoid a JOIN.</p><pre>{ &quot;orderNo&quot;: &quot;OR001&quot;, <br>&quot;customerId&quot;: &quot;c1&quot;, <br>&quot;customerSnapshot&quot;: <br>{ &quot;name&quot;: &quot;hansi&quot;, <br>• • • &quot;shippingAddress&quot;: &quot;Colombo&quot; <br>} <br>}</pre><p><strong>Why use it?</strong></p><p><strong>Fast Reads:</strong> You get the customer name without joining the Users collection.</p><p><strong>Historical Preservation:</strong> If the user changes their address today, the 2023 order still correctly reflects where it was shipped at that time.</p><p><strong>Efficiency:</strong> Use it when the full object is heavy or rarely needed</p><h3>4. The Outlier Pattern</h3><p>Standard patterns fail when your data has “Celebrity” users. 99% of your users might have 10 followers (easy to embed), but one user might have 10 million.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6t5jsqa-0Ld7pxjVhjQN9A.png" /></figure><p>The Outlier Pattern solves this by keeping the common case fast while handling edge cases separately:</p><p><strong>Normal User:</strong> Followers are embedded. hasExtraFollowers: false.</p><p><strong>Celebrity User:</strong> Embed the first 1,000 followers. hasExtraFollowers: true. Move the remaining millions to an Overflow Collection.</p><p><strong>Outcome:</strong> 99% of your database operations remain ultra-fast, while rare outliers use extra storage logic without slowing down everyone else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nojW-Kw6JkiW9TpA0C47oQ.png" /></figure><p>#MongoDB #SoftwareArchitecture #NoSQL #WebDevelopment</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b79bde70abe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Node.js: How It Actually Works Under the Hood]]></title>
            <link>https://medium.com/@dhulanjala/unlocking-node-js-how-it-actually-works-under-the-hood-b115f38ef16a?source=rss-43348a0591b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b115f38ef16a</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[how-node-work]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Hansi]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 11:23:41 GMT</pubDate>
            <atom:updated>2026-02-23T11:23:41.740Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1AXX6o8_Gwh6D_RzHFw5yA.png" /></figure><p>When we talk about Node.js, we often throw around terms like “event loop,” “single-threaded,” and “non-blocking.” But what do these actually mean in practice?</p><p>Here is a deep dive into the inner workings of Node.js, breaking down its architecture, the request-response lifecycle, and the event loop.</p><h3><strong>The Basics: What is Node.js?</strong></h3><p>At its core, a <strong>runtime environment</strong> is the infrastructure that provides the engine, APIs, and system access required to execute code. Node.js is exactly that: a runtime environment that allows JavaScript to execute <em>outside</em> of the browser.</p><p>A JavaScript runtime provides three main things:</p><ol><li>JS Engine</li></ol><p>2. APIs (files, timers, network)</p><p>3. Event Loop</p><p>To understand Node.js, it helps to compare it to the browser environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DShWH-nuH51ScPCdfKmcAQ.png" /></figure><p><strong>The Two Pillars of Node.js</strong></p><p>Node.js is built on top of two foundational pillars:</p><ol><li><strong>V8 Engine:</strong> Google’s high-performance engine (written in C++) that compiles JavaScript into machine code.</li><li><strong>Libuv:</strong> A C library that gives Node.js access to the OS kernel, file system, and networking, and provides the event loop.</li></ol><h3>T<strong>he Whole Process: From Request to Response</strong></h3><p>What actually happens when a client sends a request to your server?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_PC7Q_2Gr_30kFuiY6h_w.png" /></figure><p>Here is the lifecycle:</p><h3>1. Network to Stream</h3><p>The process begins when the client sends a request.</p><ul><li><strong>Network Layer:</strong> The request hits the server’s Network Interface Card (NIC) as TCP packets.</li><li><strong>Socket:</strong> The OS kernel manages the connection via a socket (e.g., listening on port 443).</li><li><strong>Node Layer:</strong> We don’t wait for all packets to arrive; we treat the incoming data as a stream.</li><li><strong>Libuv:</strong> uses a platform-specific demultiplexer (IOCP for Windows, epoll for Linux, kqueue for macOS) to monitor these sockets. It essentially asks the OS, <em>“Notify me if a new event comes.”</em></li></ul><h3>2. V8 and Call Stack</h3><p>The Event Loop checks if the V8 call stack is empty and ensures the microtask queue is drained. If clear, the event loop pushes the callback to the call stack, where V8 executes code using a LIFO (Last-In-First-Out) approach.</p><p>If the call stack encounters an Async task, the Event loop controls it in one of two ways:</p><ul><li><strong>Network I/O</strong> (DBs, APIs, HTTP requests): Sent to the <strong>OS Kernel</strong> -&gt; <em>Truly Non-blocking.</em></li><li><strong>File I/O, Crypto, Heavy Math</strong>: Sent to the <strong>Libuv worker pool</strong> -&gt; Handled by Worker Threads (default of 4) because OS kernels aren’t great at async CPU-intensive tasks.</li></ul><p><em>(Note: If the call stack hits synchronous code, the event loop will freeze!)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwf3_d6MFQ-nlLNhZQ7jUA.png" /></figure><h3>3. Callback Queues</h3><p>After these async tasks complete, their callbacks are pushed to the <strong>Microtask queue</strong> or <strong>Macrotask queue</strong>.</p><h3>4. The Exit</h3><p>When your code finishes and calls res.send():</p><ul><li><strong>C++ bindings</strong> act as a translator/bridge. They get our code from V8, translate it to C++, and call Libuv or the OS.</li><li><strong>Libuv</strong> tells the OS kernel to write data to the specific file descriptor (socket ID) associated with that user.</li><li>Finally, the <strong>OS kernel</strong> wraps the data back into TCP packets.</li></ul><h3>Task Queues: Micro vs. Macro</h3><p>Node.js handles scheduled tasks through specific queues. Understanding the priority here is key.</p><h3>Microtask Queue (The VIPs)</h3><p>These are tasks that need to happen <em>immediately</em> after the current execution finishes, but before the event loop moves to the next phase.</p><ul><li><strong>What’s inside:</strong> process.nextTick(), and Promises (.then(), .catch(), .finally()).</li><li><strong>The Rule:</strong> Node will exhaust this entire queue before doing anything else. If a microtask adds <em>another</em> microtask, Node will keep running them until the queue reaches zero.</li></ul><h3>Macrotasks (The Regulars)</h3><p>These represent tasks scheduled to run in the future or wait for external events, aligning with the phases of the event loop.</p><ul><li><strong>What’s inside:</strong> setTimeout(), setInterval(), setImmediate(), I/O callbacks, HTTP request handlers.</li><li><strong>The Rule:</strong> The event loop processes these one by one, according to its phase.</li></ul><h3>Execution Checkpoints in a Cycle</h3><ol><li>V8 call stack runs your code.</li><li>Call stack becomes empty.</li><li><strong>Microtask checkpoint:</strong> Node immediately executes all tasks in the microtask queue.</li><li><strong>Event loop cycle:</strong> Run <em>one</em> macrotask -&gt; Run <em>ALL</em> microtasks -&gt; Move to next macrotask.</li></ol><h3>The Event Loop Phases</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Amsupz4pTlOGXrTAP6Qbw.png" /></figure><h3>How to Break (or Freeze) Node.js</h3><p>Node.js crashes usually mean the main thread is blocked. Here are the common scenarios:</p><p><strong>1. Synchronous “Heavy” Computation.</strong> If you run heavy math, massive loops, or image processing synchronously, the call stack is occupied until those finish. The event loop can’t move. It isn’t a hard crash (the process is still alive), but to the outside world, the server is “dead” because it isn’t responding.</p><p><strong>2. Microtask Starvation</strong> Because Node won’t move to the next event loop phase until the microtask queue is empty, you can starve the event loop. For instance, if a server is executing a massive loop of microtasks (like a Promise or one process.nextTick() that recursively calls itself), the Event loop will <em>never</em> reach the poll phase to pick up new requests. The new requests will just sit in the OS kernel waiting.</p><h3>Bonus Details: Modules and Clusters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5GpUa4ZyL3dphpgL_09Pw.png" /></figure><ul><li><strong>Why put </strong><strong>require() at the top?</strong> require(&#39;fs&#39;) is inherently synchronous. It reads the file from the disk immediately. It blocks the loop <em>only once</em> at startup. This is exactly why we put require at the top of the file, not inside request handlers.</li><li><strong>If Node is single-threaded, how does the </strong><strong>cluster module work?</strong> The cluster module solves this by forking the OS process. It creates multiple copies of your application, meaning you get multiple event loops and multiple V8 instances running in parallel!</li></ul><h3>📚 Resources &amp; Further Reading</h3><p>If you want to dive even deeper into the internals of Node.js, here are the best official and community resources to explore:</p><h3>Official Documentation</h3><ul><li><a href="https://nodejs.org/en/learn/asynchronous-work/event-loop-timers-and-nexttick"><strong>Node.js Event Loop Guide</strong></a><strong>:</strong> The definitive guide to how the loop, timers, and process.nextTick() work together.</li><li><a href="https://nodejs.org/en/learn/asynchronous-work/dont-block-the-event-loop"><strong>Don’t Block Your Event Loop</strong></a><strong>:</strong> A must-read for anyone building production-grade servers to avoid performance bottlenecks.</li><li><a href="https://v8.dev/docs"><strong>V8 Engine Documentation</strong></a><strong>:</strong> Deep dives into how Google’s engine optimizes your JavaScript into machine code.</li><li><a href="https://docs.libuv.org/en/v1.x/guide/basics.html"><strong>Libuv Documentation</strong></a><strong>:</strong> For the truly curious — the official C++ documentation for the library that powers Node’s asynchronous magic.</li></ul><h3>Recommended Books</h3><ul><li><strong>“Node.js Design Patterns” by Mario Casciaro:</strong> Widely considered the “Bible” of Node.js architecture.</li><li><strong>“Distributed Systems with Node.js” by Thomas Hunter II:</strong> Great for understanding how Node scales in enterprise environments.</li><li><strong>“Node.js the Right Way” by Jim Wilson:</strong> A compact guide that focuses on modern architecture and core concepts.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b115f38ef16a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>