<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shir Brass on Medium]]></title>
        <description><![CDATA[Stories by Shir Brass on Medium]]></description>
        <link>https://medium.com/@shirbrass?source=rss-f76bd1d05dbd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oBnDCCZvILwAo8BzNEfIHA.jpeg</url>
            <title>Stories by Shir Brass on Medium</title>
            <link>https://medium.com/@shirbrass?source=rss-f76bd1d05dbd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:41:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shirbrass/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Your External Provider Goes Rogue: How We Tamed Chaos with Event Buffers]]></title>
            <link>https://medium.com/@shirbrass/when-your-external-provider-goes-rogue-how-we-tamed-chaos-with-event-buffers-31b3fd8e567b?source=rss-f76bd1d05dbd------2</link>
            <guid isPermaLink="false">https://medium.com/p/31b3fd8e567b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[doorloop]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Shir Brass]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 07:39:12 GMT</pubDate>
            <atom:updated>2026-01-19T07:39:12.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RF_HsrrvlETGH7z3i2UQTQ.png" /></figure><p>Hi everyone! I’m Shir Brass, Fullstack Tech Lead @ DoorLoop with 13+ years of experience wrangling technologies that sometimes have a mind of their own.</p><p>In this blog I want to share a war story about one of those “fun” challenges that keeps engineering teams up at night: <strong>what do you do when an external provider you depend on… just doesn’t work properly?</strong></p><h3>The Problem: When Events Play Musical Chairs</h3><p>Picture this: You’re expecting a clean, predictable event flow like this:</p><p>created → in progress → processing → completed ✅</p><p>But instead, you’re getting this chaos:</p><p>created → in progress → completed → in progress → completed → ??? 🤯</p><p><strong>Which status is real? What’s your source of truth?</strong></p><p>As users expect consistency, <em>we don’t know</em>, and “the provider is unreliable” are not acceptable answers. So what do you do?</p><h3>The Impact: Speed vs. Reliability Trade-off</h3><p>Let’s be honest: <strong>everyone wants real-time updates</strong>. Users expect to see status changes the moment they happen — no lag, no delays.</p><p>We faced a choice: give users fast updates that might be wrong, or slower updates that we know are correct.</p><p>What’s worse than a delay? Telling someone their operation transaction is “complete,” then hours later switching it back to “still processing.” That erodes trust fast.</p><p><strong>Trust is non-negotiable.</strong> The system can’t gaslight users. It can’t contradict itself.</p><p>We made the call: <strong>reliability over speed</strong>. It is important to mention that in our case, users can tolerate a reasonable delay if the answer is rock solid, and there is no business need for immediate response. What breaks their trust — and floods support — is showing the wrong status, then having to walk it back.</p><p>So… how did we solve it?</p><h3>Our Solution: Event Buffers to the Rescue 🦸</h3><h3>Step 1: Strategic Pipeline Throttling</h3><p>We couldn’t freeze the pipeline (as a downtime is not acceptable), so we <strong>introduced controlled delays</strong> and handled events in batches after a short, configurable window. If you’ve ever debounced user input on the frontend, it’s the same idea: wait for the noise to settle, then act on the latest intent.</p><p>Here’s a quick debounce analogy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*bu-xpIyj0deEVWvl16-rNg.png" /></figure><p>And here’s our backend version with an event buffer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*CCme6ZiQL7R7CVluGszIZg.png" /></figure><p>It adds some additional steps to the flow, but it bought us something priceless: <strong>determinism at the edges.</strong></p><h3>Step 2: Pattern Recognition Through Data</h3><p>With buffering in place, we could finally <strong>observe</strong> provider behavior instead of chasing it. We paused on suspicious patterns and collected evidence to tune the window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*ymOfqOsSYh5_jEmg4K2brA.png" /></figure><p><strong>Legend:</strong></p><ul><li>🟢 The first 4 bars (0–30s) are the fast resolution zone (where most operations lie)</li><li>🔴 The 72h-78h bar is that problematic cluster we identified (the lone pillar)</li><li>🟠 The final bar (78h+) represents the outliers (the longest outlier was <strong>almost 21 days</strong>)</li></ul><p><strong>Key insight:</strong> If we wait 3 days from the last “final”-class status, the state is <em>very likely</em> truly final.</p><p>As a guardrail, if new events arrive after that window, treat it as a red flag and stop automatic processing. Investigate manually.</p><p>This adds time, but gives us a <strong>secure fallback</strong>. If the 3‑day bet ever proves too short, we measure again and adjust.</p><h3>The Numbers (Because Everyone Loves Numbers 📊)</h3><h3>Before</h3><ul><li>~1,000 entities with inconsistent statuses</li><li>~20 daily customer support tickets</li><li>Ongoing doubt about the 3rd party provider’s reliability</li></ul><h3>After</h3><ul><li><strong>100% confidence</strong> in said provider’s statuses once they pass through our buffer</li><li><strong>98% reduction</strong> in said provider’s related issues</li><li><strong>Calmer ops</strong> and happier users ✨</li></ul><h3>Key Takeaways for Fellow Engineers</h3><p>External providers fail in creative ways. Your job is to <strong>contain the blast radius</strong>.</p><ul><li><strong>Instrument everything</strong>. You can’t fix what you can’t see.</li><li><strong>Keep mitigation layers separate</strong> from core business logic. Don’t tangle them.</li><li><strong>Let data lead</strong>. Pause, observe patterns, then choose windows and rules.</li><li><strong>Design for failure paths</strong>. Assume they’ll happen and make them boring.</li></ul><p>External dependencies are a given. The question isn’t <em>if</em> they’ll wobble, but <em>how ready you are when they do</em>.</p><p>Have a story about an unreliable integration? I’d love to hear what worked for you.👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31b3fd8e567b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>