<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tal Peretz on Medium]]></title>
        <description><![CDATA[Stories by Tal Peretz on Medium]]></description>
        <link>https://medium.com/@talperetz88?source=rss-c7351d02921a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BvZ4p0Qx3oePoFmBQXVEZA.jpeg</url>
            <title>Stories by Tal Peretz on Medium</title>
            <link>https://medium.com/@talperetz88?source=rss-c7351d02921a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:58:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@talperetz88/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Every Scheduling System Has This Bug. Most Teams Haven’t Found It Yet]]></title>
            <link>https://medium.com/elementor-engineers/every-scheduling-system-has-this-bug-most-teams-havent-found-it-yet-cf1c9d2f9b79?source=rss-c7351d02921a------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf1c9d2f9b79</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Tal Peretz]]></dc:creator>
            <pubDate>Wed, 13 May 2026 10:51:13 GMT</pubDate>
            <atom:updated>2026-05-13T10:51:13.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OsUrFrKaJILMk2nXZhkifw.jpeg" /></figure><p>It was another sprint planning meeting. Coffee in hand, calendar blocked, and right on cue a new change request landed in the chat.</p><p>“Can we adjust the grace period for plan X? Just a small tweak.”</p><p>We smiled. We always smile at “just a small tweak.”</p><p>Because by that point, we knew what “small tweak” really meant. It meant opening a file nobody wanted to open. It meant tracing logic that had been patched three times already. It meant writing tests to make sure that fixing plan X didn’t silently break plans A through W. And it meant that someone, somewhere, would spend their Friday afternoon praying that a subscription that should be canceled actually gets canceled.</p><p>We had built something that worked. Technically. But it had become the kind of code you’re afraid to touch the kind where every change request comes with a side of anxiety.</p><p>So instead of doing the tweak, we did something else. We stopped, looked at what we had actually built, and decided it was time to do it properly.</p><p>To understand why we ended up in that meeting, you have to understand what we started with. We run subscriptions across a growing catalog of products different plans, different billing cycles, different rules. But they all share one thing: eventually, when a customer stops paying, something has to cancel them.</p><p>One cron job. Midnight. Every night it woke up, scanned the subscriptions table, ran through a set of checks, and canceled whoever was due. That’s it. No drama, no complexity. It was the kind of code you write in an afternoon and forget about for two years because it just works.</p><p>And for a while, it did.</p><p>But “one set of checks” assumes one set of rules. As the platform grew, that assumption quietly fell apart. Different products needed different grace periods. Different plans needed different behaviors. Product managers God bless them kept coming with perfectly reasonable requests that each made complete sense in isolation. And every time we said yes, we added another branch to that midnight scan. Another condition. Another edge case. Another “wait, does this affect the other plans?”</p><p>The cron job never complained. It just kept running at midnight, carrying more and more weight, until the day someone asked for one small tweak too many and we finally looked at what we had actually built.</p><p>That’s when we realized the business complexity was actually the easy part. Hiding underneath it was a much harder problem one that our midnight cron job had been quietly masking all along.</p><p>What happens when a cancellation is already scheduled and everything that made it valid, isn’t anymore?</p><p>That’s not a product problem. That’s a distributed systems problem. And this is the story of how we solved it.</p><h3>We Stopped Coding and Grabbed a Marker</h3><p>There’s something about a whiteboard that forces clarity. When you draw the system instead of reading it, patterns you missed for months suddenly become obvious. And what we saw on that whiteboard was uncomfortable: business logic and execution logic tangled together, every grace period rule buried deep inside the cancellation flow. No wonder every “small tweak” felt like defusing a bomb.</p><p>The insight was almost embarrassingly simple once we saw it. The question of <em>when</em> to cancel a subscription and the question of <em>how many grace days to give</em> are two completely different concerns. They just happened to live in the same place.</p><p>So we separated them.</p><p>We introduced a dedicated decision layer titled as Grace Policy Resolver. Its only job is to answer one question: given this subscription’s SKU, plan type, and any customer-specific overrides, how many grace days does it get? That’s it. No scheduling logic, no cancellation triggers. Just policy.</p><p>The effect was immediate. Product teams could now change business rules without touching the execution flow. A PM wanted to adjust grace periods for a specific plan? One change, one place, no prayer required.</p><p>But the whiteboard also showed us something else. Something we hadn’t fully appreciated before.</p><p>Knowing <em>when</em> a subscription should be canceled is one thing. Making sure that decision is still valid when the scheduled action actually runs that’s a completely different problem.</p><h3>Why Simple Scheduling Fails at Scale</h3><p>Someone drew an arrow in the wrong place and the room went quiet.</p><p>We were mapping out the subscription lifecycle on the whiteboard: term starts, term ends, grace period, cancellation. Clean, linear, obvious. And then someone asked: “what happens if the customer renews <em>here</em>?” and drew an arrow pointing back from the middle of the grace period.</p><p>We stared at it for a second.</p><p>Because on paper, canceling a subscription sounds simple. The term ends, the grace period passes, the system fires the cancellation. Done. But the lifecycle is never that linear in reality. Customers don’t follow the happy path. They renew late. They upgrade mid-term. They toggle auto-renew off and then back on twenty minutes later. Any of these actions can happen at any point and any of them can invalidate a cancellation decision that was completely correct five minutes ago.</p><p>That arrow exposed something we hadn’t designed for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wGtvpI5XS5hOHv6EmRbn7g.png" /><figcaption>The subscription lifecycle. The Volatility Zone is where state keeps changing. The Middle Zone is where a scheduled cancellation sits and waits.</figcaption></figure><p>See that middle zone? A scheduled cancellation lives there, quietly waiting. But the subscription underneath it keeps changing. And when execution time finally arrives, nobody has checked whether that cancellation still makes sense.</p><p>At small scale, you get away with it. At millions of subscriptions, you can’t afford to.</p><p>That’s not a grace period problem. That’s a race condition. And it lives at the heart of every scheduling system that was built when the data was small and quiet.</p><h3>Solving Race Conditions with Versioning</h3><p>The first thing we knew we had to do was get rid of the cron job model entirely. Instead of a single job that wakes up and scans everything, we moved to a task-based model each subscription gets its own scheduled cancellation task, created at the moment we know it’s due</p><p>The obvious solution was to delete old tasks whenever a subscription changes. Also the wrong one.</p><p>Every time a subscription changes, find the scheduled cancellation task and delete it. Simple, clean, intuitive. And at small scale it even works. But the moment you’re dealing with millions of subscriptions, that model starts falling apart. You’re doing broad searches across a schedule table that never stops growing. You’re introducing coordination between systems that shouldn’t need to talk to each other. And every update becomes a race what if the task gets picked up right before you delete it?</p><p>I kept coming back to the same question: what if we didn’t touch the schedule table at all?</p><p>Instead of chasing old tasks and cleaning them up, what if we just made them safe to ignore?</p><p>That’s the idea behind versioning. We added a single field to our subscription projection, our local read model that mirrors the authoritative external state, active_schedule_version. Every time something happens that affects cancellation eligibility, that version increments. When we schedule a new cancellation task, we stamp it with the subscription&#39;s current version at that moment, so the task carries a record of the world it was born into.</p><p>Now when a worker picks up a task, it does one check: does my version match the current version on the subscription? If yes execute. If no discard. No cleanup, no coordination, no race conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vjA0Z9BcqPQCNA6Zxa7fgA.png" /><figcaption>Only the task stamped with the current version gets executed. Everything else is a zombie.</figcaption></figure><p>We started calling the outdated tasks zombie tasks. They still exist in the table. They still get picked up by workers. But the moment a worker checks the version, they’re harmless.</p><p>The code for this is almost disappointingly simple:</p><pre>const subscription = get_from_db(scheduled_task.subscription_id)<br><br>if scheduled_task.version != subscription.version:<br>    return // skip - stale task, world has moved on<br><br>process_cancellation(subscription_id)</pre><p>One field. One check. And suddenly the hardest part of the scheduling problem became the easiest part of the execution flow.</p><h3>Killing the Full Table Scan</h3><p>The midnight cron job had one more ghost we needed to kill.</p><p>Every night it woke up and scanned the entire subscriptions table. Every row. Every subscription. Just to find the handful that were actually due. At small scale that’s fine. At millions of subscriptions, you’re doing an enormous amount of work to find a tiny amount of relevant data. It’s like reading an entire dictionary to find one word.</p><p>We needed a smarter way to ask the question.</p><p>Instead of “who needs to be canceled across all subscriptions?” we wanted to ask “who needs to be canceled <em>right now</em>, in this slice of time?” That shift led us to time windows. Every scheduled task gets assigned to a bounded window based on its execution time say, an hourly bucket. Workers don’t scan the whole table. They only look at the current active window and pick up tasks whose execute_at timestamp has arrived.</p><p>Broad scans became targeted reads. The workload became predictable.</p><p>But that solved only half the problem. We didn’t want to process subscriptions one by one either that doesn’t scale. We needed workers to claim and process batches. And with multiple workers polling the same time window, we had to make sure two workers never picked up the same task.</p><p>The solution was atomic claims at the database level. A worker claims a bounded batch in a single operation, which locks those tasks away from every other worker instantly. If a worker crashes mid-batch, claimed tasks remain in CLAIMED status. A separate watchdog periodically scans for tasks stuck past a timeout threshold and resets them to PENDING, so another worker can pick them up cleanly.</p><pre>UPDATE schedule_table <br>SET status = &#39;CLAIMED&#39;, worker_id = &#39;worker_alpha&#39;, claimed_at = NOW()<br>WHERE window_id = &#39;2026-04-16-14&#39; <br>  AND status = &#39;PENDING&#39;<br>  AND execute_at &lt;= NOW()<br>LIMIT 100;</pre><p>From there, the execution pattern is simple: claim, validate, then execute or explicitly discard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*gYyKJC0Vd8lVdOcPhbWDUw.png" /><figcaption>Claim, check the version, execute or discard. No overlap, no duplicates.</figcaption></figure><p>Each worker claims its batch, checks the version on each task, executes the valid ones, and discards the zombies. No overlap, no duplicates, no single-row processing.</p><h3>Then a CX Ticket Landed</h3><p>We thought we were done.</p><p>Versioning handled the race conditions. Time windows made the scans efficient. Atomic claims prevented duplicate execution. On paper, the system was solid.</p><p>Then a CX ticket landed in our queue.</p><p>A subscription that should have been canceled wasn’t. Somewhere in the flow, a webhook had arrived late. The event never triggered the version increment. The scheduled task ran, saw a stale state, and quietly discarded itself. The system did exactly what we designed it to do and still got it wrong.</p><p>That’s the thing about distributed systems. They don’t fail loudly. They drift. Webhooks arrive late or not at all. External APIs timeout at the worst possible moment. Events that were supposed to update the local state just… don’t show up. And your perfectly designed system keeps humming along, confidently acting on a reality that no longer exists.</p><p>We needed a safety net.</p><p>But we didn’t want to build a massive reconciliation pipeline that scans every subscription every night that’s just the midnight cron job wearing a different hat. Most subscriptions don’t need special attention. Most of the time the event-driven flow works perfectly.</p><p>So we targeted only the ones where confidence was low. Subscriptions due for cancellation soon that hadn’t received a state update in days. Tasks stuck in an intermediate state without final confirmation. Jobs that had already failed multiple times and couldn’t be trusted to resolve on their own.</p><p>For those cases, we bypass local assumptions entirely and go straight to the source of truth the external billing provider. We fetch the authoritative state, compare it with our internal projection, and check if they still agree. If they do, nothing happens. If they don’t, we repair the local model and increment the version which automatically zombifies any stale pending work created under the old reality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Q6FetZgG5IC5j2AvKXgHg.png" /><figcaption>When local state and external state disagree, the verification worker repairs the projection and zombifies any stale scheduled work.</figcaption></figure><p>This is an important distinction: verification isn’t just an audit step. It’s part of the recovery model. It’s how the system heals itself when reality and local state drift apart.</p><p>That CX ticket was the last one we got about a missed cancellation.</p><h3>The End of the Midnight Cron Job</h3><p>We never set out to build a distributed scheduling system.</p><p>We set out to handle one PM’s request for a 7-day grace period. Then another PM’s request. Then another. And somewhere between the whiteboard sessions, the CX tickets, and the zombie tasks, we ended up somewhere completely different from where we started.</p><p>That’s usually how it goes.</p><p>Nobody sits down and says “today I’m going to design a version-aware, time-windowed, self-healing lifecycle engine.” You build the simple thing. It works. The business grows. The simple thing starts groaning under the weight. And one day you find yourself staring at a midnight cron job that’s been quietly holding everything together with duct tape and good intentions.</p><p>The real lesson isn’t a pattern or an architecture decision. It’s that the scheduling problem and the state management problem are not the same problem, and most systems treat them like they are, until they can’t anymore.</p><p>If you’re still running your own midnight cron job, take a look at what it’s actually carrying.</p><p>It might be holding more than you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf1c9d2f9b79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elementor-engineers/every-scheduling-system-has-this-bug-most-teams-havent-found-it-yet-cf1c9d2f9b79">Every Scheduling System Has This Bug. Most Teams Haven’t Found It Yet</a> was originally published in <a href="https://medium.com/elementor-engineers">Elementor Engineers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Didn’t See It Coming: The Redis CPU Spike That Hit Hard]]></title>
            <link>https://medium.com/elementor-engineers/we-didnt-see-it-coming-the-redis-cpu-spike-that-hit-hard-12ebf465505c?source=rss-c7351d02921a------2</link>
            <guid isPermaLink="false">https://medium.com/p/12ebf465505c</guid>
            <category><![CDATA[cpu]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cache]]></category>
            <dc:creator><![CDATA[Tal Peretz]]></dc:creator>
            <pubDate>Sun, 30 Mar 2025 14:13:19 GMT</pubDate>
            <atom:updated>2025-03-30T15:13:34.935Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwzFZP9_8niqYVV4M5lM_Q.jpeg" /></figure><p>That morning started like any other, I grabbed my coffee, sat down, and began catching up on messages. But then, complaints started coming in. Teams using our services were noticing slow response times, and frustration was growing. Everything had been running smoothly the day before, but now things were crawling. Something was clearly wrong, and we had to find out what, fast.</p><p>We jumped straight into investigating what was going on. At the same time, our monitoring channels were lighting up with alerts, Redis CPU was maxed out. At first, it caught us off guard, CPU issues in Redis? That’s not something you expect to see. Redis usually has issues with memory running out, keys getting evicted, or slow queries causing delays, not CPU spikes. But here we were, looking at a CPU bottleneck instead. It didn’t make sense at first, so we had to dig deeper to figure out what was really happening.</p><p><strong>Let’s go back a bit:</strong></p><p>Before diving into the issue, let’s take a step back and understand why we chose Redis in the first place.</p><p>We work with a lot of external vendors, and we were worried that we might hit our rate limits. To reduce the number of calls we made, we decided to use Redis as a caching layer. It was fast and efficient, making it a good fit for our needs at the time.</p><p><strong>Why Did It Take So Long to Notice?</strong></p><p>At first, we had only a handful of subscriptions, so Redis was running smoothly. We didn’t think twice about it. As we started moving old subscriptions from our legacy app to the new one, the numbers kept growing, but the increase was slow. Every day, a little more data was added, and when we looked at the Redis graphs, we saw a steady rise in usage. It made sense because more subscriptions meant more data, so we didn’t see any warning signs. At the same time, new subscription purchases were coming in, adding even more load. Since the growth was gradual, Redis seemed to handle it fine for a while. But after we migrated a large batch of subscriptions, things changed fast. The graphs, which had been climbing gradually, suddenly shot up. Redis CPU usage spiked beyond its limit, and we saw that it was completely overloaded. That was the moment we realized we had been slowly pushing Redis too far, and now it was choking.</p><p><strong>What Went Wrong When Redis Failed:</strong></p><p>Our internal services relied on the subscription service, and the subscription service depended on Redis. So when Redis started struggling, things started breaking fast. Requests were failing left and right, services that had worked perfectly fine the day before were suddenly unreliable, and lost connections started piling up. Teams were frustrated, trying to figure out why their requests weren’t going through. Then, our customers started noticing — transactions weren’t completing, pages weren’t loading, and frustration was growing. What started as a small hiccup quickly turned into chaos, and we had no choice but to act immediate</p><p><strong>What We Did to Stop the Chaos:</strong></p><ol><li><strong>Immediate Action:</strong></li></ol><p>As an emergency measure, because Redis was only a cacheing layer we were able to shut down Redis in order to prevent further disruptions. This allowed our customers to continue working while we investigated the root cause. At the same time, we had a concern that we might reach the rate limits set by our vendors. However, at this point, we didn’t have an alternative option, as our customers were already experiencing service disruptions.</p><p>2. <strong>Analysis:</strong></p><p>We carefully examined our Redis usage and found that we were frequently executing wildcard queries such as:</p><pre>subscriptionId:*:version:*</pre><p>Wildcard searches in Redis are problematic because they require scanning large portions of data to find matches. This can significantly increase CPU usage and slow down performance, especially as the dataset grows. Instead of using wildcard searches, we refactored our queries to fetch data directly using specific keys. By doing this, we eliminated unnecessary scans, reduced CPU load, and improved overall efficiency.</p><p>This approach had been implemented in the early stages of development when we were uncertain about our data structure.</p><p>Over time, we refined our data model, but we overlooked the need to update our query patterns accordingly.</p><p>3. <strong>Fixing the Queries:</strong></p><p>Instead of using wildcard searches, we refactored our queries to fetch data directly using specific keys.</p><p><em>(Previously, we used wildcard queries like </em><em>subscriptionId:*:version:*, but we optimized them to use specific keys like </em><em>subscriptionId:1234:version:1.)</em></p><p>Turns out that this was the smoking gun: refactoring the queries significantly reduced the CPU load and improved Redis efficiency.</p><p><strong>The Moment of Truth: Did It Work?</strong></p><p>The impact was immediate. The moment we rolled out the updated code with optimized queries and turned Redis back on, we saw a huge difference. CPU usage dropped back to normal, and everything started running smoothly again. Failed requests stopped, response times improved, and the chaos from earlier in the day finally settled.</p><p>Wildcard queries are bad in Redis because they force the server to scan the entire keyspace to find matches. Redis is designed for speed, but that speed depends on knowing exactly what key to look up. When you use wildcards, Redis has to do a lot more work behind the scenes, especially as your dataset grows which leads to high CPU usage and slow performance.</p><p>It was a huge relief to see the system back on track.</p><p><strong>Conclusion:</strong></p><p>This incident taught us an important lesson about query optimization and scaling considerations when using Redis.</p><p>If there’s one takeaway from this post, it’s this: <strong>never leave wildcard queries lying around.</strong> They’re fine for debugging or one-off scripts, but they should never live in production code.</p><p>Here’s how to keep your Redis setup healthy as you scale:</p><ul><li><strong>Avoid wildcard queries</strong> at all costs.</li><li><strong>Continuously review and refactor</strong> your data access patterns.</li><li><strong>Monitor Redis CPU and memory metrics</strong> as usage grows.</li></ul><p>Scaling a system isn’t just about handling more traffic; it’s also about making sure your database and caching strategies evolve with your growth. Learn from our experience, optimize your queries, monitor your Redis usage, and always expect the unexpected</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12ebf465505c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elementor-engineers/we-didnt-see-it-coming-the-redis-cpu-spike-that-hit-hard-12ebf465505c">We Didn’t See It Coming: The Redis CPU Spike That Hit Hard</a> was originally published in <a href="https://medium.com/elementor-engineers">Elementor Engineers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>