<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Satvik Singh on Medium]]></title>
        <description><![CDATA[Stories by Satvik Singh on Medium]]></description>
        <link>https://medium.com/@satvik19nitm?source=rss-13497a26d3ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CAHL4SaIiwTkQ9nw</url>
            <title>Stories by Satvik Singh on Medium</title>
            <link>https://medium.com/@satvik19nitm?source=rss-13497a26d3ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:25:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@satvik19nitm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I built the simplest key value store I could, then spent a week trying to break it]]></title>
            <link>https://medium.com/@satvik19nitm/i-built-the-simplest-key-value-store-i-could-then-spent-a-week-trying-to-break-it-f3181cdeb970?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3181cdeb970</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[blog]]></category>
            <category><![CDATA[case-study]]></category>
            <category><![CDATA[key-value-store]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 13:14:30 GMT</pubDate>
            <atom:updated>2026-09-14T13:14:30.712Z</atom:updated>
            <content:encoded><![CDATA[<p>Most write ups of this design explain how it works. This one is about what happens when it goes wrong, because that is the part I could not find written down anywhere, and it is the part that found two real bugs in my own code.</p><p>The design is called Bitcask. It came out of a company called Basho around 2010, as the storage engine underneath their database Riak. Basho went bust in 2017. Riak still exists. Neither of those facts matters much for what follows, because the design itself is lovely and worth understanding on its own.</p><p>I wrote it from scratch in Go: about 1,100 lines for the store, 600 lines of tests, and another 1,300 lines of benchmarks and failure injection. Then I corrupted files, flipped bits, truncated things, and killed the process 4.2 million writes into a run to see what survived.</p><p>Everything below is measured on one machine, an Apple M4 Max with 36 GB of memory, on macOS 26.6.2 and Go 1.25. Absolute numbers will be different on your hardware. The comparisons between settings, which is what actually matters here, were all taken on the same box with one thing changed at a time.</p><h3>The whole idea in two paths</h3><p>Here is the entire design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kf4ClDgqw5xHwcfB6HrD-g.png" /></figure><p>Two rules produce all of it.</p><p><strong>Rule one: never change a file. Only ever add to the end of it.</strong></p><p>When you store a key, the store appends it to the end of the file it currently has open. It does not go and find the old copy. It does not move anything. It writes at the end and stops.</p><p>Updating a key works the same way. You get a second copy of that key further along in the file. The old copy just sits there.</p><p>Deleting a key also works the same way. You append a small note that says “this key is gone”. Nothing is erased.</p><p><strong>Rule two: keep a map in memory that says where everything is.</strong></p><p>For every key that currently exists, memory holds a small entry: which file it is in, how many bytes into that file the value starts, and how long the value is.</p><p>That is what makes reads fast. A read is a hash map lookup, which gives you an exact byte position, followed by one read at that position. There is no tree to walk down. There is no searching. The store never looks at a byte it does not need.</p><p>It also explains the two things this design cannot do, both of which I will come back to with numbers:</p><ul><li>You cannot ask for “all keys between A and M”, because nothing on disk is in key order.</li><li>Every key you have ever stored must fit in memory. Not the values, just the keys and a little bookkeeping. But all of them, always.</li></ul><h3>One record</h3><p>Everything on disk is a sequence of these:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F7MS25LApdDo0YgFwPCcsA.png" /></figure><p>The header is 20 bytes. The interesting field is the first one, the checksum. It is computed over every byte that follows it.</p><p>The reason that matters: if the machine dies halfway through writing a record, you end up with a record whose bytes do not match its own checksum. On the next startup the store notices and refuses to use it. Without the checksum it would happily hand you half a value and a few bytes of whatever was there before.</p><p>That is the theory. Section “One bad byte” below is what actually happens.</p><h3>The first bug: a delete that came back to life</h3><p>Before benchmarking anything, I wrote the boring tests. Store a key, read it back. Overwrite it, read the new value. Delete it, confirm it is gone. Restart the store, confirm everything is still there.</p><p>That last one failed.</p><pre>after reopen: 500 keys, want 450</pre><p>I had stored 500 keys and deleted every tenth one. After a restart, all 500 were back.</p><p>Here is why, and it is a nice illustration of how these pieces interact.</p><p>Reading every record on startup is slow, so when the store shuts down cleanly it writes out a small snapshot of its in-memory index. On the next startup it reads the snapshot instead of the data file. Much less to read.</p><p>But the snapshot is built by walking the in-memory index. And a delete works by <em>removing</em> the key from that index. So by the time the snapshot gets written, the deleted key is simply not there to be written down. The snapshot records the 450 keys that still exist and says nothing at all about the 50 that were deleted.</p><p>Then on startup, the store reads the older files, finds the key alive in them, and brings it back.</p><p>The delete was on disk the whole time. It was in the data file, as a note saying “this key is gone”. The snapshot just did not mention it, and the snapshot was the thing being read.</p><p>The fix is to track deletes separately in memory so the snapshot can include them, and to drop them for good only once the clean up job has removed every older file that could still hold the old value. That last condition is the subtle half. A delete note is only there to hide something. Once the thing it was hiding is gone, the note can go too, and not before.</p><p>I like this bug because a perfectly reasonable test suite misses it. You have to test deletes <em>and</em> restarts <em>and</em> the interaction between them. Testing each on its own passes.</p><h3>What durability actually costs</h3><p>Now the measurements.</p><p>When your program writes to a file, the data does not go to the disk. It goes into memory that the operating system manages, and the operating system writes it out whenever it feels like it. To force the issue you call fsync, which means roughly &quot;do not come back until this is genuinely on the physical device&quot;.</p><p>So: how often should you call it? I built three options and measured all of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lpaYmjfkwJEwzgS5wf6Yg.png" /></figure><p>Calling fsync on every single write is about <strong>2,940 times slower</strong> than never calling it. 244 writes a second against 717,000.</p><p>One honest caveat, because that number is partly about my machine. On macOS, Go’s file sync uses something called F_FULLFSYNC, which is a genuine full flush all the way to the hardware. On Linux, plain fsync is usually cheaper, and also weaker. The gap on a Linux server would be smaller. It would still be enormous.</p><p>The per-write cost tells the story more plainly: with fsync on every write, the typical write took <strong>4 milliseconds</strong>. Not microseconds. Milliseconds. That is the physical cost of asking a device to confirm a write, and there is no clever code that avoids it.</p><h3>The typical write tells you nothing about the worst write</h3><p>Here is the chart I actually wanted to make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EbqE5NIbXfWzv1PabtHQSw.png" /></figure><p>Read the bottom three lines first. They are the three settings that do not flush on every write, and for the typical write they are indistinguishable: 1.12 microseconds against 1.17 microseconds. Flushing on a timer costs the typical write nothing at all.</p><p>Now look at the right hand edge. The slowest write in the run flushing every 10 milliseconds took <strong>8.5 milliseconds</strong>, about <strong>7,552 times slower</strong> than that setting’s typical write.</p><p>That is what batching does. It does not make writes slower on average. It takes the cost that used to be spread evenly across every write, and dumps all of it on one unlucky write. The write that happens to be holding the lock when the flush fires waits for the entire flush.</p><p>Look at what the p99 column does across those three lines: almost nothing. 3.71, 3.79, 3.67 microseconds. (“p99” means: sort every write by how long it took, and look at the one 99% of the way along. It is the standard way to talk about slow requests.)</p><p>If you were watching a dashboard of p99 write latency on this system, you would see a flat healthy line while one request in ten thousand took eight milliseconds. This is the single most common shape of production latency mystery I run into, and here it is in forty lines of storage engine.</p><p>The general version: <strong>an average hides a stall, and a p99 hides a rare stall.</strong> If a slow thing happens rarely, you need to be looking at p99.9, or at the maximum, or you will not see it at all.</p><h3>Killing the process 4.2 million writes in</h3><p>I wrote a small program that stores keys in a loop and prints a line for every write the store said had succeeded. Then a second program that launches it, lets it run for two seconds, and sends it SIGKILL, the signal a process cannot catch or clean up after. Then it reopens the store and checks every write that had been acknowledged.</p><p>Three sync settings, three runs each. Across all nine runs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*vFsj166lqWQlnbjioTMqcA.png" /></figure><p><strong>4,205,798 acknowledged writes killed mid flight. Zero lost. Zero corrupt.</strong> Including with fsync switched off entirely.</p><p>This surprises people, so it is worth being precise about why, and about what it does not prove.</p><p>When your process dies, the data you have already handed to the operating system is not lost. It is sitting in the operating system’s memory, and the operating system is still perfectly alive and will write it out in its own time. Your process dying has nothing to do with it.</p><p>fsync does not protect you from your process dying. It protects you from <strong>the machine</strong> dying: a power cut, a kernel panic, someone pulling the plug. Those lose what the operating system was still holding.</p><p>So these runs tested process death, thoroughly, and they cannot tell you anything about power loss. I cannot crash the kernel of my laptop on demand, and I am not going to pretend I did. If your reason for enabling fsync on every write is &quot;in case the service crashes&quot;, the measurements say you are paying 2,940x for nothing. If your reason is &quot;in case the datacentre loses power&quot;, you are paying it for something real.</p><p>One detail I did not expect: <strong>not a single torn record in any run.</strong> A record cut in half never happened, not once in 4.2 million chances. The reason is that each record is written with one write call, and the operating system copies that whole buffer into its memory as one operation. There is no moment where half of it has landed. To actually produce a torn record I had to go and truncate the file by hand, which is what the next section is about.</p><h3>One bad byte</h3><p>If SIGKILL will not damage a file, I will do it myself. I wrote failure tests that damage files deliberately and then check what recovery does.</p><p>The first one is the honest version of the torn write. I truncated the last record of a file partway through, deleted the index snapshot so recovery had to read the data file, and reopened.</p><p>Result: 399 of 400 keys recovered. One key lost, the one that was being written. Nothing corrupt. That is exactly the promise, and it holds.</p><p>Then I flipped a single bit in the middle of a file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLRVg_LYMErhU6ToDGXcuQ.png" /></figure><p>Read the first row. One flipped bit, 10% of the way into the file, and <strong>4,500 of 5,000 records became unreachable.</strong></p><p>The good news is in the footnote: not one corrupt value was ever handed back, in any of these runs. The checksum did its job perfectly. It is designed to detect exactly this and it detected exactly this.</p><p>The bad news is what happens next, and this is the part I think is genuinely underappreciated about append only logs.</p><p>Records have no fixed length and no markers between them. The only way to find the start of record N+1 is to read record N’s header and use its length fields to skip forward. So when a record fails its checksum, you have lost your place in the file. You cannot skip the damaged record, because the length you would need to skip is itself part of what might be damaged. You cannot scan forward looking for the next record, because there is no pattern that identifies one.</p><p>All you can safely do is stop and keep everything you read before the damage. Which means <strong>one bad byte costs you the entire rest of that file.</strong></p><p>That is why keeping files small matters far more than it looks. This is usually sold as housekeeping, so that the clean up job has manageable pieces to work with. It is actually the thing that bounds your blast radius. With one 10 GB file, one bad byte can cost you 10 GB. With 128 MB files, the same bad byte costs you at most 128 MB, because only one file lost its place.</p><p>I also appended 4 KB of zeros to the end of a good file, which is what some filesystems hand back after a crash for blocks that were allocated but never written. All 300 keys survived. A header of all zeros claims a key length of zero, which is rejected before anything else happens.</p><h3>The second bug: a snapshot that looked complete</h3><p>Then I damaged an index snapshot, expecting it to cost nothing, because a snapshot is only a cache. If it is unusable you read the data file instead and lose a bit of startup time.</p><p>I truncated one to exactly half its length. 200 of 400 keys came back, and recovery reported no problem whatsoever.</p><p>The snapshot was a series of entries, one after another, with nothing recording how many there should be. My truncation happened to land exactly on an entry boundary, because every entry in that file was the same size. So reading it hit a clean end of file after 200 entries and concluded it had read a complete snapshot of a store that contained 200 keys.</p><p>Half the database, gone, no error anywhere. Worse than the corruption case, because the corruption case at least told me something was wrong.</p><p>The fix is a footer: a marker, the number of entries, and a checksum over all of them. If the footer is missing or the count does not match what was actually read, the snapshot is thrown away and the data file is read instead. After that, the same truncation recovers all 400 keys and reports the snapshot as bad.</p><p>The lesson generalises well past this project. <strong>A file format that can be truncated at a record boundary and still parse is a file format that can lose data silently.</strong> If you have designed anything that appends fixed size entries to a file, ask what happens when it gets cut off at exactly the wrong place. “It parses fine and returns less data” is the most dangerous possible answer.</p><p>Both of my bugs were silent. Neither threw an error. Both would have passed a test suite that only checked the happy path. That is the actual argument for writing failure tests: not that they find crashes, but that they find the things that do not crash.</p><h3>The price of never overwriting anything</h3><p>Back to rule one. If updating a key appends a new copy and leaves the old one, disk usage grows with how many times you write, not with how much data you have.</p><p>I took 50,000 keys of 200 bytes each, about 11 MB of actual data, and rewrote all of them N times over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qtgDnzz8to_McO1TmZa8wg.png" /></figure><p>After 50 rewrites the directory held 638 MB. For 11 MB of data. <strong>57 times more disk than data.</strong></p><p>This is what the clean up job is for. It walks the index, copies every value that is still live into fresh files, and then deletes the old files. Everything that was an old copy or a delete note never gets copied, so it disappears. 638 MB becomes 54 MB and takes 0.65 seconds.</p><p>The other number worth noticing: to store 11 MB of data through 50 rewrites, the store wrote 51 times that amount in total, counting the clean up job’s copying. If you are on cloud storage that charges per write operation, or an SSD with a write endurance rating, that multiplier is a line item and not a detail.</p><h3>The stall nobody sees</h3><p>The clean up job has to move a lot of data, so I wanted to know what it does to a writer that is trying to work at the same time. I ran one writer flat out and started the clean up underneath it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0P-_8jduApE3CWFT7dxzMw.png" /></figure><p>The p99 does not move. 23, 23, 24 microseconds. If you only had the p99 you would conclude the clean up job is free.</p><p>The slowest write was 21 milliseconds. About <strong>879 times</strong> the p99.</p><p>I want to be clear that this is my implementation’s fault and not something inherent. I deliberately built the clean up to take the lock briefly per key rather than holding it for the whole run, precisely to avoid this. That worked, mostly: 23 microseconds at p99 is the proof that it worked. But when the clean up job fills a file and rolls over to a new one, it flushes to disk and writes a snapshot while still holding the lock. That takes about 14 milliseconds, and whatever write arrives during it waits.</p><p>So the fix is not “hold the lock for less time per key”. It is “do not do slow I/O while holding the lock at all”. Which is the same lesson as the batched flush section, arrived at from a completely different direction, and it is the thing I would actually fix next.</p><p>Notice the shape repeating. Batched flushing: median fine, tail terrible. Clean up job: p99 fine, tail terrible. Both times the cause is one thread doing something slow while holding something everyone else needs. Once you have seen it here, at this scale, you recognise it in bigger systems.</p><h3>After a crash, nothing works until the index is rebuilt</h3><p>The index lives in memory. Memory is empty when you start. So before the store can answer a single request, it has to read enough of the disk to rebuild the entire index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uiNmV5vuP5BsUY_6acTbdw.png" /></figure><p>For 5 million keys, that is 1.3 seconds after a crash. Extrapolate to 50 million and you are looking at over ten seconds of a service that is up, listening, and unable to answer anything.</p><p>The snapshots help, but much less than I expected. They hold about a third of the bytes of the data files, and they only made startup 1.6 to 2.1 times faster.</p><p>That gap led somewhere I did not expect. My first version of snapshot loading was actually <strong>slower</strong> than reading the data files it was supposed to be a shortcut for: 353 milliseconds against 276 for a million keys. An optimization that made things worse.</p><p>Two reasons, and both are about the same mistake. I was reading the whole snapshot into memory and then checksumming it separately, so every byte got processed twice. And I was putting every entry into a temporary map first so I could throw the whole thing away if the footer check failed, which meant two hash map insertions per key instead of one.</p><p>Fixing both, by streaming the file in one pass and checking it before inserting rather than after, got it to 127 milliseconds. Faster than the full scan, finally.</p><p>The lesson is in why I got it wrong. I had assumed startup was slow because of the amount of data being read, so I built a thing that reads less data. But startup is slow because of the cost of building a hash map with millions of entries in it. Reading the bytes was never the bottleneck. On an SSD with a warm page cache, reading 165 MB is close to free. Inserting a million entries into a hash map is not.</p><p>I only found this because I measured a thing I assumed I already knew the answer to.</p><h3>The ceiling</h3><p>This is the number that decides whether you can use this design at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7hayEOn_kAwLmkPcMjC_EA.png" /></figure><p><strong>117 bytes of memory per key</strong>, for a 16 byte key. 165 bytes for a 64 byte key. Flat and stable from 250,000 keys upward.</p><p>The index entry itself is 24 bytes. So where does 117 come from? 24 bytes for the entry, 16 bytes for the string header that points at the key, and 16 bytes for the key itself once Go’s allocator has rounded it up. That is 56. The remaining 61 bytes are hash map overhead: Go keeps spare slots so the map does not get too full, and you pay for the empty ones.</p><p>The consequence is the last line of the chart’s footnote. With 100 byte values, <strong>the index holds about 86% as many bytes as the data on disk.</strong> That is not an index. That is a second copy of your database that has to be in memory.</p><p>Made concrete:</p><ul><li>10 million keys: 1.2 GB of memory, before you store a single byte of value</li><li>100 million keys: 12 GB</li><li>1 billion keys: 117 GB</li></ul><p>And nothing you do to your values changes this. Values live on disk. You could have 50 KB values and terabytes of data, and the index cost would be identical, because it only depends on how many keys you have and how long they are.</p><p>This is also the one thing about this design you cannot tune your way out of. Slow flushing, you can batch. Wasted disk, you can clean up. Corruption blast radius, you can bound with smaller files. The index has to be in memory, complete, always, and that is the design.</p><h3>So should you use it</h3><p>Mostly no, and the honest reasons are all above.</p><p><strong>It fits when:</strong> your key count is bounded and you know the bound, your values are much bigger than your keys, you only ever fetch by exact key, and you want reads that are one lookup plus one read with no surprises in them. Session stores, blob metadata, caches with a known population, single machine embedded storage. There is a reason the design has lasted.</p><p><strong>It does not fit when:</strong> your key count grows without limit, because you will hit a memory wall you cannot climb. Or you need to scan ranges of keys, which it fundamentally cannot do. Or you cannot afford a startup that scales with your data. Or you have lots of tiny values, where the index approaches the size of the data.</p><p>If you want these ideas without the memory ceiling, read about LSM trees next. They keep the sorted structure on disk instead of the whole index in memory. That buys unbounded key counts and range scans. It costs you the simplicity of “one lookup, one read”.</p><h3>What I would actually take from this</h3><p>The measurements I did not expect going in:</p><ul><li>fsync costs 2,940x, and does not protect you from your own process dying. 4.2 million acknowledged writes survived SIGKILL with it switched off.</li><li>Batching a slow operation does not make things slower on average. It makes a few things 7,552x slower, and a p99 dashboard will not show you.</li><li>One flipped bit cost 90% of a file, because an append only log cannot find its place again after damage. Small files are a correctness feature, not housekeeping.</li><li>The index cost 86% as many bytes as the data on disk.</li><li>My snapshot optimization was slower than no optimization, because I optimized the wrong resource.</li><li>Both bugs I found were silent. Neither raised an error.</li></ul><p>And the general one. I wrote correctness tests first and they all passed, and the code had two silent data loss bugs in it. What found them was not more careful reading. It was deliberately breaking things: truncating a file at exactly the wrong byte, deleting a key and then restarting, flipping one bit in the middle of a file.</p><p>The tests worth writing are the ones where you have to think about what you are attacking. Test the happy path and you learn that the happy path works.</p><p><em>Every number in this post comes from a benchmark in the </em><a href="https://github.com/zen7vik/bitcask-case-study/tree/main/bitcask-study"><em>repository</em></a><em>, and the charts are generated directly from the result files, so rerunning the benchmarks regenerates the charts. Run it on your own hardware and the absolute numbers will differ, sometimes a lot. The comparisons should hold.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3181cdeb970" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Smart Reminder Cadence System: Lessons From a Surprisingly Hard Problem]]></title>
            <link>https://medium.com/@satvik19nitm/designing-a-smart-reminder-cadence-system-lessons-from-a-surprisingly-hard-problem-079b96ee96f9?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/079b96ee96f9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 16:18:25 GMT</pubDate>
            <atom:updated>2026-08-07T16:29:21.933Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7eLecwIF8NIeZtXKitktDg.png" /></figure><p>Every now and then you run into a feature that <em>looks</em> trivial on the surface… until you try to build it properly.</p><p>For me, that feature was <strong>reminder scheduling</strong>.<br>The requirement sounded innocent:</p><blockquote><em>“Send the user reminders between the creation date and the due date. More reminders as the due date approaches. Avoid weekends if possible.”</em></blockquote><p>Easy, right?</p><p>Not exactly.</p><p>What started as a “simple notifications feature” turned into a surprisingly deep engineering problem involving scheduling algorithms, UX trade-offs, data merging, weekend-awareness, and forward compatibility with timezone-based delivery.</p><p>In this post, I’ll walk through how I designed a <strong>smart reminder cadence generator, </strong>one that is deterministic, scalable, user-friendly, and robust to real-world complications.</p><h3>Why Reminder Scheduling Is Harder Than It Looks</h3><p>A simple reminder strategy might say:<br><em>“Send 3 reminders evenly spaced.”<br>“Or always remind the day before due.”</em></p><p>But real applications rarely work like that.<br>Here are the actual constraints we had to solve:</p><h3>1. The interval between creation and due date varies wildly</h3><p>Some reminders run over:<br>3 days, 10 days, 60 days, 400 days<br>A “one-size-fits-all” formula breaks immediately.</p><h3>2. Users expect reminders to cluster near the deadline</h3><p>Nobody wants a reminder:<br>at day 5, another at day 10 and nothing else until due date day 90<br>We needed <strong>density shaping,</strong> more reminders as the deadline approaches.</p><h3>3. We want to avoid weekends (but not too aggressively)</h3><p>We didn’t want hard exclusions because:<br>some users <em>might</em> be okay receiving reminders on weekends avoiding weekends too aggressively creates clustering constraints fight each other</p><p>So the rule became:</p><blockquote><em>“Try to shift reminders away from Saturday/Sunday, but don’t break the cadence if no good alternative exists.”</em></blockquote><h3>4. Reminders already sent must be preserved</h3><p>Users can change their due dates.<br> <em>We can’t resend old reminders.<br> We can’t shift past history.</em></p><p>This led to a merge algorithm:<br><strong>past reminders stay</strong>, <strong>future ones regenerate</strong>.</p><h3>5. The schedule must be deterministic</h3><p>Given the same inputs → always the same output.<br> This is important for debugging, analytics, and consistency.</p><h3>6. Eventually we’ll trigger reminders at send time using user’s timezone</h3><p>But until then, the schedule itself must be safe.</p><h3>A Smarter Way: “Cadence” Instead of “Events”</h3><p>Instead of storing timestamps, we store:</p><pre>day offsets since createdAt</pre><p>Example:</p><pre>&quot;4,7,9,13,14&quot;</pre><blockquote>…means reminders on: createdAt + 4 days, createdAt + 7 days etc.<br>This representation is clean, deterministic, and makes merges easy.</blockquote><h3>The Algorithm: Step by Step</h3><p>Let’s break down the design.</p><h3>1. Determine how many reminders to send</h3><p>The number of reminders should increase with duration, but not linearly.</p><p>We use a <strong>sub-linear growth function</strong>:</p><pre>K(T) = round(kLog * ln(T) + kRoot * sqrt(T))</pre><p>This behaves beautifully:</p><ul><li>slow growth for small T</li><li>slightly faster for long timelines</li><li>capped by an optional global max</li></ul><h3>2. Cooling-off period</h3><p>Users don’t want reminders immediately.<br>We enforce:</p><pre>coolingOffDays = clamp(10% of T, min 3, max 10)</pre><p>This ensures we start “late enough.”</p><h3>3. Density shaping near deadline</h3><p>We generate K–1 reminder points through a <strong>power curve</strong>:</p><pre>t = normalized position<br>bias = 1 - (1 - t)^exponent</pre><p>With exponent depending on T:</p><ul><li>short timelines → gentler curve</li><li>long timelines → strong clustering near deadline</li></ul><p>This guarantees reminders naturally become more frequent near the end.</p><h3>4. Always include the final reminder (T−1)</h3><p>No matter what, we send the day before due.<br>Critically: <strong>We do NOT weekend-adjust the final reminder.</strong></p><h3>5. Enforce minimum spacing (minGap)</h3><p>To avoid reminders being too close:</p><pre>minGap = max(2, floor((T - coolingOffDays) / (2 * K)))</pre><p>This ensures:</p><ul><li>longer intervals get larger gaps</li><li>reminders never cluster accidentally</li></ul><h3>6. Weekend adjustment, but with constraints</h3><p>This is where it got interesting.<br>We adjust reminders <strong>only if they are internal (not final)</strong> and <strong>only if a valid alternative exists</strong>.</p><p>Rules:<br>If Sunday (0):</p><pre>try Monday (+1)<br>else Tuesday (+2)<br>else Friday (–2)<br>else keep original</pre><p>If Saturday (0):</p><pre>try Friday (–1)<br>else Monday (+2)<br>else keep original</pre><p>A safe, minimal-shift strategy.</p><p>We also ensure:</p><ul><li>the new date is still in [1, T−1]</li><li>it respects minGap vs the previous reminder</li></ul><h3>7. After weekend adjustment: dedupe &amp; re-check minGap</h3><p>Weekend shifting can cause collisions.<br> So we re-sort, dedupe, and re-gap the list.</p><h3>8. Merging schedules when due date changes</h3><p>This was another surprisingly hard part.</p><p>When the user changes their due date:</p><ul><li><strong>Past reminders stay</strong> (already sent)</li><li><strong>Future reminders re-generate</strong></li><li><strong>We apply spacing constraints again</strong></li></ul><p>This creates a <strong>clean and predictable unified schedule</strong>.</p><h3>Putting It All Together</h3><p>The final script:</p><ul><li>handles wide date ranges</li><li>avoids weekends safely</li><li>preserves UX intuition</li><li>is deterministic</li><li>scales with future timezone-based delivery</li><li>merges cleanly after due-date changes</li><li>is safe for production systems</li></ul><p>This kind of algorithm isn’t glamorous, but it’s the backbone of a robust reminders system.</p><p>And honestly, I underestimated it at first, and yes before and after building, it looked easy, but while its implementation it made me appreciate how much nuance there is in “simple” features that interact with real user behavior.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=079b96ee96f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Thread, One Latency Budget: Every Redis Internal Is the Same Rule Wearing a Different Costume]]></title>
            <link>https://medium.com/@satvik19nitm/one-thread-one-latency-budget-every-redis-internal-is-the-same-rule-wearing-a-different-costume-42b7f75cd637?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/42b7f75cd637</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 18:35:57 GMT</pubDate>
            <atom:updated>2026-07-27T18:35:57.920Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The rule: never do an unbounded amount of work in one go. Checked against Redis 8.8.1 source, and measured.</em></p><h3>The corpus problem</h3><p>The Redis internals corpus is abundant, stale, and unmeasured. I read the 8.8.1 tree and measured it. Much of what gets repeated no longer holds:</p><ul><li>The famous active-expire rule, “if 25% of sampled keys were expired, repeat”, is wrong twice: the repeat threshold is ACTIVE_EXPIRE_CYCLE_ACCEPTABLE_STALE, default 10 (expire.c:99), applied at expire.c:447, and the 25 is a separate CPU budget constant (expire.c:98). The split landed in November 2019 (27668056, 84b01f63, antirez): posts quoting 25% are six plus years stale.</li><li>used_memory is not one atomic counter but a per-thread array (zmalloc.c:112), each slot padded to a cache line (zmalloc.c:109).</li><li>iothread.c did not exist before commit 64a40b20 (December 2024), whose message describes the 6.0 model it replaced: main thread blocked on IO threads, busy-wait burn.</li><li>hash-max-listpack-entries defaults to 512 (config.c:3349), not the 128 many posts quote.</li><li>“44 bytes or less means embstr” no longer predicts a stored key’s encoding.</li></ul><h3>How I measured</h3><p>Source tree and runtime image are both pinned at Redis 8.8.1 (image digest recorded in the results file). Every measurement ran inside a Docker container limited to 2 CPUs and 4 GB of memory, three runs per timed measurement, medians reported. The host is Apple Silicon and the container runs in a virtualized Linux guest, so absolute timings here are not bare-metal representative. I am not claiming they are. What the environment does support is A versus B comparison inside itself: same box, same image, same harness, one variable changed.</p><h3>The thesis</h3><p>Redis executes commands on one thread, and it wants to answer you in well under a millisecond. Those two facts forbid a whole category of implementation. You cannot rehash a million-bucket table in one pass, walk the heap to answer “how much memory am I using”, or free a five million element set inline without stalling every other client (section 9 measures that stall). So every internal that could take unbounded time gets a bound bolted on: one bucket per operation, a microsecond budget per cron tick, 20 keys per database per loop, five sampled keys per eviction pass, a background thread. Different costumes, same rule.</p><p>The useful consequence: once you have the rule you can predict the design. By section 4, you should be able to guess where a mechanism’s bound is before I name it.</p><h3>1. One thread, and the baggers are not the till</h3><p>Picture a supermarket with one till. io-threads hires extra baggers: they unload your trolley and pack your bags, and they scan nothing. Real parallelism, but only the till rings anything up. One customer with a trolley the length of the shop stops everybody, however many baggers you hired.</p><p>Where the metaphor misleads: the till is not slow, ringing up about 323k requests per second in this environment. Throughput is not the problem. One customer who wants the whole inventory priced is.</p><p><strong>What the source actually does.</strong> io-threads defaults to 1, is an IMMUTABLE_CONFIG (startup only), and accepts 1 to 128 (config.c:3267). At the default of 1, initThreadedIO returns immediately without spawning anything (iothread.c:877), so out of the box Redis is fully single threaded for network IO too, not just for command execution.</p><p>Turn it up and each spawned IO thread runs its own event loop via aeMain, separate from the main thread&#39;s loop, doing socket reads, protocol parsing, and writes (iothread.c:871). What it does not do is execute your command. Once a command is fully parsed, processClientsFromIOThread hands the client back to the main thread rather than re-entering processInputBuffer inside the IO thread (iothread.c:615). Read, parse, write: parallel. Execute: one thread, always.</p><p><strong>What I measured.</strong> The load: 1,000,000 random SETs to populate (roughly 632k distinct keys expected; DBSIZE not recorded), then 200,000 GETs against one fixed key while KEYS * runs ten times on a second connection. Medians of three runs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlxvjk3n6ICR-F6O-DIQWA.png" /></figure><p>The stall lives in the tail. Median max_ms goes from 2.119ms at baseline to 42.015ms during KEYS *, roughly 20x; individual runs 1.4-3.0ms baseline versus 40.4-42.0ms during the stall. Note how little p99 moved: 0.215ms to 0.335ms. Not because the stall is small: because it is brief, so few requests land inside a blocking window. Monitor p99 alone and you miss this class of problem.</p><p>io-threads 4 does not remove the stall. Read that arm on its own terms: median max goes 32.367ms without KEYS * to 72.063ms with it, the same shape as the single-threaded arm&#39;s 2.119ms to 42.015ms. Extra baggers do not help when the till is pricing the whole shop, which is what iothread.c:615 and iothread.c:871 predict.</p><p>I will not claim io-threads made the stall worse, tempting as 72.063ms against 42.015ms looks. Four IO threads plus a main thread in a two-CPU container oversubscribes the CPU, and that arm’s own baseline gives it away: 32.367ms max and 211,864.41 rps with nothing else running, against 2.119ms and 322,580.66 rps single threaded. Its noise floor starts an order of magnitude higher, so the cross-arm gap is confounded: only within-arm comparisons carry information.</p><p><strong>The bound:</strong> there isn’t one. KEYS * is your foot-gun, not Redis&#39;s. It is the constraint the rest of the post answers to: every section below caps some quantity precisely because this one cannot be capped for you.</p><h3>2. zmalloc: weigh every crate on arrival</h3><p>A warehouse has two ways to answer “how full are we”: count everything on the floor, or weigh each crate on arrival and keep a running total on a clipboard. Redis picks the clipboard: one glance, never a stocktake.</p><p>Where the metaphor misleads: there is one clipboard per worker, and the total is their sum.</p><p><strong>What the source actually does.</strong> Every allocation is accounted as it happens. PREFIX_SIZE is 0 when the allocator can report a pointer&#39;s usable size itself (HAVE_MALLOC_SIZE, true for the jemalloc build here), otherwise Redis stashes the size in a header ahead of the pointer (zmalloc.c:43). Either way the byte count is known in and out.</p><p>That count lands in a per-thread array of used_memory_entry structs, indexed by thread slot rather than one global counter (zmalloc.c:112), and each entry is padded to CACHE_LINE_SIZE so two threads incrementing their own slots do not fight over the same cache line (zmalloc.c:109). Updates go through atomicIncrGetSingleWriter or atomicIncrGet, not one global atomic add (zmalloc.c:173).</p><p>Why bother? Because the limit is enforced on the command path, not on a timer. With maxmemory set, processCommand (server.c:4391) runs an eviction pass for every command it handles, read or write (server.c:4579); only the refusal is narrower, since a command is rejected for OOM solely when it carries the denyoom flag (server.c:4591). If answering &quot;am I over the limit&quot; cost a heap walk, that cost would scale with the dataset and arrive after the deadline. O(1) accounting is not an optimization here, it is a prerequisite for having a memory limit.</p><p><strong>The kvobj finding.</strong> Here the old rule breaks. OBJ_ENCODING_EMBSTR_SIZE_LIMIT is still 44 (object.c:337) and still decides the encoding of a bare string object at creation. But on 8.8.1 a stored key and its value share one allocation, a kvobj, which re-decides the embedding: the value goes inline only when sizeof(kvobj) + (keylen+3) + (4+len) is less than or equal to CACHE_LINE_SIZE, otherwise it is allocated out of line with encoding OBJ_ENCODING_RAW whatever its own length (object.c:282-296, object.c:87). CACHE_LINE_SIZE is 64 on Linux; the 128 branch needs both __aarch64__ and __APPLE__, never true inside a Linux container even on Apple Silicon (config.h:38-44).</p><p>For a 3 byte key that leaves 64 minus 16 minus 6 minus 4, or 38 bytes. Measured, not just derived: holding the key at k:1, OBJECT ENCODING reports embstr at 36, 37, and 38, then raw at 39 through 45. Lengths 39 to 44 come back raw despite being at or under the classic limit, because the key competes with the value for the same cache line.</p><p><strong>What I measured.</strong> Bytes per key, medians of three runs, from used_memory deltas across a large batch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*GNlZOs4R2k3Rvv02rbJu-g.png" /></figure><p>Note row three: the 44 byte value is already raw. One honest gap: embstr_44 at 95 bytes per key and raw_45 at 111 differ by 16 bytes for one extra payload byte, and both are already raw, so no encoding change explains it. Suspected allocator size-class rounding, but I did not trace jemalloc&#39;s bins: unexplained, so build nothing on it.</p><p><strong>The bound:</strong> the work to answer “how much memory am I using” is pinned at O(1), so the eviction decision never becomes a stocktake. The price is bookkeeping on every allocation and a total that is a sum of shards, not one number.</p><h3>3. Encodings: the envelope and the filing cabinet</h3><p>The same contents fit in different boxes. A small hash goes in an envelope taped to the door: no index, just fields end to end, read by scanning. A big hash goes in a filing cabinet with an index: constant time lookup, a lot of steel per document. The envelope is cheaper until it is not.</p><p>Where the metaphor misleads: these are not two sizes of one container, they are different algorithms. Envelope to cabinet swaps an O(n) scan for a hash lookup, which is the real reason the switch exists. Memory is the visible symptom.</p><p><strong>What the source actually does.</strong> Every collection type carries a pair of thresholds, and the defaults are not what folklore says. hash-max-listpack-entries is 512 (config.c:3349) with hash-max-listpack-value 64 (config.c:3355). Sets get set-max-intset-entries 512 (config.c:3350) and, separately, set-max-listpack-entries 128 with set-max-listpack-value 64 (config.c:3351-3352). Sorted sets get zset-max-listpack-entries 128 (config.c:3353). Exceed either the count or the element size and the object converts, one way only: every call site converts toward the hashtable, and hashTypeConvert panics rather than downgrade one that already is (t_hash.c:1755).</p><p>The 512 is old, and the archaeology takes two steps: git log -S on the current literal finds only the 2021 listpack rename (02fd76b9), while the old spelling reaches November 2019, where hash_max_ziplist_entries already defaults to 512 (18e72c5c, Oran Agra). The 128 people quote for hashes is the set and zset entry limit, misapplied.</p><p><strong>What I measured.</strong> Hashes of increasing field count, reading MEMORY USAGE and OBJECT ENCODING. Byte-for-byte identical across all three runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j02n5e0ylzD2uEQDhyAc-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpjgb6QglNA26I3OdX6WlQ.png" /></figure><p>One field takes the hash from 6,983 bytes to 28,576 bytes, a 4.09x jump. Below the cliff, bytes per field is flat at 12 to 13 across the listpack range, which is what “no index” buys you. Above it, bytes per field starts at 55 and flattens out near 47 to 49 as the hashtable’s fixed overhead amortizes. With thousands of hashes hovering at 500 fields, you are one HSET away from quadrupling their memory, and nothing in your metrics will say why.</p><p><strong>The bound:</strong> the capped quantity is n, the length of the linear scan. A listpack lookup is O(n), so Redis caps n and pays for crossing the cap once in memory instead of repeatedly in CPU. The sacrifice is a cost cliff at a configured field count instead of a smooth curve.</p><h3>4. Incremental rehash: one box per trip, both addresses live</h3><p>You are moving house and refuse to take a day off, so you keep both addresses live and carry one box per trip. Need something? Check the new house, then the old. Every visitor carries a box over on the way out. The move takes a while. Nobody notices.</p><p>Where the metaphor misleads: nothing is ever in both places at once. A bucket lives in the old table or the new one, never both, which is why checking two tables stays correct.</p><p><strong>What the source actually does.</strong> During a resize the dict holds two tables. dictRehash(d,n) moves up to n non-empty buckets from ht[0] to ht[1], capping empty-bucket visits at n*10 so it cannot wander a sparse table forever (dict.c:405-406). Two things call it.</p><p>First, ordinary operations. _dictRehashStep calls dictRehash(d,1): exactly one bucket, opportunistically, when rehashing is not paused (dict.c:468-469). Your GET pays for one bucket of somebody else&#39;s migration. That is the visitor carrying a box.</p><p>Second, the cron. dictRehashMicroseconds(d,us) is the time-boxed variant, and the budget is in microseconds, not milliseconds (dict.c:446). kvstoreIncrementallyRehash drives it per dict with a remaining-microsecond threshold (kvstore.c:685), and serverCron gives each database&#39;s keys and expires kvstores a combined INCREMENTAL_REHASHING_THRESHOLD_US of 1000 microseconds per call (server.c:1350-1354, server.h:146). One millisecond of migration per cron tick, total.</p><p><strong>What I measured.</strong> A million keys inserted in twelve chunks while a GET benchmark ran, sampling DEBUG HTSTATS per window. Across all 36 rows, p99 median is 0.743ms. Split by whether a rehash was in progress at sample time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*2UDOPy_7CPzoAgVwEhytHA.png" /></figure><p>Mid-rehash windows do run higher: 0.104ms at the median, about 15%. A real signal, and I will not round it to zero. It is also smaller than the run’s ambient variance, since the non-rehash rows alone span 0.487ms to 1.199ms, about seven times the difference, and the worst window in the dataset (1.199ms) was not mid-rehash. Concurrency between GET traffic and migration is inferred from the chunk-per-window design, not directly timed, so I cannot cleanly attribute the 0.104ms to bucket moves rather than the insert traffic that triggers them. Over the run the keyspace went 83,334 to 999,997 keys and the table doubled three times, 131072 to 1048576.</p><p>For contrast, the alternative: stop the world and move all 999,997 keys in one pass. At this environment’s measured bulk redis-cli --pipe insert rate of about 1,153,234 keys per second, that is roughly 0.87 seconds of dead server. That figure is <strong>estimated, not measured</strong>, and errs high for a rehash: the insert rate behind it pays for protocol parse and execution per key, while a rehash only moves existing entries. Call it a loose upper bound. Even so, next to a 0.104ms median shift the trade is not subtle.</p><p><strong>The bound:</strong> two, stacked. Per operation, one bucket. Per cron slice, a microsecond budget. The sacrifice is double memory and two table checks per lookup until the move finishes. In exchange, doubling a hash table costs 0.104ms of median p99 instead of a visible pause.</p><h3>5. Active expiry: the inspector who samples twenty shops</h3><p>Health inspectors cannot visit every shop every week, so they sample: twenty shops a block, and if too many fail they redo the block. Nobody pretends that catches every violation the day it happens. It catches enough, cheaply.</p><p>Where the metaphor misleads: an inspector’s return visit is judgement. Redis’s is arithmetic, and the arithmetic is where the corpus went wrong.</p><p><strong>What the source actually does.</strong> Three constants, two endlessly conflated. ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP is 20, the keys sampled per database per loop iteration (expire.c:96). ACTIVE_EXPIRE_CYCLE_ACCEPTABLE_STALE is 10 (expire.c:99). ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC is 25 (expire.c:98).</p><p>Now the correction, plainly. The corpus says: if more than 25% of sampled keys were expired, repeat the cycle. In 8.8.1 the repeat decision compares data.expired*100/data.sampled against config_cycle_acceptable_stale (expire.c:447). It is gated on the acceptable-stale threshold, default 10, never on the CPU percentage. The 25 everyone quotes is ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC, the share of a server tick the slow cycle may burn (expire.c:98). Two constants, two jobs.</p><p>Both scale with active-expire-effort (modifiable, 1 to 10, default 1, config.c:3296): raising it lowers the repeat threshold (expire.c:292,299) and raises the CPU budget, to 43 at maximum effort (expire.c:297-298).</p><p>This is datable. git log -S on expire.c puts ACTIVE_EXPIRE_CYCLE_ACCEPTABLE_STALE in commit 27668056 (antirez, 2019-11-15), which replaced the old while (expired &gt; sampled/4), a literal 25 percent, with the dedicated 10 percent constant; 84b01f63 (antirez, 2019-11-18) added the effort scaling. Every post still teaching the 25% repeat rule describes code that changed in November 2019.</p><p><strong>What I measured.</strong> One million keys, each set with PX 5000 during a bulk load, so deadlines spread across the load rather than landing at one instant, then poll DBSIZE to zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1Wdawz-qg2edcmrT-6QVQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*W1-8wBxbRdRd68WkmeBLFg.png" /></figure><p>Effort 10 drained the same million keys end to end in a median 6,401ms against 7,900ms, roughly 19% faster. Both totals include about four seconds in which nothing was reclaimable yet: dbsize sat at exactly 1,000,000 until 3,951 to 4,217ms into every poll. Timed from the first drop to zero, the reclaim itself ran a median 3,866ms against 2,360ms, 39% faster, so the 19% understates what effort changes. What neither setting did was finish promptly. A read sees expired keys gone at once, since every lookup runs expireIfNeeded before returning (db.c:302), but the bytes come back over seconds, a sampled batch at a time.</p><p>Hence a monitoring trap: used_memory is not a live answer to &quot;did my TTLs fire&quot;: after a mass expiry it reports memory that is logically free and physically still held, for seconds.</p><p><strong>The bound:</strong> the capped quantities are keys examined per pass (20 per database per loop) and CPU share per tick. The sacrifice is timeliness: memory comes back late, and how late depends on effort, not a guarantee.</p><h3>6. Eviction: poll five voters, do not run a census</h3><p>To evict the least recently used key from a million, you would examine a million. Or sample five at random and evict the coldest. Pollsters make the same bet: ask five voters, not the electorate.</p><p>Where the metaphor misleads: a pollster estimates a population. Redis does not need the true LRU key, just a sufficiently cold one, now.</p><p><strong>What the source actually does.</strong> Eviction keeps a candidate pool sized by EVPOOL_SIZE, 16 (evict.c:36). Each pass fills it by drawing up to server.maxmemory_samples keys via kvstoreDictGetSomeKeys (evict.c:141), and the sampling loop caps sampled_keys at that same config (evict.c:605), so work per eviction is bounded by a number you set, not the keyspace. Under LFU the score is a decayed frequency counter rather than an idle time (LFULogIncr, evict.c:281; idle score is 255 minus LFUDecrAndReturn, evict.c:162), but the shape is identical: sample a few, keep 16, evict from the pool.</p><p><strong>What I measured.</strong> allkeys-lru, maxmemory 16mb, Zipf-skewed access with exponent a=1.2, 800k operations, at maxmemory-samples 5 and 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*Y8qYnyALOphUYg5q5dgeQw.png" /></figure><p>Doubling the sample bought 0.0002 of hit rate and, as a higher hit rate implies, slightly fewer evictions (49558 to 49439). Scope: one workload, one skew, one ceiling; I did not test uniform or adversarial access. But the flatness looks structural rather than lucky. Under skew most keys are cold, so any five you grab almost certainly contain one worth evicting.</p><p><strong>The bound:</strong> the capped quantity is keys sampled per eviction; the sacrifice is accuracy, since Redis evicts an approximately-coldest key. Here that lost accuracy made no measurable difference to hit rate.</p><h3>7. Fork: the photocopier that copies only the edited pages</h3><p>A photocopier duplicating a 500 page report page by page would take an hour. This one hands you the report at once and copies a page only when somebody writes on it. Duplication is nearly free; the cost arrives later, one page per edit.</p><p>Where the metaphor misleads: the pages are 4KB memory pages, the writing that triggers a copy is mostly the parent serving ordinary traffic, and Redis reports the result as the child’s private dirty pages.</p><p><strong>What the source actually does.</strong> redisFork(purpose) records start = ustime() and calls fork() (server.c:7417-7418). On the parent side server.stat_fork_time becomes ustime()-start immediately after the call returns (server.c:7448), which is exactly the pause the main thread took, and INFO publishes it as latest_fork_usec (server.c:6707). COW is accounted separately: the child reports its COW bytes over a pipe, updateChildInfo stores them (childinfo.c:106) and keeps a running peak (childinfo.c:103).</p><p><strong>What I measured.</strong> Load a dataset, dirty it under write load, BGSAVE, read latest_fork_usec and the COW peak.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wNtd9qPaoTdvfVrV4vSbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*8NM_o5w1mBe_BovPsVY-vw.png" /></figure><p>Thirty times the keys costs roughly 8x the fork time (327us to 2,562us), so the pause per million keys falls as the dataset grows. The fork is cheap and stays cheap; COW is what scales: peak observed 55,463,936 bytes, about 52.9MB, in one 3,000,000-key run; the table and the chart show medians.</p><p>One measurement note that changes what the column means: these COW figures come from rdb_last_cow_size, not the live current_cow_size. The 0.05s polls of the live counter never read higher than the completed save&#39;s figure, and current_cow_size resets to 0 the moment the save finishes.</p><p>The practical consequence: the headroom you need is not your dataset size, it is that plus the pages you dirty while the save runs. Fork does not decide that. Your write rate does.</p><p><strong>The bound:</strong> the capped quantity is work at fork time, pushed near zero by copying lazily. The sacrifice is predictability: the real cost lands later and scales with write traffic, not with anything the save controls.</p><h3>8. AOF and fsync: postbox, certified mail, or standing at the counter</h3><p>Three ways to send a letter: drop it in a postbox and walk away (no), send it certified and collect the receipts once a second (everysec), or stand at the counter until the clerk stamps it (always).</p><p>Where the metaphor misleads: the postbox is not the wilderness. Handing the letter over is write(2), which reaches the kernel page cache; the stamp is fsync. That distinction is the whole finding below.</p><p><strong>What the source actually does.</strong> flushAppendOnlyFile(int force) writes the AOF buffer out and decides fsync behaviour (aof.c:1147). Under everysec the fsync is not inline: aof_background_fsync(fd) queues it to a BIO thread via bioCreateFsyncJob (aof.c:983-984, bio.c:248), and only once 1000ms have passed with none in flight (aof.c:1347-1351).</p><p>What matters is what happens when the disk falls behind. If a background fsync is already running, Redis postpones the write itself, recording aof_flush_postponed_start instead of blocking (aof.c:1186-1194). It postpones again while less than 2000ms have elapsed, then writes anyway, increments aof_delayed_fsync and logs that the async fsync is too slow (aof.c:1196-1204). So everysec is not &quot;at most one second at risk&quot;: it is one second plus up to two more when a slow disk holds the previous fsync.</p><p><strong>What I measured.</strong> A 200,000 SET benchmark per mode, then durability: a monotonic INCR loop, record the last acked value, SIGKILL, restart, compare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IedcuBEyI8nCXb7qHazTmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LZXshp3fz_tw3-iZ1KJ8g.png" /></figure><p>Thirty times the keys costs roughly 8x the fork time (327us to 2,562us), so the pause per million keys falls as the dataset grows. The fork is cheap and stays cheap; COW is what scales: peak observed 55,463,936 bytes, about 52.9MB, in one 3,000,000-key run; the table and the chart show medians.</p><p>One measurement note that changes what the column means: these COW figures come from rdb_last_cow_size, not the live current_cow_size. The 0.05s polls of the live counter never read higher than the completed save&#39;s figure, and current_cow_size resets to 0 the moment the save finishes.</p><p>The practical consequence: the headroom you need is not your dataset size, it is that plus the pages you dirty while the save runs. Fork does not decide that. Your write rate does.</p><p><strong>The bound:</strong> the capped quantity is work at fork time, pushed near zero by copying lazily. The sacrifice is predictability: the real cost lands later and scales with write traffic, not with anything the save controls.</p><h3>8. AOF and fsync: postbox, certified mail, or standing at the counter</h3><p>Three ways to send a letter: drop it in a postbox and walk away (no), send it certified and collect the receipts once a second (everysec), or stand at the counter until the clerk stamps it (always).</p><p>Where the metaphor misleads: the postbox is not the wilderness. Handing the letter over is write(2), which reaches the kernel page cache; the stamp is fsync. That distinction is the whole finding below.</p><p><strong>What the source actually does.</strong> flushAppendOnlyFile(int force) writes the AOF buffer out and decides fsync behaviour (aof.c:1147). Under everysec the fsync is not inline: aof_background_fsync(fd) queues it to a BIO thread via bioCreateFsyncJob (aof.c:983-984, bio.c:248), and only once 1000ms have passed with none in flight (aof.c:1347-1351).</p><p>What matters is what happens when the disk falls behind. If a background fsync is already running, Redis postpones the write itself, recording aof_flush_postponed_start instead of blocking (aof.c:1186-1194). It postpones again while less than 2000ms have elapsed, then writes anyway, increments aof_delayed_fsync and logs that the async fsync is too slow (aof.c:1196-1204). So everysec is not &quot;at most one second at risk&quot;: it is one second plus up to two more when a slow disk holds the previous fsync.</p><p><strong>What I measured.</strong> A 200,000 SET benchmark per mode, then durability: a monotonic INCR loop, record the last acked value, SIGKILL, restart, compare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tH73O9vtheXJxphbDA7tKQ.png" /></figure><p>The host-measured 350ms against 58ms includes 40–100ms of docker exec overhead, identical in both arms, so the comparison stands but neither absolute is clean. INFO commandstats is clean: the DEL call cost 278,250 to 287,327 microseconds of server command time, against 13 to 25 microseconds for UNLINK.</p><p>The concurrent traffic makes it visible: max latency for unrelated GETs was a median 279.807ms during DEL (runs 277.759 to 287.743ms) against 2.143ms during UNLINK (0.775 to 3.263ms). That is section 1 from the inside: one command held the only thread for a quarter of a second while everyone else waited. lazyfree_pending_objects reads 1 right after each UNLINK and drains to 0 within a second, the night crew finishing up.</p><p><strong>The bound:</strong> the capped quantity is main-thread time per delete, cut to microseconds by handing the free to a background thread when the object is worth it. The sacrifice: the memory is not back when the command returns.</p><h3>The rule, and where it stops protecting you</h3><p>One thread, a latency budget, so never do an unbounded amount of work in one go. Every section above is that rule in a different costume: twenty keys per expire loop, five sampled keys per eviction pass, one bucket per rehash step, a microsecond budget per cron tick, a background thread for the big frees.</p><p>Two qualifications. Section 1 is not an instance of the rule, it is the constraint that forces the rest. And section 4 is what success actually looks like: bounded work does not become invisible, it gets smeared until it is smaller than the ambient variance of the same run, 0.104ms of median p99 while a hash table doubles.</p><p>Redis bounds what it controls. It cannot bound what you ask for:</p><ul><li>KEYS * walks the whole keyspace on the one thread that runs commands (section 1: 42.015ms max latency for everybody else).</li><li>SMEMBERS or HGETALL on a huge collection is the same shape: O(n) work on the single thread, plus a reply buffer sized by your data, not measured here (section 1).</li><li>A long Lua script or a large MULTI holds the thread for its full duration, by design though not measured here (section 4: the cron&#39;s rehash slice is microseconds, and it waits behind you).</li><li>DEL on a 5,000,000 element collection is unbounded work you requested synchronously (section 9: 279.807ms of collateral latency, against 2.143ms for UNLINK).</li></ul><p>The internals are all the same trade, so the technique transfers. When you must do something unbounded, you bound it: SCAN instead of KEYS, HSCAN instead of HGETALL, UNLINK instead of DEL, chunked MULTI instead of one big one. And check the thresholds yourself, because the defaults are not what the corpus says.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42b7f75cd637" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I measured OKF, RAG, a knowledge graph, and plain grep on production code]]></title>
            <link>https://medium.com/@satvik19nitm/i-measured-okf-rag-a-knowledge-graph-and-plain-grep-on-production-code-a426386651e8?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/a426386651e8</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 07:14:36 GMT</pubDate>
            <atom:updated>2026-07-18T07:14:36.936Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozttiW2We4mMo0pT479v3w.png" /></figure><p>Since Google Cloud published the Open Knowledge Format in June, my feed has been full of “OKF vs RAG” posts. I counted ten in five weeks. They all reach the same conclusion: it’s not either/or, use curated knowledge for stable facts and RAG for big unstructured corpora.</p><p>None of them contain a measurement. And none of them include, even as a baseline, the retrieval pattern that shipping coding agents (Claude Code, Cursor, Aider) already use as a default: search the tree, read the file. Practitioners obviously know grep exists. The comparison posts just don’t put it on the bench next to the things they’re comparing, and it turns out that omission matters.</p><p>So I benchmarked it. Four retrieval patterns, twelve real on-call questions, two production services at the cybersecurity company where I work (a ~500-file TypeScript risk-scoring service and a Go workflow-orchestration service). Every arm was charged for all context it ingested, including failed attempts, and every answer was scored against ground truth I verified in source.</p><p>Twelve questions and two repos is a case study, not a survey. I’ll keep every claim scoped to what the data supports, and the per-question numbers are all below so you can disagree with my scoring.</p><h3>What I found (in this benchmark)</h3><ul><li>No pattern won both question types. Grep won all five code-detail questions. Curated docs won all four architecture questions. The two things the debate compares, OKF-style bundles and RAG, each lost the categories the other won.</li><li>Grep behaved like the baseline to beat: cheapest on most questions, the only arm that never returned a poisoned answer, and one of two that failed honestly when the answer wasn’t in the repo.</li><li>Every RAG win was doc-borne. An ablation that removed the curated docs from the corpus, retrieval mechanism unchanged, dropped an architecture answer from 3/3 facts to 0/2.</li><li>The curated docs contained three confidently wrong facts, each contradicted by code that had been settled for over two years before the docs were written. Retrieval then surfaced one of those errors 26 times against 2 mentions of the correct answer.</li><li>The derived layers lagged the tree. A knowledge graph built five days earlier scored 0/3 on a two-week-old feature that was sitting in the working tree it was derived from.</li></ul><h3>Setup</h3><p>The four arms:</p><ol><li><strong>Curated docs (the OKF pattern).</strong> Both services carry an in-repo knowledgebase/ of markdown architecture docs, discoverable through an index table in the repo&#39;s agent-instructions file. This is structurally what OKF formalizes: a folder of interlinked markdown that humans and agents can read. Relevant disclosure: these docs were AI-generated by an internal doc-writing pipeline, which I&#39;d argue is the realistic OKF-at-scale condition. Nobody hand-writes and hand-maintains bundles for forty repos.</li><li><strong>Knowledge graph.</strong> The repos are indexed with graphify (an open-source tool by Safi Shamsi, not mine), which builds a typed entity/relationship graph from AST plus semantic extraction, with query, path, and explain operations.</li><li><strong>Agentic grep.</strong> Cold, no index: the greps an agent would run, then targeted file reads.</li><li><strong>BM25 RAG.</strong> A standard lexical RAG pipeline: 50-line chunks with 10-line overlap, camelCase-aware tokenization, Okapi BM25, top-5 retrieval. Production RAG often adds dense embeddings and reranking, so treat the BM25 rows as a floor for retrieval quality. The findings I lean on from this arm (the ablation, the error amplification) are properties of the corpus, not the scorer, and survive a better retriever.</li></ol><p>Accounting. Context tokens were counted with tiktoken over everything each arm ingested, including failed queries. If the graph burns a traversal on an irrelevant subgraph before finding the answer, those tokens count, because an agent would have read them. Latency isn’t reported as a differentiator: every arm ran locally in about two seconds or less per operation; the cost that matters is context.</p><p>Questions. Twelve questions I or my teammates have actually needed answered on-call, in three categories: code-detail (“what does this upsert actually do to this flag”), architecture (“map the message exchanges and HTTP dependencies”), and cross-repo traps (questions whose answers live in a <em>different</em> service, to test whether each pattern can say “not here”). The full list and scoring rule are in the appendix. Identifiers below are lightly renamed.</p><h3>Results</h3><p>Scores are key facts recovered / total, with approximate context tokens ingested. No confidence intervals: at n=12 they would be decoration, so you get the raw rows instead.</p><p><strong>Repo 1</strong> (TypeScript scoring service, 8 questions):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Udq7G1wjYr9mSvKyEnWQew.png" /></figure><p>Totals across the seven answerable questions in this repo: <em>grep ~11K tokens, BM25 ~17K, graph ~22K</em>. Docs answered six of the seven at ~17.5K charged per-question; in a real session doc reads amortize across questions (four of these questions share one doc), so the realistic figure is closer to half that.</p><p><strong>Repo 2</strong> (Go workflow service, 4 questions): the pattern repeated. Grep cheapest and clean on both code-detail questions. Docs won the architecture question but had zero coverage of a two-week-old feature. The graph, built five days before, had zero nodes for that feature and scored 0/3 on it, while BM25, which re-chunks the tree on every run, got 2/3. On the trap question the graph failed confidently again; grep passed <em>with a pointer to the owning service</em>, because a code comment named it.</p><h3>Five things the OKF-vs-RAG framing misses</h3><h3>1. Question type picked the winner, not tooling ideology</h3><p>In this benchmark, grep won all five code-detail questions: exact-string match makes the routing step nearly free, and the evidence you read is the ground truth itself. Curated docs won all four architecture questions: they were the only layer holding narrative, the deployment topology, the single-point-of-failure warnings, the “this recurring error is expected, don’t escalate” triage notes. No retrieval mechanism recovers that from raw code, because it isn’t <em>in</em> the code.</p><p>An evaluation that doesn’t split by question type will conclude whatever its question mix implies.</p><h3>2. Agentic grep should be a first-class baseline</h3><p>In this benchmark it outperformed every derived retrieval layer on code-detail tasks while requiring no indexing infrastructure, no generation pipeline, and no freshness maintenance. It was also the only arm that caught the curated layer’s factual errors, and it failed honestly on both traps.</p><p>To be precise about the target: this is not “nobody knows grep exists.” Agent builders ship it as the default. The gap is in the comparison discourse, which debates two derived layers without benchmarking either against the pattern agents already use. Any proposed knowledge layer should have to beat that baseline, on numbers.</p><h3>3. Every RAG win here depended on curated documents</h3><p>I re-ran the RAG arm with one change: the curated docs removed from the corpus. Same chunker, same scorer, same everything. The architecture question fell from 3/3 facts to 0/2, and the retrieved set contained zero occurrences of the answer’s key terms.</p><p>The conclusion is not “BM25 is bad.” A dense retriever with a reranker would raise the retrieval floor. The conclusion is that in this benchmark, every RAG win was the retriever surfacing chunks <em>of the curated docs</em>, and that property doesn’t change with a better scorer. Corpus-with-curation vs corpus-without was the variable that mattered; the retrieval mechanism was plumbing.</p><h3>4. The curated layer failed with authority, and retrieval amplified it</h3><p>Three facts in the curated docs were confidently wrong. The docs said unrecognized consumer errors default to retriable; the code retries only non-4xx errors. A routing table attributed two event types to the wrong consumer, while a sibling doc’s sequence diagram had it right, so the bundle also disagreed with itself.</p><p>Git dating made it worse: all three facts had been settled in code for over two years before the docs were written. This wasn’t staleness decay. The errors were born wrong, in an AI generation pass, into the layer agents are told to trust.</p><p>Then retrieval multiplied one of them. On the event-path question, BM25’s top-5 repeated the wrong consumer 26 times against 2 mentions of the correct one. A model reading that context has no signal for which is true. Grep, whose evidence is the code, was the only arm that contradicted the doc.</p><p>“Just curate your knowledge” is where this risk concentrates, not where it dies.</p><h3>5. Derived layers are caches without a TTL</h3><p>The knowledge graph missed a two-week-old feature entirely: honest failure on direct lookup, confident irrelevance on traversal. The curated docs had zero coverage of the same feature. Both are derived artifacts, only as good as their last regeneration, and neither signals that it’s behind.</p><p>To say it precisely: the working tree can also mislead (it lags production, branches diverge, generated code and feature flags complicate everything). The claim the data supports is narrower and still useful: the tree is the freshest artifact available to a local agent, and grep is the only arm in this benchmark that reads it directly. Every other layer adds a regeneration delay on top of whatever staleness the tree already has.</p><h3>Failure modes matter more than success modes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MakqFE7I3-GUmoNVKfcwQ.png" /></figure><p>One nuance from repo 2: trap outcomes belong to the corpus, not the retriever. BM25 failed the trap in repo 1 but passed it in repo 2, purely because a code comment in repo 2 named the service that owns the answer. Abstention has to be designed; it doesn’t emerge.</p><p><em>On-call at 3am, the difference between “no results” and “a wrong answer that looks right” is the incident.</em></p><h3>What I’d actually do</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFtnDetXGiTuBKDKcJjvTQ.png" /></figure><ol><li>Ship curated docs, with an index. They won every architecture question here, and an index line in your agent-instructions file makes their <em>absence</em> checkable, which is how the docs arm passed the trap questions. This is the part of OKF worth adopting now, and the format is fine.</li><li>Let agents grep. Don’t wrap the codebase in a retrieval layer that hides the ground truth.</li><li>Treat every curated fact as an assertion to verify, not a fact to trust. The doc errors here were only findable because an agent could check claims against code. If you generate docs with AI (at scale, you will), budget for adversarial verification against source, not just generation.</li><li>If you RAG a repo, curate first. In this benchmark the corpus was the product; the retriever was plumbing.</li><li>Demand abstention. Two of the four arms cannot say “I don’t know.” Fix that before trusting either with anything operational.</li></ol><h3>Limitations</h3><p>Twelve questions, two repos, one scorer (me, against code-verified fact checklists). The RAG arm is lexical BM25, not embeddings-plus-reranker, so its absolute rows are a floor; the ablation and amplification findings are corpus properties and survive a better retriever. The docs arm was charged full-doc reads per question, which understates docs on repeated use. The curated docs being AI-generated is both a limitation and, I’d argue, the realistic condition being debated. A skeptical reader is right to want this run on a hundred repos; consider this an existence proof and a method you can replicate on yours in an afternoon.</p><h3>Coda</h3><p>OKF is a reasonable format for the layer that won every architecture question here. RAG is reasonable plumbing when the corpus outgrows a context window. But on these two codebases, the interesting questions weren’t “OKF or RAG.” They were: does your agent read ground truth or derived truth, how does each layer fail when it can’t answer, and who verifies the curated layer against the code it describes?</p><p><em>On my codebases, the boring answer, grep plus curated docs plus verification, beat both sides of the argument. Run it on yours before believing either of us.</em></p><h3>Appendix: questions and parameters</h3><p>Code-detail: (1) What does the catalog upsert method do, and how is the custom-item flag handled when a parent record is deleted? (2) How does the consumer wrapper decide whether a failed message is retried, dead-lettered, or dropped? (3) Which feature flag enables the delayed-retry consumer, and is it evaluated via the flag service or an environment variable? (4) How does the workflow HTTP node prevent SSRF? (5) How are workflow task definitions registered and discovered by the engine?</p><p>Architecture: (6) Where does the service publish message-bus events during processing, and what HTTP calls does it make to other services? (7) Which deployment units run consumers, and which queue does each consume? (8) What is the buffer/delay/scoring queue pattern for, and what are the TTLs? (9) How does the workflow service split execution between its engine and Temporal?</p><p>Cross-repo traps: (10) What triggers the assessment-reset routine and what does it do? (answer lives in a sibling service) (11) When a sibling service publishes a control-assessment event, which consumer handles it and what happens downstream? (12) How does the admin service encrypt credential configs at rest? (answer lives in the admin service)</p><p>BM25 parameters: 50-line chunks, 10-line overlap; tokenizer lowercases and splits camelCase/snake_case with subtokens; Okapi BM25 k1=1.5, b=0.75; top-5 scored (top-10 checked for the misses); corpus = tracked text files, dot-directories and build output excluded. Scoring: one point per key fact present in (or directly pointed to by) the ingested context, against a per-question checklist verified in source. Token counting: <em>tiktoken cl100k_base</em> over all ingested context, failures included.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a426386651e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My ET Money subscription lapsed, so I built the thing myself]]></title>
            <link>https://medium.com/@satvik19nitm/my-et-money-subscription-lapsed-so-i-built-the-thing-myself-af0be8d6e98e?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0be8d6e98e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 08:50:03 GMT</pubDate>
            <atom:updated>2026-07-07T08:50:03.550Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUTn2kwnW07U9pk8EIJ1nQ.png" /></figure><p>My ET Money Genius subscription expired earlier this year. I stared at the renewal screen for a minute and realized I couldn’t articulate what I was actually paying for. Not “portfolio insights”, specifically. What data goes in, what math runs on it, and why does it say BUY?</p><p>I couldn’t answer that. So instead of renewing, I built it. This post is about what I built, how it works, and, more importantly, what I learned about what these products actually do under the hood. I’ll be blunt about the parts that don’t flatter me.</p><p>What a subscription like that actually sells you</p><p>Once I broke it down, the product was three things that is useful to me:</p><p>1.<strong> Aggregation</strong>. All your mutual funds and stocks in one place, across platforms.<br>2. <strong>A score.</strong> Some composite number that turns twenty metrics into one feeling.<br>3. <strong>Discipline. </strong>Nudges, rebalancing prompts, “your portfolio is unhealthy” alerts.</p><p>None of these require proprietary data. Fund NAVs are public (AMFI). Stock prices are public. Your own consolidated holdings are available as a CAS PDF from CAMS/KFintech, free. FII/DII flows, Put-Call Ratio, India VIX: all public. The subscription sells packaging and convenience, which is a legitimate business. I just wanted to see inside the package.</p><p>What I built</p><p><strong>InvestIQ</strong>: a self-hosted Flask app, about 11,500 lines of Python and 6,000 lines of React, running in Docker on my machine. First working version took about a week of nights. I’ve been iterating since.</p><p>The pipeline, end to end:</p><p><strong>Data layer.</strong> <em>yfinance</em> for prices and fundamentals (<em>with curl_cffi because Yahoo rate-limits naive clients aggressively</em>), <em>mftool for AMFI NAVs, RSS scraping of MoneyControl, ET, and Yahoo Finance for news</em>. <em>Upstox API v2</em> when connected, for real-time LTP instead of delayed quotes, <em>plus demat holdings sync</em>. Everything persists to SQLite in WAL mode. No Postgres, no Redis, no Kafka. For one user, SQLite is embarrassingly sufficient.</p><p><strong>Portfolio import</strong>. This was the single hardest feature, and it’s the least glamorous. Parsing a password-protected CAS PDF (casparser) so every mutual fund across every platform lands in one upload, plus auto-detecting broker <em>CSV formats from Zerodha, Groww, Upstox, and INDmoney</em>. Import shows a preview diff and reconciles before anything saves. Financial data ingestion is 10% API calls and 90% cleaning other people’s export formats.</p><p><strong>Scoring</strong>. A 17-factor composite: <em>RSI, MACD, Bollinger position, Stochastic, ADX, volume vs 20-day average, P/E vs sector median, P/B, ROE and margins, growth, debt health, analyst target gap, multi-timeframe momentum, volatility penalty, 52-week range position, Sharpe, and current macro sentiment</em>. Each factor votes bullish, neutral, or bearish; the weighted sum becomes a 0–100 score. This is the “AI score” every fintech app shows you. It is not magic. It is a spreadsheet with opinions.</p><p><strong>ML.</strong> An <em>XGBoost + RandomForest ensemble predicts 30-day direction</em>, with <em>1,000-path Monte Carlo simulation</em> for the probability bands. A separate classifier gives UP probability at 5, 10, and 20-day horizons, labeled with the triple-barrier method (profit target, stop, time limit) instead of naive future-price comparison, and validated walk-forward, meaning tested only on data the model never saw.</p><p><strong>Sizing and risk</strong>. <em>Half-Kelly position sizing</em> from measured win rate and payoff ratio. A macro regime filter that vetoes all new buys when India VIX crosses 25 or Nifty is in a confirmed downtrend. Swing signals come with ATR-based stops and a minimum 2:1 reward-to-risk, or they don’t fire.</p><p><strong>Execution</strong>. This is the riskiest and still in testing feature. An auto-invest bot that runs every signal through six validation layers (score threshold, positive Kelly edge, daily trade cap, drawdown circuit breaker, macro veto, market hours) before placing a LIMIT order via Upstox, then attaches an exchange-level GTT with trailing stop after the fill confirms. Paper mode simulates all of it with slippage before a single real rupee moves. There is a kill switch. I have used it.</p><p>The brutally honest part</p><p><strong>The ML barely beats a coin flip</strong>. Walk-forward accuracy on direction prediction sits at 55–65%. That’s the honest number, and anyone selling you a stock predictor claiming better on unseen data is measuring wrong or lying. The uncomfortable lesson: prediction is nearly worthless on its own. A 58% hit rate with disciplined sizing and 2:1 reward-to-risk makes money. A 58% hit rate with gut-feel sizing loses it. The edge lives in the boring parts.</p><p>Most backtests are fiction. My first backtest looked great until I added real Indian delivery costs: brokerage, STT, GST, and slippage. Costs quietly execute most retail strategies. The backtester now charges full freight, and the results are far humbler and far more trustworthy.</p><p>So the app grades itself. Every buy, sell, and swing suggestion is logged with the price at the moment of suggestion and re-evaluated daily against live prices. The measured hit rate is displayed in the UI and fed back to shrink the Kelly sizing conservatively. If the system is wrong, it must confront its own track record every morning. I built this feature because I caught myself remembering the wins and forgetting the losses, which is exactly the bias every “92% accuracy!” marketing claim exploits.</p><p>The economics don’t obviously work. The subscription cost a few thousand rupees a year. I spent weeks of nights building this. Priced at any reasonable hourly rate, this is the most expensive subscription replacement in history. If the only output were the software, this was a bad trade.</p><p>Why it was still worth it</p><p>Because the real product was the understanding. I now know exactly why a recommendation exists, which threshold fired, what the model’s actual accuracy is, and what costs will eat. I stopped treating market apps as oracles the day I built one. When my own app says BUY, I know precisely how little that means, and that calibration transfers to every other tool I use.</p><p>Also, honestly, it does things the subscription never did: broker execution with hard risk rails, self-graded suggestions, tax-loss harvesting checks, and a walk-forward backtest of the actual signal pipeline I run, not an idealized one.</p><p><em>Should you build one? If you want returns, no. Buy an index fund and go outside. If you want to never again be impressed by the phrase “AI-powered insights”, absolutely.</em></p><p><strong>(InvestIQ is a personal research tool, not investment advice. Nothing here is a recommendation.)</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0be8d6e98e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using POST for Search: A Practical Guide to HTTP QUERY (RFC 10008)]]></title>
            <link>https://medium.com/@satvik19nitm/stop-using-post-for-search-a-practical-guide-to-http-query-rfc-10008-728f5eedc295?source=rss-13497a26d3ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/728f5eedc295</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[http-methods]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[query]]></category>
            <dc:creator><![CDATA[Satvik Singh]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 16:06:22 GMT</pubDate>
            <atom:updated>2026-06-28T16:06:22.630Z</atom:updated>
            <content:encoded><![CDATA[<p>In June 2026, the IETF published <strong>RFC 10008, “The HTTP QUERY Method.”</strong> It is a Proposed Standard, and it closes out one of the longest-running sagas in the HTTP working group: a decade-plus effort that began life as WebDAV’s SEARCH method and finally landed as a clean, general-purpose verb.</p><p>If you build APIs, this is the first genuinely new, broadly applicable HTTP method since PATCH (RFC 5789, 2010). It deserves more than a changelog line. So let&#39;s go deep: the problem it solves, the exact semantics, how to implement it in Go today, and the parts that will bite you (caching and CORS, mostly).</p><h3>The trilemma every search endpoint hits</h3><p>Every engineer who has built a non-trivial search or query endpoint has stood at the same fork in the road, and every path is compromised.</p><p><strong>Option 1: </strong><strong>GET with a query string.</strong> Semantically perfect: GET is safe, idempotent, and cacheable. But you encode your entire query into the URI. That runs into hard length limits (proxies and servers routinely cap URLs around 4 to 8 KB), and expressing a structured query (nested filters, a JSON document, a SQL statement) as percent-encoded key/value pairs ranges from ugly to impossible.</p><p><strong>Option 2: </strong><strong>GET with a request body.</strong> Tempting. Just put the query document in the body of a GET. The problem: RFC 9110 explicitly says a body on GET has <strong>no defined semantics</strong>, and SHOULD be expected to fail. Intermediaries, caches, and servers are free to strip it, ignore it, or reject the request. It is not portable, and you cannot rely on it.</p><p><strong>Option 3: </strong><strong>POST.</strong> This is what almost everyone actually does. It works (the body holds the query, no length limit, any content type). But POST is, by definition, <strong>neither safe nor idempotent</strong>. The protocol must assume a POST might change state. The consequences are not cosmetic:</p><ul><li>Responses are effectively <strong>not cacheable</strong> by shared caches.</li><li>Intermediaries <strong>cannot transparently retry</strong> a POST after a connection drop, because they can&#39;t know it&#39;s safe to repeat.</li><li>A reader of your API can’t tell a read-only search from a mutation.</li></ul><p>Here is the whole problem in one table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYkl3fKdN9Bu_3dyEUooMA.png" /></figure><p>QUERY is the cell that was missing: a method with a request body that is <strong>both safe and idempotent</strong>.</p><h3>What QUERY actually is</h3><p>Per RFC 10008, a QUERY request asks the target resource to <strong>process the enclosed content in a safe and idempotent manner and return the result of that processing</strong>. The input lives in the request content, not the URI.</p><ul><li><strong>Safe.</strong> The client neither requests nor expects any change to resource state, the same contract as GET.</li><li><strong>Idempotent.</strong> It can be retried or restarted (after a connection failure, say) without concern for partial state changes.</li><li><strong>Content-Type is mandatory.</strong> Servers <strong>MUST</strong> fail the request if Content-Type is missing or inconsistent with the content. Crucially, <strong>content sniffing is forbidden</strong>: a server may not infer the media type from the body and proceed. No Content-Type, no query.</li></ul><p>That last rule matters. The body of a QUERY can be <em>anything</em> the resource understands (application/x-www-form-urlencoded, a JSON document, GraphQL, even application/sql), and the only thing telling the server how to parse it is the Content-Type. The media type is load-bearing.</p><p>Here is the canonical exchange from the RFC:</p><pre>QUERY /contacts HTTP/1.1<br>Host: example.org<br>Content-Type: application/x-www-form-urlencoded<br>Accept: application/json<br><br>select=surname,givenname,email&amp;limit=10&amp;match=&quot;email=*@example.*&quot;</pre><pre>HTTP/1.1 200 OK<br>Content-Type: application/json</pre><pre>select=surname,givenname,email&amp;limit=10&amp;match=&quot;email=*@example.*&quot;[<br>  { &quot;surname&quot;: &quot;Smith&quot;, &quot;givenname&quot;: &quot;John&quot;,  &quot;email&quot;: &quot;smith@example.org&quot; },<br>  { &quot;surname&quot;: &quot;Jones&quot;, &quot;givenname&quot;: &quot;Sally&quot;, &quot;email&quot;: &quot;sally.jones@example.com&quot; }<br>]</pre><p>Note the clean separation: the <em>query language</em> is negotiated by Content-Type (request) and Accept (response), independent of the verb itself.</p><h3>Doing it in Go today</h3><p>You don’t need to wait for library support. QUERY is just a method token, and Go&#39;s net/http treats methods as strings: it validates the token but does not restrict you to a fixed set. That means both the client and server work right now.</p><p>Start with a constant, since the standard library doesn’t yet define http.MethodQuery:</p><pre>const MethodQuery = &quot;QUERY&quot;</pre><p><strong>Server side.</strong> A handler that honors the spec’s hard rules (mandatory Content-Type, no sniffing, correct status codes):</p><pre>func handleContacts(w http.ResponseWriter, r *http.Request) {<br>	if r.Method != MethodQuery {<br>		w.Header().Set(&quot;Allow&quot;, &quot;GET, QUERY, HEAD, OPTIONS&quot;)<br>		http.Error(w, &quot;method not allowed&quot;, http.StatusMethodNotAllowed)<br>		return<br>	}<br>    // Content-Type is REQUIRED. Missing or malformed gives 400.<br>     ct := r.Header.Get(&quot;Content-Type&quot;)<br>     if ct == &quot;&quot; {<br>      http.Error(w, &quot;Content-Type is required&quot;, http.StatusBadRequest)<br>      return<br>     }<br>     mediaType, _, err := mime.ParseMediaType(ct)<br>     if err != nil {<br>      http.Error(w, &quot;malformed Content-Type&quot;, http.StatusBadRequest)<br>      return<br>     }<br>     // Unsupported query format gives 415, and we advertise what we DO accept.<br>     if mediaType != &quot;application/x-www-form-urlencoded&quot; {<br>      w.Header().Set(&quot;Accept-Query&quot;, &quot;application/x-www-form-urlencoded, application/sql&quot;)<br>      http.Error(w, &quot;unsupported query format&quot;, http.StatusUnsupportedMediaType)<br>      return<br>     }<br>     body, _ := io.ReadAll(r.Body)<br>     params, err := url.ParseQuery(string(body))<br>     if err != nil {<br>      // Media type was fine, but the query itself is invalid: 422.<br>      http.Error(w, &quot;unprocessable query&quot;, http.StatusUnprocessableEntity)<br>      return<br>     }<br>     results := runQuery(params) // your domain logic<br>     w.Header().Set(&quot;Content-Type&quot;, &quot;application/json&quot;)<br>     // Optional: expose this result set as a cacheable GET resource (see below).<br>     w.Header().Set(&quot;Content-Location&quot;, &quot;/contacts/stored-results/17&quot;)<br>     _ = json.NewEncoder(w).Encode(results)<br>}</pre><p>Register it with a plain path, or on Go 1.22+ with a method-aware pattern. The enhanced ServeMux accepts custom method tokens, not just the well-known ones:</p><pre>mux := http.NewServeMux()<br>mux.HandleFunc(&quot;/contacts&quot;, handleContacts)<br>// Go 1.22+ also allows:  mux.HandleFunc(&quot;QUERY /contacts&quot;, handleQuery)</pre><p><strong>Client side.</strong> http.NewRequest validates the method token, and &quot;QUERY&quot; is a valid token, so this just works:</p><pre>body := strings.NewReader(`select=surname,email&amp;limit=10&amp;match=&quot;email=*@example.*&quot;`)<br>req, _ := http.NewRequestWithContext(ctx, MethodQuery, &quot;https://example.org/contacts&quot;, body)<br>req.Header.Set(&quot;Content-Type&quot;, &quot;application/x-www-form-urlencoded&quot;)<br>req.Header.Set(&quot;Accept&quot;, &quot;application/json&quot;)<br><br>resp, err := http.DefaultClient.Do(req)<br>// Because QUERY is safe and idempotent, this request is safe to transparently retry on a transport error, unlike the POST you&#39;d otherwise have written here.</pre><p>The semantic upgrade is the whole point: that retry comment is a lie if the method is POST, and the truth if it&#39;s QUERY.</p><h3>Caching: the genuinely hard part</h3><p>This is where QUERY is more than &quot;a POST you&#39;re allowed to cache.&quot; Standard HTTP caching keys on method plus target URI (plus Vary). But two QUERY requests to the <em>same</em> URI can carry <em>different</em> bodies and deserve <em>different</em> cached responses. So RFC 10008 extends the cache key to <strong>incorporate the request content and its metadata</strong> (the Content-Type, content codings).</p><p>The immediate consequence: a cache must <strong>read the entire request body</strong> before it can compute a key. That’s a real cost, and it’s why QUERY caching is &quot;inherently more complex than caching a GET.&quot;</p><p>To avoid cache fragmentation over meaningless byte differences, the RFC permits a cache to <strong>normalize</strong> the content <em>for key generation only</em>:</p><ul><li>strip content codings (decompress before keying),</li><li>normalize per the media subtype suffix (a +json document can be canonicalized),</li><li>normalize per Content-Type semantics.</li></ul><p>The transformation affects <strong>only the cache key</strong>, never the request forwarded to the origin. Clients that need to suppress this can send the no-transform cache directive (advisory).</p><p>There’s a sharp edge here worth calling out in design reviews: if a cache normalizes the body <strong>differently</strong> from how the origin actually interprets it, you get a <strong>false-positive cache hit</strong> (two semantically different queries collapse to one key, and the wrong cached body is served). If your query language has whitespace- or ordering-sensitive semantics that a generic +json normalizer wouldn&#39;t know about, be conservative.</p><h3>The two-URI trick: Location vs Content-Location</h3><p>QUERY responses can hand the client back a URL so the <em>next</em> request avoids resending the body entirely. There are two distinct headers, and they mean different things:</p><pre>HTTP/1.1 200 OK<br>Content-Type: application/json<br>Content-Location: /contacts/stored-results/17<br>Location: /contacts/stored-queries/42<br>Last-Modified: Sat, 25 Aug 2012 23:34:45 GMT<br><br>[ ...results... ]</pre><ul><li><strong>Content-Location: /contacts/stored-results/17</strong> is a URI for <em>this specific result set</em>. A later GET on it refetches the same results, ideally via a cheap conditional request (If-None-Match or If-Modified-Since, yielding 304). There&#39;s no guarantee it persists; if it 404s, replay the original QUERY.</li><li><strong>Location: /contacts/stored-queries/42</strong> is a URI for <em>the query itself</em>. A GET on it <strong>re-runs</strong> the query against current data, so the result may differ from what you just got.</li></ul><p>One is “the answer I computed,” the other is “the question, saved.” Together they let a chatty search client degrade into ETag-gated conditional GETs.</p><p>The redirect codes carry their usual meanings, with one twist:</p><ul><li>301/308, 302/307: repeat the QUERY at the new Location.</li><li>303 See Other: the result is retrievable with a plain GET, so switch methods.</li><li><strong>Gotcha:</strong> the historical “rewrite POST as GET after a 301/302&quot; hack does <strong>not</strong> apply to QUERY. A redirected QUERY stays a QUERY.</li></ul><h3>Discovery: how a client learns a resource speaks QUERY</h3><p>Two mechanisms. OPTIONS lists the verb in Allow:</p><pre>OPTIONS /contacts HTTP/1.1            HTTP/1.1 200 OK<br>Host: example.org              -&gt;     Allow: GET, QUERY, HEAD, OPTIONS</pre><p>And the new <strong>Accept-Query</strong> response header advertises <em>which query formats</em> the resource accepts. It&#39;s an HTTP Structured Field (a list of media types), and it applies to every URI sharing that path (the query component is ignored):</p><pre>Accept-Query: application/x-www-form-urlencoded, application/sql</pre><p>It’s the QUERY analogue of Accept-Patch for PATCH. Return it on 415 responses too, so a client that guessed the wrong body format learns the right one.</p><h3>Status codes worth getting right</h3><ul><li><strong>400</strong>: missing or malformed Content-Type, or body inconsistent with it.</li><li><strong>415</strong>: media type not supported; pair it with Accept-Query.</li><li><strong>422</strong>: media type is fine, but the query is unprocessable (for example, valid application/sql that references a nonexistent table).</li><li><strong>406</strong>: you can’t produce the client’s requested Accept format.</li></ul><h3>Security and the adoption reality</h3><p>A few things to plan for before you ship QUERY to production:</p><ul><li><strong>Less leakage by design.</strong> Moving the query out of the URI and into the body means it’s far less likely to land in access logs, proxy logs, browser history, or bookmarks. That’s a real privacy win for sensitive filters.</li><li><strong>But mind temp-result URIs.</strong> If you mint a Content-Location resource to hold results of a sensitive query, the RFC says that URI <strong>SHOULD NOT</strong> embed sensitive portions of the request. Don&#39;t undo the privacy benefit by encoding the secret into the result URL.</li><li><strong>CORS preflight is mandatory.</strong> QUERY is <strong>not</strong> a CORS-safelisted method, so browsers will send an OPTIONS preflight before every cross-origin QUERY. Your CORS layer must answer it (Access-Control-Allow-Methods: QUERY).</li><li><strong>Intermediary support lags the RFC.</strong> The standard exists, but your CDN, reverse proxy, WAF, and API gateway may not key (or even pass) QUERY correctly yet. Until they do, the safe-retry and shared-cache benefits are partly theoretical. Test your actual edge.</li></ul><p>A pragmatic rollout: send QUERY, but keep a POST fallback for clients or intermediaries that choke on it, and use Accept-Query and OPTIONS to negotiate. The verb degrades gracefully because the <em>body format</em> is independent of the <em>method</em>.</p><h3>When to reach for it</h3><p>Use QUERY when you have a <strong>read-only</strong> operation whose input is <strong>too large or too structured for a URL</strong>, and you want the caching and safe-retry guarantees of GET without lying about it via POST. Search endpoints, GraphQL-style reads, reporting queries, complex filters: all the places you&#39;ve been reluctantly reaching for POST.</p><p>It won’t replace GET for simple parameterized fetches, and it won&#39;t replace POST for mutations. It fills the specific gap between them that&#39;s been empty since the beginning of HTTP. After fifteen years, the protocol finally has a verb that means <em>&quot;read me something complicated.&quot;</em></p><p><em>Sources:</em></p><ul><li><a href="https://www.rfc-editor.org/info/rfc10008/">RFC 10008, The HTTP QUERY Method (RFC Editor)</a></li><li><a href="https://datatracker.ietf.org/doc/draft-ietf-httpbis-safe-method-w-body/">draft-ietf-httpbis-safe-method-w-body (IETF Datatracker)</a></li><li><a href="https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-14.html">draft-ietf-httpbis-safe-method-w-body-14 (full text)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=728f5eedc295" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>