<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ilia Alakov on Medium]]></title>
        <description><![CDATA[Stories by Ilia Alakov on Medium]]></description>
        <link>https://medium.com/@alakov.ilia?source=rss-73cf176bb3f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z7DSyJV4yymPlhsJLrF5GA.jpeg</url>
            <title>Stories by Ilia Alakov on Medium</title>
            <link>https://medium.com/@alakov.ilia?source=rss-73cf176bb3f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:26:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alakov.ilia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[429 Isn’t About Speed. It’s About a Budget You Can’t See.]]></title>
            <link>https://medium.com/@alakov.ilia/429-isnt-about-speed-it-s-about-a-budget-you-can-t-see-150bd7e94ffd?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/150bd7e94ffd</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[telegram]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 14:01:03 GMT</pubDate>
            <atom:updated>2026-11-17T14:01:03.610Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JmKdC1yr3R1IDPusLjWVxQ.png" /></figure><p>In 2023 I wrote a Telegram publishing bot. The scheduler is still on GitHub, in <a href="https://github.com/ialakey/telegrammanager">telegrammanager</a>, and this is the whole of it:</p><pre>public void distribution() {<br>    List&lt;Note&gt; notes = dataManager.load(Note.class)<br>            .query(&quot;select n from telegrambot_Note n where n.status = :status&quot;)<br>            .parameter(&quot;status&quot;, StatusEnum.DELAYED.getId())<br>            .list();<br>    for (Note note : notes) {<br>        Date targetDate = note.getDateScheduled();<br>        Date currentDate = new Date();<br>        long delayInMillis = targetDate.getTime() - currentDate.getTime();<br>        if (delayInMillis &lt; 0) {<br>            telegramBotService.send(note.getId());<br>            delayInMillis = 0;<br>        }<br>    }<br>}</pre><p>Load everything due, send it in a loop. It worked. It published to my channels for years and I never thought about it again.</p><p>Then I spent a couple of years on a Telegram platform running at <strong>~300K monthly active users, ~75 RPS at peak and ~1.5 million events a day</strong> — and every single assumption in those fifteen lines turned out to be wrong.</p><p>The stack matters for the rest of this, so: a Laravel backend on MySQL, with Laravel’s own queue (jobs / failed_jobs) carrying the background work — generation, retries, the things that fail and get picked up again. <strong>Redis sat alongside it and carried a different job: the outbound send path and the rate-limit budget.</strong> Those are two queues with two different problems, and conflating them is the first thing to avoid. I wrote about the MySQL half of that system before, in <a href="#">the analytics piece</a> — the send path is the half that never becomes a row, and this is its story. Not <em>inefficient</em> — wrong. The loop is wrong. The absence of a delay is wrong. Even the idea that &quot;send it&quot; is an operation this process is allowed to perform on its own is wrong.</p><p>The sentence that reorganised all of it:</p><blockquote><strong><em>A rate limit is not a speed limit. It’s a budget, and you are not the only one spending it.</em></strong></blockquote><p>A speed limit is a property of <em>your</em> loop, and you satisfy it with a sleep. A budget is a property of the <em>token</em>, shared by every process holding that token, and you cannot satisfy it from inside any one of them. Almost every rate-limiting bug I have seen is that distinction collapsing.</p><p><em>Disclaimer: the platform is under NDA. Names, exact volumes and some details are changed. Orders of magnitude, configs and the incidents themselves are real. The 2023 code above is public and unchanged.</em></p><h3>What Telegram actually enforces</h3><p>This part is not a secret, it’s just rarely read. From Telegram’s own FAQ, a bot gets roughly:</p><ul><li><strong>~30 messages per second</strong> in total, across everything it does;</li><li><strong>~1 message per second into a single chat</strong>, with short bursts tolerated;</li><li><strong>~20 messages per minute</strong> into one group.</li></ul><p>Three limits with three different denominators — global, per-chat, per-group — which is the first thing that breaks the naive mental model. There is no single number to stay under. A bot sending 25 messages per second is comfortably inside the global limit and can still be hard-limited, because 24 of them went to the same chat.</p><p>And none of those numbers are the contract. The contract is this:</p><pre>{<br>  &quot;ok&quot;: false,<br>  &quot;error_code&quot;: 429,<br>  &quot;description&quot;: &quot;Too Many Requests: retry after 32&quot;,<br>  &quot;parameters&quot;: { &quot;retry_after&quot;: 32 }<br>}</pre><p>The documented figures are approximate by Telegram’s own wording — “will not allow <em>more than about</em>”. The 429 is exact. Which leads to the first mistake, and it is the most common one in every codebase I’ve looked at.</p><h3>Mistake 1: backing off instead of reading</h3><p>Here is the retry everyone writes, in every language, usually pulled from a blog post about HTTP clients:</p><pre>int attempt = 0;<br>while (attempt &lt; MAX_ATTEMPTS) {<br>    Response r = send(message);<br>    if (r.code() != 429) return r;<br>    Thread.sleep((long) Math.pow(2, attempt++) * 1000);  // 1s, 2s, 4s, 8s…<br>}</pre><p>Exponential backoff is the right default when you don’t know how long to wait. Here you <strong>do</strong> know. The server put the number in the response body. retry_after: 32 means thirty-two seconds, and sleeping 1, then 2, then 4, then 8 means you make four more requests inside a window where the server has already told you it will accept none of them.</p><p>That’s not merely wasted. On Telegram, requests made while you’re being limited extend the limit. The backoff turns a 32-second pause into a several-minute one, and the several-minute one into the thing that pages you at 3am. The loop that was supposed to protect the service is the thing degrading it.</p><pre>Response r = send(message);<br>if (r.code() == 429) {<br>    long wait = r.parameters().retryAfter();   // the server&#39;s number, not yours<br>    reschedule(message, now().plusSeconds(wait + 1));<br>    return;<br>}</pre><p>Note the second change, which matters more than the first: it does not sleep. It <strong>reschedules</strong>. I’ll come back to why in a moment, because that one is worth its own section.</p><h3>The caveat: the number is not always there, and not always true</h3><p>Telegram is well behaved. retry_after is present, it is accurate, and honouring it works. Most APIs are not Telegram, and I learned the rest of this against Steam&#39;s market API, where the numbers are in a public repo (caseforge) rather than behind an NDA:</p><ul><li>the limit is roughly <strong>20 requests per minute</strong>; past that, a 429 and a few minutes of ban;</li><li>so requests are <strong>serialised at a 3.5-second interval</strong>, search results cached for an hour and prices for half an hour;</li><li>a 429 <strong>puts the whole service to sleep for five minutes</strong>;</li><li>and <strong>negative responses are cached too</strong> — otherwise an item with no listings goes back to Steam on every single sync.</li></ul><p>That last one is the non-obvious one. An empty answer is an answer, and if you don’t cache it you spend your budget re-asking a question you already know the answer to.</p><p>Four more things that cost me time, none of which Telegram will teach you because Telegram does it properly:</p><p><strong>Retry-After is missing or lying more often than you&#39;d hope.</strong> Plenty of servers send a 429 with no header at all. Some send a constant value regardless of actual state. The spec allows an HTTP-date as well as delta-seconds, so you need both parsers, plus a jittered exponential fallback and a sane cap for when there&#39;s nothing to parse.</p><p><strong>Jitter is not a nicety.</strong> Without it, every worker that received the same retry_after wakes up in the same millisecond and reconstructs exactly the burst that got you limited. This is the single cheapest fix in the whole article and the one most often left out.</p><p><strong>Unfreeze on a ramp, not with a switch.</strong> If you freeze the queue during a block and then release it the moment a probe comes back 200, everything that piled up fires at once — and that burst looks identical to the one that got you blocked. The second block is usually longer, because plenty of services escalate. Resume at a fraction of the previous rate, increase gradually, and drop multiplicatively on the next 429. It costs a little throughput and stops the oscillation.</p><p><strong>The blocking key often isn’t the one you chose.</strong> Limits get applied per path prefix, per API key or per account at least as often as per host. Freezing a whole domain when only one endpoint is limited throws away throughput everywhere else; keying on the host when the limit is really per-token lets one token’s block stall all the others.</p><p>And the one that costs the most in practice, which is why it gets its own sentence:</p><blockquote><strong><em>A lot of rate limiting never comes back as a 429 at all.</em></strong><em> You get a 200 with a captcha page, or an empty result set, or a list quietly truncated to the first 20 items.</em></blockquote><p>Anything that only watches status codes sails straight past that and writes plausible garbage into your database, which you find out about weeks later. It is the same failure as <a href="#">a scraper that returns valid, plausible, wrong data</a> — and the defence is the same: validate the <em>shape</em> of the response, not just its status.</p><h3>Mistake 2: the limiter is in the wrong process</h3><p>This is the real one. Everything else in this article is downstream of it.</p><p>You fix the backoff. You add a proper token bucket — 30 tokens, refilled 30 per second, take one before each send. It’s correct, it’s tested, and in staging it is flawless. Then you run the sender as more than one process, and you are limited again, harder, with no code change and no traffic change.</p><p>Because each process has its own bucket. Three workers with a bucket of 30 each will attempt 90 messages per second against a limit of 30.</p><pre>what the code assumes              what Telegram sees<br>    ┌─────────────────────────┐        ┌─────────────────────────┐<br>    │  pod A · bucket(30/s)   │───┐    │                         │<br>    │  pod B · bucket(30/s)   │───┼───▶│   one token · 30/s      │<br>    │  pod C · bucket(30/s)   │───┘    │                         │<br>    └─────────────────────────┘        └─────────────────────────┘<br>       three independent limits            one shared budget</pre><p>The limit is a property of the <strong>bot token</strong>, and the token is not scoped to your process. It doesn’t care about your replica count, your autoscaler, or the one-off backfill script someone is running from a laptop against production — that script spends the same budget. If a backfill can starve your live traffic, and it can, then the live traffic and the backfill are in the same queue whether or not you modelled them that way.</p><p>So the limiter cannot live in the process. It has to live somewhere all the spenders can see — which is exactly why Redis was in that stack at all, sitting next to a perfectly good Laravel queue that could not do this job. A queue in your database orders work. It has no opinion about a budget owned by someone else’s API. The moment you need one, the thing you’re building stops being a rate limiter and starts being a queue with an admission gate on it.</p><p>This is the same shape as a database connection pool, incidentally — the reason maximumPoolSize is almost never the answer when a pool runs dry is that the pool is a local view of a budget owned by somewhere else. <a href="#">A connection pool is not the thing you think it is either.</a></p><h3>Mistake 3: INCR plus EXPIRE is not a rate limiter</h3><p>The Redis snippet that turns up in every tutorial:</p><pre>INCR  rate:{token}:{second}<br>EXPIRE rate:{token}:{second} 2</pre><p>Two problems, and the second is worse than the first.</p><p><strong>It isn’t atomic.</strong> Two commands, two round trips. A process can die between them and leave a key with no TTL, which stays at its count forever and permanently wedges that second — a self-inflicted outage that survives restarts. SET key 1 EX 2 NX followed by INCR has the same shape. Anything that needs to be atomic across more than one key or more than one command in Redis belongs in a Lua script, because Redis runs a script as one unit:</p><pre>-- KEYS[1] = bucket key<br>-- ARGV[1] = capacity   ARGV[2] = refill per second<br>-- ARGV[3] = now (ms)   ARGV[4] = tokens requested<br>local b       = redis.call(&#39;HMGET&#39;, KEYS[1], &#39;tokens&#39;, &#39;ts&#39;)<br>local tokens  = tonumber(b[1]) or tonumber(ARGV[1])<br>local ts      = tonumber(b[2]) or tonumber(ARGV[3])<br>local elapsed = math.max(0, tonumber(ARGV[3]) - ts) / 1000.0</pre><pre>tokens = math.min(tonumber(ARGV[1]), tokens + elapsed * tonumber(ARGV[2]))</pre><pre>if tokens &lt; tonumber(ARGV[4]) then<br>  -- how long until enough tokens exist, in ms; the caller schedules, it does not spin<br>  local deficit = tonumber(ARGV[4]) - tokens<br>  return {0, math.ceil(deficit / tonumber(ARGV[2]) * 1000)}<br>end</pre><pre>redis.call(&#39;HSET&#39;, KEYS[1], &#39;tokens&#39;, tokens - tonumber(ARGV[4]), &#39;ts&#39;, ARGV[3])<br>redis.call(&#39;PEXPIRE&#39;, KEYS[1], 60000)<br>return {1, 0}</pre><p><strong>It’s the wrong shape.</strong> A counter keyed by the current second is a <em>fixed window</em>, and a fixed window permits twice the limit across its boundary: 30 messages at 12:00:00.999 and 30 more at 12:00:01.001 are 60 messages in two milliseconds, and every one of them passes the check. Telegram will not be persuaded that you were technically compliant in both windows.</p><p>A token bucket doesn’t have boundaries. It has a level, and the level is continuous. That’s the entire difference, and it’s why the script above carries a timestamp instead of a window key.</p><h3>Mistake 4: the retry is a second producer</h3><p>This is the one that produced the worst incident, and it is invisible until you are in it.</p><p>A message gets a 429. The obvious thing is to put it back on the queue. So it goes back on the queue — and now it is competing with new messages for the same budget. The queue drains slower than it fills. More messages get 429s. They also go back on the queue. Each retry consumes a slot that would have carried a <em>new</em> message, so the new messages are delayed, so they are retried too.</p><p>Nothing has crashed. Every component is behaving exactly as designed. Throughput is collapsing anyway, and it will not recover on its own, because the system’s own retries are now the majority of its load. It clears when you stop accepting new work, not by healing.</p><p>The fix is the one change from the snippet earlier: <strong>a 429 is not a failure, it’s a schedule.</strong></p><pre>ZADD  send:{chat_id}  &lt;now + retry_after + 1&gt;  &lt;message_id&gt;</pre><p>A sorted set keyed by delivery time. A retried message isn’t at the front of the queue being rejected over and over; it isn’t in the queue at all until the moment it can actually be sent. The retry stops being load. This is the difference between a retry policy and a scheduler, and under a rate limit you want the scheduler.</p><p>While you’re there: cap the number of reschedules, and make the cap a business decision rather than a constant. A notification that is four hours late is not a late notification, it’s a wrong one. Dropping it is the correct behaviour and somebody other than you should choose that number.</p><h3>Mistake 5: one queue, so one slow chat stops everything</h3><p>With a single global queue, ordering is first-in-first-out, and the head of the queue is whatever arrived first. If the message at the head is going to a chat that is currently rate-limited, everything behind it waits — including messages to every other chat, all of which are perfectly free.</p><p>Classic head-of-line blocking, and the reason it’s easy to miss is that your metrics look fine. Global throughput is at the limit. The budget is fully spent. It’s being spent on one chat.</p><p>Per-chat lanes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSDl6FhcNZGLYq0RGE-m6w.png" /></figure><pre>send:ready            → a set of chat_ids that have something to send<br>send:{chat_id}        → that chat&#39;s own FIFO<br>bucket:{token}        → the one global budget</pre><p>A worker takes a chat id from send:ready, takes one token from the global bucket, sends the single message at the head of that chat&#39;s list, and puts the chat back at the end of send:ready if it still has work. Per-chat order is preserved — which is not optional, a bot that answers question two before question one is a broken bot — and no chat can hold the global budget hostage.</p><p>The per-chat limit (~1/second) then lives on the lane, not on the worker, as a readyAt timestamp on the chat. A chat that sent a message a moment ago simply isn&#39;t in send:ready yet.</p><h3>The thing you can’t engineer around: dead chats cost the same</h3><p>Some fraction of your audience has blocked the bot. That fraction only ever grows. Telegram tells you, in the clearest possible terms:</p><pre>{ &quot;ok&quot;: false, &quot;error_code&quot;: 403,<br>  &quot;description&quot;: &quot;Forbidden: bot was blocked by the user&quot; }</pre><p>A 403 costs exactly as much budget as a delivered message. It is one of your ~30 per second. For a broadcast to a large audience, the dead chats are not a rounding error — they are a straight percentage off the top of your capacity, taken before any real user is reached, every single time you broadcast.</p><p>So a 403 has to write back. Mark the chat inactive on the first one and stop sending; reactivate when the user next messages the bot, because people unblock. The ugly version of this bug is the one where nobody prunes and the broadcast gets slower every month for reasons that look like infrastructure. It isn’t infrastructure. It’s the audience leaving, rendered as latency.</p><p>This is the same failure mode as <a href="#">a scraper that quietly returns nothing</a>: the system reports success while the thing you actually cared about has stopped being true.</p><h3>What to put on the dashboard</h3><p>The metric everyone builds first is messages per second. It is nearly useless. It tells you what you spent, not what you were allowed to spend, and it looks identical whether you’re at 60% of budget on a quiet afternoon or pinned at 100% and shedding.</p><p>Four that earn their space:</p><p><strong>Budget utilisation</strong> — tokens taken over tokens available, per token, per minute. This is the only number that says how close to the wall you are. Everything else is a consequence of it.</p><p><strong>429 rate, split by which limit was hit.</strong> Global 429s and per-chat 429s have completely different causes and completely different fixes, and averaging them together hides both. Global means you’re over capacity. Per-chat means one conversation is too chatty, which is usually a bug in the conversation logic, not in the sender.</p><p><strong>Scheduled-vs-actual delivery delay, at p99.</strong> The mean is a lie here. Under a rate limit, almost everything is instant and a small tail is minutes late, and that tail is the entire user experience of the feature. If p99 is drifting up while throughput is flat, you are saturated and shedding into the future.</p><p><strong>Queue depth per chat, at the maximum.</strong> Not the total — the max. One chat with ten thousand queued messages is a runaway loop somewhere, and it is completely invisible in an aggregate that also contains hundreds of thousands of healthy chats with one message each.</p><h3>What I’d tell 2023 me</h3><p>That fifteen-line scheduler was not bad code. It was correct code with an unstated assumption: that this process is the only thing spending the budget. For one channel and one bot, that assumption held for years.</p><p>Everything expensive I learned afterwards was the same assumption failing at a larger scale. The pod count broke it. The backfill script broke it. The retry loop broke it by becoming a producer. The blocked users broke it by spending budget without being anyone.</p><p>The general form, well past Telegram:</p><blockquote><em>If a limit is enforced on a resource you share, the enforcement has to live where the sharing is visible. Any limiter inside a process is a limiter on that process’s opinion of the world.</em></blockquote><p>Which is why the fixed version isn’t a smarter loop with a better sleep. It&#39;s a queue, a shared bucket, and a scheduler — three things the original didn&#39;t have and didn&#39;t need, right up until it did.</p><p>I’ve argued here that a 429 should reschedule rather than retry, and that this is a scheduler problem rather than a retry-policy problem. There is a real counter-argument: for low-volume bots the scheduler is strictly more moving parts than a sleep, and more parts is more to get wrong.</p><p>So where’s the line? At what volume does a shared bucket stop being over-engineering and start being the only thing that works — and did you find that line the easy way or the way I did?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=150bd7e94ffd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Commission Is a Setting. Fairness Isn’t.]]></title>
            <link>https://medium.com/@alakov.ilia/the-commission-is-a-setting-fairness-isnt-1349e0200439?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1349e0200439</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 14:01:04 GMT</pubDate>
            <atom:updated>2026-11-10T14:01:04.028Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6gBS93YDKURfv8OYH5Q6A.png" /></figure><p>A while back I wrote a CS2 case site end to end and put it under MIT, mostly to find out where these things actually cheat. The answer was that they don’t have to: a verifiable roll and a 15% house edge coexist perfectly, because the lever is the ticket table, not the roll. That argument is <a href="https://medium.com/p/5bf3a252876b">a separate piece</a> and it stands on its own.</p><p>This one is what happened next. The project got four more mechanics — contracts, a daily bonus wheel with vouchers, promo codes, and a real settings layer for the operator — and each of them turned out to be the same question wearing a different hat.</p><p><strong>Where does this number come from?</strong></p><p>Not “what should it be”. Where does it <em>come from</em>. Some numbers a solver computes from a target. Some a human types, because there is nothing to solve for. And a few are not numbers at all in the sense that matters — they are the terms of a promise the site has already made, and nobody, including the person who owns the database, gets to change their mind about them later.</p><p>Sorting every constant in the project into those three piles was more useful than any single thing I built. Repository: <a href="https://github.com/ialakey/caseforge"><strong>github.com/ialakey/caseforge</strong></a> · MIT.</p><p><em>Disclaimer, to get it out of the way. Buying a case for real money is regulated as gambling in most jurisdictions: licensing, KYC, age verification, country restrictions. There are </em><strong><em>no payments</em></strong><em> in this project — the balance top-up is a stub, disabled in production. This is a teaching codebase about architecture. Running it for real money without the legal half is not a grey area, and an MIT licence does not cover you.</em></p><h3>Part I — the numbers a solver computes</h3><p>Two of these existed already. A case is balanced to a target RTP with a power law, w_i = price_i^(-k), and k is found by bisection because expected value decreases monotonically in it. An upgrade doesn&#39;t get balanced at all: chance = stake / target × 0.9 makes the expected value equal to stake × RTP at any multiplier, which is one line of algebra and one test. Both are in the previous article.</p><p>Contracts are the third, and they are where the pattern became a principle instead of a habit.</p><h3>Contracts: a payout table nobody wrote</h3><p>A contract takes between 3 and 10 items and returns exactly one. What separates it from an upgrade is that there is no losing branch at all — a contract always pays out, the only question is with what. The reward comes from the catalogue inside a corridor from 0.1× to 5× the value you put in, and the outcome is decided by the same roll from the same seed pair as everything else.</p><p>The temptation is identical to the one upgrades present: just write the weights by hand. And it is worse here, because a hand-written table quietly creates a <em>second economy</em> running next to the cases on its own mathematics. Every catalogue re-pricing then moves your margin somewhere you can’t predict, and you find out about it from a dashboard weeks later.</p><p>So the weights get solved. The target is simply that the expected reward equals stakeValue × 0.9 — the same RTP the cases and upgrades run on.</p><p>The model is a different one from the cases. There it was a power law over price; here it is an exponential tilt:</p><pre>w_i ∝ exp(-alpha × u_i)</pre><p>where u_i is the outcome&#39;s price on a logarithmic scale, normalised to [0, 1] across the whole pool.</p><p>That normalisation isn’t cosmetic. On raw log-prices the same alpha would mean different things for a pool spanning a factor of 10 and a pool spanning a factor of 10,000, and the search bounds would stop being a constant — they would have to be derived per pool, which is exactly the kind of thing you forget to do for the one pool that matters.</p><p>alpha is found by bisection, and it converges for a pleasant reason: expected value is monotonically decreasing in alpha because the derivative is minus the variance of the tilted distribution. That isn&#39;t something I had to verify numerically. It just falls out when you write it down.</p><pre>let low = -ALPHA_BOUND;<br>let high = ALPHA_BOUND;<br>for (let i = 0; i &lt; 80; i++) {<br>  const mid = (low + high) / 2;<br>  if (tiltedExpectedValue(prices, logs, span, mid) &gt; target) low = mid;<br>  else high = mid;<br>}</pre><p>Eighty halvings is enough to exhaust the mantissa. And one small thing that will hand you a NaN if you skip it: shift the exponents by their maximum before exponentiating. The absolute values mean nothing on their own — only the differences do — and the shift keeps exp() away from overflow at the ends of the range.</p><p>Here is the part I like more than the formula.</p><p><strong>In contracts the margin is a constant of the system, not a property of the catalogue.</strong> Whatever items happen to fall into the price corridor, the solver drags the expected value onto the same number. Compare that with cases, where the RTP wanders off on its own the moment a knife appreciates, and an hourly job has to chase it. Same target, same 0.9, completely different maintenance burden — and the difference is entirely in whether the number is solved at the moment of use or baked in at build time.</p><p>Turning the solved weights into whole tickets is done with the largest-remainder method. Rounding each weight independently leaves the sum a few tickets short, and a gap in ticket space is a roll with no outcome attached to it — a crash waiting for the specific player who hits it. An outcome that would get zero tickets is dropped entirely: you can’t win it, and drawing it on the reel is a lie.</p><p>And the last piece, which took me longest to see as necessary. <strong>The solved table is written into the contract row as JSON</strong> — item id, price and ticket range for every outcome. It cannot be reconstructed from the catalogue afterwards, because the prices have moved since. Without that snapshot the roll stays perfectly reproducible and stops meaning anything: you would recompute the HMAC, get the same ticket number, and have no way to say what that ticket corresponded to at the time.</p><p>Reproducible and verifiable are not the same property. That distinction cost me an afternoon.</p><h3>Part II — the numbers a human types</h3><h3>The wheel, where the solver has nothing to minimise</h3><p>I added the daily bonus last and budgeted half an hour for it. The roll machinery was already there; this was supposed to be sectors on top of it. The half hour covered the code. Working out why the approach that had carried the whole project didn’t apply here took the rest of the evening.</p><p>Everything before this had the same shape: a target expected value, and a solver that fits weights to it. Cases, a power law. Contracts, an exponential tilt. Both times the goal was one number.</p><p>The wheel’s prizes are not commensurable. Ten to the balance. A 25% discount on the next opening. A free case opening up to 300. A skin worth up to 200. There is no single number all of that reduces to. The discount in particular costs nothing at all until the player goes and opens something, and then costs exactly as much as they end up spending. There is nothing here for a solver to minimise.</p><p>So the table stays hand-written, and what gets constrained is not the return — it is the <strong>cost</strong>:</p><pre>export function wheelGrantCost(wheel = WHEEL): number {<br>  return wheel.reduce(<br>    (sum, slice) =&gt; (slice.kind === BonusKind.DISCOUNT ? sum : sum + slice.chance * slice.value),<br>    0,<br>  );<br>}</pre><p>Discounts are excluded from that sum deliberately. They cost a fraction of what the player spends afterwards, which makes them part of the <em>cases’</em> margin, not part of the wheel’s budget. Charging them to the wheel would double-count them and make the number meaningless in both places.</p><p>Everything else is counted honestly, and the current table comes to 46.50 per spin. The test fails if a re-tuning pushes that past 60. That guard isn’t about elegance. It is so that in six months somebody — me — doesn’t raise “skin up to 200” to “skin up to 5000” without noticing what it costs.</p><p>One more thing, and it is the only part of the wheel that is visual rather than arithmetic. <strong>The sectors are drawn to the exact ticket ranges they are rolled from.</strong> A prize with a 3% chance takes up 3% of the rim. On screen that means “skin up to 200” is a sliver and “10 to the balance” is nearly a third of the circle, which looks far less exciting than eight equal wedges. Eight equal wedges over unequal odds is the oldest lie in the genre, and I wasn’t going to reproduce it just because the SVG maths is easier.</p><h3>A cooldown you can’t check</h3><p>The day is a rolling 24 hours, not a calendar one. A calendar reset at midnight hands everybody in a convenient timezone two spins a few hours apart, and turns midnight into a load spike nobody asked for.</p><p>Then the part where I walked into a wall I had already walked into with balances. The first version read lastBonusAt, compared it with now, and wrote the new value if a day had passed. Two simultaneous requests both pass that check between the read and the write.</p><pre>const claimed = await tx.user.updateMany({<br>  where: { id: userId, OR: [{ lastBonusAt: null }, { lastBonusAt: { lte: cutoff } }] },<br>  data: { lastBonusAt: new Date() },<br>});<br>if (claimed.count === 0) throw badRequest(ErrorCode.BONUS_ON_COOLDOWN, &#39;...&#39;);</pre><p>Zero rows affected means somebody already took the spin. Same move as the conditional balance debit — the check and the claim are one statement, so there is no gap to race in. And the lastBonusAt column is denormalised precisely for this: deriving the last spin from the bonus table would be tidier, and you cannot capture it in a single statement.</p><p>The smoke suite fires five concurrent spin requests and asserts that exactly one is accepted.</p><h3>Vouchers, and the test that turned out to be wrong</h3><p>Cash and skins are credited immediately. A discount and a free opening are <strong>vouchers</strong>: they sit on the account until a case opening consumes them. They are consumed inside the opening’s own transaction, before the debit — releasing a voucher any earlier lets a parallel opening grab it in the gap and get the discount for nothing.</p><p>When several vouchers are open, something has to choose which one to burn. First in, first out feels neutral and is a bad rule: it will spend a 50% discount on the cheapest case in the catalogue while a free opening waits behind it. The player won’t see that happen and won’t understand where the reward went. So the choice is the voucher that saves the most <strong>on this particular basket</strong>.</p><p>And then that rule caught my own test.</p><p>The smoke suite granted an account a 50% discount, opened a case, and expected half the price to be debited. Zero was debited. My first thought was that the discount calculation had broken.</p><p>The test had broken. An unspent free opening was still sitting on the account from an earlier section, and the selection did exactly what it was told: a free opening on a 9.99 case saves the whole 9.99, where the 50% discount saves 4.99. The code was right; the test hadn’t isolated the voucher it was testing.</p><p>Two lines fixed it — the suite now clears foreign vouchers before its own assertion. But I also added a separate test for the selection rule itself: two open vouchers, one expensive and one cheap, asserting that the expensive one is spent and the cheap one is left alone. A rule that tripped its own author once is worth pinning down.</p><p>And since the server spends a voucher silently, the interface is obliged to say which one it spent. Otherwise the reward disappears from the list alongside a quietly different price, which is indistinguishable from a bug.</p><h3>The skin prize is picked by the same roll</h3><p>A “skin up to 200” sector has to become an actual item. A second, hidden draw at that point would be ironic: the entire wheel is built on outcomes derived from the seed pair, and then — “we’ll pick the item for you somehow.”</p><p>So the item comes from the same roll. Candidates are sorted by id — stable, unlike sorting by a price that drifts — and the index is the roll modulo the list length.</p><p>The case where the catalogue holds nothing under the cap needed deciding separately. Granting nothing is the one outcome a wheel must never have, so that sector pays out in cash instead, and the ledger row records why in its comment.</p><h3>And the wheel has to be visible</h3><p>You can build a perfect wheel and hide it on its own page, which a player visits once out of curiosity. A reward you have to go and find is a reward most people never collect.</p><p>So the spin status sits on the home page under the banner, and looks different depending on whether it is ready. Nothing is rendered at all when the bonus is switched off in settings or the visitor isn’t logged in — an empty placeholder promising something is worse than no placeholder.</p><h3>A promo code that doesn’t make anything cheaper</h3><p>I built promo codes knowing there are no payments yet and that one day there will be. That knowledge alone determined the shape.</p><p><strong>A code does not reduce the amount charged — it adds on top.</strong> Enter WELCOME10, top up 500, and 550 lands on the balance.</p><p>The difference looks cosmetic while there is a stub behind the button. It stops being cosmetic the day a payment provider appears, because the amount you capture has to match the amount you told the provider when you created the invoice. A code that alters the charged amount has to be threaded through the whole flow — invoice creation, webhook, reconciliation against the statement. A code that adds on top stays entirely on your side and never touches the provider at all.</p><p>The grant is a separate ledger row, not a larger deposit:</p><pre>DEPOSIT | 50000 | balanceAfter 2110093 | Top-up (stub, no real payment)<br>BONUS   |  5000 | balanceAfter 2115093 | Promo code WELCOME10</pre><p>What a player paid for and what they were given are different money. A report that merges them cannot answer the only question anyone ever asks about a campaign: what did it cost?</p><p>Codes are created by name, so an existing one is overwritten — and the redemption counter is <em>not</em> reset when that happens. Otherwise fixing a typo in a live code would silently lift its limit.</p><p>There are two limits and they are held by different mechanisms, because they race differently. The per-player limit is a count of redemption rows inside the top-up transaction, where there is no race. The global limit on the code is a conditional UPDATE on a counter: counting rows and then inserting your own means two simultaneous top-ups both see the last use as free and both take it. The redemption rows remain the source of truth; the counter exists only so the limit can be captured in one statement.</p><p>And one small thing that saves support a day: codes are deactivated, never deleted. Redemptions point at them, and a player asking where 50 on their balance came from six months ago deserves a row that still explains it.</p><h3>Part III — the numbers nobody types</h3><h3>Not everything belongs in the admin panel</h3><p>By the time there were three mechanics, the list of constants an operator would plausibly want to turn without a deploy had grown real: sell-back commission, the rate limit on openings, top-up bounds, the wheel cooldown, the wheel sectors themselves, maintenance mode.</p><p>The obvious route is a settings table you read from. I went halfway down it and came back, because every setting was costing three places: a field in the admin form, a validator on save, and a reader on the server. Three places kept in agreement by hand. That is precisely the construction that produces back-office settings which exist in the database and nowhere in the interface.</p><p>So a setting is declared once:</p><pre>&#39;economy.sellFeeBps&#39;: {<br>  group: &#39;economy&#39;,<br>  kind: &#39;int&#39;,<br>  schema: bounded(0, 5000),<br>  default: 0,<br>  label: &#39;Sell-back fee, basis points&#39;,<br>  hint: &#39;100 bps = 1%. Taken when a player sells an item back to the site.&#39;,<br>},</pre><p>The admin form is rendered from that registry, validation is taken from it, and the server reads by the same key. Adding a knob is one line in one file, and the panel grows the field by itself.</p><p>Two details that only showed up in use. Values are cached for fifteen seconds — they are read on nearly every request, and a database round trip per read for data that changes twice a month is pure waste. A write refreshes the writing instance immediately, so the operator sees the effect while still looking at the panel.</p><p>And the setting parser throws no exceptions at all. A row hand-edited into an invalid state in the database must not be able to take the site down: the value falls back to its default and the incident is logged. Validation lives at the entrance, in the admin panel, where a bad value can be rejected loudly and to the face of the person entering it.</p><h3>And these you don’t get to configure</h3><p>Which brought me to the question that turned out to be more interesting than the implementation: what do you refuse to hand the operator at all?</p><p>TICKET_SPACE. The seed algorithm. The shape of the roll.</p><p>Those are not tuning parameters. They are the terms of a promise. Every past opening was published against them, and they are exactly what the player recomputes when they check a drop. An operator with a “ticket space size” field can, with one save, make yesterday’s drops stop reconciling. Not even out of malice — curiosity is enough.</p><p>That isn’t a setting. It is a way to break the only claim the site actually makes.</p><p>The line I ended up with is simple, and I like it: <strong>you make configurable the things the site is entitled to change its mind about.</strong> The commission — yes. The rate limit on openings — yes. Fairness — no.</p><p>A case’s ticket ranges belong to the same family, and they are edited where they belong: in the builder, behind validation that refuses to save a case with a hole in its coverage or an RTP above one hundred percent. Not a field in a settings panel, but a tool that knows the rules of its own domain.</p><h3>Three bugs from the same family</h3><h3>The inventory row you never delete</h3><p>An item on an account is a row with a status, and the row is never removed:</p><pre>AVAILABLE → SOLD | WITHDRAWN | LOCKED | UPGRADED | CONTRACTED</pre><p>Actions are available only from AVAILABLE. LOCKED belongs to a withdrawal in flight; the other three are terminal and become the history tab.</p><p>Deleting the row would be simpler and is wrong from two directions at once. A player who sold a knife still wants to see what they got for it. And support cannot work a complaint about an item that vanished from the interface at the moment of the debit, because there is nothing left in the database to work from.</p><p>Every transition is the same conditional updateMany on status = AVAILABLE as in upgrades. It is the check and the claim in one: an item that has meanwhile gone to sale, withdrawal or a contract simply doesn&#39;t land in the affected rows. The call fails on a mismatch between rows affected and ids requested, and reading the row first isn&#39;t needed anywhere.</p><p>Two things that look like details and aren’t. <strong>“Sell everything” resolves its own set inside the transaction from the same filter the interface was showing</strong>, not from a list of ids sent by the browser. A list of ids would limit the operation to whichever page happened to be open and would lose any race against a change in between. A filter loses neither. The total comes back from the server for the same reason — a confirmation dialog can only ever quote an estimate.</p><p>And the price-band boundaries are defined in the base currency, not the displayed one. A band that floats with the exchange rate would re-sort the inventory on every rate refresh: an item would sit in one band on Monday and another on Tuesday without changing price at all. The labels still go through the ordinary money formatter, so the boundaries are round numbers in the base currency and converted ones elsewhere.</p><h3>One refresh for all of them</h3><p>A pair of JWTs: a short access token and a long refresh token. The access token carries id and role and is verified without touching the database, which is exactly why it has to expire quickly — otherwise a ban or a role change doesn’t take effect. The refresh token lives in an httpOnly cookie for a month.</p><p>The split only works if something actually spends the refresh token. The browser does it silently: a 401 from any call triggers one refresh and one retry of the original request, and the session ends on a failed refresh, not on a failed call.</p><p>The word doing all the work there is <em>one</em>. Loading a page fires several requests at once — profile, inventory, seeds, opening history. When the access token has expired they all get a 401 simultaneously, and if each starts its own refresh, the cookie rotates several times in a row. The losers of that race retry with a token that has already been replaced. From the player’s side this is indistinguishable from being randomly logged out, and it only reproduces after fifteen minutes of idling — which is to say, when you have stopped watching.</p><pre>let refreshing: Promise&lt;string | null&gt; | null = null;</pre><pre>async function refreshAccessToken(): Promise&lt;string | null&gt; {<br>  refreshing ??= (async () =&gt; { /* ... */ })();<br>  return refreshing;<br>}</pre><p>The ??= is the entire fix: parallel 401s share one promise.</p><p>The websocket is separate. It reads the token only at handshake, so it takes it as a callback and reconnects when it changes. A token captured at page load would break every reconnection after the first expiry.</p><p>While on contracts between the frontend and the API — one more of the same kind. The server returns a machine-readable code next to an English message; the interface translates the code and falls back to the message for codes this build doesn&#39;t know yet. An error added on the backend stays readable before its translation ships.</p><h3>The best bug in the project: a reel in a background tab</h3><p>The reel decelerates under the marker the way it does in-game, and it decides nothing — the server returned the outcome with the roll, and the animation only plays it back. All of which I already knew. Here is the one I caught recently and still enjoy.</p><p><strong>A background tab does not run animation frames. Its timers keep running.</strong></p><p>The reel’s landing was a timer. Perfectly reasonable: the duration is known in advance.</p><p>So: you open a case, switch to another tab, come back. The reel is sitting in its starting position, because requestAnimationFrame never fired and it never moved. But the timer went off on schedule, so the reel is flagged as stopped and highlighted, and the marker is pointing at nothing in particular. The item, meanwhile, was credited correctly — the server knows nothing about animations.</p><p>The fix has two halves. First, a fallback timer at the start: whichever gets there first moves the reel, and the other one does nothing.</p><pre>let started = false;<br>const start = (): void =&gt; {<br>  if (started) return;<br>  started = true;<br>  setAnimating(true);<br>  setOffset(target);<br>};</pre><pre>innerRaf = requestAnimationFrame(start);<br>const startFallback = setTimeout(start, START_FALLBACK_MS);</pre><p>The 250 ms delay is chosen so that on a visible tab this timer always arrives second and quietly retires.</p><p>Second, don’t trust the transition to carry the reel to its destination. A background tab stalls a transition even after the transform has been set, and the reel can sit at its starting offset indefinitely. So the landing — which is a timer and therefore always runs — also places the reel where the roll says it belongs. On screen this changes nothing: by then the easing has flattened out and the reel is already there.</p><p>The note I wrote down for myself: <strong>if an animation has both a timer and frames, work out separately what happens when the frames stop.</strong></p><h3>What the tests caught, including themselves</h3><p>Unit tests cover what is meaningful in isolation: the provably-fair chain, ticket ranges, balancing, upgrade odds, parsing of market and profile responses.</p><p>The smoke suites against a live API earn more. There are five of them now — general, contracts, inventory, bonuses and admin — and they check things unit tests cannot catch in principle: that balance debits are atomic, that the active server seed never leaks into a response, that an opening still recomputes after a seed rotation, that the balance reconciles with the ledger, that roles genuinely close off the admin panel, and that the server <strong>refuses</strong> to save an unprofitable case or one with a hole in its ticket ranges. A test proving the system can say no is worth five happy-path tests.</p><p>And splitting them into five suites produced an error that wasn’t in the code at all.</p><p>The inventory suite does a “sell everything” somewhere in the middle, to check the operation respects the filter. The contract suite needs at least three items on the account at the start. Separately, both are green. Run in sequence, the second fails, because the first sold everything.</p><p>One line fixes it — the contract suite now opens a couple of cheap cases if the inventory is empty. But before I saw that, I spent twenty minutes looking for a regression in contracts that was never there. <strong>A test that depends on execution order lies more convincingly than any bug.</strong></p><p>Concurrency lives in its own check: five simultaneous wheel spins, exactly one accepted. That is not catchable with a unit test — it needs a real database and real transactions.</p><h3>What’s in it and what isn’t</h3><p><strong>Works:</strong> Steam OpenID login with server-side verification and token refresh, an RU/EN interface with a currency switch, case opening (including batches of ten) with animation, provably fair with in-browser re-verification, upgrades, contracts, the daily wheel with vouchers, promo codes on top-up, an inventory with filters, history and sell-back, a live drop feed over Redis Pub/Sub batched every 300 ms, withdrawal requests with item locking and idempotency, the bot worker, a CRM with a GGR dashboard, the case builder, runtime settings and an audit log, and a nightly balance-versus-ledger reconciliation.</p><p><strong>Not there:</strong> payments, item deposits, case battles, referrals, KYC.</p><p>One honest caveat, because you would hit it on the first run otherwise. The withdrawal chain — queue, bot selection, offer, auto-confirmation, status polling — exists in the code and works, but nothing in the interface pulls it: the “Withdraw” button marks the item withdrawn and stops there. Exactly like the top-up, which credits a balance with no payment provider behind it. Both stubs are deliberately the only places that short-circuit a real path, so that wiring them up later is a deletion rather than an untangling.</p><pre>cp .env.example .env   # nothing to edit for a local run<br>pnpm setup             # install + docker + build + migrations + seed<br>pnpm dev               # api on :4000, web on :3000</pre><p>Code, an architecture document and a three-command setup: <a href="https://github.com/ialakey/caseforge"><strong>github.com/ialakey/caseforge</strong></a> · MIT.</p><h3>What I’d still argue about</h3><p>Two things I am genuinely not sure I got right, and one I would like taken apart.</p><p><strong>There are now two balancing models in one project</strong> — a power law price^(-k) for cases, an exponential tilt exp(-alpha × u) for contracts. Both are solved by bisection, both hit their target RTP, and they produce noticeably different rarity curves. I chose them separately, each for its own problem, and I am still not convinced they shouldn&#39;t have been one. If the goal is a controllable sense of <em>interestingness</em> rather than just hitting an RTP, which would you pick?</p><p><strong>The wheel’s budget is a hand-written table with a ceiling on cost.</strong> That is the one mechanic with no solver, and the reason is that a discount’s value depends on behaviour that hasn’t happened yet. I would believe there is a formulation that makes it solvable. I didn’t find one I trusted.</p><p><strong>And the line around what is configurable holds only because I drew it there.</strong> There are no permissions preventing an admin from touching the ticket space — the protection is that it isn’t in the registry at all. That is real protection in the sense that matters, and it is also one migration away from not being protection. Where would you draw that line, and would you enforce it with access control, or is leaving it out of the panel enough?</p><p>And if you find a hole in the code, say so. That was more or less the point of the whole exercise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1349e0200439" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First It Saves Your Time. Then It Starts Taking It Back.]]></title>
            <link>https://medium.com/@alakov.ilia/first-it-saves-your-time-then-it-starts-taking-it-back-f12e4048212e?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f12e4048212e</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[low-code]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 14:01:01 GMT</pubDate>
            <atom:updated>2026-11-03T14:01:01.947Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J7c4UhH4yeLNwagUXdAzHg.png" /></figure><p>At a hackathon, years ago, this is what winning looked like.</p><p>Define an entity — Order, Client, Product. Click through a couple of dialogs in the framework&#39;s Studio. Get back a list screen, an edit screen, field validation, a REST endpoint, and role-based access to all of it. Minutes, not days.</p><p>Meanwhile the team at the next table was configuring Spring Security and writing their third controller.</p><p>The framework was CUBA Platform — a full-stack Java framework for enterprise applications, built on Spring, whose lineage continues today as Jmix. But the framework barely matters, which is the point of this article. Swap in JHipster, Grails, Mendix, OutSystems, Retool, Budibase, or whatever your team is currently evaluating. <strong>They all have the same cost curve, and only the position of the inflection point differs.</strong></p><p>I used CUBA for several hackathons and a couple of real projects. The hackathons were a triumph. The real projects taught me something I’ve since watched play out in three other tools, and it’s worth naming precisely, because the failure mode is not “the tool was bad.”</p><h3>The leverage is real</h3><p>Let’s be fair to these tools first, because the dismissive take is wrong.</p><p>Out of the box you get an ORM, a UI framework, auto-generated CRUD screens, roles and permissions, a report generator, a REST API, and a working admin panel. The entire standard enterprise skeleton — the part every business application needs and nobody wants to write again — is already written.</p><p>That isn’t a trick or a demo. For MVPs, internal tools, prototypes, and anything where the end state is genuinely a set of forms over a database, it’s one of the fastest approaches available in the Java ecosystem, and the same is true of the equivalents in other stacks.</p><p>The velocity is real. The velocity is also the whole problem, because of what it hides.</p><h3>What you’re actually buying</h3><p>Here’s the mechanism, stated plainly:</p><blockquote><strong>A generator gives you enormous leverage over your domain and zero leverage over its own output.</strong></blockquote><p>You can create an entity in seconds. You cannot say “for every screen that displays a Client, change how this component behaves.&quot; The generated artifacts — dozens of XML files, screen descriptors, configuration — are not something the tool gives you a way to program <em>about</em>. They&#39;re a destination, not a material.</p><p>The concrete version, which is the single most useful thing I can tell you: <strong>you need to change one UI component across ten forms. There is no expression for that.</strong> You click through it, ten times, by hand. Then someone adds an eleventh form.</p><p>In hand-written code that’s a shared component and a one-line change. In generated code it’s ten identical manual edits with no compiler to tell you when you missed one, and a code review where the diff is a thousand lines of XML that nobody can read.</p><p>This is why the curve bends. Early on, generation is pure gain — you’re creating artifacts and the cost of each is near zero. Later, you’re <em>modifying</em> artifacts, and the cost of each modification scales with how many the tool generated on your behalf. Your early velocity becomes the size of your later maintenance surface. <strong>The thing that made month one fast is the thing that makes month twenty slow.</strong></p><h3>Three signals that you’ve crossed the inflection point</h3><p>This is the part I’d want to have had in advance. The curve is invisible while you’re on the good side of it, so you need leading indicators rather than a feeling.</p><h3>Signal 1: you write your first custom controller</h3><p>At some point a screen needs behavior the generator doesn’t express, and you drop into code:</p><pre>@Inject<br>private DataManager dataManager;</pre><pre>// custom controller, custom logic...</pre><p>This is supported. There’s inheritance, custom controllers, extension of standard screens. It works.</p><p>And it’s the moment the deal changes. Because now you’re writing an ordinary backend — the thing you adopted the framework to avoid — except <strong>on top of someone else’s abstractions, and still paying for all of them.</strong> You have the framework’s lifecycle, its data layer, its screen model, its XML, and none of the leverage that justified them, for this screen.</p><p>One custom controller is fine. It’s a signal, not a verdict. But the honest question when the third one appears is: what fraction of this application is now hand-written code wearing a framework’s constraints? If the answer is “most of the interesting parts,” you’re paying rent on an empty building.</p><p>The uncomfortable follow-up, which took me a while to say out loud: <em>why are these layers here at all?</em> And the answer isn’t comfortable either — <strong>they were useful at the beginning, and now you’re required to live with them.</strong> That’s not a bug in the tool. That’s the deal, and it was always the deal.</p><h3>Signal 2: maintaining the configuration is somebody’s job</h3><p>There’s a threshold where the generated XML stops being an implementation detail and becomes a thing the team has to actively manage: keeping it consistent, reviewing changes to it, resolving conflicts in it, onboarding people into it.</p><p>When you can name the person who is de facto the owner of the config, you’ve crossed it. Nobody plans that role. It accretes.</p><h3>Signal 3: you can’t see the SQL</h3><p>These frameworks add abstraction on top of the ORM, which is itself abstraction on top of SQL. As load grows, three things get hard in order: seeing what query was generated, changing it, and convincing the framework to let you.</p><p>Worth being precise about what this does and doesn’t mean. <strong>These tools were never built to replace manual optimization. They were built to replace manual CRUD development.</strong> Hitting a performance wall isn’t the tool failing at its job — it’s you having arrived somewhere outside its job. The mistake is not noticing that the boundary was crossed.</p><h3>Why the exit is expensive, and it isn’t the UI</h3><p>“Vendor lock-in” gets said about these tools, and it’s usually described shallowly — the UI is tied to the framework’s widget layer, so you’d have to rewrite the frontend.</p><p>That’s the cheap half. The expensive half is that <strong>your business logic was written in the framework’s paradigm.</strong> Not merely calling its APIs — shaped by its lifecycle, its transaction boundaries, its screen model, its permission checks, its assumptions about where validation happens.</p><p>That means migrating isn’t a port. Nobody can read the old code and mechanically translate it, because the rules aren’t all in the code — a meaningful share of them live in the framework’s behavior and in the configuration it generated. Migration is a <em>re-derivation of your own business rules from a system that partly encodes them implicitly.</em> That’s why these rewrites get estimated at three months and take eighteen.</p><p>And there’s an asymmetry that makes this genuinely hard to govern, which is worth stating because it’s not the tool’s fault at all:</p><blockquote><strong>The decision is made with week-one evidence. The bill arrives in year two. Usually the person who decided has moved on, and the person paying has no idea the choice was ever made.</strong></blockquote><p>The tool is at its most persuasive exactly when you have the least information about your own project. That combination is what turns a reasonable local decision into a bad global one.</p><h3>Using one of these well</h3><p>None of this is an argument against the category. It’s an argument for using it deliberately. Four things I would now insist on:</p><p><strong>1. Match the tool to the end state, not the starting point.</strong> Almost every project starts as CRUD. The question isn’t “is this CRUD today” but “in two years, is this still fundamentally forms over a database?” For an internal admin panel, a back-office tool, a document workflow — genuinely yes, and the tool will serve you for its entire life. For a product with a roadmap, the honest answer is usually no, and CRUD was just the first six weeks.</p><p><strong>2. Write down the exit condition before you start.</strong> One paragraph in the README: <em>what would have to be true for us to move off this?</em> Custom controllers past some count. Response times past some threshold. A requirement the tool structurally can’t express. The value isn’t in the specific numbers — it’s that a future team finds evidence that the trade-off was chosen rather than stumbled into, and has permission to revisit it.</p><p><strong>3. Keep the domain logic in plain services the framework calls.</strong> This is the highest-leverage habit available, and the one that converts a future rewrite back into a port. If pricing rules, eligibility logic, and state machines live in ordinary classes with no framework types in their signatures, they survive the migration untouched. If they live in screen controllers, they don’t survive at all. Almost nobody does this at the start, because at the start the framework’s way is faster — which is precisely how the trap closes.</p><p><strong>4. Treat “we needed to escape to code” as a scheduled review, not a workaround.</strong> The third custom controller should trigger a conversation, not a shrug.</p><h3>The shape is general</h3><p>CUBA was my particular version of this. Yours will be a different name and the same curve.</p><p>The tools that generate your application for you are genuinely powerful, and the power is entirely real in the window where it applies. But the deal is fixed and it doesn’t negotiate: <strong>you get velocity now against maintenance surface later, and you sign it at the exact moment you know least about how long the project will live.</strong></p><p>Use one for what it’s for and you gain an order of magnitude. Use one past its boundary and you accumulate technical debt faster than you can perceive it — not because you did anything wrong, but because the meter was running from the first generated screen and nobody was reading it.</p><p><em>If you’ve worked with one of these — Jmix, JHipster, Grails, Mendix, OutSystems, Retool — I’d be interested in where your inflection point landed and what tipped you off. My suspicion is that the signal is almost always the same one: the day someone asks for a change across N screens and the honest estimate is N times the work.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f12e4048212e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jigasi Is a SIP Gateway With Transcription Bolted On — and That Explains Every Bug]]></title>
            <link>https://medium.com/@alakov.ilia/jigasi-is-a-sip-gateway-with-transcription-bolted-on-and-that-explains-every-bug-ca2fb40149d3?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca2fb40149d3</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jitsi]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 14:01:03 GMT</pubDate>
            <atom:updated>2026-10-27T14:01:03.672Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Oy4yln063eD30l819CwRw.png" /></figure><p>We were building a pipeline to populate electronic medical records from doctor–patient video consultations. Jitsi Meet is open source, the documentation exists, so the estimate for “connect a bot, get a transcript” was about two days.</p><p>That layer took disproportionately long. Not because any individual piece was hard — because every problem we hit traced back to a single architectural fact that nothing in the setup documentation puts up front:</p><blockquote><strong>Jigasi is a SIP gateway. Transcription is a later feature built on top of telephony architecture.</strong></blockquote><p>Once you know that, the confusing parts stop being confusing. The transcriber joins your conference as a <em>participant</em>, because that’s what a phone call is. The config file is full of SIP parameters you’ll never use, because it’s a telephony config. A session “ends” when the room empties, because that’s when a call ends. The transcript arrives as a file on disk as a side effect, because that’s what a call recording is.</p><p>None of those are quirks. They’re all the same fact, showing up in five places.</p><p>And there’s a second thing that made this project unusual, which I’ll come back to at the end: the output of this pipeline becomes clinical data. A transcription error is not a UX annoyance when the sentence contains a dosage.</p><h3>The architecture, and where transcription actually lives</h3><p>Jitsi Meet isn’t a monolith with an “enable STT” button. It’s a set of services with divided responsibilities:</p><ul><li><strong>Jitsi Meet</strong> — frontend and signaling;</li><li><strong>Jitsi Videobridge (JVB)</strong> — the media server routing audio and video between participants;</li><li><strong>Jicofo</strong> — the conference manager;</li><li><strong>Prosody</strong> — the XMPP server and signaling transport;</li><li><strong>Jigasi</strong> — the SIP/XMPP gateway that joins a conference and provides transcription;</li><li><strong>STT backend</strong> — external speech recognition; in our case, Vosk.</li></ul><p>The critical placement: <strong>Jigasi sits inside the conference, not beside it.</strong> From Jitsi Meet’s perspective it’s just another peer. It receives audio streams from participants, proxies them to the STT service, and returns the transcript.</p><p>Which means “connect a bot” is not a task at the Jitsi Meet API level. There is no API for it. It’s an infrastructure task — you are provisioning a participant.</p><p>Everything below follows from that.</p><h3>Consequence 1: a participant needs an account, so failures look like absence</h3><p><em>Symptom: transcription is started in the UI, and no second participant ever appears in the conference.</em></p><p>Because Jigasi joins over XMPP using a pre-configured account, the most common cause is exactly what you’d expect for a login: the account doesn’t exist, or the password doesn’t match the config. And because a failed join is not an error anywhere in the <em>user-facing</em> path, you get silence — the UI shows transcription as started, and nothing happens.</p><p>/var/log/jitsi/jigasi.log is where the XMPP authentication error actually lives. In this stack the logs are not a debugging aid, they&#39;re the primary interface; almost nothing surfaces upward.</p><p>There’s a related historical wrinkle worth knowing if you’re reading older guides: in earlier configurations you had to manually invite the jitsi_meet_transcribe user into the room through the Jitsi Meet UI to start transcription. That is exactly what &quot;the transcriber is a participant&quot; looks like taken literally, and it was an obvious anti-pattern for anything automated. Newer configurations connect the transcriber automatically — but only when every layer below is correct.</p><h3>Consequence 2: you’re configuring a telephony product</h3><p>Jigasi’s config file is /etc/jitsi/jigasi/sip-communicator.properties. The name is not an accident, and neither is the content: it&#39;s full of SIP parameters that are irrelevant when you&#39;re doing transcription without any actual telephony.</p><p>The practical cost is that you can’t tell, by reading it, which sections apply to you. Half of it is answering a question you didn’t ask. The transcription-relevant portion:</p><pre># Enable transcription<br>org.jitsi.jigasi.ENABLE_TRANSCRIPTION=true</pre><pre># Where transcripts are written<br>org.jitsi.jigasi.transcription.DIRECTORY=/var/lib/jigasi/transcripts</pre><pre># BASE_URL is used when ADVERTISE_URL=true — for generating links to transcripts<br>org.jitsi.jigasi.transcription.BASE_URL=http://localhost/<br>org.jitsi.jigasi.transcription.jetty.port=-1<br>org.jitsi.jigasi.transcription.ADVERTISE_URL=false</pre><pre># Save formats (to disk)<br>org.jitsi.jigasi.transcription.SAVE_JSON=false<br>org.jitsi.jigasi.transcription.SAVE_TXT=true</pre><pre># Send formats (real time to participants over XMPP)<br>org.jitsi.jigasi.transcription.SEND_JSON=true<br>org.jitsi.jigasi.transcription.SEND_TXT=false</pre><pre># STT backend: Vosk over WebSocket<br>org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.VoskTranscriptionService<br>org.jitsi.jigasi.transcription.vosk.websocket_url=ws://localhost:2700</pre><h3>Consequence 3: SAVE and SEND are different products</h3><p>This is the setting that quietly determines your entire downstream architecture, and it’s easy to read past.</p><p><strong>SAVE_* writes the final transcript to disk when the session ends. </strong><strong>SEND_* pushes interim results to participants in real time over XMPP.</strong> They are not two formats of the same thing — they&#39;re two different features with different consumers and different timing.</p><p>Live captions in the meeting UI are SEND. A file you can process afterward is SAVE. Configure only SEND and you’ll watch subtitles appear on screen during a call and then find an empty transcripts directory, which reads as a bug and isn’t one.</p><p>For a post-consultation pipeline, SAVE is what matters. We kept SEND on for JSON because live captions were useful to the doctor, but nothing downstream consumes them.</p><p>The neighbouring ADVERTISE_URL=false means Jigasi won&#39;t publish a transcript link in the conference chat. With true and a correct BASE_URL, participants get a link to the transcript file at the end.</p><p>For multiple languages, websocket_url accepts a JSON mapping — {&quot;en&quot;: &quot;ws://localhost:2700&quot;, &quot;fr&quot;: &quot;ws://localhost:2710&quot;} — with a separate STT instance per language.</p><h3>Consequence 4: three configs, and partial configuration fails as silence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Id-KTxfhNtysmQDNknkPGA.png" /></figure><p>Transcription touches <strong>three independent configuration layers</strong>, and this is where most of the two days actually went.</p><p><strong>Layer one, Jigasi</strong> — above.</p><p><strong>Layer two, the STT backend.</strong> Vosk over WebSocket, which is what VoskTranscriptionService expects:</p><pre>docker run -d -p 2700:2700 alphacep/kaldi-en:latest</pre><p>Port 2700 has to be reachable from wherever Jigasi runs. Alternative backends — Deepgram, Google STT, Whisper — need a custom TranscriptionService implementation, which in practice means either forking Jigasi or building an adapter layer.</p><p><strong>Layer three, the Jitsi Meet frontend</strong>, at /etc/jitsi/meet/&lt;hostname&gt;-config.js:</p><pre>transcription: {<br>    enabled: true,<br>    useAppLanguage: true,<br>    preferredLanguage: &#39;en-US&#39;,<br>    disableStartForAll: false,<br>    autoCaptionOnRecord: false,<br>},</pre><p>Without this, <strong>the transcription option doesn’t appear in the UI even when Jigasi is configured perfectly.</strong></p><p>useAppLanguage: true takes the language from the participant&#39;s browser. With false, preferredLanguage wins. For a medical scenario with a fixed consultation language we used the explicit setting rather than letting a browser locale decide how a clinical conversation gets recognized.</p><p>Then restart all four services, Prosody first, because the others establish XMPP connections at startup:</p><pre>sudo systemctl restart prosody.service<br>sudo systemctl restart jicofo.service<br>sudo systemctl restart jitsi-videobridge2.service<br>sudo systemctl restart jigasi.service</pre><p>Here’s the general property that made this expensive, and it’s the most transferable thing in the article:</p><blockquote><strong>In a stack assembled from independent services, a partially-configured feature does not produce an error. It produces nothing.</strong></blockquote><p>Three layers, each of which fails silently and independently. Miss the frontend config and the button is missing. Miss the Jigasi config and the button exists and does nothing. Miss the STT container and the bot joins and transcribes silence. All three states look approximately like “it doesn’t work,” and none of them throw.</p><p>Directory permissions belong in this same category. Jigasi runs as the jigasi user, and if /var/lib/jigasi/transcripts doesn&#39;t belong to it, transcription runs, live captions appear, and no file is ever written:</p><pre>sudo chown -R jigasi:jigasi /var/lib/jigasi/transcripts</pre><p>Nothing complains. The captions on screen are the system working; the missing file is the system also working, at a layer nobody is watching.</p><h3>Consequence 5: “the session ended” means “the room emptied”</h3><p>Jigasi writes the final transcript when the session ends — and a session ends, in its model, when every participant has left. Which is a phone call’s definition of ending, and it is not the same as “the consultation is over.”</p><p>For downstream processing you need to reliably detect that moment yourself. The workable options are polling the transcripts directory, or webhook integration — Jitsi Meet Webhooks if you’re on Jitsi as a Service, or a custom Prosody module for on-session-end events.</p><p>That this is an integration problem at all, rather than a callback you register, is again the gateway architecture showing through: a call doesn’t notify your application when it finishes. It just finishes.</p><h3>Where the stakes change: a transcript becomes a medical record</h3><p>Everything above is ordinary integration work. This part is what made the project different.</p><p>The raw .txt from Jigasi is unstructured dialogue <em>(invented, not real patient data)</em>:</p><pre>[00:01:23] Doctor:  How long ago did the symptoms start?<br>[00:01:31] Patient: About a week ago. First a fever, then a cough.<br>[00:02:15] Doctor:  Have you taken anything? Ibuprofen, paracetamol?<br>[00:02:22] Patient: Only paracetamol, 500 milligrams three times a day.<br>[00:03:10] Doctor:  All right. Let&#39;s continue the paracetamol and add amoxicillin...</pre><p>From that, an EMR needs presenting complaints, symptom history, prescriptions with drug/dose/regimen, recommendations, and follow-up actions.</p><p>Now look at that transcript again and notice what a recognition error costs here. “500 milligrams three times a day” is a sentence where every token is load-bearing. Drug names, dosages and diagnoses are exactly the vocabulary that general-purpose acoustic models handle worst — they’re rare words, they’re often similar-sounding, and the model has no reason to prefer the right one.</p><p>The alphacep/kaldi-en:latest image ships a relatively light model. <strong>For an MVP that&#39;s fine. For clinical data it is disqualifying</strong>, and the reason is not that the accuracy number is too low — it&#39;s that the errors are concentrated in precisely the tokens that become structured fields.</p><p>The options, in increasing order of seriousness: heavier Vosk models (alphacep/kaldi-en-streaming:latest), a commercial STT with a medical language profile, or LLM post-processing with domain normalization.</p><p>We used the third, and it’s worth being precise about what that layer was doing. <strong>The LLM was not adding intelligence. It was compensating for a known deficiency in the layer beneath it</strong> — reconstructing correct drug names and prescription structures from surrounding context, because context is exactly the information the acoustic model discarded. That’s a legitimate mitigation and a bad foundation. If I were doing this again with a budget, I’d fix the STT rather than paper over it, and keep the LLM for the structuring job it actually deserves.</p><p>The extracted result was mapped onto FHIR-compatible structures and handed to a React form for the doctor to validate before anything was written to the EMR.</p><p><strong>Fully automatic recording without review was never on the table.</strong> Not as a caution — as a design constraint. A pipeline whose input is acoustically unreliable and whose output is a clinical record requires a human at the last step, and that requirement doesn’t relax as the models improve; it relaxes when someone is willing to own the liability, which is a different kind of decision entirely.</p><h3>The decision that made all of this tractable</h3><p>One architectural choice was worth more than any configuration detail: <strong>processing is post-consultation, not real time.</strong></p><p>That was a deliberate simplification. Live prompts to a doctor during a call require an entirely different level of latency, orchestration and fault tolerance — you’re now in a system where being slow is being wrong. For populating a chart afterward, the asynchronous model removes almost all of that: a stage can retry, a slow LLM call costs nobody anything, and a failure is recoverable rather than visible.</p><p>If your goal is filling in records rather than assisting live, take the asynchronous option. The temptation to do it in real time is strong and buys very little.</p><h3>What generalizes</h3><p>Very little of this is specific to Jitsi, and one thing is worth carrying to the next integration:</p><p><strong>When you adopt a component from an open-source stack, find out what it originally was.</strong> Its original purpose determines its vocabulary, its config file, its failure modes, and — most importantly — its definition of “done.” Jigasi’s definition of a finished session is a telephony definition, and it was telephony’s definitions that cost us the time, not any of the code we wrote.</p><p>The second thing, which I’d now check on any multi-service integration before writing a line: <strong>find out which layers fail silently.</strong> In this stack, three of them do. Knowing that in advance turns a two-day estimate that becomes two weeks into a two-day estimate that’s approximately right, because you stop debugging by inference and start reading the one log that actually knows.</p><p><em>If you’ve wired Jitsi transcription to a different STT backend — especially Whisper or a domain-specific medical model — I’d be interested in how you handled the adapter layer without forking Jigasi. That’s the seam I never found a clean answer for.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca2fb40149d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discord Doesn’t Give You a Recording. It Gives You 20-Millisecond Frames.]]></title>
            <link>https://medium.com/@alakov.ilia/discord-doesnt-give-you-a-recording-it-gives-you-20-millisecond-frames-bbbd2a6a7aba?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbbd2a6a7aba</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[discord]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 13:01:03 GMT</pubDate>
            <atom:updated>2026-10-20T13:01:03.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0ov91CrUoftpKPfDu18fA.png" /></figure><p>I built a Spring Boot service that plays audio into Discord voice channels, synthesizes speech, records those channels, ships the recordings to Telegram, and exposes all of it through a web panel. Three front ends, one process.</p><p>The features took a weekend. The next several weeks went somewhere else entirely.</p><p>Looking back at what actually cost time, there’s a pattern: <strong>almost none of it was my logic. It was other people’s contracts.</strong> Discord decides what a “recording” is. Telegram decides how long a file URL lives. The JVM decides what happens when you accumulate 48,000 samples per second in a ByteArrayOutputStream. Each of those made a decision for me, silently, and I found out later.</p><p>That’s what this article is about. Not a tour of the codebase — the three boundaries, what each one actually demanded, and what it cost to find out.</p><p>For scale, so you can calibrate: a server with ~50 active users, peak load around 20 simultaneous voice connections. Small. Every problem below still happened.</p><h3>Boundary 1: Discord</h3><h3>What Discord means by “recording”</h3><p>Here’s the thing that reframes the whole project. Discord does not give you a recording. There’s no “download the session” call. What you get is a <strong>raw PCM stream: 20-millisecond frames, 48 kHz, stereo, signed 16-bit, little-endian.</strong> Frames arrive, forever, until you stop asking.</p><p>Everything a recording normally implies — a container, a codec, a duration header, a file — is yours to build:</p><pre>public class SimpleAudioRecorder extends AudioReceiveHandler {<br>    private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();</pre><pre>    @Override<br>    public boolean canReceiveCombined() {<br>        return true;   // give me the already-mixed stream, not per-user tracks<br>    }</pre><pre>    @Override<br>    public void handleCombinedAudio(@NotNull CombinedAudio audio) {<br>        try {<br>            buffer.write(audio.getAudioData(1.0f));<br>        } catch (IOException e) {<br>            log.error(&quot;Failed to write audio data&quot;, e);<br>        }<br>    }<br>}</pre><p>That is the entire recorder, and it is also the bug. Hold that thought — it detonates in Boundary 3.</p><p>Turning the accumulated bytes into a file means declaring Discord’s format by hand, because nothing in the stream tells you what it is:</p><pre>AudioFormat format = new AudioFormat(<br>        48000,   // sampleRate  — Discord&#39;s, not negotiable<br>        16,      // sampleSizeInBits<br>        2,       // channels (stereo)<br>        true,    // signed<br>        false    // bigEndian<br>);</pre><p>Get any of those five numbers wrong and you don’t get an error. You get audio that plays at the wrong speed, or static. The format isn’t validated against anything, because there’s nothing to validate it against — you’re describing bytes you already have.</p><h3>The intent that makes the bot blind</h3><p>Second Discord contract, and this one is a genuinely silent failure:</p><pre>JDA jda = JDABuilder.createDefault(token)<br>        .enableIntents(<br>                GatewayIntent.GUILD_MESSAGES,<br>                GatewayIntent.MESSAGE_CONTENT,     // ← without this, since April 2022<br>                GatewayIntent.GUILD_VOICE_STATES,<br>                GatewayIntent.GUILD_MEMBERS<br>        )<br>        .addEventListeners(listener)<br>        .build();</pre><pre>jda.awaitReady();</pre><p>Without MESSAGE_CONTENT, the bot connects successfully, appears online, receives MessageReceivedEvent for every message — and every one of them has empty content. No exception, no warning. Your command handler runs, finds nothing that starts with !, and returns. The bot looks alive and does nothing.</p><p>GUILD_VOICE_STATES has the same property one layer down: without it, voice connect/disconnect events simply never fire, and the moderation layer that&#39;s supposed to kick blocked users silently never triggers.</p><p>There’s a general shape here worth naming: <strong>a permissions model you didn’t opt into fails as absence, not as error.</strong> You don’t get “access denied,” you get an empty string, and empty strings flow through code happily.</p><h3>One AudioPlayerManager, forever</h3><p>Third: LavaPlayer’s threading contract. AudioPlayerManager owns thread pools. Create one per request — which is what feels natural when each command is self-contained — and you leak threads until the process dies.</p><p>One manager for the whole bot, injected as a singleton, and individual AudioPlayer instances per playback. That&#39;s the whole rule, and it&#39;s the kind that&#39;s invisible until you look at a thread dump wondering why a 50-user bot has 400 threads.</p><p>The adapter between LavaPlayer and Discord is AudioPlayerSendHandler — it pulls frames from the player and pushes them to Discord&#39;s gateway over UDP. Wire it once per connection:</p><pre>audioManager.openAudioConnection(channel);<br>AudioPlayer player = playerManager.createPlayer();      // per playback<br>audioManager.setSendingHandler(new AudioPlayerSendHandler(player));</pre><pre>playerManager.loadItem(filepath, new AudioLoadResultHandler() {<br>    @Override<br>    public void trackLoaded(AudioTrack track) {<br>        player.playTrack(track);<br>        player.addListener(new AudioEventAdapter() {<br>            @Override<br>            public void onTrackEnd(AudioPlayer p, AudioTrack t, AudioTrackEndReason reason) {<br>                audioManager.closeAudioConnection();     // or you hold the channel forever<br>            }<br>        });<br>    }</pre><pre>    @Override<br>    public void loadFailed(FriendlyException e) {<br>        log.error(&quot;Failed to load track: {}&quot;, filepath, e);<br>        audioManager.closeAudioConnection();             // including on failure<br>    }<br>});</pre><p>Note that closeAudioConnection() appears in both callbacks. Omit it from loadFailed and one bad file leaves the bot sitting silently in a voice channel indefinitely. Cleanup on the success path is obvious; cleanup on the failure path is where these bugs live.</p><h3>Boundary 2: Telegram</h3><h3>Polling isn’t the lazy option</h3><p>The recordings had to reach Telegram, and Telegram offers two ways to receive updates: a webhook or long polling. Webhooks are the “proper” answer and the wrong one here.</p><p>A webhook requires a public HTTPS endpoint, a valid certificate, handling Telegram’s IP range, and nginx or a load balancer for TLS termination. For a self-hosted bot on one machine behind NAT, that’s an entire infrastructure layer bolted on to receive a handful of messages a day.</p><p>Long polling works behind NAT and a firewall, needs no reverse proxy, and retries on network failure by construction. For this deployment shape it isn’t the shortcut — it’s the correct choice.</p><p>But polling has its own contract, and one parameter carries all of it:</p><pre>String url = String.format(<br>        &quot;https://api.telegram.org/bot%s/getUpdates?offset=%d&amp;timeout=25&quot;,<br>        botToken, lastUpdateId + 1<br>);</pre><p>timeout=25 is what makes it <em>long</em> polling: Telegram holds the connection open for up to 25 seconds waiting for an update, and returns immediately when one arrives. Drop that parameter and the call returns instantly with an empty array, every time, and your 2-second timer becomes a request every 2 seconds forever — the same latency, a hundred times the traffic, and a decent chance of getting rate-limited for it.</p><p>The offset half matters just as much: it acknowledges everything up to lastUpdateId. Forget to advance it and you reprocess the same updates in a loop.</p><pre>for (JsonElement update : json.getAsJsonArray(&quot;result&quot;)) {<br>    JsonObject obj = update.getAsJsonObject();<br>    lastUpdateId = obj.get(&quot;update_id&quot;).getAsLong();     // advance, or reprocess forever<br>    processUpdate(obj);<br>}</pre><h3>The URL that expires in an hour</h3><p>The nicest feature in the bot pipes a Telegram voice message into a Discord channel. Reply to a voice message with /sendvoice General and it plays in Discord. The flow:</p><ol><li>extract file_id from reply_to_message;</li><li>call getFile to obtain a file_path;</li><li>build <a href="https://api.telegram.org/file/bot%7Btoken%7D/%7Bfile_path%7D;">https://api.telegram.org/file/bot{token}/{file_path};</a></li><li>hand the URL to LavaPlayer, which streams it straight into Discord.</li></ol><p>Elegant — LavaPlayer speaks HTTP, so no download step is needed at all.</p><p>Except <strong>file_path is valid for one hour.</strong> Play a message from this morning and you get a 403 from inside LavaPlayer&#39;s loader, surfacing as a generic load failure with no hint about why. The fix is caching the file locally on first fetch, and the lesson is more general: when a third party hands you a URL, find out what its lifetime is <em>before</em> you store it anywhere or hand it to another system.</p><h3>And the format is not negotiable either</h3><p>Telegram voice messages must be OGG Opus. Not MP3, not WAV, not “audio.” So the recording pipeline ends with an external process:</p><pre>ProcessBuilder pb = new ProcessBuilder(<br>        &quot;ffmpeg&quot;, &quot;-i&quot;, wavFile.getAbsolutePath(),<br>        &quot;-c:a&quot;, &quot;libopus&quot;,<br>        &quot;-b:a&quot;, &quot;64k&quot;,<br>        oggFile.getAbsolutePath()<br>);<br>Process process = pb.start();<br>int exitCode = process.waitFor();</pre><pre>if (exitCode != 0) {<br>    throw new RuntimeException(&quot;ffmpeg failed: &quot; + exitCode);<br>}</pre><p>Opus is the right codec on the merits anyway — roughly 10:1 against WAV at comparable quality, and it’s designed for speech (variable bitrate, DTX). But it’s worth being clear that this wasn’t an engineering choice. Telegram made it.</p><p>One thing that line does wrong, and I left it wrong in the repo so I’d remember: it reports the exit code and throws away ffmpeg’s stderr. ffmpeg tells you <em>exactly</em> what went wrong, in detail, on stderr, and &quot;ffmpeg failed: 1&quot; tells you nothing. Capture and log stderr on any non-zero exit. That&#39;s a five-minute change that saves an hour every time it fires.</p><h3>Boundary 3: the JVM</h3><p>Now the recorder from the top of the article detonates.</p><p>ByteArrayOutputStream accumulating 48 kHz stereo 16-bit PCM: 48,000 samples × 2 channels × 2 bytes = <strong>about 192 KB per second, ~11.5 MB per minute.</strong> The buffer also doubles its backing array as it grows, so peak usage during a copy is roughly twice the current size.</p><p>Ten minutes of recording is ~115 MB of live bytes and a ~230 MB spike while resizing. On a default heap, that’s an OutOfMemoryError — and it takes the whole process down, along with the Discord connection, the Telegram poller and the web panel, because all three share one JVM.</p><p>That’s the price of the monolith, paid in full, in one place.</p><p>The fix is to stop treating the recording as a value in memory and start treating it as a stream to disk:</p><pre>public class StreamingAudioRecorder extends AudioReceiveHandler {<br>    private final FileOutputStream fileOutput;<br>    private final DataOutputStream dataOutput;</pre><pre>    public StreamingAudioRecorder(String filename) throws IOException {<br>        this.fileOutput = new FileOutputStream(filename);<br>        this.dataOutput = new DataOutputStream(fileOutput);<br>        writeWavHeader();   // header with a placeholder where the size will go<br>    }</pre><pre>    @Override<br>    public void handleCombinedAudio(@NotNull CombinedAudio audio) {<br>        try {<br>            dataOutput.write(audio.getAudioData(1.0f));<br>        } catch (IOException e) {<br>            log.error(&quot;Failed to write audio chunk&quot;, e);<br>        }<br>    }</pre><pre>    public void close() throws IOException {<br>        dataOutput.close();<br>        fileOutput.close();<br>        updateWavHeader();   // seek back and patch the real size in<br>    }<br>}</pre><p>Constant memory, any duration. But look at what it costs: writeWavHeader() writes a header containing a size you don&#39;t know yet, and updateWavHeader() reopens the file to patch it. The WAV format wants the length up front; a stream doesn&#39;t have one. So you write a lie and correct it on close.</p><p>And that introduces a new failure mode the buffered version didn’t have: <strong>if the process dies mid-recording, the file on disk has a header claiming a size it doesn’t have.</strong> The audio data is all there and most tools will refuse to open it. The buffered version lost everything on a crash; the streaming version keeps everything and makes it look corrupt. Which is better, but it’s a different problem, not the absence of one.</p><p>Hence the shutdown hook — not as a nice-to-have, but as the thing that closes the header:</p><pre>@PreDestroy<br>public void onShutdown() {<br>    recordingService.stopAllRecordings();     // this is what patches the headers</pre><pre>    jda.getGuilds().forEach(guild -&gt; {<br>        AudioManager am = guild.getAudioManager();<br>        if (am.isConnected()) {<br>            am.closeAudioConnection();<br>        }<br>    });<br>}</pre><p>A @PreDestroy in a bot that mostly plays sound effects looks like ceremony. It isn&#39;t. It&#39;s the difference between a valid file and a broken one on every deploy.</p><h3>The one decision that kept paying</h3><p>Not everything was somebody else’s contract. One design choice was mine and it aged better than anything else in the project.</p><p>Command dispatch falls back to the filesystem:</p><pre>String prefix = parts[0].toLowerCase();<br>Command command = commands.get(prefix);</pre><pre>if (command != null) {<br>    command.execute(event, parts.length &gt; 1 ? parts[1] : &quot;&quot;);<br>} else {<br>    handleAudioFallback(event, prefix);   // try to play &lt;prefix&gt;.mp3<br>}</pre><p>An unrecognized command isn’t an error — it’s a lookup for &lt;prefix&gt;.mp3 in the audio directory. Which means <strong>adding a new sound requires no code, no deploy, and no restart.</strong> Drop victory.mp3 in the folder and !victory works.</p><p>That one branch is responsible for essentially all of the bot’s actual day-to-day use. Every deliberate feature I built got used occasionally; the sounds get used constantly, by people who have never asked me for anything, because adding one costs them nothing and costs me nothing.</p><p>The transferable version: when you have a namespace of user-invoked names, consider making the miss case do something useful instead of reporting failure. It converts a feature request queue into a folder.</p><h3>Why one process, and what that cost</h3><p>Three front ends — Discord commands, Telegram, and a Thymeleaf web panel — share one service layer. Business operations (list channels, pick a voice channel, synthesize and play audio, manage block lists) are Spring services, reused from all three.</p><p>For 50 users, the monolith is straightforwardly correct: one container, one process, one log, one place to look. Operations like “block a user and kick them from voice” are atomic without any distributed machinery. A stack trace runs from Discord event to ffmpeg invocation unbroken.</p><p>But the OOM section above <em>is</em> the bill for that decision, and it’s worth stating plainly: <strong>one heap means one blast radius.</strong> A recording that runs long doesn’t degrade recording — it kills the Telegram poller and the admin panel too. At this scale that’s an acceptable trade, made knowingly. If load grew an order of magnitude, the first thing to leave would be audio processing, precisely because it’s the component whose resource usage is unbounded by user input.</p><p>Two other bills from the same decision, still unpaid:</p><p><strong>Block lists live in memory.</strong> Every block is lost on restart. That’s a blocked_users table and a Flyway migration away from fixed, and it stays unfixed because restarts are rare and the moderators are three people who remember.</p><p><strong>Nothing rate-limits </strong><strong>!speak.</strong> One user can queue TTS indefinitely and monopolize a channel. On a server where everyone knows everyone, social pressure does the job. On a public server it would be the first thing exploited, and a token bucket per user would be non-optional.</p><p>Both of those are honest technical debt: known, cheap to fix, deliberately not fixed because the actual deployment doesn’t need them yet. Worth writing down, because the version of this article that lists them as “roadmap items” would be pretending they’re planned work rather than accepted risk.</p><h3>What I’d tell myself at the start</h3><p>The features were never the hard part. Playing an MP3 into a voice channel is fifteen lines. The hard part was everything that a system I don’t control had already decided:</p><ul><li><strong>Discord decided</strong> that a recording is raw frames, that message content requires an opt-in that fails as silence, and that one thread pool means one manager.</li><li><strong>Telegram decided</strong> that voice is Opus, that file URLs die in an hour, and that one query parameter separates efficient polling from hammering.</li><li><strong>The JVM decided</strong> what happens when audio arrives faster than you write it anywhere.</li></ul><p>None of those are documented where you look for them. All of them are documented somewhere. The pattern I’d want to internalize is: <strong>when you integrate with a system, the interesting questions aren’t “what can it do” but “what has it already decided for me” — the formats, the lifetimes, the failure modes, and which failures show up as errors versus as nothing at all.</strong></p><p>The silent ones cost the most. An empty message string, a 403 an hour later, a header with a wrong number in it. Every one of those looked like working software right up until it didn’t.</p><p><em>If you’ve run a recording bot at real scale, I’d like to hear how you handled the WAV header problem — patching it on close works but leaves a window where a crash produces a valid-looking broken file, and I suspect writing to a container that doesn’t need the length up front is the actual answer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbbd2a6a7aba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Video URL Isn’t a Resource. It’s a Permission.]]></title>
            <link>https://medium.com/@alakov.ilia/a-video-url-isnt-a-resource-it-s-a-permission-6d480fca7805?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d480fca7805</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 13:01:03 GMT</pubDate>
            <atom:updated>2026-10-13T13:01:03.250Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*316ZANoq2n8l_zvX_bRu_g.png" /></figure><p>The idea was simple: watch something with friends so that one person pausing pauses it for everybody. Not Discord Go Live with its artefacts and mush, not “press space on three” — an actual shared player, where whoever clicks, everyone seeks.</p><p>I thought it was an evening’s work. It turned into four non-trivial problems, three of which I couldn’t find described anywhere, and one night spent working out why hls.js was silently doing nothing at all.</p><p>The thing that took the longest was also the thing I had most wrong going in, and it’s worth stating up front, because it generalises well past video:</p><blockquote><em>A media URL from a CDN is not a resource you can address. It is a </em><strong><em>permission</em></strong><em>, issued to one IP, one </em><em>Referer, and a window of about an hour. Handing it to a browser hands over none of those.</em></blockquote><p>Everything below follows from that. What I built: a FastAPI app with rooms, a shared player, Discord login with guild membership checks, chat, and automatic episode tracking. The links themselves come from anime-dl-core, <a href="https://medium.com/@alakov.ilia/c229dc51ba95">a library I wrote for exactly that</a> job. Everything here is what happens between &quot;the library returned a link&quot; and &quot;video plays in sync for five people&quot;.</p><p>The whole thing: <a href="https://github.com/ialakey/anime-watch-together"><strong>github.com/ialakey/anime-watch-together</strong></a> — MIT, one docker compose up.</p><pre>Browser ──WebSocket──► FastAPI ──► RoomManager (room state in memory)<br>   │                      │<br>   │                      ├──► Catalog ──► anime-dl-core ──► the source sites<br>   │                      │                                  └► the embedded players<br>   │                      │<br>   └──HTTP(S) video──────►└──► StreamProxy ──► the player&#39;s CDN</pre><h3>Problem 1: the direct link is useless</h3><p>The naive plan: call the library, get https://cdn.example/…/720.mp4:hls:manifest.m3u8, put it in &lt;video src&gt;, go home.</p><p>It doesn’t work. Three reasons, each sufficient on its own.</p><p><strong>The CDN requires the right </strong><strong>Referer.</strong> Without it you get a 403. And the browser will not let you set an arbitrary Referer — it&#39;s a forbidden header, fetch drops it silently.</p><p><strong>The link is bound to the IP that obtained it.</strong> Your <em>server</em> made the request for that link, so the link is valid for the server’s IP. Your user’s browser arrives from a different address and gets turned away.</p><p><strong>The links are short-lived.</strong> An hour or two, then they expire. Even if the first two problems vanished, a user who opened a tab in the morning would find a dead player by the evening.</p><p>The conclusion is unpleasant but unambiguous: <strong>the video has to go through your own server.</strong> Which immediately raises the next question — how do you avoid becoming an open relay that strangers use to pull whatever they like?</p><h3>Signed tokens</h3><p>The answer: for each external link, the server issues a token — HMAC-SHA256 over the URL, the headers and an expiry.</p><pre>def sign(self, url: str, headers: dict[str, str], *, is_playlist: bool) -&gt; str:<br>    payload = {<br>        &quot;u&quot;: url,<br>        &quot;h&quot;: headers or {},<br>        &quot;p&quot;: 1 if is_playlist else 0,<br>        &quot;e&quot;: int(time.time()) + self.settings.stream_token_ttl,<br>    }<br>    body = _b64encode(json.dumps(payload, separators=(&quot;,&quot;, &quot;:&quot;)).encode(&quot;utf-8&quot;))<br>    signature = hmac.new(self._secret, body.encode(&quot;ascii&quot;), hashlib.sha256).digest()<br>    return f&quot;{body}.{_b64encode(signature)}&quot;</pre><p>Verification is symmetric: compare the signature with hmac.compare_digest, check the expiry, and reject anything that isn&#39;t http/https — otherwise you&#39;ll be serving file:///etc/passwd on day one.</p><p>The client only ever sees /api/stream/segment?t=&lt;token&gt;. Forging a token without the key is impossible, so only our server can mint links, and the proxy doesn&#39;t become an open relay.</p><h3>Rewriting HLS on the fly</h3><p>With mp4 it’s easy: proxy the bytes, pass Range through, done. HLS is harder, because a playlist is a text file with more links inside it. And there are a lot of them: quality variants, segments, AES-128 encryption keys, initialisation segments in EXT-X-MAP.</p><p>Every one has to be resolved to absolute (they’re often relative) and replaced with a link to the same proxy:</p><pre>def rewrite_playlist(self, content: str, playlist_url: str, headers: dict[str, str]) -&gt; str:<br>    lines: list[str] = []<br>    for raw_line in content.splitlines():<br>        line = raw_line.strip()<br>        if not line:<br>            lines.append(raw_line)<br>        elif line.startswith(&quot;#&quot;):<br>            lines.append(self._rewrite_tag(line, playlist_url, headers))<br>        else:<br>            absolute = urljoin(playlist_url, line)<br>            lines.append(<br>                self.proxy_url(https://medium.com/absolute,%20headers,%20is_playlist=_looks_like_playlist(absolute))<br>            )<br>    return &quot;\n&quot;.join(lines) + &quot;\n&quot;</pre><p>The subtle case is tags where the URI hides inside an attribute:</p><pre>#EXT-X-KEY:METHOD=AES-128,URI=&quot;key.bin&quot;,IV=0x0</pre><p>Miss the encryption key and the player goes for it directly and collects its well-deserved 403. This one isn’t obvious from the symptom: the manifest parses, the segments download, and there’s simply no picture.</p><p>On a real episode that came to 252 rewritten links in one playlist. It works.</p><h3>What it costs</h3><p>Honestly: proxying isn’t free. Every HLS segment is a separate request through your server, so an episode of 252 segments is 252 proxied requests <em>per viewer</em>, and the traffic transits twice.</p><p>You can soften it: serve the mp4 track instead of HLS where one exists (a single Range request instead of hundreds), and don&#39;t set a quality ceiling higher than you need — mine is STREAM_MAX_QUALITY=1080. But fundamentally this is the price of the CDN refusing to talk to the browser directly.</p><h3>Problem 2: synchronisation without echo</h3><p>The naive shared player: on play, broadcast &quot;play&quot;; on receiving &quot;play&quot;, call video.play().</p><p>And you immediately get an infinite loop: calling video.play() fires a play event, which sends a message, which calls video.play() on your neighbour, which… you get it.</p><p>Plus a second problem: someone has a slower connection, buffers longer, and ten minutes in the group has drifted apart by half a minute.</p><p>What I did:</p><p><strong>The server is the source of truth.</strong> A client doesn’t command other clients, it reports a fact to the server: “I pressed play at 12.5 seconds.” The server stores the position <strong>together with the moment in time</strong>:</p><pre>@dataclass<br>class Playback:<br>    playing: bool = False<br>    position: float = 0.0<br>    updated_at: float = field(default_factory=time.time)</pre><pre>    def current(self, *, at: float | None = None) -&gt; float:<br>        &quot;&quot;&quot;The position *now*, corrected for elapsed time.&quot;&quot;&quot;<br>        if not self.playing:<br>            return self.position<br>        now = at if at is not None else time.time()<br>        return max(0.0, self.position + (now - self.updated_at))</pre><p>Because of that, a latecomer joining mid-episode lands on the right second by themselves — the server computes it at the moment they ask.</p><p><strong>An event suppressor.</strong> Before applying somebody else’s action, the client opens a short window in which its own events are ignored:</p><pre>const suppress = (ms = 600) =&gt; { state.suppressUntil = performance.now() + ms; };<br>const suppressed = () =&gt; performance.now() &lt; state.suppressUntil;</pre><pre>video.addEventListener(&#39;play&#39;, () =&gt; {<br>  if (suppressed()) return;<br>  send({ type: &#39;play&#39;, position: video.currentTime });<br>});</pre><p>The echo dies, the loop never starts. The initiator gets a short ack rather than the broadcast — there&#39;s nothing for them to apply.</p><p><strong>Drift correction.</strong> Every 8 seconds the server broadcasts a reference time. A client only pulls itself into line if it has drifted more than ROOM_SYNC_TOLERANCE (1.5 seconds by default) — otherwise the player would twitch constantly.</p><p><strong>Clock offset.</strong> Client clocks lie. I estimate the offset from ping/pong at half the round trip:</p><pre>function applyPong(message) {<br>  const now = Date.now() / 1000;<br>  const roundTrip = now - Number(message.client_time);<br>  state.clockOffset = Number(message.server_time) + roundTrip / 2 - now;<br>}</pre><p>Measured across two tabs in a real browser: 253.2 and 253.3 seconds. A 0.1 s gap — well inside what a person doesn’t notice.</p><p>The general lesson here is the one I’d keep: <strong>synchronisation is about time, not events.</strong> As soon as the server started storing a position together with the instant it was true, everything else fell out on its own.</p><h3>Problem 3: hls.js does nothing, silently</h3><p>This debugging session cost me the most, so I’ll spell it out — it might save somebody an evening.</p><p>Symptom: the page opens, the player is there, the poster shows, and there are <strong>zero</strong> errors in the console. The video doesn’t load. video.readyState === 0, the buffer is empty.</p><p>Going deeper. hls.js is attached and the manifest is parsed:</p><pre>[[&quot;manifest&quot;, 1], [&quot;level&quot;, 252, false]]</pre><p>252 fragments, VOD. So the playlist was read and understood. But FRAG_LOADED never fires, and the server logs show not a single request for a segment.</p><p>Internal state:</p><pre>{&quot;state&quot;: &quot;IDLE&quot;, &quot;started&quot;: true, &quot;media&quot;: true}</pre><p>Sitting in IDLE. A manual hls.startLoad(0) changes nothing. Still no errors.</p><p>Into the buffer controller:</p><pre>{&quot;ms&quot;: &quot;closed&quot;, &quot;sbKeys&quot;: [], &quot;mediaReady&quot;: 0}</pre><p>There it is. <strong>MediaSource is </strong><strong>closed.</strong> It never opened, so there&#39;s nothing to create a SourceBuffer in, so there&#39;s no reason to fetch segments, so: IDLE. And all of it without a single error, because from hls.js&#39;s point of view nothing bad has happened — it&#39;s just waiting.</p><p>There were two causes, and both look harmless.</p><p><strong>1. </strong><strong>preload=&quot;none&quot; on the </strong><strong>&lt;video&gt; element.</strong> I&#39;d set it to avoid fetching anything before the user pressed play. With MSE that is fatal: with preload=&quot;none&quot;, Chrome doesn&#39;t begin loading the source at all, so the blob URL from MediaSource never opens. Fix: preload=&quot;metadata&quot;.</p><p><strong>2. Call order.</strong> I had:</p><pre>hls.loadSource(stream.url);<br>hls.attachMedia(video);</pre><p>The hls.js docs permit either order. In practice — here are two instances on the same playlist in the same tab:</p><pre>// loadSource → attachMedia<br>{&quot;log&quot;: [[&quot;manifest&quot;,1], [&quot;level&quot;,252,false]], &quot;ms&quot;: &quot;closed&quot;, &quot;ready&quot;: 0}</pre><pre>// attachMedia → loadSource<br>{&quot;log&quot;: [[&quot;attached&quot;], [&quot;frag&quot;,1], [&quot;bufcreated&quot;], [&quot;frag&quot;,2], [&quot;frag&quot;,3]], &quot;ms&quot;: &quot;open&quot;, &quot;ready&quot;: 4}</pre><p>The only difference is the order of two lines. The final version:</p><pre>// attachMedia first, then loadSource: in the other order Chrome doesn&#39;t get<br>// MediaSource open in time and hls.js sits silently in IDLE.<br>hls.attachMedia(video);<br>hls.loadSource(stream.url);</pre><p>The moral: if hls.js “does nothing” and doesn’t complain, go and read hls.bufferController.mediaSource.readyState. closed means the culprit isn&#39;t hls.js — it&#39;s the &lt;video&gt; element that never started loading.</p><h3>Problem 4: small traps that cost real time</h3><p>Three short ones, each instructive in its own way.</p><h3>display beats the hidden attribute</h3><p>The overlay above the player was marked up like this:</p><pre>&lt;div class=&quot;player-overlay&quot; data-overlay hidden&gt;</pre><p>and styled like this:</p><pre>.player-overlay { position: absolute; inset: 0; display: grid; }</pre><p>The overlay sat on top of the video permanently. Because hidden is nothing more than display: none from the user-agent stylesheet, and any display of your own overrides it. Meanwhile the attribute is honestly there in the DOM, el.hidden === true, and you cannot find the bug by looking.</p><p>One line at the top of your styles fixes it:</p><pre>[hidden] { display: none !important; }</pre><h3>lstrip(&quot;/&quot;) eats an absolute path</h3><p>The container died on startup:</p><pre>PermissionError: [Errno 13] Permission denied: &#39;data&#39;</pre><p>The culprit:</p><pre>path = urlsplit(database_url).path.lstrip(&quot;/&quot;)</pre><p>In a SQLAlchemy URL, exactly one slash separates the scheme from the path:</p><ul><li>sqlite:///./data/x.db → path /./data/x.db → relative</li><li>sqlite:////data/x.db → path //data/x.db → <strong>absolute</strong> /data/x.db</li></ul><p>lstrip(&quot;/&quot;) removes <em>all</em> leading slashes and turns the absolute path into a relative one. In a container that meant trying to create a data directory inside /app, where a non-root user cannot write. The correct thing is to remove exactly one character:</p><pre>return Path(raw[1:])</pre><h3>/anime/103 is not anime #103</h3><p>AnimeGO identifiers look like naruto-uragannye-hroniki-103. I decided the trailing number was enough and started requesting /anime/103.</p><p>The site answered 200. And served <strong>page 103 of the catalogue</strong>. Which is why a room would open with the title “Anime list — watch online … page 103”.</p><p>The funny part is that the tests didn’t catch it: the catalogue stub had its own identifier and returned the right title. Only a live run found it.</p><p>The lesson is banal and regularly forgotten: <strong>if a service doesn’t 404 on a meaningless request, that doesn’t make the request meaningful.</strong></p><h3>Discord login with a guild check</h3><p>Since the site is for a specific group, login goes through Discord and only admits members of one guild.</p><p>There’s a non-obvious point about scope. Most examples online suggest requesting guilds and checking whether the server is in the list. That works, but the guild list is a lot of extra information about a person, and roles aren&#39;t visible in it.</p><p>Better: guilds.members.read and one targeted request:</p><pre>response = await self._client.get(<br>    f&quot;{self.settings.discord_api_base}/users/@me/guilds/{guild_id}/member&quot;,<br>    headers={&quot;Authorization&quot;: f&quot;Bearer {access_token}&quot;},<br>)</pre><p>One request gives you membership, roles and the server nickname. 404 means not a member. 401/403 means the user didn&#39;t grant the scope, and then it&#39;s worth falling back to the guild list.</p><p>A third mode is checking via a bot that’s in the guild itself. Then the user only needs identify, and roles are always visible — useful if you want to admit holders of a specific role rather than all members.</p><p>All three modes are one setting, and authentication as a whole is disabled by another — the site then runs in guest mode where a name is enough. That turned out to be convenient: local development needs no OAuth fiddling at all.</p><p>The token is revoked after the check — the site has its own session by then, and there’s no reason to keep somebody else’s:</p><pre>finally:<br>    if access_token:<br>        await discord.revoke(access_token)</pre><h3>Other things that mattered</h3><p><strong>Rooms live in memory.</strong> A deliberate decision: room state is playback, viewers and chat, all ephemeral. The price is that the app runs strictly in one worker. Two workers would mean viewers landing in different copies of the same room. It scales horizontally with sticky sessions keyed on the room code; I didn’t build proper sharing over Redis pub/sub.</p><p><strong>The library is synchronous.</strong> anime-dl-core is built on requests, so all its calls go to a thread pool via anyio.to_thread.run_sync, and results land in a TTL cache with stampede protection — so five people opening the same page at once don&#39;t produce five identical requests to the source.</p><p><strong>The link cache has to be short.</strong> Episode lists are cached for 10 minutes, direct links for 4. The temptation to raise the second one is strong, but the links expire and the user gets a 403 instead of video.</p><p><strong>Tracking fills itself in.</strong> An episode is marked watched after 85%, and a title is added to your list on first view. Nothing to press — which is exactly why people abandon trackers.</p><h3>Infrastructure</h3><p>Multi-stage Dockerfile, non-root user, healthcheck, a 361 MB final image. Three compose files: SQLite by default, a PostgreSQL overlay, and a dev overlay with auto-reload. The entrypoint waits for the database and applies migrations.</p><p>CI: ruff, mypy, pytest on Python 3.11, 3.12 and 3.13, alembic check against a real PostgreSQL (which catches models drifting from migrations), and an image build with a /healthz smoke test inside the running container. 90 tests, none of which touch the network — sources are stubbed, the database is in-memory.</p><p>One detail for anyone developing on Windows and deploying to Linux: a .gitattributes with *.sh text eol=lf is mandatory. Otherwise checkout turns the entrypoint into CRLF and the container greets you with a cryptic bad interpreter.</p><p>And a small surprise at the end. While the repository was private, the image publishing workflow failed on the provenance attestation step:</p><pre>Failed to persist attestation: Feature not available for user-owned private repositories.</pre><p>The image built and pushed fine — only the last step was red. It turns out artifact attestations aren’t available for private repositories on personal accounts: you need a public repository or an organisation.</p><p>Fixed with a condition, after which the workflow behaves correctly in both states — quiet when private, signing when public:</p><pre>- name: Attest provenance<br>  if: github.event.repository.visibility == &#39;public&#39;<br>  uses: actions/attest-build-provenance@v2</pre><p>A small thing, but a telling one: “green CI” and “CI that does what you think it does” are not the same. Until you read the log of the failed step, you don’t learn that the image was actually published.</p><h3>What I took away</h3><ul><li><strong>“Just use the link” is almost never just.</strong> Referer, IP binding, expiry — three independent reasons, any one of which sinks the naive plan.</li><li><strong>A silent library is worse than a noisy one.</strong> hls.js produced not one error while sitting in IDLE. Only reading its internal state saved me.</li><li><strong>A live run catches what tests don’t.</strong> Three of the four bugs above could not reproduce against stubs at all.</li><li><strong>Synchronisation is about time, not events.</strong> Once the server stored a position together with its timestamp, the rest fell into place.</li></ul><p>The project is FastAPI + SQLAlchemy 2 + Jinja2 and vanilla JS with no bundler. The frontend is deliberately framework-free: six pages, not much interactivity, and skipping the npm stage simplifies both Docker and CI enormously.</p><p>If you build something similar — start with the proxy. It’s the most underestimated part of the job and the one without which nothing moves at all.</p><p>Sources: <a href="https://github.com/ialakey/anime-watch-together"><strong>github.com/ialakey/anime-watch-together</strong></a></p><p><strong>Has anyone got an elegant way to share room state across workers without dragging in Redis?</strong> I read the responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d480fca7805" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Before You Fix the Parser, Check You're Still on the Right Website]]></title>
            <link>https://medium.com/@alakov.ilia/before-you-fix-the-parser-check-youre-still-on-the-right-website-c229dc51ba95?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c229dc51ba95</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[reverse-engineering]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 13:01:04 GMT</pubDate>
            <atom:updated>2026-10-06T13:01:04.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ICgfeMiOn1cKjtBTOQElTg.png" /></figure><p>I took a popular open-source library of video-player extractors, ran it, and got 500 Internal Server Error from Kodik. Not &quot;token expired&quot;, not &quot;IP blocked&quot; — a 500 on every single request.</p><p>The investigation took an evening and the cause was one character: the library was sending the ref parameter empty, and the server requires it filled in <strong>and percent-decoded</strong>.</p><p>Then it turned out that wasn&#39;t a one-off breakage, it was the normal condition of the genre. One player had moved to a different page format six months earlier. Another answers with an infinite redirect to itself. A third had its <strong>domain taken over</strong> — there&#39;s an Indonesian online store there now, not video. A fourth hides its links in a WebSocket assembled inside a 600 KB obfuscated bundle, and I couldn&#39;t crack that one.</p><p>Here&#39;s the thesis, and it&#39;s the reason this write-up isn&#39;t just a list of fixes:</p><blockquote><em>A parser doesn&#39;t rot. Its assumptions expire. And the assumption most likely to have expired isn&#39;t in the HTML — it&#39;s about who owns the domain.</em></blockquote><p>Below: eight players, how each one serves video, where the existing extractors break, and what follows for anyone writing something similar. The library came out of it as a by-product and it&#39;s public: <a href="https://github.com/ialakey/anime-dl-core"><strong>github.com/ialakey/anime-dl-core</strong></a>, MIT.</p><pre>import anime_dl_core as ap<br><br>result = ap.extract(&quot;https://aniboom.one/embed/9G1MJ6NMV8z?episode=1&amp;translation=30&quot;)<br><br>print(result.qualities)                   # [360, 480, 720, 1080]<br>stream = result.best(kind=&quot;hls&quot;)<br>print(stream.url)                         # https://.../media_6.m3u8<br>print(stream.headers)                     # {&#39;Referer&#39;: &#39;https://aniboom.one/&#39;, ...}<br>print(stream.ffmpeg_command(&quot;ep1.mp4&quot;))   # a ready-to-run ffmpeg command</pre><p>Eight players, one dependency (requests), Python 3.9+, 86 offline tests against real saved responses plus 18 live ones.</p><p>The library deliberately does one thing: players. It doesn&#39;t search for titles, doesn&#39;t maintain a catalogue, doesn&#39;t download anything itself and doesn&#39;t bypass paid restrictions — it reads exactly what an ordinary web player reads in a browser.</p><h3>Why bother</h3><p>These sites are all built the same way: the site itself is a storefront — description, ratings, episode list — and the video lives in somebody else&#39;s player embedded in an &lt;iframe&gt;. The same title on different sites runs through the same four or five players.</p><p>So if you want your own client — a TV, a downloader on a NAS, offline for a train journey, a screen without ads — the problem isn&#39;t &quot;scrape the site&quot;, it&#39;s &quot;understand the player&quot;. There&#39;s a finite number of players, and unlike storefronts, they last for years.</p><p>One at a time.</p><h3>Kodik: three steps, a Caesar cipher, and that empty parameter</h3><p>Kodik is the most widespread player here and the best protected of the ones I met. Getting a link out of it takes three requests.</p><p><strong>Step one.</strong> The embed page (https://kodikplayer.com/seria/&lt;id&gt;/&lt;hash&gt;/720p) contains a block of signatures:</p><pre>var urlParams = &#39;{&quot;d&quot;:&quot;animego.org&quot;,<br>                  &quot;d_sign&quot;:&quot;45f22a7face01f12...:2609080656&quot;,<br>                  &quot;pd&quot;:&quot;kodikplayer.com&quot;,<br>                  &quot;pd_sign&quot;:&quot;31aa713fde164488...:2609080656&quot;,<br>                  &quot;ref&quot;:&quot;https%3A%2F%2Fanimego.org%2F&quot;,<br>                  &quot;ref_sign&quot;:&quot;f99556604d27d0ff...:2609080656&quot;}&#39;;<br>var vInfo = {};<br>vInfo.type = &#39;seria&#39;;<br>vInfo.hash = &#39;932d5da818729ec5ccc9be7968ee3717&#39;;<br>vInfo.id   = &#39;1304528&#39;;</pre><p>The signatures are single-use and timestamped — the page fetch and the next request have to happen back to back.</p><p><strong>Step two.</strong> The endpoint&#39;s address is hidden in the player script:</p><pre>$.ajax({type:&quot;POST&quot;, url:atob(&quot;L2Z0b3I=&quot;), cache:!1, dataType:&quot;json&quot;, data:e, ...})</pre><p>atob(&quot;L2Z0b3I=&quot;) → /ftor. Obfuscation at the &quot;so it isn&#39;t googleable&quot; level — but you have to find it with a regex over the atob call, not by offset in the file. The original library cut the address out as js[js.find(&quot;$.ajax&quot;)+30 : js.find(&quot;cache:!1&quot;)-3], which any minifier change breaks.</p><p><strong>Step three — and here&#39;s where it was broken.</strong> A POST to /ftor with the signatures. The original library sent:</p><pre>params = {..., &#39;ref&#39;: &#39;&#39;, &#39;ref_sign&#39;: urlParams[&#39;ref_sign&#39;], ...}</pre><p>An empty ref, and the server answers 500. The correct version:</p><pre>from urllib.parse import unquote<br>params = {..., &#39;ref&#39;: unquote(url_params[&#39;ref&#39;]),   # &#39;https://animego.org/&#39;<br>               &#39;ref_sign&#39;: url_params[&#39;ref_sign&#39;], ...}</pre><p>ref has to be taken from urlParams <strong>and decoded</strong> from percent-encoding: ref_sign is computed over the decoded value. One line, and instead of a 500 you get JSON with links.</p><p>Encrypted links, that is:</p><pre>Tg9rjO91HK5hj2fdHOVsjq5rj20dlFVtkvDejO9pHPUdVLQeUhIgV2GbGENtUg00HODqTERtU2Qb...</pre><p>That&#39;s base64 with a Caesar shift applied over the Latin alphabet. The shift changes — at the time I looked into it, it was 18. You find it by trying all 26 and checking the result:</p><pre>def decode_kodik_url(https://medium.com/value,%20known_shift=None):<br>    for shift in ([known_shift] if known_shift is not None else []) + list(range(26)):<br>        try:<br>            decoded = b64decode_padded(caesar_shift(value, shift)).decode(&quot;utf-8&quot;)<br>        except Exception:<br>            continue<br>        if decoded.startswith(&quot;//&quot;) or decoded.startswith(&quot;http&quot;):<br>            return decoded, shift<br>    raise DecryptionError(...)</pre><p>The discovered shift is cached on the player object, so the brute force happens once per session. What comes out:</p><pre>//cloud.solodcdn.com/useruploads/4203027f-.../858b661430c3...:2026090706/720.mp4:hls:manifest.m3u8</pre><p>And the direct mp4 is right there — chop off :hls:manifest.m3u8. Verified: it serves an honest 202 MB file.</p><p>As a bonus, the page carries the opening and ending timecodes:</p><pre>playerSettings.skipButton = parseSkipButton(&quot;0:30-1:50,22:55-24:05&quot;, &quot;anime&quot;);</pre><p><strong>Worth knowing:</strong> none of this needs a Kodik API token. The token is only for searching Kodik&#39;s catalogue; extracting video from an embed link you already have doesn&#39;t need one.</p><h3>Aniboom: everything in one attribute</h3><p>After Kodik, a holiday. The embed page contains:</p><pre>&lt;video id=&quot;video&quot; data-parameters=&quot;{&amp;quot;id&amp;quot;:&amp;quot;QK8d1LbNX6l&amp;quot;,<br>  &amp;quot;dash&amp;quot;:&amp;quot;{\&amp;quot;src\&amp;quot;:\&amp;quot;https:\/\/.../o9hwpavtcccz7.mpd\&amp;quot;}&amp;quot;,<br>  &amp;quot;hls&amp;quot;:&amp;quot;{\&amp;quot;src\&amp;quot;:\&amp;quot;https:\/\/.../master.m3u8\&amp;quot;}&amp;quot;,<br>  &amp;quot;duration&amp;quot;:&amp;quot;2971&amp;quot;, &amp;quot;qualityVideo&amp;quot;:1080, ...}&quot;&gt;</pre><p>Unescape the HTML entities, parse the JSON, and inside hls and dash find JSON strings again. Then a master playlist, which I expand into individual qualities:</p><pre>#EXT-X-STREAM-INF:BANDWIDTH=593867,RESOLUTION=640x360,CODECS=&quot;avc1.640028,mp4a.40.2&quot;<br>media_0.m3u8<br>#EXT-X-STREAM-INF:BANDWIDTH=1593867,RESOLUTION=1280x720,...<br>media_4.m3u8</pre><p>The one subtlety is headers: without Referer: https://aniboom.one/ and Origin, the CDN refuses. That&#39;s common to every player here, and it&#39;s why in my library the headers aren&#39;t an implementation detail — they are part of the result:</p><pre>stream.headers                 # {&#39;Referer&#39;: ..., &#39;Origin&#39;: ..., &#39;User-Agent&#39;: ...}<br>stream.ffmpeg_args(&quot;ep.mp4&quot;)   # already includes -headers</pre><h3>CVH: an open API and keys you&#39;d never guess</h3><p>CVH (CdnVideoHub) — two open endpoints, no auth at all:</p><pre>GET /api/v1/player/sv/playlist?pub=747&amp;aggr=mali&amp;id=51019   → all episodes and dubs<br>GET /api/v1/player/sv/video/&lt;vkId&gt;                          → stream links</pre><p>The playlist is pleasantly structured: each entry is {season, episode, voiceStudio, voiceType, vkId}, so you can see straight away which episode has which dubs.</p><p>The response with the links, though, holds a surprise:</p><pre>&quot;sources&quot;: {<br>  &quot;hlsUrl&quot;: &quot;https://vd576.okcdn.ru/video.m3u8?...&quot;,<br>  &quot;dashUrl&quot;: &quot;https://vd576.okcdn.ru/?...&quot;,<br>  &quot;mpegHighUrl&quot;: &quot;...&quot;, &quot;mpegFullHdUrl&quot;: &quot;...&quot;, &quot;mpegMediumUrl&quot;: &quot;...&quot;,<br>  &quot;mpegLowUrl&quot;: &quot;...&quot;, &quot;mpegLowestUrl&quot;: &quot;...&quot;, &quot;mpegTinyUrl&quot;: &quot;...&quot;<br>}</pre><p>Video is served by Odnoklassniki&#39;s CDN, hence the naming. The mapping: mpegTinyUrl is 144p, mpegLowestUrl 240p, mpegLowUrl 360p, mpegMediumUrl 480p, <strong>mpegHighUrl 720p</strong>, and <strong>mpegFullHdUrl 1080p</strong>.</p><p>The original library looked for keys starting with url (url360, url720 — the <em>old</em> response format), and therefore lost <strong>every</strong> direct mp4, keeping only HLS. A textbook silent breakage: no error, data present, half the result missing.</p><h3>Sibnet: one line and a mandatory Referer</h3><pre>player.src([{src: &quot;/v/2305c05076f006222ee9e551c08c7e03/2589828.mp4&quot;, type: &quot;video/mp4&quot;}]);</pre><p>A relative path → https://video.sibnet.ru/v/...mp4, then a 302 to the real CDN:</p><pre>Location: //dv97.sibnet.ru/25/89/82/2589828.mp4?st=I3P2Jhxp...&amp;e=1788750000&amp;noip=1</pre><p>The signed address is short-lived and pinned to an IP, so by default I return the stable video.sibnet.ru/v/... link, which redirects on its own, and made redirect resolution an option.</p><h3>VK Video: the page moved, and the parsers don&#39;t know</h3><p>Every VK parser I know of looks for this on the video_ext.php page:</p><pre>var playerParams = {&quot;params&quot;:[{&quot;url240&quot;:&quot;...&quot;,&quot;url720&quot;:&quot;...&quot;,&quot;hls&quot;:&quot;...&quot;}]};</pre><p>It isn&#39;t there any more. The data now lives in a prefetched internal API response:</p><pre>window.cur = Object.assign(window.cur || {}, {<br>  &quot;apiPrefetchCache&quot;:[{<br>    &quot;method&quot;:&quot;video.get&quot;,<br>    &quot;response&quot;:{&quot;items&quot;:[{<br>      &quot;files&quot;:{<br>        &quot;mp4_144&quot;:&quot;https://vkvd419.okcdn.ru/?...&quot;,<br>        &quot;mp4_720&quot;:&quot;https://vkvd419.okcdn.ru/?...&quot;,<br>        &quot;hls_ondemand&quot;:&quot;https://vkvd419.okcdn.ru/.../hls4_....m3u8&quot;,<br>        &quot;dash_ondemand&quot;:&quot;https://vkvd419.okcdn.ru/.../dash4_....mpd&quot;<br>      },<br>      &quot;title&quot;:&quot;...&quot;, &quot;duration&quot;:1440, &quot;image&quot;:[...]<br>    }]}<br>  }]<br>});</pre><p>I found it the dumb way, which I recommend: save the page to a file and count which keys sit next to https:// at all.</p><pre>keys = re.findall(r&#39;&quot;([a-zA-Z0-9_]+)&quot;:&quot;https:\\/\\/[^&quot;]{20,}&quot;&#39;, text)<br>print(Counter(keys).most_common(30))</pre><p>Pulling JSON out of the middle of a page is neatest with raw_decode — it reads exactly one object and tells you where it stopped:</p><pre>start = text.find(&quot;[&quot;, text.find(&#39;&quot;apiPrefetchCache&quot;&#39;))<br>entries, _ = json.JSONDecoder().raw_decode(text[start:])</pre><p>Useful details: the hash from the embed code is only needed for &quot;available by link&quot; videos — public ones need just oid and id; a live stream has only hls in files, which is how you detect live; and the video page (vkvideo.ru/video-…) prefetches nothing useful — it&#39;s video_ext.php that works. I kept the old format as a fallback, because mirrors still serve it.</p><h3>SovetRomantica: the site is gone and the domain changed hands</h3><p>This is where it becomes a detective story. The SovetRomantica player lives at sovetromantica.com/embed/episode_&lt;id&gt;_&lt;episode&gt;-&lt;dubbed|subtitles&gt;. I request it and get back 394 KB of HTML containing no anime and no player — but a children&#39;s goods store, in Indonesian.</p><p>First thought: DNS interception by my ISP. So I check the certificate:</p><pre>cert = ssl_socket.getpeercert()<br># subject: www.qris.sovetromantica.com<br># SAN: *.sovetromantica.com, sovetromantica.com,<br>#      www.panduan.sovetromantica.com, www.qris.sovetromantica.com</pre><p>The certificate is genuine — Let&#39;s Encrypt, issued for the domain, together with subdomains qris (an Indonesian payment system) and panduan (&quot;guide&quot;). This isn&#39;t interception: the domain simply <strong>changed owner</strong> and went into an SEO scheme. The team&#39;s CDN (scu*.sovetromantica.com) doesn&#39;t resolve at all.</p><p>You&#39;d think: cross the player off. But the markup is recoverable — the Wayback Machine&#39;s CDX API will list saved pages by pattern:</p><pre>requests.get(&quot;https://web.archive.org/cdx/search/cdx&quot;, params={<br>    &quot;url&quot;: &quot;sovetromantica.com/embed/*&quot;, &quot;output&quot;: &quot;json&quot;,<br>    &quot;filter&quot;: &quot;statuscode:200&quot;, &quot;from&quot;: &quot;2023&quot;, &quot;collapse&quot;: &quot;urlkey&quot;})</pre><p>Eighty-five saved embed pages. Take a recent one (the id_ suffix in the archive URL returns the original without the archive&#39;s own wrapper) and there it is, the real markup:</p><pre>var nextEpisode = &#39;https://sovetromantica.com/embed/episode_1073_2-dubbed&#39;;<br>var skips = [<br>  {&#39;start&#39;: 174.7996, &#39;end&#39;: 184.7996, &#39;skip_to&#39;: 262.9293},<br>  {&#39;start&#39;: 1316.7738, &#39;end&#39;: 1326.7738, &#39;skip_to&#39;: 1398.5638},<br>];<br>var config = {<br>  &quot;id&quot;:&quot;sovetromantica_player&quot;,<br>  &quot;file&quot;:&quot;https://scu2.sovetromantica.com/anime/1073_.../episodes/dubbed/episode_1/episode_1.m3u8&quot;,<br>  &quot;poster&quot;:&quot;...&quot;, &quot;thumbnails&quot;:&quot;....vtt&quot;,<br>  &quot;title&quot;:&quot;...&quot;, &quot;points&quot;: points<br>};</pre><p>Note config: it is <strong>not valid JSON</strong> — it ends with a JS variable, points. So the fields come out one at a time with regexes, not with a single json.loads.</p><p>skips has a non-obvious structure too: start–end is when to show the &quot;skip&quot; button, and skip_to is where it jumps. So the skipped segment is start → skip_to, and that&#39;s how I record it.</p><p>The player is in the library, its parsing verified against real archived pages, and the domain is a parameter:</p><pre>with SovetRomanticaPlayer(base_url=&quot;https://new-domain&quot;) as player:<br>    result = player.extract(&quot;episode_1073_1-dubbed&quot;)</pre><p>When the team comes back, it works with no code change. Their new releases currently go out on VK, which the vk player handles.</p><p><strong>The moral, which cost me an evening:</strong> before fixing a parser, check you&#39;re on the right site at all. Two hundred lines of regexes can &quot;stop working&quot; simply because there&#39;s a different person behind the domain.</p><h3>Animedia: an infinite redirect and a relocation</h3><p>Links like online.animedia.tv/embed/14938/1/1 are still handed out by aggregators. I request one and requests dies with TooManyRedirects. Looking without auto-follow:</p><pre>GET https://online.animedia.tv/embed/14938/1/1<br>301 → https://online.animedia.tv/embed/14938/1/1</pre><p>A redirect to itself. Neither cookies nor User-Agent help, and it&#39;s identical from another network — so it&#39;s broken for everybody, not just me.</p><p>The live site turned up by guessing domains: animedia.online redirects to <strong>amd.online</strong>, which is a working &quot;Animedia Online&quot;. The player there is now their own, at aser.pro/vod/&lt;id&gt;, and everything is on one line:</p><pre>var player = new Playerjs({<br>  id: &quot;videoplayer20182&quot;,<br>  file: &quot;https://aser.pro/content/stream/.../hls/index.m3u8&quot;<br>});</pre><p>Playerjs is a popular player and its file parameter comes in three shapes: a single link, [720]url1,[360]url2, and a JSON playlist of episodes. I support all three, because it costs ten lines and breaks silently otherwise.</p><p>Episode links live on the title page, one per episode:</p><pre>&lt;a data-vid=&quot;1&quot; data-vlnk=&quot;https://aser.pro/vod/1067&quot; href=&quot;...&quot;&gt;1&lt;/a&gt;<br>&lt;a data-vid=&quot;2&quot; data-vlnk=&quot;https://aser.pro/vod/1068&quot; href=&quot;...&quot;&gt;2&lt;/a&gt;</pre><h3>Alloha: where I stopped</h3><p>Alloha is a large aggregator with an open API. By Kinopoisk/IMDb/TMDb id or by title, it returns everything you need:</p><pre>{&quot;status&quot;:&quot;success&quot;,&quot;data&quot;:{<br>  &quot;name&quot;:&quot;Attack on Titan&quot;, &quot;year&quot;:2013, &quot;id_kp&quot;:749374,<br>  &quot;seasons&quot;:{&quot;1&quot;:{&quot;episodes&quot;:{&quot;1&quot;:{&quot;iframe&quot;:&quot;https://...&quot;,&quot;translation&quot;:{...}}}}},<br>  &quot;iframe&quot;:&quot;https://miyagi-as.stravers.live/?token_movie=...&amp;token=...&quot;}}</pre><p>I open the iframe: the HTML has a fileList with file ids, qualities and dubs — and <strong>not a single video link</strong>. Searching further: no m3u8 in the HTML; /api/..., /playlist?..., /get?... all 404; and in the player bundle (600 KB, obfuscated with a string array and hex constants) this shows up:</p><pre>window.__ws_factory(&#39;&#39;.concat(aS, ...).concat(encodeURIComponent(aO), &#39;&amp;v=2.1&amp;t=&#39;).concat(Date.now()))</pre><p>The links arrive over a <strong>WebSocket</strong> whose address is assembled inside obfuscated code. That isn&#39;t &quot;hard to parse&quot;, that&#39;s &quot;you need to execute their JS&quot;. A headless browser would solve it and would also add Chromium to a library with one dependency.</p><p>So I stopped, and wrote that in the README in plain words. Alloha is in the library as a <strong>source</strong>, not a player: it returns metadata, the list of seasons and dubs, and a ready iframe URL to embed or open in a webview.</p><p>Worth saying why I think that&#39;s the right call. The temptation was to write &quot;supported&quot; and hand back an iframe disguised as a stream. But the library has one contract: Stream.url is something you can feed to ffmpeg. A line in the README saying &quot;I couldn&#39;t, here&#39;s why&quot; is more honest than a method that returns something other than what it promises.</p><p>One more that didn&#39;t make it: jut.su. Its &lt;source&gt; elements now contain placeholders —</p><pre>&lt;source src=&quot;https://gen.jut.su/templates/school/images/pixel.png?1080&quot; res=&quot;1080&quot;/&gt;</pre><p>— with the real links loaded dynamically. Same story: no way around executing JS.</p><h3>What each player actually protects</h3><p><strong>Aniboom</strong> — How it serves the link: JSON in a tag attribute; Barrier: Referer/Origin</p><p><strong>CVH</strong> — How it serves the link: open API; Barrier: none</p><p><strong>Kodik</strong> — How it serves the link: POST with single-use signatures; Barrier: base64 + Caesar, signatures, ref</p><p><strong>Sibnet</strong> — How it serves the link: JS string on the page; Barrier: Referer, signed redirect</p><p><strong>Animedia</strong> — How it serves the link: new Playerjs({file}); Barrier: none</p><p><strong>AniLibria</strong> — How it serves the link: open API; Barrier: none</p><p><strong>VK</strong> — How it serves the link: prefetched API response in HTML; Barrier: the format changes</p><p><strong>Alloha</strong> — How it serves the link: WebSocket from an obfuscated bundle; Barrier: <strong>not passed</strong></p><p>The general pattern: the bigger the service, the less it relies on obfuscation and the more on signatures, short lifetimes and IP binding. Kodik encrypts its link with a Caesar cipher — which isn&#39;t protection, it&#39;s a &quot;do not enter&quot; sign — but the signatures in urlParams live for minutes, and <em>that</em> is real protection, because it makes caching pointless.</p><p>Three conclusions for anyone writing this kind of thing:</p><ul><li><strong>Headers are part of the result.</strong> A link without its Referer isn&#39;t a link. Return them with the URL, or the user of your library will collect 403s and assume you broke something.</li><li><strong>Links live for minutes.</strong> Caching them is pointless; fetch them immediately before downloading.</li><li><strong>An error must name what wasn&#39;t found.</strong> &quot;Couldn&#39;t find urlParams on the Kodik page&quot; is fixed in a minute. AttributeError: &#39;NoneType&#39; object has no attribute &#39;group&#39; takes an evening.</li></ul><p>That last one connects to something I&#39;ve written about before: <a href="https://medium.com/@alakov.ilia/a-scraper-doesnt-crash-it-returns-a-file-full-of-confident-nonsense-df10c9b0ea8d">a scraper doesn&#39;t crash</a>, it returns valid, plausible, wrong data. The CVH case above is exactly that failure mode — no exception, a populated result, and half the streams missing.</p><h3>How it&#39;s put together</h3><p>The library is built around a player registry: extract() identifies the player by the link&#39;s domain and returns one unified result.</p><pre>src/anime_dl_core/<br>    registry.py   the registry, extract() / extract_async()<br>    base.py       BasePlayer: clients, URL matching, lifecycle<br>    models.py     Stream, PlayerResult, SkipSegment, StreamKind<br>    http.py       HttpClient (requests) and AsyncHttpClient (aiohttp) with a shared Response<br>    utils.py      html/js parsing, m3u8, Kodik link decryption<br>    players/      one file per player<br>    sources/      optional helpers (AnimeGO, Animedia, Alloha)</pre><p>A new player is about twenty lines:</p><pre>@register<br>class MyPlayer(BasePlayer):<br>    name = &quot;myplayer&quot;<br>    domains = (&quot;myplayer.example&quot;,)<br>    playback_headers = {&quot;Referer&quot;: &quot;https://myplayer.example/&quot;}<br><br>    def extract(self, url, **kwargs):<br>        page = self.client.get(url).raise_for_status()<br>        playlist = search(r&#39;file:&quot;([^&quot;]+\.m3u8)&quot;&#39;, page.text, what=&quot;the m3u8 link&quot;)<br>        return PlayerResult(player=self.name, source_url=url,<br>                            streams=[Stream(playlist, StreamKind.HLS, None,<br>                                            dict(self.playback_headers))])</pre><p>The sync and async modes perform the same steps and call the same parsing functions, so their behaviour can&#39;t diverge. HTML is parsed with regexes: for a dozen specific pages that&#39;s more reliable than depending on bs4/lxml, and it leaves the library with exactly one dependency.</p><h3>Tests: fixtures from real responses</h3><p>Eighty-six offline tests run the parsing against <strong>real</strong> player responses saved in tests/fixtures/. Not invented ones — the actual pages I reverse-engineered, including a genuine encrypted Kodik response, so the decryption is tested for real rather than against a string I encrypted myself.</p><p>Eighteen live tests go out to the internet. <strong>They deliberately don&#39;t run in CI:</strong> a live test fails when a particular title is pulled from a site or a service is unreachable from a GitHub data centre — that&#39;s not a regression in the library, it&#39;s a red build for no reason. They run separately:</p><pre>ANIME_DL_CORE_LIVE=1 pytest -m live</pre><p>I&#39;d recommend that split to anyone writing parsers: offline tests answer &quot;did I break the parsing&quot;, live tests answer &quot;did the site break&quot;. Those are different questions, and mixing them in one run is a reliable way to train yourself to ignore a red CI.</p><h3>CI: SHA-256 in the release and trusted publishing</h3><p>Since the library is public, CI does releases on a tag: it builds the sdist and wheel, installs the built wheel into a clean environment (a package that only imports from source is a broken package), computes the SHA-256 of each file, and puts the hashes with the commit into the release notes.</p><p>Publishing to PyPI goes through Trusted Publishing: PyPI accepts a short-lived OIDC token issued to one specific repository and workflow. There is no long-lived secret at all, and the files carry attestations (PEP 740) proving the wheel was built by that workflow from that commit:</p><pre>{&quot;kind&quot;: &quot;GitHub&quot;, &quot;repository&quot;: &quot;ialakey/anime-dl-core&quot;,<br> &quot;workflow&quot;: &quot;publish-pypi.yml&quot;, &quot;environment&quot;: null}</pre><p>The rake I stepped on: a <em>pending publisher</em> on PyPI only works for projects that <strong>don&#39;t exist yet</strong>. If the project was already created by a token upload, the trusted publisher has to be added on the project&#39;s own page (Manage → Publishing), or the token exchange fails with invalid-publisher: valid token, but no corresponding publisher — with entirely correct claims, which is thoroughly confusing.</p><h3>What&#39;s next</h3><p>What&#39;s missing, and what I might do — or someone else might, it&#39;s MIT:</p><ul><li><strong>Alloha via a headless browser</strong>, as a separate optional package, so the core keeps its one dependency.</li><li><strong>More players.</strong> The format for adding one is a file, a dozen regexes and a fixture from a real response.</li><li><strong>Downloading.</strong> Right now the library hands back a link and an ffmpeg command; a downloader with resume and HLS joining is a separate layer.</li></ul><p>If you have a public example of a player I didn&#39;t support — or supported, but it doesn&#39;t work for you — that is the most useful feedback there is: it&#39;s almost always either a new page format or one more relocated domain.</p><p><em>The library doesn&#39;t bypass paid restrictions and doesn&#39;t break DRM: it reads exactly the data an ordinary web player is served in a browser. Responsibility for use lies with the user; respect copyright and site terms.</em></p><p>Code: <a href="https://github.com/ialakey/anime-dl-core"><strong>github.com/ialakey/anime-dl-core</strong></a> · Package: anime-dl-core on PyPI · MIT.</p><p><strong>What&#39;s the strangest reason one of your parsers &quot;stopped working&quot; that turned out not to be your code at all?</strong> I read the responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c229dc51ba95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Email Address Is in Your Own Commits]]></title>
            <link>https://medium.com/@alakov.ilia/your-email-address-is-in-your-own-commits-6a6418b4af0d?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a6418b4af0d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[job-search]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 13:01:02 GMT</pubDate>
            <atom:updated>2026-09-29T13:01:02.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0WFPnH1PQMZnpWPEc4hNfg.png" /></figure><p>A week ago I published <a href="https://medium.com/@alakov.ilia/i-dont-apply-to-jobs-i-email-the-humans-behind-them-so-i-built-something-that-finds-them-553543179399">a piece here</a> about <strong>resume2human</strong> — a tool that turns your CV into a list of matching job openings and, for each one, finds one to three actual humans with a name, a role and an address, so you can write to them instead of to a form.</p><p>Since then: 21 commits across nine pull requests, the test count went from 345 to 746, and I broke exactly half the promises I made in that article.</p><p>This is the follow-up. Three things in it are worth your time even if you never touch the tool: a free contact-discovery tier that runs on git instead of a rented database; a family of bugs where the program was honestly searching for something other than what you asked for; and a short lesson about why coverage of the empty set equals 1, and what that cost me.</p><p><strong>For anyone who missed </strong><a href="https://medium.com/@alakov.ilia/i-dont-apply-to-jobs-i-email-the-humans-behind-them-so-i-built-something-that-finds-them-553543179399"><strong>the first article</strong></a>, one paragraph. Applying through a form is a letter to an inbox read by a script; what works is a letter to a specific engineering manager or recruiter. The problem with direct letters is purely mechanical: finding a relevant opening takes half an hour, finding the human behind it takes another half hour, and by hand that doesn&#39;t scale. resume2human is a Windows app that takes your CV, walks 24 collectors (ATS boards for 111 companies, regional boards, LinkedIn, Telegram, Threads, web search), computes an explainable match between CV and opening, and brings back one to three people with contacts for each good match. No AI, no API keys, no automated sending. You write the letter.</p><h3>Half of my &quot;it will never do this&quot; list is now false</h3><p>The first article had a section titled <em>what this tool doesn&#39;t do — and won&#39;t</em>. Six days later:</p><p><strong>&quot;It doesn&#39;t send emails for you&quot; — still true.</strong> And it will stay true; the reason is at the bottom and hasn&#39;t changed.</p><p><strong>&quot;It doesn&#39;t generate addresses from a template&quot; — true, with an asterisk.</strong> A guessing tier now exists, but it&#39;s off by default and labelled separately.</p><p><strong>&quot;It doesn&#39;t track application status and isn&#39;t a CRM&quot; — broken.</strong> There&#39;s now a funnel, statuses, reminders, and six analytical views.</p><p><strong>&quot;It doesn&#39;t send your CV to any service&quot; — still true.</strong></p><p>About that third line, plainly: it was never a principled position. It was a boundary I drew because I hadn&#39;t got to it yet. It sat in the spec from day one and the applications table had been in the schema, empty, since the first commit. All it took was running my own tool a few times in a row to discover that almost everything missing in practice was the same hole: the program was a search engine that ended in a file, and it knew absolutely nothing about what happened to an opening afterwards.</p><p>The worse part is how I found out. Not by using it — by going to update the project&#39;s showcase page and noticing that the README promised readers something that was already untrue. Two PRs had passed between &quot;the tool doesn&#39;t do X&quot; and &quot;the tool does X&quot;, with zero documentation changes in between.</p><h3>The week&#39;s main event: the address is in the commits</h3><p>The first version of the contact ladder answered <em>who is hiring here</em>. On the next question — <em>how do I write to this person</em> — it stopped: the report contained a name, a role and a profile link you can&#39;t send anything to.</p><p>There&#39;s an entire industry that closes exactly that step: ContactOut, Lusha, Apollo, RocketReach — a rented database plus a subscription. I spent a day looking in that direction, then reframed the question:</p><blockquote><em>What share of these addresses did the person publish themselves, with their own hands, somewhere public?</em></blockquote><p>For engineers, the answer turned out to be indecently high. Here&#39;s why.</p><h3>Git is an open database of engineering contacts that everyone forgot about</h3><p>Every commit carries author.email. Not <em>may carry</em> — carries; the field is mandatory. A person can hide their address in their GitHub profile settings, but the commit they pushed three years ago from their laptop — before noreply mode, before they thought about privacy at all — sits in a public repository and is served by a plain GET /repos/{owner}/{repo}/commits with no key at all.</p><p>Hence the collectors/person_contacts.py tier. It runs after the ladder, and only for the two or three people who will actually appear in the report:</p><pre>GitHub: profile → public push events → top repositories<br>      ↓<br>GitLab: profile, website, organisation<br>      ↓<br>Telegram: your own session, MTProto<br>      ↓<br>Web search with page fetching<br>      ↓<br>hunter.io  ← paid, last, and only if a key exists</pre><p><strong>The order is the design.</strong> Not &quot;most accurate source first&quot;, but <strong>free and unattached to anyone first</strong>. The paid provider fills a gap rather than shaping the result: with no key, *from*hunter is a no-op and the other four channels behave identically. That follows directly from a requirement in the first article: the moment a tool has a mandatory API key, it also has an instruction that begins &quot;register, attach a card, deposit $5&quot;, and a person actively looking for work simply will not run that program.</p><p>Three refinements produced most of the hits:</p><p><strong>Public push events.</strong> GET /users/{login}/events/public returns recent pushes together with commit objects, and those carry the author&#39;s address. Empty profile, talkative events. One request.</p><p><strong>Own repositories, not forks.</strong> If neither the profile nor recent events yield an address, the tool reads the person&#39;s most recently pushed repositories and takes the <em>earliest</em> commit. The first commit of somebody&#39;s pet project was almost always made before they configured user.email carefully — so what&#39;s in there is a real personal address, not a corporate one. The difference matters: a work inbox stops working the day someone leaves; a personal one doesn&#39;t. Personal addresses are labelled as such in the report.</p><p><strong>Fetching the page instead of trusting the snippet.</strong> The single biggest lever on findability, full stop. If a search result names the person but truncates the contact in the snippet, the page is downloaded and read in full. The address is almost always on the page, not in its summary. Social networks and login walls are skipped, and the budget is capped: PERSON_LOOKUP_MAX_PAGE_FETCHES = 6 per run.</p><h3>Half the module is rejection logic, and it has to be</h3><p>An error here looks like a success. The letter goes to a namesake, you never find out, and a stranger receives a CV from someone who got them confused with somebody else. So every channel has its own ownership rule:</p><ul><li><strong>GitHub profile</strong> — accepted only if the name <strong>and</strong> the company match. verified</li><li><strong>Address from a commit</strong> — accepted only if author.name matches. verified</li><li><strong>GitLab profile</strong> — company confirmed by the profile or by the website domain. verified</li><li><strong>Telegram, handle resolution</strong> — the account&#39;s name matches the person&#39;s name. verified</li><li><strong>Telegram, search by name</strong> — the name matches <strong>and</strong> the @username is built from the name. likely</li><li><strong>Address from search results</strong> — the local part of the address is derivable from the name. likely</li><li><strong>Phone</strong> — international format <strong>and</strong> the page names the person. likely</li><li><strong>hunter.io</strong> — confidence comes from the provider, as given.</li></ul><p>The rule in models/contact.py is untouched: <strong>nothing is synthesised</strong>. The program never shows an address it hasn&#39;t seen printed somewhere. Every contact in the report carries a provenance tag — [personal_github/verified], [search/likely] — and that isn&#39;t decoration: it&#39;s how you know whether to write confidently or check with your own eyes first.</p><h3>Is this legal? Is it decent?</h3><p>This is the first comment on every version of this article, so I&#39;ll answer before it&#39;s asked, without dodging.</p><p>Everything this tier reads, the person published themselves: a profile on a forge, a commit in a public repository, a &quot;team&quot; page on a company site, their own handle. No leaks, no rented databases, no SMTP probing (more on that below). The data isn&#39;t accumulated into a database for sale — it lives in a local SQLite file next to the executable, and deleting it means deleting the folder.</p><p>Then there&#39;s the place where legality and decency diverge. GDPR under legitimate interest doesn&#39;t forbid writing to a person at a published address, but that doesn&#39;t mean you may write to everyone. The difference between <em>&quot;I found the hiring manager for a role I&#39;m actually qualified for and wrote to them about it&quot;</em> and <em>&quot;I exported a department&#39;s contacts and started a campaign&quot;</em> is not a technical difference. The tool holds that line the only way that works: <strong>there is no bulk sending in it and there never will be.</strong> Three people per opening, copy-paste, letter written by hand.</p><p>And yes — that&#39;s exactly the constraint I left myself in the first article, and the easiest one to have quietly relaxed this week.</p><h3>The scoring gave away 50 points for free</h3><p>Now the part I actually sat down to write about.</p><p>The matcher had a line that looked like a harmless default. In two places — in the fast score and in the full verdict:</p><pre>must = profile.must_have_skills or profile.skills[:4]</pre><p>Reads as: &quot;if no must-have gate is defined, take the first four skills the parser returned.&quot; Sounds reasonable. There are two bugs in it, and the second is much worse.</p><p><strong>Bug one: half the score hung on list order.</strong> <em>Which</em> four skills gate an opening is a decision. skills[:4] delegates that decision to the parser&#39;s output order. A CV that happened to list Python second started matching Python jobs as if Python were a hard requirement.</p><p><strong>Bug two: coverage of the empty set is 1.</strong> If must_have_skills is empty <em>and</em> skills is empty or shorter, the fraction of covered must-haves computes as &quot;all of the ones that existed are covered&quot; — that is, 100%. And must-have is half the scale. A profile with no gate at all scored <strong>50 points out of 100 simply for being a profile</strong>, which turned the 60% threshold into a formality: almost any opening can find another ten points among the nice-to-haves.</p><p>Now the key part, the reason the tests never caught it: <strong>the gate was only ever assigned on one code path.</strong> A resume.md → profile.json file went through an enrichment step where the must-have decision is made once, deliberately. A CV uploaded through the app window, or sent to the bot, was saved <em>raw</em> — straight from the parser: must_have_skills empty, domains empty, deal_breakers empty. Which means <strong>on every run started from the GUI, the deal-breaker filter did not work at all</strong>, and half the score was handed out for free. The developer tested the CLI; the user ran the window; those were two different scoring systems.</p><p>The fix was shorter than the diagnosis: gating_skills() uses only declared fields, and when the gate is empty the must-have share <strong>moves over</strong> to nice-to-have rather than being handed out. profile.loader. enrich() now runs on every input and is idempotent, so a hand-edited profile.json passes through untouched.</p><p><strong>The moral:</strong> x or fallback in a scoring function is not a default — it&#39;s a second, undeclared algorithm that nobody tests. And if your metric is a coverage ratio, check separately what it equals on the empty set. Mathematically it&#39;s one. Semantically it&#39;s &quot;no data&quot;. Those are very different numbers to be multiplying by half your scale.</p><h3>Location can&#39;t be an addend</h3><p>The second bug of the same run, and more annoying because it hits the most visible thing.</p><p>Location participated in the score as an addend: a miss cost a few points out of twelve. Formally fair — &quot;the place isn&#39;t ideal but the job is strong.&quot; In practice, a run for a candidate who lives in Serbia and said so in three fields returned a job in Mexico City above the threshold. Because the stack matched beautifully, and a continent miss is minus five points.</p><p>Location became a filter. An opening passes if it names one of your countries, or a broader market that contains your country, or — if you specified a market — a country inside it. Then the details, of which there were unexpectedly many:</p><ul><li><strong>&quot;Remote&quot; is not a reason.</strong> &quot;Remote, United States&quot; is work in the US, with their time zone and their right-to-work requirements.</li><li><strong>A location that isn&#39;t stated at all keeps the opening.</strong> So does one that isn&#39;t in the alias table. A false keep costs one extra line in the report; a false drop silently removes the one you needed. The asymmetry is obvious and the filter is tuned to it.</li><li><strong>Countries are compared, not spellings</strong> (config.REGION_ALIASES) — and two-letter codes only as whole fields. Otherwise de is found inside Rio de Janeiro.</li><li><strong>The checkbox is part of the scorer&#39;s signature.</strong> Otherwise the verdict cache would survive toggling it, and a disabled filter would keep filtering with yesterday&#39;s answers.</li><li><strong>The number of filtered-out openings is printed in the report footer.</strong> A filter that silently removes a third of a run is indistinguishable from a broken source.</li></ul><p>On 35 openings from my own database: 33 → 27, six dropped on location — Prague, Bangalore, San Francisco, Coimbra, Mexico City, the US. As a bonus, Threads spam that had been scoring 67% and reliably making the report stopped clearing the threshold.</p><h3>One profile per application</h3><p>The third bug in the same family, and probably the most embarrassing.</p><p>The &quot;Search&quot; button in the window read the profile from the database. Scheduled runs and the CLI read profile/profile.json. The copies drifted silently: the file had the gate, the hand-edited deal-breakers and the domains; the database had whatever the parser returned and nothing else. The same person with the same CV got two different runs depending on which button they pressed.</p><p>The window now writes and reads the same file, and a profile left in the database by an older build is migrated into it on first read. No manual migration: anyone who unzips the new version over the old folder notices nothing.</p><p>Living next door was another bug of the same class, which I only found when I unticked a source myself and saw no difference. <strong>The &quot;Find jobs&quot; button did not apply the form.</strong> The pipeline reads parameters from search.json, and the widgets only got there after a separate &quot;Save&quot; button — in a different panel, below the search button itself. You untick a source, press Find, and get a run with the previous parameters while looking at the new ones. Same for keywords, locations, job titles and the threshold.</p><p>Three bugs, one shape: <strong>one fact had two places to live, and the program didn&#39;t know which one was true.</strong> If you&#39;re looking for this class of bug in your own code, start with fields written from two different paths.</p><h3>Three bugs that have nothing to do with the domain</h3><p>A week of fixing things always produces a collection of findings unrelated to the subject matter. These are my favourites.</p><h3>A flaky test that lied about what it was testing</h3><p>A build on main failed on test_each_refusal_doubles_the_pause. The test had been flaky from the start and failed on double rounding.</p><p>The pause stores monotonic() + delay. On Windows, monotonic() advances in 15.6 ms ticks, so both calls in the test read the same instant, and the difference between deadlines is exactly two delays. Exactly — except for floating point: at an uptime of 596.093 s, &quot;+300&quot; lands on 896.093 and &quot;+600&quot; lands on 1196.0929999999998, so the difference comes out as 299.9999999999999. The test demanded &quot;no less than 300&quot; and failed on a machine whose uptime rounded the wrong way. The numbers from the CI log reproduced exactly.</p><p>The clock is now frozen and the assertion got stronger: the first pause equals the constant, the second is twice that, and the twelfth hits the ceiling rather than continuing to double. That last property wasn&#39;t checked at all before — meaning the test that kept failing also wasn&#39;t catching the real bug.</p><h3>Argument list too long — and a showcase linking to nine images that didn&#39;t exist</h3><p>Publishing the showcase failed like this:</p><pre>/usr/bin/gh: Argument list too long<br>Error: Process completed with exit code 126.</pre><p>The step uploaded files via -f content=&quot;$(base64 -w0 &quot;$file&quot;)&quot;. Linux has a limit on the length of <strong>one</strong> argv element — MAX_ARG_STRLEN, 128 KiB — and it is not configurable by ulimit or anything else. Base64 adds a third, so any file over roughly 96 KB killed the step. Of the twelve showcase files exactly two exceeded it: the main window screenshot in both languages, 111 KB on disk, 148,244 bytes after base64 against 131,072 allowed.</p><p>Worse than the failure was <strong>how</strong> it failed. Files are uploaded one at a time, and by the time it errored, the README in two languages and one image had already landed: the public repository was left with a new README and links to nine images that weren&#39;t there. <strong>A partial deploy is worse than a failed one.</strong></p><p>The request body is now assembled by a separate script and passed as a file via gh api --input: only the path, the commit message and the blob SHA go into argv, while the content is read from disk and printed to stdout, bypassing both the command line and the environment — the limit on environment strings is the same one.</p><h3>Ctrl+C doesn&#39;t work on a Cyrillic keyboard layout</h3><p>You couldn&#39;t copy text out of the application log. The panel wasn&#39;t at fault: <strong>Tk binds </strong><strong>&lt;&lt;Copy&gt;&gt; to the letter </strong><strong>c, not to the key</strong>, and on a Cyrillic layout Ctrl+C arrives as Control-Cyrillic_es and matches nothing.</p><p>In a window that is entirely in Russian, that&#39;s not an edge case — it&#39;s the normal state of the keyboard. And it broke more than the log: Ctrl+V in the field where you paste your CV, and in every token field, went the same way. So the first step of the primary flow (&quot;paste your CV&quot;) did not work for a user with a Cyrillic layout, and I learned this on day seven.</p><p>The fix is a handler that reads the key <em>code</em> and only handles presses Tk never saw (returning None for Latin, or it would paste twice). One binding, on the all tag, fixing every field and panel at once.</p><h3>i18n without gettext: Russian strings as their own keys</h3><p>An English interface was requested on day one. The project is written in Russian: strings in source, comments, reports. The standard path is gettext, _(), and moving every string into a catalogue behind artificial keys like btn.search.label.</p><p>I did it differently: <strong>the Russian string is the key.</strong></p><pre>ttk.Button(row, text=t(&quot;Найти вакансии&quot;), ...)</pre><p>The button still reads as a button. There&#39;s no file where btn.search.label becomes a riddle six months later. The English catalogue sits next to it — 552 entries today.</p><p>The interesting part is the test. tests/test_i18n.py walks the sources for every t(&quot;…&quot;) and for tables whose values act as keys, and fails on:</p><ul><li>an untranslated string;</li><li>an orphaned catalogue entry (the string in the code was rewritten, the translation stayed);</li><li>a lost {placeholder} — the classic, where a translation drops a substitution and format() blows up in the user&#39;s face;</li><li>a module that calls t without importing it. That last one immediately found a live NameError in profile/search.py — code that would have crashed on its first call.</li></ul><p>The first translation bug showed up exactly where it hurts most: the &quot;here&#39;s what I understood&quot; block was built with f-strings, so under the English heading &quot;Here is what I understood&quot; sat &quot;Позиция: … Грейд: … Стек: …&quot;. That&#39;s the first thing a person sees after switching languages, and it reads like an unfinished translation, which is what it was.</p><h3>Applications: a funnel, two templates, and a Wilson interval</h3><p>The broken promise. Since the boundary got rewritten, it got rewritten properly.</p><p><strong>Status ladder:</strong> new → written → replied → screening → interview → offer, plus terminal rejected/skipped. Forward by any number of rungs is allowed; backwards is not — you can&#39;t un-reply to a letter, and correcting a mistake is a cancellation, not a rollback.</p><p>A few decisions I think are non-obvious:</p><ul><li><strong>The key is a hash of the opening, not a rowid.</strong> A database id survives neither a reinstall nor an imported archive; the hash is identical in the database, the archive and the report.</li><li><strong>&quot;Ghosted&quot; is derived, not stored.</strong> It&#39;s written plus silence longer than GHOSTED_AFTER_DAYS = 21. A status you have to set by hand is a status nobody sets, and the funnel then lies in the most important place.</li><li><strong>Status is not written into the run archive.</strong> An archive is a snapshot, and it&#39;s honest precisely because it doesn&#39;t change. A frozen status would turn a week-old report into a file confidently claiming &quot;written&quot; where a rejection has since arrived. Applications are overlaid on the report at render time.</li><li><strong>Two letter templates from the start.</strong> One template isn&#39;t an A/B test, it&#39;s just a template. When someone presses &quot;Copy letter&quot; and then &quot;Sent&quot;, the version is recorded — without that, the &quot;by message version&quot; view is unmeasurable in principle.</li><li><strong>A percentage is never shown without a sample size.</strong> Below eight letters the row says &quot;not enough data&quot;; above it, it prints the percentage, n, and a Wilson interval. At thirty applications, the difference between 17% and 11% is noise, and a decision based on it is worse than random — because it&#39;s also confident. This is the one place where I deliberately made the product look <em>less</em> impressive: percentages everywhere look more serious than an honest &quot;not enough data&quot;.</li><li><strong>Stages are counted literally, from the transition history.</strong> &quot;An interview implies a screen&quot; is false: people get invited to interviews without one. Crediting the screen would turn &quot;this stage gets skipped for me&quot; into &quot;this stage exists for me&quot;.</li><li><strong>The funnel bars are drawn on a </strong><strong>tk.Canvas.</strong> matplotlib would have added tens of megabytes on top of a 184 MB build for seven rectangles.</li></ul><p>Scheduling arrived alongside: JobHunter.exe --scan is the same run without the window. Only &quot;when the user is logged on&quot;, though — anything else needs a stored account password, and running as SYSTEM breaks paths and Playwright. LinkedIn is disabled in background runs: a browser step at night with nobody there is pure account risk and nothing else.</p><h3>Trust: SHA-256, a signature, and a 404 in the most important paragraph</h3><p>Downloading a zip with an executable from a stranger on the internet is a dubious pleasure, and I know it. Here&#39;s what went in that direction this week.</p><p><strong>A hash you can actually check.</strong> Previously the showcase README sent you for the SHA-256 to an Actions page in a <em>private</em> repository. For everyone except me, that link returned 404 — in precisely the paragraph suggesting you verify the file hadn&#39;t been tampered with. Now the hash is computed from the downloaded artifact, written into the release notes, and the README shows the Get-FileHash command.</p><p><strong>A signed executable.</strong> Set-AuthenticodeSignature from stock PowerShell, no Windows SDK, no signtool — the build still requires only Python. With an RFC 3161 timestamp, without which the signature stops verifying the day the certificate expires. In CI the step activates on secrets and, without them, prints a notice and skips: a fork&#39;s build shouldn&#39;t fail for lack of somebody else&#39;s certificate.</p><p>And immediately the honest part: <strong>the certificate is self-signed, and it will not clear SmartScreen for a stranger.</strong> It says &quot;this file hasn&#39;t changed since it was signed&quot;, not &quot;this publisher is trustworthy&quot;. That&#39;s what the README says too — the temptation to write &quot;signed&quot; and stop there was real.</p><p><strong>What else is verifiable without taking my word for it:</strong> builds come from CI on a commit, not from my machine; --selftest runs inside the built exe; there&#39;s no telemetry; the showcase sources and screenshots leave the repository rather than being assembled by hand.</p><p>On screenshots specifically. A screenshot goes stale silently: a button gets renamed, a column is added, and the picture stays and lies — and the first person to notice is the first person to download. So both sets (RU and EN, five images each) are produced by one script. It copies the sources to a temporary folder and launches the window there: a run made for screenshots must not overwrite a real database and real search parameters. The data is fictional and the domains end in .example — those are the images that go into a public repository, and a real contact has no business being in them.</p><h3>The CV parser stopped being backend-only</h3><p>An admission I should have written sooner: the skill taxonomy was tuned to my own stack. Java, Spring, Kafka, PostgreSQL. Any other CV — frontend, mobile, data/ML, QA — lost half its keywords, and matching for those people was noticeably worse. A tool that &quot;works for everyone&quot; worked for me.</p><p>Two changes:</p><p><strong>Capturing unknown declared skills.</strong> A new extract_declared_terms reads the skills section and inline &quot;Stack:&quot; lines and pulls out terms that aren&#39;t in the taxonomy at all: svelte, airflow, rstudio, sqlalchemy. They enter the profile at a lower weight than recognised skills, but a non-zero one — so an opening naming exactly that term still scores. Prose under a &quot;Skills&quot; heading isn&#39;t shredded: there&#39;s a filter on stop words, length and word count.</p><p><strong>An expanded taxonomy:</strong> languages (swift, dart, elixir), frameworks (next.js, svelte, flutter, pytorch, tensorflow, spark, airflow), databases (dynamodb, snowflake, bigquery), cloud/devops (helm, ansible, argocd), practices (tdd, qa automation, oauth, bi).</p><p>And matching precision along with it. An unknown skill is now matched on word boundaries rather than as a substring: java no longer matches javascript, go no longer matches category. The ultra-short c and r were removed from the taxonomy entirely — a bare c catches &quot;C.&quot;, &quot;Plan C&quot; and essentially everything; their ecosystems get picked up as declared terms instead.</p><h3>The week in numbers</h3><p><strong>Tests</strong> — 2 Sep: 345; 8 Sep: <strong>746</strong></p><p><strong>Job collectors</strong> — 2 Sep: 24; 8 Sep: 24</p><p><strong>Personal-contact channels</strong> — 2 Sep: 0; 8 Sep: <strong>5</strong></p><p><strong>…of which need a paid key</strong> — 2 Sep: —; 8 Sep: <strong>1, last in the chain</strong></p><p><strong>Report formats</strong> — 2 Sep: txt; 8 Sep: <strong>txt, csv, xlsx</strong></p><p><strong>Interface languages</strong> — 2 Sep: 1; 8 Sep: <strong>2</strong></p><p><strong>Settings editable from the window</strong> — 2 Sep: 0; 8 Sep: <strong>59</strong></p><p><strong>Application tracking</strong> — 2 Sep: none; 8 Sep: funnel + 6 views</p><p><strong>Scheduled runs</strong> — 2 Sep: none; 8 Sep: yes</p><p>The collectors row is deliberate. Not a single new job source was added this week: everything went into making what was already collected stop lying.</p><h3>Honest limitations, updated</h3><p>The section without which this would be an advert.</p><ul><li><strong>Free contact lookup doesn&#39;t find everyone.</strong> It rests on what people published themselves. A backend engineer with an active GitHub: excellent odds. A recruiter with no public code: much worse, and the honest answer there is LinkedIn or nothing. There will never be &quot;100% coverage&quot; here — that&#39;s a property of the data, not of the parser.</li><li><strong>There is no SMTP probing and there won&#39;t be.</strong> The most common way to &quot;verify&quot; a guessed address is RCPT TO against the mail server. It doesn&#39;t work: Workspace and M365 accept everything, port 25 is closed at consumer ISPs, and probing gets your own IP greylisted. So guessed addresses never enter the &quot;People&quot; list — they&#39;re printed in a separate block marked <em>unverified</em>, and the whole guessing tier is off by default.</li><li><strong>The scoring still doesn&#39;t understand context.</strong> It&#39;s a keyword weighting algorithm: &quot;Java&quot; in the requirements and &quot;Java&quot; in a list of the team&#39;s technologies are the same thing to it. It&#39;s explainable and free, but it&#39;s dumb, and sometimes you can tell. This week it became more honest (the gate, the location). It did not become smarter.</li><li><strong>Web search breaks.</strong> DuckDuckGo returns an anti-bot page from the second request off one IP; Bing silently ignores site:. Engines back off with doubling pauses, the query goes to the next one, and a warning appears in the report — because a silently empty list of people looks like &quot;there&#39;s nobody at these companies&quot;.</li><li><strong>LinkedIn is still against the ToS.</strong> It runs on your own session with hard pause budgets and a cap on navigations per run, it&#39;s entirely optional, and the risk to your account is real. I&#39;m not going to pretend otherwise.</li><li><strong>Windows.</strong> The prebuilt release targets Windows 10/11. From source it runs anywhere with Python 3.11+.</li><li><strong>It will not find you a job.</strong> It saves the &quot;half an hour plus half an hour&quot; per opening. You still write the letter, and the letter is still the part that matters.</li></ul><h3>Trying it</h3><p><strong>Download (Windows, no Python needed):</strong> the resume2human-windows.zip release asset on GitHub.</p><p>Unpack the archive <strong>whole</strong> — there&#39;s a folder inside, not a single file, and the exe doesn&#39;t start without its neighbours — and run JobHunter\JobHunter.exe. The archive&#39;s SHA-256 is in the release notes and checks in one command:</p><pre>Get-FileHash .\resume2human-windows.zip -Algorithm SHA256<br>JobHunter.exe --selftest      # result in data\selftest.log</pre><p>Everything the program creates — database, reports, LinkedIn and Telegram sessions — appears in the folder next to the exe. Deleting it means deleting the folder.</p><p>Repository, README in two languages, releases: <a href="https://github.com/ialakey/resume2human"><strong>github.com/ialakey/resume2human</strong></a></p><h3>What&#39;s next</h3><ul><li><strong>More forges for free lookup.</strong> Bitbucket, Codeberg, sourcehut, public mailmaps. The logic is already generalised; each new one is a function and an ownership rule.</li><li><strong>Adapters for Workday, Comeet, Gupy, PeopleForce.</strong> 83 companies on my list still fall off on those.</li><li><strong>An honest measurement of the matcher.</strong> A second backend on an LLM, and a comparison: does it beat the weighted one by enough to justify a key? After the skills[:4] story I&#39;m far more interested in a different question — how many other defaults are handing out points for free.</li></ul><p>If you run it on your own CV, tell me what it found and what broke. Non-backend CVs are the most useful of all: the expanded taxonomy was validated against my fixtures, not against your actual résumé, and the chance of a silent failure there is high.</p><p>And two claims I&#39;m willing to argue about in the responses, both from this week:</p><ol><li><strong>Free contact discovery through git covers most of what people buy those subscriptions for.</strong> Not all of it. But the part a person job-hunting for themselves needs — yes.</li><li><strong>x or fallback in any metric is a hidden second algorithm.</strong> It isn&#39;t documented, it isn&#39;t tested, and it switches itself on for exactly the users who arrived by a different path than the developer did.</li></ol><p><strong>Where&#39;s the </strong><strong>x or fallback in your codebase?</strong> I read the responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a6418b4af0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA["Provably Fair" and "Rigged" Are Not Opposites]]></title>
            <link>https://medium.com/@alakov.ilia/provably-fair-and-rigged-are-not-opposites-5bf3a252876b?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bf3a252876b</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[cryptography]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[game-development]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 13:01:04 GMT</pubDate>
            <atom:updated>2026-09-23T13:01:04.620Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpePw0a7BpspcaA0PoUxhA.png" /></figure><p>A couple of months ago I caught myself unable to answer a question clearly: when you open a case on one of those CS2 skin sites, are you being cheated <em>on the roll</em>, or somewhere else entirely?</p><p>There are dozens of these sites. Real money moves through them. Not one publishes its source. From the outside you get a spinning reel and a button. Inside there are Steam bots, a client emulation layer, HMACs, integer accounting, and some very specific mathematics arranged so that the house wins on average.</p><p>Rather than settle for &quot;they&#39;re all scammers&quot;, I built one end to end and put it under MIT. Not an animated demo — the whole cycle: Steam login, case opening with a verifiable roll, selling and upgrading an item, a withdrawal request that a trade bot actually sends, and an admin panel where a case is balanced against a target RTP before it ever ships.</p><p>And while writing it, the thing I&#39;d actually gone looking for became visible. It isn&#39;t cheating in the roll. <strong>It&#39;s that cheating in the roll isn&#39;t necessary.</strong></p><p>Repository: <a href="https://github.com/ialakey/caseforge"><strong>github.com/ialakey/caseforge</strong></a> · MIT.</p><p><em>Disclaimer, to get it out of the way. Buying a case for real money is regulated as gambling in most jurisdictions: licensing, KYC, age verification, country restrictions. There are </em><strong><em>no payments</em></strong><em> in this project — the balance top-up is a stub, disabled by default in production. This is a research and teaching codebase about architecture, not a turnkey business. Running it for real money without the legal half is not a grey area, it&#39;s a straight violation, and an MIT licence does not help you.</em></p><h3>The claim, stated plainly</h3><p>The industry-standard &quot;provably fair&quot; scheme answers exactly one question:</p><blockquote><em>Was </em>this specific roll<em> tampered with after the fact?</em></blockquote><p>It answers it well, with real cryptography, and you can check it yourself with a calculator. What it does not answer — what it is not designed to answer, and what people consistently believe it answers — is:</p><blockquote><em>Is this game good for me?</em></blockquote><p>That second question is answered by RTP, the return to player, and RTP lives in a completely different place in the code. A site can be flawlessly, verifiably honest on every roll and still build a case at 85% RTP, taking 15% of every opening on average. Both statements are true at the same time, and the second one decides the outcome.</p><p>That&#39;s the whole article. Everything below is the machinery that makes it concrete — plus, for the engineers, the half of the project that has nothing to do with games at all, because it turns out that talking to Steam is harder than everything else combined.</p><h3>1. The least obvious technical fact: the backend has to be Node</h3><p>I started out thinking I&#39;d write it in Go. Websockets, load, fine. That didn&#39;t survive contact with the problem, and the reason is technical rather than aesthetic.</p><p><strong>Trading items in Steam does not go through the official Web API.</strong> The Web API can only read: inventory, history, profile. Creating a trade offer, confirming it with the mobile authenticator, tracking its status — those are internal steamcommunity.com endpoints, and working with them means emulating a Steam client, protocols and all.</p><p>There is exactly one living, maintained ecosystem for that: DoctorMcKay&#39;s Node libraries.</p><ul><li><strong>steam-user</strong> — logging in as a Steam client, sessions, refresh tokens</li><li><strong>steamcommunity</strong> — steamcommunity.com cookies and session, mobile confirmations</li><li><strong>steam-tradeoffer-manager</strong> — creating, tracking and confirming trade offers</li><li><strong>steam-totp</strong> — Steam Guard codes and confirmation keys from shared_secret</li><li><strong>globaloffensive</strong> — connecting to the CS2 game coordinator: float, pattern, stickers</li></ul><p>Python equivalents (steam, steampy) exist but lag noticeably: they get fixed later after each Valve change, and mobile-confirmation coverage is weaker. For a project where the bot farm <em>is</em> the cash register, a week behind Valve means a week with no withdrawals.</p><p>Which settles the whole stack. <strong>The trading layer is going to be Node regardless.</strong> Given that, writing the rest of the backend in a second language means two data models, two sets of types and a bridge between them, for a few percent of RPS. So the entire project is TypeScript:</p><pre>apps/web          Next.js 15 (App Router), React 19, Tailwind, Zustand, socket.io-client<br>apps/api          NestJS 11 on Fastify, Prisma 6, Postgres 16, Redis 7, BullMQ, socket.io<br>apps/bot          Node worker: the Steam bot farm, consumer of the withdrawal queue<br>packages/shared   shared types, zod schemas, translations, tickets and provably fair</pre><p>packages/shared isn&#39;t a &quot;utils folder&quot; here, it&#39;s load-bearing: the same upgrade-chance function runs on the server when rolling and in the browser when drawing the odds bar. They cannot drift apart by construction.</p><p>One monorepo detail worth a saved evening: the package is deliberately split into two entry points. The main one is isomorphic; server-side crypto lives behind the @caseforge/shared/node subpath. Otherwise node:crypto ends up in the browser bundle and the frontend simply won&#39;t build.</p><h3>2. Provably fair: where you actually could cheat</h3><p>The scheme is standard and you can verify it by hand.</p><ul><li>The server generates a serverSeed (32 random bytes) and publishes only sha256(serverSeed).</li><li>The player sets their own clientSeed (or gets a random one) and can change it whenever they like.</li><li>The seed pair has a nonce counter that increments on every opening.</li><li>The roll:</li></ul><pre>export function computeRoll(serverSeed: string, clientSeed: string, nonce: number): number {<br>  const hmac = createHmac(&#39;sha256&#39;, serverSeed)<br>    .update(`${clientSeed}:${nonce}`)<br>    .digest(&#39;hex&#39;);<br>  return parseInt(hmac.slice(0, 8), 16) % TICKET_SPACE; // TICKET_SPACE = 1_000_000<br>}</pre><ul><li>The item whose ticket range covers roll is the one you get.</li><li>When the server seed rotates, the old one is revealed in full — any past roll can be recomputed and checked against the published hash.</li></ul><p><strong>Why ticket ranges instead of weights:</strong> a range is verifiable in your head. [0, 749999] is 75%, and the order in which the array is walked has no effect on the result. With weights, the user would have to trust your normalisation. The ticket-space size is fixed forever: changing it would make every past opening unverifiable.</p><p>The browser-side re-check is deliberately an <strong>independent implementation on Web Crypto</strong> (packages/shared/src/verify.ts), not a call into the same server function. If the verification page called the same code as the roll, it would only be checking the server&#39;s consistency with itself.</p><h3>Honest about modulo bias</h3><p>2^32 doesn&#39;t divide evenly by 1,000,000. The first 967,296 tickets get 4,295 hashes each, the rest get 4,294. The skew is about <strong>0.023%</strong>, identical for everyone, and orders of magnitude below price granularity. I deliberately don&#39;t do rejection sampling: it breaks the property the whole scheme exists for — that you can verify a roll with a calculator and one line in a console. That tradeoff is written in a comment in the code, not hidden.</p><h3>The real lever is not the roll</h3><p>Here&#39;s the thing that flipped my everyday understanding of what &quot;fair&quot; means on these sites.</p><p><strong>Nobody needs to rig the roll.</strong> It&#39;s risky — revealing the seed turns any rigging into a mathematical proof of fraud — and it&#39;s pointless, because the lever that decides your outcome isn&#39;t the roll, it&#39;s <strong>the ticket table</strong>. A site can be entirely honest in the provably-fair sense and still assemble a case at 85% RTP.</p><p>So the project enforces two hard rules:</p><ul><li>the serverSeed depends on neither the item nor the user, and the drop decision is not allowed to look at the player&#39;s balance;</li><li>if the economy needs adjusting, it gets adjusted through the ranges — in the open, in the admin panel, with the RTP recomputed.</li></ul><p>And correspondingly, every case on the storefront publishes its contents: rarity, current price, and the <strong>exact chance</strong> of each item. Not &quot;approximately&quot; — the actual ranges the server rolls against.</p><h3>3. The economy: &quot;make this case profitable&quot; is one equation with N unknowns</h3><p>A case&#39;s RTP = Σ(item chance × item price) / case price.</p><p>The working corridor is 85–95%. Above 100% the case loses money by construction; below 80% nobody buys it. The hard ceiling is 98% — above that the server refuses to save the case at all.</p><p>The operator&#39;s job sounds like &quot;arrange the odds so the player finds it interesting and the site finds it profitable&quot;. Formally that&#39;s one equation with N unknowns, so there are infinitely many solutions. You need a model. I used the one that&#39;s natural for cases: <strong>the more expensive an item, the rarer it is.</strong></p><p>Formally: item weight w_i = price_i^(-k), chance s_i = w_i / Σw. At k = 0 all items are equally likely; as k grows the cheap ones crowd out the expensive ones. Expected value decreases monotonically in k, so the right exponent is found by binary search:</p><pre>const evAt = (k: number): number =&gt; {<br>  // Normalise prices by the minimum: price^(-k) overflows a double for large<br>  // prices and k, and normalising doesn&#39;t change the ratios between weights.<br>  const weights = prices.map((p) =&gt; Math.pow(p / minPrice, -k));<br>  const total = weights.reduce((a, b) =&gt; a + b, 0);<br>  return prices.reduce((sum, p, i) =&gt; sum + (weights[i]! / total) * p, 0);<br>};<br><br>let lo = -60, hi = 60;                   // k = -60 pushes everything to the most expensive item,<br>for (let iter = 0; iter &lt; 200; iter++) { // k = 60 to the cheapest<br>  const mid = (lo + hi) / 2;<br>  if (evAt(mid) &gt; targetEv) lo = mid;<br>  else hi = mid;<br>}</pre><p>The achievable RTP range is bounded hard by the extreme items: no distribution can return less than the cheapest item or more than the most expensive one. If the target falls outside that corridor, the builder doesn&#39;t silently approximate — it tells you what to change:</p><blockquote><em>At a case price of 250.00 the achievable RTP is 12.4%…940.0% (items from 31.00 to 2350.00). Your target is lower — raise the case price or add a cheaper item.</em></blockquote><p>The inverse problem — &quot;here&#39;s the loot table, what should the case cost&quot; — is trivial: price = expected drop value / target RTP. That&#39;s how cases actually get built in practice: contents first, price second.</p><h3>The problem you only see in production</h3><p>Chances are fractions; tickets are integers. Rounding down leaves a remainder of tickets, and it has to go to somebody. The obvious answer — &quot;give it to the last item in the list&quot; — quietly breaks the economy: the list is sorted by price, the last item is the knife, and the remainder invisibly raises both the knife&#39;s odds and the whole case&#39;s RTP.</p><pre>// The remainder goes to the item with the widest share, i.e. the cheapest one.<br>let widest = 0;<br>for (let i = 1; i &lt; widths.length; i++) {<br>  if (widths[i]! &gt; widths[widest]!) widest = i;<br>}<br>widths[widest]! += remainder;</pre><p>Separately: an item whose share rounds to zero still gets one ticket. A slot in a case with a zero chance is a lie on the storefront.</p><h3>Price drift</h3><p>A case built at 92% RTP is at 105% a month later because a knife appreciated. So:</p><ul><li>an hourly job pulls Steam Market prices for items in active cases;</li><li>after every re-pricing, the RTP of every active case is recomputed;</li><li>leaving the corridor logs a warning; exceeding 98% logs an error.</li></ul><p>A separate hazard is <strong>unconfirmed prices</strong>. An item whose price Steam has never returned (no listings, a malformed name) carries a made-up number, yet participates in the RTP calculation on equal terms. Those items are flagged both in the case list and in the builder — this is the quietest way to drive an economy underwater.</p><h3>4. An upgrade you don&#39;t have to balance separately</h3><p>The player stakes their skin against a more expensive one: win and you take the expensive item, lose and you give up yours.</p><p>The temptation is to set the odds by hand. I derive them from the prices:</p><pre>chance = stakeValue / targetValue × UPGRADE_RTP,  UPGRADE_RTP = 0.9</pre><p>The same 0.9 the cases run on. Under that definition, the expected value of an upgrade equals stake × RTP <strong>at any multiplier</strong> — there&#39;s a test for it. Which means the upgrade automatically lives in the same economy as the cases and needs no separate balancing. This is exactly the place where a tidy formula saves an analyst a month of future work.</p><p>The roll is the same roll, from the same seed pair and the same shared nonce:</p><pre>winThreshold: Math.floor(chance * TICKET_SPACE)  // a win when roll &lt; winThreshold</pre><p>That&#39;s not code reuse for its own sake, it&#39;s a product property: the user has <strong>one</strong> fairness verification page for every mechanic, and an upgrade is checked exactly the way a drop is.</p><p>Boundaries: multipliers under 1.05 are rejected (that&#39;s not an upgrade, it&#39;s a swap with commission), chances above 85% too (a near-guaranteed swap), and price gaps wide enough to push the chance below 0.5%. The list of available targets on the frontend is computed with the same formula as the chance — so the UI can never offer an option the server will then refuse.</p><p>The stake burns regardless of outcome: it&#39;s a stake, not a deposit. And it&#39;s debited by a conditional updateMany on status = AVAILABLE — if the item has meanwhile been sold or sent to withdrawal, the upgrade doesn&#39;t happen, rather than the item being handed out twice.</p><p>And a bug I caught on myself. I&#39;ll leave the code comment as it stands:</p><pre>// Check the chance BEFORE the clamp, not after. Clamping first makes the<br>// comparison always pass — the minimum has already been substituted by the<br>// clamp — and an excessive price gap goes unnoticed.<br>if (raw &lt; UPGRADE_MIN_CHANCE) { ... }<br>const chance = Math.min(UPGRADE_MAX_CHANCE, raw);</pre><p>A classic: a clamp before validation turns validation into decoration.</p><h3>5. Money: integer minor units, conditional UPDATE, and reserving a block of nonces</h3><p>No Float, anywhere. Every amount is an <strong>integer in minor units</strong> (Int). Every balance change creates a row in Transaction; the balance on User is a denormalised cache that is <em>required</em> to reconcile with the sum of transactions. A nightly cron checks; a mismatch raises an alert.</p><p>Debiting is not read-compute-write, it&#39;s a conditional UPDATE. Zero rows affected means insufficient funds, and the race between two openings in two browser tabs disappears:</p><pre>UPDATE users SET balance = balance - $1 WHERE id = $2 AND balance &gt;= $1</pre><p>Then a detail I didn&#39;t get to immediately. Cases can be opened in batches of up to ten, and the whole batch is one transaction: either all of them are paid for and all are granted, or nothing happens at all — including the nonce counter. You cannot increment nonce in a loop: a parallel upgrade or an opening from another tab will wedge into the middle and take a number out of your batch, and nonces have to be contiguous or the history stops recomputing in order. So the whole block is reserved with one UPDATE:</p><pre>// Reserve the entire block of nonces with a single UPDATE.<br>const seedRows = await tx.$queryRaw`<br>  UPDATE server_seeds<br>  SET nonce = nonce + ${count}<br>  WHERE &quot;userId&quot; = CAST(${userId} AS uuid) AND &quot;isActive&quot; = true<br>  RETURNING id, seed, &quot;seedHash&quot;, nonce<br>`;<br>// RETURNING gives the already-incremented value — our block is the last `count` numbers before it.<br>const firstNonce = serverSeed.nonce - count + 1;</pre><p>Two small things on top. A batch writes <strong>one</strong> ledger row, not ten: reconciliation looks at sums, and ten rows for one click only clutter the history. And the rate limiter counts <strong>cases, not requests</strong> — one press of &quot;×10&quot; is ten openings, and script protection has to see it that way.</p><h3>6. Steam: the ten problems that cost me the most time</h3><p>This is the part that isn’t collected anywhere, and it’s where half the project goes.</p><ol><li><strong>Steam has no OAuth.</strong> Only OpenID 2.0. Redirect to steamcommunity.com/openid/login, come back with parameters, and then a <strong>mandatory server-side verification</strong> via check_authentication. Without it the parameters are trivially forged — that step isn&#39;t a formality, it <em>is</em> the security of the login.</li><li><strong>OpenID returns only a SteamID64.</strong> No nickname, no avatar. Those are separate calls.</li><li><strong>Two sources for the profile, and the second isn’t a hack.</strong> GetPlayerSummaries needs a STEAM_API_KEY (100k requests/day per key); /profiles/&lt;id&gt;?xml=1 needs nothing and returns the same thing. Web API is the main path, XML is the fallback: if the key expires or the Web API is down, users still see their own name and avatar instead of user_123456.</li><li><strong>Parse that XML carefully.</strong> It contains nested friend lists with their own &lt;steamID&gt; and &lt;avatarFull&gt;. A naive &quot;take the first match&quot; gives somebody else&#39;s avatar. Debugging that is a treat.</li><li><strong>Steam never returns an email.</strong> Not by any path. There isn’t one until the user types it. Every account-recovery flow has to be built around the Steam account itself.</li><li><strong>Market search doesn’t accept </strong><strong>market_hash_name.</strong> The | character and the exterior parentheses return zero results even for items that definitely sell. The name has to be normalised into keywords.</li><li><strong>Knives and gloves carry a </strong><strong>★ prefix.</strong> Karambit | Doppler (Factory New) does not exist as far as Steam is concerned. ★ Karambit | Doppler (Factory New) does. Without the star, neither price nor image resolves.</li><li><strong>Two endpoints with different properties.</strong> market/search/render returns name, image, rarity and listing count — and <strong>always in dollars</strong>, ignoring the currency parameter. market/priceoverview returns only a price, but respects currency and demands an exact market_hash_name. Hence the split: search for picking items and metadata, priceoverview for any price in the project&#39;s currency.</li><li><strong>Take the median, not the minimum.</strong> lowest_price jumps around on single dumped listings, and a case priced off it undervalues its own contents.</li><li><strong>The limit is roughly 20 requests per minute.</strong> Past that: 429 and a few minutes of ban. So requests are serialised with a 3.5-second interval, search is cached for an hour, prices for half an hour, and a 429 puts the service to sleep for five minutes. <strong>Negative responses are cached too</strong> — otherwise an item with no listings goes back to Steam on every sync.</li></ol><p>Because of item 10, the command that populates a catalogue of 20 themed cases honestly warns you it will take about ten minutes. In exchange, the contents aren’t hardcoded by name: items are picked via market search, and every one of them is guaranteed to have an image and a real price.</p><h3>7. Display currency is not accounting currency</h3><p>The header has two switches: <strong>RU / EN</strong> and <strong>₽ / $</strong>. There’s no default language — it comes from the browser, currency follows language, and both choices are remembered.</p><p>The key rule: the currency switch is <strong>display only</strong>. Everything is computed and stored in roubles, as integer minor units, and <strong>no ledger row ever contains a converted amount</strong>. The dollar figure is that same number divided by the rate at render time.</p><p>Why this deserves its own section: store a converted value once and you have two truths about one amount, and after the rate moves, the site starts selling items below cost. The rate comes from the Russian Central Bank’s public daily feed (no key, no quota, refreshed every six hours, cached in Redis); if the feed is down, the previous rate is served — <strong>a stale rate is better than a broken price list</strong>.</p><h3>8. The bot farm and withdrawals</h3><p>One bot is a separate Steam account with the mobile authenticator enabled, whose shared_secret and identity_secret come from its maFile (without them, offers can&#39;t be confirmed automatically). Secrets aren&#39;t stored in plaintext: AES-256-GCM under a key from the environment in dev, an external secret store in production.</p><p>The whole withdrawal design is shaped by four Steam constraints: <strong>an inventory holds 1,000 slots</strong> per account (so there are several bots, and a request must be routed to the one that actually holds the item); <strong>a trade hold of up to 15 days</strong> if the recipient has no mobile authenticator or enabled it recently (check this <em>before</em> sending the offer, or the item sits in escrow); a bot that recently changed its password or device goes into hold by itself; and Valve rate-limits offer creation, so a throttled queue is mandatory.</p><p>Two places where a mistake costs you an incident. First: the whole chain is idempotent on withdrawalId — a job retry without idempotency is an item handed out twice. Second: the trade URL is validated against the account — partner in the link is the low 32 bits of the SteamID64, and someone else&#39;s link is rejected before an item travels down it.</p><h3>9. The reel decides nothing</h3><p>The strip of items decelerates under the marker exactly as it does in-game. The important property: <strong>the outcome is already known</strong> — the server returned it together with the roll, and the animation only plays the result back. You neither need nor are able to “adjust” the animation for fairness: the roll was computed from the seed pair before the strip started moving.</p><p>Three things I had to learn by hand:</p><p><strong>Two frames, not one.</strong> The first frame gives the browser the starting position with no transition; the second gives the target with the transition. In a single frame the browser collapses both styles and nothing scrolls at all.</p><p><strong>A small random offset within the tile.</strong> Landing dead-centre every time looks fake. But the spread has to be narrow: too wide and the marker lands on a seam between tiles, and it’s unclear which of the two you won.</p><p><strong>Eager image loading.</strong> The strip is around 8,900 px long and the winner sits at the very end of it, far outside the preload window for lazy images. On a cold cache, the tile under the marker would arrive <em>after</em> the strip stopped. Here eager costs nothing: 60 tiles are drawn from the case pool and repeat a dozen identical URLs.</p><p>And when you open ten cases at once, the strips stop <strong>one after another</strong>, and losing tiles <strong>dim</strong> once stopped: with ten strips, a single highlight isn’t enough — the eye loses the winner anyway.</p><h3>10. The trap that made the database seed stop creating cases</h3><p>Originally db:seed created items <em>and</em> cases, with prices eyeballed by me. That turned out to be a trap in two parts.</p><p>First: after the very first sync, real Steam prices differed by an order of magnitude and the cases went underwater — meaning the site pays players more than it takes in.</p><p>Second, and worse: re-running the seed silently overwrote edits made in the admin panel.</p><p>Now db:seed creates <strong>only a demo admin and their seed pair</strong>. The catalogue is populated by a separate command that talks to Steam and assembles cases exactly the way a human would in the builder: auto-balancing to a target RTP and saving through the same validation. No &quot;special&quot; write paths that bypass business logic — which is probably the single biggest architectural lesson of the whole project.</p><h3>11. What checks it</h3><p>Unit tests cover what’s worth checking in isolation: provably fair, ticket ranges, balancing, upgrade odds, parsing of market and profile responses.</p><p>The most valuable thing, though, is a smoke test against the live API. It checks things unit tests cannot catch in principle:</p><ul><li>that balance debits are atomic;</li><li>that the active server seed <strong>never</strong> leaks into a response;</li><li>that an opening recomputes correctly after seed rotation;</li><li>that the balance reconciles with the ledger;</li><li>that roles actually close off the admin panel;</li><li>that every item in an active case has an image and a Steam-confirmed price;</li><li>that the server <strong>refuses</strong> to save an unprofitable case, or one with a hole in its ticket ranges.</li></ul><p>That last one is my favourite. A test that proves the system can say “no” is worth five happy-path tests.</p><h3>What’s in it and what isn’t</h3><p><strong>Works:</strong> Steam OpenID login with server-side verification, RU/EN interface with a currency switch, case opening (including batches of ten) with animation, provably fair with in-browser re-verification, upgrade, inventory and sell-back, a live drop feed over Redis Pub/Sub batched every 300 ms, withdrawal requests with item locking and idempotency, the bot worker, an admin panel with a GGR dashboard, a case builder and an audit log, nightly balance-versus-ledger reconciliation.</p><p><strong>Not there:</strong> payments, item deposits, case battles, contracts, promo codes, KYC.</p><pre>cp .env.example .env   # nothing to edit for a local run<br>pnpm setup             # install + docker + build + migrations + seed<br>pnpm dev               # api on :4000, web on :3000</pre><p>Code, an architecture document and a three-command setup: <a href="https://github.com/ialakey/caseforge"><strong>github.com/ialakey/caseforge</strong></a> · MIT.</p><h3>What to take away</h3><p>I went into this industry expecting to find cheating in the roll. It isn’t there — and not because everyone’s honest, but because it isn’t needed: cryptographically verifiable fairness coexists very comfortably with 85% RTP.</p><p><strong>If you play rather than write code.</strong> “Provably fair” answers exactly one question: <em>was this roll tampered with?</em> It says nothing at all about whether the game is good for you. RTP answers that, and its value is usually printed somewhere near the button — if it’s printed at all. A “provably fair” badge and your negative expected value are entirely compatible. That isn’t a bug, it’s the design.</p><p><strong>If you write code.</strong> The interesting part here was never the game logic. It’s the layer between you and Steam: OpenID with no OAuth, a market with two endpoints in two different currencies, stars in knife names, 1,000 inventory slots and trade holds. Exactly the case where “just wire up the API” takes longer than the entire product around it.</p><p><strong>Where else have you seen a verifiable mechanism used to imply a guarantee it never actually made?</strong> I read the responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bf3a252876b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Connection Is Not Available” Is a Symptom, Not a Diagnosis]]></title>
            <link>https://medium.com/@alakov.ilia/connection-is-not-available-is-a-symptom-not-a-diagnosis-ae82ba3a0b4a?source=rss-73cf176bb3f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae82ba3a0b4a</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Ilia Alakov]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 13:01:03 GMT</pubDate>
            <atom:updated>2026-09-17T13:01:03.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2nwm1sh2uwtL-rM5Rngjg.png" /></figure><p><a href="https://medium.com/@alakov.ilia/explain-describes-a-query-production-is-everything-else-thats-running-6b307b63c330">I wrote up three PostgreSQL incidents</a> recently — bloat, a lock queue, a deadlock — all on the database side. Several people replied with the same thought: <em>fine, that’s the database. What was the application doing? The pool must have been on fire too.</em></p><p>It was.</p><p>This is the same post-mortem one layer up, about <strong>HikariCP</strong> — the default connection pool in Spring Boot, the one that just works until it doesn’t. Same two projects: a banking system after <a href="https://medium.com/@alakov.ilia/a-terabyte-of-banking-data-from-oracle-to-postgresql-with-zero-downtime-bea2c701085d">its migration off Oracle</a> (a terabyte, 70+ tables, 500–3,000 RPS on reads and 50–300 on writes at peak), and a Java/Spring enterprise platform running PostgreSQL under Hibernate. Different teams, different load profiles, and every time the same opening scene:</p><pre>HikariPool-1 - Connection is not available, request timed out after 30000ms.</pre><p>A thread waited thirty seconds for a free connection, didn’t get one, and died. Which reads like a capacity problem, so the first thing anyone does is go and raise maximumPoolSize.</p><p>That’s the thesis of this article: <strong>Connection is not available is not a diagnosis, it&#39;s a symptom class.</strong> Three incidents below produce that identical line, and they have three unrelated causes. In none of them was the pool too small. In one of them the pool was too <em>large</em>, and enlarging it is what broke production.</p><p><em>Disclaimer: both projects are under NDA. Names, exact volumes and some details are changed. Orders of magnitude, configs and the incidents themselves are real.</em></p><h3>Incident 1: we raised the pool and it got worse</h3><p>Starting with the counterintuitive one, because this is where most people’s intuition breaks.</p><p><strong>Symptom.</strong> Banking system, weekday, peak load. The service starts returning timeouts, and the log shows the line above. Hikari’s default maximumPoolSize is 10 and we were running six application instances, so we raised the pool from 10 to 50 per instance.</p><p>It got <strong>worse</strong>. Latency rose, throughput dropped, and on the PostgreSQL side we now had close to three hundred connections (six instances × fifty), of which a handful were doing anything at all. The rest sat in active, competing for the same pages, the same buffers, the same locks. A database that had comfortably served 3,000 read RPS started choking.</p><p><strong>What was actually happening.</strong> A connection pool is not a pipe, where a wider pipe carries more water. Every active PostgreSQL connection is a separate backend <em>process</em> on the database server, fighting for CPU, for shared_buffers, for locks. When the database has 16 cores and you send it 300 concurrent queries, those 300 do not run in parallel — they run in groups of roughly core-count, and the rest wait in a queue <em>inside</em> the database, where you can neither see nor control them.</p><p>Now let’s do the arithmetic nobody does before touching the setting. 3,000 read RPS sounds like “we need lots of connections”. But after we’d fixed the bloat and the plans from the previous article, the average query took about <strong>2 ms</strong>. One connection at 2 ms per query serves roughly 500 queries per second. Three thousand reads per second therefore needs <strong>six or seven</strong> continuously busy connections, plus headroom for write peaks (50–300 RPS) and for variance in response time.</p><p>Not three hundred. Not even fifty.</p><p>HikariCP’s own wiki has a page on this, with a starting-point formula:</p><pre>connections = (core_count * 2) + effective_spindle_count</pre><p>For our 16-core database server on SSD that’s <strong>roughly 20–35 connections for the entire application cluster</strong>, not 300. In HikariCP’s own benchmark, 10,000 front-end users are served happily by a pool of about ten — because the actual work of a query is measured in milliseconds.</p><p><strong>How we fixed it.</strong> We did the exact opposite of the first instinct: we <strong>shrank</strong> the pool. From 50 back to 20 per instance… and even that turned out to be generous, so we settled at 15. Plus we fixed a couple of slow queries that held connections longer than they needed to (hello again, EXPLAIN). The timeouts went away, latency returned to its usual ~300 ms on the dashboard, and the database stopped choking.</p><p><strong>The conclusion that cost us a week.</strong> If you’re running out of connections under load, first find out <strong>why your queries hold connections for so long</strong>, and only then touch the pool size. Nine times out of ten the problem isn’t that the pool is small — it’s that connections aren’t coming back in time.</p><p>Which brings us to incident 2.</p><h3>Incident 2: a connection left the pool and never came back</h3><p>This one is a direct relative of the lock incident in the database article. There, one root cause — a heavy external call inside @Transactional — surfaced as a lock queue on the loans table and a forty-minute idle in transaction. Here the same class of bug showed itself differently: the pool simply emptied. One mistake, two symptoms, depending on which side you&#39;re standing on.</p><p><strong>Symptom.</strong> Enterprise platform, document generation module. The same Connection is not available, but intermittent: fine during the day, degrading during reporting hours, then recovering on its own. A restart buys a few hours. The textbook picture of a leak.</p><p><strong>Where we looked and what we found.</strong> First thing: turn on leak detection. It’s one line, and I now add it to every project on day one:</p><pre>spring:<br>  datasource:<br>    hikari:<br>      leak-detection-threshold: 20000   # 20 seconds</pre><p>The log immediately filled with Apparent connection leak detected and a stack trace, pointing straight at the <a href="https://medium.com/@alakov.ilia/apache-poi-is-a-parser-not-a-renderer-5b9cd8193624">document conversion path</a> — the one I&#39;ve written about separately:</p><pre>@Transactional<br>public void buildReport(Long reportId) {<br>    Report report = repository.findById(reportId);<br>    // ... and here, a render through LibreOffice in headless mode<br>    byte[] pdf = converterService.docxToPdf(report.getTemplate()); // up to 30–40 seconds<br>    report.setRenderedPdf(pdf);<br>    repository.save(report);<br>}</pre><p>See it? The method is @Transactional. That means a connection is taken from the pool at the <em>start</em> of the method and held until the end of it — until the transaction commits. And in the middle sits a synchronous call into LibreOffice headless, which on a heavy template and a bad day spent 30–40 seconds rendering. For all of that time the PostgreSQL connection is <strong>checked out and doing nothing</strong>, waiting for a subprocess.</p><p>During reporting hours, dozens of these ran at once. The pool drained not because there were many database queries, but because each connection was idling inside somebody else&#39;s I/O. With a pool of 15, fifteen concurrent report generations were enough to take production down.</p><p><strong>How we fixed it.</strong> Exactly as on the database side: split the database work and the heavy I/O into separate transactions, and where that&#39;s architecturally impossible, move to an outbox.</p><pre>public void buildReport(Long reportId) {<br>    Report report = readReport(reportId);                          // short transaction: read<br>    byte[] pdf = converterService.docxToPdf(report.getTemplate()); // render — no transaction, no connection<br>    saveRenderedPdf(reportId, pdf);                                // short transaction: write<br>}</pre><p>The connection is now checked out for the milliseconds of a read and the milliseconds of a write, instead of the full forty seconds of a render. The pool stopped draining. On the database side the seatbelt from the previous article was already in place:</p><pre>ALTER ROLE app_backend SET idle_in_transaction_session_timeout = &#39;5min&#39;;</pre><p>It would have killed the forgotten transaction after five minutes anyway — but the pool had already degraded by then, which is exactly why the fix had to be in the code and not in a database timeout.</p><p><strong>Conclusion.</strong> Nothing inside @Transactional should ever wait on a network, a disk, an external process or a human. No HTTP calls, no rendering, no file processing &quot;while we hold the transaction&quot;. A pooled connection is the most expensive resource in that chain. And turn on leak-detection-threshold now, before the incident: it costs nothing and will one day save you a night.</p><h3>Incident 3: the pool is full, the connections are alive, and every Nth one fails</h3><p>This was the nastiest of the three, because every metric said everything was fine. And it happened in the place where the network is guarded most seriously — the banking perimeter.</p><p><strong>Symptom.</strong> Every few minutes, a random request failed with a network error:</p><pre>org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.<br>Caused by: java.net.SocketException: Connection reset</pre><p>The pool was full. Free connections were available. The database was alive, the network was alive. Some of the connections in the pool were simply… dead. Hikari handed one to the application, the application tried to send on it, and got Connection reset. Nights were worse, when traffic in that region dropped and connections sat idle for long stretches.</p><p><strong>What was happening.</strong> Between the application and PostgreSQL sat a firewall that <strong>silently dropped TCP connections idle for more than 15 minutes</strong>. No RST, no polite close — it just removed the connection from its state table. From the application&#39;s point of view, the connection was still perfectly alive.</p><p>Now look at HikariCP&#39;s defaults:</p><ul><li>maxLifetime = 1,800,000 ms = <strong>30 minutes</strong></li><li>idleTimeout = 600,000 ms = 10 minutes</li><li>keepaliveTime = 0 (disabled by default)</li></ul><p>Hikari considered a connection valid for up to 30 minutes. The firewall killed it at 15. In the window between minute 15 and minute 30, the pool was full of connections that no longer existed on the other end.</p><p><strong>How we fixed it.</strong> The rule is simple: <strong>maxLifetime must be meaningfully shorter than the shortest timeout anywhere on the path to the database</strong> — firewall, NAT, load balancer, or PostgreSQL&#39;s own idle_session_timeout. We also enabled keepalive so Hikari pings idle connections itself and doesn&#39;t let the network quietly rot them:</p><pre>spring:<br>  datasource:<br>    hikari:<br>      max-lifetime: 600000       # 10 minutes — below the firewall&#39;s 15-minute limit<br>      keepalive-time: 120000     # touch idle connections every 2 minutes<br>      idle-timeout: 300000<br>      connection-timeout: 10000  # fail in 10s and be visible, instead of 30s of silence</pre><p>We dropped connection-timeout from the default 30 seconds to 10 at the same time: thirty seconds of silent waiting masked the problem, while a ten-second failure lit it up in metrics and alerts immediately.</p><p><strong>Conclusion.</strong> HikariCP cannot read your network topology. If there&#39;s a device between your application and your database that severs idle connections on a timer, you are obliged to tell the pool about it via maxLifetime and keepaliveTime. Otherwise the pool will keep confidently handing out connections that no longer exist.</p><h3>Bonus: HikariCP and PgBouncer in transaction pooling</h3><p>A short trap people hit when they try to solve incident 1 by adding PgBouncer in front of the database.</p><p>If PgBouncer runs in transaction pooling mode, server-side prepared statements break: the underlying connection to the real database changes between transactions, while a prepared statement is bound to one specific backend. The result is intermittent prepared statement &quot;S_1&quot; does not exist.</p><p>Fix it either by moving PgBouncer to session mode (which gives up half the point of a pooler), or by disabling server-side prepared statements in the driver:</p><pre>jdbc:postgresql://.../db?prepareThreshold=0</pre><p>The moral is the same as incident 3: two independent layers that both believe they manage connections must know the other exists. Otherwise they take turns pulling the rug out.</p><h3>What we ended up putting in the default config</h3><p>After all three incidents we have a template that every new service starts from. Not a silver bullet, but a starting point that closes the three classes of problem above:</p><pre>spring:<br>  datasource:<br>    hikari:<br>      maximum-pool-size: 15           # derived from DB cores and query time, not &quot;bigger, to be safe&quot;<br>      minimum-idle: 15                # keep the pool flat, no warm-up dips<br>      connection-timeout: 10000       # fail fast and visibly<br>      max-lifetime: 600000            # below the shortest timeout on the path to the DB<br>      keepalive-time: 120000          # don&#39;t let idle connections rot<br>      idle-timeout: 300000<br>      leak-detection-threshold: 20000 # on from day one, not after the incident</pre><p>And three rules that together matter more than any config:</p><ul><li><strong>The pool isn&#39;t too small because the number is small — it&#39;s too small because connections don&#39;t come back fast enough.</strong> Profile the queries first (3,000 RPS was covered by about ten connections), then touch maximumPoolSize.</li><li><strong>Inside a transaction, only database work.</strong> Any network or disk I/O goes outside it, or into an outbox.</li><li><strong>Synchronise </strong><strong>maxLifetime with the shortest infrastructure timeout.</strong> Firewall, NAT, load balancer, idle_session_timeout — whichever is shortest sets your ceiling.</li></ul><p>The unifying idea, if you want one sentence to take away: a connection pool&#39;s real capacity isn&#39;t measured in connections, it&#39;s measured in connection-<em>time</em>. Almost everything that looks like &quot;we need a bigger pool&quot; is actually &quot;something is holding a connection for far longer than it does useful work&quot;.</p><p><strong>What&#39;s your </strong><strong>Connection is not available story — and what did it actually turn out to be?</strong> I read the responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae82ba3a0b4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>