Who froze the event loop?

In an asyncio server, every request shares one event loop. So when a new request spends about 80 ms building TLS contexts and loading a tokenizer, everyone else’s stream just stops for that time. I reproduced this with a small streaming gateway, caught the loop in the act with a watchdog thread, and moved the setup to startup. Before, two out of every three new requests froze the loop. After, none did, and the slowest chunks went from about 240 ms late to under 20. The part that surprised me: the stack traces blamed the tokenizer, when the TLS setup was costing almost as much.

If you run an asyncio service that streams to many clients at once, like an LLM gateway pushing tokens, every client shares one event loop. When a new request does 100 ms of synchronous setup, every stream on that process stops for 100 ms, including the ones that were halfway through a reply.

Four of the 40 streams, all on one event loop. One new request starts its setup and nobody can send anything. When it finishes, every stream fires off a late chunk at the same time.

I built a small gateway to show this and to measure the fixes. The code is here: GitHub gist. 40 clients hold open token streams, each expecting a chunk every 20 ms, while a new request arrives every 250 ms on average. Each new request does setup that looks harmless in code review.

In the naive version, the loop spent about a third of the run stuck in stalls longer than 50 ms. The slowest 1% of chunks arrived 227 to 249 ms late. After the fixes the watchdog caught no stalls at all, and the slowest 1% of chunks arrived 11 to 19 ms late.

How late the slowest chunk was, every tenth of a second, over a 20-second run. Each orange spike is a new request starting up. The blue line is the same traffic after the fixes.

All numbers above are from four runs on an M4 Pro with Python 3.14.

What each request did on the loop

The median request spent 81 to 90 ms in setup before streaming anything:

Step                                                  Median per request
Creating 3 TLS contexts (LLM, moderation, tracing)    27 to 28 ms
Loading a 200k-entry tokenizer from JSON              34 to 36 ms
Compressing a 1 MB payload already compressed         13 ms
Scoring 400 retrieval candidates in pure Python       6.5 ms

The TLS line surprises people. ssl.create_default_context() reads and parses the whole CA bundle every time you call it, and httpx, for one, builds a new context for every client unless you pass your own. If you build clients per request, you pay that per request.
None of these steps is slow by itself, and that’s why they’re easy to miss. Each one is fine when you test one request at a time. They add up only when many requests share a loop.

A lag graph tells you it’s happening

The usual first tool is a loop-lag monitor, schedule a tick, measure how late it fires. That shows you the stalls exist. It can’t show what was running, because by the time the late tick fires, the blocking code has returned and its stack is gone.

You can see the stack from outside the loop while it is stuck. A Python thread can read another thread’s current frame with sys._current_frames(), so a separate watchdog thread that notices the loop is overdue can grab the loop's stack while the blocking code is still on it:

import asyncio
import sys
import threading
import time
import traceback


class LoopWatchdog:
    """Create on the event-loop thread. That thread is the one sampled."""

    def __init__(self, loop, tick=0.5, margin=0.1, poll=0.025, report=print):
        self.loop, self.report = loop, report
        self.tick, self.limit, self.poll = tick, tick + margin, poll
        self.loop_thread = threading.get_ident()
        self.heartbeat = self.seen = time.monotonic()
        self.stacks = []
        threading.Thread(target=self._watch, daemon=True).start()

    async def beat(self):
        # The loop's only cost: one float write per tick.
        while True:
            self.heartbeat = time.monotonic()
            await asyncio.sleep(self.tick)

    def _watch(self):
        while True:
            time.sleep(self.poll)
            beat = self.heartbeat
            if beat != self.seen:  # the loop ticked
                gap, self.seen = beat - self.seen, beat
                stacks, self.stacks = self.stacks, []
                if gap > self.limit:
                    blocked_ms = (gap - self.tick) * 1000
                    self.loop.call_soon_threadsafe(self.report, blocked_ms, stacks)
                continue
            overdue = time.monotonic() - beat > self.limit
            if overdue and len(self.stacks) < 2:
                frame = sys._current_frames().get(self.loop_thread)
                if self.heartbeat == beat:  # still stuck, so the stack is the culprit
                    self.stacks.append(traceback.extract_stack(frame, limit=15))

The loop just keeps writing a heartbeat. When the heartbeat goes quiet for too long, a second thread looks at what the loop is stuck on and grabs its stack. It reports back once the loop is moving again.

The loop’s only cost is one float write per tick. The watchdog wakes on a short poll and does nothing unless the heartbeat is overdue, and then it takes at most two stacks per stall. When the loop resumes, it hands the record back with call_soon_threadsafe, so your logging runs on the loop with its normal context.

Two details are easy to get wrong. The heartbeat gap includes the normal tick, so subtract the tick and report only the time the loop was blocked. And keep stalls that end with no stack. An empty stack means the watchdog thread couldn’t run either, which usually means a C extension held the GIL for the whole stall.

Reading the stacks

The watchdog put 67% to 98% of stalls on the tokenizer load. The step timings above say TLS setup costs almost as much. Both are right about what they measure.

One request’s setup, laid out in time. The watchdog takes its snapshot around 40 to 55 ms in, which is right in the middle of the tokenizer load. The TLS work that ran just before never shows up.

A sampled stack shows whatever is running at the moment the stall crosses the threshold. The tokenizer load ran right after the TLS setup, so it was usually the code on the stack at the 50 ms mark, while the 27 ms of TLS work just before it never showed up.

Get Swayam Gupta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Some of the tokenizer’s blame was probably garbage collection. Loading 200k entries allocates a lot of objects, and allocation is what triggers a collection, so a GC pause shows up in the stack of whatever code was allocating when it started. The naive runs’ worst GC pause was 176 to 191 ms.

So treat the watchdog’s stacks as leads to check. I’d log GC time and step timings on every stall record next to the stack, which also lets you separate your own blocking from GC and from container CPU throttling.

The fixes: build once per process, before traffic

Create one TLS context at startup and share it, along with pooled clients, across requests. If your clients keep cookies, give the shared ones no cookie jar so one request’s cookies never reach another.

Load the tokenizer (or model, or index) once per process, not per request. Anything immutable that every request needs belongs in startup.

Move numeric loops to numpy. Scoring 400 candidates went from 6.5 ms to under 0.5 ms.

Stop compressing data that doesn’t compress. Already compressed payloads, images, and raw formats gain nothing from zlib, so the 13 ms was pure CPU cost.

At startup, call gc.collect() and then gc.freeze(). Freezing moves everything alive at that point into a permanent generation the collector never scans again. In an isolated test (gc_probe.py) with a 1.5M object startup heap, the worst GC pause dropped from 155 ms to under 1 ms on Python 3.14, and from 145 ms to 59 ms on 3.12. It isn't free, on 3.14 the median request's bookkeeping took about 5 ms instead of 2 ms. One long pause turned into slightly more steady work, which is the right trade for a streaming service.

The longest the garbage collector froze everything, with and without gc.freeze(). On Python 3.14 it went from 155 ms to under 1 ms.

Before and after

Same arrival times in both modes, 75 requests per run, after four runs:

Metric                                 Naive                Fixed
Stalls per request (heartbeat >50ms)   0.67 to 0.69         0
Median stall                           95 to 116 ms         none
Worst stall                            261 to 277 ms        none
Time the loop spent in stalls          33% to 36%           0%
Chunks more than 50 ms late            7.4% to 7.8%         0%
Slowest 1% of chunks                   227 to 249 ms late   11 to 19 ms late
Median request setup                   81 to 90 ms          11 to 16 ms
Worst GC pause                         176 to 191 ms        3 to 4 ms

What a real server adds

A real service has costs this one doesn’t model. The first request after a process starts often pays for lazy imports and caches that a warm-up at startup would build. Containers get CPU-throttled, which looks exactly like a stall from inside the process. And the right GC settings depend on your heap, so measure before copying gc.freeze() everywhere.

Checklist

If you run streaming or real-time work on asyncio, check for these:

1. HTTP clients or SSL contexts created per request.
2. Tokenizers, models, or indexes loaded inside a request handler.
3. Per-element Python loops over numeric data.
4. Compression on data that is already compressed.
5. A large startup heap with default GC settings.

Then add a watchdog that captures the loop’s stack, and log step timings and GC time next to it, so the stacks have something to be checked against.