Building an LLM Inference Simulator

I built a small LLM server that has no model and no GPU, but slows down under load like a real one does.

Most of what we build around LLMs sits in front of the model: gateways, routers, rate limiters, streaming clients and observability dashboards. Testing those layers against real GPUs is expensive. Testing it against a mock is cheap, but a mock either answers instantly at any load or adds a fixed delay or random jitter. It never queues, never runs out of KV cache and never preempts a request halfway through. So the failure modes we need to engineer never show up.

I wanted something in between: an OpenAI-compatible server that is deterministic, runs on a laptop or in CI, and whose time to first token (TTFT) and inter-token latency (ITL) respond to load like a real serving engine’s.

I followed the rule of predicting numbers before running load tests, then measuring the diversion and fixing the fidelity issues. The source code is shared at the end of this article.

Scope: What it is, and what it isn’t

It answers one question: how does my system behave when the model slows down? It does not predict how fast a particular model runs on particular GPUs. Only the real engine, or a simulator built from profiled kernels, can answer that.

A simulator is only useful if you know where to stop trusting it. Therefore, I would like to clarify the fidelity before we go through the architecture.

Architecture: what is said vs when it is sent

A real LLM server answers two things at once: what the model says, and when each token arrives. The simulator splits them.

• The generator (llm/) decides what is said. It counts prompt tokens, matches the last user message against scripted responses in responses.yml, picks a variant deterministically and truncates to max_tokens. It never deals with time.
• The engine (simulator/) decides when each token is sent. It takes the generator's plan (prompt length, output tokens) and the request's arrival time, and yields one TokenEvent per token with its simulated time. It never deals with content.
• The two share only types.py, which imports nothing from the project, so neither layer can reach into the other.
• runtime.py is the composition root. It is the only module that reads the parsed config, and it builds one generator and one engine per model at startup. A broken config stops the server before it accepts traffic.
• Engines report metrics through callbacks wired in runtime.py, so the simulator never imports the metrics layer.

The split is what made the project tractable. The content side stayed a simple YAML lookup while the timing side grew from fixed delays to a full scheduler, without either side changing the other.

The generator orchestrates resulting matcher and tokenizer:

def generate(self, request: GenerationRequest) -> GenerationPlan:
        raw = _last_user_message(request.messages)
        # Matching and selection ignore surrounding whitespace, so "hello\n" and
        # "hello" get the same answer; {last_user_message} keeps the raw text.
        text = raw.strip()

        prompt_tokens = self._tokenizer.count_messages(request.messages)
        # Checked before matching, so rejected requests cost nothing more.
        limit = self._completion_limit(prompt_tokens, request.max_tokens)

        result = self._matcher.match(text)
        seed = self._seed if request.seed is None else request.seed
        digest = _digest(seed, result.entry_id, text)

        choices = result.body.choices
        index = _hash_int(digest, 0) % len(choices)
        variant = choices[index]
        variables = {
            **result.groups,
            "last_user_message": raw,
            "model": request.model,
        }
        tokens = self._tokenizer.encode(_render(variant.parts, variables))
        return GenerationPlan(
            prompt_tokens=prompt_tokens,
            tokens=tuple(tokens),
            finish_reason="length" if truncated else "stop",
            entry_id=result.entry_id,
            variant=index,
        )

The matcher matches the incoming prompt with a deterministic response configuration. The response configuration follows a custom DSL:

version: 1
seed: 42                          # default seed for choosing variants

responses:                        # checked top to bottom; the first match wins
  - id: greeting
    match: { exact: ["hi", "hello"] }
    content: "Hello! How can I help you today?"

  - id: weather
    match: { regex: "weather in (?P<city>\\w+)" }
    variants:                     # one is chosen per request, deterministically
      - "It's sunny in {city} today."
      - "Expect light rain in {city}."

  - id: echo
    match: { starts_with: "/echo" }
    content: "You said: {last_user_message}"

  - id: long-answer
    match: { contains: "explain" }
    synthetic: { tokens: [200, 400] }   # filler text of 200 to 400 tokens

default:                          # required: used when nothing matches
  variants:
    - "I'm a simulated model ({model})."
    - "This is a canned response."

The tokenizer is a regex implementation that enables processing the pattern:

_PATTERN = re.compile(
    r"""
    '(?:[sdmt]|ll|ve|re)            # contractions: 's 'd 'm 't 'll 've 're
    | [ ]?[%(cjk)s]                 # CJK, kana and Hangul, one character per token
    | [ ]?(?:(?![%(cjk)s])[^\W\d_])+  # words (other letters), with an optional leading space
    | [ ]?\d{1,3}                   # numbers, in groups of up to three digits
    | [ ]?[^\s\w]+                  # runs of punctuation and symbols
    | \s+(?!\S)                     # whitespace, leaving the last space to start the next token
    | \s+                           # whitespace left over, such as a newline before a word
    | [\s\S]                        # anything else (e.g. underscores), so encoding is lossless
    """ % {"cjk": _CJK},
    re.IGNORECASE | re.VERBOSE,
)

def encode(self, text: str) -> List[str]:
    return self._PATTERN.findall(text)

def count(self, text: str) -> int:
    return sum(1 for _ in self._PATTERN.finditer(text))

There are three engines behind one interface:

The engine is chosen per model in models.yml, so one server can serve a fixed-rate model and a batched one side by side. The configuration looks like:

hardware:
  h100-80gb:
    hbm_gib: 80
    memory_bandwidth_gbps: 3000
    peak_tflops: 989

models:
  - name: qwen-mock-batched
    context_length: 32768
    tokenizer: regex
    engine: scheduler
    hardware: h100-80gb
    weights:
      parameters_billions: 6.74
      bytes_per_parameter: 2
    kv_cache:
      bytes_per_token: 524288
      block_size: 16
      gpu_memory_utilization: 0.9
      watermark: 0.01
    scheduler:
      max_batch_tokens: 8192
      max_num_sequences: 64
    latency:
      step_overhead_ms: 0
    efficiency:
      compute: 1.0
      memory: 1.0

The engine: one interface, two very different timelines

Every engine does one job: turn a plan into a timed stream of tokens. The interface is small on purpose:

class Engine(ABC):
    """Turns a plan into a timed stream of token events.
    Engines never change what is said, only when."""

    name: ClassVar[str]
    # models.yml sections a model must define to use this engine; checked at startup
    required_sections: ClassVar[Tuple[str, ...]] = ()

    @abstractmethod
    def run(self, plan: GenerationPlan, arrival: float) -> AsyncIterator[TokenEvent]:
        """Yield the plan's tokens in order; deadlines are computed from arrival."""

    async def start(self) -> None: ...   # e.g. launch a background step loop
    async def stop(self) -> None: ...

The two types that cross the boundary carry no HTTP and no config:

• GenerationPlan comes from the generator: prompt_tokens, the output tokens, finish_reason, and which response entry answered.
• TokenEvent goes back to the SSE layer: index, text, sim_time (simulated seconds after arrival when the token is due) and lag (how late it was actually released).

The API layer pulls tokens with async for event in engine.run(plan, arrival) and writes a chunk for each. It doesn't know or care which engine is behind it. When a client disconnects, the API closes the iterator, and the engine's finally: block frees whatever the request held.

Engines take plain parameters, never config classes. runtime.py reads models.yml, checks required_sections, and builds each engine with numbers. That keeps every engine testable with a fake clock and a few arguments.

Fixed-rate: each request owns its timeline

The fixed-rate engine has no shared state. A request’s whole timeline is known the moment it arrives:

def timeline(self, plan: GenerationPlan) -> Tuple[float, ...]:
    """Simulated seconds after arrival at which each token is due."""
    ttft = self._overhead + plan.prompt_tokens / self._prefill_rate
    return tuple(ttft + index * self._itl for index in range(len(plan.tokens)))

async def run(self, plan, arrival):
    for index, (text, sim_time) in enumerate(zip(plan.tokens, self.timeline(plan))):
        lag = await sleep_until(arrival + sim_time, self._clock, self._sleep)
        yield TokenEvent(index, text, sim_time, lag)

That is the whole engine. A thousand concurrent requests means a thousand independent coroutines, each sleeping to its own deadlines. Load changes nothing about the simulated timeline.

Scheduler: one step loop owns the GPU

The scheduler inverts this. No request knows its timeline in advance, because when its tokens come out depends on who else is in the batch. So run() only registers the request and waits on its own queue, and one background loop, started by start(), decides everything:

async def run(self, plan, arrival):
    stream = _Stream(next(self._ids), arrival, plan)
    self._streams[stream.id] = stream
    heapq.heappush(self._arrivals, (arrival, stream.id, stream))
    self._wake.set()                                   # wake the step loop
    try:
        while (item := await stream.queue.get()) is not _DONE:
            index, sim_time, queue_time = item
            lag = max(0.0, self._clock() - (arrival + sim_time))
            yield TokenEvent(index, plan.tokens[index], sim_time, lag, queue_time)
    finally:
        self._policy.abort(stream.id)                  # free its KV blocks if the client left

async def _steps(self):                                # simplified
    end = float("-inf")                                # simulated time the last step ended
    while True:
        # (when idle: wait for a request, then start at its arrival, not now)
        while self._arrivals and self._arrivals[0][0] <= end:
            _, _, s = heapq.heappop(self._arrivals)    # only requests that have arrived
            self._policy.add(s.id, s.arrival, s.plan.prompt_tokens, len(s.plan.tokens))

        plan = self._policy.schedule(end)              # admit, batch, preempt
        cost = self._cost.step_cost(plan.shape)        # roofline: max(compute, memory) + overhead
        end += cost.duration                           # anchored: lateness never shifts the timeline
        step_lag = await sleep_until(end, self._clock, self._sleep)
        await asyncio.sleep(0)                         # an overdue loop still yields

        result = self._policy.complete(plan)
        for seq in result.emitted:                     # one token to every sequence in the batch
            self._streams[seq.id].queue.put_nowait((seq.generated - 1, end - seq.arrival, ...))
        for seq in result.finished:
            self._streams[seq.id].queue.put_nowait(_DONE)
        self._on_step(self._stats(plan, cost, step_lag))   # metrics, via a callback

The output length goes to the policy only so it knows when to stop the sequence, like an end-of-sequence token. Admission never looks at it, because a real server can’t know it.

Both engines share one helper, sleep_until, and that is where the timing model lives.

The timing model: deadlines, not sleeps

The obvious way to fake a token stream is sleep(ttft), then sleep(itl) per token. It has two problems:

1. Overhead leaks into the metrics. Parsing, validation and matching happen before the first sleep, so observed TTFT becomes overhead plus simulated TTFT, and that overhead varies by prompt.
2. Errors accumulate. Each sleep wakes slightly late. Over thousands of tokens the lateness adds up and throughput drifts below the configured rate.

Instead, the middleware stamps arrival as early as possible, and the engine sleeps until absolute deadlines:

deadline_i = arrival + ttft + i * itl
await asyncio.sleep(max(0, deadline_i - time.monotonic()))

Overhead is absorbed whenever it is shorter than the simulated time to the first token, and lateness never accumulates, because every token targets its own deadline.

Lag is the simulator’s error bar. When a deadline has already passed, the token goes out at once and the engine records lag = release time − deadline. Lag measures how faithfully the simulator carried out its own timeline, not anything about the simulated GPU. Since only the last token's lag reaches end-to-end time, late tokens catch up rather than stretch the request.

Time is measured in three places, and each layer measures its own:

The gap between server and client is delivery cost; the gap between simulated and server is the simulator running late. Keeping them apart is what lets every result later in this article say which one it is about.

The scheduler: continuous batching on one simulated GPU

The fixed-rate engine treats every request as if it had a GPU to itself. Real servers don’t. Requests share a batch, compete for KV-cache memory, wait when it is full and get evicted when it runs out. The scheduler engine models that, following vLLM's rules.

KV capacity is computed at startup from the hardware profile and the model:

With 80 GiB of HBM, 90% utilisation, a 6.74B-parameter fp16 model (13.48 GB of weights) and 512 KiB of KV per token, that is 7,609 blocks of 16 tokens: about 121,700 tokens, or roughly 29 concurrent requests at 4k context.

Then, every step:

1. Admit. Waiting requests join in arrival order (preempted ones first) while their whole prompt fits, keeping 1% of blocks free as a watermark, and while fewer than max_num_sequences are running.
2. Batch. Each running sequence contributes one decode token, or its next prefill chunk. New prompts fill the rest of the max_batch_tokens budget. Prefill is always chunked, so a long prompt spreads over several steps while every other stream keeps decoding.
3. Grow. A sequence needs a new block every 16 tokens. If none is free, the newest admitted sequence is preempted: its blocks are freed and it goes back to the front of the queue to recompute everything it had.
4. Duration. The step takes as long as its slowest resource, a roofline: max(compute, memory) + step_overhead, where compute is 2 × parameters × scheduled tokens over peak FLOPS, and memory is the weights plus every sequence's KV over memory bandwidth. Both can be scaled by calibrated efficiencies.
5. Emit. Sleep until the step’s absolute deadline, then release one token to every sequence whose tokens are computed.

One rule I held to: admission never uses a plan’s real output length. The simulator knows it, but a real server cannot, so it is used only to stop the sequence, like an end-of-sequence token.

Get Umang Aggarwal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

How KV pressure reaches the client:

In the example above, one short request decodes at about 4.5 ms per token. Twenty-nine requests at 4k context read about 62 GB of KV per step on top of the weights, so ITL rises to about 25 ms for everyone. Decode is memory-bound; compute takes well under a millisecond.

Code layout. The scheduler has three tiers, with dependencies pointing down only. sequence.py, kv_cache.py and cost_model.py hold resources and cost; policy.py makes the decisions; driver.py is the only code that touches time and asyncio. Because the core is pure, I can run it in virtual time, with no server and no clock, to compute exactly what a load test should produce. That turned out to be the most useful design decision in the project.

Building a benchmark I could trust

My first throughput estimate, 5,500–6,000 output tokens per second, came from an ad-hoc script. It was partly measuring the script. So before load-testing the scheduler I built a load generator, llmsim-bench, that works against any OpenAI-compatible server and checks itself.

Three things mattered most:

• Open loop, not just closed loop. A closed loop keeps N requests in flight, sending the next only when one finishes, so a slow server is offered less load and queueing stays hidden. An open loop sends requests on a seeded Poisson schedule whether or not earlier ones have finished, like independent users.
• Latency from the scheduled arrival, not from the send. If the client caps concurrency, an arrival may wait for a free slot. Measuring only from the actual send hides that wait, a trap known as coordinated omission. In one run at about 82% utilisation, TTFT p99 was 12 ms from the send and 65 ms from the scheduled arrival, because 66 requests had queued for a slot.
• The client checks its own health. A monitor wakes every 10 ms and records how late each wake-up is, plus the client’s CPU. Above 5 ms at p99, the report warns that the client, not the server, inflated the latencies. It caught a real problem in my own tests: with the server in a thread of the test process, both shared the GIL and the client’s loop ran 10 ms late at p99.

A few smaller choices:

• It uses aiohttp, not the OpenAI SDK, which builds a typed object for every chunk.
• Each synthetic prompt is distinct, and exactly N tokens long, so a real server’s prefix cache cannot flatter TTFT.
• After warm-up it resets the server’s /metrics/summary, and after the run it reads it back. Client, server and simulated values then cover the same requests and can sit in one table.

First limits: how far one process can go

Before the scheduler existed, I load-tested the fixed-rate engine to find how much traffic one Python process can stream before it stops keeping time. The simulated TTFT was 7.12 ms and TPOT 4.00 ms; the question was how far the observed values drift from those.

Up to 200 requests per second, about 13,000 output tokens per second, lag p99 stays under 4 ms. Past that, the TTFT tail grows first (p99 49 ms at 250 req/s), and by 350 req/s the median falls behind too. Late tokens then go out back to back, so TPOT reads below the configured 4 ms while end-to-end time stays within 1–4% of schedule.

Two fixes got it there:

• uvloop and httptools. uvicorn’s defaults are a pure-Python HTTP parser and the standard asyncio loop. Switching to uvicorn[standard] cut mean lag at 200 streams from 122 ms to 18.6 ms, and mean TTFT from 183 ms to 56 ms.
• A regex pre-filter. With a realistic responses.yml, matching every prompt against every pattern collapsed the server at about 60 req/s. Matching held the GIL for about 88% of the time at 60 req/s and stalled the event loop. The fix parses each regex once at load time for the plain strings every match must contain, then skips the regex when the prompt lacks them, with a substring test. Matching a 512-token prompt fell from 10.7 ms to 0.47 ms, and the server held 300 req/s at +2.6 ms median TTFT.

The takeaway: generating a token was never the bottleneck. In-process, the engine releases a token in about 0.7 µs, and tracing plus SSE serialisation bring it to about 9 µs. Delivering it over HTTP costs about 170 µs. A real GPU replica produces a few thousand to low tens of thousands of tokens per second, so one process can stand in for one replica; beyond that, run more processes, as you would run more servers.

Validating the scheduler: predict, then measure

Six of seven experiments passed, most within 0.1–3% of a number written down before the run.

A load test that only describes what happened can’t tell you whether the simulator is right. So each experiment checks one scheduler decision against a prediction, with four rules:

1. Predict before running. Each prediction comes from the cost model and the scheduler’s rules: a formula when one is enough, otherwise the pure scheduler core run step by step in virtual time.
2. One variable at a time. Each test model differs from the base model in exactly one field, and a unit test checks that it does.
3. When a prediction fails, stop. Work out whether the prediction or the simulator is wrong, and never adjust a prediction to fit a result.
4. An explanation is not proof. An explanation that fits the numbers stays marked as inferred until a control experiment removes the suspected cause.

The base model is the 6.74B one from the scheduler section: 13.48 GB of weights, 512 KiB of KV per token, 3,000 GB/s and 989 TFLOPS. Every response has an exact length, so --max-tokens sets output length precisely.

What each passing result means for someone testing a gateway:

• Batching (E1). Sixty-four streams get 38× the throughput of one, for only 1.56× the step time, because decode is memory-bound and the weights are read once per step for the whole batch.
• Context (E2). At the same batch size, 16× the prompt length makes every stream’s step 2.8× slower. Long-context users slow everyone down.
• Chunked prefill (E3). One long prompt arriving among 16 decoding streams stalls them all for one step. Cutting the token budget from 8,192 to 2,048 lowered that stall from 112 ms to 28 ms, and served the same prompts and decode tokens in the same time. This is the trade-off behind vLLM’s max_num_batched_tokens.
• KV admission (E4). With a small cache and 2,000-token prompts, the KV cache, not the batch limit of 64, decides how many run: 9. TTFT becomes almost all queueing, 4.46 s of 4.57 s.
• Preemption (E5). Sixty-four requests that outgrow the cache together force about 130 preemptions. The 1% watermark cut them by 9% at no cost in time. The median ITL was 7.9 ms, but a few preempted streams froze for up to 50 s. That is exactly the case a gateway’s timeouts should be tested against.

What the predictions missed

The misses taught me more than the passes. Two stood out.

1. Fixed-length benchmarks phase-lock. In E1, step times matched, but simulated TTFT grew to about 100 ms at 64 streams with nothing blocked. Every request had the same length and the loop was closed, so streams that started together finished together and sent their next requests together. About every 200 steps, one step prefilled most of the batch’s new prompts at once. The largest observed steps were 74–88% of a fully synchronised prefill at every level.

A real engine under the same benchmark would phase-lock too. It suggests that fixed-length, closed-loop benchmarks, a very common setup, overstate TTFT compared with real traffic. For now this is inferred, not proven. If phase-locking is the cause, TTFT p50 should drop to about 10–15 ms.

2. My own request handling leaked into simulated time. E7 swept open-loop load from 10 to 100 req/s. Batch size and step time matched at every rate, but two results failed:

• TTFT was 3.5–5 ms higher than predicted at every rate. The arrival is stamped before the body is parsed and the generator runs. An idle engine absorbs that time; a busy one can’t add a request to a step that is already running. So the processing time, rounded up to the next step boundary, shows up as queue time.
• Capacity was 9% lower: 74 req/s against 81.7. New prompts reached the engine about three at a time instead of one per step. One 512-token prompt riding with 64 decodes is free: its 7.9 ms of compute hides under the step’s 10.6 ms of memory time. Three together make the step compute-bound at 21.8 ms. Feeding the virtual run the same arrivals in 20–30 ms bunches reproduced the real run on every measure, capacity 70–77 req/s, which confirms the bunching is enough to explain the gap.

Neither is a scheduling bug. Both are the cost of the HTTP process in front of the scheduler, which is exactly the part of the system I had not yet optimised.

Where the time goes

With the scheduler, lag grows with batch size, not with throughput: token lag p99 is about 1.7 ms plus 0.15 ms per sequence in the batch.

At 64 streams that is only 3,100 output tokens per second, a quarter of what the fixed-rate engine sustains within the same 5 ms bar. The cause is the shape of the work. The fixed-rate engine spreads each request’s tokens over its own timeline. A scheduler step releases one token to every sequence at the same instant, as a real engine does, and one event loop then delivers them one after another. Tokens late in that queue show it as lag.

Simulated values are unaffected. The timeline is anchored, so step times and simulated TTFT and E2E are exact at every level; observed ITL p50 matched the simulated step time (16.72 vs 16.82 ms at 64 streams). Lag only adds jitter to what clients observe.

To see where each token’s CPU goes, I profiled 64 streams with py-spy:

One lesson about the instrument itself. py-spy’s default mode pauses the process for every sample, and under this load it inflated CPU per token 1.7× and doubled the step loop’s share. I trusted the profile only after it matched two independent measurements: process CPU from /proc (89–91 µs per token) and timers around the step loop inside the server (13.4 µs).

The breakdown also sets realistic expectations for the fix. Prebuilt SSE templates (about 22 → 3 µs) and lighter per-token metrics (about 15 → 7 µs) bring the total to about 60 µs.

The socket write, uvloop and the task switch, about 25–30 µs together, remain. So lag p99 should roughly halve, and observed timing should stay within 5 ms up to about 40–50 streams instead of 25.

Known limits and what’s next

On the same workload (512-token prompts, 64 output tokens), one process keeps observed timing within 5 ms of the simulated timeline up to:

• about 200 req/s with the fixed-rate engine, about 13,000 output tokens per second;
• about 25 req/s with the scheduler, a batch of about 10 and 1,675 output tokens per second.

Beyond that, simulated step times, batches and preemptions stay right, but observed per-token times carry the process’s own overhead, and the bench report flags it. Lag is always measured, so a result never silently drifts outside the envelope.

Next, I would study:

1. Running an experiment to confirm or rule out phase-locking.
2. Prebuilt SSE templates and lighter per-token metrics, measured the same way as the breakdown above, with a test that the bytes sent stay identical.
3. gc.freeze() after startup. One full garbage collection took 13.7 ms, the largest lag outlier.
4. Further out: prefix caching in the scheduler, tensor parallelism, several processes as several replicas, and a virtual-time mode for capacity planning without real sleeps.

Try it

docker/bench.sh

That builds the images, starts a simulator and runs a rate sweep against it, writing JSON results to results/. Point any OpenAI client at http://localhost:8080/v1 and it works unchanged.

The takeaway: the scheduler’s rules were the easy part to get right; the predictions matched within a few percent. What the predictions missed was the HTTP process in front of it, the same layer real serving stacks spend so much effort on.

A fun exercise in writing the number down before putting load on the system, and then finding out why it was wrong.

Source Code & Full Results:

GitHub - umangaggarwal2/llm-simulator: A deterministic fake LLM that streams tokens through OpenAI…