Batch the Work, Not the Requests: Async Batcher 0.3.0

One caller, one item, one result. Async Batcher groups nearby requests into one bulk operation, so your model or database does less repeated work.

Four requests, one function call, four separate results.

A machine learning model can process many inputs in one call. But users usually send one request at a time, and each request expects its own answer.

Async Batcher solves this problem. Each caller sends one item and waits for one result. Behind that call, the batcher groups nearby items and sends them to your function as one batch:

result = await batcher.process(item)

The client code stays the same. Only the service makes the batches.

I started the project in February 2024. Version 0.3.0 is now available, so this is a good time to explain the idea, the tradeoffs, and the results so far.

Why I built it

My starting point was machine learning inference. If a service sends each request to the model separately, it repeats the same overhead for each request. That costs money and wastes compute.

A model can support batches, but independent requests do not arrive together. Something must collect them, call the model once, and return the correct prediction to each caller.

Energy use was also important to me. In France and in Europe, environmental responsibility is a strong priority, and that influenced how I thought about the problem. Better use of compute can reduce both cost and energy demand.

Lower energy use is a goal, not a promise. The actual saving depends on the model, the hardware, the traffic, and the batch settings. You must measure it for your workload.

The first example served a TensorFlow model with FastAPI. One endpoint called the model for each request. A second endpoint sent each input to a batcher, which collected inputs before it called the model.

In both endpoints, each HTTP request waited for its own prediction. That was the interface I wanted: each caller asks for one result, and the service shares the work behind it.

Not only for machine learning

The same problem exists outside inference:

• A database can accept many writes in one transaction.
• An API can accept many records in one call.
• A message client can send many messages in one operation.

In each case, every caller still needs to know what happened to its own item. The early project description already listed database operations and messaging next to inference.

How a request becomes part of a batch

Suppose that four requests arrive close together. If the backend has a bulk operation, and the requests belong together, they can share one backend call.

Async Batcher does the coordination in four steps:

1. Each caller submits an item with await batcher.process(item).
2. The batcher collects items until the batch is full or the collection deadline arrives.
3. Your batch function processes the collected items.
4. The batcher sends each result to the correct caller.

Your batch function has one rule: return one result for each input, in the same order. For a write-only operation, it can return None. The function guide gives the details.

The tradeoff: throughput against latency

A short wait can make larger batches and reduce repeated backend overhead. But the wait also adds latency, mostly when requests arrive slowly.

For example, max_queue_time=0.01 gives a collection window of 10 ms. This is not a promise of a response in 10 ms. A request can also wait for free capacity and for the backend.

Try it: a small example

Async Batcher 0.3.0 requires Python 3.11 or later. The core package has no runtime dependencies.

python -m pip install 'async-batcher==0.3.0'

Each caller submits one input and receives its own prediction. Async Batcher combines the four requests into one function call. The arithmetic here stands in for a model. In a real application, replace it with your model’s batch prediction function.

import asyncio
from async_batcher import AsyncBatcher


def predict_batch(values):
    print(f"One call for {len(values)} inputs")
    return [2 * value + 1 for value in values]  # Toy predictor.


async def main():
    async with AsyncBatcher.from_callable(
        predict_batch, max_batch_size=4, max_queue_time=0.01
    ) as batcher:
        predictions = await asyncio.gather(
            batcher.process(1),
            batcher.process(2),
            batcher.process(3),
            batcher.process(4),
        )
        print(predictions)


asyncio.run(main())

The output shows one call for four inputs, and each caller gets its own prediction:

One call for 4 inputs
[3, 5, 7, 9]

Three details make this example work:

• The function runs in a thread. predict_batch is synchronous, so the batcher runs it in a thread executor. Thus, a slow model call does not block the event loop.
• A full batch goes at once. With max_batch_size=4, the fourth item fills the batch, so the batcher does not wait for the 10 ms window.
• Requests must overlap. asyncio.gather submits the four requests together. If you await each request before you submit the next one, you usually get batches of one item.

The context manager processes all accepted work before it exits. For a large file or stream, use the bounded map API. It does not create a task for every item at once.

Get Hussein Awala’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If you upgrade from 0.2.2, read the migration guide first. Version 0.3.0 removes Python 3.10 support and rejects invalid configuration.

What the benchmarks show

Under heavy load, batching kept almost every request and cut p95 latency from 17.5 ms to 2.3 ms. With quiet traffic, it added a few milliseconds of wait. Both results come from the same test.

Database writes under load

On November 7, 2026, I compared three ways to write to SQLite:

• Direct: one transaction for each item
• Small custom batcher: a simple batcher written for the test
• Async Batcher: a development checkout just before the 0.3.0 release

All three used one writer on an Apple M2 Max, batches of at most 32 items, and a 2 ms collection window. Each trial sent requests for three seconds, and each result is the median of three trials. All three rejected requests at the same limit of 128 outstanding requests.

At 12,000 offered requests per second:

• Direct: 9,091 completed requests per second, 24.0% rejected
• Small custom batcher: 11,992 completed requests per second, 0% rejected
• Async Batcher: 11,971 completed requests per second, 0.16% rejected

The number of transactions explains the difference. Async Batcher used about 1,430 transactions for about 36,000 accepted rows. The direct writer committed each row in its own transaction.

Median p95 latency of three trials. Full report on GitHub.

The p95 latency is the time below which 95% of responses complete. In the burst profile, requests arrive at 1,000 per second in groups of 32.

In this test, Async Batcher did not beat the small custom batcher. The value of the library is different: it packages the same coordination with queue limits and lifecycle controls, so you do not write and maintain them yourself.

The first inference test (2024)

On February 18, 2024, I compared the direct and batched endpoints of the TensorFlow and FastAPI example. The test used version 0.1.0 on an Apple M2 Max with 12 Uvicorn workers. Each run lasted seven minutes.

• Direct prediction: 1,374 ms mean, 3,800 ms p95, 41.3% failed requests (109,987 requests)
• Batched prediction: 121 ms mean, 300 ms p95, 0.21% failed requests (133,568 requests)

Failed requests include client timeouts and connection failures. The saved reports give the full data.

This test compares two application designs, not only batching. The direct endpoint blocked the event loop during inference, but the batched endpoint ran inference in an executor. Part of the gain comes from that change, and these are 0.1.0 results, not measurements of 0.3.0.

What matters beyond batch size

A list of items is only one part of the problem. A service also needs limits when requests arrive faster than it can process them. Version 0.3.0 gives you these controls:

• Bounded queues: max_queue_size limits how many items can wait. Callers can wait until the queue has space.
• Concurrency control: concurrency sets how many batches run at the same time.
• Cost budgets: a batch can have a limit on estimated cost, such as tokens or bytes, not only on item count. Ten short texts and ten long documents need very different work.
• Grouped batching: requests that must stay apart go into separate batches, for example different tenants, model versions, or destinations.
• In-flight coalescing: equivalent requests that run at the same time can share one call. It does not cache completed results.
• Statistics and batch events: you can see what the batcher does without logging request payloads.
• Adapters: optional adapters for SQLAlchemy, ScyllaDB, MongoDB, DynamoDB, Kafka, Kinesis and Firehose, Pinecone, Keras, and scikit-learn. The core still works with your own batch function.

The release notes describe these features and the other changes in 0.3.0.

When to use it, and when not to

Async Batcher fits applications where many individual requests can share a useful bulk operation:

• Model serving: combine nearby prediction requests, and return a separate result to each caller.
• Data imports: process a file or stream in bounded batches, and track the result of each item.
• Database and API clients: combine writes, lookups, or records when the backend has a bulk operation.

Batching helps only when the shared operation gives a benefit. If the work cannot share overhead, a queue only adds delay.

Async Batcher coordinates work inside one running process. Your application still needs its own policies for provider rate limits, retries, and recovery after a crash. The provider quotas recipe shows one approach, but it is an example, not a built-in feature.

Before you adopt it, measure. The benchmark guide shows how. Use the same backend and workload, and compare throughput, p95 latency, and failures.

Downloads so far

ClickPy recorded 205,635 downloads of Async Batcher from February 19, 2024 to November 7, 2026. The last month (October 8 to November 7) had 16,562 downloads.

These numbers include mirrors, automated downloads, and repeated installs, so they do not count unique users or companies. They also come from before the 0.3.0 release.

Try it, and tell me what you batch

I love the open source community. I am a member of the Apache Airflow Project Management Committee (PMC), and Async Batcher is another way for me to share something useful and learn from the people who use it.

If this matches a problem in your application, start with one batch function and the 0.3.0 documentation.

If you already use Async Batcher, I want to know what you batch and where it helps. Share your experience in the project discussion.

Links

• GitHub repository
• Documentation for 0.3.0
• Release notes
• Benchmark guide