Python Threading + FastAPI: Zero to Hero

Threading in Python is the topic most likely to make an experienced engineer say “wait, that’s not what I expected.” This guide covers why — and exactly what FastAPI does with your threads in production.

If you’ve written multithreaded code in Java or C++, Python threading will feel familiar on the surface and behave completely differently underneath. The reason is one object almost every Python developer has heard of and few have internalized correctly: the Global Interpreter Lock (GIL). It shapes every decision in this guide, including one you’re making right now without realizing it — whether your FastAPI routes should be def or async def.

This is a from-scratch guide to Python concurrency, built toward one specific, practical destination: understanding exactly what happens inside FastAPI when a request hits a synchronous route, why the default thread pool has exactly 40 slots, and when that default will quietly become your production bottleneck.

What you’ll walk away with:

• What the GIL actually is, and why it makes CPU-bound threading mostly pointless but I/O-bound threading genuinely useful
• Threads, locks, and race conditions — with the classic lost-update bug made concrete
• ThreadPoolExecutor, the API you should reach for instead of raw threading.Thread
• A clear decision rule for threading vs. multiprocessing vs. asyncio
• Exactly how FastAPI’s run_in_threadpool and AnyIO's 40-token limiter work
• How to monitor and size that thread pool in production instead of guessing

Let’s get into it.

1. The GIL: The One Fact That Explains Everything Else

The Global Interpreter Lock is a mutex that allows only one thread to execute Python bytecode at a time, per process — no matter how many CPU cores the machine has.

import threading
import time

def cpu_bound_work():
    total = 0
    for i in range(50_000_000):
        total += i
    return total

# Two threads doing CPU-bound work
start = time.perf_counter()
t1 = threading.Thread(target=cpu_bound_work)
t2 = threading.Thread(target=cpu_bound_work)

t1.start(); t2.start()
t1.join(); t2.join()

print(f"Two threads: {time.perf_counter() - start:.2f}s")
# On most machines, this takes roughly the SAME time as running
# cpu_bound_work() twice sequentially - sometimes slightly longer.

Why doesn’t this parallelize? Because both threads need the GIL to execute Python bytecode, and only one can hold it at a time — they take turns, switching every ~5ms (sys.setswitchinterval) or on I/O, but never run Python bytecode simultaneously on separate cores.

Get Jitendra Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here’s the part that makes threading worth learning anyway: the GIL is released whenever a thread is waiting on I/O — a network call, a disk read, a time.sleep(). During that wait, other threads run freely.

import httpx, threading, time

def fetch(url):
    httpx.get(url)  # GIL released while waiting for the network

urls = ["https://httpbin.org/delay/1"] * 5
start = time.perf_counter()
threads = [threading.Thread(target=fetch, args=(u,)) for u in urls]
for t in threads: t.start()
for t in threads: t.join()
print(f"5 requests, threaded: {time.perf_counter() - start:.2f}s")
# ~1s total, not 5s - because the GIL was free during each network wait

The rule that follows directly from this: threading gives you real concurrency for I/O-bound work and essentially none for CPU-bound work. Keep that asymmetry in mind for the rest of this guide — it’s the axis everything else is measured against.

2. Creating and Managing Threads

The low-level API is threading.Thread. You'll rarely reach for it directly in production code (Section 4 covers the API you actually want), but understanding it helps the higher-level APIs make sense.

import threading

def worker(name: str, delay: float):
    print(f"{name} starting")
    time.sleep(delay)
    print(f"{name} done")

t = threading.Thread(target=worker, args=("Worker-1", 2.0), daemon=True)
t.start()          # New -> Runnable -> Running

print("main thread continues immediately")
t.join(timeout=5)   # blocks HERE until t finishes (or timeout)
print("main thread resumes after join")

Three details that cause real bugs when missed:

• Forgetting .join() — the main thread can exit before a non-daemon thread finishes, but a daemon thread gets killed abruptly when the main thread exits, mid-work, with no cleanup. Use daemon=True only for genuinely disposable background work (like a heartbeat), never for anything that must complete (like a database write).
• .join(timeout=...) doesn't kill the thread. Python has no built-in way to forcibly terminate a thread — if it's stuck, join() with a timeout just gives up waiting; the thread keeps running. Design for cooperative cancellation (a shared threading.Event the loop checks) instead of assuming you can kill it.
• Exceptions inside a thread don’t propagate to the caller. An unhandled exception in worker() prints a traceback to stderr and silently kills that thread — your main thread has no idea anything went wrong unless you explicitly capture and re-raise it.

3. Race Conditions and Locks

The GIL protects individual bytecode operations, but not multi-step operations like counter += 1, which is actually read-modify-write under the hood — three separate bytecode instructions that can be interleaved between threads.

import threading

counter = 0
def increment():
    global counter
    for _ in range(100_000):
        counter += 1   # NOT atomic: read, add 1, write back
threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counter)   # Expected 400_000. Actual: something less, and non-deterministic.

The fix is a Lock, which forces threads to take turns through the critical section:

counter = 0
lock = threading.Lock()

def safe_increment():
    global counter
    for _ in range(100_000):
        with lock:          # only one thread inside this block at a time
            counter += 1

threads = [threading.Thread(target=safe_increment) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counter)   # Always exactly 400_000

A few related primitives worth knowing, briefly:

• RLock (reentrant lock) — the same thread can acquire it multiple times without deadlocking itself; useful when a locked method calls another locked method on the same object.
• Semaphore(n) — like a Lock but allows up to n threads through simultaneously; useful for capping concurrent access to a limited resource (e.g., "at most 5 threads hitting this rate-limited API at once").
• Deadlock — two threads each holding a lock the other needs. The classic fix is always acquiring multiple locks in the same global order across your entire codebase.

4. ThreadPoolExecutor: The API You Should Actually Use

Manually creating and joining Thread objects doesn't scale past a handful of threads. concurrent.futures.ThreadPoolExecutor is the high-level, production-grade API: it manages a fixed pool of worker threads, queues excess work, and gives you Future objects for clean result retrieval.

from concurrent.futures import ThreadPoolExecutor, as_completed
import httpx

urls = [f"https://httpbin.org/delay/1" for _ in range(10)]

def fetch(url: str) -> int:
    resp = httpx.get(url, timeout=10)
    return resp.status_code

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(fetch, url): url for url in urls}
    for future in as_completed(futures):
        url = futures[future]
        try:
            status = future.result()
            print(f"{url} -> {status}")
        except Exception as exc:
            print(f"{url} raised {exc!r}")

Why this beats raw threads for almost every real use case:

• Bounded concurrency. max_workers=5 means at most 5 threads run at once, regardless of how many URLs you submit — the rest queue automatically. Unbounded Thread creation is how you accidentally spawn 10,000 OS threads and crash a process.
• Exceptions propagate correctly. future.result() re-raises whatever exception happened inside the worker — no more silently swallowed tracebacks.
• executor.map() for the simple case where you just want results in order:

with ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(fetch, urls))

This is also, not coincidentally, extremely close to what FastAPI does internally for synchronous routes — which brings us to the second half of this guide.

5. Threading vs. Multiprocessing vs. Asyncio: Picking the Right Tool

This is the decision that trips people up most, because all three look similar on the surface (“run things concurrently”) but solve completely different problems.

# CPU-bound -> multiprocessing (separate processes, separate GILs, real parallelism)
from concurrent.futures import ProcessPoolExecutor

def crunch(n):
    return sum(i * i for i in range(n))

with ProcessPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(crunch, [10_000_000] * 4))
# This DOES parallelize across cores - each process has its own GIL.

# I/O-bound, blocking library -> threading  
with ThreadPoolExecutor(max_workers=20) as pool:
    results = list(pool.map(fetch, urls))
# GIL releases during each network wait - threads overlap productively.

# I/O-bound, async-native library -> asyncio
import asyncio, httpx

async def fetch_async(client, url):
    resp = await client.get(url)
    return resp.status_code

async def main():
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(*(fetch_async(client, u) for u in urls))
# Single thread, cooperative scheduling - cheapest concurrency of the three.

The rule in one line: multiprocessing buys you real parallelism at the cost of process overhead and no shared memory; threading buys you I/O concurrency cheaply but never CPU parallelism; asyncio buys you the highest I/O concurrency per unit of memory, but only for libraries written to support it. Mixing them badly — synchronous blocking code inside asyncio, or CPU-heavy work inside a request-handling thread pool — is where most production concurrency bugs actually come from.

6. The Producer-Consumer Pattern with queue.Queue

Coordinating multiple threads with raw Lock and Condition objects is error-prone. queue.Queue is a thread-safe FIFO that handles all of that internally — it's the standard way to hand work between threads safely.

import queue, threading

work_queue: queue.Queue = queue.Queue(maxsize=100)
SENTINEL = object()

def producer(items):
    for item in items:
        work_queue.put(item)   # blocks if the queue is full — natural backpressure
    work_queue.put(SENTINEL)

def consumer():
    while True:
        item = work_queue.get()
        if item is SENTINEL:
            work_queue.put(SENTINEL)  # let other consumers see it too
            break
        process(item)
        work_queue.task_done()

producers = [threading.Thread(target=producer, args=(batch,)) for batch in batches]
consumers = [threading.Thread(target=consumer) for _ in range(3)]
for t in producers + consumers: t.start()
for t in producers + consumers: t.join()

Queue.put() blocking when full is a built-in backpressure mechanism — a slow consumer naturally throttles a fast producer instead of memory growing unbounded. This is the same shape of problem FastAPI's thread pool solves at a different layer, which is exactly where we're headed next.

7. FastAPI’s Threading Model: What Actually Happens to a def Route

Here’s where this stops being general Python knowledge and becomes something you need to know to run FastAPI correctly in production.

When a request hits a route defined with async def, it runs directly on Uvicorn's single event loop, interleaved with every other in-flight coroutine. When a request hits a route defined with plain def, Starlette (which FastAPI is built on) does not run it on the event loop — it offloads it:

# Starlette's actual mechanism, simplified
from starlette.concurrency import run_in_threadpool

async def route_wrapper(request):
    if is_async_def(endpoint):
        return await endpoint(request)
    else:
        return await run_in_threadpool(endpoint, request)

run_in_threadpool calls anyio.to_thread.run_sync() under the hood, which draws from AnyIO's default worker thread limiter — a CapacityLimiter with exactly 40 tokens. Each token represents one blocking call allowed to run concurrently; a 41st concurrent sync call simply queues until a token frees up.

@app.get("/report")
def get_report():                 # plain def -> goes to the threadpool
    return legacy_sync_sdk.fetch()  # blocking call, isolated to a worker thread — safe

@app.get("/report-async")
async def get_report_async():      # async def -> runs on the event loop
    return legacy_sync_sdk.fetch()  # BLOCKS THE ENTIRE EVENT LOOP for every request — dangerous

You can observe and tune the limiter directly:

from contextlib import asynccontextmanager
import anyio

@asynccontextmanager
async def lifespan(app: FastAPI):
    limiter = anyio.to_thread.current_default_thread_limiter()
    limiter.total_tokens = 100   # raise the ceiling — see Section 9 for whether you should
    yield

app = FastAPI(lifespan=lifespan)

The gotcha almost nobody expects: class-based dependencies are always resolved synchronously, because __init__ can't be async. That means every Depends(SomeClass) in your route signature also draws a token from the same 40-slot pool — stack four class-based dependencies on a hot endpoint and you've quietly multiplied your thread pool pressure by four, even if every route itself is async def.

8. Mixing Async Code with Threads on Purpose

Sometimes you need to call a blocking library from inside an async def route without freezing the event loop. The tool for this is explicit thread offloading — not run_in_threadpool (that's Starlette's internal mechanism for whole routes), but the same underlying idea, exposed directly:

import asyncio

def blocking_report_generation(report_id: int) -> bytes:
    # some CPU-light but blocking legacy call - e.g. an old PDF library
    return legacy_pdf_lib.render(report_id)

@app.get("/reports/{report_id}")
async def get_report(report_id: int):
    # Python 3.9+: the clean, modern way
    pdf_bytes = await asyncio.to_thread(blocking_report_generation, report_id)
    return Response(content=pdf_bytes, media_type="application/pdf")

Or, if you need a specific executor (say, a dedicated pool sized differently from the default):

from concurrent.futures import ThreadPoolExecutor

pdf_executor = ThreadPoolExecutor(max_workers=4)

@app.get("/reports/{report_id}")
async def get_report(report_id: int):
    loop = asyncio.get_running_loop()
    pdf_bytes = await loop.run_in_executor(pdf_executor, blocking_report_generation, report_id)
    return Response(content=pdf_bytes, media_type="application/pdf")

This pattern — an async def route that explicitly hands one blocking call to a thread — is more precise than making the whole route def, because it only pays the thread-pool cost for the specific blocking segment, not the entire request lifecycle including response serialization.

9. Common Threading Bugs in Production

Race conditions in shared application state. A module-level cache dict mutated by multiple request-handling threads without a lock is the same counter += 1 bug from Section 3, just harder to spot because it's spread across a real codebase:

# WRONG — shared mutable state, no lock, accessed from many threadpool threads
_cache: dict[str, Any] = {}

def get_cached(key: str):
    if key not in _cache:
        _cache[key] = expensive_computation(key)   # race: computed twice under load
    return _cache[key]

Thread pool exhaustion from a slow dependency. One upstream service getting slow doesn’t just slow its own endpoint — every sync route sharing that 40-token pool backs up behind it, because the pool is global to the process, not per-route.

Using threads for CPU-bound work and being confused it’s not faster. Revisit Section 1 — this is directly the GIL, and it’s the single most common “why isn’t my threading code fast” question in Python.

Forgetting that requests (the library) is fully blocking. It has no async variant; calling it inside async def is a guaranteed event-loop freeze. Use httpx.AsyncClient in async code, or explicitly asyncio.to_thread() around requests calls if you must keep the library.

10. Production Hardening: Sizing and Monitoring the Thread Pool

Treat AnyIO’s 40-token limiter as a capacity number you actively manage, not a default you forget about.

Monitor utilization directly, not just latency after the fact:

import anyio

async def monitor_thread_pool():
    limiter = anyio.to_thread.current_default_thread_limiter()
    while True:
        utilization = limiter.borrowed_tokens / limiter.total_tokens
        # export as a Prometheus gauge, e.g. thread_pool_utilization
        if utilization > 0.8:
            logger.warning("thread_pool_saturation", borrowed=limiter.borrowed_tokens, total=limiter.total_tokens)
        await anyio.sleep(5)

Alert on sustained saturation, not a momentary spike — a brief burst to 100% during a traffic spike is normal; staying there for minutes means requests are queuing behind the pool, not just behind the network.

When you actually should raise total_tokens: if your workload is dominated by many short blocking calls (not long ones) and you've confirmed via monitoring that you're hitting the ceiling, raising to 100-200 is a reasonable, well-precedented move — several teams have documented exactly this fix for throughput problems traced to the default 40.

When raising the token count is the wrong fix: if individual blocking calls are slow (seconds, not milliseconds), more threads just means more slow things running at once, competing for the same downstream resource (a database, an external API) — the actual fix is reducing the blocking call’s latency or moving to an async-native client, not widening the pool.

The longer-term fix is almost always migration, not tuning: every sync route converted to async def with an async-native library (httpx, asyncpg, motor) removes load from the shared thread pool entirely and moves it to the much cheaper event loop. Treat the thread pool as a bridge for libraries that don't have an async equivalent yet, not a permanent architecture.

Closing: The Mental Model to Keep

Python threading isn’t “concurrency, but with more footguns than Java” — it’s a tool precisely shaped by the GIL: nearly useless for parallelizing CPU work, genuinely valuable for overlapping I/O waits. ThreadPoolExecutor is almost always the right level of abstraction over raw Thread objects, and queue.Queue almost always beats hand-rolled lock coordination.

And in FastAPI specifically, async def vs. def isn't a style choice — it's a direct decision about whether your code runs on the event loop or draws a token from a 40-slot thread pool shared by your entire application. Know which one you're choosing, and you'll understand exactly why your service behaves the way it does under real load.

Questions, corrections, or a threading war story from production? Drop them in the comments — always happy to go deeper on any one of these sections.