Skip to content

Segfault in contextvars HAMT during isolation_scope() / new_scope() with uvicorn multiprocessing workers #5190

@sysradium

Description

@sysradium

How do you use Sentry?

Self-hosted/on-premise

Version

2.45.0

Steps to Reproduce

Environment

  • sentry-sdk version: 2.45.0
  • Python version: 3.10.x
  • OS: Linux (x86_64)
  • Framework: FastAPI + uvicorn with --workers 10 (multiprocessing spawn)
  • Event loop: uvloop 0.21.0

Problem

Workers crash with SIGSEGV when Sentry SDK calls _current_scope.set() inside isolation_scope() or new_scope(). The crash occurs in Python's HAMT (Hash Array Mapped Trie) implementation used by contextvars.

GDB analysis shows that the HAMT node pointer has been corrupted - it points to a PyFunctionObject (_safe_repr_wrapper from sentry_sdk/serializer.py) instead of a HAMT node. When Python tries to clone this "node", it reads the function's vectorcall field as if it were b_array, then crashes trying to Py_INCREF a C function pointer.

Stack trace:

#0  _Py_INCREF (op=0x7b4d9a872260 <_PyFunction_Vectorcall>) at ./Include/object.h:472
#1  _Py_XINCREF (op=0x7b4d9a872260 <_PyFunction_Vectorcall>) at ./Include/object.h:558
#2  hamt_node_bitmap_clone (node=0x7b4c301c7e20) at Python/hamt.c:583
#3  hamt_node_bitmap_assoc (...) at Python/hamt.c:771
#4  _PyHamt_Assoc (...) at Python/hamt.c:2308
#5  contextvar_set (var=0x7b4d9916a5c0, val=<Scope at remote 0x7b4c30d62b60>) at Python/context.c:738
#6  PyContextVar_Set (ovar=<_contextvars.ContextVar at remote 0x7b4d9916a5c0>, ...) at Python/context.c:285
...
#12 _PyEval_EvalFrame (...) for file sentry_sdk/scope.py, line 1785, in isolation_scope

Python stack trace

File "sentry_sdk/scope.py", line 1785, in isolation_scope
    current_token = _current_scope.set(forked_current_scope)
File "contextlib.py", line 135, in __enter__
    return next(self.gen)
File "sentry_sdk/integrations/asgi.py", line 198, in _run_app
    with sentry_sdk.isolation_scope() as sentry_scope:

Corrupted HAMT node:

(gdb) p *node
$2 = {
  ob_base = {ob_type = 0x7b4d9aa8a7a0 <PyFunction_Type>},
  b_bitmap = 2587471808,
  b_array = {'_safe_repr_wrapper'}  # Does not look like HAMT data
}

(gdb) p node->b_array[0]
$5 = '_safe_repr_wrapper'

(gdb) x/20x 0x7b4d9a872260
0x7b4d9a872260 <_PyFunction_Vectorcall>: ....

Sentry configuration

sentry_sdk.init(
    dsn=settings.SENTRY_DSN,
    traces_sample_rate=1.0,
    profiles_sample_rate=0.0,
    enable_tracing=True,
    send_default_pii=True,
    integrations=[
        FastApiIntegration(),
        HttpxIntegration(),
        AioHttpIntegration(),
        RedisIntegration(),
    ],
)

How to reproduce

  1. FastAPI app with sentry-sdk 2.45.0
  2. Run with uvicorn app:app --workers 10 (multiprocessing spawn)
  3. Send concurrent HTTP requests
  4. Workers eventually crash with SIGSEGV

Observations

  • Crash happens in spawned worker processes, not the main process
  • The corrupted pointer always points to Sentry SDK's _safe_repr_wrapper function or _PyFunction_Vectorcall
  • Workers that don't use Sentry SDK (different project, same infrastructure) don't crash
  • Same codebase works fine in single-process mode

We tried setting OPENBLAS_NUM_THREADS=1 and it did not help.

It appears that a function object created during Sentry serialization (_safe_repr_wrapper) is somehow being written to a location that should contain a HAMT node pointer.

This could be:

  • A use-after-free where HAMT memory is reused for function objects
  • A buffer overflow in serialization code
  • Race condition in scope/context handling with multiprocessing

Expected Result

No segfault

Actual Result

Eventual segfault

Metadata

Metadata

Assignees

Labels

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions