Data race between _PyMonitoring_RegisterCallback and _Py_call_instrumentation_2args in instrumentation.c

[vfdev-5]

Bug report

Bug description:

I built cpython 3.13 from source with TSAN and running the following code:

import sys
import concurrent.futures
import threading


if __name__ == "__main__":
    num_workers = 20
    num_runs = 100

    barrier = threading.Barrier(num_workers)

    tool_id = 3
    event_id = sys.monitoring.events.CALL

    sys.monitoring.use_tool_id(tool_id, "test_tool_1")
    sys.monitoring.set_events(tool_id, event_id)


    def closure():
        barrier.wait()

        def my_callback(code, instruction_offset, callable, arg0):
            pass

        sys.monitoring.register_callback(tool_id, event_id, my_callback)

        def example_function():
            a = 1
            b = 2
            c = a + b

        for _ in range(num_runs):
            example_function()

    with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
        futures = []
        for i in range(num_workers):
            futures.append(executor.submit(closure))
        assert len(list(f.result() for f in futures)) == num_workers

TSAN reports the following data race: https://gist.github.com/vfdev-5/e80f9b4528993eb1543e45a9216350e3#file-repro-log

cpython version:

Python 3.13.2+ experimental free-threading build (heads/3.13:9e0fce413a9, Mar 12 2025, 00:48:15) [Clang 18.1.3 (1ubuntu1)]

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-131141: fix data race in instrumentation while registering callback #131142
• [3.13] gh-131141: fix data race in instrumentation while registering callback #131166

[kumaraditya303]

On current main TSAN report:

==================
WARNING: ThreadSanitizer: data race (pid=76663)
  Atomic write of size 8 at 0x55dfbb90bfc0 by thread T1:
    #0 _Py_atomic_exchange_ptr Include/cpython/pyatomic_gcc.h:195 (python+0x3ea0d4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #1 exchange_callables Python/instrumentation.c:3014 (python+0x3ea0d4)
    #2 _PyMonitoring_RegisterCallback Python/instrumentation.c:3044 (python+0x3ef67d) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #3 monitoring_register_callback_impl Python/instrumentation.c:2289 (python+0x3efb95) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #4 monitoring_register_callback Python/clinic/instrumentation.c.h:152 (python+0x3efcbc) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #5 cfunction_vectorcall_FASTCALL Objects/methodobject.c:436 (python+0x1f7699) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #6 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x171175) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #7 PyObject_Vectorcall Objects/call.c:327 (python+0x1712be) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #8 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6141 (python+0x355edd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #9 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #10 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #11 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #12 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #13 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #14 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #15 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6302 (python+0x356fa8) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #16 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #17 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #18 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #19 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #20 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #21 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #22 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6302 (python+0x356fa8) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #23 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #24 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #25 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #26 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x1754c4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #27 method_vectorcall Objects/classobject.c:72 (python+0x1757e4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #28 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #29 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #30 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #31 thread_run Modules/_threadmodule.c:351 (python+0x4e448f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #32 pythread_wrapper Python/thread_pthread.h:242 (python+0x43ac31) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)

  Previous read of size 8 at 0x55dfbb90bfc0 by main thread:
    #0 call_one_instrument Python/instrumentation.c:977 (python+0x3e9036) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #1 call_instrumentation_vector Python/instrumentation.c:1171 (python+0x3ea8cc) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #2 _Py_call_instrumentation_2args Python/instrumentation.c:1226 (python+0x3eca2c) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #3 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6074 (python+0x355de0) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #4 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x19cd2a) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #5 gen_send_ex2 Objects/genobject.c:255 (python+0x19d307) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #6 gen_iternext Objects/genobject.c:629 (python+0x19f367) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #7 list_extend_iter_lock_held Objects/listobject.c:1230 (python+0x1c21fb) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #8 _list_extend Objects/listobject.c:1403 (python+0x1c2343) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #9 list___init___impl Objects/listobject.c:3448 (python+0x1c263c) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #10 list_vectorcall Objects/listobject.c:3472 (python+0x1c27db) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #11 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x171175) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #12 PyObject_Vectorcall Objects/call.c:327 (python+0x1712be) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #13 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6141 (python+0x355edd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #14 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #15 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #16 PyEval_EvalCode Python/ceval.c:771 (python+0x372d5b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #17 run_eval_code_obj Python/pythonrun.c:1362 (python+0x418fa0) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #18 run_mod Python/pythonrun.c:1433 (python+0x419217) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #19 pyrun_file Python/pythonrun.c:1290 (python+0x419c7a) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #20 _PyRun_SimpleFileObject Python/pythonrun.c:518 (python+0x41bbd4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #21 _PyRun_AnyFileObject Python/pythonrun.c:78 (python+0x41bdd4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #22 pymain_run_file_obj Modules/main.c:394 (python+0x45659b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #23 pymain_run_file Modules/main.c:413 (python+0x45670f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #24 pymain_run_python Modules/main.c:679 (python+0x45753f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #25 Py_RunMain Modules/main.c:760 (python+0x457899) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #26 pymain_main Modules/main.c:790 (python+0x457950) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #27 Py_BytesMain Modules/main.c:814 (python+0x457aaf) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    #28 main Programs/python.c:15 (python+0x85b3b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)

  Location is global '_PyRuntime' of size 377920 at 0x55dfbb8b4400 (python+0x7ebfc0)

The data race is because the callback is being read here without atomic whereas it was exchanged with atomics.

cpython/Python/instrumentation.c

Line 976 in e0637ce

PyObject *instrument = interp->monitoring_callables[tool][event];