multiprocessing Manager error: send() called for a closed connection

[wevsty]

BPO	27749
Nosy	@pfmoore, @tjguk, @bitdancer, @zware, @eryksun, @zooba, @applio, @AraHaan
Files	python.exe.8228.dmp debug_AA1F93C2.JPG: multiprocessing debug Screenshot

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-08-12.18:59:30.581>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = 'multiprocessing Manager error:  send() called for a closed connection'
updated_at = <Date 2021-03-13.22:29:18.086>
user = 'https://bugs.python.org/wevsty'

bugs.python.org fields:

activity = <Date 2021-03-13.22:29:18.086>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2016-08-12.18:59:30.581>
creator = 'wevsty'
dependencies = []
files = ['44099', '44116']
hgrepos = []
issue_num = 27749
keywords = []
message_count = 14.0
messages = ['272557', '272577', '272581', '272584', '272589', '272615', '272616', '272647', '272653', '272743', '272861', '273007', '334464', '334628']
nosy_count = 10.0
nosy_names = ['paul.moore', 'tim.golden', 'r.david.murray', 'zach.ware', 'eryksun', 'steve.dower', 'davin', 'Decorater', 'wevsty', 'Michael Jacob2']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27749'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-71936: Fix race condition in multiprocessing.Pool #98274
• gh-71936: Fix race condition in multiprocessing.Pool #124973
• [3.13] gh-71936: Fix race condition in multiprocessing.Pool (GH-124973) #126869
• [3.12] gh-71936: Fix race condition in multiprocessing.Pool (GH-124973) #126870

[wevsty]

When I use Python 3.5.2 some of the time will be a crash.displayed crash module is python35.dll.
I run Python environment is Windows Server 2012R2.I think crash dump maybe will be helpful.Report.wer and carsh dump can be downloaded via the following link.
http://o8z0k9748.bkt.clouddn.com/Python_crash_Dump_20160813_AF1A29AF.7z

I have another question when i using managers lib multiprocessing Maybe have exception.
my code is：
Session_lock.G_Process_Lock = multiprocessing.Manager().Lock()
def execute_select_command(command):
try:
with Session_lock.G_Process_Lock:
values = []
conn = sqlite3.connect(SQLite_DB_NAME)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
finally:
pass

This code is possible to print out the information.

Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 44, in execute_select_command
    return values
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
    return self._callmethod('release')
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 280, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
TypeError: WriteFile() argument 1 must be int, not None

This maybe seem like a lib problem.

[eryksun]

To diagnose the access violation, it may help if you install the debug binaries and try to reproduce the crash using python_d.exe. Attach the dump file to this issue.

Regarding the multiprocessing question, if its a separate issue you need to file it on its own and nosy davin. At first glance, I don't see why the handle is None in the lock's __exit__ method. After you acquire the lock, try logging the connection information. For example:

    import multiprocessing as mp
    import multiprocessing.util

    def test(lock):
        with lock:
            mp.util.info('test:lock:token: %r' % (lock._token,))
            mp.util.info('test:lock:handle: %r' % (lock._tls.connection._handle,))

    if __name__ == '__main__':
        mp.util.log_to_stderr(mp.util.INFO)
        lock = mp.Manager().Lock()
        p = mp.Process(target=test, args=(lock,))
        p.start()
        p.join()

[wevsty]

About the crash,python_d.exe crash dump download address is

http://o8z0k9748.bkt.clouddn.com/python_d_from_vs_20160813_930A2640.7z

Crash dump display,at the

python35_d.dll!_Py_ForgetReference(_object * op) Line 1758 C
python35_d.dll!_Py_Dealloc(_object * op) Line 1786 C
python35_d.dll!tupledealloc(PyTupleObject * op) Line 236 C

I hope this can be helpful to you.

About the multiprocessing question
I modified my code becomes
def execute_select_command(command):
values = []
try:
with Session_lock.G_Process_Lock:
mp.util.info('test:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('test:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
conn = sqlite3.connect(SQLite_DB_NAME,check_same_thread = False)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
mp.util.info('Exception:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('Exception:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
pass
return values
finally:
pass

After run，Console output

[INFO/SpawnPoolWorker-18] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-18] test:lock:handle: 720
[INFO/SpawnPoolWorker-17] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-17] test:lock:handle: 908
[INFO/SpawnPoolWorker-16] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] test:lock:handle: 744
WriteFile() argument 1 must be int, not None
[INFO/SpawnPoolWorker-16] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] Exception:lock:handle: 744
Could not find thread pid_2720_id_389212239784

this problem and crash both the use of multiprocessing lib after appears.Maybe these two issues are related.
If you need my full code or need to open a new issue, please let me know.Thinks

[AraHaan]

Crashes also happen from Recursion if not careful. Idk why but it happens. Although some of it is probably by not using try / catch probably anywhere in the python source code as a safeguard. I always like try / catch even in C++ as you never know when crap might happen to make things crash. A Perfect Example is a Access violation.

(Yes I have this is a Standard for me and it gets me to great quality code)
(this makes me wonder if Windows does not use try / catch anywhere. It would be nice if Microsoft Required try / catch in every function for drivers as well to hopefully make them more stable.)

All I know is I like having a lot of Checks (even if not required) that can actually prevent any and all exceptions to begin with.

And yes from what I learned from experiance myself. Always use try and catch anything before it is to late will help you in the long run.

(not to mention I blame Microsoft for not replacing their current Windows Update for when Updates Fail with a Detailed list of the names of the update(s) that failed and a description of what the reason is without Error code(s) as the Codes does not tell crap about why they failed. So, yeah I point my fingers at whoever wrote Windows Update, to fix their crap. As this is a issue in windows 7 SP1, 8, 8.1, and 10.)

[bitdancer]

Decorator: In python code you generally want to let the exception bubble up so that you can see what the error was. Sometimes you do want to catch general errors, but mostly you don't.

As for C++ try/catch, CPython is written in C, not C++.

So, your comment doesn't really have any relevance to this issue, I'm afraid.

[AraHaan]

I can confirm that I also get the same crash.

[AraHaan]

I do know for sure the crash happens in python.exe and blames ntdll.dll for it.

[wevsty]

To Decorater
If you are found to have crashes, I'm sorry, you can follow the previous suggestion to use python_d.exe then open a new issue uploading crash dump.
It would be better.

About the multiprocessing question，I would like to add some information.

My code uses a multiprocessing Pool，I used multiprocessing.util.debug opened the debug information.debug information is as follows.

[DEBUG/SpawnPoolWorker-7] thread 'MainThread' does not own a connection
[DEBUG/SpawnPoolWorker-7] making connection to manager
[DEBUG/SyncManager-2] starting server thread to service 'SpawnPoolWorker-7'
[INFO/SpawnPoolWorker-4] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-4] test:lock:handle: 712
[INFO/SpawnPoolWorker-7] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] test:lock:handle: 768
[DEBUG/SpawnPoolWorker-7] DECREF 'e44c9bc720'
[INFO/SpawnPoolWorker-4] leave:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
[DEBUG/SpawnPoolWorker-7] DECREF 'b0182059f8'
[DEBUG/SyncManager-2] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'e44c9bc720'
handle is closed
'ForkAwareLocal' object has no attribute 'connection'
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 51, in execute_select_command
    conn.close()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
    return self._callmethod('release')
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 204, in send
    self._check_closed()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed
[INFO/SpawnPoolWorker-7] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] Exception process:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SyncManager-1] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-4] DECREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-7] INCREF 'e44c9bc720'

This time the message and not the same，From the information looks like two processes simultaneously locking the same lock object, and then release happened the problems.I do not know if this is correct，
But i will continue to follow the issue.

[AraHaan]

I use code that does multithreading to from a library that uses ffmpeg for exact it makes daemon threads so it could be from that.

[wevsty]

I try to change the multiprocessing lib codes，upload the file is debugger display.I try to used while loop make sure connection hanle value is not none,but connection hanle value will still becomes none.
I think there may be other threads to modify the handle，But it not my code.

[wevsty]

today，Find a new crash
python_d_crash_dump_20160816_A901A6F3.7z，is full crash dump
Download address
http://o8z0k9748.bkt.clouddn.com/python_d_crash_dump_20160816_A901A6F3.7z
dump display crash in the

python35_d.dll!visit_decref(_object * op, void * data) Line 379 C
python35_d.dll!dict_traverse(_object * op, int(*)(_object *, void *) visit, void * arg) Line 2534 C
python35_d.dll!subtract_refs(_gc_head * containers) Line 401 C
python35_d.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 959 C

[wevsty]

A similar situation also occurs in the following Linux，A similar situation on stackoverflow

http://stackoverflow.com/questions/29277150/python-3-4-multiprocessing-bug-on-lock-acquire-typeerror-integer-required

[MichaelJacob2]

Does this ticket track the issue in its title or the native crash? If it's the latter, is there a new ticket for the None _handle or shall I create one?

[MichaelJacob2]

So, I'm experiencing the issue in the title, too. The pipe handle stays valid for between 5 and 60 minutes, then it goes None when written to.

I'm far from understanding that code, but this crude re-connect code seems to solve the issue for me:

In BaseProxy._callmethod, I changed:

        conn.send((self._id, methodname, args, kwds))

to:

    try:
        conn.send((self._id, methodname, args, kwds))
    except TypeError:
        self._connect()
        conn = self._tls.connection
        conn.send((self._id, methodname, args, kwds))

[basnijholt]

I can consistently reproduce this problem (unfortunately with a third party library that I maintain) on Python 3.12.3 on Ubuntu.

The code:

from concurrent.futures import ProcessPoolExecutor
from pipefunc import pipefunc, Pipeline

for _ in range(100):

    @pipefunc(output_name="c", cache=True)
    def f(a, b):
        return a + b

    pipeline = Pipeline([f], cache_type="lru")
    func = pipeline.func("c")
    sequence = 100 * [{"a": 2, "b": 3}]
    with ProcessPoolExecutor(max_workers=1) as executor:
        results = executor.map(func.call_with_dict, sequence)
        list(results)

Which outputs:

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/process.py", line 263, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/process.py", line 212, in _process_chunk
    return [fn(*args) for args in chunk]
            ^^^^^^^^^
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_pipeline.py", line 1304, in call_with_dict
    return self(**kwargs)
           ^^^^^^^^^^^^^^
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_pipeline.py", line 1274, in __call__
    return self.pipeline.run(output_name=self.output_name, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_pipeline.py", line 490, in run
    self._run(
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_pipeline.py", line 422, in _run
    return_now, result_from_cache = _get_result_from_cache(
                                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_pipeline.py", line 1398, in _get_result_from_cache
    if cache_key is not None and cache_key in cache:
                                 ^^^^^^^^^^^^^^^^^^
  File "/home/bas.nijholt/repos/pipefunc/pipefunc/_cache.py", line 316, in __contains__
    return key in self._cache_dict
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in __contains__
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/multiprocessing/managers.py", line 820, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/multiprocessing/connection.py", line 427, in _send_bytes
    self._send(header + buf)
  File "/home/bas.nijholt/micromamba/envs/pipefunc/lib/python3.12/multiprocessing/connection.py", line 384, in _send
    n = write(self._handle, buf)
        ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object cannot be interpreted as an integer
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[1], line 19
     17 with ProcessPoolExecutor(max_workers=1) as executor:
     18     results = executor.map(func.call_with_dict, sequence)
---> 19     list(results)

File ~/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/process.py:642, in _chain_from_iterable_of_lists(iterable)
    636 def _chain_from_iterable_of_lists(iterable):
    637     """
    638     Specialized implementation of itertools.chain.from_iterable.
    639     Each item in *iterable* should be a list.  This function is
    640     careful not to keep references to yielded objects.
    641     """
--> 642     for element in iterable:
    643         element.reverse()
    644         while element:

File ~/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/_base.py:619, in Executor.map.<locals>.result_iterator()
    616 while fs:
    617     # Careful not to keep a reference to the popped future
    618     if timeout is None:
--> 619         yield _result_or_cancel(fs.pop())
    620     else:
    621         yield _result_or_cancel(fs.pop(), end_time - time.monotonic())

File ~/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/_base.py:317, in _result_or_cancel(***failed resolving arguments***)
    315 try:
    316     try:
--> 317         return fut.result(timeout)
    318     finally:
    319         fut.cancel()

File ~/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/_base.py:456, in Future.result(self, timeout)
    454     raise CancelledError()
    455 elif self._state == FINISHED:
--> 456     return self.__get_result()
    457 else:
    458     raise TimeoutError()

File ~/micromamba/envs/pipefunc/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self)
    399 if self._exception:
    400     try:
--> 401         raise self._exception
    402     finally:
    403         # Break a reference cycle with the exception in self._exception
    404         self = None

TypeError: 'NoneType' object cannot be interpreted as an integer

I applied the patch in #98274 which seems to have solve the problem.

[encukou]

Merged in main. I'll watch the buildbots for a bit before backporting.

[basnijholt]

@encukou, sorry that it took me this long to reply but using the code in #71936 (comment) and your patch from #124973, I can no longer reproduce the problem!

Thanks a lot!

[encukou]

The backports are in. Thank you to everyone involved!