forked process in multiprocessing does not honour atexit

[gaborbernat]

BPO	39675
Nosy	@pitrou, @applio, @pablogsal, @gaborbernat

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = None
created_at = <Date 2020-02-18.11:35:55.954>
labels = ['type-bug', 'library', '3.9']
title = 'forked process in multiprocessing does not honour atexit'
updated_at = <Date 2020-02-18.12:27:31.963>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2020-02-18.12:27:31.963>
actor = 'pablogsal'
assignee = 'pablogsal'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-02-18.11:35:55.954>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 39675
keywords = []
message_count = 1.0
messages = ['362197']
nosy_count = 4.0
nosy_names = ['pitrou', 'davin', 'pablogsal', 'gaborjbernat']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39675'
versions = ['Python 3.9']

Linked PRs

• gh-83856: Honor atexit for all multiprocessing start methods #114279

[gaborbernat]

I've talked with Pablo about this in person, and as advised opening the issue here now.

I've discovered that forked processes do not honour atexit registrations. See the following example code:

from multiprocessing import Process, set_start_method
import time
import os
import atexit


def cleanup():
    print(f"cleanup {os.getpid()}")


atexit.register(cleanup)


def run():
    time.sleep(0.1)
    print(f"process done {os.getpid()}")
    # atexit._run_exitfuncs()


if __name__ == "__main__":
    set_start_method("fork")
    process = Process(target=run)
    process.start()
    process.join()
    print("app finished")

In case of a forked process childs the atexit is never executed (note it works if I ran them manually at the end of the child process; so they're registered correctly). Switching to spawn method makes it work as expected. The behaviour is the same even if you call register within the child process (as opposed to being inherited during forking). Also found this StackOverflow question that mentions this https://stackoverflow.com/a/26476585. At the very least the documentation should explain this; though I'd expect atexit to be called before finalization of the fork processes (assuming the child process exits with 0 exit code). d

[pablogsal]

CC: @gpshead @pitrou @ambv

See the PR and this comment for more context

[gpshead]

The discussion on the PR was useful. This is merged in time for 3.13beta1. It is a bit of a behavior change and a corner case that has existed for a long time and most code does not hit so I do not believe we should be backporting it as a bugfix despite it also being reasonably to call a bug. Thanks for the discussion and PR!