Runtime error on scp.pp.import_fragments #58
Description
Thanks for the intereseting tools! I am currently using scPrintero, following the vignette of bulkATAC data.
I faced runtime rror onscp.pp.import_fragments()when I try to run the pipeline again after failed run.
Python version which I used is 3.12.8 and the environment of my workstation is [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux.
The error messages are as below:
Multiple fragments files detected, it is suggested to provide sample names to avoid barcode collision
You are now using the beta auto_detect_shift function, this overwrites the plus_shift and minus_shift you provided
If you believe the auto_detect_shift is wrong, please set auto_detect_shift=False
Importing fragments: 0%| | 0/6 [00:00<?, ?it/s]Detecting the shift in the paired end fragments file
If you think the above message is wrong, please check the input file format
Minimum MSE is 0.000598303155434603, shift detected
Minimum MSE is 0.0005461446782905552, shift detected
detected plus_shift and minus_shift are 4 -5 for /globalsc/ulb/iribhm/ysong/scPRINTER_test/01_bulkATAC//fragments/Bcell_0_frags.tsv.gz
Importing fragments: 17%|█▋ | 1/6 [00:07<00:35, 7.19s/it]
Importing fragments: 17%|█▋ | 1/6 [00:07<00:35, 7.19s/it]Detecting the shift in the paired end fragments file
If you think the above message is wrong, please check the input file format
Minimum MSE is 0.000529748210377538, shift detected
Minimum MSE is 0.000614607207130894, shift detected
detected plus_shift and minus_shift are 4 -5 for /globalsc/ulb/iribhm/ysong/scPRINTER_test/01_bulkATAC//fragments/Bcell_1_frags.tsv.gz
Importing fragments: 33%|███▎ | 2/6 [00:21<00:45, 11.41s/it]
Importing fragments: 33%|███▎ | 2/6 [00:21<00:45, 11.41s/it]Multiple fragments files detected, it is suggested to provide sample names to avoid barcode collision
You are now using the beta auto_detect_shift function, this overwrites the plus_shift and minus_shift you provided
If you believe the auto_detect_shift is wrong, please set auto_detect_shift=False
Importing fragments: 0%| | 0/6 [00:00<?, ?it/s]Multiple fragments files detected, it is suggested to provide sample names to avoid barcode collision
You are now using the beta auto_detect_shift function, this overwrites the plus_shift and minus_shift you provided
If you believe the auto_detect_shift is wrong, please set auto_detect_shift=False
Importing fragments: 0%| | 0/6 [00:00<?, ?it/s]Detecting the shift in the paired end fragments file
If you think the above message is wrong, please check the input file format
Minimum MSE is 0.0006106771692154147, shift detected
Minimum MSE is 0.0005871743434437336, shift detected
detected plus_shift and minus_shift are 4 -5 for /globalsc/ulb/iribhm/ysong/scPRINTER_test/01_bulkATAC//fragments/Bcell_0_frags.tsv.gz
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen runpy>", line 287, in run_path
File "<frozen runpy>", line 98, in _run_module_code
File "<frozen runpy>", line 88, in _run_code
File "/globalsc/users/y/s/ysong/scPRINTER_test/01_bulkATAC/bulkATAC.py", line 47, in <module>
printer = scp.pp.import_fragments(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/logomaker/../scprinter/preprocessing.py", line 240, in import_fragments
pool.submit(
File "/usr/lib64/python3.12/concurrent/futures/process.py", line 822, in submit
self._adjust_process_count()
File "/usr/lib64/python3.12/concurrent/futures/process.py", line 781, in _adjust_process_count
self._spawn_process()
File "/usr/lib64/python3.12/concurrent/futures/process.py", line 799, in _spawn_process
p.start()
File "/usr/lib64/python3.12/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/multiprocessing/context.py", line 289, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/lib64/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/lib64/python3.12/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
File "/usr/lib64/python3.12/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Importing fragments: 0%| | 0/6 [00:22<?, ?it/s]
['Bcell_0', 'Bcell_1', 'Monocyte_0', 'Monocyte_1', 'Tcell_0', 'Tcell_1']
Multiple fragments files detected, it is suggested to provide sample names to avoid barcode collision
You are now using the beta auto_detect_shift function, this overwrites the plus_shift and minus_shift you provided
If you believe the auto_detect_shift is wrong, please set auto_detect_shift=False
Importing fragments: 0%| | 0/6 [00:00<?, ?it/s]Detecting the shift in the paired end fragments file
If you think the above message is wrong, please check the input file format
Minimum MSE is 0.0001534636937659554, shift detected
Minimum MSE is 0.00013151955026936068, shift detected
detected plus_shift and minus_shift are 4 -5 for /globalsc/ulb/iribhm/ysong/scPRINTER_test/01_bulkATAC//fragments/Monocyte_0_frags.tsv.gz
Importing fragments: 50%|█████ | 3/6 [00:45<00:51, 17.12s/it]
Importing fragments: 50%|█████ | 3/6 [00:45<00:51, 17.12s/it]Detecting the shift in the paired end fragments file
If you think the above message is wrong, please check the input file format
Minimum MSE is 0.00012713128718992635, shift detected
Minimum MSE is 0.00012540037802311342, shift detected
detected plus_shift and minus_shift are 4 -5 for /globalsc/ulb/iribhm/ysong/scPRINTER_test/01_bulkATAC//fragments/Monocyte_1_frags.tsv.gz
Traceback (most recent call last):
File "/globalsc/users/y/s/ysong/scPRINTER_test/01_bulkATAC/bulkATAC.py", line 48, in <module>
printer = scp.pp.import_fragments(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/scprinter/preprocessing.py", line 240, in import_fragments
pool.submit(
File "/usr/lib64/python3.12/concurrent/futures/process.py", line 805, in submit
raise BrokenProcessPool(self._broken)
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore
Importing fragments: 50%|█████ | 3/6 [00:54<00:54, 18.03s/it]
/usr/lib64/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 5 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
I already tried to re-start my session on jupyter notebook, forcing the fork+threads with multiprocessing module. However, nothing really works and generating same issue.
For this, can you give me some advice? Thanks in advance!