Repository navigation
multiprocessing forkserver does not flush output before fork (was: preloading '__main__' with forkserver has been broken for a long time) #98552
Description
Bug report
The forkserver start method provides the ability to call set_forkserver_preload on the multiprocessing context to load modules into and configure the forkserver process. By doing this carefully, you can avoid having to do module loading and other work each time the forkserver process is forked to create a new process. Without doing such work, forkserver can be way slower than the traditional fork start method
You can specify the module '__main__' in the set_forkserver_preload list, and the forkserver source has special code when you do this. It ensures that the main file path does not have to be configured/loaded after each fork. To do this, in multiprocessing.forkserver.ensure_running, it calls multiprocessing.spawn.get_preparation_data and then uses the main_path entry that may be in the returned dict.
Unfortunately, 3 months after it was introduced, this functionality was broken in commit 9a76735. That commit renamed the main_path dictionary entry returned in get_preparation_data to init_main_from_path, but didn't update the use in multiprocessing.fork_server
Not having the ability to load and configure main on the forkserver ends up being unusually painful for my recent scenario, which led to tracking this down. I have a python program on a share that spawns short-lived processes at a high rate. Then multiple machines run this program from the share. Huge slowdown ensues as smbd processes on the server go crazy responding to every new process on every client reading the file and stat-ing the directory the file is contained in.
A simple fix in multiprocessing.forkserver accounting for the changed name rectifies the problem. I'll work on putting that PR together. Any thoughts on a workaround that doesn't require modifying the python source are welcome, as I imagine it will be a while until I'm on a python with the fix.
Here's a simple repro.
import time
import multiprocessing
print('hi from forkserver_repro')
def _silly(i):
time.sleep(0.2)
return i%2
def run_subprocesses():
process_list=[]
for i in range(10):
process_list.append(multiprocessing.Process(target=_silly,args=[i]))
for process in process_list:
process.start()
for process in process_list:
process.join()
return 0
if __name__ == '__main__':
multiprocessing.set_start_method('forkserver')
ctx = multiprocessing.get_context('forkserver')
ctx.set_forkserver_preload(['__main__',])
print('Only one more "hi from forkserver_repro" should print! More means a bug!')
run_subprocesses()
Your environment
Ubuntu 20.04.4 LTS, CPython 3.8
Python source code examination indicates this bug is still present in the current version of CPython.
- PR: [3.14] gh-98552: flush std streams in the multiprocessing forkserver before fork (GH-141849) #141851
Activity
gpshead commentedon Oct 23, 2022
A PR fixing this and adding unittest coverage would be great.
BTW as you are using forkserver, be aware of the recently disclosed #97514 if you ever run on 3.9 or later. (that issue can be worked around)
aggieNick02 commentedon Oct 24, 2022
Thanks for the pointer on the CVE.
The simple fix for this is pretty straightforward and I'll try to get it together quickly (first cpython PR so may take a little longer). It is just to deal with the renamed dict entry.
However, I've discovered some other "quirks" with forkserver, and was wondering if they are known or being worked on, if they could use improvement, or are expected and it's just my newness in looking at it. (I'm on linux but trying to use multiprocessing in programs that also use threading, and have gotten burned a few times by deadlocks because of it, leading me to move to using forkserver. But maybe I should be considering another way forward?)
Anyways, with the break fixed by using the right dict key to get
main_pathinto main, preloading__main__works, but imports in the main module can fail, becausesys.pathis not set in forkserver's main before importing main. (The correctsys_pathis actually fed intoforkserver.main, but not used?! It's been that way since forkserver was added 9 years ago...)Not having
sys.pathset leads to other weirdness even when not importing main. The modules specified to preload will succeed/fail depending on the cwd of the process, sopython a/foo.pybehaves different thencd a; python foo.py. And I'm pretty sure I've seen the same module imported twice, once at the name given as preload and once as the name used by another module (e.g.,a.my_moduleandmy_module).The
try/catcharound each preloaded module hides all the failures too. In my case, I'd much prefer to get the exception, because if the preload fails, then every process created by the forkserver is going to have to do expensive module loading itself, and I don't realize it until I notice the performance problem.When
spawnis used, we set/clear_inheritingand callpreparewith the result ofget_preparation_datainspawn._main. My completely uninformed inclination is to do the same inforkserver.main, instead of what's currently happening. Is that a reasonable thing to look into?aggieNick02 commentedon Oct 24, 2022
I did some experiments. Using
spawn.prepareinforkserver.pyto consume the dict fromget_prepare_dataseems to work well. It fixessys.pathto be correct so imports work regardless of cwd, fixes preloading__main__in a less brittle way, and does a bunch of other stuff that I'm guessing should be happening too. But I don't know enough about forkserver and spawn to be certain.I'll put together a PR for this approach too, to solicit input.
forkserver.maingets its parameters from the string created inforkserver.ensure_running, so the dict fromget_prepare_datahas to be turned into a string. A quick way to do this and not have to worry about escaping quotes/etc is to remove theauthkeyfrom the dict, thenjson.dumpstoutf-8tob64encodeit. Thenmaindoes the reverse to get it back. (Thought aboutpicklebut don't know enough to know the security implications here.)The downside (for me) of this more general solution is that it alters code on the new process side, versus the not as complete fix that is on the original process side. This means the latter can be done with monkey-patching, but the former cannot, as far as I understand.
aggieNick02 commentedon Oct 26, 2022
Updating the tests for these changes led to some comical confusion. Turns out there was another bug waiting... forkserver does not flush its stdout/stderr before each fork, and that makes things really confusing.
I was using print statements in files to make sure preload was working and they were only imported in the forkserver and not in the new forked processes. But the output said otherwise, even though everything was actually working...
pythongh-98552: Fix preloading '__main__' with forkserver being broke…
thesamesam commentedon Oct 9, 2025
I think this one is fixed now: #99515 (comment).
duaneg commentedon Nov 22, 2025
Ah, apologies @aggieNick02 for not finding this issue and your fix when I looked at #126631. You even discovered and fixed the same stdout buffering issue that so confused me!
gpshead commentedon Nov 22, 2025
preloading main got fixed as noted above, while this one was overlooked as having had a fix for it already in the shuffle. thanks for tying things together. does the output buffering issue still remain?
[-]preloading '__main__' with forkserver has been broken for a long time[/-][+]multiprocessing forkserver does not flush output before fork (was: preloading '__main__' with forkserver has been broken for a long time)[/+]3 remaining items
[3.14] gh-98552: flush std streams in the multiprocessing forkserver …
[3.13] gh-98552: flush std streams in the multiprocessing forkserver …
Revert (unneeded, already done elsewhere) "pythongh-98552: flush std …
gpshead commentedon Nov 23, 2025
the remaining purpose i had for keeping this open wound up a dupe of #135335 which was already done elsewhere in the code that I hadn't noticed. reverting those. closing this out. a new issue #141860 tracks the continuation of your feature to be able to see the errors from preload module imports in the forkserver process.
gh-98552: Revert (unneeded, already done elsewhere) "flush std stream…
pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …
pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …
[3.13] gh-98552: Revert (unneeded, already done elsewhere) "flush std…
[3.14] gh-98552: Revert (unneeded, already done elsewhere) "flush std…
duaneg commentedon Nov 23, 2025
As you noted, this was fixed under #135335. There is still an open PR, #138686, which simplifies the unit test added for #126631. With the buffering bug fixed we can print from
__main__, which makes things much easier. It would be lovely if that could be merged, assuming it looks good ofc 🙂pythongh-98552: flush std streams in the multiprocessing forkserver b…
pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …
pythongh-98552: flush std streams in the multiprocessing forkserver b…
pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …