Repository navigation
Start the deprecation cycle for subprocess preexec_fn #82616
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2019-10-10.18:18:17.382>
labels = ['type-feature', 'library', '3.11']
title = 'Start the deprecation cycle for subprocess preexec_fn'
updated_at = <Date 2022-03-16.16:54:37.730>
user = 'https://github.com/gpshead'bugs.python.org fields:
activity = <Date 2022-03-16.16:54:37.730>
actor = 'markmentovai'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-10-10.18:18:17.382>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 38435
keywords = ['patch']
message_count = 18.0
messages = ['354397', '354404', '354407', '354439', '383708', '383720', '383724', '383725', '383726', '383727', '383728', '383736', '383740', '383863', '383988', '400498', '403026', '415352']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'vstinner', 'diekhans', 'markmentovai', 'serhiy.storchaka', 'izbyshev']
pr_nums = ['23930', '23936']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38435'
versions = ['Python 3.11']
Activity
gpshead commentedon Oct 10, 2019
subprocess's preexec_fn feature needs to enter PendingDeprecationWarning state in 3.9, to become a DeprecationWarning in 3.10, with a goal of removing it in 3.11.
Rationale: We now live in a world full of threads, it is entirely unsafe to call back into the python interpreter within the forked child before exec per POSIX specification.
We've also already made preexec_fn no longer supported from CPython subinterpreters in 3.8.
If there are not already issues open for desired features of subprocess that do not yet have replacements or workarounds for *specific* actions that preexec_fn is being used for in your application, please open feature requests for those. (ex: calling umask is https://bugs.python.org/issue38417, and group, uid, gid setting has already landed in 3.9)
vstinner commentedon Oct 10, 2019
What is the recommanded way to replace preexec_fn?
gpshead commentedon Oct 10, 2019
With task specific arguments. cwd, start_new_session, group, extra_groups,
user, etc..
We cannot provide a general do everything replacement and should not try.
It not possible.
vstinner commentedon Oct 11, 2019
Well, I proposed a solution at:
https://bugs.python.org/issue38417#msg354242
I know that this solution has multiple flaws, but a bad solution may be better than no solution: breaking applications when upgrading to Python 3.11.
diekhans commentedon Dec 24, 2020
calling setpgid() is a common post-fork task that would need to be an explicit parameter to Popen when preexec_fn goes away
gpshead commentedon Dec 25, 2020
PR up to add setpgid support. From what I've come across, some setpgid() users can use setsid() already available via start_new_session= instead. But rather than getting into the differences between those, making both available makes sense to allow for anyone's case where setsid() isn't desired.
gpshead commentedon Dec 25, 2020
https://bugs.python.org/issue42736 filed to track adding Linux prctl() support.
gpshead commentedon Dec 25, 2020
Another preexec_fn use to consider:
Using an intermediate shell script wrapper that changes the rlimit and exec's the actual process is also an alternative.
gpshead commentedon Dec 25, 2020
I'm also seeing a lot of os.setpgrp() calls, though those are more likely able to use start_new_session to do setsid() as a dropin replacement.
gpshead commentedon Dec 25, 2020
signal.signal use case:
Calls to signal.signal(x, y) to sometimes to set a non SIG_DFL behavior on a signal. SIGINT -> SIG_IGN for example.
I see a lot of legacy looking code calling signal.signal in prexec_fn that appears to set SIG_DFL for the signals that Python otherwise modifies. Which restore_signals=True should already be doing.
19 remaining items
gpshead commentedon Feb 14, 2024
Creating your own intermediate executable that does the same thing as your preexec code before calling exec to replace itself with the ultimate desired process rather than expecting it to be done in the restricted environment of process between fork+exec should work well for things like that.
hauntsaninja commentedon Feb 15, 2024
My work codebase has a fair amount of preexec_fn use, more than half of it is prctl PR_SET_PDEATHSIG
vstinner commentedon Feb 15, 2024
Would it make sense to have a portable and generic way to:
Rather than
preexec_fn=<function>, it would be something likepreexec_code: str.We should design carefully "preexec" about signals, threads, file descriptor inheritance, but also report properly exec() errors to the parent process!
The main difference is that
preexec_fnis called between fork() and exec(), rather thanpreexec_codewould be called after fork()+exec("python").astrand commentedon Mar 26, 2024
Hi all. As the original subprocess module author, I'm a bit concerned over the expansion of the API. The number of arguments for the subprocess.Popen constructor is, to be honest, crazy. I am grateful for all work and bugfixes that has been done over the years, but the API is now quite far from the my original vision of a "small and clean API", which was the reason to "start from scratch", thus create the subprocess module.
Removing preexec_fn and then adding additional arguments for cases where preexec_fn was used will further contribute to the "bloat" of the API, if I may use that word.
I do understand the technical difficulties, but as always there are multiple alternatives. Better documentation as well as #82616 (comment) are very interesting, as I see it.
gpshead commentedon Mar 26, 2024
Major kudos for its creation @astrand! Your work largely succeeded! If it had not, it would've needed replacing again. =)
The original overall
subprocesspublic API design was sound with one major exception:preexec_fn=was a conflicted with the existence of threads, thus why I label it as a fundamentally flawed feature. Yet its existence also allowed for a lot of use cases that were otherwise unimplemented by subprocess and would've resulted in people coming up with even worse custom (read: buggy x1000) workarounds. So despite the problem, please don't consider this a criticism. It made the most sense at the time.The remaining conundrum:
A library using the
subprocessAPI withpreexec_fn=is often not be owned by the same people owning code elsewhere in a broader application that uses threads. Both code owners believe they are right to do what they did. Afterall, they were just using documented public APIs with an expectation that they just work thanks to Python. Python programmers are often operating at a high level and not expected to need to know low level system programming details for what appear to them to be normal mundane tasks like spawning a process or using threads.If told they cannot expect do either they will have no good options that satisfy their needs. Their mere existence of both within the same process is the source of conflict. Use of threading is far more important to the world than realistic
preexec_fn=use cases, thus nobody is advising "nobody should use threads, a minority of people won't be able to do these esoteric things with child processes easily!". 😄In absence of ways to replace the functionality people use it for,
preexec_fn=continues to exist, just with the giant warning about the random deadlocks that will happen to end users of such unfortunate applications when the entire application stack is not tightly controlled.Thus our dirty plethora of options added to replace many common things people were using preexec_fn= for. Those also greatly increase subprocess launching performance (done in C, vfork can be used, etc). We've provided for some common cases. More exist. There is no way for us to predict and provide everything.
This is the largest detractor to "small and clean" that I hear users complain about today. It is true! But most people do not need these options in daily life. So that is perhaps best considered a documentation organization issue.
gpshead commentedon Mar 26, 2024
Stepping back from a "What do users actually need?" point of view, nothing has provided that:
exec'd.Idea from above: Launching an intermediate Python interpreter process to run the preexec_fn code.
This is a good idea in terms of code compatibility. It would add a large Python interpreter startup time to the child process launch. But functionality wise it would work and the caveat can be documented. It also requires that a
sys.executableinterpreter binary to launch exists (excluding users of embedded Python, and possibly users running under an emulator? A deliberate choice we could make).Brainstorming further Idea: A smaller binary to execute as the intermediate is one possibility. A microscopic tiny Python-ish interpreter with a restricted set of APIs (no import capability, presume
osandsysexist, etc) sufficient to just allow the kinds of system calls people need to make? I don't like the complexity, but it'd be entertaining to see how feasible this is. People can already manually use things like/bin/shfor some fraction of pieces of this concept today (very system dependent).astrand commentedon Mar 31, 2024
@gpshead thanks for your comments. preexec_fn is just a way to execute code between fork and exec, so as far as I can tell, the problem is exactly the same if you manually code fork() and exec() and some code in between. I assume that neither fork() nor exec() will be removed from the Python API, so why should preexec_fn be removed? As I understand it, Python 3.12 already has a mechanism in place to raise a DeprecationWarning if threads are detected. Why not use the same approach for preexec_fn?
gpshead commentedon Mar 31, 2024
The same warning can and should be added there (I'll get to it).
eli-schwartz commentedon May 21, 2024
I've been trying to figure out if I can change from a preexec_fn that does os.setuid "to irrevocably drop permissions", to the user= kwarg. It makes for fairly obtuse reading, since the subprocess module documentation somewhat vaguely says "the setreuid() system call will be made" but not what arguments the low-level description of syscalls will be called with, and my underlying goal was actually to just "irrevocably switch to that user" (a high-level goal).
I think it is safe "depending on your version of POSIX conformance" assuming that cpython passes the user kwarg twice to setreuid, (rather than 3 times to setresuid ;)). Do I understand correctly?
ofek commentedon Feb 7, 2025
To create a daemon process on Linux that continues running after the current process exits, does not block from I/O and is immune to signals from the current process group, is the following sufficient? Or does one need to manually do that double fork technique?subprocess.Popen(command, start_new_session=True, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, close_fds=True)I asked on the forum instead.
dmitmel commentedon Dec 30, 2025
I was trying to use
preexec_fnto reset the dispositions ofSIGINTandSIGQUITtoSIG_DFLin the subprocess because my script which sets those signals toSIG_IGNin the parent, so that the parent does not die when CTRL+C is issued, but the child receives that signal (basically to emulate the way interactive shells spawn foreground jobs). I wish thesubprocess.PopenAPI exposed something akin toPOSIX_SPAWN_SETSIGMASKandPOSIX_SPAWN_SETSIGDEFofposix_spawn, both of which are accessible withos.posix_spawnfrom Python, by the way.JackKelly commentedon Jan 14, 2026
Thanks for everyone's work on this. Just to mention that, like some other folks in the comments above, I'm using
preexec_fnto set thePDEATHSIGof the child process. (In my case, I'm setting it toSIGTERMso the child dies if the parent is killed).Instead of having to write code to kill the child process if the parent is killed, I'd much prefer to be able to pass an argument to
Popento saykill_child_if_parent_is_killed=True(although I'm sure you can think of a better name for that argument!).ndahn commentedon Aug 10, 2026
Instead of adding more and more function args, how about a
ProcessSettingsdataclass-like object that's handled by the internals? This would be very easy to extend without making the API more messy.Background: I'm the author of better_launch, a process launcher and manager for ROS2. Right now I'm modifying the
process groupandniceness, but additional customization may be needed in the future.