Repository navigation
asyncio proc.kill() and proc.wait() are counter intuitive #119710
Description
Activity
It's a surprising feature. communicate() waits until stdout (and stderr) is closed. If a child process spawns "sleep 20", the "sleep 20" inherits stdout and keeps it open for 20 seconds.
Maybe you should create a new process group and kill the whole process group, rather than only killing the "parent" process (parent of "sleep 20"). See the start_new_session parameter of subprocess.Popen and then use
os.killpg().That's what I did in test.libregrtest to make sure that CTRL+C kills immediately all processes, and not only direct child processes.
- Indeed the process groups can help, if the child process does not create another process group. But that is another story. Here my point is that we should clarify The documentation ton avoid any surprise. The same behavior applies to proc.wait(). For the record, the crurent documentation says "Wait for child process to terminate" which looks missleading to me.
wait() and communicate() are very different.
Indeed. This issue is about proc.wait() documentation. I updated the code snippet to male it clearer.
I really consider this a pretty bad bug. It is absolutely impossible to wait for the executed process. I specifically wanted to wait for the launched process and not the pipe to be closed. Unfortunately, with the current API that is pretty much impossible.
There is not even a
timeout=parameter forwait(), so I am now resorting to having a second task that specifically closes the pipe usingproc.stdout._transport._protocol.pipe.close()(after killing the process just in case). I consider this a really nasty workaround, as I just need to wait for the process to quit. However, adding a second child watch is also not really possible …Considering we have the mess, maybe one could:
- Properly document the
wait()behaviour - Fix
Process.wait()to not immediately return if the process already quit (BaseSubprocessTransport._waitneeds to testself._finishednotself._returncodeto be consistent) - Add a
Process.waitpid()function with the expected behaviour that does not wait for the pipes to be closed - Maybe add a
close()function that can be used to explicitly close all pipes. Though I suppose that will happen anyway when the object is deleted.
Reacted by hazyfossa, Alisa Sireneva and Anatoly ParshintsevReacted by Anatoly Parshintsev- Properly document the
I've dug around and the bug seems to be a race condition depending on if
proc.wait()was called before or after the process exited and could set its returncode.import asyncio import sys async def main1(): proc = await asyncio.create_subprocess_exec( sys.executable, "-c", "import os;os.system('sleep 5')", stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) await asyncio.sleep(0.5) proc.kill() await asyncio.wait_for(proc.wait(), 2.0) async def main2(): proc = await asyncio.create_subprocess_exec( sys.executable, "-c", "import os;os.system('sleep 5')", stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) await asyncio.sleep(0.5) proc.kill() while proc.returncode is None: await asyncio.sleep(0.1) await asyncio.wait_for(proc.wait(), 2.0) try: asyncio.run(main1()) print("main1 finished") except asyncio.TimeoutError: print("main1 Timed out") try: asyncio.run(main2()) print("main2 finished") except asyncio.TimeoutError: print("main2 Timed out")
When run gives:
$python test_asyncio_sub.py main1 Timed out main2 finished Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7e6b57e227a0> Traceback (most recent call last): ... # <snip> raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7e6b57e227a0> Traceback (most recent call last): ... # <snip> raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed
I've checked relevant issues and changes and it seems to boil down to #32073 replacing the notifying of blocked waiters from
BaseSubprocessTransport._process_exitedwith instead closing pipes and delaying notifying readers when the closed pipes eventually leads to theconnection_lostcallback being triggered.The pipe-closing was then removed by https://github.com/python/cpython/pull/100154/changes since it led to lost output but didn't re-instate the wakeups.
I've tried simply re-instating the wake-ups in
_process_exited(with minor tweaks and confirmed it solves the above as well as allows the rest of the test-suite to still pass (on linux at least)Reacted by JuliaChecking the rationale for the original change of https://github.com/python/cpython/pull/32073/changes it's explained here
The issue it tried to solve was that pipes can outlive the process and if they do and you need to wait for both to complete before closing the loop to not have resource errors. The question is really if that is the responsibility of
Process.wait)(I'd argue not since we don't guarantee it anyway as above examples show and the synchronoussubprocess.Popenworks the same way including grandchildren causing problems if you use kill)Any thoughts @kumaraditya303
@benzea (Or anyone else needing to wait for just the process exit as is currently)
It's possible to use the builtin notification of process exit by subclassing the protocol class
asyncio.subprocess.SubprocessStreamProtocoland overridingprocess_exitedto for example set anasyncio.Eventyou can wait on.You'll have to use the lower-level
AbstractEventLoop.subprocess_*()methods rather thanasyncio.create_subprocess_execthough to supply your own protocol factory.class _ProcessStreamProtocol(asyncio.subprocess.SubprocessStreamProtocol): """ A subprocess protocol that allows us to be notified of ``process_exited`` """ def __init__(self) -> None: """Match standard factory for asyncio.create_process""" super().__init__(limit=2**16, loop=asyncio.get_running_loop()) self.exited = asyncio.Event() def process_exited(self) -> None: super().process_exited() self.exited.set() async def main(): # Use loop.subprocess_shell()/subprocess_exec() rather than their # asyncio.create_subprocess_*() counterparts to get access to # transport/protocol. loop = asyncio.get_running_loop() transport, protocol = await loop.subprocess_exec( _ProcessStreamProtocol, [sys.executable, "-c", "print('hello!')"] ) process = asyncio.subprocess.Process(transport, protocol, loop) # Should return when process is finished regardless of pipe status. await protocol.exited.wait()
I've dug through the various iterations of this and it seems like simply reinstating the code to wake waiters when the process exits makes it consistent in both cases (neither wait before or after exit are held up by open streams) which also matches uvloop.
All tests passes as well with those changes since the issues with the pipes closing after the loop is closed seems to have been mitigated in other ways.
If that sounds reasonable I'll publish my PR for the fix and would mainly want some help with ensuring reasonable regression tests.
@gvanrossum (you've been involved in a lot of prior review of affected code)
I'll try to look at this soon.
Reacted by Tobias Alex-Petersen- added 3 commits that reference this issue
on Jun 23, 2026 I did some work in #151983 as a suggestion or at least material for discussion.
- Sorry, I am on vacation.
- added a commit that references this issue
on Jul 19, 2026 - added 4 commits that reference this issue
on Jul 20, 2026
Bug report
Bug description:
On Linux
proc.wait()does not return afterproc.kill()when there as sub/sub processes.This only happens when the standard streams are piped, it looks like
wait()blocks until the pipes are closed.I don't know if it's a documentation issue or a bug.
proc.wait()after aproc.kill()returned when the process exited, and does not depend on other processes completion.CPython versions tested on:
3.11
Operating systems tested on:
Linux
Linked PRs