Repository navigation
cProfile/profile do not output results if profiling terminates because of SIGTERM #104676
Description
Bug report
cProfile and profile do not produce any results if the process being profiled is terminated with SIGTERM, although the documentation for cProfile and profile and the behavior of the profiler in the case of an exception (KeyboardInterrupt in particular) do make it seem like the profiler will always write output "at the end" of profiling, and the actual behavior on this edge case is somewhat surprising as a result.
I discovered this behavior while trying to profile a worker subprocess in an application that sends SIGTERM to that subprocess during shutdown. I was able to write my own profiling helper that registers a signal handler, but thought it made sense to open an issue and ask if this is a case of slightly murky documentation, or something that can be improved in profile. Considering the complexity of signal handling, I'm guessing this is more likely to be considered a missing stair in the documentation, but I'm happy to send a PR for either case if others agree that this behavior isn't entirely obvious.
The closest issue I can find in the CPython issue history is #50228 which doesn't involve signals.
Reproduction
$ cat repro.py
import time
x = 0
while True:
x += 1
print(x)
time.sleep(0.1)
$ python3.9 -m cProfile --outfile out.prof repro.py & PID=$!; sleep 0.5; kill -s SIGTERM ${PID}
[2] 1113607
1
2
3
4
[2]- Terminated python3 -m cProfile --outfile out.prof repro.py
$ ls out.prof
ls: cannot access 'out.prof': No such file or directoryEnvironment
- CPython versions tested on: 3.9, 3.8
- Operating system and architecture: Ubuntu 20.04
Activity
sunmy2019 commentedon May 20, 2023
The only way you handle signals (except SIGINT) from the Python side is to use the
signalmodule.sunmy2019 commentedon May 20, 2023
If this really needs a fix, the only thing we can do is register signal handlers.
But is this really needed? Pdb doesn't handle signals either.
SnoopJ commentedon May 20, 2023
Right, this is what the helper I linked to does.
It may not be, but a note in the docs that the profiler will not leave any record if terminated by signal would have saved me some time, and I thought that I should ask about adding one.
gaogaotiantian commentedon May 22, 2023
Well
SIGINTworks :)Handling common cases for
SIGTERMon Linux is definitely possible, but there are a couple of considerations here:SIGTERMis basically useless on Windows, so you still can't handle the case on Windows, which causes inconsistent behavior on different platforms, but that's not a deal breaker.SIGTERMcould potentially be overwritten by user code. Still not a deal breaker, but even more inconsistent behavior.SIGTERMis debatable - depending on how you understand the signal.So the question is not whether we can achieve this, is whether it's worth it to do it - how many new cases are covered with the new feature, how much confusion is caused by it?
cProfiledoes not support multiprocess by default, you'll need some wrappers anyway and you can do this in your wrappers. For the single process usage, it's not quite easy for me to imagine a common use case for this feature(compared toCtrl+Cbreak).Again, I'm not opposed to the feature, just bring up some thoughts on it.
SnoopJ commentedon May 22, 2023
It seems very possible that I'm the only person who is surprised/confused by this behavior. 😅 Since filing this, I've switched[1] from
profileto the third-party profilerpy-spy, and it turns outpy-spyhas the same behavior if the profiler is sentSIGTERM. Maybe it's obvious to others that the profiler just drops its data on the ground in this case.I don't think very many "real" user cases would be covered by extending the profiler to worry about signals, and the complexity it would add is appreciable for the reasons you point out. I think I'm erring in favor of fleshing out the existing docs to make it clear what "end of the profile run" means when using
-m profileor-m cProfile, if any change is worth making.[1] for reasons unrelated to this issue, I realized I need a time-resolved profile, not just call/time statistics
gaogaotiantian commentedon May 23, 2023
Not sure what you meant by "time resolved" profile. The major difference between the results of
py-spyandcProfileis the call stack.cProfileis not able to generate call stacks by design.py-spygenerates a flamegraph which still does not include data like order of events(calls) or duration of individual event.If you are looking for a profiler that generates that kind of data(events on a timeline), supports multiprocessing/subprocess natively and handles
SIGTERM, try viztracer. (Yes, it's my project).Back to the topic, yes we can add some notes on cProfile/profile to mention that data won't be generated if it's terminated by
SIGTERM, that could be helpful. Do you want to submit a PR? I can do it as well.pythongh-104676: Fix error location for constructs with spaces and pa…
pythongh-104676: Fix error location for constructs with spaces and pa…
pythongh-104676: Fix error location for constructs with spaces and pa…
pythongh-104676: Fix error location for constructs with spaces and pa…