Repository navigation
test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler? #109096
Description
Bug report
Every test class in test_httpservers starts a thread to run a server. CGIHTTPServerTestCase tests trigger htt.server.CGIHTTPRequestHandler.run_cgi() which executes a CGI script in a separate process. Using os.fork() if possible. But os.fork() now emits a DeprecationWarning if used in multi-threaded environment, and in future it will be an error.
$ ./python -Wa -m test -v test_httpservers -m CGIHTTPServerTestCase
...
test_accept (test.test_httpservers.CGIHTTPServerTestCase.test_accept) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_authorization (test.test_httpservers.CGIHTTPServerTestCase.test_authorization) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_cgi_path_in_sub_directories (test.test_httpservers.CGIHTTPServerTestCase.test_cgi_path_in_sub_directories) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
...
We should do something with this. First, the tests are polluted with presumably false alerts. Second, these alerts may be not false. Third, these tests will be broken in future, when the warning is turned into error.
Also, what actions are expected from users of http.server when they see a warning? How they can solve the problem on their side? If http.server is deprecated, it should be documented. If it remains, it should not produce scary warnings.
cc @gpshead
Activity
gpshead commentedon Sep 12, 2023
At a minimum we should at least silence this particular warning in this particular
CGIHTTPServerTestCaseclass. If this part of the test suite starts hanging on particular platform configurations due to the technically not allowed by posix issue, we can add explicit skips for those. To fix the underlying issue and not silence it:The real bug exposed is that
http.server.CGIHTTPRequestHandlerhas never been safe to use on a multithreaded server. This code - https://github.com/python/cpython/blob/main/Lib/http/server.py#L1172-L1192 - where the warning comes from needs to be rewritten to usesubprocess.But given that
http.serveris not considered a good idea to use in serious applications (aka "production") per the warning at the top of its docs, nobody has been paying attention to that kind of maintenance on it. It does not see much use. https://docs.python.org/3/library/http.server.htmlserhiy-storchaka commentedon Sep 13, 2023
We should not simply silence deprecation warnings. We should make
os.fork()only be used if it is safe (only in single-threaded program) or allow the user to control this by an option. We should rewrite tests and test the fork-using code without threads, and subprocess-using code with threads. We should document the limitations of using CGI in http.server.gpshead commentedon Sep 13, 2023
I wonder if we need CGIHTTPRequestHandler at all? It isn't the '90s any more, CGI is... extremely dated and often problematic. Deprecating it could be another option.
gpshead commentedon Sep 13, 2023
the
cgiandcgitbmodules were removed in 3.13 for https://peps.python.org/pep-0594/#deprecated-modules.Searching internally (which covers a large PyPI/github third_party corpus as well) I only found a single use of
CGIHTTPRequestHandlerin actual code in an internal demo test server. It was a misunderstanding, they didn't need to use that class and should've usedSimpleHTTPRequestHandleras they implemented their owndo_POSTmethod.[-]test_httpservers produces DeprecationWarning[/-][+]test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler?[/+]gpshead commentedon Sep 13, 2023
If anyone can find actual uses of CGIHTTPRequestHandler in real world code, it'd be good to see what still uses it and why.
gh-109096: Deprecate `http.server.CGIHTTPRequestHandler` (#109387)
gpshead commentedon Sep 15, 2023
The deprecation is in for 3.13. I'll work up a small 3.12 specific test-only change to silence the warning on the CGIHTTPRequestHandler test there.
[3.12] pythongh-109096: Silence test_httpservers fork + threads Depre…
[3.12] gh-109096: Silence test_httpservers fork + threads Deprecation…
pythongh-109096: Deprecate `http.server.CGIHTTPRequestHandler` (pytho…