Repository navigation
test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4) #91169
Description
Activity
Since build 298, test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10:
https://buildbot.python.org/all/#/builders/694/builds/2982 changes of build 298:
- bpo-46986: Upgrade bundled setuptools to 60.9.3 (bpo-46986: Upgrade bundled setuptools to 60.9.3 #31820)(21 hours ago)
- [3.10] bpo-46985: Upgrade bundled pip to 22.0.4 (bpo-46985: Upgrade bundled pip to 22.0.4 #31819) ([3.10] bpo-46985: Upgrade bundled pip to 22.0.4 (GH-31819) #31849)(21 hours ago)
Logs:
0:02:03 load avg: 9.95 [213/427/1] test_distutils failed (uncaught exception) -- running: test_concurrent_futures (42.4 sec) test test_distutils crashed -- Traceback (most recent call last): File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 335, in _runtest_inner refleak = _runtest_inner2(ns, test_name) File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 280, in _runtest_inner2 the_module = importlib.import_module(abstest) File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_distutils.py", line 15, in <module> import distutils.tests ModuleNotFoundError: No module named 'distutils.tests'
======================================================================
FAIL: test_skip (test.test_bdb.StateTestCase)
----------------------------------------------------------------------Traceback (most recent call last): File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 730, in test_skip with TracerRun(self, skip=skip) as tracer: File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 448, in __exit__ self.test_case.fail(err_msg) File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 582, in fail raise self.failureException(msg) from None AssertionError: Wrong event type at expect_set item 2, got 'call' Expected: ('line', 3, 'tfunc_import') Got: ('call', 84, 'find_spec'), ('quit',),
Reproduce:
$ ./configure --prefix /opt/py310 && make clean && make && make install $ cd /somewhere/else/ $ /opt/py310/bin/python3 -m test -v test_bdb ... FAIL: test_skip (test.test_bdb.StateTestCase) ... $ /opt/py310/bin/python3 -m test -v test_distutils ... ModuleNotFoundError: No module named 'distutils.tests' ...
setuptools installs this file:
$ cat /opt/py310/lib/python3.10/site-packages/distutils-precedence.pth import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();
See: [BUG] Having setuptools installed causes cpython stdlib build to fail
pypa/setuptools#3007x86 Gentoo Installed with X 3.9:
https://buildbot.python.org/all/#builders/527/builds/4744 tests failed:
test_bdb test_distutils test_importlib test_peg_generatorPython 3.9 gets two more issues: test_importlib and test_peg_generator.
======================================================================
FAIL: test_package_discovery (test.test_importlib.test_main.DiscoveryTests)
----------------------------------------------------------------------Traceback (most recent call last): File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_importlib/test_main.py", line 159, in test_package_discovery assert all( AssertionError 0:06:27 load avg: 3.96 [ 83/425/3] test_peg_generator failed (uncaught exception) -- running: test_multiprocessing_forkserver (1 min 27 sec) Warning -- warnings.filters was modified by test_peg_generator Failed to import test module: test.test_peg_generator.test_c_parser Traceback (most recent call last): File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 436, in _find_test_path module = self._get_module_from_name(name) File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name __import__(name) File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_peg_generator/test_c_parser.py", line 4, in <module> from distutils.tests.support import TempdirManager ModuleNotFoundError: No module named 'distutils.tests'
- changed the title
[-]3.10: test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10[/-][+]test_bdb and test_distutils fail on installed Python 3.9 and 3.10 (setuptools 60.9.3, pip 22.0.4)[/+]on Mar 14, 2022 - changed the title
[-]3.10: test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10[/-][+]test_bdb and test_distutils fail on installed Python 3.9 and 3.10 (setuptools 60.9.3, pip 22.0.4)[/+]on Mar 14, 2022 AMD64 Fedora Stable Clang Installed 3.x:
https://buildbot.python.org/all/#builders/350/builds/16212 re-run tests:
test_bdb test_distutils- changed the title
[-]test_bdb and test_distutils fail on installed Python 3.9 and 3.10 (setuptools 60.9.3, pip 22.0.4)[/-][+]test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4)[/+]on Mar 14, 2022 10 remaining items
From pypa/setuptools#3007:
If there is a
pybuilddir.txtnext tosetup.pythen it's a CPython Unix build. All the other factors are not reliable.If tiran is right, then Setuptools is already doing everything it can not to interfere with CPython builds.
SETUPTOOLS_USE_DISTUTILSis not an option. We cannot reliable inject the env var into our tests harness. The problem has to be solved in setuptools.It appears that Setuptools is unable to do anything to detect that the stdlib distutils is to be preferred.
Trying to think outside the box, maybe there's a way that the distutils hook could be extended to detect the import of any module matching
test.test_, and if that's the case, disable the hook as well.It's not simply enough to detect an attempt to import
test, becausetest.supportis available and used by third party packages.In fact, maybe Setuptools could simply disable the hook if the specifically-implicated modules are encountered.
I can try that, maybe using the repro above.
I was able to repro at least some of the failures with this dockerfile:
FROM jaraco/multipy-tox RUN apt install -y libarchive-tools libssl-dev RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf - WORKDIR cpython-3.9 RUN sh ./configure --prefix /opt/python RUN make install WORKDIR / RUN /opt/python/bin/python3 -m pip install --upgrade pip setuptools CMD /opt/python/bin/python3 -m test
Here's an excerpt from the output:
5 tests failed: test_bdb test_distutils test_importlib test_peg_generator test_zipfileAttempting to install a pre-release of setuptools, I find that the custom build of Python is inadequate because
_ctypesisn't available, soimport wheel.macosx_libfilefails. I need to figure out what dependencies are needed to get ctypes to build.I suspect
libffi-devwas the required dependency, but I've just copied all of the dependencies from the developer guide:FROM jaraco/multipy-tox RUN apt install -y libarchive-tools RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf - WORKDIR cpython-3.9 RUN sh ./configure --prefix /opt/python RUN make install WORKDIR / RUN /opt/python/bin/python3 -m pip install --upgrade pip RUN /opt/python/bin/python3 -m pip install git+https://github.com/pypa/setuptools@experiment/sensitive-tests CMD /opt/python/bin/python3 -m test
My attempts to have setuptools selectively disable the distutils override worked for all of the tests except
test_bdb. The failing test is test_skip. I suspected and then confirmed that that test fails even if the distutils patching is disabled but the Finder is still present. It seems the presence of another finder on meta_path is sufficient to breaktest_bdb.StateTestCase.test_skip.Here's a repro not involving Setuptools showing how
test_bdbis sensitive tosys.meta_pathmanipulation:FROM jaraco/multipy-tox RUN apt install -y libarchive-tools RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf - WORKDIR cpython-3.9 RUN sh ./configure --prefix /opt/python RUN make install WORKDIR / RUN /opt/python/bin/python3 -m pip install --upgrade pip RUN /opt/python/bin/python3 -m pip uninstall -y setuptools RUN echo "import sys; sys.meta_path.insert(0, type('NullFinder', (), dict(find_spec=lambda *args: None))())" > /opt/python/lib/python3.9/site-packages/null-finder.pth CMD /opt/python/bin/python3 -m test_bdbIt fails with a similar error:
AssertionError: Wrong event type at expect_set item 2, got 'call' Expected: ('line', 3, 'tfunc_import') Got: ('call', 1, '<lambda>'), ('quit',),Based on this finding, I don't think there's anything that Setuptools can do short of completely removing the hook, which is a non-starter because (a) it would break all users relying on the hook to provide distutils when it's not available in the stdlib and (b) it would remove the ability for use cases still reliant on stdlib to opt out.
I suggest we'll want to find a fix for test_bdb independent of the concerns introduced by setuptools.
With this Dockerfile, I can confirm the proposed fix in Setuptools has the desired effect:
FROM jaraco/multipy-tox RUN apt install -y libarchive-tools RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf - WORKDIR cpython-3.9 RUN sh ./configure --prefix /opt/python RUN make install RUN chmod u+x /opt/python/lib/python3.9/test/ziptestdata/exe_with_* WORKDIR / RUN /opt/python/bin/python3 -m pip install --upgrade pip RUN /opt/python/bin/python3 -m pip install git+https://github.com/pypa/setuptools@bugfix/3383-cpython-sensitive-tests CMD /opt/python/bin/python3 -m test
$ docker run -it -v /Users/jaraco/p/pypa/setuptools:/src @$(docker build -q .) /opt/python/bin/python3 -m test test_importlib test_distutils test_peg_generator 0:00:00 load avg: 0.11 Run tests sequentially 0:00:00 load avg: 0.11 [1/3] test_importlib 0:00:01 load avg: 0.11 [2/3] test_distutils /opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") 0:00:02 load avg: 0.11 [3/3] test_peg_generator /opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") == Tests result: SUCCESS == All 3 tests OK. Total duration: 3.0 sec Tests result: SUCCESSUnfortunately, it also creates its own regression in
test_venv:$ docker run -it -v /Users/jaraco/p/pypa/setuptools:/src @$(docker build -q .) /opt/python/bin/py thon3 -m test test_venv 0:00:00 load avg: 0.04 Run tests sequentially 0:00:00 load avg: 0.04 [1/1] test_venv test test_venv failed -- Traceback (most recent call last): File "/opt/python/lib/python3.9/test/test_venv.py", line 544, in test_with_pip self.do_test_with_pip(True) File "/opt/python/lib/python3.9/test/test_venv.py", line 511, in do_test_with_pip out, err = check_output([envpy, File "/opt/python/lib/python3.9/test/test_venv.py", line 44, in check_output raise subprocess.CalledProcessError( subprocess.CalledProcessError: Command '['/tmp/tmpqts0cln1/bin/python3', '-W', 'ignore::DeprecationWarning', '-I', '-m', 'ensurepip._uninstall']' returned non-zero exit status 1. test_venv failed (1 error) == Tests result: FAILURE == 1 test failed: test_venv Total duration: 5.7 sec Tests result: FAILUREAnd annoyingly, that error message is all but worthless for diagnosis.
After adding a breakpoint where the error is occurring (pytest would do this automatically with
--pdb), I'm able to inspect the error:(Pdb) print(exc.stderr.decode()) Traceback (most recent call last): File "<string>", line 6, in <module> File "/opt/python/lib/python3.9/runpy.py", line 221, in run_module mod_name, mod_spec, code = _get_module_details(mod_name) File "/opt/python/lib/python3.9/runpy.py", line 130, in _get_module_details spec = importlib.util.find_spec(mod_name) File "/opt/python/lib/python3.9/importlib/util.py", line 103, in find_spec return _find_spec(fullname, parent_path) File "<frozen importlib._bootstrap>", line 925, in _find_spec File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 79, in find_spec return method() File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 100, in spec_for_pip if self.pip_imported_during_build(): File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 111, in pip_imported_during_build return any( File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 112, in <genexpr> frame.f_globals['__file__'].endswith('setup.py') KeyError: '__file__' Traceback (most recent call last): File "/opt/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/python/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/opt/python/lib/python3.9/ensurepip/_uninstall.py", line 31, in <module> sys.exit(_main()) File "/opt/python/lib/python3.9/ensurepip/_uninstall.py", line 27, in _main return ensurepip._uninstall_helper(verbosity=args.verbosity) File "/opt/python/lib/python3.9/ensurepip/__init__.py", line 152, in _uninstall_helper return _run_pip(args + [p[0] for p in reversed(_PROJECTS)]) File "/opt/python/lib/python3.9/ensurepip/__init__.py", line 34, in _run_pip return subprocess.run([sys.executable, "-c", code], check=True).returncode File "/opt/python/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/tmp/tmpntevhrty/bin/python3', '-c', '\nimport runpy\nimport sys\nsys.path = [] + sys.path\nsys.argv[1:] = [\'uninstall\', \'-y\', \'--disable-pip-version-check\', \'pip\', \'setuptools\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.It seems there's a new issue, probably brought on by changes in Setuptools.
The failure looks suspiciously similar to pypa/setuptools#2940.
Okay, so Setuptools fixed that issue and released it in 60.0.3, but the version of Setuptools installed into the virtualenv doesn't have that fix. But that doesn't explain why the issue only emerges when the version of Setuptools is bumped.
Aah. I get it. This test is running a venv with
--system-site-packages, so thedistutils-precedence.pthfor the system site packages is taking effect, enablingSETUPTOOLS_USE_DISTUTILS=local, but then because the venv gets an old version of Setuptools with the bug in it, that version gets invoked. This bug will go away when the version of Setuptools installed into the venvs is updated.At this point, pypa/setuptools#3386 (to be released with Setuptools>=62.5.0) and gh-93962 should fix the issue.
@jaraco I upgraded to setuptools 64.0.3 and still get these errors. Is there a solution?
Using docker pull registry.gitlab.com/python-devs/ci-images:active from README for dev container
python3 -m test test_importlib test_distutils test_peg_generatorAll 3 tests fail due to no module found. Pip install --upgrade setuptools doesn't resolve although version is 64.0.3. I also tried make from source and it throws the error at test_peg_generator.
Thanks!
@yilun11 I suspect your issue is different than this one. I can indeed produce an error if I use your repro:
~ $ docker run -it registry.gitlab.com/python-devs/ci-images:active python3 -m test test_importlib | grep ModuleNotFound ModuleNotFoundError: No module named 'test.test_importlib'However, I also observe that I get the same error for any test module:
~ $ docker run -it registry.gitlab.com/python-devs/ci-images:active python3 -m test test_bool | grep ModuleNotFound ModuleNotFoundError: No module named 'test.test_bool'When I look at the Python lib directory, I see a
supportsubpackage, but no modules for the tests themselves. I suspect the image you're using simply doesn't have the Python tests installed, so you shouldn't expect it to work. TheModuleNotFoundErroris a fairly generic error not specific to this issue. This issue is specific to a few select modules not working when Setuptools is present. You can tell if you're affected by the issue if you can work around it by settingSETUPTOOLS_USE_DISTUTILS=stdlib.To close the loop on this issue, I've confirmed that the repro that previously failed no longer fails with the latest Setuptools:
draft $ cat Dockerfile FROM jaraco/multipy-tox RUN apt install -y libarchive-tools RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf - WORKDIR cpython-3.9 RUN sh ./configure --prefix /opt/python RUN make install RUN chmod u+x /opt/python/lib/python3.9/test/ziptestdata/exe_with_* WORKDIR / RUN /opt/python/bin/python3 -m pip install --upgrade pip setuptools CMD /opt/python/bin/python3 -m test draft $ docker run -it @$(docker build -q .) /opt/python/bin/python3 -m test test_importlib test_distutils test_peg_generator 0:00:00 load avg: 0.63 Run tests sequentially 0:00:00 load avg: 0.63 [1/3] test_importlib 0:00:01 load avg: 0.63 [2/3] test_distutils /opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") 0:00:02 load avg: 0.66 [3/3] test_peg_generator /opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") == Tests result: SUCCESS == All 3 tests OK. Total duration: 2.9 sec Tests result: SUCCESS
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: