test_glob: test_selflink() fails randomly on Linux

AMD64 RHEL8 Refleaks 3.x:

0:09:26 load avg: 9.08 [406/463/1] test_glob failed (1 failure) -- running (7): (...)
beginning 6 repetitions
123456
.....test test_glob failed -- Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_glob.py", line 396, in test_selflink
    self.assertIn(path, results)
AssertionError:
'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/' not found in
{'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'}

The tested path dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link has one less link/ than the first path of the test: 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.

# Tested path
>>> len('dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.split('/'))
33

# First (shorted) expected path
>>> len('dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'.split('/'))
34

build: https://buildbot.python.org/all/#/builders/259/builds/892

Linked PRs

I didn't see this failure recently, I close the issue.

Seen recently in a PR CI run.

Dumping the expected value and the set content from the failed run.

The shortest value in the set has one more link/ entry than the expected value that the test is looking for, so it's definitely the same failure mode, but the absolute numbers are different (27 vs 28 here, rather than the 33 vs 34 that Victor saw):

AssertionError:
'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
not found in
{
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/',
 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
}

>>> s = ... pasted set data ...
>>> expected = 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/'
>>> expected in s
False
>>> expected + 'link/' in s
True
>>> len(expected.split("/"))
27

I assume it gets up to a max path length, so the OS & CWD will affect that number.

That was my first thought too, but it doesn't seem to be the case, since all the paths in the set were accessible on disk (the set being checked is derived from the glob result). The largest value in the set will be dictated by the OS and CWD, but the missing value isn't the largest one, it's somewhere in the middle. In the latest failure, the remaining 15 iterations would have worked:

>>> for remaining in sorted(s):
...     print(remaining.count("/"), end=", ")
...
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,

I also confirmed both tracebacks are pointing at the same assertion:

Sep 2023 code:

cpython/Lib/test/test_glob.py

Line 396 in ae36cd1

self.assertIn(path, results)
Jun 2024 code:

cpython/Lib/test/test_glob.py

Line 553 in a86e625

self.assertIn(path, results)

To get the reported failures:

every entry up to the failing depth was in the set (otherwise the expected value would be shorter, as the loop starts at depth 0)
this path is shorter than the possible maximum path length, otherwise the set would be empty

I'm tempted to change the way this test works to build a full set of expected values and then do a set comparison rather than checking the entries one by one. Something like:

            # Check "**/" (or platform equivalent) has no duplicates or skipped entries
            results = glob.glob(os.path.join('**', ''), recursive=True)
            self.assertEqual(len(results), len(set(results)))
            results = set(results)
            max_depth = len(results)
            expected_results = {}
            for depth in range(max_depth):
                expected_results.add(os.path.join(*(['dir'] + ['link'] * depth + [''])))
            self.assertEqual(results, expected_results)

(and similarly for the other two search loops)

It won't fix the problem, but it should make it clearer how many entries are actually missing when the test does fail (since it will only show the diff, rather than all the remaining entries after the first missing entry).

It's hard to see any possible source of discrepancies between "directories scanned" and "paths reported" in a recursive glob, since _rlistdir tries to unconditionally recurse down into every reported name: https://github.com/python/cpython/blob/main/Lib/glob.py#L197

The check for whether that recursion operation is valid then appears to just be whether os.scandir throws OSError or not: https://github.com/python/cpython/blob/main/Lib/glob.py#L152

What a mysterious bug!

The pattern in question ("**/") triggers this bit of code:

cpython/Lib/glob.py

Lines 54 to 60 in 6b280a8

    
           if not pathname or recursive and _isrecursive(pathname[:2]): 
        
               try: 
        
                   s = next(it)  # skip empty string 
        
                   if s: 
        
                       it = itertools.chain((s,), it) 
        
               except StopIteration: 
        
                   pass

I suppose it's possible there's a bug in itertools.chain(), or the way iglob() uses it.

(edit: removing the topic-pathlib label as pathlib isn't relevant here)

Hypothesis tests on Ubuntu: https://github.com/python/cpython/actions/runs/12415820872/job/34663070210?pr=128097

FAIL: test_selflink (test.test_glob.SymlinkLoopGlobTests.test_selflink)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_glob.py", line 543, in test_selflink
    self.assertIn(path, results)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file' not found in {'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file', 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file', 'dir/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/file'}

pythongh-109959: Log the current directory in test_glob.test_selflink()

gh-109959: Log the current directory in test_glob.test_selflink() (#1…

Does that imply that it's a glibc or kernel bug?

I can see the behavior at the syscall level, so it's the behavior of the Linux kernel. I don't know if it's a bug. I don't know if Linux has guarantee that all path operations support up to 40 levels of links.

My man path_resolution says:

In order to protect the kernel against stack overflow, and also to protect against denial of service, there are limits on the maximum recursion depth, and on the maximum number of symbolic links followed. An ELOOP error is returned when the maximum is exceeded ("Too many levels of symbolic links").

As currently implemented on Linux, the maximum number of symbolic links that will be followed while resolving a pathname is 40. In kernels before 2.6.18, the limit on the recursion depth was 5. Starting with Linux 2.6.18, this limit was raised to 8. In Linux 4.2, the kernel's pathname-resolution code was reworked to eliminate the use of recursion, so that the only limit that remains is the maximum of 40 resolutions for the entire pathname.

I'm on 5.15.0-126-generic FWIW.

Could SELinux somehow cause this?

Could SELinux somehow cause this?

I saw the issue on the Ubuntu Hypothesis CI, and Ubuntu doesn't use SELinux (but AppArmor).

pythongh-109959: Skip test_glob.test_selflink() flaky test

gh-109959: Skip test_glob.test_selflink() flaky test (#128812)

pythongh-109959: Skip test_glob.test_selflink() flaky test (pythonGH-…

[3.13] gh-109959: Skip test_glob.test_selflink() flaky test (GH-128812)…

pythongh-109959: Skip test_glob.test_selflink() flaky test (pythonGH-…

The flaky test is now skipped. Since the root issue comes from the Linux kernel, maybe we should just remove the test. Or the glob module should be modified to detect symlink loops and don't visit broken links. I close the issue.

[3.12] gh-109959: Skip test_glob.test_selflink() flaky test (GH-128812)…

FWIW this might be fixed by #116392, because it no longer stat()s paths discovered by os.scandir()

pythongh-109959: Remove test_glob.test_selflink()

gh-109959: Remove test_glob.test_selflink() (#130551)

Navigation Menu

Repository navigation

test_glob: test_selflink() fails randomly on Linux #109959

Linked PRs

12 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if not pathname or recursive and _isrecursive(pathname[:2]):
	try:
	s = next(it) # skip empty string
	if s:
	it = itertools.chain((s,), it)
	except StopIteration:
	pass

test_glob: test_selflink() fails randomly on Linux #109959

Description

Linked PRs

Activity

vstinner commented on Nov 8, 2023

encukou commented on Jun 19, 2024

ncoghlan commented on Jun 19, 2024

encukou commented on Jun 19, 2024

ncoghlan commented on Jun 19, 2024

ncoghlan commented on Jun 19, 2024

ncoghlan commented on Jun 19, 2024

barneygale commented on Jun 29, 2024

vstinner commented on Dec 19, 2024

12 remaining items

barneygale commented on Jan 8, 2025

vstinner commented on Jan 8, 2025

barneygale commented on Jan 8, 2025

vstinner commented on Jan 8, 2025

vstinner commented on Jan 14, 2025

barneygale commented on Jan 14, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions