Repository navigation
Improve performance of os.walk() #119169
Description
There are a couple of minor performance improvements possible in os.walk():
- We don't need to manually pump the
os.scandiriterator, given we handle exceptions fromnext()like exceptions fromscandir()itself, i.e. by ignoring the problematic directory and moving on. We can use aforloop like filthy casuals. In bottom-up mode, we can handle exceptions fromentry.is_symlink()in the same block as those fromentry.is_dir(), which avoids a few temporary variables.- In top-down mode, we can call
os.path.join()once on a parent directory rather than for each child path.
Linked PRs
- GH-119169: Speed up
os.walk(topdown=False)#119186 - GH-119169: Skip reversing sibling directories in
os.[f]walk(topdown=False)#119473 - GH-119169: Implement
pathlib.Path.walk()usingos.walk()#119573 - [3.13] GH-119169: Implement
pathlib.Path.walk()usingos.walk()(GH-119573) #119750 - GH-119169: Slightly speed up
os.walk(topdown=True)#121431 - GH-119169: Simplify
os.fwalk()exception handling #121432 - GH-119169: Speed up
os.fwalk(topdown=False)#121433 - GH-119169: Simplify
os.walk()exception handling #121435
Activity
barneygale commentedon May 19, 2024
Upon closer inspection, I think I was wrong. Symlinks to directories where
is_symlink()raises should still be categorised asdirs.pythonGH-119169: Speed up `os.walk()`
pythonGH-119169: Skip reversing sibling directories in `os.walk(topdo…
pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()`
pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()`
GH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (#119573)
pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (p…
[3.13] GH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (G…
pythonGH-119169: Slightly speed up `os.fwalk()`
pythonGH-119169: Speed up `os.fwalk(topdown=False)`
serhiy-storchaka commentedon Jul 8, 2024
There are too many open PRs. Are they independent? Alternative? What should I review?
GH-119169: Simplify `os.walk()` exception handling (#121435)
barneygale commentedon Jul 8, 2024
The two draft PRs (GH-121433 and GH-119186) depend on GH-119473. The three non-draft PRs are independent and can be reviewed/accepted/rejected in any order or combination.
pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (p…
pythonGH-119169: Simplify `os.walk()` exception handling (python#121435)
pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (p…
pythonGH-119169: Simplify `os.walk()` exception handling (python#121435)
GH-119169: Slightly speed up `os.walk(topdown=True)` (GH-121431)
pythonGH-119169: Slightly speed up `os.walk(topdown=True)` (pythonGH-…