Repository navigation
Speed up pathlib.Path.glob() by removing redundant regex matching #115060
Description
In #104512 we made pathlib.Path.glob() use a "walk-and-filter" strategy for expanding ** wildcards in patterns: when we encounter a ** segment, we immediately consume subsequent segments and use them to build a regex that is used to filter results. This saves a bunch of scandir() calls.
However! We actually build a regex for the entire pattern given to glob(), rather than just the segments following ** wildcards. And so when evaluating a pattern like dir*/**/file*, the dir* part is needlessly matched twice against each path. @zooba noted this in a review comment at the time.
We should be able to improve performance by building an re.Pattern only for segments following ** wildcards, and not the entire glob() pattern.
Linked PRs
- GH-115060: Speed up
pathlib.Path.glob()by removing redundant regex matching #115061 - GH-115060: Speed up
pathlib.Path.glob()by skipping directory scanning #116152 - GH-115060: Speed up
pathlib.Path.glob()by not scanning literal parts #117732 - GH-115060: Speed up
pathlib.Path.glob()by omitting initialstat()#117831
Activity
pythonGH-115060: Speed up `pathlib.Path.glob()` by removing redundant…
GH-115060: Speed up `pathlib.Path.glob()` by removing redundant regex…
pythonGH-115060: Speed up `pathlib.Path.glob()` by removing redundant…
barneygale commentedon Feb 26, 2024
Re-opening: there's another optimization possible.
Previous versions of pathlib used
path.exists()orpath.is_dir()to check whether non-wildcard segments exist, rather than callingpath._scandir()on the parent and filtering through a regex. This was dropped when we added the case_sensitive argument.If we can somehow check that the underlying filesystem is case-sensitive, and the user sets
case_sensitive=True(or leaves it at the default ofcase_sensitive=Noneon POSIX), then we can restore this more direct check for existence. And we can do better! For patterns like*/foo/bar.py, we can skip checking whetherfoois a directory, because we're going to checkfoo/bar.pyexists anyway. Likewise forfoo/*, where the_scandir()call failing makes anis_dir()check irrelevant.Alternatively, we could add a normalize_case argument to
glob(), defaulting toTrue. If set toFalse, and case_sensitive is effectivelyFalse, then literal segments in results wouldn't be case-normalized.Alternatively alternatively, we could enable this behaviour specifically when case_sensitive is set to
None(the default), and so users would need to setcase_sensitive=Falseto get fully case-normalized results.I need to think about this more.
pythonGH-115060: Speed up `pathlib.Path.glob()` by skipping directory…
barneygale commentedon Apr 6, 2024
Closed because I'm planning to make pathlib use the
globmodule for globbing, see #116392barneygale commentedon Apr 6, 2024
Re-opening because I'm a prize eejit.
pythonGH-117586: Speed up `pathlib.Path.glob()` by working with strings
GH-117586: Speed up `pathlib.Path.glob()` by working with strings (#1…
pythonGH-115060: Speed up `pathlib.Path.glob()` by not scanning liter…
GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal par…
barneygale commentedon Apr 13, 2024
Sorry for the close/open spam, I keep spotting things to do.
pythonGH-115060: Speed up `pathlib.Path.glob()` by omitting initial `…
GH-115060: Speed up `pathlib.Path.glob()` by omitting initial `stat()` (
pythonGH-117586: Speed up `pathlib.Path.glob()` by working with strin…
pythonGH-115060: Speed up `pathlib.Path.glob()` by not scanning liter…
pythonGH-115060: Speed up `pathlib.Path.glob()` by omitting initial `…