Repository navigation
'*' matches entire path in fnmatch #72904
Description
| BPO | 28718 |
|---|---|
| Nosy | @serhiy-storchaka, @MojoVampire, @decibel, @Hooloovoo, @tovrstra |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2016-11-16.20:12:07.121>
labels = ['library']
title = "'*' matches entire path in fnmatch"
updated_at = <Date 2019-03-31.12:59:09.371>
user = 'https://github.com/decibel'bugs.python.org fields:
activity = <Date 2019-03-31.12:59:09.371>
actor = 'Toon Verstraelen'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-11-16.20:12:07.121>
creator = 'Jim Nasby'
dependencies = []
files = []
hgrepos = []
issue_num = 28718
keywords = []
message_count = 8.0
messages = ['280985', '281017', '281018', '288608', '290624', '307867', '339054', '339256']
nosy_count = 6.0
nosy_names = ['serhiy.storchaka', 'josh.r', 'Jim Nasby', 'Alberto Galera', 'aaron-whitehouse', 'Toon Verstraelen']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue28718'
versions = []
Activity
decibel commentedon Nov 16, 2016
A '*' in fnmatch.translate is converted into '.*', which will greedily match directory separators. This doesn't match shell behavior, which is that * will only match file names:
From a posix standpoint, this would easily be fixed by using '[^/]*' instead of '.*'. I'm not sure how to make this work cross-platform though.
It's worth noting that some programs (rsync, git) support **, which would correctly translate to '.*'.
MojoVampire commentedon Nov 17, 2016
Presumably something like:
r'(?:' + r'|'.join({re.escape(os.path.sep), re.escape(os.path.altsep)}) + r')'
would cover it completely. I switched to using non-capturing groups over a character class both to deal with the fact that escaping doesn't work the same way for character classes and to cover the possibility (no idea here) that some terrible OS might have a multicharacter path separator.
MojoVampire commentedon Nov 17, 2016
Oops, altsep is None, not the empty string when there is only one separator. And I didn't handle inverting the match. Sigh. You get the idea.
Hooloovoo commentedon Feb 26, 2017
Note that somebody has forked the standard library to implement this:
https://github.com/kianxineki/python-wildcard
This shows that the actual changes would be pretty small (though pywildcard is based on 2.x code and does not handle the cross-platform slashes you have been discussing).
It is also worth noting that the glob standard library:
https://docs.python.org/3.7/library/glob.html
implements a "recursive" option that has similar behaviour (* does not span path separators whereas ** does) and essentially builds this on top of fnmatch for the actual filename matching.
I do not think we can change the default behaviour of fnmatch at this point, but I would like to see this behaviour triggered by an optional argument to the various functions, e.g.:
fnmatch.fnmatch(filename, pattern, glob_asterisks=False)
fnmatch.fnmatchcase(filename, pattern, glob_asterisks=False)
fnmatch.filter(names, pattern, glob_asterisks=False)
fnmatch.translate(pattern, glob_asterisks=False)
In each case, if glob_asterisks (or whatever other name we came up with) is true, the behaviour would match the pywildcard behaviour, i.e.:
** matches everything
* matches in one path level
I look after the glob matching code in duplicity and would like to start using the standard library to do filename matching for us, but we need the above behaviour. I am happy to do the patching if there is a realistic chance of it being accepted.
[-]'*' matches entire path in fnmatch.translate[/-][+]'*' matches entire path in fnmatch[/+]Hooloovoo commentedon Mar 27, 2017
Posted to the [Python-ideas] mailing list, as it is proposing a change to a standard library:
https://mail.python.org/pipermail/python-ideas/2017-February/044880.html
Nobody has responded so far, however. I take this as at least no vehement objection to the idea.
AlbertoGalera commentedon Dec 8, 2017
I see that they have commented on the lib that I made a few years ago (python-wildcard).
The reason for the creation of that little fork started in this issue:
https://bugs.python.org/issue25734
tovrstra commentedon Mar 28, 2019
For consistency with the corresponding feature in the glob function since Python 3.5, I would suggest to add an extra optional argument 'recursive' instead of 'glob_asterisks'. With the default recursive=False, one gets the old behavior, with recursive=True, it can handle the '**' and '*' as in pywildcard.
I realize that with recursive=False, the behavior is not exactly consistent with glob, but I'd still prefer the same name for the optional argument. It is the common terminology for this type of feature. See https://en.wikipedia.org/wiki/Matching_wildcards
tovrstra commentedon Mar 31, 2019
Just for reference, here are a few more implementations of the same idea, next to pywildcard, sometimes combined with other useful features:
The last one is rather active, with regular releases, last one on March 24, 2019.
barneygale commentedon Mar 15, 2023
I have an implementation of this for pathlib:
pathlib.PurePath.match()#101398It exploits a simple trick: swapping path separators and newlines, and then matching without setting
re.DOTALL. This causes*(and other wildcards) to not match directory separators.If folks thought it was a good idea, we could instead put the implementation in an
fnmatch.globmatch()orglob.match()function.barneygale commentedon Jul 2, 2023
Some other differences between
fnmatchandglob:**/has a special meaning inglob.glob(). You might think it means "any number of characters, followed by one slash". In fact it means "any number of characters, as long as we terminate after a slash". Importantly, this means it can match nothing at all, and so a pattern like/etc/**/**/**/hostswill match/etc/hosts.*as a standalone path segment also has a special meaning. You might think it means "any number of any non-slash characters". In fact it means "at least one non-slash character", i.e. it always consumes precisely one path component.pythonGH-72904: Add optional *seps* argument to `fnmatch.translate()`
3 remaining items
Merge branch 'main' into pythongh-72904-fnmatch-seps
barneygale commentedon Aug 13, 2023
I've posted about this on discuss.python.org: https://discuss.python.org/t/add-glob-translate-convert-path-with-shell-wildcards-to-regular-expression/31549
Merge branch 'main' into pythongh-72904-fnmatch-seps
wimglenn commentedon Sep 25, 2023
Adding to the helpful list from tovrstra, I can recommend pathspec.
pythonGH-72904: Simplify implementation of `fnmatch.translate()`
Merge branch 'main' into pythongh-72904-fnmatch-seps
Merge branch 'main' into pythongh-72904-fnmatch-seps
Merge branch 'main' into pythongh-72904-fnmatch-seps
GH-72904: Add `glob.translate()` function (#106703)
barneygale commentedon Nov 13, 2023
Addressed in Python 3.13 / cf67ebf / #106703 -- we've added a new
glob.translate()function that converts a glob expression into a regular expression.pythonGH-72904: Add `glob.translate()` function (python#106703)
hauntsaninja commentedon May 17, 2024
@barneygale minor thing, I noticed that the
patargument is documented aspathname. Should we make the arg positional-only or rename the arg to match the documentation?pythonGH-72904: Add `glob.translate()` function (python#106703)