Description of '\w' behavior is vague in re documentation #82747

Activity

1. snoopjedi commented on Oct 23, 2019

   

   snoopjedimannequin

   on Oct 23, 2019

   MannequinAuthor

   More actions

   The documentation for the re library¹ describes the behavior of the specifier '\w' as matching "Unicode word characters," which is very vague. The closest thing I can find that corresponds to this language is the guidance offered in Unicode Technical Standard #18², which defines the class <word_character> to include all alphabetic and decimal codepoints, as well as U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER. This does not appear to be a correct description of re, however, as these zero-width characters are not counted when matching '\w', e.g.:

   >>> re.match('\w*', 'Auf\u200Clage')
   <re.Match object; span=(0, 3), match='Auf'>
   

   It seems from examining the CPython source³ that SRE treats '\w' as meaning any alphanumeric character OR U+005F SPACING UNDERSCORE, which does not match any Unicode class definition I've been able to find.

   Can anyone provide clarification on what part of Unicode this documentation is referring to? If there is some other definition, the documentation should be more specific about referring to it (and including a link would be preferred). If instead the documentation is incorrect, this language should be changed to describe the true meaning of \w.

   ¹ https://docs.python.org/3/library/re.html#index-32
   ² http://unicode.org/reports/tr18/
   ³ https://github.com/python/cpython/blob/master/Modules/_sre.c#L125
2. snoopjedi

   assigned

   docspython

   on Oct 23, 2019
3. snoopjedi

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   on Oct 23, 2019
4. MojoVampire commented on Oct 23, 2019

   

   MojoVampiremannequin

   on Oct 23, 2019

   Mannequin

   More actions

   The definition of \w, historically, has corresponded to the set of characters that can occur in legal variable names in C (alphanumeric ASCII plus underscores, making it equivalent to [a-zA-Z0-9_] for ASCII regex). That's why, on top of the definitely wordy alphabetic characters, and the arguably wordy numerics, it includes the underscore, _.

   That definition predates Unicode entirely, and Python is just building on it by expanding the definition of "alphanumeric" to encompass all alphanumeric characters in Unicode.

   We definitely can't remove underscores from the definition without breaking existing code which assumes a common subset of PCRE support (every regex flavor I know of includes underscores in \w). Adding the zero width characters seems of limited benefit (especially in the non-joiner case; if you're trying to pull out words, presumably you don't want to group letters across a non-joining boundary?). Basically, you're parsing "Unicode word characters" as "Unicode's definition of word characters", when it's really meant to mean "All word characters, not just ASCII".

   You omitted the clarifying remarks from the documentation though, the full description is:

   Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.

   That's about as precise as I think we can make it (because technically, some of the things that count as "word characters" aren't actually part of an "alphabet" in the technical definition). If you think there is a clearer way of expressing it, please suggest a better phrasing, and this can be fixed as a documentation bug.
5. snoopjedi commented on Oct 23, 2019

   

   snoopjedimannequin

   on Oct 23, 2019

   MannequinAuthor

   More actions

   Cheers for the additional context. My recommendation would be to change the language to avoid confusion with the consortium's formal specifications. Describing what SRE does should be fine:

   Matches any alphanumeric Unicode character, as well as '_'. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.

   I think it'd also be nice for the term "alphanumeric Unicode character" to link to the documentation for str.isalnum(), which provides enough clarity for the user to work out exactly what Unicode category properties will end up qualifying as a match.
6. tirkarthi

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 24, 2019
7. ezio-melotti

   transferred this issue fromon Apr 10, 2022
8. AlexWaygood

   added

   topic-regex

   on Apr 19, 2022
9. JelleZijlstra commented on Apr 30, 2022

   

   JelleZijlstra

   on Apr 30, 2022

   Member

   More actions

   Duplicate of #69929
10. JelleZijlstra

    marked this as a duplicate of [doc] Clarify exactly what \w matches in UNICODE mode #69929 on Apr 30, 2022
11. JelleZijlstra

    closed this as completedon Apr 30, 2022