Skip to content

"z" format specifier is treated differently in unicode and bytes #104018

Closed
Closed
"z" format specifier is treated differently in unicode and bytes#104018
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@navytux
navytux
opened on Apr 30, 2023

Hello up there. I've hit a discrepancy in how z flags is handled by % in unicode and bytes:

kirr@deca:~$ python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> '%zf' % 1                                       <--   unicode
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unsupported format character 'z' (0x7a) at index 1

>>> b'%zf' % 1                                      <--   bytes
b'1.000000'

>>> b'%zf' % 0.0                                    <--   +0 -> 0
b'0.000000'

>>> b'%zf' % -0.0                                   <--   -0 -> 0
b'0.000000'

>>> b'%f' % -0.0                                    <--   -0 -> -0 if run without 'z'
b'-0.000000'

In other words there is inconsistency in how 'z' is handled by '%' for unicode and bytes, and there is also inconsistency in how 'z' was supposed to be handled by .format and not handled by '%' as originally discussed on BPO-45995.

'z' handling was implemented in #30049 and indeed there I see b'%z' being fully handled:

b0b836b20cb5#diff-f6d440aad34e1c4535c0d898c0197a95490766c745991caace6f64b5dd1ece51

but u'%z' being only partly handled internally without corresponding frontend parsing that bytes has:

b0b836b20cb5#diff-34c966e7876d6f8bf801dd51896327e4f68bba02cddb95fbf3963f0b2e39c38a

In my view the fix should be either a) to add '%z' handling to unicode, or b) to remove '%z' handling from bytes.

Thanks beforehand,
Kirill

  • CPython versions tested on: 3.11.2
  • Operating system and architecture: Debian GNU/Linux 12 on AMD64

/cc @belm0, @mdickinson

Linked PRs

Activity

  1. navytux
    added
    type-bugAn unexpected behavior, bug, or error
    on Apr 30, 2023
  2. arhadthedev
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Apr 30, 2023

  3. belm0 commented on May 1, 2023

    @belm0
    belm0
    on May 1, 2023
    Contributor

    Good catch. I think having that enabled was left over from an early incarnation of the PR, before it was decided that %-format would not be supported.

    For now, I confirmed that the tests still pass after disable of case 'z' in _PyBytes_FormatEx().

    Fix should include tests to confirm that "z" format is not accepted for %-formatting.

  4. belm0
    added a commit that references this issue on May 1, 2023

    pythongh-104018: disallow "z" format specifier in %-format of byte st…

    f697407
  5. mdickinson
    added a commit that references this issue on May 1, 2023

    gh-104018: disallow "z" format specifier in %-format of byte strings (G…

    3ed8c88
  6. miss-islington
    added a commit that references this issue on May 1, 2023

    pythongh-104018: disallow "z" format specifier in %-format of byte st…

    e18148a

  7. mdickinson commented on May 1, 2023

    @mdickinson
    mdickinson
    on May 1, 2023
    Member

    Fixed in #104018 and fix backported to 3.11 in #104033. Thanks @navytux for the report and @belm0 for the quick fix.

  8. mdickinson
    closed this as completedon May 1, 2023
  9. mdickinson
    added a commit that references this issue on May 1, 2023

    [3.11] gh-104018: disallow "z" format specifier in %-format of byte s…

    10db28b
  10. carljm
    added a commit that references this issue on May 1, 2023

    Merge branch 'main' into inlinecomp2

    1274e2b

  11. navytux commented on May 2, 2023

    @navytux
    navytux
    on May 2, 2023
    Author

    @belm0, @mdickinson, thanks for the prompt fix.

    May I ask why static formatfloat() in bytesobject.c remains with F_NO_NEG_0 handling? Offhand it looks like that flag bit could never make it into that function, but I might be missing something. The same question applies to unicodeobject.c .

  12. belm0
    added a commit that references this issue on May 2, 2023

    pythongh-104018: remove unused format "z" handling in string objects

    35ba4e4

  13. belm0 commented on May 2, 2023

    @belm0
    belm0
    on May 2, 2023
    Contributor

    May I ask why static formatfloat() in bytesobject.c remains with F_NO_NEG_0 handling? Offhand it looks like that flag bit could never make it into that function, but I might be missing something. The same question applies to unicodeobject.c .

    Thank you, please see #104107

  14. belm0
    added a commit that references this issue on May 2, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    a3be962

  15. navytux commented on May 3, 2023

    @navytux
    navytux
    on May 3, 2023
    Author

    Thanks

  16. kumaraditya303
    added a commit that references this issue on May 7, 2023

    gh-104018: remove unused format "z" handling in string formatfloat() (#…

    69621d1
  17. miss-islington
    added a commit that references this issue on May 7, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    eea8edc
  18. kumaraditya303
    added a commit that references this issue on May 7, 2023

    [3.11] gh-104018: remove unused format "z" handling in string formatf…

    15ffcf7
  19. jbower-fb
    added a commit that references this issue on May 8, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    06707b9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions