Skip to content

Getting a buffer from a Unicode array uses invalid format #57281

Closed
Closed
Getting a buffer from a Unicode array uses invalid format#57281
Assignees
vstinner
Labels
release-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Sep 30, 2011
BPO 13072
Nosy @loewis, @birkenfeld, @mdickinson, @ncoghlan, @pitrou, @vstinner, @skrah, @meadori
Files
  • array_revert_pep393.patch
  • array_revert_pep393-2.patch
  • array_unicode_format.patch
  • array_deprecate_u.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vstinner'
closed_at = <Date 2012-08-24.18:22:37.401>
created_at = <Date 2011-09-30.00:09:50.065>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Getting a buffer from a Unicode array uses invalid format'
updated_at = <Date 2012-08-24.18:22:37.400>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2012-08-24.18:22:37.400>
actor = 'skrah'
assignee = 'vstinner'
closed = True
closed_date = <Date 2012-08-24.18:22:37.401>
closer = 'skrah'
components = ['Library (Lib)']
creation = <Date 2011-09-30.00:09:50.065>
creator = 'vstinner'
dependencies = []
files = ['26646', '26647', '26704', '26892']
hgrepos = []
issue_num = 13072
keywords = ['patch']
message_count = 45.0
messages = ['144658', '144812', '144814', '144817', '144818', '158381', '158892', '167091', '167109', '167112', '167119', '167122', '167165', '167173', '167520', '167521', '167522', '167540', '167545', '167546', '167547', '167549', '167551', '167561', '167566', '167571', '167673', '167702', '167703', '167708', '167732', '167936', '167947', '167997', '168005', '168369', '168373', '168558', '168561', '168567', '168571', '168575', '169026', '169063', '169065']
nosy_count = 10.0
nosy_names = ['loewis', 'georg.brandl', 'mark.dickinson', 'ncoghlan', 'pitrou', 'vstinner', 'Arfrever', 'skrah', 'meador.inge', 'python-dev']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13072'
versions = ['Python 3.3']

    Linked PRs

    Activity

    1. vstinner commented on Sep 30, 2011

      @vstinner
      vstinner
      on Sep 30, 2011
      MemberAuthor

      In Python 3.2, when you get a buffer from array.array('u'), "u" is used as buffer format. The format is supposed to be a format from the struct module, and "u" is an invalid struct format. "w" is used on wide mode.

      I just upgraded the array module to use the new Unicode API (PEP-393). The array now uses a Py_UCS4 buffer. I used "I" or "L" format depending on the size of int and long C types.

      It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.

      For Python 2.7 and 3.2, I don't know if it is really a bug or not.

    2. vstinner
      added
      stdlibStandard Library Python modules in the Lib/ directory
      on Sep 30, 2011

    3. skrah commented on Oct 3, 2011

      @skrah
      skrahmannequin
      on Oct 3, 2011
      Mannequin

      The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
      (PEP-3118 repo) to fail:

      # Not implemented formats. Ugly, but inevitable. This is the same as
# issue python/cpython#46783: equality is also used for membership testing and must
# return a result.
a = array.array('u', 'xyz')
v = memoryview(a)
self.assertNotEqual(v, a)
self.assertNotEqual(a, v)

      I don't have a better idea though what to do about 'u' except
      officially implementing it for struct and memoryview as well.

    4. skrah commented on Oct 3, 2011

      @skrah
      skrahmannequin
      on Oct 3, 2011
      Mannequin

      It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.

      PEP-3118 suggests for the extended struct syntax:

      'c' -> ucs-1 (latin-1) encoding
      'u' -> ucs-2
      'w' -> ucs-4

    5. vstinner commented on Oct 3, 2011

      @vstinner
      vstinner
      on Oct 3, 2011
      MemberAuthor

      The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
      (PEP-3118 repo) to fail:

      Not implemented formats. Ugly, but inevitable. This is the same as

      issue bpo-2531: equality is also used for membership testing and must

      return a result.

      a = array.array('u', 'xyz')
      v = memoryview(a)
      self.assertNotEqual(v, a)
      self.assertNotEqual(a, v)

      I don't understand: a buffer format is a format for the struct module,
      or for the array module?

    6. skrah commented on Oct 3, 2011

      @skrah
      skrahmannequin
      on Oct 3, 2011
      Mannequin

      STINNER Victor <report@bugs.python.org> wrote:

      > # Not implemented formats. Ugly, but inevitable. This is the same as
      > # issue bpo-2531: equality is also used for membership testing and must
      > # return a result.
      > a = array.array('u', 'xyz')
      > v = memoryview(a)
      > self.assertNotEqual(v, a)
      > self.assertNotEqual(a, v)

      I don't understand: a buffer format is a format for the struct module,
      or for the array module?

      It's like this: memoryview follows the current struct syntax, which
      doesn't have 'u'. memory_richcompare() does not understand 'u', but
      is required to return something for __eq__ and __ne__, so it returns
      'not equal'.

      This isn't so important, since I discovered (see my later post)
      that 'u' and 'w' were scheduled for inclusion in the struct
      module anyway.

      So I think we should focus on whether the proposed 'c', 'u' and 'w'
      format specifiers still make sense after the PEP-393 changes.

    7. vstinner commented on Apr 16, 2012

      @vstinner
      vstinner
      on Apr 16, 2012
      MemberAuthor

      @Stefan: What is the status of this issue?

    8. skrah commented on Apr 20, 2012

      @skrah
      skrahmannequin
      on Apr 20, 2012
      Mannequin

      I'm not sure what to do. Martin's opinion was that the change should
      be reverted:

      http://mail.python.org/pipermail/python-dev/2012-March/117390.html

    9. vstinner commented on Aug 1, 2012

      @vstinner
      vstinner
      on Aug 1, 2012
      MemberAuthor

      Should we do something before Python 3.3 final?

    10. skrah commented on Aug 1, 2012

      @skrah
      skrahmannequin
      on Aug 1, 2012
      Mannequin

      Is it possible without too much effort to keep the old behavior
      ('u' -> Py_UNICODE)? Then I'd say that should go into 3.3.

      The problem with the current behavior is that it's neither backwards
      compatible nor PEP-3118 compliant.

      If it is too much work to restore the status quo, we could leave this
      change with the excuse that 'u' is probably not used very often.

    11. vstinner commented on Aug 1, 2012

      @vstinner
      vstinner
      on Aug 1, 2012
      MemberAuthor

      Here is a patch reverting changes of the PEP-393, as suggested by Martin von Loewis. With the patch, array uses Py_UNICODE* type for the 'u' format. So array.array('u', '\u0010ffff')[0] should return '\uDBFF' on Windows.

    12. skrah commented on Aug 1, 2012

      @skrah
      skrahmannequin
      on Aug 1, 2012
      Mannequin

      The diff between b9558df8cc58 and default with array_revert_pep393.patch
      applied is small, but I noticed that in some places you switched back to
      Py_UNICODE typecode and in others not. For instance, in struct arraydescr
      typecode is still char.

      I'm not sure why typecode was originally Py_UNICODE though.

    13. vstinner commented on Aug 1, 2012

      @vstinner
      vstinner
      on Aug 1, 2012
      MemberAuthor

      The diff between b9558df8cc58 and default with array_revert_pep393.patch
      applied is small, but I noticed that in some places you switched back to
      Py_UNICODE typecode and in others not.

      I just copied code from Python 3.2, I forgot to update typecode type
      (Py_UNICODE => char). I attach a new patch which changes also the
      documentation of the "u" format.

    14. skrah commented on Aug 1, 2012

      @skrah
      skrahmannequin
      on Aug 1, 2012
      Mannequin

      array_revert_pep393-2.patch looks good (checked against 7042a83f37e
      and all following commits that should be kept).

    15. vstinner commented on Aug 1, 2012

      @vstinner
      vstinner
      on Aug 1, 2012
      MemberAuthor

      @georg: are you ok with this change? It reverts the behaviour of Python 3.2 and avoids to have to maintain an API that nobody wants to use ('u' format using Py_UCS4, 32 bits unsigned).

    16. 36 remaining items

    17. loewis commented on Aug 24, 2012

      @loewis
      loewismannequin
      on Aug 24, 2012
      Mannequin

      Stefan, your patch array_deprecate_u.diff is fine. If you get to it, please also rephrase the clause "Python's unicode type"; not sure what the convention is to refer to Py_UNICODE now (perhaps "historical unicode type").

    18. python-dev commented on Aug 24, 2012

      @python-dev
      python-devmannequin
      on Aug 24, 2012
      Mannequin

      New changeset 9c7515e29219 by Stefan Krah in branch 'default':
      Issue bpo-13072: The array module's 'u' format code is now deprecated and
      http://hg.python.org/cpython/rev/9c7515e29219

    19. skrah commented on Aug 24, 2012

      @skrah
      skrahmannequin
      on Aug 24, 2012
      Mannequin

      Good, I think this can be closed then.

    20. skrah
      closed this as completedon Aug 24, 2012
    21. skrah
      added
      type-bugAn unexpected behavior, bug, or error
      on Aug 24, 2012
    22. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    23. StanFromIreland commented on Apr 28, 2025

      @StanFromIreland
      StanFromIreland
      on Apr 28, 2025
      Member

      This has been deprecated for a long time, I was unable to find any uses in the top 1000 pypi projects or on gh. This should be set for removal in 3.15.

    24. vstinner commented on Apr 28, 2025

      @vstinner
      vstinner
      on Apr 28, 2025
      MemberAuthor

      Its removal is scheduled for Python 3.16, see the array C code:

      if (PyErr_WarnEx(PyExc_DeprecationWarning,
                         "The 'u' type code is deprecated and "
                         "will be removed in Python 3.16",
                         1)) {
            return NULL;
        }
    }

    25. StanFromIreland commented on Apr 28, 2025

      @StanFromIreland
      StanFromIreland
      on Apr 28, 2025
      Member

      I see now why I was mislead, the note is under pending-removal-in-future-versions too. I sent a pr.

    26. vstinner
      added a commit that references this issue on Apr 29, 2025

      gh-57281: Fix duplicate deprecation note: array 'u' format (#133113)

      59ff421
    27. miss-islington
      added a commit that references this issue on Apr 29, 2025

      pythongh-57281: Fix duplicate deprecation note: array 'u' format (pyt…

      151e54e
    28. vstinner
      added a commit that references this issue on Apr 29, 2025

      [3.13] gh-57281: Fix duplicate deprecation note: array 'u' format (GH…

      dcbda41
    29. serhiy-storchaka
      added this to Struct, memoryview and array issues 🏗️on Jul 13, 2026
    30. github-project-automation
      moved this to Todo in Struct, memoryview and array issues 🏗️on Jul 13, 2026
    31. github-project-automation
      moved this from Todo to Done in Struct, memoryview and array issues 🏗️on Jul 13, 2026
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    release-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions