Using the public PyUnicodeWriter C API made the json module slower #133968

Activity

1. vstinner

   added

   performancePerformance or resource usagePerformance or resource usage

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on May 13, 2025
2. vstinner commented on May 13, 2025

   

   vstinner

   on May 13, 2025

   MemberAuthor

   More actions

   See also #133186.
3. vstinner

   added 2 commits that reference this issue on May 13, 2025

   pythongh-133968: Add fast path to PyUnicodeWriter_WriteStr()

   ...

   8613170

   gh-133968: Add fast path to PyUnicodeWriter_WriteStr() (#133969)

   ...

   fe9f6e8
4. miss-islington

   added a commit that references this issue on May 13, 2025

   pythongh-133968: Add fast path to PyUnicodeWriter_WriteStr() (pythonG…

   ...

   616ee64
5. vstinner

   added 3 commits that reference this issue on May 13, 2025

   [3.14] gh-133968: Add fast path to PyUnicodeWriter_WriteStr() (GH-133969

   ...

   5632e93

   pythongh-133968: Add PyUnicodeWriter_WriteASCII() function

   ...

   fa9d9df

   pythongh-133968: Add PyUnicodeWriter_WriteASCII() function

   ...

   2aa2e87
6. encukou commented on May 15, 2025

   

   encukou

   on May 15, 2025

   Member

   More actions

   Could you try overoptimizing PyUnicodeWriter_WriteUTF8 and benchmarking that, to see how fast the existing function can “theoretically” be?
   I'm thinking something like:

   • in the JSON module, force the constants ("null" etc) to be size_t-aligned
   • add private (ABI-only) function _PyUnicodeWriter_WriteUTF8_SmallAligned, which requires size_t-aligned input and size <= sizeof(size_t)
   • make PyUnicodeWriter_WriteUTF8 a macro that calls _PyUnicodeWriter_WriteUTF8_SmallAligned if the need are met, and PyUnicodeWriter_WriteUTF8 otherwise
7. picnixz

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on May 15, 2025
8. picnixz

   added this to JSON issueson May 15, 2025
9. vstinner commented on May 15, 2025

   

   vstinner

   on May 15, 2025

   MemberAuthor

   More actions

   in the JSON module, force the constants ("null" etc) to be size_t-aligned

   I don't know how to guarantee that.

   add private (ABI-only) function _PyUnicodeWriter_WriteUTF8_SmallAligned, which requires size_t-aligned input and size <= sizeof(size_t)

   Why do you want to limit the size to sizeof(size_t)? Which kind of optimization are you thinking of? find_first_nonascii() already takes care of the different cases with different optimizations.

   ascii_decode() is already highly optimized.
10. encukou commented on May 17, 2025

    

    encukou

    on May 17, 2025

    Member

    More actions

    ascii_decode() is already highly optimized.

    Indeed it is! After playing around a bit, I could only optimize short-ASCII-strings at the expense of other strings, or a lot of code duplication.
    Sorry for the noise!

    I don't know how to guarantee that.

    For the record: alignas; in old C it could be union { char my_string[N]; size_t _align_me_please; }

    👍React with 👍1Reacted by Victor Stinner
11. 6 remaining items

    Load more
12. vstinner

    added a commit that references this issue on Jun 9, 2025

    gh-133968: Update PyUnicodeWriter_WriteASCII() documentation (#135297)

    ...

    aaad2e8
13. vstinner commented on Jun 9, 2025

    

    vstinner

    on Jun 9, 2025

    MemberAuthor

    More actions

    I close the issue. Multiple changes optimized the json module which should be "less slow" than 3.13 in 3.14 and main (3.15) branches now.

    If someone has more optimization ideas, I suggest opening a new issue.
14. vstinner

    closed this as completedon Jun 9, 2025
15. github-project-automation

    moved this to Done in JSON issueson Jun 9, 2025
16. lkollar

    added a commit that references this issue on Jun 19, 2025

    pythongh-133968: Update PyUnicodeWriter_WriteASCII() documentation (p…

    ...

    8956434
17. Pranjal095

    added 4 commits that reference this issue on Jul 12, 2025

    pythongh-133968: Add fast path to PyUnicodeWriter_WriteStr() (python#…

    ...

    1c428ef

    pythongh-133968: Add PyUnicodeWriter_WriteASCII() function (python#13…

    ...

    85425a2

    pythongh-133968: Create the Unicode writer on demand in json (python#…

    ...

    71aa7bf

    pythongh-133968: Update PyUnicodeWriter_WriteASCII() documentation (p…

    ...

    6fc4ecc
18. taegyunkim

    added 4 commits that reference this issue on Aug 4, 2025

    pythongh-133968: Add fast path to PyUnicodeWriter_WriteStr() (python#…

    ...

    a9164b0

    pythongh-133968: Add PyUnicodeWriter_WriteASCII() function (python#13…

    ...

    3a33b93

    pythongh-133968: Create the Unicode writer on demand in json (python#…

    ...

    bd6baaf

    pythongh-133968: Update PyUnicodeWriter_WriteASCII() documentation (p…

    ...

    544a389
19. Agent-Hellboy

    added a commit that references this issue on Aug 19, 2025

    pythongh-133968: Update PyUnicodeWriter_WriteASCII() documentation (p…

    ...

    80b1e5e
20. gostak-dd

    added a commit that references this issue on Jun 2, 2026

    pythongh-133968: Add PyUnicodeWriter_WriteASCII() function (python#13…

    ...

    c58b376