Document PyUnicode_* API

@malemburg

BPO	1944
Nosy	@malemburg, @birkenfeld, @vstinner, @avassalotti, @berkerpeksag, @vadmium, @serhiy-storchaka, @shihai1991, @furkanonder
PRs	bpo-1944: wrap functions with macro #20011
Files	unicode.patch: docs for PyUnicodes C-API functions: FromFormat, FromFormatV, FromString, FromStringAndSize, Partition, RPartition and RSplit

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-01-27.06:26:42.019>
labels = ['type-feature', 'docs']
title = 'Document PyUnicode_* API'
updated_at = <Date 2020-06-21.12:39:22.129>
user = 'https://github.com/avassalotti'

bugs.python.org fields:

activity = <Date 2020-06-21.12:39:22.129>
actor = 'shihai1991'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2008-01-27.06:26:42.019>
creator = 'alexandre.vassalotti'
dependencies = []
files = ['13717']
hgrepos = []
issue_num = 1944
keywords = ['patch']
message_count = 9.0
messages = ['61734', '86116', '86121', '89100', '185552', '185725', '264571', '264575', '368582']
nosy_count = 11.0
nosy_names = ['lemburg', 'georg.brandl', 'vstinner', 'alexandre.vassalotti', 'donlorenzo', 'docs@python', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'shihai1991', 'furkanonder']
pr_nums = ['20011']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1944'
versions = ['Python 3.5', 'Python 3.6']

[edit @encukou]: Converting the lists below to checkboxes:

Linked PRs

I was wandering whether the pointer returned by PyUnicode_AsString needs
to be freed after usage (It turned it doesn't since the result is
cached). However, I found out that there isn't any documentation on
docs.python.org about the PyUnicode_AsString and
PyUnicode_AsStringAndSize functions. Although, both are documented in
the public unicodeobject.h header.

I notice that the documentation for several other unicode functions is
missing. Quickly, I see:

PyUnicode_Resize
PyUnicode_InternImmortal
PyUnicode_GetDefaultEncoding
PyUnicode_SetDefaultEncoding
PyUnicode_BuildEncodingMap
PyUnicode_FromFormatV
PyUnicode_*UTF7*
PyUnicode_AsEncodedObject
PyUnicode_FromOrdinal
PyUnicode_DecodeFSDefault
PyUnicode_DecodeFSDefaultAndSize

It would probably be a good idea to polish up the documentation for
PyUnicode as much as possible for Python 3000, since extension
developers will certainly need to refer to it a lot during the
transition from 2.x.

In addition to the above mentioned functions I found these to be
undocumented:

PyUnicode_DecodeUTF7
PyUnicode_DecodeUTF7Stateful
PyUnicode_EncodeDecimal
PyUnicode_EncodeUTF7
PyUnicode_FromFormat
PyUnicode_FromString
PyUnicode_FromStringAndSize
PyUnicode_GetMax
PyUnicode_Partition
PyUnicode_RPartition
PyUnicode_RSplit

From the original list the following functions seem to have been removed:

PyUnicode_InternImmortal
PyUnicode_DecodeFSDefault
PyUnicode_DecodeFSDefaultAndSize

I try to put together a patch for some of these during the weekend.

Ok, here is my shot at a patch for at least some of the undocumented
functions. Namely the following functions are being documented in the patch:

PyUnicode_FromFormat
PyUnicode_FromFormatV
PyUnicode_FromString
PyUnicode_FromStringAndSize
PyUnicode_Partition
PyUnicode_RPartition
PyUnicode_RSplit

Please thoroughly review this patch since I didn't really digg into the
source to find out what the functions do but rather just copied old
PyString documentation or derived it from the docs for the Python API.

The patch looks alright. I don't like the documentation for
PyUnicode_FromFormatV, however. Here's my attempt to document it:

.. cfunction:: PyObject* PyUnicode_FromFormatV(const char *format,
va_list vargs)

Equivalent to the function :cfunc:`PyUnicode_FromFormat`, except that
it takes a va_list instead of variable number of arguments.

Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

On 30.03.2013 13:09, Mark Lawrence wrote:

Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

PEP-393 only changed the way Unicode is internally stored.
The Unicode API is mostly unaffected by this change.

Remaining undocumented functions:

From this issue:

PyUnicode_RSplit
PyUnicode_Partition
PyUnicode_RPartition

From bpo-10435:

PyUnicode_IsIdentifier
PyUnicode_Append
PyUnicode_AppendAndDel
PyUnicode_GetDefaultEncoding
PyUnicode_FromOrdinal
PyUnicode_Resize
PyUnicode_GetMax
PyUnicode_InternImmortal
PyUnicode_CHECK_INTERNED

From bpo-18688:

Py_UNICODE_REPLACEMENT_CHARACTER
PyUnicodeIter_Type
PyUnicode_AsDecodedObject
PyUnicode_AsDecodedUnicode
PyUnicode_AsEncodedObject
PyUnicode_AsEncodedUnicode
PyUnicode_BuildEncodingMap

I agree, though they have been deprecated for many years but there is no planned removal date that I can find. Should we just remove them now?

IMO now it's too late for the 3.14 release cycle. It would be better to schedule such removal at the start of a new dev cycle, such as Python 3.15. Someone also has to check if these functions are used in the wild (ex: run a code search).

@serhiy-storchaka

cc @serhiy-storchaka who originally deprecated them in 0093907

A new issue is probably best for this, scheduling for 3.15 seems good for now.

gh-46236: Document `PyUnicodeIter_Type` (GH-132925)

pythongh-46236: Document `PyUnicodeIter_Type` (pythonGH-132925)

[3.13] gh-46236: Document `PyUnicodeIter_Type` (GH-132925) (GH-133149)

The list needs to be updated, just PyUnicode_BuildEncodingMap is left.

gh-46236: Document PyUnicode_BuildEncodingMap (#133270)

pythongh-46236: Document PyUnicode_BuildEncodingMap (pythonGH-133270)

The list is complete. I close the issue. Thanks to everybody who was involved in this old issue (2008!).

[3.14] gh-46236: Document PyUnicode_BuildEncodingMap (GH-133270) (#13…

[3.13] gh-46236: Document PyUnicode_BuildEncodingMap (#133770)

pythongh-46236: Document PyUnicode_BuildEncodingMap (python#133270)

Navigation Menu

Repository navigation

Document PyUnicode_* API #46236

Linked PRs

38 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Document PyUnicode_* API #46236

Description

Linked PRs

Activity

avassalotti commented on Jan 27, 2008

donlorenzo commented on Apr 18, 2009

donlorenzo commented on Apr 18, 2009

avassalotti commented on Jun 8, 2009

BreamoreBoy commented on Mar 30, 2013

malemburg commented on Apr 1, 2013

berkerpeksag commented on Apr 30, 2016

38 remaining items

vstinner commented on Apr 22, 2025

StanFromIreland commented on Apr 22, 2025

StanFromIreland commented on May 1, 2025

vstinner commented on May 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions