[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type

@vstinner

BPO	39573
Nosy	@vstinner
PRs	bpo-39573: Use Py_REFCNT() macro #18388 bpo-39573: Add Py_SET_REFCNT() function #18389 bpo-39573: Use Py_TYPE() in abstract.c #18390 bpo-39573: Use Py_TYPE() macro in Python and Include directories #18391 bpo-39573: Use Py_TYPE() macro in Objects directory #18392 bpo-39573: Use Py_TYPE() macro in Modules directory #18393 bpo-39573: Add Py_SET_TYPE() function #18394 bpo-39573: Use Py_TYPE() macro in object.c #18398 bpo-39573: Add Py_SET_SIZE() function #18400 bpo-39573: Use Py_SET_SIZE() function #18402 bpo-39573: Use Py_TYPE() macro in ctypes.h #18411 bpo-39573: Use Py_SIZE() in s_set() of cfield.c #18419 bpo-39573: Add Py_IS_TYPE macro #18488 bpo-39573: Fix bad copy-paste in Py_SET_SIZE. #18496 bpo-39573: Update clinic to use Py_IS_TYPE macro #18507 bpo-39573: Update Include/* to use Py_IS_TYPE #18508 bpo-39573: Clean up modules and headers to use Py_IS_TYPE #18521 bpo-39573: Finish converting to new Py_IS_TYPE macro. #18601 bpo-39573: Use Py_IS_TYPE for type checking #18789 bpo-39573: Use Py_IS_TYPE for negative type checks #18798 bpo-39573: Make Py_IS_TYPE take const args. Add _PyObject_CAST_CONST. #18799 bpo-39873: PyObject_Init() uses PyObject_INIT() #18804 bpo-39573: Use Py_IS_TYPE to check for types #18809 bpo-39573: Use Py_IS_TYPE to check for types #19882 bpo-30459: Cast the result of PyList_SET_ITEM() to void #19975 bpo-39573: Convert Py_TYPE() to a static inline function #20290 bpo-39573: Fix buildbot failure for tupleobject.c #20391 bpo-39573: Convert Py_REFCNT and Py_SIZE to functions #20429 bpo-39573: Porting to Python 3.10: Py_SET_SIZE() macro #20610 [WIP, DO NOT MERGE] bpo-41188: Prepare CPython for opague PyObject structure. #21262 bpo-39573: Use the Py_TYPE() macro #21433 bpo-39573: Convert Py_TYPE() and Py_SIZE() back to macros #23366 bpo-39573: Remove What's new entry for Py_SIZE() #23375 bpo-39573: Py_TYPE becomes a static inline function #26493 bpo-11105: use a lower recursion limit for infinite recursion tests #26550 bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" #26596 bpo-39573: Py_TYPE becomes a static inline function #28128

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-08.16:32:15.871>
created_at = <Date 2020-02-06.23:07:12.131>
labels = ['expert-C-API', '3.11']
title = '[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type'
updated_at = <Date 2022-01-20.00:24:00.180>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-01-20.00:24:00.180>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-08.16:32:15.871>
closer = 'vstinner'
components = ['C API']
creation = <Date 2020-02-06.23:07:12.131>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39573
keywords = ['patch']
message_count = 96.0
messages = ['361513', '361514', '361515', '361516', '361517', '361518', '361519', '361522', '361523', '361526', '361527', '361529', '361531', '361540', '361549', '361555', '361557', '361590', '361593', '361607', '361611', '361626', '361631', '361639', '361904', '361960', '361961', '361963', '361964', '361965', '361971', '361977', '361987', '361988', '362033', '362034', '362133', '362134', '362166', '362212', '362216', '362445', '363345', '363494', '363564', '365690', '366473', '366493', '368047', '369896', '369898', '370074', '370303', '370638', '370663', '370665', '370666', '370671', '370729', '370902', '370932', '372308', '373460', '379675', '379679', '379680', '379757', '379759', '381337', '381345', '381365', '381374', '381403', '381404', '382260', '382534', '382539', '382780', '382781', '382783', '394954', '394971', '395018', '395205', '395206', '395287', '395323', '395536', '401365', '401370', '401378', '401395', '401396', '401399', '403252', '410995']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['18388', '18389', '18390', '18391', '18392', '18393', '18394', '18398', '18400', '18402', '18411', '18419', '18488', '18496', '18507', '18508', '18521', '18601', '18789', '18798', '18799', '18804', '18809', '19882', '19975', '20290', '20391', '20429', '20610', '21262', '21433', '23366', '23375', '26493', '26550', '26596', '28128']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39573'
versions = ['Python 3.11']

Linked PRs

gh-83754: Use the Py_TYPE() macro #120599

Today, CPython is leaking too many implementation through its public C API. We cannot easily change the "default" C API, but we can enhance the "limited" C API (when Py_LIMITED_API macro is defined). Example of leaking implementation details: memory allocator, garbage collector, structure layouts, etc.

Making PyObject an opaque structure would allow in the long term of modify structures to implement more efficient types (ex: list specialized for small integers), and it can prepare CPython to experiment tagged pointers.

Longer rationale:

I propose to incremental evolve the existing limited C API towards opaque PyObject, by trying to reduce the risk of breakage.

We may test changes on PyQt which uses the limited C API.

Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

First issues that I met when I tried that:

C code generated by Argument Clinic is incompatible the limited C API: METH_FASTCALL, _PyArg_CheckPositional(), static _PyArg_Parser, etc. are excluded from the limited C API.
PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like:
Py_TYPE(self)->tp_free((PyObject*)self);
_Py_IDENTIFIER() is not part of the limited C API

New changeset a93c51e by Victor Stinner in branch 'master':
bpo-39573: Use Py_REFCNT() macro (GH-18388)
a93c51e

it can prepare CPython to experiment tagged pointers

In September 2018, Neil Schemenauer did an experiment:

More recent discussion on the capi-sig list:

https://mail.python.org/archives/list/capi-sig@python.org/thread/JPUNPN3AILGXOA3C2TTSLMOFNSWJE3QX/

See also my notes:
https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doable

Wikipedia article: https://en.wikipedia.org/wiki/Tagged_pointer

In the limited C API, Py_REFCNT() should be converted to:

static inline Py_ssize_t _Py_REFCNT(const PyObject *ob)
{ return ob->ob_refcnt; }
#define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))

It would enforce the usage of newly added Py_SET_REFCNT() (PR 18389) and advertise that the object is not modified (const).

That would only be the first step towards a really opaque Py_REFCNT() function.

TODO: Add Py_IS_TYPE() macro:

#define Py_IS_TYPE(ob, tp) (Py_TYPE(ob) == (tp))

For example, replace:

    #define PyBool_Check(x) (Py_TYPE(x) == &PyBool_Type)

with:

    #define PyBool_Check(x) Py_IS_TYPE(x, &PyBool_Type)

IMHO it makes the code more readable.

nascheme@c156300

New changeset c86a112 by Victor Stinner in branch 'master':
bpo-39573: Add Py_SET_REFCNT() function (GH-18389)
c86a112

New changeset 0d76d2b by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() in abstract.c (GH-18390)
0d76d2b

Py_TYPE() is commonly used to render the type name in an error message. Example:

PyErr_Format(PyExc_TypeError,
             "cannot convert '%.200s' object to bytearray",
             Py_TYPE(arg)->tp_name);

This code has multiple issues:

It truncates type name to 200 characters: there is no Python exception, not even a marker to indicate that the string has been truncated
It's only the short name: the qualified name (tp_qualname) would be more helpful. The best would be to generate the fully qualified name: module + qualname.
Py_TYPE() returns a borrowed reference which is causing multiple issues: https://pythoncapi.readthedocs.io/bad_api.html#borrowed-references

In September 2018, I created bpo-34595: "PyUnicode_FromFormat(): add %T format for an object type name". But there was disagreement, so I rejected my change.

I started "bpo-34595: How to format a type name?" thread on python-dev:

https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

I didn't continue this work (until now), since it wasn't my priority.

New changeset a102ed7 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Python and Include directories (GH-18391)
a102ed7

Make PyObject an opaque structure is also a first step towards the more ambitious project "HPy" project which is fully opaque:
https://github.com/pyhandle/hpy

This API is written from scratch and currently implemented on top on the existing C API.

The following article is a nice introduction to the overall idea:
https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.html

From my point of view, the long term goal would be to get better performance on PyPy and having a single API for C extension which would be efficient on all Python implementations (not only CPython).

Currently, the C API is not only a performance issue to run C extensions on PyPy. It's also an issue in CPython. Because the C API leaks too many implementation details, we cannot experiment optimizations.

See also: https://pythoncapi.readthedocs.io/rationale.html

New changeset 58ac700 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Objects directory (GH-18392)
58ac700

New changeset daa9756 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Modules directory (GH-18393)
daa9756

I changed the issue title to restrict its scope: "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type".

Making PyObject and PyVarObject structures opaque is a broader topic which should be splited into sub-issues.

"Py_TYPE(obj)=type;" is now disallowed. I consider that the work of this issue is now completed and I close the issue.

Thanks everyone who help to fix these tedious issues!

You can continue to use this issue if you need my help to adapt your C extensions to Py_SET_TYPE()/Py_SET_SIZE().

See also the upgrade_pythoncapi.py script of the pythoncapi_compat project which helps to port your C extensions without losing support for old Python versions:
https://github.com/pythoncapi/pythoncapi_compat

See also the Py_TYPE() change announcement on the capi-sig list:
https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/

I wrote an article about these changes:
https://vstinner.github.io/c-api-abstract-pyobject.html

It elaborates the rationale for making these changes.

@victor

@victor, git bisect tells me the change f3fa63e caused test_exceptions.ExceptionTests.test_recursion_in_except_handler to stack overflow only on windows debug builds.

FYI this regression was handled last year in bpo-44348 "test_exceptions.ExceptionTests.test_recursion_in_except_handler stack overflow on Windows debug builds" and fixed at 2021-09-07 by using the trashcan mecanism in the BaseException deallocator function:

New changeset fb30509 by Victor Stinner in branch 'main':
bpo-44348: BaseException deallocator uses trashcan (GH-28190)
fb30509

Replace Py_TYPE(obj)=type with Py_SET_TYPE()

pythongh-83754: Use the Py_TYPE() macro

gh-83754: Use the Py_TYPE() macro (#120599)

pythongh-83754: Use the Py_TYPE() macro (python#120599)

Navigation Menu

Repository navigation

[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754

Linked PRs

81 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754

Description

Linked PRs

Activity

vstinner commented on Feb 6, 2020

vstinner commented on Feb 6, 2020

vstinner commented on Feb 6, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

vstinner commented on Feb 7, 2020

81 remaining items

vstinner commented on Sep 8, 2021

vstinner commented on Oct 5, 2021

vstinner commented on Jan 20, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions