C API: Investigate how the PyTypeObject members can be removed from the public C API

I propose to investigate an incompatible C API change: make the PyTypeObject and PyHeapTypeObject structures opaque, remove their members from the public C API (move them to the internal C API). We have to investigate how it's outside in 3rd party C extensions (ex: Cython, pybind11, etc.), and design a smooth migration plan.

The PyTypeObject structure is exposed as part of the public Python C API. For example, Py_TYPE(obj)->tp_name directly gets a type name (as a UTF-8 encoded byte string, char*).

The PyTypeObject members are NOT part of the limited C API (PEP 384).

In Python 3.9 (2020), I reworked the C API to avoid accessing directly PyTypeObject members at the ABI level: issue #84351. For example, Python 3.8 implements PyObject_IS_GC() as a macro:

/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
    (PyType_IS_GC(Py_TYPE(o)) \
     && (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))

whereas Python 3.9 only provides an opaque function call:

/* Test if an object implements the garbage collector protocol */
PyAPI_FUNC(int) PyObject_IS_GC(PyObject *obj);

At the ABI level, the direct access to the PyTypeObject.tp_is_gc member became an opaque function call.

Changing PyTypeObject API and ABI caused a lot of troubles in the past. Example:

issue C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed #81431: Cython broken by PyTypeObject.tp_print removal -- related to PyTypeObject.tp_vectorcall_offset addition
issue Remove cross-version binary compatibility #76569: ABI issue with the addition of PyTypeObject.tp_finalize (PEP 442). Is Py_TPFLAGS_HAVE_FINALIZE needed for ABI compatibility? See also the python-dev thread.
issue Remove Py_TPFLAGS_HAVE_VERSION_TAG flag? #86913: remove Py_TPFLAGS_HAVE_VERSION_TAG flag (ABI compatibility)
issue Documentation inconsistency with the stable ABI #91271: Documentation inconsistency with the stable ABI (PyTypeObject)
The PEP 620 uses PyTypeObject incompatible changes in its rationale.

For many years, there is a work-in-progress to convert all Python built-in types and types of stdlib extensions from static types to heap types. See for example issue #84258 and PEP 630.

The API and ABI for heap type was also enhanced over the years. Examples:

Python 3.12 adds PyType_FromMetaclass()
Add the buffer protocol to the limited C API (to PyType_FromSpec()): issue Limited API support for Py_buffer #89622
issue [C API] PyType_GetSlot cannot get tp_name #86201: get tp_name with PyType_GetSlot()

In the past, other structure members were removed:

PyInterpreterState: Python 3.8
PyGC_Head: Python 3.9
PyFrameObject: Python 3.11

The work was also prepared for:

PyObject: issue [C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754, Python 3.10 and Python 3.11, add Py_SET_REFCNT() and Py_SET_TYPE()
PyTheadState: issue [C API] Prepare the C API to make PyThreadState opaque: add getter functions #84128, Python 3.11

For Py_TYPE(obj)->tp_name pattern, I proposed adding %T format to PyUnicode_FromFormat(): issue GH-78776. See related python-dev discussion. The change was reverted.

One thing I don't know how to do with the spec-based API is adding additional C fields to a type without exposing them to Python. That is something I have a use-case for in PyObjC and can currently do by embedding a PyTypeObject (or rather PyHeapTypeObject in a larger struct). Storing the data out of line, for example in some kind of hash table, likely results in additional overhead in what is in the end fairly hot code.

I want to try moving PyObjC to the spec-based API during the 3.13 release cycle if I have enough free time, but that's a fairly big "if". The longer term goal with that is to support sub interpreters.

@ronaldoussoren

@ronaldoussoren:

One thing I don't know how to do with the spec-based API is adding additional C fields to a type without exposing them to Python

Isn't it the purpose of PEP 697 – Limited C API for Extending Opaque Types? It's said to be fully implemented in Python 3.12.

cc @encukou

Changing PyTypeObject and PyHeapTypeObject API is complex because they have 112 members in total.

In general, most members can be get with PyType_GetSlot() and many can be get and set with PyObject_GetAttrString() and PyObject_SetAttrString(). I listed members with more specialized way to access them.

Attributes (read-only):

tp_base: get "__base__" attribute
tp_basicsize: _PyObject_SIZE()
tp_doc: get/set "__doc__" attribute
tp_flags: PyType_GetFlags(), PyType_HasFeature()
tp_itemsize: _PyObject_VAR_SIZE()
tp_vectorcall_offset: PyVectorcall_Function(), PyObject_Vectorcall()

Attributes:

tp_bases: get/set "__bases__" attribute, _PyType_GetBases()
tp_name: PyType_GetName(), PyType_GetQualName(), set "__name__" attribute

Internal functions for the GC:

tp_clear: used indirectly by Py_Finalize()
tp_traverse: call gc.get_objects() function

Internal attributes and methods:

tp_dealloc: _Py_Dealloc(), used by Py_DECREF()
tp_members
tp_mro (read-only): PyObject_GetAttrString(type, "__mro__")
tp_subclasses: call __subclasses__() method, _PyType_GetSubclasses()
tp_version_tag: method cache, PyUnstable_Type_AssignVersionTag(), _PyType_Lookup()
tp_watched: PyType_Watch(), PyType_Unwatch()
tp_weaklist: used by tp_weaklistoffset
tp_weaklistoffset: get "__weakrefoffset__" attribute
tp_vectorcall: used by tp_vectorcall_offset

Async methods:

tp_as_async.am_await: call __await__() method, _PyCoro_GetAwaitableIter()
tp_as_async.am_aiter: call __aiter__() method
tp_as_async.am_anext: call __anext__() method
tp_as_async.am_send: PyIter_Send()

Buffer methods:

tp_as_buffer.bf_getbuffer: PyObject_GetBuffer()``
tp_as_buffer.bf_releasebuffer: PyBuffer_Release()

Mapping methods:

tp_as_mapping.mp_length: PyMapping_Size()
tp_as_mapping.mp_subscript: PyObject_GetItem()
tp_as_mapping.mp_ass_subscript: PyObject_SetItem()

Number methods:

tp_as_number.nb_add: PyNumber_Add()
tp_as_number.nb_subtract: PyNumber_Subtract()
tp_as_number.nb_multiply: PyNumber_Multiply()
tp_as_number.nb_remainder: PyNumber_Remainder()
tp_as_number.nb_divmod: PyNumber_Divmod()
tp_as_number.nb_power: PyNumber_Power()
tp_as_number.nb_negative: PyNumber_Negative()
tp_as_number.nb_positive: PyNumber_Positive()
tp_as_number.nb_absolute: PyNumber_Absolute()
tp_as_number.nb_bool: PyObject_IsTrue()
tp_as_number.nb_invert: PyNumber_Invert()
tp_as_number.nb_lshift: PyNumber_Lshift()
tp_as_number.nb_rshift: PyNumber_Rshift()
tp_as_number.nb_and: PyNumber_And()
tp_as_number.nb_xor: PyNumber_Xor()
tp_as_number.nb_or: PyNumber_Or()
tp_as_number.nb_int: PyNumber_Long()
tp_as_number.nb_float: PyNumber_Float()
tp_as_number.nb_inplace_add: PyNumber_InPlaceAdd()
tp_as_number.nb_inplace_subtract: PyNumber_InPlaceSubtract()
tp_as_number.nb_inplace_multiply: PyNumber_InPlaceMultiply()
tp_as_number.nb_inplace_remainder: PyNumber_InPlaceRemainder()
tp_as_number.nb_inplace_power: PyNumber_InPlacePower()
tp_as_number.nb_inplace_lshift: PyNumber_InPlaceLshift()
tp_as_number.nb_inplace_rshift: PyNumber_InPlaceRshift()
tp_as_number.nb_inplace_and: PyNumber_InPlaceAnd()
tp_as_number.nb_inplace_xor: PyNumber_InPlaceXor()
tp_as_number.nb_inplace_or: PyNumber_InPlaceOr()
tp_as_number.nb_floor_divide: PyNumber_FloorDivide()
tp_as_number.nb_true_divide: PyNumber_TrueDivide()
tp_as_number.nb_inplace_floor_divide: PyNumber_InPlaceFloorDivide()
tp_as_number.nb_inplace_true_divide: PyNumber_InPlaceTrueDivide()
tp_as_number.nb_index: PyNumber_Index()
tp_as_number.nb_matrix_multiply: PyNumber_MatrixMultiply()
tp_as_number.nb_inplace_matrix_multiply: PyNumber_InPlaceMatrixMultiply()

Sequence methods:

tp_as_sequence.sq_length: PySequence_Size()
tp_as_sequence.sq_concat: PySequence_Concat()
tp_as_sequence.sq_repeat: PySequence_Repeat()
tp_as_sequence.sq_item: PySequence_GetItem()
tp_as_sequence.sq_ass_item: PySequence_SetItem()
tp_as_sequence.sq_contains: PySequence_Contains()
tp_as_sequence.sq_inplace_concat: PySequence_InPlaceConcat()
tp_as_sequence.sq_inplace_repeat: PySequence_InPlaceRepeat()

Attribute lookup, get/attr instance attributes:

tp_descr_get
tp_descr_set
tp_dict: get "__dict__" attribute, PyObject_GenericGetAttr(), PyObject_GenericSetDict(), _PyObject_GetDictPtr(). See also Python 3.10 Py_TPFLAGS_MANAGED_DICT.
tp_dictoffset: get "__dictoffset__" attribute
tp_getattr: PyObject_GetAttr()
tp_getattro: PyObject_GetAttr()
tp_setattr: PyObject_SetAttr()
tp_setattro: PyObject_SetAttr()
tp_getset
tp_methods

Methods and how to call them:

tp_alloc: PyType_GenericNew()
tp_call: PyObject_Call()
tp_del: call __del__() method (?), subtype_dealloc()
tp_finalize: call PyObject_CallFinalizer()
tp_free: get PyType_GetSlot(Py_tp_free)
tp_hash: PyObject_Hash()
tp_init: get PyType_GetSlot(Py_tp_init), call a type to create an instance, type_call()
tp_is_gc: PyObject_IS_GC()
tp_iter: PyObject_GetIter()
tp_iternext: PyIter_Next()
tp_new: get PyType_GetSlot(Py_tp_new), call a type to create an instance, type_call()
tp_repr: PyObject_Repr()
tp_richcompare: PyObject_RichCompare()
tp_str: PyObject_Str()

PyHeapTypeObject internal members:

_ht_tpname
_spec_cache
ht_cached_keys
ht_slots

PyHeapTypeObject members:

ht_type: ?
as_async: copy of PyTypeObject.tp_as_async
as_number: copy of PyTypeObject.tp_as_number
as_mapping: copy of PyTypeObject.tp_as_mapping
as_sequence: copy of PyTypeObject.tp_as_sequence
as_buffer: copy of PyTypeObject.tp_as_buffer
ht_name: PyType_GetName(), set "__name__" attribute
ht_qualname: PyType_GetQualName()
ht_module: PyType_GetModule(), PyType_GetModuleByDef()

Unused members:

tp_cache
tp_as_number.nb_reserved
tp_as_sequence.was_sq_slice
tp_as_sequence.was_sq_ass_slice

Creating a static type PyTypeObject my_type = {...}; requires accessing directly all PyTypeObject members. This API, like PyType_Ready(), should be deprecated in favor of heap types.

Creating a heap type can be done with the PyType_Spec structure and PyType_FromSpec() function. This API doesn't access PyTypeObject or PyHeapTypeObject members on purpose. It helps the ABI backward/forward compatibility: the stable ABI used it for example.

The Python stdlib is a bad example, it still has many static types in Python 3.13: issue #84258. Moreover, some static types may remain implemented as static types since they are exposed in the C API, such as &PyUnicode_Type (issue #84781).

What changed recently is that many data are now "static": generated at build type and constant. "Static" types means more and more something really "static", like it's no longer needed to release their memory at Python exit (issue #103276).

tp_dict: get "dict" attribute, PyObject_GenericGetAttr(), PyObject_GenericSetDict(), _PyObject_GetDictPtr(). See also Python 3.10 Py_TPFLAGS_MANAGED_DICT.
tp_dictoffset: get "dictoffset" attribute

PyType_GetDict() is excluded from the limited C API, and _PyObject_GetDictPtr() is part of the internal API.

Using tp_name to format an error message is a very common code pattern. I created issue gh-111696 for that.

Let's revisit that once PEP 737 is accepted and implementated, it's one of the most common consumer of the PyTypeObject.

Navigation Menu

Repository navigation

C API: Investigate how the PyTypeObject members can be removed from the public C API #105970

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

C API: Investigate how the PyTypeObject members can be removed from the public C API #105970

Description

Activity

vstinner commented on Jun 21, 2023

ronaldoussoren commented on Jun 22, 2023

vstinner commented on Jun 22, 2023

vstinner commented on Jun 23, 2023

vstinner commented on Jun 23, 2023

vstinner commented on Oct 17, 2023

vstinner commented on Nov 8, 2023

vstinner commented on Dec 20, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions