Repository navigation
PyType_GetModuleByDef family for binary functions performance #117578
Description
Feature or enhancement
Proposal:
When implementing an extension module with a module state enabled,
binary tp slot functions (e.g. nb_add) need two PyType_GetModuleByDef() to be used like the following to compare the given types with a heap-type object in the module state, which can be slow:
static PyObject *
foo_add(PyObject *left, PyObject *right)
{
...
PyObject *module = PyType_GetModuleByDef(Py_TYPE(left), &module_def);
if (module == NULL) {
PyErr_Clear();
module = PyType_GetModuleByDef(Py_TYPE(right), &module_def);
}
...
}- 3.13.0 alpha5
_decimal(module state ver.)
from timeit import timeit
f = lambda s: timeit(s, 'import _decimal; d = _decimal.Decimal(1)')
a = f('d + 1')
b = f('1 + d')
print('d + 1:', a)
print('1 + d:', b, b / a)d + 1: 0.11202071857405826
1 + d: 0.49533294743326095 4.421797625818569
The difference mainly comes from a TypeError emission from PyType_GetModuleByDef(), so it would be nice if we could have a new function which receives two types and emits an error after checking them.
cc @encukou
Activity
encukou commentedon Apr 8, 2024
Make it private at first, then perhaps expose it in another issue.
gvanrossum commentedon Apr 8, 2024
@neonene Are you mainly interested in fixing the specific case for
_decimal? Other stdlib modules? 3rd party modules? Limited API? I agree with @encukou that if you want this mainly for_decimal, a private function would be best, and we should start there.neonene commentedon Apr 8, 2024
It seems that extension modules such as
_decimaland_ctypescannot use any privateexternfunctions, so my proposal (a private function) might be related to built-in modules. I'm thinking I should take back if not so beneficial.gvanrossum commentedon Apr 8, 2024
Isn't this a recent side effect caused by a desire to use only the Limited API? @vstinner should know more. I think the slowdown you've observed is rather significant, so maybe we need to make an exception that restriction (if that's what's causing your observation).
In modules not restricted to the Limited API, private APIs can be used as long as they use the
PyAPI_FUNC()macro.neonene commentedon Apr 9, 2024
Tested
_decimalwith #117661, where the private function was temporarily exposed:vstinner commentedon Apr 9, 2024
Yep, I agree. Also, there is maybe no need to expose it in public right now :-)
Python has multiple stdlib extensions built as shared extensions: you should export internal functions using
PyAPI_FUNC()for them.Example in
Include/internal/pycore_object.h:_decimaland_ctypesare not built with the limited C API. If tomorrow there is a great advantage to use the internal C API in a stdlib extension built with the limited C API, just remove the few lines aroundPy_LIMITED_APIand that's it.neonene commentedon Apr 9, 2024
Thanks. I've been very confused around the
PyAPI_FUNC()macro.gh-117578: Introduce _PyType_GetModuleByDef2 private function (GH-117661
picnixz commentedon Aug 17, 2024
Re-opening until #123100 is merged (just for tracking purposes).
neonene commentedon Sep 26, 2024
Superseded by
PyType_GetBaseByToken(): d7248cd