Repository navigation
Split up _testcapimodule.c #93649
Description
Activity
ideally without polluting the namespace of top-level modules
Can hypothetical
Lib/_testcapimodule/*.cexport their objects astest.capi.*?As far as I know they can, as long as
test.capiitself is not an extension module. It would need some extra support in both make and Windows builds, though.- added a commit that references this issue
on Jul 4, 2022 I tried to group de 299 attributes of the _testcapi extension module. I'm not
sure about each attribute, I only looked quickly at the name, sometimes at the
implementation. But it should give a coarse idea of which "groups" make sense
or not.Python/getargs.c and Python/modsupport.c (parse and build arguments) (51):
- parse_tuple_and_keywords
- getargs_B
- getargs_C
- getargs_D
- getargs_H
- getargs_I
- getargs_K
- getargs_L
- getargs_S
- getargs_U
- getargs_Y
- getargs_Z
- getargs_Z_hash
- getargs_b
- getargs_c
- getargs_d
- getargs_es
- getargs_es_hash
- getargs_et
- getargs_et_hash
- getargs_f
- getargs_h
- getargs_i
- getargs_k
- getargs_keyword_only
- getargs_keywords
- getargs_l
- getargs_n
- getargs_p
- getargs_positional_only_and_keywords
- getargs_s
- getargs_s_hash
- getargs_s_hash_int
- getargs_s_star
- getargs_tuple
- getargs_u
- getargs_u_hash
- getargs_w_star
- getargs_y
- getargs_y_hash
- getargs_y_star
- getargs_z
- getargs_z_hash
- getargs_z_star
- test_L_code
- test_k_code
- test_s_code
- test_buildvalue_N
- test_buildvalue_issue38913
- test_empty_argparse
- argparsing
datetime, pytime.c (33):
- PyDateTime_DATE_GET
- PyDateTime_DELTA_GET
- PyDateTime_GET
- PyDateTime_TIME_GET
- PyTime_AsMicroseconds
- PyTime_AsMilliseconds
- PyTime_AsSecondsDouble
- PyTime_AsTimespec
- PyTime_AsTimespec_clamp
- PyTime_AsTimeval
- PyTime_AsTimeval_clamp
- PyTime_FromSeconds
- PyTime_FromSecondsObject
- datetime_check_date
- datetime_check_datetime
- datetime_check_delta
- datetime_check_time
- datetime_check_tzinfo
- get_date_fromdate
- get_date_fromtimestamp
- get_datetime_fromdateandtime
- get_datetime_fromdateandtimeandfold
- get_datetime_fromtimestamp
- get_delta_fromdsu
- get_time_fromtime
- get_time_fromtimeandfold
- get_timezone_utc_capi
- get_timezones_offset_zero
- make_timezones_capi
- pytime_object_to_time_t
- pytime_object_to_timespec
- pytime_object_to_timeval
- test_datetime_capi
Types, metaclasses, type inheritance (wide category, not sure if it makes sense) (30):
- _test_structmembersType
- awaitType
- instancemethod
- ipowType
- create_type_from_repeated_slots
- matmulType
- NullTpDocType
- Generic
- GenericAlias
- MyList
- ContainerNoGC
- HeapCTypeMetaclass
- HeapCTypeMetaclassCustomNew
- HeapCTypeSetattr
- HeapCTypeSubclass
- HeapCTypeSubclassWithFinalizer
- HeapCTypeWithBuffer
- HeapCTypeWithDict
- HeapCTypeWithNegativeDict
- HeapCTypeWithWeakref
- HeapDocCType
- HeapGcCType
- test_from_spec_invalid_metatype_inheritance
- pytype_fromspec_meta
- test_from_spec_metatype_inheritance
- test_get_statictype_slots
- test_get_type_name
- test_get_type_qualname
- test_lazy_hash_inheritance
- test_type_from_ephemeral_spec
Function calls and spawn threads to call functions (25):
- _test_thread_state
- call_in_temporary_c_thread
- get_kwargs
- get_args
- create_cfunction
- pyobject_vectorcall
- pyvectorcall_call
- pyobject_fastcall
- pyobject_fastcalldict
- meth_fastcall
- meth_fastcall_keywords
- meth_noargs
- meth_o
- meth_varargs
- meth_varargs_keywords
- MethClass
- MethInstance
- MethStatic
- MethodDescriptor2
- MethodDescriptorBase
- MethodDescriptorDerived
- MethodDescriptorNopGet
- return_null_without_error
- return_result_with_error
- stack_pointer
Unicode, codecs (12):
- codec_incrementaldecoder
- codec_incrementalencoder
- test_unicode_compare_with_ascii
- test_widechar
- test_string_from_format
- unicode_asucs4
- unicode_asutf8
- unicode_asutf8andsize
- unicode_aswidechar
- unicode_aswidecharstring
- unicode_copycharacters
- unicode_findchar
Memory allocators (17):
- WITH_PYMALLOC
- pyobject_malloc_without_gil
- pymem_api_misuse
- pymem_buffer_overflow
- pymem_getallocatorsname
- pymem_malloc_without_gil
- test_pymem_alloc0
- test_pymem_setallocators
- test_pymem_setrawallocators
- test_pyobject_new
- test_pyobject_setallocators
- set_nomemory
- check_pyobject_forbidden_bytes_is_freed
- check_pyobject_freed_is_freed
- check_pyobject_null_is_freed
- check_pyobject_uninitialized_is_freed
- remove_mem_hooks
Exceptions, "error handling", C "errno" variable and signals (13):
- RecursingInfinitelyError
- raise_memoryerror
- make_exception_with_doc
- write_unraisable_exc
- traceback_print
- set_exception
- raise_exception
- set_exc_info
- exception_print
- fatal_error
- error
- set_errno
- raise_SIGINT_then_send_None
C types limit and size (20):
- CHAR_MAX
- CHAR_MIN
- INT_MAX
- INT_MIN
- LLONG_MAX
- LLONG_MIN
- LONG_MAX
- LONG_MIN
- PY_SSIZE_T_MAX
- PY_SSIZE_T_MIN
- SHRT_MAX
- SHRT_MIN
- UCHAR_MAX
- UINT_MAX
- ULLONG_MAX
- ULONG_MAX
- USHRT_MAX
- SIZEOF_TIME_T
- test_config
- test_sizeof_c_types
Float, PyLongObject and PyNumber C API (16):
- test_long_and_overflow
- test_long_api
- test_long_as_size_t
- test_long_as_unsigned_long_long_mask
- test_long_long_and_overflow
- test_long_numbits
- test_longlong_api
- pynumber_tobase
- DBL_MAX
- DBL_MIN
- FLT_MAX
- FLT_MIN
- float_pack
- float_unpack
- test_string_to_double
- test_long_as_double
Docstring (10):
- docstring_empty
- docstring_no_signature
- docstring_with_invalid_signature
- docstring_with_invalid_signature2
- docstring_with_signature
- docstring_with_signature_and_extra_newlines
- docstring_with_signature_but_no_doc
- docstring_with_signature_with_defaults
- no_docstring
- test_with_docstring
Garbage collector (3):
- test_gc_control
- without_gc
- with_tp_del
Tracemalloc (3):
- tracemalloc_get_traceback
- tracemalloc_track
- tracemalloc_untrack
os module helpers (1):
- W_STOPCODE
Subinterpreters (1):
- run_in_subinterp
C API (57):
-
PyObject and PyVarObject C API (9):
- negative_refcount
- test_decref_doesnt_leak
- test_incref_decref_API
- test_incref_doesnt_leak
- test_refcount_funcs
- test_refcount_macros
- test_xdecref_doesnt_leak
- test_xincref_doesnt_leak
- test_set_type_size
-
PyObject_Bytes(), PyObject_Repr(), PyObject_Str() (3):
- pyobject_bytes_from_null
- pyobject_repr_from_null
- pyobject_str_from_null
-
PyTypeObject C API (2):
- negative_dictoffset
- type_get_version
-
C API macros (3):
- test_macros
- test_py_is_funcs
- test_py_is_macros
-
PyCodeObject C API (2):
- test_code_api
- code_newempty
-
PyFrameObject C API (5):
- frame_getbuiltins
- frame_getgenerator
- frame_getglobals
- frame_getlasti
- frame_getlocals
-
PyDictObject C API (3):
- dict_get_version
- dict_getitem_knownhash
- test_dict_iteration
-
PyThreadState C API (2):
- test_tstate_capi
- crash_no_current_thread
-
PyListObject C API (1):
- test_list_api
-
PySequence and PyMapping C API, PyObject_GetItem() (6):
- sequence_getitem
- get_mapping_items
- get_mapping_keys
- get_mapping_values
- getitem_with_error
- bad_get
-
C API "feature" macros like HAVE_FORK (1):
- get_feature_macros
-
HAMT C API (1):
- hamt
-
PyBuffer and memoryview C API (5):
- getbuffer_with_null_view
- make_memoryview_from_NULL_pointer
- test_from_contiguous
- PyBuffer_SizeFromFormat
- test_pep3118_obsolete_write_locks
-
PyCapsule C API (1):
- test_capsule
-
PyThread_tss C API (1):
- test_pythread_tss_key_state
-
PyStructSeq C API (2):
- test_structseq_newtype_doesnt_leak
- test_structseq_newtype_null_descr_doc
-
PyEval_SetTrace C API (1):
- settrace_to_record
-
PyMarshal C API (6):
- pymarshal_read_last_object_from_file
- pymarshal_read_long_from_file
- pymarshal_read_object_from_file
- pymarshal_read_short_from_file
- pymarshal_write_long_to_file
- pymarshal_write_object_to_file
-
Py_AddPendingCall C API (1):
- _pending_threadfunc
-
Misc C API (2):
- Py_Version
- Py_CompileString
Misc (7):
__doc____file____loader____name____package____spec__- the_number_three
Reacted by Petr Viktorin, Gregory P. Smith and Erlend E. AaslandIt might be interesting to split the large _testcapi module into multiple modules. But I like the idea of starting by splitting the long C file into multiple C files.
You do not need to include
_testcapi/vectorcall.cinMODULE__TESTCAPI_DEPS. It is already listed as dependency for_testcapiin the Setup file.Thank you for the fix! I'll 🔨 test-with-buildbots next time I touch the buildsystem.
(I wonder why this issue didn't get the broken buildbot notification, though...)Thank you for the fix! I'll hammer test-with-buildbots next time I touch the buildsystem. (I wonder why this issue didn't get the broken buildbot notification, though...)
We think that there was a problem in the buildbot hook, python/buildmaster-config#333
- added a commit that references this issue
on Jul 13, 2022 68 remaining items
- added 4 commits that reference this issue
on Feb 7, 2025 You do not need to include
_testcapi/vectorcall.cinMODULE__TESTCAPI_DEPS. It is already listed as dependency for_testcapiin the Setup file.It appears not to be the same as what you said. https://github.com/python/cpython/blob/main/Modules/Setup#L285
I encounter this:
/usr/bin/ld: Modules/_testcapimodule.o: in function `PyInit__testcapi': /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4168:(.text+0x5f59): undefined reference to `_PyTestCapi_Init_Vectorcall' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4171:(.text+0x5f69): undefined reference to `_PyTestCapi_Init_Heaptype' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4174:(.text+0x5f79): undefined reference to `_PyTestCapi_Init_Abstract' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4177:(.text+0x5f89): undefined reference to `_PyTestCapi_Init_Bytes' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4180:(.text+0x5f99): undefined reference to `_PyTestCapi_Init_Unicode' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4183:(.text+0x5fa9): undefined reference to `_PyTestCapi_Init_GetArgs' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4186:(.text+0x5fb9): undefined reference to `_PyTestCapi_Init_DateTime' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4189:(.text+0x5fc9): undefined reference to `_PyTestCapi_Init_Docstring' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4192:(.text+0x5fd9): undefined reference to `_PyTestCapi_Init_Mem' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4195:(.text+0x5fe9): undefined reference to `_PyTestCapi_Init_Watchers' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4198:(.text+0x5ff9): undefined reference to `_PyTestCapi_Init_Long' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4201:(.text+0x6009): undefined reference to `_PyTestCapi_Init_Float' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4204:(.text+0x6019): undefined reference to `_PyTestCapi_Init_Complex' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4207:(.text+0x6029): undefined reference to `_PyTestCapi_Init_Numbers' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4210:(.text+0x6039): undefined reference to `_PyTestCapi_Init_Dict' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4213:(.text+0x6049): undefined reference to `_PyTestCapi_Init_Set' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4216:(.text+0x6059): undefined reference to `_PyTestCapi_Init_List' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4219:(.text+0x6069): undefined reference to `_PyTestCapi_Init_Tuple' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4222:(.text+0x6079): undefined reference to `_PyTestCapi_Init_Structmember' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4225:(.text+0x6089): undefined reference to `_PyTestCapi_Init_Exceptions' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4228:(.text+0x6099): undefined reference to `_PyTestCapi_Init_Code' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4231:(.text+0x60a9): undefined reference to `_PyTestCapi_Init_Buffer' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4234:(.text+0x60b9): undefined reference to `_PyTestCapi_Init_File' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4237:(.text+0x60c9): undefined reference to `_PyTestCapi_Init_Codec' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4240:(.text+0x60d9): undefined reference to `_PyTestCapi_Init_Immortal' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4243:(.text+0x60e9): undefined reference to `_PyTestCapi_Init_GC' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4246:(.text+0x60f9): undefined reference to `_PyTestCapi_Init_PyAtomic' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4249:(.text+0x6109): undefined reference to `_PyTestCapi_Init_Run' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4252:(.text+0x6119): undefined reference to `_PyTestCapi_Init_Hash' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4255:(.text+0x6129): undefined reference to `_PyTestCapi_Init_Time' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4258:(.text+0x6139): undefined reference to `_PyTestCapi_Init_Monitoring' /usr/bin/ld: /home/runner/.xbuilder/run/2939/python3/src/./Modules/_testcapimodule.c:4261:(.text+0x6149): undefined reference to `_PyTestCapi_Init_Object' collect2: error: ld returned 1 exit status gmake: *** [Makefile:1681: Programs/_freeze_module] Error 1It appears not to be the same as what you said.
You respond to a comment, dated 2022. Maybe something was changed? See Modules/Setup.stdlib.in.
I encounter this:
We can't guess how you do this. Even CPython version, which you are using. E.g.:
$ wc -l Modules/_testcapimodule.c 3396 Modules/_testcapimodule.cAnyway, if you encounter a build failure - you should open a new issue. This issue is a wrong place for that.
I run
sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.localI built the python-3.13.2
It seem that
Modules/Setupis not keep sync withModules/Setup.stdlib.inI run sed -n -E 's/^#([a-z_*].*)$/\1/p' Modules/Setup > Modules/Setup.local
I built the python-3.13.2Could you, please, show us where on https://docs.python.org/3/ or https://devguide.python.org/ you have found such instruction?
@leleliu008, if you think you have found a bug in CPython, please open an issue.
But first you may want do start a thread in https://discuss.python.org/c/help/7 and describe in details what you are trying to do and why, and what's happened. Please follow to the documentation, if you are trying to build CPython.
@skirpichev I'm not sure if this is a bug. It might be due to a misunderstanding of the
Modules/Setup.*mechanism on my part. I couldn't find any detailed documents about the relationship betweenModules/Setup.*, I only found https://devguide.python.org/developer-workflow/extension-modules/#configuring-the-cpython-project , it just motionsModules/Setup.bootstrap.inandModules/Setup.stdlib.in, it doesn't motion others. I guessModules/Setup.localwould override others. I resolved my problem via runninggsed -i "/^_testcapi/d" Modules/Setup.local
Modules/_testcapimodule.cis a nearly-8000-line behemoth with no clear structure or organization. It is getting hard to maintain.It also doesn't work well with testing (a) feature macros that affect
Python.hand (b) module initialization, so we have additional C-API testing modules:_testmultiphase,_testimportmultiple,_testinternalcapi._testbufferis already split out, but there are many other aspects of the API that would use a similar dedicated test suite.We should split and combine these, ideally without polluting the namespace of top-level modules.