Skip to content

Isolate the _io extension module #101819

Description

@erlend-aasland

Isolate the _io extension module by moving all global variables to module state, porting static types to heap types, and implementing multi-phase init. All global variables in the _io module are static types:

  • Modules/_io/bufferedio.c: PyBufferedIOBase_Type
  • Modules/_io/bufferedio.c: PyBufferedRWPair_Type
  • Modules/_io/bufferedio.c: PyBufferedRandom_Type
  • Modules/_io/bufferedio.c: PyBufferedReader_Type
  • Modules/_io/bufferedio.c: PyBufferedWriter_Type
  • Modules/_io/bytesio.c: PyBytesIO_Type
  • Modules/_io/bytesio.c: _PyBytesIOBuffer_Type
  • Modules/_io/fileio.c: PyFileIO_Type
  • Modules/_io/iobase.c: PyIOBase_Type
  • Modules/_io/iobase.c: PyRawIOBase_Type
  • Modules/_io/textio.c: PyIncrementalNewlineDecoder_Type
  • Modules/_io/textio.c: PyTextIOBase_Type
  • Modules/_io/textio.c: PyTextIOWrapper_Type
  • Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type

Converting the static types to heap types involves applying PEP-687 to _io.
Adapting multi-phase init involves applying PEP-489 to _io.

Linked PRs

Activity

erlend-aasland commented on Feb 11, 2023

@erlend-aasland
ContributorAuthor

erlend-aasland commented on Feb 13, 2023

@erlend-aasland
ContributorAuthor

I see that _PyWindowsConsoleIO_Type is marked with PyAPI_DATA, but I can't find it in Misc/stable_abi.toml. @zooba or @eryksun, do you know why this is so? I did a quick code search on GitHub, but I only got hits in various clones of the CPython code base.

#ifndef Py_LIMITED_API
#ifdef MS_WINDOWS
extern PyTypeObject PyWindowsConsoleIO_Type;
PyAPI_DATA(PyObject *) _PyWindowsConsoleIO_Type;
#define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), (PyTypeObject*)_PyWindowsConsoleIO_Type))
#endif /* MS_WINDOWS */
#endif /* Py_LIMITED_API */

UPDATE: I see it was added by Steve in 312cef7, probably because of the PyWindowsConsoleIO_Check checks in PC/_testconsole.c and Python/pylifecycle.c:

if (!PyWindowsConsoleIO_Check(file)) {
PyErr_SetString(PyExc_TypeError, "expected raw console object");
return NULL;
}

extern PyTypeObject PyWindowsConsoleIO_Type;
# define PyWindowsConsoleIO_Check(op) \
(PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type))
#endif

cpython/Python/pylifecycle.c

Lines 2359 to 2363 in 2db2c4b

#ifdef MS_WINDOWS
/* Windows console IO is always UTF-8 encoded */
if (PyWindowsConsoleIO_Check(raw))
encoding = L"utf-8";
#endif

Perhaps exposing _PyWindowsConsoleIO_Type through a CPython internal capsulated C API could be an option?

erlend-aasland commented on Feb 13, 2023

@erlend-aasland
ContributorAuthor

Other issues that must be resolved before continuing: _PyBytesIOBuffer_Type is used by _testcapimodule. Since that type is not exposed in _io, we cannot simply use PyImport_Import and look it up. One possibility is to expose it through a CPython internal capsule C API.

erlend-aasland commented on Feb 13, 2023

@erlend-aasland
ContributorAuthor

... or we could simply expose the types mentioned in #101819 (comment) and #101819 (comment) in the _io module.

zooba commented on Feb 13, 2023

@zooba
Member

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

encukou commented on Feb 13, 2023

@encukou
Member

The type is explicitly not part of the limited API/stable ABI, see #ifndef Py_LIMITED_API.

erlend-aasland commented on Feb 13, 2023

@erlend-aasland
ContributorAuthor

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

IIUC, we can remove this from the DLL and instead export it via _io and look it up from the module using the C API (PyImport_Import, etc.).

zooba commented on Feb 13, 2023

@zooba
Member

Yeah, if it's got a leading underscore then we can do whatever we need to make our tests work.

50 remaining items

added a commit that references this issue on May 11, 2023
added a commit that references this issue on May 11, 2023
added 2 commits that reference this issue on May 12, 2023
added a commit that references this issue on May 12, 2023
added a commit that references this issue on May 15, 2023
added a commit that references this issue on May 15, 2023

vstinner commented on May 15, 2023

@vstinner
Member

@erlend-aasland erlend-aasland closed this as completed

Great achievement!

serhiy-storchaka commented on Oct 23, 2023

@serhiy-storchaka
Member

It causes a crash during garbage collection. See #111049, #111211.

added 2 commits that reference this issue on Sep 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions