_PyImport_SwapPackageContext would be helpful if exposed

Feature or enhancement

In Nuitka, we have our own meta-path-based loader, which loads packages that contain extension modules with custom code that emulates the load process. For an extension module in a package, we used to set _Py_PackageContext, which has since been removed. Currently, _PyRuntime.imports.pkgcontext is not used because we fall back to a thread-local pkgcontext variable, which Nuitka cannot access on Windows or where static linking to Python is not available.

This leads to some issues when Nuitka goes to load extensions that are stored in a submodule or create submodules, because the package context is not correct. For example, an extension module under x.y will create submodule z as y.z rather than x.y.z. There are currently some workarounds to repair the damage, but this is tedious and sometimes unreliable. If _PyImport_SwapPackageContext were exposed, Nuitka could just call that and make PyModule_Create know what package it lives in.

Nuitka doesn't need _PyImport_SwapPackageContext to be exposed as public or PyUnstable, we would just like to have it available in the ABI under PyAPI_FUNC. It would be very helpful if this could happen before 3.14.0, but that's up to Hugo.

Linked PRs

This was discussed in https://discuss.python.org/t/43396 for another loader, but we found a workaround for that use case.
IMO it would be helpful to discuss Nuitka's case too.

I think Meta's case only involved statically linked extensions, but this applies to all extensions inside or with submodules in Nuitka's case. I do agree that we should get _PyImport_SwapPackageContext as PyUnstable_Import_SwapPackageContext eventually, but that ship has already sailed for 3.14.

Is it Nuitka-compiled modules only (which could be fixable by using multi-phase-init), or also third-party ones?

Third-party ones.

OK, that's not as easy. A 3.14 change is in the hands of the RM no, I guess.

Other than that, it looks like the process is broken. How did this behaviour manage to survive the beta period and three RCs?

Other than that, it looks like the process is broken. How did this behaviour manage to survive the beta period and three RCs?

This has been the case since 3.12, as far as I can tell. Again, Nuitka has been working around it using some hacks, but we've been hesitant to ask for upstream changes as we're relying on internals. It'd be fine if it didn't make it to 3.14, but just a little inconvenient.

It'd be fine if it didn't make it to 3.14, but just a little inconvenient.

If that's the case, you can pretty much close the PR -- a code change after the final RC should need an exceptionally good reason.

Instead, shall we aim for Unstable API? That means it needs documentation & tests.
The new docs single-phase init should make it easier to explain why you'd need this function.

I can just update the PR to use the unstable API.

Instead, shall we aim for Unstable API? That means it needs documentation & tests.

Yeah, but that makes it a lot more difficult to justify, and adds more maintenance burden on us (CPython, not Nuitka -- I've been back and forth with "we"/"us"). Explicit tests for something like this are going to be complex and won't catch much anyway, because if we were to break _PyImport_SwapPackageContext then our import system would fail anyway.

I'm not opposed to making it PyUnstable, but we should perhaps be more relaxed with our requirements for unstable APIs here.

Would GH-139196 help here?

If we want 3rd parties like Nuitka to use something, we should expose it as a proper interface, document how it should be used, and test it so that we don't break it in a patch release.

if we were to break _PyImport_SwapPackageContext then our import system would fail anyway.

Not necessarily. We could, for example, add an argument to it and update all the callers.

This is more urgent now that 3.15 breaks the ability to access pkgcontext from outside CPython.

Would #139196 help here?

It could, but the devil's in the details. Nuitka does a bunch of PGO shenanigans and adds other shims that would probably break if we switched to that. At this point, it'd be much simpler if we simply had access to _PyImport_SwapPackageContext.

That said, I don't think I can reasonably come up with a test case to exercise a PyUnstable_Import_SwapPackageContext, at least short of writing an entire importer, which is not ideal for a single unstable C API function. Maybe that's indicative of this being too low-level for the unstable API? I'm not sure.

I don't understand well the purpose of this "package context". It's made of multiple parts:

_PyRuntime.imports.pkgcontext
static _Py_thread_local const char *pkgcontext = NULL; in Python/import.c
const char* _PyImport_ResolveNameWithPackageContext(const char *name)
const char* _PyImport_SwapPackageContext(const char *newcontext)

To understand how it works, I commented out _PyImport_SwapPackageContext() calls in Python/importdl.c. Problem: the whole test suite pass without it :-( It doesn't help me to understand its purpose... So this API is not tested by Python test suite.

From what I understood, the "package context" is used to override the module name in _PyModule_CreateInitialized(). For example, PyModuleDef.m_name is "module", but if the package context is set to "package.module", the module name becomes "package.module".

Why PyModuleDef.m_name is not "package.module" at the beginning?

Note: _PyRuntime.imports.pkgcontext is no longer used and can be removed. I wrote PR gh-151490 to remove it.

_PyImport_SwapPackageContext() stores a raw pointer to a bytes string without copying the string. If we expose it, we should clearly describe this limitation: the caller must ensure that the pointer remains valid until the next _PyImport_SwapPackageContext() call.

_PyImport_RunModInitFunc() calls _PyImport_SwapPackageContext() with info->newcontext argument:

    /* Package context is needed for single-phase init */
    const char *oldcontext = _PyImport_SwapPackageContext(info->newcontext);
    PyObject *m = p0();
    _PyImport_SwapPackageContext(oldcontext);

The info is a struct _Py_ext_module_loader_info* structure which is set in a caller function:

int
_Py_ext_module_loader_info_init(struct _Py_ext_module_loader_info *p_info,
                                PyObject *name, PyObject *filename,
                                _Py_ext_module_origin origin)
{
    ...
    info.name = Py_NewRef(name);
    info.name_encoded = get_encoded_name(info.name, &info.hook_prefixes);
    if (info.name_encoded == NULL) {
        ...
        return -1;
    }

_PyImport_SwapPackageContext() just stores a raw pointer to a string, it doesn't copy the string:

const char *
_PyImport_SwapPackageContext(const char *newcontext)
{
    const char *oldcontext = PKGCONTEXT;
    PKGCONTEXT = newcontext;
    return oldcontext;
}

So PKGCONTEXT remains valid until info->name is valid (strong reference).

gh-139227: Remove unused _PyRuntime.imports.pkgcontext (#151490)

pythongh-139227: Remove unused _PyRuntime.imports.pkgcontext (python#…

I was able to implement this in Nuitka using 3.15's new PyImport_CreateModuleFromInitfunc API, so I no longer need this. Thanks for the insight, both of you!

Navigation Menu

Repository navigation

`_PyImport_SwapPackageContext` would be helpful if exposed #139227

Feature or enhancement

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

_PyImport_SwapPackageContext would be helpful if exposed #139227

Description

Feature or enhancement

Linked PRs

Activity

encukou commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

encukou commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

encukou commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

encukou commented on Sep 22, 2025

ZeroIntensity commented on Sep 22, 2025

encukou commented on Sep 23, 2025

ZeroIntensity commented on Jun 15, 2026

vstinner commented on Jun 15, 2026

vstinner commented on Jun 15, 2026

ZeroIntensity commented on Jul 7, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`_PyImport_SwapPackageContext` would be helpful if exposed #139227