_socket extension C capsule cannot be visited by the GC and so _socket.socket type may stay alive longer than expected

Bug report

Checklist

I am confident this is a bug in CPython, not a bug in a third-party project
I have searched the CPython issue tracker,
and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]

A clear and concise description of the bug:

The following script will never delete _socket.socket type:

import _socket

# uncomment to workaround the bug
#_socket.CAPI = None

_socket = None

import sys
del sys.modules['_socket']

import gc
gc.collect()

print("exit")

Example on a Python debug build:

$ ./python -X showrefcount script.py 
exit
[408 refs, 260 blocks]

The GC cannot visit the strong reference to the type stored in _socket.CAPI capsule (C API). In Python, creating a heap type creates a reference cycle. For example, a type MRO contains the type. Methods also contain a strong reference to their type.

$ ./python
Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
>>> import _socket
>>> t=_socket.socket
>>> t.__mro__[0] is t
True

Linked PRs

pythongh-108240: Fix a reference cycle in _socket.CAPI capsule

I wrote PR #108241 to fix the issue for the _socket extension: avoid strong references in the capsule object.

_curses extension is affected by the same bug: _curses._C_API capsule stores a strong reference to _curses.window heap type.

Extensions not affected by the issue, using borrowed references:

_datetime.datetime_CAPI stores borrowed references to _datetime heap types
pyexpat.expat_CAPI does not store types
unicodedata._ucnhash_CAPI does not store types
_testcapi capsules don't store types

By the way, commit 46366ca may have introduced a leak: see PR #107577.

I closed issue #107789 as a duplicate of this issue.

@Eclips4

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

@Eclips4

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

Probably related: #107994

Probably related: #107994

Oh right, the _decimal extension has no capsule, so the refleak is unrelated to this issue.

@erlend-aasland

cc @erlend-aasland @corona10 @pablogsal: yet another complex GC issue involving extension multi-phase initialization and the garbage collector.

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

You mean leak in the _curses is a known issue?

import importlib
import sys

def foo():
    name = "_curses"
    importlib.import_module(name)
    del sys.modules[name]


for _ in range(4):
    foo()

./python.exe -X showrefcount example.py
[1006 refs, 491 blocks]

You mean leak in the _curses is a known issue?

Yes. It should be ported to multi-phase init and its static types should be converted to heap types.

Yes. It should be ported to multi-phase init and its static types should be converted to heap types.

FYI, it seems that work on isolating the _curses extension module is suspended. #101714 (comment)

@chgnrdv

FYI, it seems that work on isolating the _curses extension module is suspended. #101714 (comment)

So, should we re-open @chgnrdv PR about isolating _curses? Seems work almost done, but isolating module only for avoid reference leak looks kinda weird.

Yes, storing the state in the interpreter is an option, but I don't think we want to do that for tkinter, readline, and curses.

pythongh-108240: Add PyCapsule_SetTraverse() function

pythongh-108240: Add _PyCapsule_SetTraverse() function

pythongh-108240: Add _PyCapsule_SetTraverse() interal function

gh-108240: Add _PyCapsule_SetTraverse() internal function (#108339)

@Eclips4

Fixed by commit 513c89d

Thanks for your bug report @Eclips4.

pythongh-108240: _PyCapsule_SetTraverse() rejects NULL callbacks

gh-108240: _PyCapsule_SetTraverse() rejects NULL callbacks (#108417)

pythongh-108240: Add pycore_capsule.h internal header file

gh-108240: Add pycore_capsule.h internal header file (#108596)

Navigation Menu

Repository navigation

_socket extension C capsule cannot be visited by the GC and so _socket.socket type may stay alive longer than expected #108240

Bug report

Checklist

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

A clear and concise description of the bug:

Linked PRs

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

_socket extension C capsule cannot be visited by the GC and so _socket.socket type may stay alive longer than expected #108240

Description

Bug report

Checklist

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

A clear and concise description of the bug:

Linked PRs

Activity

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

Eclips4 commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

Eclips4 commented on Aug 21, 2023

vstinner commented on Aug 21, 2023

CharlieZhao95 commented on Aug 22, 2023

Eclips4 commented on Aug 22, 2023

4 remaining items

erlend-aasland commented on Aug 22, 2023

vstinner commented on Aug 23, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions