Numerous refleaks on main

@sunmy2019

https://buildbot.python.org/all/#/release_status shows that the refleak buildbots are failing on current main. Example output on https://buildbot.python.org/all/#/builders/75/builds/745/steps/5/logs/warnings__111_.

Failing tests:

Details

test_functools leaked [1, 1, 2] references, sum=4
test_warnings leaked [4, 4, 4] references, sum=12
test_ipaddress leaked [6, 6, 6] references, sum=18
test_zoneinfo leaked [7, 6, 11] references, sum=24
FAIL: test_flock (__main__.FNTLEINTRTest.test_flock)
test_random leaked [1, 2, 4] references, sum=7
test_statistics leaked [175, 174, 175] references, sum=524
test_dataclasses leaked [3, 3, 2] references, sum=8
test_argparse leaked [69, 70, 76] references, sum=215
/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/support/__init__.py:743: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=2, type=1, proto=6>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
test_tarfile leaked [17, 17, 16] references, sum=50
test_typing leaked [95, 121, 95] references, sum=311
test_typing leaked [1, 14, 1] memory blocks, sum=16
test_lib2to3 leaked [5, 8, 6] references, sum=19
test_shutil leaked [2, 2, 2] references, sum=6
test_unittest leaked [8, 12, 15] references, sum=35
test_logging leaked [3, 2, 1] references, sum=6
test__xxsubinterpreters leaked [46, 45, 45] references, sum=136
test_mailbox leaked [2, 3, 2] references, sum=7
test_socket leaked [24, 23, 23] references, sum=70
test_htmlparser leaked [2, 2, 2] references, sum=6
test_regrtest leaked [9, 12, 8] references, sum=29
test_datetime leaked [11, 11, 11] references, sum=33
test_nntplib leaked [1, 3, 2] references, sum=6
test_ftplib leaked [10, 13, 9] references, sum=32
test_interpreters leaked [54, 54, 54] references, sum=162
test_zipfile leaked [3, 4, 4] references, sum=11
test_sys_settrace leaked [6, 6, 6] references, sum=18
test_descr leaked [3, 65, 67] references, sum=135
test_compile leaked [7, 7, 7] references, sum=21
test_bytes leaked [2, 2, 2] references, sum=6
test_ast leaked [9, 9, 9] references, sum=27
test_poplib leaked [3, 3, 2] references, sum=8
test_types leaked [8, 50, 43] references, sum=101
test_concurrent_futures leaked [11, 12, 11] references, sum=34
test_genericalias leaked [2, 2, 3] references, sum=7
test_math leaked [32, 31, 33] references, sum=96
test_numeric_tower leaked [2, 1, 2] references, sum=5
test_capi leaked [99, 99, 99] references, sum=297
test_import leaked [170, 171, 168] references, sum=509
test_import leaked [71, 71, 70] memory blocks, sum=212
.test test_monitoring failed -- multiple errors occurred; run in verbose mode for details
test_selectors leaked [5, 6, 4] references, sum=15
test_email leaked [27, 26, 25] references, sum=78
test_subprocess leaked [1, 1, 1] references, sum=3
test_struct leaked [1, 1, 1] references, sum=3
test_weakref leaked [1, 1, 1] references, sum=3
test_imaplib leaked [2, 2, 1] references, sum=5
0:45:54 load avg: 0.13 Re-running failed tests in verbose mode
0:45:54 load avg: 0.13 Re-running test_functools in verbose mode
test_functools leaked [1, 1, 2] references, sum=4
0:45:55 load avg: 0.13 Re-running test_warnings in verbose mode
test_warnings leaked [4, 4, 4] references, sum=12
0:45:59 load avg: 0.20 Re-running test_ipaddress in verbose mode
test_ipaddress leaked [6, 6, 6] references, sum=18
0:46:00 load avg: 0.20 Re-running test_zoneinfo in verbose mode
test_zoneinfo leaked [7, 6, 11] references, sum=24

I saw similar leaks triggering the refleak buildbots on #103866 and #103764, before I realized the issue was probably on main.

Based on @sunmy2019's work in #103764 (comment), we're likely leaking references to the object.__setattr__ wrapper descriptor.

Linked PRs

gh-103879: Fix refleak in super specialization #103882

@carljm

The failure on test.test_htmlparser bisects to

ef25febcf2ede92a03c5ea00a13e167e0b5cb274 is the first bad commit
commit ef25febcf2ede92a03c5ea00a13e167e0b5cb274
Author: Carl Meyer <carl@oddbird.net>
Date:   Tue Apr 25 11:45:51 2023 -0600

    gh-87729: specialize LOAD_SUPER_ATTR_METHOD (#103809)

@carljm

pythongh-103879: Fix refleak in super specialization

gh-103879: Fix refleak in super specialization (#103882)

Given that refleak buildbots passed on #103882, it seems this is fixed now.

@JelleZijlstra

Thanks @JelleZijlstra for fixing!

No problem!

https://buildbot.python.org/all/#/release_status is still red but I assume the buildbots are running now. Given that the refleak buildbots were green on my PR, we probably did fix the issue.

pythongh-103879: Fix refleak in super specialization (python#103882)

We might have another refleak on test_import, visible on main and on #103866 link.

I wasn't able to reproduce the new leak locally (tried both MacOS and Linux). It seems to happen only on one buildbot.

The leak of test_import is still visible on that buildbot, and the issue persists for a long time

Visible at
https://buildbot.python.org/all/#/builders/205/builds/696
... (note: logs will be cleared for old tests, soon the log will not be visible)

Usually, rerunning the tests will let it pass. (Warning, no errors)

I suspect something changed in the test environment.

It seems to happen only on one buildbot.

This is not true. test_import leaked on the first run and passed in the second run. Visible on various arch*os combinations.

https://buildbot.python.org/all/#/builders/766/builds/770
https://buildbot.python.org/all/#/builders/474/builds/1161
https://buildbot.python.org/all/#/builders/411/builds/1216
https://buildbot.python.org/all/#/builders/802/builds/814
https://buildbot.python.org/all/#/builders/210/builds/1196
https://buildbot.python.org/all/#/builders/18/builds/1233
https://buildbot.python.org/all/#/builders/409/builds/1169

...

@JelleZijlstra

@JelleZijlstra I can reproduce it with

./python -m test -j1 -R 3:3 test_import

Single processing will not cause leakage. -j is necessary.

Good catch, that reproduces it for me too.

@ericsnowcurrently

Current status:

Found most leakage is from test_import.SinglephaseInitTests.

ref change of type dict: +7
ref change of type float: +21
ref change of type getset_descriptor: +5
ref change of type int: +16
ref change of type str: ... (immortal str not counted)
ref change of type tuple: +7
ref change of type type: +12
ref change of type weakref.ReferenceType: +5

I took a quick look, ref change of type type: +12 contains many instances created by

    state->error = PyErr_NewException("_testsinglephase.error", NULL, NULL);

Modules/_testsinglephase.c:init_module is called several times during -j tests, but only once with the single-threaded one.

cpython/Modules/_testsinglephase.c

Lines 125 to 148 in 738c226

    
           static int 
        
           init_module(PyObject *module, module_state *state) 
        
           { 
        
               if (PyModule_AddObjectRef(module, "error", state->error) != 0) { 
        
                   return -1; 
        
               } 
        
               if (PyModule_AddObjectRef(module, "int_const", state->int_const) != 0) { 
        
                   return -1; 
        
               } 
        
               if (PyModule_AddObjectRef(module, "str_const", state->str_const) != 0) { 
        
                   return -1; 
        
               } 
        
               double d = _PyTime_AsSecondsDouble(state->initialized); 
        
               PyObject *initialized = PyFloat_FromDouble(d); 
        
               if (initialized == NULL) { 
        
                   return -1; 
        
               } 
        
               if (PyModule_AddObjectRef(module, "_module_initialized", initialized) != 0) { 
        
                   return -1; 
        
               } 
        
               return 0; 
        
           }

At least, I found initialized is leaked here. PyModule_AddObjectRef creates a new reference. The previous leak statistics indicate state->error may also be leaked elsewhere.

@ericsnowcurrently Can you please take a look since the last change is yours?

@ericvsmith

@ericvsmith I tracked down the problem. Hope you can fix it and add me as a co-author.

1

initialized is leaked as mentioned above.

2

_testsinglephase_with_state module. Its md_state is not released by _testinternalcapi.clear_extension or _clear_globals.

When I manually add and call the clean-up function. Ref leaks are gone.

3

cpython/Lib/test/test_import/__init__.py

Lines 2458 to 2462 in 738c226

    
                   # Start with an interpreter that gets destroyed right away. 
        
                   base = self.import_in_subinterp(postscript=''' 
        
                       # Attrs set after loading are not in m_copy. 
        
                       mod.spam = 'spam, spam, mash, spam, eggs, and spam' 
        
                   ''')

No cleanup was made against base, which is leaking ref. This line looks intentional from the comment, but it does leak things.

Most of the ref leaks in test_import are here. Hopefully, it should be all.

@sunmy2019

@sunmy2019, there are known refleaks in test_import. See gh-102251. Are you talking about more than that?

Also, your analysis is helpful. Thanks!

Are you talking about more than that?

Yes, I am talking about test_import. The link you give seems to be test_imp.

The ref leak in test_import is introduced at least two months ago:
https://buildbot.python.org/all/#/builders/205/builds/696

and still visible and reproducible on main:

./python -m test -j1 -R 3:3 test_import

The above guide should help you clear major ref leaks in test_import.

The tests in question were moved from test_imp to test_import.

I think the main refleaks are fixed now, closing this as the test_import refleaks are tracked in #102251.

Navigation Menu

Numerous refleaks on main #103879

Linked PRs

1

2

3

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	static int
	init_module(PyObject module, module_state state)
	{
	if (PyModule_AddObjectRef(module, "error", state->error) != 0) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "int_const", state->int_const) != 0) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "str_const", state->str_const) != 0) {
	return -1;
	}

	double d = _PyTime_AsSecondsDouble(state->initialized);
	PyObject *initialized = PyFloat_FromDouble(d);
	if (initialized == NULL) {
	return -1;
	}
	if (PyModule_AddObjectRef(module, "_module_initialized", initialized) != 0) {
	return -1;
	}

	return 0;
	}

	# Start with an interpreter that gets destroyed right away.
	base = self.import_in_subinterp(postscript='''
	# Attrs set after loading are not in m_copy.
	mod.spam = 'spam, spam, mash, spam, eggs, and spam'
	''')

Numerous refleaks on main #103879

Description

Linked PRs

Activity

JelleZijlstra commented on Apr 26, 2023

carljm commented on Apr 26, 2023

carljm commented on Apr 26, 2023

JelleZijlstra commented on Apr 26, 2023

JelleZijlstra commented on Apr 27, 2023

JelleZijlstra commented on Apr 27, 2023

sunmy2019 commented on Apr 28, 2023

sunmy2019 commented on Apr 28, 2023

sunmy2019 commented on Apr 28, 2023

JelleZijlstra commented on Apr 28, 2023

sunmy2019 commented on Apr 29, 2023

sunmy2019 commented on Apr 29, 2023

1

2

3

ericsnowcurrently commented on May 1, 2023

sunmy2019 commented on May 1, 2023

ericsnowcurrently commented on May 2, 2023

JelleZijlstra commented on May 12, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions