test_free_threading.test_monitoring flaky (occasional reference leak)

Bug report

When running ./python -E -m test --timeout=2700 -R3:3 -u-cpu -j40 test_free_threading.test_monitoring -F, I get a failure every ~50-100 test runs:

0:00:36 load avg: 122.12 [ 55/1] test_free_threading.test_monitoring failed (reference leak)
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XX. 243
test_free_threading.test_monitoring leaked [2, 4, 3] memory blocks, sum=9

I think the issue is that we rely on Thread.is_alive() instead of joining the threads. I think there are still some objects that may not be freed until the thread is joined (like the handle):

cpython/Lib/test/test_free_threading/test_monitoring.py

Lines 65 to 77 in cb59eae

    
           while True: 
        
               any_alive = False 
        
               for t in threads: 
        
                   if t.is_alive(): 
        
                       any_alive = True 
        
                       break 
        
               if not any_alive: 
        
                   break 
        
               self.during_threads() 
        
           self.after_test()

Linked PRs

There's a second source of flakiness: sys.monitoring lazily allocates monitoring data the first time a code object is run under instrumentation. The test_free_threading.test_monitoring flips turns on and off monitoring during the test run. It's possible that the test case finishes with monitoring still enabled and tearDown turns it off.

cpython/Lib/test/test_free_threading/test_monitoring.py

Lines 125 to 142 in 180b3eb

    
           def tearDown(self): 
        
               monitoring.set_events(self.tool_id, 0) 
        
               super().tearDown() 
        
           def callback(self, *args): 
        
               self.called = True 
        
           def after_test(self): 
        
               self.assertTrue(self.called) 
        
           def during_threads(self): 
        
               if self.set: 
        
                   monitoring.set_events( 
        
                       self.tool_id, monitoring.events.CALL | monitoring.events.LINE 
        
                   ) 
        
               else: 
        
                   monitoring.set_events(self.tool_id, 0) 
        
               self.set = not self.set

This can lead to a false reported reference leak. If you are unlucky, the first three warmup tests will finish with monitoring disabled. If a subsequent run finishes with monitoring still enabled, then parts of unittest get instrumented before tearDown() is called leading to new lazily allocated monitoring data.

Adding a monitoring.set_events(self.tool_id, 0) to after_test() helps, but there are similar (but rare) issues with other code paths getting non-deterministically executed under monitoring in some runs but not others.

@gaogaotiantian

@gaogaotiantian, I'd like to get your take on this:

We have some flakiness in refleak tests that use threads and sys.monitoring. Basically, there's some non-determinism in terms of which code objects get run under monitoring so sometimes _co_monitoring and other monitoring-related fields get initialized after the "warmup" runs, and get (mis)reported as refleaks.

To avoid flakiness, I'm thinking about clearing all unused monitoring data in either sys._clear_internal_caches or test.libregrtest.utils.clear_caches, at least in the free threading build. What do you think about this?

How do you clear all unused monitoring data? How do you get all the code objects that are instrumented? Are you going to have a list for all the code objects?

How do you clear all unused monitoring data?

If monitoring is disabled globally and on the code object, we call instrument_lock_held on the code object. Otherwise, we leave the code object alone.

How do you get all the code objects that are instrumented?

In the free threading build, we traverse the mimalloc heaps to find code objects. We already do this to find and clear unused thread-local bytecode in sys._clear_internal_caches.

I think we limit this to the free threading build for now, because I don't really want a list of all code objects.

Why do you need to check if monitoring is disabled globally? I think instrument_lock_held checks it internally right? To avoid the extra lock cost? I think we only need to lock once and iterate through all the code objects right?

#120158 was marked as a duplicate, but the error there looks very different to the me -- a crash, rather than a refleak:

test_instrumentation (test.test_free_threading.test_monitoring.SetProfileMultiThreaded.test_instrumentation) ... python: Python/instrumentation.c:1305: _Py_call_instrumentation_line: Assertion `debug_check_sanity(tstate->interp, code)' failed.
python: Python/instrumentation.c:1305: _Py_call_instrumentation_line: Assertion `debug_check_sanity(tstate->interp, code)' failed.
Fatal Python error: Aborted
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/case.py", line 666 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/case.py", line 723 in __call__
  File "python: Python/instrumentation.c:1105: get_tools_for_instruction: Assertion `debug_check_sanity(interp, code)' failed.
/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py"
1 test failed again:
    test_free_threading

pythongh-136692: Provide extra debug info when monitoring assertion f…

I'm not sure if the refleak and crash have the same underlying cause. It happens fairly often in the buildbots, but I've had a hard time reproducing the crash locally.

I think what's happening is that we're overflowing the global monitoring version, so we bail out in an inconsistent state:

cpython/Python/instrumentation.c

Lines 2024 to 2029 in 92972ae

    
           set_events(&interp->monitors, tool_id, events); 
        
           uint32_t new_version = global_version(interp) + MONITORING_VERSION_INCREMENT; 
        
           if (new_version == 0) { 
        
               PyErr_Format(PyExc_OverflowError, "events set too many times"); 
        
               return -1; 
        
           }

We already set interp->monitors but don't instrument code objects.

Navigation Menu

Repository navigation

`test_free_threading.test_monitoring` flaky (occasional reference leak) #136692

Bug report

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	while True:
	any_alive = False
	for t in threads:
	if t.is_alive():
	any_alive = True
	break

	if not any_alive:
	break

	self.during_threads()

	self.after_test()

	def tearDown(self):
	monitoring.set_events(self.tool_id, 0)
	super().tearDown()

	def callback(self, *args):
	self.called = True

	def after_test(self):
	self.assertTrue(self.called)

	def during_threads(self):
	if self.set:
	monitoring.set_events(
	self.tool_id, monitoring.events.CALL \| monitoring.events.LINE
	)
	else:
	monitoring.set_events(self.tool_id, 0)
	self.set = not self.set

	set_events(&interp->monitors, tool_id, events);
	uint32_t new_version = global_version(interp) + MONITORING_VERSION_INCREMENT;
	if (new_version == 0) {
	PyErr_Format(PyExc_OverflowError, "events set too many times");
	return -1;
	}

test_free_threading.test_monitoring flaky (occasional reference leak) #136692

Description

Bug report

Linked PRs

Activity

colesbury commented on Jul 16, 2025

colesbury commented on Jul 21, 2025

gaogaotiantian commented on Jul 23, 2025

colesbury commented on Jul 23, 2025

gaogaotiantian commented on Jul 24, 2025

encukou commented on Nov 11, 2025

colesbury commented on Nov 21, 2025

colesbury commented on Nov 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`test_free_threading.test_monitoring` flaky (occasional reference leak) #136692