test_free_threading.test_dict on TSAN/free-threading is flaky

Bug report

Bug description:

We have:

0:04:15 load avg: 7.63 [20/23/2] test_free_threading worker non-zero exit code (Exit code -6 (SIGABRT)) -- running (2): test_socket (1 min 38 sec), test_threading (38.8 sec)
python: Objects/obmalloc.c:1219: void process_queue(struct llist_node *, struct _qsbr_thread_state *, _Bool, delayed_dealloc_cb, void *): Assertion `buf->rd_idx == buf->wr_idx' failed.
Fatal Python error: Aborted

<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/home/runner/work/cpython/cpython/Lib/test/test_free_threading/test_dict.py", line 184 in writer_func
  File "/home/runner/work/cpython/cpython/Lib/threading.py", line 996 in run
  File "/home/runner/work/cpython/cpython/Lib/threading.py", line 1054 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython/Lib/threading.py", line 1016 in _bootstrap

Extension modules: _testinternalcapi, _testcapi (total: 2)

I'm not sure if it's a real bug or not. Victor suggested me to open an issue for this one. Maybe there's already one that exists though.

See https://github.com/python/cpython/actions/runs/13225352402/job/36915529351?pr=129175#step:12:44 for the log (hopefully it will stay).

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

gh-130519: Fix crash in QSBR when destructor reenters QSBR #130553

I'm a bit stuck with my last PR. Do you want me to look into this?

@ZeroIntensity

You can investigate if you want but maybe free-threading experts have an idea of why it fails.

cc @ZeroIntensity maybe you've encountered this beforehand?

I haven't seen it. I think there's a thread somewhere where you're supposed to report flaky TSan failures.

@DinoV

This is a good place to report it, thanks.

I think the problem is that free_work_item can be re-entrant because it calls Py_DECREF(). Originally, the QSBR code just made calls to PyMem_Free() or PyObject_Free(), which are never re-entrant. In:

gh-124470: Fix crash when reading from object instance dictionary while replacing it #122489

We extended it to also executing Py_DECREF() on objects, which can execute arbitrary code via destructors, including re-entering QSBR (possibly during a GC).

This invalidates some assumptions. For example, the head of the work queue can change during a call to free_work_item:

cpython/Objects/obmalloc.c

Lines 1258 to 1268 in c5f925c

    
           struct _mem_work_chunk *buf = work_queue_first(head); 
        
           while (buf->rd_idx < buf->wr_idx) { 
        
               struct _mem_work_item *item = &buf->array[buf->rd_idx]; 
        
               if (!_Py_qsbr_poll(qsbr, item->qsbr_goal)) { 
        
                   return; 
        
               } 
        
               free_work_item(item->ptr, cb, state); 
        
               buf->rd_idx++; 
        
           }

cc @DinoV

pythongh-130519: WIP

pythongh-130519: Fix crash in QSBR when destructor reenter QSBR

gh-130519: Fix crash in QSBR when destructor reenters QSBR (gh-130553)

This should be fixed now.

pythongh-134679: Fix assertion failure in QSBR

gh-134679: Fix assertion failure in QSBR (gh-134811)

pythongh-134679: Fix assertion failure in QSBR (pythongh-134811)

[3.14] gh-134679: Fix assertion failure in QSBR (gh-134811) (gh-134814)

pythongh-134679: Fix assertion failure in QSBR (pythongh-134811)

Navigation Menu

Repository navigation

`test_free_threading.test_dict` on TSAN/free-threading is flaky #130519

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	struct _mem_work_chunk *buf = work_queue_first(head);

	while (buf->rd_idx < buf->wr_idx) {
	struct _mem_work_item *item = &buf->array[buf->rd_idx];
	if (!_Py_qsbr_poll(qsbr, item->qsbr_goal)) {
	return;
	}

	free_work_item(item->ptr, cb, state);
	buf->rd_idx++;
	}

test_free_threading.test_dict on TSAN/free-threading is flaky #130519

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

sergey-miryanov commented on Feb 24, 2025

picnixz commented on Feb 24, 2025

ZeroIntensity commented on Feb 24, 2025

colesbury commented on Feb 25, 2025

colesbury commented on Mar 2, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`test_free_threading.test_dict` on TSAN/free-threading is flaky #130519