Regression segfault under CPython 3.12a5 in the SymPy test suite

Crash report

This comes from sympy/sympy#24776 which adds CPython 3.12 prerelease testing in SymPy's CI.

This is seen with CPython 3.12a5 but not with 3.11 or earlier versions.

The reproducer is to run the SymPy test suite:

pip install sympy==1.11.1

Then

>>> import sympy
>>> sympy.test(subprocess=False)
...
(it takes about 5 minutes)
...
sympy/functions/special/tests/test_delta_functions.py[3] ...                                                                  [OK]
sympy/functions/special/tests/test_elliptic_integrals.py[4] ....                                                              [OK]
sympy/functions/special/tests/test_error_functions.py[24] ................Segmentation fault: 11

I don't yet have a simpler reproducer for this because it seems to be non-deterministic but the SymPy test suite reliably invokes a segfault under 3.12 alpha 5 after about 5 minutes. The tests that are running at the time of the segfault will pass if run in isolation. Running the whole test suite though will cause it to fail randomly at one of a few specific places. I don't have a simpler reproducer because running a smaller part of the test suite does not reproduce the problem.

Error messages

Usually:

Segmentation fault: 11

In one case I have also seen (on OSX):

Bus error: 10

Your environment

CPython versions tested on: 3.12 alpha 5 (the problem is not seen with 3.11 or earlier).
Operating system and architecture: Ubuntu and OSX, both on x84-64 CPU.

The problem was seen initially in GitHub Actions CI on an Ubuntu 20.04 runner but I have also reproduced it locally in OSX (an Intel-CPU Macbook).

I don't immediately have a setup that I can use to bisect this but I will get one set up soon to narrow this down.

Linked PRs

gh-102250: Fix double-decref in COMPARE_AND_BRANCH error case #102287

@oscarbenjamin

@oscarbenjamin do you have time to debug it to help us? Which test exacty fails? Or is it a combination / sequence of tests?

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment.
(Buf AFAIK, sympy is pure-python code..)


sympy/core/tests/test_relational.py[64] ...............f.....................f..
...Modules/gcmodule.c:115: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x1064b25e0
object refcount : 6
object type     : 0x145c63630
object type name: floor
object repr     : floor(x)

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x000000010197c580 (most recent call first):
  Garbage-collecting
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in <genexpr>
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in deduce_all_facts
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 546 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/add.py", line 703 in _eval_is_zero
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 539 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1561 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1555 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 629 in __new__
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/tests/test_relational.py", line 967 in test_issue_10927
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1281 in test_file
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1146 in test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 568 in _test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 498 in test
  File "<stdin>", line 1 in <module>
[1]    21140 abort      python

main (sympy.__version__ == '1.11.1'):

Python 3.12.0a5+ (heads/main:a35fd38b57, Feb 25 2023, 23:27:09) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.12.0-alpha-5) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
sympy/functions/combinatorial/tests/test_comb_numbers.py[24] ....zsh: segmentation fault  ./python.exe

Second run

sympy/functions/special/tests/test_bsplines.py[11] .........wzsh: segmentation fault  ./python.exe

Not on 3.11.2 (Feb 8 Gzipped source tarball, sympy.__version__ == '1.11.1') though:

Python 3.11.2 (main, Feb 26 2023, 15:08:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.11.2-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
True

Moreover

Python 3.11.2+ (heads/3.11:7a0dc8a802, Feb 26 2023, 16:22:23) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
...
True

It looks like if CPython is the culprit, it probably should be bisectable in 3.12. Unfortunately, the test suite is quite long! :)

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment. (Buf AFAIK, sympy is pure-python code..)

The crash you showed suggests this is a segfault arising from a refcounting bug. Yes, SymPy is pure Python. It can be used with other optional dependencies that are not pure Python like numpy, gmpy2 etc but this bug is seen without any of those. A refcounting bug here is definitely a CPython bug.

do you have time to debug it to help us? Which test exacty fails? Or is it a combination / sequence of tests?

I am trying to bisect it currently but it's a slow process because of the time taken to build cpython and then run the sympy test suite. Also it is difficult because the failure itself is not completely deterministic. As I said in the OP running individual tests does not reproduce the failure so it is not possible to narrow it down to less than running the whole test suite.

Actually I have narrowed it down a little bit. It seems that the first 1/6 of the test suite is enough to reproduce this:

import sympy
sympy.test(subprocess=False, split="1/6")

That does not reduce the time taken for the crash to show up but it does reduce the time taken to reproduce the crash not happening which is useful for bisecting.

I haven't quite finished bisecting but I'm currently down to this range:

commit df10571a130c46d67756239e76ad69a3d730a990 (refs/bisect/bad)
Author: Steve Dower <steve.dower@python.org>
Date:   Mon Jan 16 16:05:39 2023 +0000

    gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location (GH-100947)

commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <mark@hotpy.org>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

commit b1a74a182d8762bda51838401ac92b6ebad9632a
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Mon Jan 16 13:46:07 2023 +0300

    gh-101056: Fix memory leak in `formatfloat()` in `bytesobject.c` (#101057)

commit b82049993f74185da71adf2eb8d6c8f15db063e1
Author: scrazzz <scruzism@mail.com>
Date:   Mon Jan 16 13:44:50 2023 +0300

    Docs: Remove duplicate `TraversableResources` reference (#101068)

commit 74c20b6ecee125e76c187c3f78c55f39501702ae (HEAD)
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Mon Jan 16 16:11:57 2023 +0530

    GH-100997: fix refleak in `_testinternalcapi.c` (#101058)

commit 8e9d08b062bbabfe439bc73f82e3d7bb3800189e
Author: Oleg Iarygin <oleg@arhadthedev.net>
Date:   Mon Jan 16 00:36:31 2023 +0400

    gh-101041: Fix a misspelled name of `utctimetuple` in a doc warning (GH-101042)

commit 5e9df888dd5ab0e59d1cebc30c998a17aa65a3e2
Author: Cristián Maureira-Fredes <cmaureir@users.noreply.github.com>
Date:   Sun Jan 15 16:12:37 2023 +0100

    Fix `ctypes` typo in `set_exception` (#101040)

commit f6307d44167e9c110a1db442f7a33deb365e89a8 (refs/bisect/good-f6307d44167e9c110a1db442f7a33deb365e89a8)
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Sun Jan 15 20:39:26 2023 +0530

    GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953)

Tests pass in 3.12.0a4 (Head 3d5d3f7, Jan 10)

@markshannon

Bisect complete:

sympy/functions/elementary/tests/test_miscellaneous.py[18] ......s...........                                                                             [OK]
sympy/functions/elementary/tests/test_piecewise.py[63] ....ww...w..............................................w......                                    [OK]
sympy/functions/elementary/tests/test_trigonometric.py[86] ./bug.sh: line 12: 30073 Bus error: 10           ./python.exe -c 'import sympy; sympy.test(subprocess=False, split="1/6")'
$ git bisect bad
7b14c2ef194b6eed79670aa9d7e29ab8e2256a56 is the first bad commit
commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <mark@hotpy.org>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

:040000 040000 a01bf959d999386154fb4dfef5d568c0288cb316 dbd2b3a4a61424abfa5f9e726973b9d03bd24bfd M	Doc
:040000 040000 0fac1d9d6fc2e2066a40e527653ecdae70011c23 d3479ece49974dc513a50d3bd1d5684834e95970 M	Include
:040000 040000 58458255181b3044deeb7c7f852c89e37d398508 f24643aa3a7f1afbdd58639b6e4496f59f1da4f9 M	Lib
:040000 040000 3a605b820cd18bd683911c0a274405ab875a6c79 11052d23b2fb46e6cb0964518d8c215dbc5b3bcd M	Misc
:040000 040000 9364b4ad219faab45e789b146bd93c5694865d79 9a9702c99b36e043768cfde5b3dbdb97ba8972f3 M	Objects
:040000 040000 40b1d7e8c2b8c5af2fd1186256c49bebd3e9c954 83b3b65a9a27d8cb683313f6d280e9bfd20f16ca M	Python
:040000 040000 0cae5606a7c50631954bb014dd67b0384f201210 12b0ea6eb5eafad436ccf7564bacd0a72f40ab59 M	Tools

That's 7b14c2e from gh-100983.

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

CC @markshannon

Full test suite green for me in b1a74a1.

Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?

Attempting to run the full test suite at 7b14c2e, I encounter a segfault at

...
sympy/diffgeom/tests/test_diffgeom.py[16] ................                                               [OK]
sympy/diffgeom/tests/test_function_diffgeom_book.py[4] ..zsh: segmentation fault  ./python.exe

However, running sympy/diffgeom/tests/test_function_diffgeom_book.py repeatedly (100 times) doesn't seem to reproduce the segfault - likewise for sympy/functions/special/tests/test_error_functions.py.

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

I confirm the full test suite passes with the parent commit so I'm confident that the problem begines with 7b14c2e.

Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?

I tried something similar to this. Exactly where it crashes is different on different runs but for example it sometimes crashes in the diffgeom tests so I made a script to run just the diffgeom tests 100 times and that didn't give any crash. I'll try again with:

import sympy

for n in range(1000):
    print()
    print('-------------------------------------- n =', n)
    print()
    sympy.test('test_error_functions', subprocess=False)

It's possible that it would be different by setting the environment variable SYMPY_USE_CACHE=no although that makes everything much slower.

How many repetitions should I expect to need?

@corona10

The crash @corona10 mentioned is reproducible with --with-pydebug enabled. A simpler reproducer is

from sympy.core.symbol import symbols
x = symbols('x')
assert x > 0

The bytecode for both is below

b1a74a1 (good):


  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_OP              68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

7b14c2e (bad):


  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_AND_BRANCH      68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

with the difference between the two being COMPARE_OP vs COMPARE_AND_BRANCH.

I can't cause this alone to cause a segfault when --with-pydebug is not enabled.

Edit: Removing https://github.com/sympy/sympy/blob/sympy-1.11.1/sympy/core/tests/test_relational.py#L860-L869 from the sympy test suite allow sthe subset of the test suite which I was running on (split='1/12') to run without segfaulting anymore. Adding the test back in reintroduces the segfault that appears later on.

A simpler reproducer is

from sympy.core.symbol import symbols
x = symbols('x')
assert x > 0

Okay this starts to make sense. A comparison like x > 0 will call x.__gt__(0) which would usually be expected to return a bool. However Symbol.__gt__ instead returns a symbolic GreaterThan(x, 0) object to represent the inequality symbolically:

>>> cond = x > 0
>>> cond
x > 0
>>> type(cond)
<class 'sympy.core.relational.StrictGreaterThan'>
>>> bool(cond)  # it isn't known if cond is True or False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oscar/current/sympy/sympy.git/sympy/core/relational.py", line 510, in __bool__
    raise TypeError("cannot determine truth value of Relational")
TypeError: cannot determine truth value of Relational

The COMPARE_AND_BRANCH opcode evaluates this here:

cpython/Python/bytecodes.c

Lines 1854 to 1857 in 7b14c2e

    
           PyObject *cond = PyObject_RichCompare(left, right, oparg>>4); 
        
           Py_DECREF(left); 
        
           Py_DECREF(right); 
        
           ERROR_IF(cond == NULL, error);

Now I think:

PyObject_RichCompare will raise an exception so it returns NULL.
Py_DECREF(left) decrefs the Symbol object x.
cond == NULL evaluates as true so we do the ERROR_IF.

I'm not sure what ERROR_IF does:

cpython/Python/bytecodes.c

Line 68 in 7b14c2e

#define ERROR_IF(cond, labelname) ((void)0)

I guess it means goto error;. I'm not sure what happens then.

2. Py_DECREF(left) decrefs the Symbol object x.

To be clear it is the Symbol object refcount that causes the crash. Using --with-pydebug and the short repro above:

$ ./python.exe t.py 
Traceback (most recent call last):
  File "/Users/enojb/current/sympy/cpython.git/t.py", line 3, in <module>
    assert x > 0
           ^^^^^
  File "/Users/enojb/current/sympy/cpython.git/sympy/core/relational.py", line 511, in __bool__
    raise TypeError("cannot determine truth value of Relational")
TypeError: cannot determine truth value of Relational
Modules/gcmodule.c:115: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x103808410
object refcount : 10
object type     : 0x7f80fd043a30
object type name: Symbol
object repr     : x

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000000101fe7000)

Current thread 0x0000000109b74600 (most recent call first):
  Garbage-collecting
  <no Python frame>
Abort trap: 6

PyObject_RichCompare will raise an exception so it returns NULL.

No it won't. It will return the GreaterThan:

cpython/Python/bytecodes.c

Lines 1854 to 1869 in 7b14c2e

    
           PyObject *cond = PyObject_RichCompare(left, right, oparg>>4); 
        
           Py_DECREF(left); 
        
           Py_DECREF(right); 
        
           ERROR_IF(cond == NULL, error); 
        
           assert(_Py_OPCODE(next_instr[1]) == POP_JUMP_IF_FALSE || 
        
                  _Py_OPCODE(next_instr[1]) == POP_JUMP_IF_TRUE); 
        
           bool jump_on_true = _Py_OPCODE(next_instr[1]) == POP_JUMP_IF_TRUE; 
        
           int offset = _Py_OPARG(next_instr[1]); 
        
           int err = PyObject_IsTrue(cond); 
        
           Py_DECREF(cond); 
        
           if (err < 0) { 
        
               goto error; 
        
           } 
        
           if (jump_on_true == (err != 0)) { 
        
               JUMPBY(offset); 
        
           }

Then PyObject_IsTrue returns -1 because GreaterThan.__bool__ raises. Then we hit the err < 0 and goto error;.

Before 7b14c2e the COMPARE_OP code did not call PyObject_IsTrue:

cpython/Python/bytecodes.c

Lines 1839 to 1854 in b1a74a1

    
           inst(COMPARE_OP, (unused/1, left, right -- res)) { 
        
               _PyCompareOpCache *cache = (_PyCompareOpCache *)next_instr; 
        
               if (ADAPTIVE_COUNTER_IS_ZERO(cache->counter)) { 
        
                   assert(cframe.use_tracing == 0); 
        
                   next_instr--; 
        
                   _Py_Specialize_CompareOp(left, right, next_instr, oparg); 
        
                   DISPATCH_SAME_OPARG(); 
        
               } 
        
               STAT_INC(COMPARE_OP, deferred); 
        
               DECREMENT_ADAPTIVE_COUNTER(cache->counter); 
        
               assert((oparg >> 4) <= Py_GE); 
        
               res = PyObject_RichCompare(left, right, oparg>>4); 
        
               Py_DECREF(left); 
        
               Py_DECREF(right); 
        
               ERROR_IF(res == NULL, error); 
        
           }

Here is a simple reproducer not involving SymPy:

class SymbolicBool:
    def __bool__(self):
        raise TypeError

class Symbol:
    def __gt__(self, other):
        return SymbolicBool()

x = Symbol()

assert x > 0

With --with-pydebug this crashes:

$ ./python.exe t2.py 
Traceback (most recent call last):
  File "/Users/enojb/current/sympy/cpython.git/t2.py", line 11, in <module>
    assert x > 0
           ^^^^^
  File "/Users/enojb/current/sympy/cpython.git/t2.py", line 3, in __bool__
    raise TypeError
TypeError
Modules/gcmodule.c:450: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x10a16bbf0
object refcount : 5
object type     : 0x109ac23e0
object type name: dict
object repr     : Segmentation fault: 11

With --with-pydebug this crashes:

In fact it crashes with an ordinary build as well including the OSX 3.12a5 build that I installed from python.org.

gh-102250: Fix double-decref in COMPARE_AND_BRANCH error case (GH-102287

@sweeneyde

Fixed in #102287 by @sweeneyde. Thanks for reporting @oscarbenjamin!

Thanks for fixing and thanks everyone for the help pinning this down!

Merge branch 'main' into inlinecomp2

Thanks again to all.

The 3.12a6 release rolled out to SymPy CI which is now all green for CPython 3.12 (although not the latest mypy release...):
https://github.com/sympy/sympy/actions/runs/4367781443

I also would like to say thanks to all involved with the faster-CPython project. I know that any significant changes will cause churn and some bugs like this one but SymPy and many other projects stand to benefit significantly from the speed improvements in CPython.

Now that 3.12 tests pass I have collected the total time taken (in CI) to run the SymPy test and doctest suite under different CPython versions:

3.8: 3439 secs
3.9: 2914 secs
3.10: 3132 secs
3.11: 3009 secs
3.12a6: 2503 secs

(That's the time for tests1+tests2+doctests in the linked CI job. There "latest" means 3.11.)

According to those timings 3.12 looks like showing the biggest effect from recent releases and the cumulative effect from 3.8 to 3.12 is a 30% reduction in runtime.

Navigation Menu

Repository navigation

Regression segfault under CPython 3.12a5 in the SymPy test suite #102250

Crash report

Error messages

Your environment

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	PyObject *cond = PyObject_RichCompare(left, right, oparg>>4);
	Py_DECREF(left);
	Py_DECREF(right);
	ERROR_IF(cond == NULL, error);

	inst(COMPARE_OP, (unused/1, left, right -- res)) {
	_PyCompareOpCache cache = (_PyCompareOpCache )next_instr;
	if (ADAPTIVE_COUNTER_IS_ZERO(cache->counter)) {
	assert(cframe.use_tracing == 0);
	next_instr--;
	_Py_Specialize_CompareOp(left, right, next_instr, oparg);
	DISPATCH_SAME_OPARG();
	}
	STAT_INC(COMPARE_OP, deferred);
	DECREMENT_ADAPTIVE_COUNTER(cache->counter);
	assert((oparg >> 4) <= Py_GE);
	res = PyObject_RichCompare(left, right, oparg>>4);
	Py_DECREF(left);
	Py_DECREF(right);
	ERROR_IF(res == NULL, error);
	}

Regression segfault under CPython 3.12a5 in the SymPy test suite #102250

Description

Crash report

Error messages

Your environment

Linked PRs

Activity

sobolevn commented on Feb 26, 2023

corona10 commented on Feb 26, 2023

OTheDev commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

OTheDev commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

OTheDev commented on Feb 26, 2023

markshannon commented on Feb 26, 2023

OTheDev commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

dtrodrigues commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

oscarbenjamin commented on Feb 26, 2023

AlexWaygood commented on Feb 27, 2023

oscarbenjamin commented on Feb 27, 2023

oscarbenjamin commented on Mar 9, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions