Repository navigation
Some tests consume too much memory #133454
Description
Several years ago, all tests, except bigmem tests could be ran sequentially with just 650 MB of memory. There were issues with some buldbots (e.g. AIX) which small RAM, because excess memory consumption not always caused a quick test failure, but could cause swapping and failing tests at timeout 2 hours later. So all tests with high memory consumption were decorated with bigmemtest.
Now, some tests need more than 1 GB of memory, or even more than 2 GB. Not all buildbots have such much.
The main culprit is running too many threads. Every thread needs 35, 70 or even 90 MB, depending on test.
There are two possible solutions:
- Decrease the number of threads. Some tests can perform their function with fewer threads, but this must be decided individually so as not to make the test useless.
- Decorate the test with
bigmemtest. This will skip it on most buildbots.
Linked PRs
- gh-133454: Mark tests with many threads that use much memory as bigmem #133456
- gh-133454: Reduce the number of threads in test_racing_getbuf_and_releasebuf #133458
- [3.14] gh-133454: Mark tests with many threads that use much memory as bigmem (GH-133456) #133663
- [3.13] gh-133454: Mark tests with many threads that use much memory as bigmem (GH-133456) #133664
- gh-133454: Mark test_queue tests with many threads as bigmem #134575
- [3.14] gh-133454: Reduce the number of threads in test_racing_getbuf_and_releasebuf (GH-133458) #134589
- [3.14] gh-133454: Mark test_queue tests with many threads as bigmem (gh-134575) #135046
- [3.13] gh-133454: Mark test_queue tests with many threads as bigmem (gh-134575) #135047
Activity
pythongh-133454: Mark tests with many threads that use much memory as…
pythongh-133454: Mark tests with many threads that use much memory as…
pythongh-133454: Reduce the number of threads in test_racing_getbuf_a…
gh-133454: Mark tests with many threads that use much memory as bigmem (
pythongh-133454: Mark tests with many threads that use much memory as…
pythongh-133454: Mark tests with many threads that use much memory as…
[3.13] gh-133454: Mark tests with many threads that use much memory a…
[3.14] gh-133454: Mark tests with many threads that use much memory a…
pythongh-133454: Mark test_queue tests with many threads as bigmem
gh-133454: Reduce the number of threads in test_racing_getbuf_and_rel…
pythongh-133454: Reduce the number of threads in test_racing_getbuf_a…
[3.14] gh-133454: Reduce the number of threads in test_racing_getbuf_…
serhiy-storchaka commentedon May 23, 2025
I think it's all for now.
gh-133454: Mark test_queue tests with many threads as bigmem (gh-134575)
pythongh-133454: Mark test_queue tests with many threads as bigmem (p…
pythongh-133454: Mark test_queue tests with many threads as bigmem (p…
[3.13] gh-133454: Mark test_queue tests with many threads as bigmem (g…
[3.14] gh-133454: Mark test_queue tests with many threads as bigmem (g…
pythongh-133454: Mark tests with many threads that use much memory as…
pythongh-133454: Reduce the number of threads in test_racing_getbuf_a…
pythongh-133454: Mark test_queue tests with many threads as bigmem (p…
pythongh-133454: Mark tests with many threads that use much memory as…
pythongh-133454: Reduce the number of threads in test_racing_getbuf_a…
pythongh-133454: Mark test_queue tests with many threads as bigmem (p…
pythongh-133454: Mark test_queue tests with many threads as bigmem (p…
pythongh-133454: Reduce the number of threads in test_racing_getbuf_a…