Repository navigation
Memory leak on executables embedded with 3.13 #113055
Description
Activity
That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free.
- changed the title
[-]Memory leak on embedded Windows executable with 3.12 and later[/-][+]Memory leak on Windows executable embedded with 3.12 and later[/+]on Dec 15, 2023 I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to
67807cfcit didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.A possible way to fix this: when
free_interpreter()gets called, walk through obmalloc arenas and free them. Thepymalloc_print_stats()function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.If you free the arenas and there is allocated blocks inside of them, you can't call
pymalloc_free()on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters.Reacted by Eric SnowBuilding gh-113218 with the address sanitizer and then running
./Programs _testembed test_repeated_simple_initshows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this.Reacted by neonene and Eric Snow- changed the title
[-]Memory leak on Windows executable embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.12 and later[/+]on Dec 16, 2023 _testembed test_repeated_simple_init shows it doesn't leak anymore
I can also personally confirm that on Windows, including the 32-bit version of Python.
ea2c001 has a memory leak to be fixed, which is found at gh-113412.
I've seen a few reports about likely suspects: #110411 (comment), #113190 (comment)Your
test_repeated_simple_initcase leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_simple_initTrace log:
https://gist.github.com/nascheme/fac2dd566f09fe3d75c98134f18c6170
E.g.
==1275635== 502,840 bytes in 9,800 blocks are definitely lost in loss record 421 of 422 ==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381) ==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56) ==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917) ==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238) ==1275635== by 0x2FBAC3: unicode_decode_utf8 (unicodeobject.c:4701) ==1275635== by 0x2FC017: PyUnicode_DecodeUTF8Stateful (unicodeobject.c:4834) ==1275635== by 0x2F0E38: PyUnicode_FromString (unicodeobject.c:1893) ==1275635== by 0x3135D9: PyUnicode_InternFromString (unicodeobject.c:14943) ==1275635== by 0x289EBA: PyObject_SetAttrString (object.c:1062) ==1275635== by 0x28504F: _add_methods_to_object (moduleobject.c:174) ==1275635== by 0x2857D1: PyModule_FromDefAndSpec2 (moduleobject.c:384) ==1275635== by 0x3FB5C6: create_builtin (import.c:1396) ==1275635== by 0x4002F6: _imp_create_builtin (import.c:3400) ==1275635== by 0x284483: cfunction_vectorcall_O (methodobject.c:512) ==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273) ==1275635== by 0x21B94C: _PyObject_Call (call.c:348) ==1275635== by 0x21BA23: PyObject_Call (call.c:373) ==1275635== by 0x3856F4: _PyEval_EvalFrameDefault (generated_cases.c.h:1250) ==1275635== by 0x37E2B1: _PyEval_EvalFrame (pycore_ceval.h:115) ==1275635== by 0x3AA774: _PyEval_Vector (ceval.c:1771)And
==1275635== 339,080 bytes in 6,920 blocks are definitely lost in loss record 420 of 422 ==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381) ==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56) ==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917) ==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238) ==1275635== by 0x2F11FE: _PyUnicode_FromUCS1 (unicodeobject.c:2025) ==1275635== by 0x2F16F9: PyUnicode_FromKindAndData (unicodeobject.c:2096) ==1275635== by 0x414B4E: r_object (marshal.c:1157) ==1275635== by 0x414DC0: r_object (marshal.c:1222) ==1275635== by 0x415AC6: r_object (marshal.c:1399) ==1275635== by 0x414DC0: r_object (marshal.c:1222) ==1275635== by 0x415A8A: r_object (marshal.c:1393) ==1275635== by 0x416028: read_object (marshal.c:1524) ==1275635== by 0x416339: PyMarshal_ReadObjectFromString (marshal.c:1641) ==1275635== by 0x3FCDF0: unmarshal_frozen_code (import.c:2062) ==1275635== by 0x4008B1: _imp_get_frozen_object_impl (import.c:3575) ==1275635== by 0x3F8E6B: _imp_get_frozen_object (import.c.h:282) ==1275635== by 0x284098: cfunction_vectorcall_FASTCALL (methodobject.c:425) ==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273) ==1275635== by 0x21B94C: _PyObject_Call (call.c:348) ==1275635== by 0x21BA23: PyObject_Call (call.c:373)The following change seems to greatly reduce the leaks. The
_Py_hashtable_set(INTERNED_STRINGS, s, s)call looks like a problem to me.--- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -14870,6 +14870,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p) return; } +#if 0 /* Look in the global cache first. */ PyObject *r = (PyObject *)_Py_hashtable_get(INTERNED_STRINGS, s); if (r != NULL && r != s) { @@ -14885,6 +14886,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p) } return; } +#endif /* Look in the per-interpreter cache. */ PyObject *interned = get_interned_dict(interp);Reacted by neoneneMy memory leak seems to come from
_PyUnicode_ClearInterned()when thePy_DEBUGflag is not specified. I'm not sure of the root cause, though.
cpython/Objects/unicodeobject.c
Lines 14952 to 14973 in 0c57454
For the record, my memory gain reduced from 2M bytes to 200K bytes per cycle with gh-113412 (non-debug).
Regarding the crash after re-initialization due to the full cleanup by
_PyUnicode_ClearInterned():main
(leaks)#113218 #113412 datetime (current) crash crash crash datetime (isolated) ctypes crash crash crash _ctypes crash? crash _lsprof faulthandler tracemalloc _testcapi crash A dangling pointer in a non-isolable (small) extension could be worked around by making the interned string statically allocated, if any.
Original issue: #100911
My previous comment about disabling the
INTERNED_STRINGSglobal hash table was incorrect. I must have built with Py_DEBUG enabled and so I erroneously thought it fixed the leak. You seem to be correct about_PyUnicode_ClearInterned(). There is this comment:/* TODO: * Currently, the runtime is not able to guarantee that it can exit without * allocations that carry over to a future initialization of Python within * the same process. i.e: * ./python -X showrefcount -c 'import itertools' * [237 refs, 237 blocks] * * Therefore, this should remain disabled for until there is a strict guarantee * that no memory will be left after `Py_Finalize`.If Py_DEBUG is enabled, then that function will free interned strings with the flag
SSTATE_INTERNED_IMMORTAL. That comment was from Eddie's immortal refcnt change (gh-19474, ea2c001).3 remaining items
- changed the title
[-]Memory leak on executables embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.13[/+]on Oct 15, 2024 GH-118618 would fix the most serious memory leak in 3.12. For the last bugfix release of 3.12, @Yhg1s decided it was too large/disruptive to merge a backport and so that PR is closed. Assuming we are not changing our minds on merging that, I think the memory leak due to the obmalloc state will remain unfixed in 3.12. In theory there might be a less invasive and simpler fix for 3.12, I'd have to research that though.
It's getting confusing to keep track of these embedding bugs and what fixes have been backported. There are two classes of them: memory leaks and crashers. The crashers are the more serious ones. I believe we have fixes the known crashers (see GH-124865, GH-124536, GH-125314). For the memory leaks, the obmalloc state one is the worst. The leaks due to interned strings also depend on the mortal string change (49f6beb). Running the 3.12 head with ASAN enabled would show what's still leaking.
In the long term, i think it would be nice to have CPython running with no ASAN warnings. Then we have a build-bot to ensure we don't regress on that.
Reacted by Erlend E. Aasland and Hu Yueh-WeiFYI, 3.12.7 was not the last bugfix release of 3.12. We have scheduled bugfix releases of 3.12 until April 2025.
I cannot reproduce this issue anymore on the main branch:
diff --git a/Programs/_testembed.c b/Programs/_testembed.c index d15dd519dbf..3f9455a197f 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -38,7 +38,7 @@ char **main_argv; #define PROGRAM_NAME L"./_testembed" #define PROGRAM_NAME_UTF8 "./_testembed" -#define INIT_LOOPS 4 +#define INIT_LOOPS 100 // Ignore Py_DEPRECATED() compiler warnings: deprecated functions are // tested on purpose here. @@ -220,6 +220,10 @@ static int test_repeated_simple_init(void) _testembed_Py_Initialize(); Py_Finalize(); printf("Finalized\n"); // Give test_embed some output to check + + char cmd[1000]; + sprintf(cmd, "grep ^VmRSS /proc/%i/status", getpid()); + system(cmd); } return 0; }
Output:
--- Loop #1 --- Finalized VmRSS: 9424 kB --- Loop #2 --- Finalized VmRSS: 9932 kB --- Loop #3 --- Finalized VmRSS: 9952 kB (...) --- Loop #98 --- Finalized VmRSS: 9588 kB --- Loop #99 --- Finalized VmRSS: 9588 kB --- Loop #100 --- Finalized VmRSS: 9576 kBSince 3.12 branch is not going to be fixed, can we close the issue?
Bug report
Bug description:
Since 67807cf,
_testembed.exe(x64 Release) increases the memory usage in the cycle ofPy_InitializeandPy_Finalize.I monitored the Task Manager (
taskmgr.exe) on Windows10/11, with a change againstPrograms/_testembed.c:Commands:
_testembed.exe test_repeated_simple_initorpython -m test test_embed -m test_simple_initialization_api(no log)With RADIX TREE (as-is)
No RADIX TREE (as-is)
Recent
obmalloc.clooks ready for finalization. Just adding my rough (invalid?) experiment made the leaks even. So, I hope that they share the same issue. Otherwise, ea2c001 or 15d4c9f has another leak. #98359 (comment)patch
With RADIX TREE (patched)
No RADIX TREE (patched)
cc @ericsnowcurrently @nascheme
CPython versions tested on:
3.12, 3.13, CPython main branch: 3aea6c4
Operating systems tested on:
Windows
Linked PRs