Py_Finalize() doesn't clear all Python objects at exit

@amauryfa

BPO	1635741
Nosy	@amauryfa, @orsenthil, @vstinner, @tiran, @phsilva, @encukou, @ambv, @ericsnowcurrently, @serhiy-storchaka, @phmc, @DimitrisJim, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @koubaa, @h-vetinari, @csabesz88, @kumaraditya303, @CharlieZhao95
PRs	bpo-1635741: Port _json extension module to multiphase initialization(PEP 489) #17835 bpo-1635741: Remove redundant _Py_INC_TPALLOCS in _Py_NewReference() #17883 bpo-1635741: Port _abc extension module to multiphase initialization(PEP 489) #18030 bpo-1635741: Port _asyncio extension module to multiphase initialization(PEP 489) #18032 bpo-1635741: Port _bisect extension module to multiphase initialization(PEP 489) #18049 bpo-1635741: Port _bz2 extension module to multiphase initialization(PEP 489) #18050 bpo-1635741: Port _codecs extension module to multiphase initialization (PEP 489) #18065 bpo-1635741: Port _collections extension module to multiphase initialization(PEP 489) #18066 bpo-1635741: Port _locale extension module to multiphase initialization (PEP 489) #18358 bpo-1635741: Calling Py_INCREF() after PyModule_AddObject() success to run #18365 bpo-1635741: Port _contextvars extension module to multiphase initialization (PEP … #18374 bpo-1635741: Port _crypt extension module to multiphase initialization(PEP 489) #18404 bpo-1635741: Fix refleaks of timemoudle in PyInit_time() #18486 bpo-1635741: Port audioop extension module to multiphase initialization(PEP 489) #18608 bpo-1635741: Fix potential refleaks in binascii module #18613 bpo-1635741: Port _ctypes_test extension module to multiphase initialization(PEP … #19012 bpo-1635741: Port statistics module to multiphase initialization #19015 bpo-1635741: Fix possible refleaks in tupleobject.c #19018 [3.7] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19022 [3.8] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19021 bpo-1635741: Port itertools module to multiphase initialization. #19044 bpo-1635741: Port _heapq module to multiphase initialization. #19057 bpo-39337: Add a test case for normalizing of codec names #19069 [WIP] bpo-1635741: Port _collections module to multiphase initialization. #19071 bpo-1635741: Port _collections module to multiphase initialization. #19074 bpo-1635741: Port _weakref extension module to multiphase initialization(PEP 489) #19084 bpo-1635741: Enhance get_binascii_state() #19100 bpo-1635741: Port time module to multiphase initialization (PEP 489) #19107 bpo-1635741: Port _datetime module to multiphase initialization (PEP 489) #19122 Revert "bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084)" #19128 bpo-40050: Fix importlib._bootstrap_external #19135 bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084) #19140 bpo-1635741: Port operator module to multiphase initialization (PEP 489) #19150 bpo-1635741: Port _functools module to multiphase initialization (PEP-489) #19151 bpo-1635741: Port _uuid module to multiphase initialization. #19242 bpo-1635741: Port math module to multiphase initialization #19243 bpo-1635741: Port resource extension module to multiphase initialization(PEP 489) #19252 bpo-1635741: Enhance PyModule_AddIntMacro by ADD_INT macro #19307 bpo-1635741: Port _lzma module to multiphase initialization #19382 bpo-1635741: Port mmap module to multiphase initialization #19459 bpo-1635741: Port _stat module to multiphase initialization #19798 bpo-1635741: Fix compiler warning in _stat.c #19822 bpo-1635741: Port syslog module to multiphase initialization #19907 bpo-1635741: Port errno module to multiphase initialization #19923 bpo-1635741: Port fcntl module to multiphase initialization #20540 [WIP] bpo-1635741: Py_Finalize() finalizes builtin static types #20763 bpo-1635741: Port nis module to multiphase initialization. #20808 bpo-1635741: Port _dbm module to multiphase initialization #20848 bpo-1635741: Port _gdbm module to multiphase initialization #20920 bpo-1635741: Enable unicode_release_interned() without insure or valgrind. #21087 bpo-1635741: Port termios module to multiphase initialization #21168 bpo-1635741: Port sha256 module to multiphase init (PEP 489) #21189 bpo-1635741: Release Unicode interned strings at exit #21269 bpo-1635741: Fix unicode_dealloc() for mortal interned string #21270 bpo-1635741: Port faulthandler module to multiphase initialization #21294 bpo-1635741 - try to port pickle to multi-stage init #21319 bpo-1635741: port winapi to multi-stage init #21371 bpo-1635741 unicodedata #21375 bpo-1635741: port multiprocessing to multiphase init #21378 bpo-1635741: prepare for multi-phase init for unicodedata #21418 bpo-1635741: Clean sysdict and builtins of interpreter #21605 bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818 bpo-1635741: port sha3 module to multi-phase init #21855 bpo-1635741: port blake2 module to multi-phase init #21856 bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases #21896 bpo-1635741: Fix reference cycle by calling explicit gc collection in main interpreter #21902 bpo-1635741 port some simple modules to multi-phase initialization #21985 bpo-1635741 port _curses_panel to multi-phase init (PEP 489) #21986 bpo-1635741 port zlib module to multi-phase init #21995 bpo-1635741 port _testbuffer to multi-phase init #22003 bpo-1635741 port signalmodule to multi-phase init (PEP 489) #22049 bpo-1635741: port opcode module to multi-phase init (PEP 489) #22050 bpo-1635741: - port overlapped to multi-phase #22051 bpo-1635741: testbuffer heap types #22131 bpo-1635741: sha256 heap types #22134 bpo-1635741: Port the termios to multi-phase init (PEP 489) #22139 bpo-1635741 - port unicodedata to multi-phase init #22145 bpo-1635741: Port _string module to multi-phase init #22148 bpo-1635741: Port mashal module to multi-phase init #22149 bpo-1635741: port scproxy to multi-phase init #22164 bpo-1635741 port cmath to multi-phase init (pep 489) #22165 bpo-1635741 port _lsprof to multi-phase init (PEP 489) #22220 bpo-1635741: port pyexpat to multi-phase init (PEP 489) #22222 bpo-1635741: convert an _lsprof method to argument clinic #22240 gh-85956: update error in documentation #22242 bpo-1635741: replace UCD_Check with Py_IS_TYPE #22328 bpo-1635741: Port _bisect module to multi-phase init #22415 bpo-41861: Convert _sqlite3 CursorType and ConnectionType to heap types #22478 bpo-1635741: pyexpat error handling #22489 bpo-1635741: convert unicode ucd type to heap type #22490 bpo-1635741: Add a global module state to unicodedata #22712 bpo-1635741: Move _PyUnicode_Name_CAPI to the internal C API #22713 bpo-1635741: Port _csv to multi-phase init #22838 bpo-1635741: Fix NULL ptr deref in multiprocessing #22880 gh-103092: Prep curses module for multi-phase init #23091 bpo-1635741: Add PyModule_AddObjectRef() function #23122 bpo-1635741: _ast uses PyModule_AddObjectRef() #23146 bpo-1635741: _contextvars uses PyModule_AddType() #23147 bpo-1635741: _sqlite3 uses PyModule_AddObjectRef #23148 bpo-1635741: _warnings uses PyModule_AddObjectRef() #23151 bpo-1635741: sqlite3 now uses Py_NewRef and Py_XNewRef #23170 bpo-1635741: Enhance _datetime error handling #23139 bpo-1635741: port pickle to multi-phase init (PEP 489) #23188 bpo-1635741: Fix typo in PyModule_AddObjectRef() doc #23234 bpo-40170: Fix PyType_Ready() refleak on static type #23236 bpo-1635741: In pickle module, inject module state from class methods #23304 bpo-1635741: Port _hashlib to multiphase initialization (GH-23358) #23358 bpo-1635741: Port _random to multiphase initialization #23359 bpo-1635741: Port grp and pwd to multiphase initialization #23360 bpo-1635741: Port symtable module to multiphase initialization #23361 bpo-1635741: Port _queue to multiphase initialization #23376 bpo-1635741: Port gc module to multiphase initialization #23377 bpo-1635741: Convert _imp to multi-phase init #23378 bpo-1635741: Port _warnings to the multi-phase init #23379 bpo-1635741: Port _random to the multi-phase init #23381 bpo-1635741: Port spwd to multiphase initialization (GH-23390) #23390 bpo-1635741: Convert _sre types to heap types and establish module state (PEP 384) #23393 bpo-1635741: Port _struct to multiphase initialization #23398 Revert "bpo-1635741: Port _struct to multiphase initialization" #23401 bpo-1635741: Fix _struct for build bot error #23402 bpo-1635741: Port _posixshmem extension module to multiphase initialization #23404 bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406) #23406 bpo-1635741: Port select module to multiphase init (GH-23409) #23409 bpo-1635741: Port resource extension module to module state (GH-23462) #23462 bpo-42327: Add PyModule_Add() (smaller). #23443 bpo-42327: Add PyModule_Add() (smaller). #23443 bpo-1635741: port _elementtree to multi-phase init (PEP 489) #23535 Add symbols of the stable ABI to python3dll.c #23598 bpo-1635741: Refactor _threadmodule.c #23793 bpo-1635741: Port _thread to multiphase init #23811 bpo-1635741: Fix ref. leak introduced by bf64d90 #23972 bpo-1635741: Adapt _multibytecodec to multi-phase initialization #24095 bpo-1635741: Convert socket.SocketType to heap type, and establish global state #24175 bpo-1635741: Fix PyModule_AddObjectRef to use EXPORT_FUNC #24205 bpo-1635741: Port _datetime extension module to multiphase initialization (PEP 489) #30522 bpo-1635741: port _ctypes extension module to multiphase initialization #30525 bpo-46417: Call _PyDebug_PrintTotalRefs() later #30744 bpo-1635741: test_embed cheks that Python does not leak #31555 bpo-1635741: Fix winreg reference leaks #31560

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-27.13:42:43.202>
created_at = <Date 2007-01-15.10:26:05.000>
labels = ['expert-subinterpreters', 'interpreter-core', '3.11', 'performance']
title = "Py_Finalize() doesn't clear all Python objects at exit"
updated_at = <Date 2022-02-25.11:34:18.230>
user = 'https://bugs.python.org/kylotan'

bugs.python.org fields:

activity = <Date 2022-02-25.11:34:18.230>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-27.13:42:43.202>
closer = 'vstinner'
components = ['Interpreter Core', 'Subinterpreters']
creation = <Date 2007-01-15.10:26:05.000>
creator = 'kylotan'
dependencies = []
files = []
hgrepos = []
issue_num = 1635741
keywords = ['patch']
message_count = 185.0
messages = ['61054', '110895', '111024', '130729', '248761', '355187', '355189', '355191', '355193', '355194', '355201', '359342', '359482', '359830', '360063', '361427', '361428', '361466', '361798', '362068', '362124', '362143', '362144', '362195', '363935', '363936', '363937', '363940', '363941', '364234', '364330', '364379', '364463', '364521', '364609', '364656', '364833', '364836', '364845', '364871', '364883', '364906', '364909', '364951', '364952', '364968', '364971', '364972', '364973', '364975', '364987', '365008', '365017', '365043', '365386', '365388', '365484', '365584', '365611', '367686', '367796', '368096', '368318', '370561', '370763', '371119', '371569', '371683', '372061', '372077', '372096', '372102', '372103', '372797', '372912', '372938', '372942', '375169', '375272', '375286', '375287', '375288', '375290', '375311', '375567', '376218', '376221', '376240', '376288', '376360', '376461', '376474', '376477', '376480', '376493', '376500', '376552', '376554', '376555', '376556', '376566', '376567', '376571', '376572', '376616', '376687', '376993', '377253', '377364', '377365', '377523', '378115', '378685', '379280', '379287', '379667', '380216', '380270', '380280', '380317', '380318', '380331', '380332', '380340', '380346', '380351', '380733', '380758', '380854', '380857', '381124', '381300', '381317', '381318', '381351', '381352', '381376', '381379', '381391', '381392', '381395', '381396', '381400', '381411', '381423', '381426', '381431', '381432', '381434', '381476', '381477', '381571', '381572', '383155', '383168', '383276', '383283', '383285', '383287', '383454', '383636', '383785', '383799', '383801', '383844', '383847', '383880', '384039', '384042', '384326', '384352', '384646', '384978', '384987', '385010', '388728', '410809', '411116', '411278', '411311', '411872', '411961', '411997', '413938', '413979']
nosy_count = 26.0
nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'kylotan', 'vstinner', 'christian.heimes', 'tlesher', 'phsilva', 'petr.viktorin', 'ysj.ray', 'santoso.wijaya', 'lukasz.langa', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pconnell', 'isoschiz', 'Jim Fasarakis-Hilliard', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'koubaa', 'h-vetinari', 'TCsaba', 'kumaraditya', 'CharlieZhao']
pr_nums = ['17835', '17883', '18030', '18032', '18049', '18050', '18065', '18066', '18358', '18365', '18374', '18404', '18486', '18608', '18613', '19012', '19015', '19018', '19022', '19021', '19044', '19057', '19069', '19071', '19074', '19084', '19100', '19107', '19122', '19128', '19135', '19140', '19150', '19151', '19242', '19243', '19252', '19307', '19382', '19459', '19798', '19822', '19907', '19923', '20540', '20763', '20808', '20848', '20920', '21087', '21168', '21189', '21269', '21270', '21294', '21319', '21371', '21375', '21378', '21418', '21605', '21818', '21855', '21856', '21896', '21902', '21985', '21986', '21995', '22003', '22049', '22050', '22051', '22131', '22134', '22139', '22145', '22148', '22149', '22164', '22165', '22220', '22222', '22240', '22242', '22328', '22415', '22478', '22489', '22490', '22712', '22713', '22838', '22880', '23091', '23122', '23146', '23147', '23148', '23151', '23170', '23139', '23188', '23234', '23236', '23304', '23358', '23359', '23360', '23361', '23376', '23377', '23378', '23379', '23381', '23390', '23393', '23398', '23401', '23402', '23404', '23406', '23409', '23462', '23443', '23443', '23535', '23598', '23793', '23811', '23972', '24095', '24175', '24205', '30522', '30525', '30744', '31555', '31560']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1635741'
versions = ['Python 3.11']

This C code:

#include <Python.h>
int main(int argc, char *argv[])
{
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
}

Produces this output:
[7438 refs]
[7499 refs]
[7550 refs]
[7601 refs]
[7652 refs]
[7703 refs]
[7754 refs]

A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
...
[58295 refs]
[58346 refs]
[58397 refs]

This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.

Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, some objects are not cleaned in finalization.
This is not a problem in usual cases though, when the interpreter is
started only once.

Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, here is the running result on current 3.3 latest code:
[37182 refs]
[39415 refs]
[41607 refs]
[43799 refs]
[45991 refs]
[48183 refs]
[50375 refs]

This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize

Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

0:004> !heap -p -a 0x000000DB144346F0
address 000000db144346f0 found in
_HEAP @ db0cae0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
000000db14434690 030a 0000 [00] 000000db144346c0 03074 - (busy)
7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
5a582047 python36_d!new_keys_object+0x0000000000000077
5a57f7c5 python36_d!dictresize+0x0000000000000085
5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
5a57bf33 python36_d!PyDict_Update+0x0000000000000023
5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
5a778f9e python36_d!Py_Finalize+0x000000000000005e

I tested on the master branch of Python:
---

#include <Python.h>

void func()
{
    Py_Initialize(); Py_Finalize();
    Py_ssize_t cnt = _Py_GetRefTotal();
    printf("sys.gettotalrefcount(): %zd\n", cnt);
}

int main(int argc, char *argv[])
{
    Py_SetProgramName(L"./_testembed");
    for (int i=0; i < 10; i++) {
        func();
    }
}

Each iteration leaks around 5,000 Python objects:
---
sys.gettotalrefcount(): 15113
sys.gettotalrefcount(): 19527
sys.gettotalrefcount(): 23941
sys.gettotalrefcount(): 28355
sys.gettotalrefcount(): 32769
sys.gettotalrefcount(): 37183
sys.gettotalrefcount(): 41597
sys.gettotalrefcount(): 46011
sys.gettotalrefcount(): 50425
sys.gettotalrefcount(): 54839
---

I marked bpo-6741 as a duplicate of this issue.

Thanks to recent enhancements, epecially in bpo-46417, the last memory blocks are now released at Python exit! The initial issue has been fixed!!! This bug was 15 years old! Current main branch:

$ ./python -I -X showrefcount -c pass
[-5 refs, 0 blocks]

"0 blocks" means that Python no longer leaks memory at exit. You can use Valgrind to check it ;-)

The negative reference count is being discussed in bpo-46449.

I close this issue since it has an insane history: too many pull requests, changes, etc.

While the work is not 100% done (we need to convert remaining static types to heap types, and convert extensions to multi-phase init), I prefer to address remaining issues in (existing or new) separated issues. See for example the bpo-40077: "Convert static types to heap types: use PyType_FromSpec()".

I would like to thank everybody who helped to fix this issue! It's hard to list all names since this issue is a meta issues made of many sub-issues which also have sub-issues. Authors of PEP-573 and PEP-630 also helped fixing this issue!

I marked bpo-35774 as a duplicate of this issue.

The negative refcount issue has been fixed ;-)

At commit 9a24127, Python now leaks exactly 0 reference count and 0 memory block at exit:

$ ./python -I -X showrefcount -c pass
[0 refs, 0 blocks]

New changeset c9c178f by Victor Stinner in branch 'main':
bpo-1635741: test_embed cheks that Python does not leak (GH-31555)
c9c178f

New changeset 4657bf7 by Victor Stinner in branch 'main':
bpo-1635741: Fix winreg reference leaks (GH-31560)
4657bf7

I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call
Py_IsInitialized();
Py_FinalizeEx();

This issue is closed, please open a new issue.

I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call

Ah, it seems like it has been reported as a new issue: #96853

Navigation Menu

Repository navigation

Py_Finalize() doesn't clear all Python objects at exit #44470

199 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Py_Finalize() doesn't clear all Python objects at exit #44470

Description

Activity

kylotan commented on Jan 15, 2007

BreamoreBoy commented on Jul 20, 2010

amauryfa commented on Jul 21, 2010

ysjray commented on Mar 13, 2011

AlexBudovski commented on Aug 18, 2015

vstinner commented on Oct 23, 2019

vstinner commented on Oct 23, 2019

199 remaining items

vstinner commented on Jan 27, 2022

vstinner commented on Jan 28, 2022

vstinner commented on Jan 28, 2022

vstinner commented on Feb 24, 2022

vstinner commented on Feb 25, 2022

AlexSoft73 commented on Sep 19, 2022

vstinner commented on Sep 19, 2022

vstinner commented on Sep 19, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions