test_lzma: test_refleaks_in_decompressor___init__() leaks 100 handles on Windows

@pfmoore

BPO	33916
Nosy	@pfmoore, @vstinner, @tjguk, @methane, @zware, @serhiy-storchaka, @zooba, @ZackerySpytz, @miss-islington
PRs	gh-67413: Fix segfaults and multiple leaks in the lzma and bz2 modules #7822 bpo-33916: Fix bz2 and lzma init when called twice #7843 [3.7] bpo-33916: Fix bz2 and lzma init when called twice (GH-7843) #7871 [3.6] bpo-33916: Fix bz2 and lzma init when called twice (GH-7843) #7872

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-23.13:07:46.389>
created_at = <Date 2018-06-20.15:59:40.375>
labels = ['3.8', '3.7', 'tests', 'OS-windows', 'performance']
title = 'test_lzma: test_refleaks_in_decompressor___init__() leaks 100 handles on Windows'
updated_at = <Date 2018-06-23.13:07:46.388>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-06-23.13:07:46.388>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-23.13:07:46.389>
closer = 'vstinner'
components = ['Tests', 'Windows']
creation = <Date 2018-06-20.15:59:40.375>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33916
keywords = ['patch']
message_count = 10.0
messages = ['320087', '320091', '320093', '320094', '320095', '320096', '320300', '320301', '320306', '320307']
nosy_count = 9.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'methane', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'ZackerySpytz', 'miss-islington']
pr_nums = ['7822', '7843', '7871', '7872']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue33916'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

Using my PR 7827, I saw that test_refleaks_in_decompressor___init__() of test_lzma leaks 100 handles on Windows.

Extract of the code:

    @support.refcount_test
    def test_refleaks_in_decompressor___init__(self):
        gettotalrefcount = support.get_attribute(sys, 'gettotalrefcount')
        lzd = LZMADecompressor()
        refs_before = gettotalrefcount()
        for i in range(100):
            lzd.__init__()
        self.assertAlmostEqual(gettotalrefcount() - refs_before, 0, delta=10)

The test comes from bpo-31787: commit d019bc8.

In _lzmamodule.c

self->lock = PyThread_allocate_lock();

This seems leak some resource. But I'm not sure this lock contains refcnt on Windows.

Have a look at bpo-23224 and PR 7822.

This seems leak some resource. But I'm not sure this lock contains refcnt on Windows.

PR 7827 checks for leaks of Windows handles, this issue is about Windows handles, not Python reference leaks. But a Windows lock may use a Windows handle internally, I don't know :-)

It seems like test_bz2 has the same issue, but I'm not sure:
#7827 (comment)

I four times tried to update this issue, and only the forth one was successful. Other attempts was failed because "Edit Error: someone else has edited this issue (nosy, components, versions)." This is an extreme case of race condition in real life.

New changeset 9b7cf75 by Victor Stinner in branch 'master':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843)
9b7cf75

New changeset efc6bf6 by Miss Islington (bot) in branch '3.7':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843)
efc6bf6

New changeset 1094891 by Victor Stinner in branch '3.6':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843) (GH-7872)
1094891

It seems like Python 2.7 is inaffected by the bug:

static int
BZ2File_init(BZ2FileObject *self, PyObject *args, PyObject *kwargs)
{
    ...
#ifdef WITH_THREAD
    if (!self->lock) {
        self->lock = PyThread_allocate_lock();
    }

    if (!self->lock) {
        PyErr_SetString(PyExc_MemoryError, "unable to allocate lock");
        goto error;
    }
#endif
    ...
}

I fixed the leak in Python 3.6, 3.7 and master. I close the issue.

Navigation Menu

Repository navigation

test_lzma: test_refleaks_in_decompressor_init() leaks 100 handles on Windows #78097

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_lzma: test_refleaks_in_decompressor___init__() leaks 100 handles on Windows #78097

Description

Activity

vstinner commented on Jun 20, 2018

methane commented on Jun 20, 2018

ZackerySpytz commented on Jun 20, 2018

vstinner commented on Jun 20, 2018

vstinner commented on Jun 20, 2018

serhiy-storchaka commented on Jun 20, 2018

vstinner commented on Jun 23, 2018

miss-islington commented on Jun 23, 2018

vstinner commented on Jun 23, 2018

vstinner commented on Jun 23, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

test_lzma: test_refleaks_in_decompressor_init() leaks 100 handles on Windows #78097