Skip to content

Running the test suite with --junit-xml and -R incorrectly reports refleaks #83434

Closed
Closed
Running the test suite with --junit-xml and -R incorrectly reports refleaks#83434
Labels
3.11only security fixes3.12only security fixes3.13only security fixestestsTests in the Lib/test dir

Description

@pablogsal
pablogsal
opened on Jan 7, 2020
BPO 39253
Nosy @vstinner, @zooba, @pablogsal

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-07.23:54:53.841>
labels = ['tests', '3.9']
title = 'Running the test suite with --junit-xml and -R incorrectly reports refleaks'
updated_at = <Date 2020-01-08.17:31:56.794>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-01-08.17:31:56.794>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2020-01-07.23:54:53.841>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 39253
keywords = []
message_count = 9.0
messages = ['359561', '359562', '359563', '359606', '359608', '359610', '359611', '359612', '359615']
nosy_count = 3.0
nosy_names = ['vstinner', 'steve.dower', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39253'
versions = ['Python 3.9']

Linked PRs

Activity

  1. pablogsal commented on Jan 7, 2020

    @pablogsal
    pablogsal
    on Jan 7, 2020
    MemberAuthor

    ./python -m test test_list -R : --junit-xml something
    0:00:00 load avg: 0.52 Run tests sequentially
    0:00:00 load avg: 0.52 [1/1] test_list
    beginning 9 repetitions
    123456789
    .........
    test_list leaked [798, 798, 798, 798] references, sum=3192
    test_list leaked [345, 345, 345, 345] memory blocks, sum=1380
    test_list failed

    == Tests result: FAILURE ==

    1 test failed:
    test_list

    Total duration: 3.4 sec
    Tests result: FAILURE

  2. pablogsal
    added
    3.9 (EOL)end of life
    testsTests in the Lib/test dir
    on Jan 7, 2020

  3. pablogsal commented on Jan 7, 2020

    @pablogsal
    pablogsal
    on Jan 7, 2020
    MemberAuthor

    This is going to be very noisy now that we are using --junit-xml in the buildbots...

  4. vstinner commented on Jan 8, 2020

    @vstinner
    vstinner
    on Jan 8, 2020
    Member

    Oh. That explains these strange results on AMD64 RHEL7 Refleaks 3.8:

    https://buildbot.python.org/all/#/builders/219/builds/41

    argv: [b'make', b'buildbottest', b'TESTOPTS=-R 3:3 -u-cpu --junit-xml test-results.xml -j10 ${BUILDBOT_TESTOPTS}', b'TESTPYTHONOPTS=', b'TESTTIMEOUT=11700']

    ...
    test_quopri leaked [168, 168, 168] references, sum=504
    test_quopri leaked [75, 75, 75] memory blocks, sum=225
    1:50:20 load avg: 1.06 Re-running test_future5 in verbose mode
    test_future5 leaked [42, 42, 42] references, sum=126
    test_future5 leaked [20, 20, 20] memory blocks, sum=60
    ...

    The previous build is successful and it didn't use --junit-xml option:

    https://buildbot.python.org/all/#/builders/219/builds/40

  5. zooba commented on Jan 8, 2020

    @zooba
    zooba
    on Jan 8, 2020
    Member

    We can stop using --junit-xml if we want. GitHub Actions doesn't support displaying test results anyway, so once Azure Pipelines is switched off we'll just be reading the console all the time anyway (unless Buildbot has a way of rendering test results that I haven't noticed?)

  6. zooba commented on Jan 8, 2020

    @zooba
    zooba
    on Jan 8, 2020
    Member

    My guess would be that the reference tracking needs some kind of awareness for logging results. The results are collected as regular Python objects and formatted later, as we need to calculate totals before writing the XML file.

    I guess keeping those objects around looks like a leak?

  7. pablogsal commented on Jan 8, 2020

    @pablogsal
    pablogsal
    on Jan 8, 2020
    MemberAuthor

    We can stop using --junit-xml if we want. GitHub Actions doesn't support displaying test results anyway, so once Azure Pipelines is switched off we'll just be reading the console all the time anyway (unless Buildbot has a way of rendering test results that I haven't noticed?)

    We are using --junit-xml on the buildbots to analyze failures and in the future we can use it to better display the summary, as is more reliable than parsing the longs.

    I guess keeping those objects around looks like a leak?

    Yeah, because it seems that we collect those objects in the innermost runner, while the refleak runner runs the whole "test suite" (the test file, like test_list.py) as one unit.

    To be honest, the easiest fix here maybe is just deactivating the --junit and make a warning as in bpo-27103

  8. pablogsal commented on Jan 8, 2020

    @pablogsal
    pablogsal
    on Jan 8, 2020
    MemberAuthor

    deactivating the --junit

    I mean deactivating the --junit *if* is used together with -R

  9. vstinner commented on Jan 8, 2020

    @vstinner
    vstinner
    on Jan 8, 2020
    Member

    FYI test_io is skipped in Python 2.7 when using -R, because test_io is not reliable. Extract of Lib/test/test_io.py:

        if huntrleaks:
        # FIXME: bpo-31731: test_io hangs with --huntrleaks
        print("Warning: bpo-31731: test_io hangs with --huntrleaks: "
              "exclude the test")
        nottests.add('test_io')

    I'm not shocked by disabling JUnit feature when using -R. But I suggest to raise an error and tune the buildbot configuration to not use it, rather than silently ignore the option.

  10. pablogsal commented on Jan 8, 2020

    @pablogsal
    pablogsal
    on Jan 8, 2020
    MemberAuthor

    But I suggest to raise an error and tune the buildbot configuration to not use it, rather than silently ignore the option.

    I already configured the buildbots to not use --junit when running with =R

  11. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  12. furkanonder
    added
    3.11only security fixes
    on Mar 22, 2024

  13. 5 remaining items

  14. vstinner commented on Mar 25, 2024

    @vstinner
    vstinner
    on Mar 25, 2024
    Member

    The problem is that writing XML is done outside the code checking for refleaks, and creating the XML stores objects in memory. Pseudo-code:

    xml = XML()
for i in range(6):
    run_test(xml)  # add data to xml
    check_for_refleaks()
write_xml(xml)

    Maybe a solution would be to write partial XML in run_test() and then cumulate results in a final XML. Pseudo-code:

    for i in range(6):
    run_test(temp_xml_filename=f"partial{i}.xml")  # write data to disk, not to memory
    check_for_refleaks()
create_xml([f"partial{i}.xml" for i in range(6)])

    Or partial results can be Python objects written with pickle.

  15. zooba commented on Mar 25, 2024

    @zooba
    zooba
    on Mar 25, 2024
    Member

    Disabling result collection during leak tests is probably best. I don't think we collect the number of leaked objects in the results, so why would you turn it on? By mistake.

  16. vstinner
    added a commit that references this issue on Mar 25, 2024

    pythongh-83434: Disable XML in regrtest when -R option is used

    9547276

  17. vstinner commented on Mar 25, 2024

    @vstinner
    vstinner
    on Mar 25, 2024
    Member

    @zooba:

    Disabling result collection during leak tests is probably best.

    Ok, I wrote PR gh-117232 for that.

  18. vstinner
    added 3 commits that reference this issue on Mar 26, 2024

    gh-83434: Disable XML in regrtest when -R option is used (#117232)

    d52bdfb

    pythongh-83434: Disable XML in regrtest when -R option is used (pytho…

    21c0484

    pythongh-83434: Disable XML in regrtest when -R option is used (pytho…

    4612d11

  19. vstinner commented on Mar 26, 2024

    @vstinner
    vstinner
    on Mar 26, 2024
    Member

    Fixed by change d52bdfb.

  20. vstinner
    closed this as completedon Mar 26, 2024
  21. vstinner
    added a commit that references this issue on Mar 26, 2024

    [3.12] gh-83434: Sync libregrtest and test_regrtest with the main bra…

    477ef90
  22. miss-islington
    added a commit that references this issue on Mar 26, 2024

    [3.12] pythongh-83434: Sync libregrtest and test_regrtest with the ma…

    23a5deb
  23. vstinner
    added a commit that references this issue on Mar 26, 2024

    [3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the m…

    04b26ee
  24. NGRsoftlab
    added 2 commits that reference this issue on Apr 2, 2024

    Squashed commit of the following:

    1b5fe4c

    Squashed commit of the following:

    a016a54
  25. diegorusso
    added a commit that references this issue on Apr 17, 2024

    pythongh-83434: Disable XML in regrtest when -R option is used (pytho…

    e9d5ef0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixes3.12only security fixes3.13only security fixestestsTests in the Lib/test dir

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions