Skip to content

test_unicode always fails during a PGO-optimised build (because it does not exist) #110276

Closed
Closed
test_unicode always fails during a PGO-optimised build (because it does not exist)#110276
Assignees
AlexWaygood
Labels
buildThe build process and cross-buildtestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@AlexWaygood
AlexWaygood
opened on Oct 3, 2023

Bug report

Bug description:

As part of a PGO-optimised build, a selection of tests are run in order for the PGO build to take place (if I understand correctly, these tests are used to gather data on which functions in Python are performance-critical and should be optimised). Exactly which tests are run is hardcoded here:

cpython/Lib/test/libregrtest/pgo.py

Lines 1 to 51 in bb2e96f

# Set of tests run by default if --pgo is specified. The tests below were
# chosen based on the following criteria: either they exercise a commonly used
# C extension module or type, or they run some relatively typical Python code.
# Long running tests should be avoided because the PGO instrumented executable
# runs slowly.
PGO_TESTS = [
'test_array',
'test_base64',
'test_binascii',
'test_binop',
'test_bisect',
'test_bytes',
'test_bz2',
'test_cmath',
'test_codecs',
'test_collections',
'test_complex',
'test_dataclasses',
'test_datetime',
'test_decimal',
'test_difflib',
'test_embed',
'test_float',
'test_fstring',
'test_functools',
'test_generators',
'test_hashlib',
'test_heapq',
'test_int',
'test_itertools',
'test_json',
'test_long',
'test_lzma',
'test_math',
'test_memoryview',
'test_operator',
'test_ordered_dict',
'test_patma',
'test_pickle',
'test_pprint',
'test_re',
'test_set',
'test_sqlite3',
'test_statistics',
'test_struct',
'test_tabnanny',
'test_time',
'test_unicode',
'test_xml_etree',
'test_xml_etree_c',
]

test_unicode is specified as one of the tests to be run as part of this selection. However, test_unicode always fails during a PGO build. Why? Because it was renamed as test_str in #13172 (by @asqui). This hasn't been noticed until now because the PGO build is still marked as a "success" even if some tests fail during the analysis phase.

The fix for the immediate issue is simple: don't try to run test_unicode as part of the PGO-optimised build (it doesn't exist anymore); run test_str instead. Longer term, though, we might want to look at making the PGO build fail if it tries to run nonexistent tests. It's probably correct for the build to continue even if some tests fail during the analysis phase, but perhaps we could make an exception for this specific kind of failure (attempting to run nonexistent tests).

Cc. @hugovk as the primary reviewer for #13172, and @vstinner as a libregrtest expert.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

Activity

  1. AlexWaygood
    added
    type-bugAn unexpected behavior, bug, or error
    testsTests in the Lib/test dir
    buildThe build process and cross-build
    on Oct 3, 2023
  2. AlexWaygood
    self-assigned this
    on Oct 3, 2023
  3. AlexWaygood
    added 2 commits that reference this issue on Oct 3, 2023

    pythongh-110276: Run `test_str`, not `test_unicode`, as part of the P…

    3b61502

    gh-110276: Run `test_str`, not `test_unicode`, as part of the PGO bui…

    dddc757

  4. vstinner commented on Oct 3, 2023

    @vstinner
    vstinner
    on Oct 3, 2023
    Member

    I don't understand this issue. We have many PGO buildbots.

    How is it possible that a test was removed and nothing broke?

    If I replace PGO_TESTS with PGO_TESTS=['test_xxx'], tests fail as expected:

    $ ./python -m test --pgo; echo $?
0:00:00 load avg: 1.80 Run 1 test sequentially
0:00:00 load avg: 1.80 [1/1] test_xxx
test_xxx failed (uncaught exception)

Total duration: 54 ms
Total tests: run=0
Total test files: run=1/1 failed=1
Result: FAILURE
2

    Maybe the problem is that Makefile.pre.in ignores failures :-(

            $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
(...)
        # Run instrumented binaries to collect data.
        $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true

    @corona10: Any idea why we ignore failures?

  5. AlexWaygood commented on Oct 3, 2023

    @AlexWaygood
    AlexWaygood
    on Oct 3, 2023
    MemberAuthor

    FWIW, the failure wasn't "just on my machine": you can see the "test_unicode failed (uncaught exception)" message in the logs for our PGO buildbots too. Example: https://buildbot.python.org/all/#/builders/249/builds/6518/steps/3/logs/stdio.

    Maybe the problem is that Makefile.pre.in ignores failures :-(

    Hmm, I'm on Windows, and the test failure doesn't cause a PGO build (which doesn't use the Makefile on Windows) to fail on my machine either. The test fails as part of the build, the failure is reported and logged to the terminal, but the build then continues and is marked as a success.

  6. vstinner commented on Oct 3, 2023

    @vstinner
    vstinner
    on Oct 3, 2023
    Member

    FWIW, the failure wasn't "just on my machine": you can see the "test_unicode failed (uncaught exception)" message in the logs for our PGO buildbots too.

    A log is not really what I expect when I say "failure". I expected a red buildbot :-) Otherwise, it's too easy to miss.

  7. AlexWaygood commented on Oct 3, 2023

    @AlexWaygood
    AlexWaygood
    on Oct 3, 2023
    MemberAuthor

    FWIW, the failure wasn't "just on my machine": you can see the "test_unicode failed (uncaught exception)" message in the logs for our PGO buildbots too.

    A log is not really what I expect when I say "failure". I expected a red buildbot :-) Otherwise, it's too easy to miss.

    Oh, we agree! There's definitely a problem here that this issue went unseen for so long! :D

    This is the reason why I haven't closed the issue yet, even though #110277 has now been merged, fixing the immediate problem.

  8. vstinner
    added 2 commits that reference this issue on Oct 3, 2023

    pythongh-110276: No longer ignore PROFILE_TASK fail silently

    dcc36c1

    pythongh-110276: No longer ignore PROFILE_TASK failure silently

    a26ba58

  9. vstinner commented on Oct 3, 2023

    @vstinner
    vstinner
    on Oct 3, 2023
    Member

    I wrote PR #110295 to no longer ignore PROFILE_TASK failure silently.

  10. vstinner
    added a commit that references this issue on Oct 3, 2023

    gh-110276: No longer ignore PROFILE_TASK failure silently (#110295)

    6ab6040

  11. vstinner commented on Oct 3, 2023

    @vstinner
    vstinner
    on Oct 3, 2023
    Member

    My commit 6ab6040 no longer ignores PROFILE_TASK failure silently.

  12. AlexWaygood commented on Oct 3, 2023

    @AlexWaygood
    AlexWaygood
    on Oct 3, 2023
    MemberAuthor

    My commit 6ab6040 no longer ignores PROFILE_TASK failure silently.

    It sounds like there is little appetite among the core devs who maintain the Windows build for an equivalent PCBuild fix, so I'll close this out now. Thanks @vstinner!

  13. AlexWaygood
    closed this as completedon Oct 3, 2023

  14. vstinner commented on Oct 4, 2023

    @vstinner
    vstinner
    on Oct 4, 2023
    Member

    PGO builds are tested daily on Unix by buildbots. So now we should get any similar issue get reported way earlier.

  15. vstinner
    added a commit that references this issue on Nov 10, 2023

    pythongh-110276: No longer ignore PROFILE_TASK failure silently (pyth…

    6911381
  16. Glyphack
    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-110276: Run `test_str`, not `test_unicode`, as part of the P…

    b417e23

    pythongh-110276: No longer ignore PROFILE_TASK failure silently (pyth…

    0c1a713
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

buildThe build process and cross-buildtestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions