Repository navigation
test_peg_generator leaks references #105063
Copy link
Copy link
Open
Labels
testsTests in the Lib/test dirTests in the Lib/test dirtopic-parsertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Tried on curent main branch
Traceback
PS C:\Users\KIRILL-1\CLionProjects\cpython> ./python -m test -v test_peg_generator
Running Debug|x64 interpreter...
== CPython 3.13.0a0 (heads/main:1668b41dc4, May 29 2023, 18:02:21) [MSC v.1933 64 bit (AMD64)]
== Windows-10-10.0.19043-SP0 little-endian
== Python build: debug
== cwd: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ
== CPU count: 16
== encodings: locale=cp1251, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_peg_generator
Run: C:\Users\KIRILL-1\CLionProjects\cpython\PCbuild\amd64\python_d.exe -X dev -m venv venv
Run: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Scripts\python_d.exe -X dev -m pip install C:\U
sers\KIRILL-1\CLionProjects\cpython\Lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\w
heel-0.40.0-py3-none-any.whl
Processing c:\users\kirill-1\clionprojects\cpython\lib\test\setuptools-67.6.1-py3-none-any.whl
C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_
dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'}
return cls(files, info_location)
Processing c:\users\kirill-1\clionprojects\cpython\lib\test\wheel-0.40.0-py3-none-any.whl
Installing collected packages: wheel, setuptools
C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_
dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'}
return cls(files, info_location)
Successfully installed setuptools-67.6.1 wheel-0.40.0
C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\self_outdated_check.
py:232: DeprecationWarning: datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware obje
cts to represent datetimes in UTC: datetime.now(datetime.UTC).
current_time=datetime.datetime.utcnow(),
C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\utils\misc.py:130: D
eprecationWarning: onerror argument is deprecated, use onexc instead
shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\encodings\cp1251.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe6' in position 63: character maps to <undefined>
setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ... ERROR
test_advance_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_advance_left_recursion) ... ok
test_alternatives (test.test_peg_generator.test_first_sets.TestFirstSets.test_alternatives) ... ok
test_epsilon_production_in_start_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_epsilon_production_in_start_rule
) ... ok
test_gather (test.test_peg_generator.test_first_sets.TestFirstSets.test_gather) ... ok
test_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_left_recursion) ... ok
test_multiple_nullable_rules (test.test_peg_generator.test_first_sets.TestFirstSets.test_multiple_nullable_rules) ... ok
test_mutual_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_mutual_left_recursion) ... ok
test_nasty_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_nasty_left_recursion) ... ok
test_negative_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_negative_lookahead) ... ok
test_nullable_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_nullable_rule) ... ok
test_optional_after (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_after) ... ok
test_optional_before (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_before) ... ok
test_optional_literal (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_literal) ... ok
test_optional_operator (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_operator) ... ok
test_optionals (test.test_peg_generator.test_first_sets.TestFirstSets.test_optionals) ... ok
test_positive_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_positive_lookahead) ... ok
test_repeat_0 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0) ... ok
test_repeat_0_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0_with_group) ... ok
test_repeat_1 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1) ... ok
test_repeat_1_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1_with_group) ... ok
test_repeat_with_separator (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_with_separator) ... ok
test_rule_with_collision_after_some_other_rules (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_collision
_after_some_other_rules) ... ok
test_rule_with_no_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_no_collision) ... ok
test_rule_with_simple_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_simple_collision) ... ok
test_deep_nested_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_deep_nested_rule) ... ok
test_multiple_rules (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_multiple_rules) ... ok
test_simple_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_simple_rule) ... ok
test_advanced_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_advanced_left_recursive) ... ok
test_alt_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_alt_optional_operator) ... ok
test_bad_token_reference (test.test_peg_generator.test_pegen.TestPegen.test_bad_token_reference) ... ok
test_cut (test.test_peg_generator.test_pegen.TestPegen.test_cut) ... ok
test_dangling_reference (test.test_peg_generator.test_pegen.TestPegen.test_dangling_reference) ... ok
test_expr_grammar (test.test_peg_generator.test_pegen.TestPegen.test_expr_grammar) ... ok
test_forced (test.test_peg_generator.test_pegen.TestPegen.test_forced) ... ok
test_forced_with_group (test.test_peg_generator.test_pegen.TestPegen.test_forced_with_group) ... ok
test_gather (test.test_peg_generator.test_pegen.TestPegen.test_gather) ... ok
test_invalid_rule_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_rule_name) ... ok
test_invalid_variable_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name) ... ok
test_invalid_variable_name_in_temporal_rule (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name_in_temporal_r
ule) ... ok
test_left_recursion_too_complex (test.test_peg_generator.test_pegen.TestPegen.test_left_recursion_too_complex) ... ok
test_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_left_recursive) ... ok
test_locations_in_alt_action_and_group (test.test_peg_generator.test_pegen.TestPegen.test_locations_in_alt_action_and_group) ... o
k
test_long_rule_str (test.test_peg_generator.test_pegen.TestPegen.test_long_rule_str) ... ok
test_lookahead (test.test_peg_generator.test_pegen.TestPegen.test_lookahead) ... ok
test_missing_start (test.test_peg_generator.test_pegen.TestPegen.test_missing_start) ... ok
test_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_mutually_left_recursive) ... ok
test_named_lookahead_error (test.test_peg_generator.test_pegen.TestPegen.test_named_lookahead_error) ... ok
test_nasty_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_nasty_mutually_left_recursive) ... ok
test_nullable (test.test_peg_generator.test_pegen.TestPegen.test_nullable) ... ok
test_opt_sequence (test.test_peg_generator.test_pegen.TestPegen.test_opt_sequence) ... ok
test_optional_literal (test.test_peg_generator.test_pegen.TestPegen.test_optional_literal) ... ok
test_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_optional_operator) ... ok
test_parse_grammar (test.test_peg_generator.test_pegen.TestPegen.test_parse_grammar) ... ok
test_python_expr (test.test_peg_generator.test_pegen.TestPegen.test_python_expr) ... ok
test_repeat_0_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_complex) ... ok
test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple) ... ok
test_repeat_1_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_complex) ... ok
test_repeat_1_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_simple) ... ok
test_repeat_with_sep_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_with_sep_simple) ... ok
test_soft_keyword (test.test_peg_generator.test_pegen.TestPegen.test_soft_keyword) ... ok
test_start_leader (test.test_peg_generator.test_pegen.TestPegen.test_start_leader) ... ok
test_typed_rules (test.test_peg_generator.test_pegen.TestPegen.test_typed_rules) ... ok
test_unreachable_explicit (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_explicit) ... ok
test_unreachable_implicit1 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit1) ... ok
test_unreachable_implicit2 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit2) ... ok
test_unreachable_implicit3 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit3) ... ok
======================================================================
ERROR: setUpClass (test.test_peg_generator.test_c_parser.TestCParser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\test_peg_generator\test_c_parser.py", line 95, in setUpClass
sitepackages = subprocess.check_output(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\KIRILL-1\\CLionProjects\\cpython\\build\\test_python_11752æ\\tempcwd\\venv\\S
cripts\\python_d.exe', '-c', "import sysconfig; print(sysconfig.get_path('platlib'))"]' returned non-zero exit status 1.
----------------------------------------------------------------------
Ran 66 tests in 22.073s
FAILED (errors=1)
test test_peg_generator failed
test_peg_generator failed (1 error)
== Tests result: FAILURE ==
1 test failed:
test_peg_generator
Total duration: 22.3 sec
Tests result: FAILURELinked PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
testsTests in the Lib/test dirTests in the Lib/test dirtopic-parsertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Projects
- StatusShow more project fieldsTodo
Activity
Eclips4 commentedon May 29, 2023
seems to be introduced in #104798
cc @lysnikolaou
Eclips4 commentedon May 29, 2023
I think, this traceback more helpful:
Traceback
erlend-aasland commentedon May 29, 2023
Thanks, Lysandros is already working on this since yesterday. BTW, the failure is not Windows specific: https://buildbot.python.org/all/#/builders/320/builds/730/steps/5/logs/stdio
[-]test_peg_generator fails on Windows[/-][+]test_peg_generator leaks references[/+]erlend-aasland commentedon May 29, 2023
FTR, it is verified that #104798 introduced this.
Eclips4 commentedon May 29, 2023
Indeed.
I couldn't reproduce it on my WSL (Ubuntu 20.04), and thought that it's related to Windows system.So, thanks for this point and removing os-windows label.
UPD: I was running test as
./python -m test -v test_peg_generatoron Linux, which doesn't cause test fail.Eclips4 commentedon May 29, 2023
I'm somewhat unsure about changing the issue title. Seems there two issues:
1 - which described in this issue - test fail (Windows issue)
2 - refleak test fail, which reproducible on Linux (but on Windows we cannot get refleak test results, because of test failing)
So, seems that there two problems, which probably will be solved by one PR
erlend-aasland commentedon May 29, 2023
Enclosing my response in
<details>in order to not bring too much clutter and noise into this issue:Details
Kirill, please see CI on
main. You can see it using the GitHUb UI by clicking the "Code" tab. When in the "Code" tab, you can explore the CI run of the most recent commit:When you click the red cross or green check mark, you can inspect the CI run for the commit:
As you can see, both the 32-bit and 64-bit Windows CI are completely green, which implies there is no failure on current
main.Perhaps the failed test you are seeing is because of an issue with your local setup? Did you try a fresh build? If no, please do so. In case of test failures, you should always try
git clear -fdx(in case of in-tree builds), orrm -rf <builddir>first.However, what we know from our CI, is that there's a reference leak.
Eclips4 commentedon May 29, 2023
Thanks Erlend for your answer!
I know about it all.
After one situation with asyncio(eager-task merging) I've git clone and rebuilt interpreter every time when I find some fail.
So, there's a some proof of my words:
Details
erlend-aasland commentedon May 29, 2023
@Eclips4, try with a UTF-8 setup.
Eclips4 commentedon May 30, 2023
Seems that with UTF-8 encoding test passes. But, cp1251 is a common encoding for Windows.
10 remaining items
lysnikolaou commentedon Jun 2, 2023
@sunmy2019 Thanks a lot for spending time on this! I'll work on opening separate issues if needed and fixing those leaks, probably early next week.
erlend-aasland commentedon Jun 23, 2023
@lysnikolaou, any updates on this? Perhaps we should skip rerunning these tests for now?
lysnikolaou commentedon Jun 23, 2023
Unfortunately I didn't have the time to look into this further.
Yup, that's probably a good idea.
pythongh-105063: Disable test_peg_generator.TestCParser bco. ref leaks
gh-105063: Disable test_peg_generator.TestCParser bco. ref leaks (#10…
pythongh-105063: Disable test_peg_generator.TestCParser bco. ref leaks (
[3.12] gh-105063: Disable test_peg_generator.TestCParser bco. ref lea…
ambv commentedon Jul 5, 2023
Keeping this open until a real fix is reintroduced.
Revert "pythongh-105063: Disable test_peg_generator.TestCParser bco. …