Repository navigation
GitHub Actions CI/CD Coverage job is broken on master #88054
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-04-30.22:41:40.416>
created_at = <Date 2021-04-19.11:50:46.444>
labels = []
title = 'GitHub Actions CI/CD `Coverage` job is broken on master'
updated_at = <Date 2021-04-30.22:41:40.415>
user = 'https://github.com/webknjaz'bugs.python.org fields:
activity = <Date 2021-04-30.22:41:40.415>
actor = 'ammar2'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-30.22:41:40.416>
closer = 'ammar2'
components = []
creation = <Date 2021-04-19.11:50:46.444>
creator = 'webknjaz'
dependencies = []
files = []
hgrepos = []
issue_num = 43888
keywords = ['patch']
message_count = 7.0
messages = ['391373', '391412', '391455', '391836', '391837', '392514', '392527']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'webknjaz', 'ammar2', 'pablogsal', 'miss-islington', 'BTaskaya']
pr_nums = ['25471', '25679']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43888'
versions = []
Activity
webknjaz commentedon Apr 19, 2021
I noticed that https://github.com/python/cpython/runs/2378199636 (a coverage job on the last commit on master at the time of writing) takes suspiciously long to complete.
I did some investigation and noticed that this job on the 3.9 branch succeeds (all of the job runs on the first page in the list are green — https://github.com/python/cpython/actions/workflows/coverage.yml?query=branch%3A3.9)
But then I took a look at the runs on master and discovered that the last successful run was 4 months ago — https://github.com/python/cpython/actions.html?query=is%3Asuccess+branch%3Amaster&workflow_file_name=coverage.yml.
The last success is https://github.com/python/cpython/actions/runs/444323166 and after that, starting with https://github.com/python/cpython/actions/runs/444405699, if fails consistently.
Notably, all of the failures are caused by the job timeout after *6 hours* — GitHub platform just kills those, 6h is a default per-job timeout in GHA.
It's also important to mention that before every job starting timing out effectively burning 6 hours of GHA time for each merge and producing no useful reports, there were occasional 6h-timeouts but they weren't consistent.
Looking into the successful runs from the past, on master and other jobs, I haven't noticed it taking more than 1h35m to complete with a successful outcome. Taking into account this as a baseline, I suggest changing the timeout of the whole job or maybe just one step that actually runs coverage.
Action items:
I'm going to send a PR addressing the first point but feel free to pick up the investigation part — I don't expect to have time for this anytime soon.
P.S. FTR the last timeout of this type happened two months ago — https://github.com/python/cpython/actions.html?page=4&query=branch%3A3.9&workflow_file_name=coverage.yml.
brettcannon commentedon Apr 19, 2021
It might be time to just kill the coverage report since people are obviously not looking at the results.
miss-islington commentedon Apr 20, 2021
New changeset 077a2e7 by Sviatoslav Sydorenko in branch 'master':
bpo-43888: Reduce coverage collection timeout to 1h40m in GHA (GH-25471)
077a2e7
isidentical commentedon Apr 25, 2021
Coverage runs are still failing on the master, and I think at least we should do something like allow failure or other wise github will send notifications for this flaky run.
ammaraskar commentedon Apr 25, 2021
For what it's worth I think Brett's suggestion of just removing the coverage build entirely is good too since it seems like no one actually looks at the results and they take up valuable CI time.
brettcannon commentedon Apr 30, 2021
New changeset 726c931 by Ammar Askar in branch 'master':
bpo-43888: Remove coverage builds from CI (GH-25679)
726c931
ammaraskar commentedon Apr 30, 2021
Coverage builds have been removed from the CI, closing this now.
feat!: remove fullcoverage, it doesn't work in 3.13
feat!: remove fullcoverage, it doesn't work in 3.13
Remove mention of fullcoverage
feat!: remove fullcoverage, it doesn't work in 3.13
Remove mention of fullcoverage (#1173)
Update dependency coverage to v7.3.2 (#347)
chore(dev): update dependency coverage to v7.3.2 (master) (#721)
chore(dev): update dependency coverage to v7.3.2 (v5) (#724)
chore(deps): update dev (#793)