Repository navigation
Sphinx linkcheck and broken/redirect occurrences in Python Docs #103484
Description
Running make linkcheck in Doc folder outputs thousands of broken and redirected status. It would be nice to clean this up so we can link-check Python in CI/CD for each commit, but right now it is too polluted.
Some of these occurrences could/should be fixed in the docs itself, others can benefit from Sphinx's linkcheck configs e.g. linkcheck_ignore and linkcheck_allow_redirect.
See linkcheck-log.txt for the full log as of commit f2b7ecb.
Fun stats obtained from the above file:
- Of its 8327 lines, where 7824 are related to BPO -> GH issues. Of 7824, 5744 lines are from whatsnew/changelog and only 20 are not from whatsnew/
- 241 lines are redirection of CPython CVS URL, fixing /tree/ to /blob/ in GitHub URL
- 28 lines are broken links, where one broken link is 'https://' from an example in whatsnet/changelog
The way I see, this steps divide in:
- Clean BPO to GH redirection messages
- Fix broken links
- Clean CPython CVS URL redirections
- Clean GH issues to GH PR redirections
- Fix the rest of the occurrences.
linkcheck_allow_redirect and linkcheck_ignore can be very handy in this case. linkcheck_allow_redirect makes 'ok' status a redirect that is being spotted by linkcheck, and we have linkcheck_ignore as the last resource.
Questions I have before implementing the solution:
- Documentation hosted by Read The Docs may have language enabled so example.com is redirected to example.com/en/latest. To handle occurrences, I could add them to linkcheck_allow_redirect or we can use sphinx-ext-intersphinx to map a keyword to the documentation URL (e.g 'rtd' for read-the-docs docs). The last option allows to map to proper language of the target URL linked, similar on how Weblate did)
- Is there any restrictions to fix broken/redirect links in old whatsnew/.rst?
- Is there any restrictions to fix broken/redirect links in old whatsnew/changelog.rst (i.e. Misc/NEWS)?
- Should I create a single Pull Request for all the fixes?
Linked PRs
- GH-103484: Docs: add linkcheck allowed redirects entries for most cases #103569
- [3.11] GH-103484: Docs: add linkcheck allowed redirects entries for most cases (GH-103569) #103602
- GH-103484: Fix broken links reported by linkcheck #103608
- [3.11] GH-103484: Fix broken links reported by linkcheck (GH-103608) #103683
- GH-103484: Fix redirected permanently URLs #104001
- [3.11] GH-103484: Fix redirected permanently URLs (GH-104001) #104088
- GH-103484: Tell linkcheck to ignore debian manpage redirects #123019
- [3.12] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124136
- [3.13] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124137
- GH-103484: Fix permanently redirects reported by linkcheck #124144
- [3.13] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124151
- [3.12] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124152
- GH-103484: Fix broken links reported by linkcheck #124169
- [3.13] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124179
- [3.12] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124180
- [3.12]: gh-103484: Relink _xxsubinterpretersmodule.c in whatsnew/3.12.rst #124183
- gh-103484: Fix links reported as "broken" by linkcheck #140012
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsTodo
Activity
merwok commentedon Apr 12, 2023
What about ignoring the 7824 bpo redirects? They work fine, we control bpo and can keep the redirector working, we don’t need to change 8000 lines in historical whatsnew docs.
The CVS redirections can be fixed, as well as the broken links.
hugovk commentedon Apr 12, 2023
(For reference, some earlier discussion at https://discuss.python.org/t/sphinx-linkcheck-and-broken-redirect-occurrences-in-python-docs/25687)
Yes, let's ignore the BPO redirects. There's so many, I think either
linkcheck_allow_redirectorlinkcheck_ignore, whichever is quicker for linkcheck to process.We actually have a
linkcheck_ignorefor BPO, but the regex doesn't match, I guess it's gone out of sync:cpython/Doc/conf.py
Line 258 in 7f3c106
Please could you give some examples of these?
If there's not too many, we could add to
linkcheck_allow_redirect, similar to the devguide. There may be some other useful rules to copy over:https://github.com/python/devguide/blob/79e1a1f2961ba0882d5f0274f7c8876e36ca5dff/conf.py#L60-L107
Not as such, although effort is usually prioritised for the newer/more active pages. Cleaning them up would mean linkcheck output is more useful in the future.
...
Maybe a PR for each of these steps? And split smaller if you feel they're too big. Smaller PRs are easier to review and get merged.
rffontenelle commentedon Apr 13, 2023
Sure. The file Doc/about.rst has
:source:`Misc/ACKS`, which links to that file in CPython's source code repository. The custom role "source" is defined in Doc/tools/extensions/pyspecific.py to https://github.com/python/cpython/tree/main/ + the suffix.GitHub URL for browsing files is /blob/ and directories is /tree/. When browsing Misc/ACKS, the /tree/ redirects to /blob/.
From linkcheck stdout:
See in the above log how the redirect was from /tree/ to /blob/.
Great, I'll focus on the cleanup then.
Will do, thanks for the suggestion.
hugovk commentedon Apr 13, 2023
Thanks, we can add the tree->blob redirects to
linkcheck_allow_redirectlike: https://github.com/python/devguide/blob/main/conf.py#L73-L74merwok commentedon Apr 13, 2023
Why not do the one-line fix to pyspecific rather?
rffontenelle commentedon Apr 13, 2023
Because in
:source:`something`the URL has /tree/ if something is a directory and /blob/ if it is a file. GitHub redirects both ways. If we put /blob/ in pyspecific.py, then a:source:`Misc/`would result in /blob/ being redirected to /tree/.EDIT: See some examples:
rffontenelle commentedon Apr 13, 2023
Suggestion please?
The text talks about OpenSSL 1.1.0 and the URL for version 1.1.0 is broken (HTTP 404), but there are valid URLs for the same resource pointing to OpenSSL version 1.1.1 and to 3.1.0. Should I update the URL to either one, or use wayback machine to make sure to match the OpenSSL version?
merwok commentedon Apr 13, 2023
If openssl 1.1.1 is a bugfix release of the 1.1 branch, then it would be fine to point there.
rffontenelle commentedon Apr 15, 2023
Created the first PR. I'm planning to create other 2 PRs, with more file changes than this one.
rffontenelle commentedon Apr 16, 2023
I noticed the docs.yml workflow file runs for all branches from main to 3.7. Should I back-port the fixes to all these versions (cases were the bot won't be able to back-port automatically) or is there some code freeze that prevents it?
merwok commentedon Apr 16, 2023
Backporting doc fixes to 3.11 is fine, but older branches only accept security fixes.
rffontenelle commentedon Apr 16, 2023
Got it. It means a linkcheck job in docs.yml will require a condition check to exclude versions < 3.11. Thanks.
42 remaining items
rffontenelle commentedon Sep 17, 2024
@vstinner I was hoping to include at least another PR for "redirect with found" reported by linkcheck, (see above comment #103484 (comment)), and then we can close this. Can you please keep this open for now?
pythonGH-103484: Fix permanently redirects reported by linkcheck (pyt…
pythonGH-103484: Fix broken links reported by linkcheck (python#124169)
[3.13] GH-103484: Tell linkcheck to ignore debian manpage redirects (G…
[3.13] GH-103484: Fix broken links reported by linkcheck (GH-124169) (#…
[3.13] GH-103484: Fix permanently redirects reported by linkcheck (GH…
[3.12]: gh-103484: Relink _xxsubinterpretersmodule.c in whatsnew/3.12…
StanFromIreland commentedon Oct 12, 2025
I ran
make linkcheckon main (which took a while!), the full log: anseo!Broken cases, ignoring 403's and false positives they are all fixed by #140012:
warnings