Repository navigation
One HTML-only cron + one everything-but-HTML cron #131
Description
To help python/docsbuild-scripts#169.
Current situation
Right now the docs server is taking over 40 hours to build a full set of 3.12-3.14 docs, plus 12 translations each:
List of versions/languages
3.14/zh-tw3.14/zh-cn3.14/uk3.14/tr3.14/pt-br3.14/pl3.14/ko3.14/ja3.14/it3.14/id3.14/fr3.14/es3.14/en3.13/zh-tw3.13/zh-cn3.13/uk3.13/tr3.13/pt-br3.13/pl3.13/ko3.13/ja3.13/it3.13/id3.13/fr3.13/es3.13/en3.12/zh-tw3.12/zh-cn3.12/uk3.12/tr3.12/pt-br3.12/pl3.12/ko3.12/ja3.12/it3.12/id3.12/fr3.12/es3.12/en
Nearly all these include HTML, plain text, PDF, Texinfo and EPUB (Ukrainian is HTML only). HTML-only is fast to build, about 3-4 minutes. The full set of artifacts is much slower to build, between 40 minutes and two hours, depending on the language, and is mostly due to building latex for PDFs.
What happens is:
A cron goes off at 7 minutes past the hour and starts a new full build loop.
- If there's a build running (= lockfile found), the new one exits and allows the running one to continue.
- If there's no build running, it creates its own lockfile and starts a new build.
For each language/version, we only do a build if the docs have changed since last time, or if the translation has changed since last time. This is good, there's no point rebuilding something that hasn't changed.
However, because the full loop takes over 40 hours, inevitably there have been docs or translation changes since the last time, and we get a full rebuild each time.
This results in long delays between docs being updated, not to mention the high server resources usage.
HTML vs. PDF
We have download stats for the HTML docs, but we don't have download numbers for the other artifacts to compare.
However, I'm certain the HTML is by far the most used, and there's the most benefit to getting fresh HTML up quickly.
An affordance of websites is being able to look up just the pages you need, on-demand. Compared with PDF, where you can download it once and use it as an offline reference. Maybe you'll re-download again later, but there's less benefit in updating often, as the one you usually consult is an old, offline copy.
Proposal
I suggest we have two cron jobs:
-
The current hourly job only builds HTML.
-
A new job builds everything else except HTML.
1. HTML only
When there are new changes, they will be built and uploaded much sooner. It will run much quicker.
It's more likely that on the next pass, some languages can be skipped because there's nothing to update this time round.
2. Everything but HTML
This will be much slower than the HTML-only job, and will take about the same as the current loop does now.
Maybe it'll be a bit quicker due to not needing to build HTML, but maybe a bit slower because we'll sometimes be using CPU to build HTML at the same time. However, the majority of the time is spent running a latex command on a single CPU, so it might not make much difference.
We also don't need to update the non-HTML as often, so its cron could be every few days?
Activity
hugovk commentedon Sep 12, 2024
TODO
If we do this,
build_docs.pyalready has--quickto build HTML-only:https://github.com/python/docsbuild-scripts/blob/56d72d43e5759cc0ed600827b56e81d8310bcaca/build_docs.py#L525-L530
make dist-no-htmlcpython#124383)--select-outputdocsbuild-scripts#199)cron.presententry: (Restore the HTML docsbuild cron job psf-salt#497)Anything else?
hugovk commentedon Sep 23, 2024
It's been mentioned previously, one downside of this sort of approach is:
https://docs.python.org/x.y/archives/python-x.y.z-docs-pdf-a4.zipThis has been partially addressed by adding a 404 that says "The archive you're trying to download has not been built yet. Please try again later or consult the archives for earlier versions."
Perhaps we could also mitigate this by renaming the files so they only have x.y in the filename and not x.y.z?
So
https://docs.python.org/x.y/archives/python-x.y-docs-pdf-a4.zipinstead.For example, on the day we release 3.13.0, instead of getting a 404 page for the 3.13.0 PDF, they get the 3.13.0rc2 PDF.
I think this is fine. There's usually not much that's changed, and the benefit is everyone gets the HTML and PDF files sooner (all the time, not just for releases).
AA-Turner commentedon Sep 23, 2024
I seem to remember that the release process includes building the docs + PDF etc, or perhaps it used to. If this is still the case, can part of the release process be to upload a copy of that archive to the docs server as well as the release server?
@ned-deily did this I think for rc2? Is this something we can formalise?
A
ned-deily commentedon Sep 23, 2024
This subject is confusingly complex for various reasons (some due to attempts to provide compatibility with older end-of-life releases) so there's a good chance that some or all of what follows is wrong but, to the best of my knowledge, it works today like this.
The release process currently does produce a quick build of the untranslated docs (html and PDF et als) for all rc and final releases (but not alpha and beta releases) built from the source release git tag (i.e.
v3.13.0rc2). As mostly discussed in PEP 101, the release process saves the unpacked html files of the docset for every release on the doc server (under/srv/docs.python.org/release/and saves the downloadable files to the download server (under/srv/www.python.org/ftp/python/doc/). The archived html is linked to from the Python Documentation by Version web page and, importantly, the links in theirDownload these documentspages, for example Download Python 3.12.6 Documentation, link to these archived downloadable files. Links to these archived versions of the docs may also be used by the RM on the individual release pages, often for the link to the changelog. This is all completely independent of the cron doc builds under discussion here.At the same time, the cron jobs do their things and (try to) produce their daily/3-hour builds on the docs server that are served under various python.org URLs. The actual file names of the download files do include the version number but are served under the branch-specific directories (
/srv/docs.python.org/x.y/archives/) resulting in the URLs mentioned earlier (likehttps://docs.python.org/x.y/archives/python-x.y.z-docs-pdf-a4.zip). But this does result in a curious set of files in thesex.y/archivesdirectories because the file name changes following the first cron run after the release manager merges the release engineering branch back into the main repo. The effect is that, at some somewhat random time following a release, the download files built by the cron jobs will switch from...-x.y.z-...to...-x.y.z+1..file names (or even to...-x.y+1.0a0...) in the same directory and the older files are never deleted even though they imprecisely imply their source, i.e.3.12/archives/python-3.12.5-docs-pdf-a4.zipneither reflects the state of the documentation at the time of the 3.12.5 release nor necessarily the final state of the 3.12 docs prior to the 3.12.6 release; it's just what existed in the 3.12 git branch the last time the cron started building (and successfully finished) before the 3.12.6 release was merged into the 3.12 branch.A partial look at a docs server branch directory shows this:
AFAICT, there is no reason to be keeping the older files in this directory and this would be solved/mitigated, along with the sync problem alluded to above, if the download file names were changed as suggested above by @hugovk. The trick, of course, is to eliminate or minimize any compatibility issues with user expectations and with the separate release archives produced by the RMs.
To the other point:
I believe that what I did for 3.13.0rc2 was to add a temporary link on the Python Documentation by Version web page to the rc2 documentation produced by the release process; normally that page does not include links to pre-release versions. Maybe there was something else on the release page, too.
AA-Turner commentedon Sep 25, 2024
See python/cpython#124489 to alter the build process.
We'll need to consider what to do with the existing
/archives/files. It likely doesn't hurt to keep them, even though they are in effect random snapshots of the documentation at some point in time. If we do remove them, we can create redirects in the docs server configuration so that links don't break.A
AA-Turner commentedon Sep 25, 2024
In terms of splitting the build into HTML and non-HTML, we have a triumvirate of patches:
--select-outputdocsbuild-scripts#199 to allow choosing between HTML or non-HTML outputs inbuild_docs.pyautobuildtargets that docsbuild-scripts uses.The first can be merged and the builds will carry on as they are now without any change. The second will start building HTML files twice and could potentially overwite itself when copying to
/srv/...if a full job and HTML-only job finish at the same time. The CPython change should be merged third, and fixes this by splitting the non-HTML and HTML-only builds into disjoint sets.A
ned-deily commentedon Sep 25, 2024
Thanks, that looks good to me. The only potential issue I can think of is that there might be users/scripts out there that might be periodically expecting to directly download the current built artifacts using the old URL formats. I would guess that is not common and I don't think we've ever provided any guarantees about the URLs other than linking through the
download.htmlpages. ...... And the links that would break, besides the above-mentioned possible scripts, would be from previously downloaded copies of the html-format documentation (the only artifact where the download links appear?) and from embedded copies of the HTML documentation that are provided, for example, in the python.org macOS installer. Others? So, if we did add redirects from the x.y.z to the new x.y URLs for releases where we apply this change (and are still building docs), that should solve the problem for all of those cases, I think. If it would make the redirecting easier, we could perhaps do a one-time create x.y symlinks for EOL releases.
AA-Turner commentedon Sep 25, 2024
If we want to add redirects, I've opened python/psf-salt#498 as a draft.
A
ned-deily commentedon Sep 25, 2024
I think this brings up another related issue inspired by the above discussion and a comment in the PR:
That is, like the file names and corresponding URLs, the Python version displayed in the daily document HTML and downloads is also imprecise and potentially misleading. The daily builds currently show the
x.y.zversion (i.e. 3.12.6) but the actual documentation reflects the current state of the branch (3.12, say) which is likely newer than the last release (3.12.6). If one builds Python from the current head of the branch, the version insys.versionand thus displayed in the REPL is the last release with a '+' appended, i.e.3.12.6+. This comes from the#define PY_VERSIONstring in Include/patchlevel.h. The+is automatically added as one of the final steps of the release process when the release manager triggers the release engineering branch to be merged into the cpython release branch. We could consider using that more precise version for the daily builds. I thought we did do that at some time in the past but I may be misremembering (I know RMs used to do it manually for theREADMEfile in the repo itself).AA-Turner commentedon Sep 25, 2024
An alternative is to go the other way with less precision, and advertise the daily downloads page as for "Python 3.12" (helpfully this is also easier to achieve). The static
/release/versions would keep the full release (e.g. https://docs.python.org/release/3.12.0rc3/download.html).A
merwok commentedon Oct 1, 2024
To be interpreted as «Python 3.12 as that branch looks today» ?
… seems good!
AA-Turner commentedon Oct 2, 2024
On Monday (30 September 2024) we split the server into a HTML-only and non-HTML cron task, the formed scheduled hourly and the latter daily. After some initial teething problems, we've had a successful full rebuild of the non-HTML job, hence this note.
First, two tables of statistics with build times and durations:
Build times (HTML only)
Build times (no HTML)
Taking the most recent 16 rebuilds for HTML-only:
These have an average (mean) time of 8344s, or 2h 19m 4s. Excluding the builds for which no work was done (5), we have 11 builds with an average of 3h 32m 3s.
These numbers are significantly skewed by the Chinese languages, which take more than an hour each. Excluding the Chinese, we have 109 HTML-only builds at an average (mean) time of 4m 15s.
We haven't yet observed a full rebuild for all versions and languages, which is to be expected as a benefit of splitting the workers is that the HTML job will have no work to do more frequently. The expected time for a full rebuild of all 13 languages and 3 versions is (11 x 4m15s + 2 x 66m) x 3, or just under nine hours (8:56:15).
This is a significant improvement to the status quo ante (c. 30 hours per rebuild), and will further improve dramatically when we resolve the issue with Chinese builds.
The Non-HTML archive builds are currently scheduled daily, and a basic projection estimates that a full rebuild of 12 languages x 3 versions would take just under 19 hours, so we have headroom here (though faster is of course better!).
Thank you to everyone involved in making this work happen, I'll now close this issue.
A
hugovk commentedon Oct 2, 2024
And thank you @AA-Turner for all your work here!
Another idea to consider: a third cron for only the English HTML for the default
/3version. It takes about 3 minutes. We know from Plausible stats (numbers below from the July 2023 trial) that this one is served an order of magnitude more than the others.But I agree, let's first try and figure out why the Chinese builds are so slow.
By language or version:
By language and version: