Repository navigation
Failure affecting 2 French sites: "Latexmk: Errors, so I did not complete making targets" #187
Description
Follow on from #186, the other 3 "Badly handled exception, human, please help" lines in yesterday's 24-hour log are caused by:
2024-08-11 15:12:53,206 DEBUG fr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev"
2024-08-11 15:44:37,172 ERROR fr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev" KO:
Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex': Reasons for rerun
Changed files or newly in use/created:
library.aux
library.ind
library.toc
Latexmk: Summary of warnings from last run of *latex:
Latex found 2 multiply defined reference(s)
=====Latex reported missing or unavailable character(s).
=====See log file for details.
Latexmk: Errors, so I did not complete making targets
make[2]: *** [Makefile:29: library.pdf] Error 12
make[2]: Leaving directory '/srv/docsbuild/cpython/Doc/build/latex'
make[1]: *** [Makefile:216: dist] Error 2
make[1]: Leaving directory '/srv/docsbuild/cpython/Doc'
make: *** [Makefile:274: autobuild-dev] Error 2
make: Leaving directory '/srv/docsbuild/cpython/Doc'
2024-08-11 15:44:37,176 ERROR fr/3.13: Badly handled exception, human, please help.
Traceback (most recent call last):
File "/srv/docsbuild/scripts/build_docs.py", line 655, in run
self.build()
File "/srv/docsbuild/scripts/build_docs.py", line 762, in build
run(
File "/srv/docsbuild/scripts/build_docs.py", line 247, in run
result.check_returncode()
File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['make', '-C', '/srv/docsbuild/cpython/Doc', 'PYTHON=/srv/docsbuild/venv-3.13/bin/python', 'SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build', 'BLURB=/srv/docsbuild/venv-3.13/bin/blurb', 'VENVDIR=/srv/docsbuild/venv-3.13', 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0', 'SPHINXERRORHANDLING=', 'autobuild-dev']' returned non-zero exit status 2.
2024-08-11 15:44:37,178 DEBUG fr/3.13: Resetting dropped connection: sentry.ioIt's happening for three French builds, the 3.13 and 3.14 builds:
2024-08-11 00:09:54,728 ERROR fr/3.14: Badly handled exception, human, please help.
2024-08-11 15:44:37,176 ERROR fr/3.13: Badly handled exception, human, please help.
2024-08-11 23:54:51,221 ERROR fr/3.14: Badly handled exception, human, please help.
https://docs.python.org/fr/3.13/ and https://docs.python.org/fr/3.14/ were last updated on 8th August. fr/3.12 is building fine.
Activity
hugovk commentedon Aug 12, 2024
A fr/3.13 build happened to be running now and there's this matching the warnings above:
I'm not sure if they're the errors causing it to fail, but are a good place to start.
"multiply defined" means "defined multiple times".
AA-Turner commentedon Aug 12, 2024
This also seems suspicious? Is there more detail in the log file?
A
hugovk commentedon Aug 12, 2024
Here's the whole library.log, assuming it's referring to this one.
This is the contents of the
build/latexI managed to copy before the next build cleared the directory.library.*do appear to be the last ones processed/created, but I don't know if anything was before the next build:[-]Failure affecting 2 sites: "Latexmk: Errors, so I did not complete making targets"[/-][+]Failure affecting 2 French sites: "Latexmk: Errors, so I did not complete making targets"[/+]picnixz commentedon Sep 11, 2024
Not sure if it's you or GitHub but the logs seem truncated. Can you confirm that there are 5049 lines?
By the way, I don't think the issue is with the labels being redefined. Normally, labels being redefined are only warnings and shouldn't affect the build.
hugovk commentedon Sep 11, 2024
Yes, this is how long the copy is on the server:
hugovk commentedon Sep 17, 2024
fr/3.13 now has another error:
'pdflatex' needed too many passes2024-09-17 15:17:53,682 ERROR fr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev" KO: 'pdflatex' needed too many passes ---------------------- Latexmk: Sometimes, the -f option can be used to get latexmk to try to force complete processing. But normally, you will need to correct the file(s) that caused the error, and then rerun latexmk. In some cases, it is best to clean out generated files before rerunning latexmk after you've corrected the files. Latexmk: Summary of warnings from last run of *latex: Latex found 2 multiply defined reference(s) =====Latex reported missing or unavailable character(s). =====See log file for details. make[2]: *** [Makefile:29: library.pdf] Error 12 make[2]: Leaving directory '/srv/docsbuild/cpython/Doc/build/latex' make[1]: *** [Makefile:226: dist] Error 2 make[1]: Leaving directory '/srv/docsbuild/cpython/Doc' make: *** [Makefile:289: autobuild-dev] Error 2 make: Leaving directory '/srv/docsbuild/cpython/Doc' 2024-09-17 15:17:53,687 ERROR fr/3.13: Badly handled exception, human, please help. Traceback (most recent call last): File "/srv/docsbuild/scripts/build_docs.py", line 655, in run self.copy_build_to_webroot(http) File "/srv/docsbuild/scripts/build_docs.py", line 762, in build run( File "/srv/docsbuild/scripts/build_docs.py", line 247, in run File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode raise CalledProcessError(self.returncode, self.args, self.stdout, subprocess.CalledProcessError: Command '['make', '-C', '/srv/docsbuild/cpython/Doc', 'PYTHON=/srv/docsbuild/venv-3.13/bin/python', 'SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build', 'BLURB=/srv/docsbuild/venv-3.13/bin/blurb', 'VENVDIR=/srv/docsbuild/venv-3.13', 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0', 'SPHINXERRORHANDLING=', 'autobuild-dev']' returned non-zero exit status 2.However, https://docs.python.org/fr/3.14/ has now recovered: "Mis à jour le Sep 17, 2024 (10:36 UTC)."
Its last error was also
'pdflatex' needed too many passeson 2024-09-12 14:04.Its first successful build was 2024-09-13 23:29.
(fr/3.12 is still building fine.)
picnixz commentedon Sep 19, 2024
Some hypotheses:
Now, do the logs have some "LaTeX Error" message? Have you tried using xelatex instead of pdflatex? (why is it saying pdflatex but you are specifying xelatex as the engine????) because I think that the regex section requires xelatex because of a special character somewhere in the document.
hugovk commentedon Sep 19, 2024
No "LaTeX Error" in the logs. Here's the whole fr/3.13 part:
2024-09-18 16:19:13,943 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/cpython fetch' 2024-09-18 16:19:14,144 INFO fr/3.13: Running. 2024-09-18 16:19:14,144 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/cpython show-ref -s origin/3.13' 2024-09-18 16:19:14,148 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/cpython reset --hard 112b1704fa6165ce6386ce2d9363fb73dd799de5 --' 2024-09-18 16:19:14,174 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/cpython clean -dfqx' 2024-09-18 16:19:14,188 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES fetch' 2024-09-18 16:19:14,506 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES branch -r' 2024-09-18 16:19:14,511 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES show-ref -s origin/3.11' 2024-09-18 16:19:14,514 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES reset --hard 2d42a3c0d181f19ab1bd62400bfafc9a8304a4b1 --' 2024-09-18 16:19:14,568 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES clean -dfqx' 2024-09-18 16:19:14,726 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/cpython rev-parse HEAD' 2024-09-18 16:19:14,729 DEBUG fr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/fr/LC_MESSAGES rev-parse HEAD' 2024-09-18 16:19:14,732 INFO fr/3.13: Should rebuild: new translations (from ceaeb373626e282ba17220752aa2c522fd046891 to 2d42a3c0d181f19ab1bd62400bfafc9a8304a4b1) 2024-09-18 16:19:14,732 DEBUG fr/3.13: Run: '/srv/docsbuild/venv/bin/python -m venv /srv/docsbuild/venv-3.13' 2024-09-18 16:19:17,377 DEBUG fr/3.13: Run: '/srv/docsbuild/venv-3.13/bin/python -m pip install --upgrade --upgrade-strategy=eager python-docs-theme jieba -rrequirements.txt' 2024-09-18 16:19:20,902 DEBUG fr/3.13: Run: '/srv/docsbuild/venv-3.13/bin/python -m pip freeze --all' 2024-09-18 16:19:21,494 INFO fr/3.13: Build start. 2024-09-18 16:19:21,494 INFO fr/3.13: Running make autobuild-dev 2024-09-18 16:19:21,494 DEBUG fr/3.13: Run: "sed -i 's/ *-A switchers=1//' /srv/docsbuild/cpython/Doc/Makefile" 2024-09-18 16:19:21,502 DEBUG fr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev" 2024-09-18 16:49:43,031 ERROR fr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev" KO: 'pdflatex' needed too many passes ---------------------- Latexmk: Sometimes, the -f option can be used to get latexmk to try to force complete processing. But normally, you will need to correct the file(s) that caused the error, and then rerun latexmk. In some cases, it is best to clean out generated files before rerunning latexmk after you've corrected the files. Latexmk: Summary of warnings from last run of *latex: Latex found 2 multiply defined reference(s) =====Latex reported missing or unavailable character(s). =====See log file for details. make[2]: *** [Makefile:29: library.pdf] Error 12 make[2]: Leaving directory '/srv/docsbuild/cpython/Doc/build/latex' make[1]: *** [Makefile:226: dist] Error 2 make[1]: Leaving directory '/srv/docsbuild/cpython/Doc' make: *** [Makefile:289: autobuild-dev] Error 2 make: Leaving directory '/srv/docsbuild/cpython/Doc' 2024-09-18 16:49:43,116 ERROR fr/3.13: Badly handled exception, human, please help. Traceback (most recent call last): File "/srv/docsbuild/scripts/build_docs.py", line 654, in run self.build() File "/srv/docsbuild/scripts/build_docs.py", line 762, in build run( File "/srv/docsbuild/scripts/build_docs.py", line 245, in run result.check_returncode() File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode raise CalledProcessError(self.returncode, self.args, self.stdout, subprocess.CalledProcessError: Command '['make', '-C', '/srv/docsbuild/cpython/Doc', 'PYTHON=/srv/docsbuild/venv-3.13/bin/python', 'SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build', 'BLURB=/srv/docsbuild/venv-3.13/bin/blurb', 'VENVDIR=/srv/docsbuild/venv-3.13', 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0', 'SPHINXERRORHANDLING=', 'autobuild-dev']' returned non-zero exit status 2. 2024-09-18 16:49:43,121 DEBUG fr/3.13: Starting new HTTPS connection (1): sentry.io:443I've not tried changing anything, it has this in
config.toml:picnixz commentedon Sep 19, 2024
This is really weird. In order to pinpoint the issue we may try to manually build the docs and see whether they fail (namely manually invoking make latex and possibly manually invoking the latex commands). There's a way to increase the number of passes but I don't remember the option.
I'll try to first compile the library.tex file (or maybe one file at a time) when I'm back home (next week). By default I think latemk does 5 passes (which is also one reason why latex builds can take a long time).
jfbu commentedon Sep 23, 2024
This by itself can not cause a LaTeX build failure (it would with
\tracinglostchars=3added to .tex file) but it must be investigated by the French translation team as it means the PDF is missing glyphs (maybe replaced by square boxes). It is strange though that this should arise only in French translation.It should first be checked if the English language build has the same. Latexmk by default will not end with positive return status when emitting such a thing. It is not the reason there is build failure.
this only indicates some usually minor problem
it may mean though that internal hyperlinks will target wrong location
yes, agreed. Such latex warnings should be fixed but are not by themselves causing a build failure.
Unfortunately we can only deduce that the error occurred somewhere after page 250, probably on page 251, but this refers to output not input. And the problem is that contrarily to dvi/xdv format there can not be a partial pdf, if xelatex fails during pdf build, there is nothing at all. If latexmk is run with option -xelatex, it will make an xdv, which may still be validly processed by xdvipdfmx.
Anyway, the sole way is to build locally as said by @picnixz.
5 passes is the default maximal number. But most documents require usually 3 or 4 passes only (3 being often a minimum).
This is due to some peculiarity (for legacy reasons of compatiblity with old Latexmk or other reasons now forgotten) of how Sphinx uses here Latexmk, the
pdflatexhere is a Perl variable$pdflatex = 'xelatex ' . $ENV{'LATEXOPTS'} . ' %O %S';(you can see this in filelatexmkrcin build directory). The log file did show that it is xelatex. Withmake latexpdf LATEXMKOPTS="-xelatex"it would have referred to'xelatex'. This option would have left in the build directory an xdv file containing the 250 first pages. This would help locate in library.tex where to look for for manual investigation.As @picnixz explains, the sole way to handle this is to recover the full build directory and (for example) add manually
\stopinside library.tex and trigger manually the build inside the build directory. (because the nature of the crash seems to be such that we do not get a log file going til the error; but normally doing the build manually with an added\errorcontextlines=10in the source should help). This is painful because aux file (also toc file etc) may have been corrupted and one has to remove them first each time. It is a bit rare when the LaTeX log file is truncated like this but it happens. Assuming it really is truncated which perhaps we will see it is not if building manually.(I don't have much time myself and can only perhaps look at this in a few days, and after that I can't for at least 2 to 3 weeks).
hugovk commentedon Sep 23, 2024
cc @JulienPalard
jfbu commentedon Sep 24, 2024
I did a local build using Sphinx 8.0.2 (as installed in the venv created by make venv) of library.pdf for the non-translated cpython project. Indeed a few duplicated labels and missing characters are reported:
Turns out
And 3 more --- see log file 'library.log'is a bit misleading as I don't find any other Missing character line in library.log but rather one finds a number of "undefined references" in the log, in particularAnd rather curiously I find only one line starting with
Missing character: There is no ٤ (U+0664)not two as Latexmk reports, which is utterly bizarre.The latex log file for build of library.pdf contains a number of Warnings:
The
a4thing indicates a broken custom Makefile in project source, not the Sphinx default one, it should bea4paper.As per the Missing characters I can perhaps contribute a patch another day (too late here European time for me and busy) but it is a problem of finding fonts supporting them and using these fonts.
The characters are ARABIC-INDIC DIGIT FOUR, TAMIL DIGIT FOUR, TELUGU DIGIT FOUR, and LAO DIGIT FOUR.
Chances are the Missing characters are reported as such in most translated builds. This means the PDF will not display them correctly.
jfbu commentedon Sep 25, 2024
Apologies for forgetting to mention I did this on 3.13 branch.
And apologies for the partially off-topic report but I don't know off-hand the needed steps to test the French version, which is the main concern here, so I haven't done it.
AA-Turner commentedon Sep 25, 2024
This is a simplified version of what docsbuild-scripts runs, which should work from the
Doc/directory of your CPython clone:make 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\usepackage{fontspec} -D locale_dirs=<locale_dirs_path> -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-devFor reference, the full command is:
make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc=\\usepackage{fontspec} -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=fr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-devA
jfbu commentedon Sep 25, 2024
Thanks Adam! I git cloned
https://git.afpy.org/AFPy/python-docs-fr.git, which has a somewhat complex Makefile and which itselfs clones cpython. I started along their guidelines, created a virtual env for installing dependencies. They don't have a 3.13 branch. I remained on 3.11. I started the make, which ended up in a failure related to blurb, but thelocalesdir had been created and populated forfr.So I went back to my clone of
cpythonwith its 3.13 checkout from yesterday evening and then issued your customized simplified commandThe html was built, then CTRL-C because it was doing all builds, and I wanted only latex so I did
All went fine and I checked build/latex/library.tex did contains French translation, which it did.
At last I am doing
make -C build/latex library.pdf. I did not experience build failure:My cpython is at
commit e9b00cc78853373623031c657193cbe557488c0a (HEAD -> 3.13, origin/3.13).My python-docs-fr is at
commit 2d42a3c0d181f19ab1bd62400bfafc9a8304a4b1 (HEAD -> 3.11, origin/HEAD, origin/3.11). I did not find a 3.13 branch there.The library.pdf does look partially French.
Could it be that the partially parallel builds encounter an error with another file, which causes all builds to abort (and would explain also the partial
library.log). I would need to launch withall-pdftarget.I am doing that now, but will go for lunchbreak. After that I may have no time to check result until about 7PM EST.
jfbu commentedon Sep 25, 2024
all-pdftarget in build/latex with French-translated latex output completed fine and did not cause any build failure. I thus can not reproduce the problem. I saw a few more "Missing characters" than those already reported. All pdf's from .tex files are build:(
turtle-star.pdfis actually an image file it is not among those obtained from xelatex build)AA-Turner commentedon Sep 25, 2024
Update: somewhat annoyingly, after spending a large proportion of today keeping an eye on the server to monitor for
fr/3.13, everything seems to be fine. Most likely something in the unmerged backports fixed the issue, if I had to guess.Accordingly, https://docs.python.org/fr/3.13/index.html shows "Mis à jour le Sep 25, 2024 (13:37 UTC)." and https://docs.python.org/fr/3.13/archives/python-3.13.0rc2-docs-pdf-a4.zip is up-to-date.
A
hugovk commentedon Sep 25, 2024
Hooray! Thank you both for your investigations!