Repository navigation
Windows wheel builds run out of memory #10074
Description
Windows wheel builds started running out of memory. Here's an extract from the logs:
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\RUNNER~1\AppData\Local\Temp\pip-req-build-b3o88uhy\mypyc\lib-rt -Ibuild -Ic:\cibw\python\python.3.5.4\tools\include -Ic:\cibw\python\python.3.5.4\tools\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /Tcbuild\__native_35888ab3ae83eb1dd0bf.c /Fobuild\temp.win-amd64-3.5\Release\build\__native_35888ab3ae83eb1dd0bf.obj /O2 /wd4102 /wd4101 /wd4146 /GL- /wd9025
cl : Command line warning D9025 : overriding '/GL' with '/GL-'
cl : Command line warning D9014 : invalid value '9025' for '/wd'; assuming '5999'
__native_35888ab3ae83eb1dd0bf.c
build\__native_35888ab3ae83eb1dd0bf.c(4558): fatal error C1060: compiler is out of heap space
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
Building wheel for mypy (PEP 517): finished with status 'error'
ERROR: Failed building wheel for mypy
Activity
JukkaL commentedon Feb 11, 2021
It looks like this started happening after the 0.800 release, so this doesn't block the 0.810 release.
To investigate this further, we can try bisecting the commits since 0.800 was released. This might have been caused by the organic growth of the compiled codebase, or by a mypyc regression. If my first hypothesis is correct, a possible workaround is to not compile some parts of the codebase to get the build again under the CI memory limit.
JukkaL commentedon Feb 11, 2021
This started happening some time after 68a0c65. After this commit wheels builds were broken, so we don't know which commit caused the failure. There is at least one suspicious mypyc commit which is worth checking (7ec1455).
hauntsaninja commentedon Feb 11, 2021
There was some Windows code that I removed when writing the new wheel build script, because I didn't understand it and it worked without it. In case it helps: https://github.com/mypyc/mypy_mypyc-wheels/blob/5638de415c9fc7fa9076b2d9a429e887817943a0/appveyor.yml#L32-L39 (mentioned in #9536 (comment))
emmatyping commentedon Feb 11, 2021
Ah the important part is:
- call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"By default MSVC uses a 32 bit compiler executable, which often OOMs on larger compiles,
vcvars64.batsets up the environment to use the 64 bit compiler executable.E: Also is it correct to remove
"SET MYPYC_OPT_LEVEL=2"?hauntsaninja commentedon Feb 11, 2021
The default is
MYPYC_OPT_LEVEL=3, which seemed better / consistent with what we do on other platforms (https://github.com/mypyc/mypy_mypyc-wheels/blob/a6cee7828a842ac2ffd2c5fafc7f2227094b0923/.github/workflows/build.yml#L60). Since things just worked when I set it up, I didn't mess around further.JukkaL commentedon Feb 15, 2021
I vaguely remember that using
MYPYC_OPT_LEVEL=3and a 32-bit compiler caused problems on Windows (slow compiles and/or running out of memory). It's possible that we are now using a more recent compiler and things got better, but only barely so -- and now we have hit some code size threshold and regressed again.Switching to consistent settings was a good idea, since it did work. However, I think that we should now try reverting to the original settings to get the build working again. I can start by trying to switch the optimization level on Windows.
Attempt to fix wheel builds by using optimization level 2 on Windows (#…
JukkaL commentedon Feb 15, 2021
Changing the optimization level didn't help. It would be great if somebody can figure out how to invoke
vcvars64.batin the GitHub action.emmatyping commentedon Feb 15, 2021
I believe it should simply be
JukkaL commentedon Feb 15, 2021
Since we build all platforms using the same file, I think that we need to skip this on non-Windows platforms. Maybe we can use an if statement?
JukkaL commentedon Feb 15, 2021
I can't figure how to use an if statement for this -- it might not work.
Another idea would be to use
pwshas the shell, since it should be supported on all platforms. I've never used PowerShell, however. I guess we can also always use a Python script.emmatyping commentedon Feb 15, 2021
You can add an
if: startsWith(matrix.os, 'windows')to the item.4 remaining items
JukkaL commentedon Feb 16, 2021
Closed this accidentally. I moved the wheel build logic to a script in the mypy repository (#10096). This should make it easier to experiment with things and to add more complex logic. Once the new script works as well as the old build logic, I'll try to finally fix the Windows wheel build.
Use script in the mypy repository to build wheels (#18)
Use a 64-bit C compiler on Windows to build wheels
Use a 64-bit C compiler on Windows to build wheels (#10097)
JukkaL commentedon Feb 16, 2021
Running
vcvars64.batbefore invoking cibuildwheel didn't help. The 32-bit compiler seems to be still used.I'm running out of options. I would appreciate any help here, particularly since I don't have a Windows dev environment set up, so debugging issues is slow. With the most recent changes, it should be possible to test things locally. I tried some things in mypyc/mypy_mypyc-wheels#20, but they didn't help.
Here are a few other ideas that might plausibly help:
VsDevCmd.batinstead ofvcvars64.bat.vcvars64.bat. Make sure this works as expected. If not, figure out what needs to be done to propagate them. If nothing else helps, updatingCIBW_ENVIRONMENT_WINDOWSmight do the trick.Akuli commentedon Feb 16, 2021
In the past, I have used this github action.
hauntsaninja commentedon Feb 16, 2021
I was surprised mypyc/mypy_mypyc-wheels#20 didn't work. Try setting
DISTUTILS_USE_SDK=1? (Agreed that this is easiest done by someone with access to a Windows dev machine)Use 64-bit compiler on Windows (#20)
JukkaL commentedon Feb 18, 2021
This seems to enable the 64-bit compiler, but we still run out of memory.
My next attempt is reducing the size of the build: #10107
If that doesn't help, bisecting to find the commit that caused the breakage could be helpful. This will be somewhat tricky, since we'll need to merge changes to the build scripts on top of the mypy commit to test before the build works.
I'm also working on reducing the size of the generated C code, which may also help with memory use. This will take some time, however, but this could be a good longer-term solution.
If nothing else works, we can perhaps split the build into multiple smaller files, or get a bigger machine for running Windows builds.
JukkaL commentedon Feb 19, 2021
Reducing the size of the build did not help. The build was already using multiple files.
JukkaL commentedon Feb 19, 2021
It now seems like 7ec1455 (or the follow-up commit) caused the failures. I've reverted the commit now.
My initial impression is that MSVC doesn't like the code we are generating and runs out of memory, either because of some major inefficiency or a bug. I have an idea about how to modify the generated code to not cause problems. I will have to verify that the Windows wheel build no longer is affected before landing the change again.
JukkaL commentedon Feb 19, 2021
We now have Windows wheels again: https://github.com/mypyc/mypy_mypyc-wheels/releases/tag/v0.820%2Bdev.b07c564043328ded5686d1aac4d7d3e5c8a5bfb0
Add Python script to build wheels using cibuildwheel (#10096)