Skip to content

Do not start nested parallel work from inside a thread-server job - #6260

Open
apullin wants to merge 3 commits into
OpenMathLib:developfrom
apullin:nested-threading
Open

apullin wants to merge 3 commits into
OpenMathLib:developfrom
apullin:nested-threading

Conversation

@apullin

@apullin apullin commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Several level-1 kernels split vectors longer than 10,000 elements across num_cpu_avail(1) threads.

Affected level-1 kernels:

  • x86_64: kernel/x86_64/ddot.c, kernel/x86_64/zdot.c (Haswell, Zen, SkylakeX/Cooperlake/Sapphire Rapids, Sandybridge, Bulldozer family)
  • arm64: kernel/arm64/dot.c, kernel/arm64/zdot_thunderx2t99.c, kernel/arm64/copy_thunderx2t99.c (Neoverse N1/N2/V1/V2, SVE, SME, A64FX, Vortex, ThunderX2/3)

Threaded level-2 drivers call those kernels from inside each worker job.

Affected level-2 callers on columns longer than 10,000 elements (DOT):

  • dgbmv, zgbmv (and sgbmv, cgbmv on arm64) with trans = 'T' or 'C'
  • dtbmv, ztbmv (and stbmv, ctbmv on arm64) with trans = 'T' or 'C'
  • dtpmv, ztpmv (and stpmv, ctpmv on arm64) with trans = 'T' or 'C'
  • zhpmv (and chpmv on arm64)

Affected level-2 callers on strided vectors longer than 10,000 elements on arm64 (COPY_K):

  • sger, dger, cgeru, cgerc, zgeru, zgerc
  • strmv, dtrmv, ctrmv, ztrmv
  • stbmv, dtbmv, ctbmv, ztbmv
  • stpmv, dtpmv, ctpmv, ztpmv
  • ssyr, dsyr, csyr, zsyr, cher, zher
  • ssyr2, dsyr2, csyr2, zsyr2, cher2, zher2
  • sspr, dspr, cspr, zspr, chpr, zhpr
  • sspr2, dspr2, cspr2, zspr2, chpr2, zhpr2

In pthreads builds num_cpu_avail() had no nesting check, so each worker re-entered exec_blas(). Once all pool threads were occupied by the outer call, exec_blas_async() spun forever waiting for idle worker slots while holding exec_queue_lock before signalling any assigned sub-jobs.

This PR tracks whether the current thread is executing a thread-server job and makes num_cpu_avail() return 1 when set. Covered execution paths:

  • pool workers in the pthreads and win32 servers
  • exec_threads() for OpenMP and registered threads callbacks
  • the calling thread while it runs its own share in exec_blas()

Tests

utest/test_nested_thread.c covers:

  • dgbmv('T'), sgbmv('T'), zgbmv('C') with column DOT > 10,000 elements
  • dger, dtbmv with strided COPY > 10,000 elements

With the POSIX pthreads server each case runs in a child process under a timeout so a regression fails cleanly instead of hanging CI.

Verification (before → after on develop)

  • x86_64 (Zen, Haswell, Sandybridge, SkylakeX under Intel SDE): dgbmv('T'), zgbmv('C'), zhpmv, dtpmv('T') hang → pass (170/170 openblas_utest, 1545/1545 openblas_utest_ext)
  • arm64 (NEOVERSEN1 under qemu): all 5 nested_thread cases hang → pass (166/166 openblas_utest)
  • USE_OPENMP=1, USE_THREAD=0, and CMake BUILD_SHARED_LIBS=ON builds: all pass

Several level-1 kernels split vectors longer than 10000 elements across
num_cpu_avail(1) threads:
- x86_64: ddot.c, zdot.c
- arm64: dot.c, zdot_thunderx2t99.c, copy_thunderx2t99.c

Threaded level-2 drivers call those kernels from inside each worker job.

Callers reaching threaded DOT on columns longer than 10000 elements:
- ?GBMV('T'/'C'), ?TBMV('T'/'C'), ?TPMV('T'/'C'), ?HPMV

Callers reaching threaded COPY_K on strided vectors longer than 10000
elements on arm64:
- ?GER, ?TBMV, ?TRMV, ?TPMV
- ?SYR/?HER, ?SYR2/?HER2, ?SPR/?HPR, ?SPR2/?HPR2

In pthreads builds num_cpu_avail() had no nesting check, so each worker
re-entered exec_blas(). Once all pool threads were occupied by the outer
call, exec_blas_async() spun forever waiting for idle worker slots while
holding exec_queue_lock before signalling any assigned sub-jobs.

Track whether the current thread is executing a thread-server job and
make num_cpu_avail() return 1 when set. This covers pool workers in the
pthreads and win32 servers, exec_threads() for OpenMP and registered
threads callbacks, and the calling thread while it runs its own share in
exec_blas().
Cover threaded level-2 calls whose worker jobs invoke threaded level-1
kernels above 10000 elements:
- DGBMV('T'), SGBMV('T'), ZGBMV('C') (column DOT)
- DGER, DTBMV (strided COPY)

With the POSIX pthreads server each case runs in a child process under a
timeout so a regression fails cleanly instead of hanging the test run.
openblas_utest.h includes <stdlib.h> before common.h defines _GNU_SOURCE
and __USE_XOPEN, which leaves __WAIT_STATUS undefined when <sys/wait.h>
is included afterwards in the manylinux1 (CentOS 5 / glibc 2.5) build.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant