Skip to content

test_hashlib fails when run on Free Threading with TSAN #153201

Closed
Closed
test_hashlib fails when run on Free Threading with TSAN#153201
Labels
extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@vstinner
vstinner
opened on Jul 6, 2026

Crash report

TSAN CI job: https://github.com/python/cpython/actions/runs/28747943876/job/85377269260?pr=152717

test_hashlib fails when run on Free Threading with TSAN on the main branch.

Commands:

export CC=clang
export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"

./configure --config-cache --with-thread-sanitizer --with-pydebug --disable-gil
make -j4
./python -m test --tsan-parallel --parallel-threads=4 -j4 -W --timeout=600 --slowest

Tests output (truncated):

Using random seed: 1463135961
0:00:00 load avg: 2.65 mem: 267.1 MiB Run 2 tests in parallel using 2 worker processes (timeout: 10 min, worker timeout: 15 min)
0:00:06 load avg: 2.76 mem: 611.2 MiB [1/2] test_abc passed
0:00:14 load avg: 2.78 mem: 278.3 MiB [2/2/1] test_hashlib worker non-zero exit code (Exit code 66)
test_algorithms_available (test.test_hashlib.HashLibTestCase.test_algorithms_available) [threads=4] ... ok
test_algorithms_guaranteed (test.test_hashlib.HashLibTestCase.test_algorithms_guaranteed) [threads=4] ... ok
(...)
test_scrypt_types (test.test_hashlib.TestScrypt.test_scrypt_types) [threads=4] ... ok
test_scrypt_validate (test.test_hashlib.TestScrypt.test_scrypt_validate) [threads=4] ... ok

----------------------------------------------------------------------
Ran 88 tests in 9.698s

OK (skipped=56)

== Tests result: FAILURE ==

TSAN logs (truncated):

WARNING: ThreadSanitizer: data race (pid=30628)
  Read of size 8 at 0x721c0000e3a8 by thread T456:
    #0 memcmp <null> (python+0x10d49e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 <null> <null> (libcrypto.so.3+0x24f2d1) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)

  Previous write of size 8 at 0x721c0000e3a8 by thread T454 (mutexes: write M0, write M1):
    #0 memcpy <null> (python+0xfc022) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 <null> <null> (libcrypto.so.3+0x24f0e6) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)

  Mutex M0 (0x721000000500) created at:
    #0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)

  Mutex M1 (0x7210000008c0) created at:
    #0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)

Full TSAN logs: tsan_full.log.

Linked PRs

Activity

  1. vstinner
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-free-threading
    on Jul 6, 2026

  2. vstinner commented on Jul 6, 2026

    @vstinner
    vstinner
    on Jul 6, 2026
    MemberAuthor

    I can reproduce the issue with "OpenSSL 3.5.7 9 Jun 2026" on Fedora 44 with commands:

    export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"
./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log

    It's the test_pbkdf2_hmac_c() test which triggers the data race.

  3. vstinner commented on Jul 6, 2026

    @vstinner
    vstinner
    on Jul 6, 2026
    MemberAuthor

    See also issue gh-143750.

  4. vstinner commented on Jul 6, 2026

    @vstinner
    vstinner
    on Jul 6, 2026
    MemberAuthor

    I tried but failed (so far) to reproduce the issue with OpenSSL built with TSAN.

    I applied this patch (extracted from #143750):

    diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py
index 1a213187b89..d5e38993d97 100755
--- a/Tools/ssl/multissltests.py
+++ b/Tools/ssl/multissltests.py
@@ -163,6 +163,12 @@
     dest='keep_sources',
     help="Keep original sources for debugging."
 )
+parser.add_argument(
+    '--tsan',
+    action='store_true',
+    dest='tsan',
+    help="Build with thread sanitizer. (Disables fips in OpenSSL 3.x)."
+)
 
 
 class AbstractBuilder(object):
@@ -317,6 +323,8 @@ def _build_src(self, config_args=()):
         """Now build openssl"""
         log.info("Running build in {}".format(self.build_dir))
         cwd = self.build_dir
+        if self.args.tsan:
+            config_args += ("-fsanitize=thread",)
         cmd = [
             "./config", *config_args,
             "shared", "--debug",

    I ran commands:

    ./Tools/ssl/multissltests.py --steps=library --base-directory ~/multissl --openssl 3.5.7 --system Linux --tsan
export LD_LIBRARY_PATH=~/multissl/openssl/3.5.7/lib/ 

./configure --with-thread-sanitizer --with-pydebug --disable-gil --with-openssl=/home/vstinner/multissl/openssl/3.5.7
make clean
make

./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log

    Sadly, so far I failed to reproduce the test_hashlib failure this way.

  5. kumaraditya303 commented on Jul 7, 2026

    @kumaraditya303
    kumaraditya303
    on Jul 7, 2026
    Contributor

    I think it might be a false-positive race because openssl was not compiled with tsan enabled.

  6. sobolevn commented on Jul 7, 2026

    @sobolevn
    sobolevn
    on Jul 7, 2026
    Member

    This failure now blocks the CI for multiple PRs :(
    Example: https://github.com/python/cpython/actions/runs/28853850174?pr=153253 Even 3rd re-run is failing.

    And since san job is required, no PRs are able to get merged which touch some specific files.
    Should we disable san job's required status for now?

  7. picnixz commented on Jul 7, 2026

    @picnixz
    picnixz
    on Jul 7, 2026
    Member

    The race could be in get_openssl_evp_md_by_utf8name which is a known failure however that I recently fixed #153019. I want to emit a nice commit message which I unfortunately didn't have time to write for now but it was also very hard to reproduce in general.

  8. picnixz
    added
    extension-modulesC modules in the Modules dir
    on Jul 7, 2026
  9. kumaraditya303
    closed this as completedon Jul 10, 2026
  10. sobolevn
    reopened this on Jul 10, 2026

  11. kumaraditya303 commented on Jul 10, 2026

    @kumaraditya303
    kumaraditya303
    on Jul 10, 2026
    Contributor

    @sobolevn Why did you reopen this?

  12. kumaraditya303
    added 2 commits that reference this issue on Jul 10, 2026

    gh-153201: Make TSan CI mandatory (#153491)

    d071438

    [3.15] gh-153201: Make TSan CI mandatory (GH-153491) (#153497)

    bcc1a36

  13. sobolevn commented on Jul 10, 2026

    @sobolevn
    sobolevn
    on Jul 10, 2026
    Member

    So we can merge #153201 and backports :)

  14. kumaraditya303
    closed this as completedon Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions