Skip to content

Bugfixes and RFC compliance for SymCrypt-OpenSSL 1.11 - #185

Open
Maxwell Moyer-McKee (mamckee) wants to merge 14 commits into
scossl-1.11from
mamckee/scossl-1.11-validation-fixes
Open

Maxwell Moyer-McKee (mamckee) wants to merge 14 commits into
scossl-1.11from
mamckee/scossl-1.11-validation-fixes

Conversation

@mamckee

@mamckee Maxwell Moyer-McKee (mamckee) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes a handful of bugs in the provider, as well as behavior discrepancies between this implementation and the default OpenSSL provider.

ML-DSA / ML-KEM encoders and decoders

  • Update PQ encoders and decoders to match the now-standardized formats in officially published RFCs
  • Private keys are encoded/decoded with OpenSSL ASN.1 templates as the CHOICE { seed [0] IMPLICIT OCTET STRING, expandedKey OCTET STRING, both SEQUENCE { seed, expandedKey } } structure.
  • For the both form, the expanded key is verified against the key derived from the seed.
  • AlgorithmIdentifier parameters must be absent for ML-DSA keys.

Keygen selection

  • ECC, X25519, DH, ML-KEM and hybrid ML-KEM keygen honor the requested selection, so EVP_PKEY_paramgen returns a params-only key.
  • X25519 export, ECDH derive and ECC match handle params-only keys (fixes a NULL dereference when exporting/deriving with a key from paramgen).

Other

  • RSA decrypt robustness fix, KeysInUse fixes, version bump.

Testing

  • Build: release + ASan/UBSan, no new warnings.
  • Codec interop with OpenSSL 3.5 default provider: 282/282 (all ML-DSA/ML-KEM parameter sets, seed/expanded/both private key forms, PEM/DER, PKCS#8/SPKI, both directions).
  • Negative decoding tests: malformed, truncated and mismatched seed/expanded encodings are rejected. Non-DER (BER) encodings of the inner CHOICE are still accepted; to be addressed separately.
  • Crypto interop (sign/verify, encap/decap across providers): 140/140.
  • TLS 1.3 e2e with openssl s_server/s_client: 144/144 (ML-DSA-44/65/87 certs × X25519MLKEM768, SecP256r1MLKEM768, SecP384r1MLKEM1024, ML-KEM-512/768/1024 × all SCOSSL/OpenSSL server/client pairings); 48/48 under ASan.
  • Paramgen/keygen selection stress across EC, X25519, DH, ML-KEM and hybrids, exercising every operation on params-only keys: no crashes under ASan; behavior matches OpenSSL.
  • SslPlay and KeysInUseTest pass.

…than the SymCrypt output bound

scossl_rsa_decrypt() ignored the caller-supplied cbDst and always passed a
size derived from cbModulus to the underlying SymCrypt decrypt APIs. For
RSA_PKCS1_WITH_TLS_PADDING, the provider only validates outsize >= 48
(SSL_MAX_MASTER_KEY_LENGTH), so a TLS <=1.2 static-RSA ClientKeyExchange
with an oversized PKCS#1 message (up to cbModulus - 11 bytes) causes
SymCryptRsaPkcs1Decrypt to write past the 48-byte premaster-secret buffer
allocated by libssl, with fully attacker-controlled overflow content.

Bound the SymCrypt output size to min(cbDst, cbModulus - padding overhead)
for PKCS1/TLS and OAEP decrypt, and add an explicit cbDst < cbModulus guard
before the raw (no-padding) decrypt, which otherwise always requested a
full-modulus-sized write regardless of the caller's buffer size.

No behavioral change for the engine (cbDst == -1 sentinel still maps to
cbModulus) or for the provider's non-TLS decrypt paths, which already
enforce outsize >= cbModulus upstream in p_scossl_rsa_cipher.c before
calling into this shared function.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1e2e58ee-b90f-43af-aa4c-c290aceaadb7
KeysInUse initializes lazily on the first call to keysinuse_is_running(),
so when the provider config set enabled = 0 (or had no keysinuse section)
the first sign/decrypt still started KeysInUse and logged events. This
contradicts the documented default (off unless enabled by config) and
caused the readlink() from crypto free paths that breaks sandboxed
processes. Explicitly call keysinuse_disable() in that case, unless an
earlier provider instance in the process already enabled KeysInUse.

KeysInUseTest loads the provider without a keysinuse config, so it now
opts in through the KEYSINUSE_ENABLED environment variable. Since KeysInUse
now starts when the provider is loaded, check the logged process start
time against a timestamp taken before providers are loaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 388f7776-048b-41e5-8c6c-8f9175ddf05d
The PKCS#8 privateKey for ML-DSA is the CHOICE
  seed [0] IMPLICIT OCTET STRING, expandedKey OCTET STRING,
  both SEQUENCE { seed, expandedKey }.
The encoder wrapped the 32 byte seed in a universal OCTET STRING (04 20),
which OpenSSL and other implementations treat as a malformed expandedKey,
so private keys written by the provider could not be read elsewhere.

Encode the seed as [0] (80 20), matching RFC 9881 and OpenSSL. The
decoder now parses exactly the three CHOICE forms, preferring the seed
and verifying that the expanded key in the "both" form matches the key
derived from the seed. Trailing data and malformed encodings, including
the previous non-standard seed encoding, are rejected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 388f7776-048b-41e5-8c6c-8f9175ddf05d
The RSA decrypt tests decrypted into a buffer sized for the plaintext,
which is smaller than the RSA modulus. The provider rejects this with
"output buffer too small", aborting the run before the remaining key
types were tested, and with the legacy engine path RSA_private_decrypt
could write past the end of the buffer. Decrypt into a separate buffer
sized once from the key, resetting the in/out length before each call.

main() also returned 1 on success and 0 on failure, so failures were
not visible to callers. Return 0 only when all tests pass.

Rename processStartTime to testStartTime to reflect that it marks the
start of the tests rather than the process.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 388f7776-048b-41e5-8c6c-8f9175ddf05d
RFC 9881 and draft-ietf-lamps-pq-composite-sigs require the parameters
field of the ML-DSA and composite ML-DSA AlgorithmIdentifier to be
absent in both PrivateKeyInfo and SubjectPublicKeyInfo. The decoders
accepted keys with parameters present (e.g. NULL), unlike OpenSSL.
Reject them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 388f7776-048b-41e5-8c6c-8f9175ddf05d
The symlink check on the log file used stat(), which follows symlinks,
so S_ISLNK() could never be true and a dangling symlink was treated as
a missing file. The log file was then opened with O_CREAT and without
O_NOFOLLOW. Since /var/log/keysinuse is world-writable and log file
names are predictable, a local user could plant a symlink at another
user's log path to create or append to arbitrary files.

Use lstat() so symlinks are detected and removed, and open the log file
with O_NOFOLLOW so a symlink placed after the check is not followed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 388f7776-048b-41e5-8c6c-8f9175ddf05d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cb2bed9a-250a-4787-9cda-95ae7623ed3e
Keygen now honors the selection, so EVP_PKEY_paramgen can return
uninitialized ECC/X25519 keys. Guard X25519 export and ECDH derive
against keys without key material, and only compare initialization
state in match when the keypair is selected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 27da34cb-485f-4fad-a249-fb6e7efdc300

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change can silently suppress requested KeysInUse logs across provider contexts and breaks the OpenSSL 1.1.1 test build by calling the OpenSSL 3.0-only EVP_PKEY_get_size API.

2 open issues
What changed in this PR

This PR prepares SCOSSL 1.11.3 with post-quantum key-format compliance fixes, parameter-only key generation support, and robustness updates across the OpenSSL provider and shared code.

Changes:

  • Updates ML-KEM/ML-DSA codecs for standard private-key forms, seed/expanded-key consistency checks, and absent algorithm parameters.
  • Honors key-generation selections and handles parameter-only keys safely.
  • Tightens RSA output-buffer handling, hardens KeysInUse logging, and updates its tests.
File Description
test/​KeysInUseTest/​KeysInUseTest.cpp Updates logging timestamps, decrypt buffers, opt-in setup, and exit codes.
SymCryptProvider/​src/​p_scossl_ecc.c Computes encoded public-key size from curve parameters.
SymCryptProvider/​src/​p_scossl_base.c Adds explicit default-off handling for KeysInUse.
SymCryptProvider/​src/​keymgmt/​p_scossl_mlkem_keymgmt.c Honors generation selection for parameter-only ML-KEM keys.
SymCryptProvider/​src/​keymgmt/​p_scossl_mlkem_hybrid_keymgmt.c Honors generation selection and computes ciphertext sizes without key material.
SymCryptProvider/​src/​keymgmt/​p_scossl_ecc_keymgmt.c Supports parameter-only EC/X25519 generation and guards key operations.
SymCryptProvider/​src/​keymgmt/​p_scossl_dh_keymgmt.c Supports parameter-only DH generation and parameter queries.
SymCryptProvider/​src/​keyexch/​p_scossl_ecdh.c Rejects derivation with uninitialized key material.
SymCryptProvider/​src/​encoder/​p_scossl_encode_mlkem.c Corrects seed encoding, public-key buffer ownership, and text export format.
SymCryptProvider/​src/​encoder/​p_scossl_encode_mldsa.c Encodes private seeds with the required implicit tag.
SymCryptProvider/​src/​decoder/​p_scossl_decode_mlkem.c Parses private-key choices and validates combined forms and algorithm parameters.
SymCryptProvider/​src/​decoder/​p_scossl_decode_mldsa.c Parses private-key choices and validates combined forms and algorithm parameters.
ScosslCommon/​src/​scossl_rsa.c Enforces caller-provided decryption buffer limits.
KeysInUse/​keysinuse.c Detects log-file symlinks and prevents following them when opening files.
CMakeLists.txt Bumps the release version to 1.11.3.

Review effort: Balanced

Comment thread SymCryptProvider/src/p_scossl_base.c Resolved
Comment thread test/KeysInUseTest/KeysInUseTest.cpp Resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 27da34cb-485f-4fad-a249-fb6e7efdc300
Without a validate function OpenSSL treats EVP_PKEY_*_check as success,
so params-only or public-only keys passed public/private/pairwise checks.
Key material is validated by SymCrypt on import, so validate only checks
that the selected components are present, matching ECC and ML-DSA.

Hybrid ML-KEM intentionally has no validate function, matching the
OpenSSL default provider's MLX keymgmt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 27da34cb-485f-4fad-a249-fb6e7efdc300

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues allow concurrent provider loads to disable logging, EC parameter comparisons to equate different curves, and DH parameter generation to lose requested private-key lengths.

3 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Preserve the requested DH private-key length in paramgen

SymCryptProvider/​src/​keymgmt/​p_scossl_dh_keymgmt.c:600

Preserve genCtx->nBitsPriv in the returned context when generating parameters. p_scossl_dh_keymgmt_new_ctx initializes it to the default, and this path never copies the requested OSSL_PKEY_PARAM_DH_PRIV_LEN. Consequently, paramgen with a custom private-key length returns parameters reporting the default length, and subsequent keygen from those parameters loses the request because p_scossl_dh_keygen_set_template copies tmplCtx->nBitsPriv. Copy the requested length before returning the context.

1 resolved since last review

Review effort: Balanced

Comment thread SymCryptProvider/src/p_scossl_base.c Resolved
EVP_PKEY_parameters_eq selects only OSSL_KEYMGMT_SELECT_DOMAIN_PARAMETERS,
so a params-only key could match a key on a different curve. Compare
curves for any parameter selection, matching OpenSSL ec_match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 27da34cb-485f-4fad-a249-fb6e7efdc300
@mamckee Maxwell Moyer-McKee (mamckee) changed the title SCOSSL 1.11 validation fixes: ML-KEM/ML-DSA codec compliance and keygen selection Bugfixes and RFC compliance for SymCrypt-OpenSSL 1.11 Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unsynchronized process-wide initialization can permanently disable logging during concurrent provider loads, and DH parameter checks incorrectly accept unvalidated caller-supplied groups.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

High severity Validate custom DH parameters before reporting success

SymCryptProvider/​src/​keymgmt/​p_scossl_dh_keymgmt.c:1054

EVP_PKEY_param_check() now reports success for caller-supplied DH parameters merely because ctx->pDlGroup is set. These parameters are imported with SYMCRYPT_DLGROUP_FIPS_NONE, and parameters-only imports skip key import entirely, so this path does not establish that the group is valid. Validate custom groups, or return failure for groupSetByParams until those checks exist, while retaining the known-group fast path.

3 resolved since last review

Review effort: Balanced

}

if (!scossl_dh_generate_keypair(ctx->keyCtx, genCtx->nBitsPriv, genCtx->pDlGroup))
if ((genCtx->selection & OSSL_KEYMGMT_SELECT_KEYPAIR) != 0 &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping keypair generation leaves parameter-only DH objects uninitialized, so the export callback rejects OpenSSL’s OSSL_KEYMGMT_SELECT_ALL request. This makes EVP_PKEY_paramgen() followed by EVP_PKEY_get1_DH() fail. Recommend allowing export of available domain parameters and skipping key-material export when uninitialized.

// disabled when not enabled by config or environment. Skip this if a
// previous provider instance in this process already enabled it, since
// the KeysInUse state is process-wide.
keysinuse_disable();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading an unconfigured provider first permanently disables KeysInUse, preventing a later instance from enabling logging even when explicitly configured. Recommend using reversible default-off handling or provider-local gating instead of keysinuse_disable().

keysinuse_init();
p_scossl_keysinuse_started = TRUE;
}
else if (!p_scossl_keysinuse_started)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concurrent provider loads access p_scossl_keysinuse_started without synchronization. An unconfigured instance can permanently disable logging between another instance’s initialization and flag update. Recommend synchronizing the whole transition; an atomic flag alone would not protect it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants