Skip to content

Wycheproof: Fix incorrect assertion for invalid encaps/decaps tests#1636

Open
mkannwischer wants to merge 1 commit intomainfrom
fix-wycheproof-test-assertions
Open

Wycheproof: Fix incorrect assertion for invalid encaps/decaps tests#1636
mkannwischer wants to merge 1 commit intomainfrom
fix-wycheproof-test-assertions

Conversation

@mkannwischer
Copy link
Copy Markdown
Contributor

The encaps and combined test functions asserted that K should NOT match the expected value for invalid test cases where the binary succeeded. This is wrong: for invalid inputs, the binary should reject the input via input validation, not succeed and produce a different K.

Fix this by restructuring all test functions to branch on tc["result"] first, then verify the binary's behavior:

  • "invalid": assert the binary rejected the input (_error or decode_error)
  • "valid": assert outputs match expected values
  • anything else: fail with unexpected result

Additional changes:

  • Remove "acceptable" as a result type (not used in any test vectors)
  • In keygen_seed tests, only "valid" results exist; simplify accordingly
  • In semi_expanded_decaps tests, assert that invalid tests are rejected rather than silently ignoring errors on valid tests
  • In combined tests, distinguish keygen decode errors from runtime errors, and always verify ek when keygen succeeds

@mkannwischer mkannwischer force-pushed the fix-wycheproof-test-assertions branch from f2e623f to a87dddf Compare March 23, 2026 05:48
@oqs-bot
Copy link
Copy Markdown
Contributor

oqs-bot commented Mar 23, 2026

CBMC Results (ML-KEM-512)

Full Results (187 proofs)
Proof Status Current Previous Change
**TOTAL** 1285s 1261s +1.9%
mlk_indcpa_keypair_derand 158s 160s -1%
mlk_indcpa_enc 147s 145s +1%
mlk_keccak_squeezeblocks_x4 123s 123s +0%
mlk_rej_uniform_c 81s 76s +7%
mlk_polyvec_basemul_acc_montgomery_cached_c 69s 70s -1%
mlk_poly_rej_uniform 43s 40s +7%
poly_ntt_native 39s 35s +11%
polyvec_basemul_acc_montgomery_cached_native 23s 22s +5%
keccakf1600x4_permute_native_x4 20s 18s +11%
mlk_poly_reduce_native 17s 15s +13%
mlk_polyvec_add 17s 18s -6%
mlk_poly_decompress_d10_native 15s 13s +15%
mlk_ntt_layer 13s 12s +8%
mlk_poly_decompress_d4_native 13s 13s +0%
mlk_indcpa_dec 11s 12s -8%
mlk_poly_frommsg 10s 11s -9%
mlk_fqmul 9s 8s +12%
mlk_ntt_butterfly_block 9s 7s +29%
mlk_poly_frombytes_native 9s 10s -10%
mlk_poly_rej_uniform_x4 9s 8s +12%
mlk_polymat_permute_bitrev_to_custom 9s 7s +29%
mlk_keccak_absorb_once_x4 8s 8s +0%
mlk_keccak_squeeze_once 8s 6s +33%
keccakf1600_permute_native 7s 4s +75%
mlk_keccak_squeezeblocks 7s 9s -22%
mlk_poly_cbd_eta2 6s 6s +0%
mlk_poly_tobytes_c 6s 3s +100%
kem_dec 5s 4s +25%
mlk_invntt_layer 5s 7s -29%
mlk_matvec_mul 5s 3s +67%
mlk_poly_add 5s 5s +0%
mlk_poly_decompress_d10_c 5s 5s +0%
mlk_poly_mulcache_compute_c 5s 3s +67%
mlk_shake256x4 5s 5s +0%
poly_decompress_d10_native_x86_64 5s 4s +25%
poly_decompress_d4_native_x86_64 5s 5s +0%
poly_frombytes_native_x86_64 5s 7s -29%
intt_native_x86_64 4s 4s +0%
mlk_keccakf1600_extract_bytes 4s 1s +300%
mlk_poly_compress_d10_c 4s 4s +0%
mlk_poly_compress_d4 4s 3s +33%
mlk_poly_frombytes_c 4s 2s +100%
mlk_poly_reduce_c 4s 4s +0%
mlk_polyvec_frombytes 4s 1s +300%
mlk_polyvec_permute_bitrev_to_custom 4s 2s +100%
mlk_scalar_compress_d10 4s 2s +100%
mlk_value_barrier_u32 4s 1s +300%
poly_decompress_d5_native_x86_64 4s 4s +0%
poly_getnoise_eta1122_4x_native 4s 2s +100%
poly_mulcache_compute_native_x86_64 4s 3s +33%
polyvec_basemul_acc_montgomery_cached_k4_native_aarch64 4s 3s +33%
rej_uniform_native 4s 5s -20%
keccak_f1600_x1_native_aarch64 3s 1s +200%
keccak_f1600_x1_native_aarch64_v84a 3s 2s +50%
keccakf1600x4_xor_bytes_native 3s 3s +0%
kem_enc 3s 3s +0%
kem_enc_derand 3s 3s +0%
mlk_ct_cmask_nonzero_u8 3s 1s +200%
mlk_ct_get_optblocker_u8 3s 2s +50%
mlk_gen_matrix 3s 3s +0%
mlk_gen_matrix_serial 3s 2s +50%
mlk_keccak_absorb_once 3s 4s -25%
mlk_keccakf1600_xor_bytes 3s 2s +50%
mlk_keccakf1600x4_extract_bytes 3s 2s +50%
mlk_poly_compress_d10_native 3s 1s +200%
mlk_poly_compress_d5_c 3s 2s +50%
mlk_poly_decompress_d11 3s 2s +50%
mlk_poly_decompress_d4 3s 2s +50%
mlk_poly_decompress_d5_c 3s 2s +50%
mlk_poly_getnoise_eta1122_4x 3s 3s +0%
mlk_poly_getnoise_eta1_4x 3s 2s +50%
mlk_poly_getnoise_eta1_4x_native 3s 3s +0%
mlk_poly_mulcache_compute 3s 2s +50%
mlk_poly_ntt_c 3s 2s +50%
mlk_poly_tobytes 3s 2s +50%
mlk_polyvec_compress_du 3s 3s +0%
mlk_polyvec_decompress_du 3s 2s +50%
mlk_polyvec_mulcache_compute 3s 5s -40%
mlk_polyvec_ntt 3s 5s -40%
mlk_polyvec_permute_bitrev_to_custom_native 3s 2s +50%
mlk_polyvec_reduce 3s 4s -25%
mlk_polyvec_tobytes 3s 4s -25%
mlk_scalar_decompress_d11 3s 2s +50%
mlk_scalar_decompress_d5 3s 1s +200%
mlk_shake128_squeezeblocks 3s 2s +50%
poly_compress_d11_native_x86_64 3s 3s +0%
poly_compress_d4_native_x86_64 3s 3s +0%
poly_compress_d5_native_x86_64 3s 2s +50%
poly_invntt_tomont_native 3s 3s +0%
poly_tobytes_native_aarch64 3s 2s +50%
poly_tomont_native_x86_64 3s 1s +200%
polyvec_basemul_acc_montgomery_cached_k3_native_x86_64 3s 2s +50%
rej_uniform_native_x86_64 3s 3s +0%
sys_check_capability 3s 1s +200%
intt_native_aarch64 2s 3s -33%
keccak_f1600_x4_native_aarch64_v84a 2s 2s +0%
keccak_f1600_x4_native_aarch64_v8a_v84a_scalar_hybrid 2s 2s +0%
keccak_f1600_x4_native_avx2 2s 1s +100%
keccakf1600x4_extract_bytes_native 2s 2s +0%
kem_check_pk 2s 3s -33%
kem_check_sk 2s 3s -33%
kem_keypair 2s 3s -33%
kem_keypair_derand 2s 4s -50%
mlk_barrett_reduce 2s 3s -33%
mlk_check_pct 2s 3s -33%
mlk_ct_cmask_neg_i16 2s 2s +0%
mlk_ct_cmask_nonzero_u16 2s 1s +100%
mlk_ct_cmov_zero 2s 2s +0%
mlk_ct_get_optblocker_u32 2s 3s -33%
mlk_ct_sel_int16 2s 3s -33%
mlk_keccakf1600_permute 2s 3s -33%
mlk_keccakf1600x4_permute 2s 1s +100%
mlk_keccakf1600x4_xor_bytes 2s 1s +100%
mlk_montgomery_reduce 2s 3s -33%
mlk_poly_cbd_eta1 2s 2s +0%
mlk_poly_compress_d10 2s 2s +0%
mlk_poly_compress_d11_native 2s 2s +0%
mlk_poly_compress_d4_c 2s 6s -67%
mlk_poly_compress_d5 2s 1s +100%
mlk_poly_compress_dv 2s 1s +100%
mlk_poly_decompress_d11_c 2s 3s -33%
mlk_poly_decompress_d11_native 2s 2s +0%
mlk_poly_decompress_d4_c 2s 1s +100%
mlk_poly_decompress_d5 2s 2s +0%
mlk_poly_decompress_d5_native 2s 2s +0%
mlk_poly_decompress_du 2s 2s +0%
mlk_poly_frombytes 2s 2s +0%
mlk_poly_getnoise_eta2 2s 2s +0%
mlk_poly_mulcache_compute_native 2s 2s +0%
mlk_poly_ntt 2s 3s -33%
mlk_poly_reduce 2s 3s -33%
mlk_poly_sub 2s 1s +100%
mlk_poly_tobytes_native 2s 2s +0%
mlk_poly_tomont 2s 2s +0%
mlk_poly_tomont_native 2s 4s -50%
mlk_poly_tomsg 2s 1s +100%
mlk_polyvec_invntt_tomont 2s 3s -33%
mlk_rej_uniform 2s 3s -33%
mlk_scalar_compress_d1 2s 3s -33%
mlk_scalar_compress_d11 2s 1s +100%
mlk_scalar_compress_d4 2s 3s -33%
mlk_scalar_decompress_d10 2s 3s -33%
mlk_scalar_decompress_d4 2s 2s +0%
mlk_scalar_signed_to_unsigned_q 2s 4s -50%
mlk_sha3_256 2s 2s +0%
mlk_sha3_512 2s 2s +0%
mlk_shake128_absorb_once 2s 2s +0%
mlk_shake128x4_squeezeblocks 2s 2s +0%
mlk_value_barrier_i32 2s 3s -33%
mlk_value_barrier_u8 2s 3s -33%
ntt_native_aarch64 2s 2s +0%
ntt_native_x86_64 2s 2s +0%
nttunpack_native_x86_64 2s 4s -50%
poly_compress_d10_native_x86_64 2s 2s +0%
poly_mulcache_compute_native_aarch64 2s 2s +0%
poly_reduce_native_aarch64 2s 2s +0%
poly_reduce_native_x86_64 2s 1s +100%
poly_tobytes_native_x86_64 2s 3s -33%
poly_tomont_native_aarch64 2s 2s +0%
polyvec_basemul_acc_montgomery_cached_k2_native_aarch64 2s 1s +100%
polyvec_basemul_acc_montgomery_cached_k2_native_x86_64 2s 3s -33%
polyvec_basemul_acc_montgomery_cached_k3_native_aarch64 2s 4s -50%
polyvec_basemul_acc_montgomery_cached_k4_native_x86_64 2s 5s -60%
rej_uniform_native_aarch64 2s 3s -33%
keccak_f1600_x4_native_aarch64_v8a_scalar_hybrid 1s 1s +0%
mlk_ct_get_optblocker_i32 1s 2s -50%
mlk_ct_memcmp 1s 2s -50%
mlk_ct_sel_uint8 1s 2s -50%
mlk_keccakf1600_extract_bytes (big endian) 1s 1s +0%
mlk_keccakf1600_xor_bytes (big endian) 1s 1s +0%
mlk_keypair_getnoise_eta1 1s 3s -67%
mlk_poly_compress_d11 1s 3s -67%
mlk_poly_compress_d11_c 1s 1s +0%
mlk_poly_compress_d4_native 1s 3s -67%
mlk_poly_compress_d5_native 1s 3s -67%
mlk_poly_compress_du 1s 2s -50%
mlk_poly_decompress_d10 1s 1s +0%
mlk_poly_decompress_dv 1s 1s +0%
mlk_poly_invntt_tomont 1s 2s -50%
mlk_poly_invntt_tomont_c 1s 1s +0%
mlk_poly_tomont_c 1s 4s -75%
mlk_polyvec_basemul_acc_montgomery_cached 1s 2s -50%
mlk_polyvec_tomont 1s 2s -50%
mlk_scalar_compress_d5 1s 3s -67%
mlk_shake128x4_absorb_once 1s 2s -50%
mlk_shake256 1s 1s +0%
poly_decompress_d11_native_x86_64 1s 1s +0%

@oqs-bot
Copy link
Copy Markdown
Contributor

oqs-bot commented Mar 23, 2026

CBMC Results (ML-KEM-768)

Full Results (187 proofs)
Proof Status Current Previous Change
**TOTAL** 1441s 1386s +4.0%
mlk_indcpa_enc 267s 250s +7%
mlk_indcpa_keypair_derand 214s 190s +13%
mlk_keccak_squeezeblocks_x4 121s 118s +3%
mlk_rej_uniform_c 71s 65s +9%
polyvec_basemul_acc_montgomery_cached_native 61s 57s +7%
mlk_polyvec_basemul_acc_montgomery_cached_c 48s 48s +0%
mlk_poly_rej_uniform 35s 32s +9%
mlk_polyvec_add 27s 26s +4%
poly_ntt_native 24s 22s +9%
keccakf1600x4_permute_native_x4 18s 21s -14%
mlk_poly_decompress_d4_native 15s 13s +15%
mlk_poly_reduce_native 15s 14s +7%
mlk_poly_decompress_d10_native 13s 10s +30%
mlk_indcpa_dec 12s 13s -8%
mlk_ntt_layer 12s 11s +9%
mlk_poly_frommsg 11s 12s -8%
mlk_keccak_absorb_once_x4 9s 9s +0%
mlk_poly_frombytes_native 9s 7s +29%
mlk_fqmul 8s 7s +14%
mlk_ntt_butterfly_block 8s 8s +0%
keccakf1600_permute_native 7s 7s +0%
mlk_keccak_squeezeblocks 7s 6s +17%
mlk_invntt_layer 6s 8s -25%
mlk_keccak_squeeze_once 6s 8s -25%
mlk_poly_add 6s 6s +0%
mlk_poly_rej_uniform_x4 6s 8s -25%
mlk_polymat_permute_bitrev_to_custom 6s 4s +50%
mlk_polyvec_mulcache_compute 6s 2s +200%
poly_frombytes_native_x86_64 6s 3s +100%
keccak_f1600_x1_native_aarch64_v84a 5s 2s +150%
kem_dec 5s 5s +0%
mlk_barrett_reduce 5s 2s +150%
mlk_ct_cmask_nonzero_u16 5s 4s +25%
mlk_keccakf1600_permute 5s 3s +67%
mlk_polyvec_permute_bitrev_to_custom_native 5s 3s +67%
keccakf1600x4_xor_bytes_native 4s 4s +0%
kem_enc_derand 4s 3s +33%
mlk_gen_matrix 4s 2s +100%
mlk_keccak_absorb_once 4s 4s +0%
mlk_keccakf1600_extract_bytes 4s 2s +100%
mlk_matvec_mul 4s 2s +100%
mlk_montgomery_reduce 4s 2s +100%
mlk_poly_cbd_eta1 4s 2s +100%
mlk_poly_compress_d11_native 4s 1s +300%
mlk_poly_decompress_d10 4s 1s +300%
mlk_poly_decompress_d10_c 4s 3s +33%
mlk_poly_decompress_d11 4s 3s +33%
mlk_poly_decompress_d4 4s 1s +300%
mlk_poly_getnoise_eta1122_4x 4s 3s +33%
mlk_poly_getnoise_eta1_4x 4s 7s -43%
mlk_poly_getnoise_eta2 4s 3s +33%
mlk_poly_mulcache_compute_c 4s 3s +33%
mlk_polyvec_frombytes 4s 2s +100%
mlk_polyvec_ntt 4s 4s +0%
mlk_scalar_compress_d1 4s 1s +300%
poly_decompress_d10_native_x86_64 4s 4s +0%
poly_mulcache_compute_native_aarch64 4s 3s +33%
intt_native_x86_64 3s 4s -25%
keccak_f1600_x1_native_aarch64 3s 2s +50%
keccak_f1600_x4_native_aarch64_v84a 3s 2s +50%
keccak_f1600_x4_native_aarch64_v8a_v84a_scalar_hybrid 3s 1s +200%
kem_keypair 3s 3s +0%
mlk_ct_memcmp 3s 2s +50%
mlk_ct_sel_int16 3s 4s -25%
mlk_keccakf1600x4_permute 3s 3s +0%
mlk_poly_compress_d10_c 3s 3s +0%
mlk_poly_compress_d11 3s 1s +200%
mlk_poly_compress_d11_c 3s 3s +0%
mlk_poly_compress_d5_c 3s 1s +200%
mlk_poly_compress_du 3s 2s +50%
mlk_poly_decompress_d5 3s 1s +200%
mlk_poly_mulcache_compute 3s 3s +0%
mlk_poly_mulcache_compute_native 3s 2s +50%
mlk_poly_tomsg 3s 4s -25%
mlk_polyvec_basemul_acc_montgomery_cached 3s 3s +0%
mlk_polyvec_invntt_tomont 3s 2s +50%
mlk_polyvec_tomont 3s 2s +50%
mlk_scalar_compress_d11 3s 3s +0%
mlk_scalar_compress_d5 3s 3s +0%
mlk_scalar_decompress_d10 3s 3s +0%
mlk_scalar_decompress_d11 3s 2s +50%
mlk_shake128_absorb_once 3s 3s +0%
mlk_shake128x4_squeezeblocks 3s 1s +200%
mlk_shake256 3s 2s +50%
mlk_shake256x4 3s 5s -40%
mlk_value_barrier_i32 3s 2s +50%
ntt_native_x86_64 3s 4s -25%
nttunpack_native_x86_64 3s 3s +0%
poly_compress_d10_native_x86_64 3s 2s +50%
poly_compress_d4_native_x86_64 3s 3s +0%
poly_compress_d5_native_x86_64 3s 4s -25%
poly_decompress_d4_native_x86_64 3s 3s +0%
poly_mulcache_compute_native_x86_64 3s 3s +0%
poly_reduce_native_x86_64 3s 3s +0%
polyvec_basemul_acc_montgomery_cached_k4_native_aarch64 3s 2s +50%
polyvec_basemul_acc_montgomery_cached_k4_native_x86_64 3s 2s +50%
rej_uniform_native_x86_64 3s 3s +0%
sys_check_capability 3s 1s +200%
intt_native_aarch64 2s 2s +0%
keccakf1600x4_extract_bytes_native 2s 3s -33%
kem_check_pk 2s 4s -50%
kem_check_sk 2s 2s +0%
kem_enc 2s 1s +100%
kem_keypair_derand 2s 1s +100%
mlk_ct_cmask_neg_i16 2s 2s +0%
mlk_ct_cmov_zero 2s 2s +0%
mlk_ct_get_optblocker_u32 2s 3s -33%
mlk_ct_sel_uint8 2s 4s -50%
mlk_gen_matrix_serial 2s 5s -60%
mlk_keccakf1600_extract_bytes (big endian) 2s 2s +0%
mlk_keccakf1600_xor_bytes 2s 3s -33%
mlk_keccakf1600_xor_bytes (big endian) 2s 2s +0%
mlk_keccakf1600x4_extract_bytes 2s 2s +0%
mlk_poly_cbd_eta2 2s 1s +100%
mlk_poly_compress_d10_native 2s 4s -50%
mlk_poly_compress_d4 2s 3s -33%
mlk_poly_compress_d4_native 2s 2s +0%
mlk_poly_compress_d5 2s 2s +0%
mlk_poly_compress_dv 2s 2s +0%
mlk_poly_decompress_d5_c 2s 4s -50%
mlk_poly_decompress_du 2s 2s +0%
mlk_poly_getnoise_eta1_4x_native 2s 4s -50%
mlk_poly_invntt_tomont 2s 3s -33%
mlk_poly_invntt_tomont_c 2s 2s +0%
mlk_poly_ntt 2s 3s -33%
mlk_poly_ntt_c 2s 3s -33%
mlk_poly_reduce 2s 1s +100%
mlk_poly_sub 2s 2s +0%
mlk_poly_tobytes_c 2s 2s +0%
mlk_poly_tobytes_native 2s 2s +0%
mlk_poly_tomont_native 2s 3s -33%
mlk_polyvec_compress_du 2s 2s +0%
mlk_polyvec_decompress_du 2s 3s -33%
mlk_polyvec_permute_bitrev_to_custom 2s 2s +0%
mlk_polyvec_reduce 2s 3s -33%
mlk_scalar_compress_d10 2s 2s +0%
mlk_scalar_signed_to_unsigned_q 2s 2s +0%
mlk_sha3_256 2s 3s -33%
mlk_sha3_512 2s 2s +0%
mlk_value_barrier_u32 2s 2s +0%
ntt_native_aarch64 2s 3s -33%
poly_compress_d11_native_x86_64 2s 2s +0%
poly_decompress_d11_native_x86_64 2s 3s -33%
poly_decompress_d5_native_x86_64 2s 3s -33%
poly_getnoise_eta1122_4x_native 2s 1s +100%
poly_invntt_tomont_native 2s 1s +100%
poly_reduce_native_aarch64 2s 3s -33%
poly_tobytes_native_aarch64 2s 3s -33%
poly_tomont_native_aarch64 2s 2s +0%
polyvec_basemul_acc_montgomery_cached_k2_native_aarch64 2s 2s +0%
polyvec_basemul_acc_montgomery_cached_k2_native_x86_64 2s 3s -33%
polyvec_basemul_acc_montgomery_cached_k3_native_aarch64 2s 1s +100%
polyvec_basemul_acc_montgomery_cached_k3_native_x86_64 2s 2s +0%
rej_uniform_native 2s 2s +0%
rej_uniform_native_aarch64 2s 5s -60%
keccak_f1600_x4_native_aarch64_v8a_scalar_hybrid 1s 3s -67%
keccak_f1600_x4_native_avx2 1s 1s +0%
mlk_check_pct 1s 3s -67%
mlk_ct_cmask_nonzero_u8 1s 2s -50%
mlk_ct_get_optblocker_i32 1s 1s +0%
mlk_ct_get_optblocker_u8 1s 4s -75%
mlk_keccakf1600x4_xor_bytes 1s 3s -67%
mlk_keypair_getnoise_eta1 1s 4s -75%
mlk_poly_compress_d10 1s 2s -50%
mlk_poly_compress_d4_c 1s 3s -67%
mlk_poly_compress_d5_native 1s 1s +0%
mlk_poly_decompress_d11_c 1s 3s -67%
mlk_poly_decompress_d11_native 1s 1s +0%
mlk_poly_decompress_d4_c 1s 2s -50%
mlk_poly_decompress_d5_native 1s 2s -50%
mlk_poly_decompress_dv 1s 3s -67%
mlk_poly_frombytes 1s 3s -67%
mlk_poly_frombytes_c 1s 3s -67%
mlk_poly_reduce_c 1s 1s +0%
mlk_poly_tobytes 1s 3s -67%
mlk_poly_tomont 1s 3s -67%
mlk_poly_tomont_c 1s 1s +0%
mlk_polyvec_tobytes 1s 2s -50%
mlk_rej_uniform 1s 2s -50%
mlk_scalar_compress_d4 1s 2s -50%
mlk_scalar_decompress_d4 1s 3s -67%
mlk_scalar_decompress_d5 1s 3s -67%
mlk_shake128_squeezeblocks 1s 1s +0%
mlk_shake128x4_absorb_once 1s 4s -75%
mlk_value_barrier_u8 1s 2s -50%
poly_tobytes_native_x86_64 1s 2s -50%
poly_tomont_native_x86_64 1s 4s -75%

@oqs-bot
Copy link
Copy Markdown
Contributor

oqs-bot commented Mar 23, 2026

CBMC Results (ML-KEM-1024)

Full Results (187 proofs)
Proof Status Current Previous Change
**TOTAL** 1381s 1500s -7.9%
mlk_indcpa_enc 250s 264s -5%
polyvec_basemul_acc_montgomery_cached_native 125s 132s -5%
mlk_keccak_squeezeblocks_x4 120s 130s -8%
mlk_indcpa_keypair_derand 78s 84s -7%
mlk_rej_uniform_c 69s 82s -16%
mlk_polyvec_basemul_acc_montgomery_cached_c 66s 76s -13%
mlk_poly_rej_uniform 34s 33s +3%
poly_ntt_native 28s 38s -26%
mlk_polyvec_add 26s 24s +8%
keccakf1600x4_permute_native_x4 20s 18s +11%
mlk_indcpa_dec 13s 12s +8%
mlk_ntt_layer 13s 16s -19%
mlk_poly_reduce_native 13s 16s -19%
mlk_polyvec_ntt 13s 14s -7%
mlk_poly_decompress_d11_native 12s 15s -20%
mlk_poly_decompress_d5_native 12s 14s -14%
mlk_poly_rej_uniform_x4 10s 10s +0%
mlk_polymat_permute_bitrev_to_custom 10s 11s -9%
mlk_gen_matrix 9s 9s +0%
mlk_poly_compress_d11_c 9s 11s -18%
mlk_poly_frommsg 9s 11s -18%
kem_dec 8s 9s -11%
mlk_fqmul 8s 6s +33%
mlk_keccak_absorb_once_x4 8s 7s +14%
mlk_keccak_squeezeblocks 8s 6s +33%
mlk_gen_matrix_serial 7s 6s +17%
keccakf1600_permute_native 6s 7s -14%
mlk_keccak_squeeze_once 6s 7s -14%
mlk_keccakf1600_permute 6s 2s +200%
mlk_ntt_butterfly_block 6s 8s -25%
mlk_poly_frombytes_native 6s 10s -40%
poly_decompress_d5_native_x86_64 6s 5s +20%
keccakf1600x4_xor_bytes_native 5s 3s +67%
mlk_invntt_layer 5s 6s -17%
mlk_keypair_getnoise_eta1 5s 3s +67%
mlk_poly_add 5s 6s -17%
mlk_poly_mulcache_compute_c 5s 3s +67%
mlk_poly_tomont_c 5s 2s +150%
mlk_shake128x4_absorb_once 5s 2s +150%
poly_frombytes_native_x86_64 5s 6s -17%
poly_reduce_native_x86_64 5s 2s +150%
kem_check_sk 4s 2s +100%
mlk_check_pct 4s 3s +33%
mlk_ct_cmov_zero 4s 2s +100%
mlk_poly_compress_d11_native 4s 3s +33%
mlk_poly_tobytes_native 4s 3s +33%
mlk_polyvec_tobytes 4s 3s +33%
mlk_polyvec_tomont 4s 1s +300%
mlk_value_barrier_u32 4s 3s +33%
poly_tomont_native_aarch64 4s 2s +100%
rej_uniform_native_x86_64 4s 2s +100%
intt_native_x86_64 3s 2s +50%
keccak_f1600_x4_native_aarch64_v8a_scalar_hybrid 3s 3s +0%
keccak_f1600_x4_native_avx2 3s 2s +50%
kem_check_pk 3s 5s -40%
mlk_barrett_reduce 3s 3s +0%
mlk_ct_cmask_nonzero_u16 3s 5s -40%
mlk_ct_cmask_nonzero_u8 3s 5s -40%
mlk_ct_get_optblocker_u8 3s 2s +50%
mlk_ct_sel_int16 3s 2s +50%
mlk_keccakf1600x4_xor_bytes 3s 2s +50%
mlk_matvec_mul 3s 2s +50%
mlk_poly_compress_d10_native 3s 2s +50%
mlk_poly_compress_d11 3s 2s +50%
mlk_poly_compress_d4_c 3s 3s +0%
mlk_poly_compress_d5 3s 3s +0%
mlk_poly_decompress_d11_c 3s 2s +50%
mlk_poly_decompress_d4 3s 1s +200%
mlk_poly_decompress_d5_c 3s 4s -25%
mlk_poly_decompress_dv 3s 3s +0%
mlk_poly_getnoise_eta1122_4x 3s 2s +50%
mlk_poly_getnoise_eta1_4x 3s 5s -40%
mlk_poly_getnoise_eta1_4x_native 3s 3s +0%
mlk_poly_mulcache_compute_native 3s 1s +200%
mlk_poly_ntt_c 3s 3s +0%
mlk_poly_reduce_c 3s 2s +50%
mlk_poly_tomont_native 3s 1s +200%
mlk_polyvec_decompress_du 3s 2s +50%
mlk_polyvec_permute_bitrev_to_custom 3s 3s +0%
mlk_polyvec_permute_bitrev_to_custom_native 3s 4s -25%
mlk_rej_uniform 3s 4s -25%
mlk_scalar_compress_d11 3s 2s +50%
mlk_scalar_decompress_d11 3s 2s +50%
mlk_shake256x4 3s 3s +0%
poly_compress_d10_native_x86_64 3s 2s +50%
poly_compress_d5_native_x86_64 3s 4s -25%
poly_decompress_d11_native_x86_64 3s 8s -62%
poly_invntt_tomont_native 3s 3s +0%
poly_mulcache_compute_native_x86_64 3s 1s +200%
poly_reduce_native_aarch64 3s 2s +50%
poly_tobytes_native_aarch64 3s 2s +50%
polyvec_basemul_acc_montgomery_cached_k2_native_x86_64 3s 2s +50%
rej_uniform_native 3s 4s -25%
rej_uniform_native_aarch64 3s 4s -25%
intt_native_aarch64 2s 3s -33%
keccak_f1600_x4_native_aarch64_v8a_v84a_scalar_hybrid 2s 2s +0%
keccakf1600x4_extract_bytes_native 2s 3s -33%
kem_enc 2s 2s +0%
kem_enc_derand 2s 3s -33%
kem_keypair 2s 2s +0%
mlk_ct_cmask_neg_i16 2s 3s -33%
mlk_ct_get_optblocker_u32 2s 3s -33%
mlk_ct_memcmp 2s 3s -33%
mlk_ct_sel_uint8 2s 1s +100%
mlk_keccak_absorb_once 2s 5s -60%
mlk_keccakf1600_extract_bytes 2s 3s -33%
mlk_keccakf1600_xor_bytes 2s 1s +100%
mlk_keccakf1600_xor_bytes (big endian) 2s 3s -33%
mlk_keccakf1600x4_extract_bytes 2s 2s +0%
mlk_poly_cbd_eta1 2s 3s -33%
mlk_poly_cbd_eta2 2s 4s -50%
mlk_poly_compress_d10_c 2s 2s +0%
mlk_poly_compress_d4 2s 4s -50%
mlk_poly_compress_d4_native 2s 2s +0%
mlk_poly_compress_d5_c 2s 1s +100%
mlk_poly_compress_d5_native 2s 4s -50%
mlk_poly_compress_dv 2s 2s +0%
mlk_poly_decompress_d10_native 2s 1s +100%
mlk_poly_decompress_d11 2s 2s +0%
mlk_poly_decompress_d4_c 2s 2s +0%
mlk_poly_decompress_d5 2s 1s +100%
mlk_poly_decompress_du 2s 2s +0%
mlk_poly_frombytes 2s 2s +0%
mlk_poly_frombytes_c 2s 1s +100%
mlk_poly_invntt_tomont 2s 2s +0%
mlk_poly_invntt_tomont_c 2s 1s +100%
mlk_poly_reduce 2s 2s +0%
mlk_poly_sub 2s 3s -33%
mlk_poly_tobytes 2s 3s -33%
mlk_polyvec_basemul_acc_montgomery_cached 2s 2s +0%
mlk_polyvec_compress_du 2s 2s +0%
mlk_polyvec_frombytes 2s 3s -33%
mlk_polyvec_invntt_tomont 2s 3s -33%
mlk_polyvec_mulcache_compute 2s 3s -33%
mlk_polyvec_reduce 2s 2s +0%
mlk_scalar_compress_d4 2s 3s -33%
mlk_scalar_decompress_d10 2s 2s +0%
mlk_scalar_decompress_d5 2s 4s -50%
mlk_sha3_256 2s 2s +0%
mlk_shake128_absorb_once 2s 1s +100%
mlk_shake128x4_squeezeblocks 2s 2s +0%
mlk_shake256 2s 3s -33%
mlk_value_barrier_u8 2s 1s +100%
ntt_native_aarch64 2s 3s -33%
ntt_native_x86_64 2s 5s -60%
nttunpack_native_x86_64 2s 4s -50%
poly_compress_d4_native_x86_64 2s 2s +0%
poly_decompress_d10_native_x86_64 2s 2s +0%
poly_getnoise_eta1122_4x_native 2s 2s +0%
poly_mulcache_compute_native_aarch64 2s 1s +100%
poly_tobytes_native_x86_64 2s 3s -33%
polyvec_basemul_acc_montgomery_cached_k3_native_aarch64 2s 3s -33%
polyvec_basemul_acc_montgomery_cached_k4_native_x86_64 2s 3s -33%
sys_check_capability 2s 2s +0%
keccak_f1600_x1_native_aarch64 1s 3s -67%
keccak_f1600_x1_native_aarch64_v84a 1s 4s -75%
keccak_f1600_x4_native_aarch64_v84a 1s 1s +0%
kem_keypair_derand 1s 3s -67%
mlk_ct_get_optblocker_i32 1s 2s -50%
mlk_keccakf1600_extract_bytes (big endian) 1s 1s +0%
mlk_keccakf1600x4_permute 1s 1s +0%
mlk_montgomery_reduce 1s 3s -67%
mlk_poly_compress_d10 1s 3s -67%
mlk_poly_compress_du 1s 3s -67%
mlk_poly_decompress_d10 1s 4s -75%
mlk_poly_decompress_d10_c 1s 2s -50%
mlk_poly_decompress_d4_native 1s 2s -50%
mlk_poly_getnoise_eta2 1s 4s -75%
mlk_poly_mulcache_compute 1s 1s +0%
mlk_poly_ntt 1s 4s -75%
mlk_poly_tobytes_c 1s 2s -50%
mlk_poly_tomont 1s 2s -50%
mlk_poly_tomsg 1s 2s -50%
mlk_scalar_compress_d1 1s 3s -67%
mlk_scalar_compress_d10 1s 3s -67%
mlk_scalar_compress_d5 1s 2s -50%
mlk_scalar_decompress_d4 1s 2s -50%
mlk_scalar_signed_to_unsigned_q 1s 3s -67%
mlk_sha3_512 1s 3s -67%
mlk_shake128_squeezeblocks 1s 2s -50%
mlk_value_barrier_i32 1s 3s -67%
poly_compress_d11_native_x86_64 1s 2s -50%
poly_decompress_d4_native_x86_64 1s 2s -50%
poly_tomont_native_x86_64 1s 4s -75%
polyvec_basemul_acc_montgomery_cached_k2_native_aarch64 1s 1s +0%
polyvec_basemul_acc_montgomery_cached_k3_native_x86_64 1s 2s -50%
polyvec_basemul_acc_montgomery_cached_k4_native_aarch64 1s 3s -67%

@mkannwischer mkannwischer marked this pull request as ready for review March 23, 2026 06:15
@mkannwischer mkannwischer requested a review from a team as a code owner March 23, 2026 06:15
The encaps and combined test functions asserted that K should NOT match
the expected value for invalid test cases where the binary succeeded.
This is wrong: for invalid inputs, the binary should reject the input
via input validation, not succeed and produce a different K.

Fix this by restructuring all test functions to branch on tc["result"]
first, then verify the binary's behavior:
- "invalid": assert the binary rejected the input (_error or decode_error)
- "valid": assert outputs match expected values
- anything else: fail with unexpected result

Additional changes:
- Remove "acceptable" as a result type (not used in any test vectors)
- In keygen_seed tests, only "valid" results exist; simplify accordingly
- In semi_expanded_decaps tests, assert that invalid tests are rejected
  rather than silently ignoring errors on valid tests
- In combined tests, distinguish keygen decode errors from runtime errors,
  and always verify ek when keygen succeeds

Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
tc["result"] == "invalid"
), f"binary error on non-invalid tcId={tc['tcId']}"
elif tc["result"] in ("valid", "acceptable"):
"_error" in out or "decode_error" in out
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the "_error" branch supposed to catch in addition to decode_error?

out["K"].upper() != tc["K"].upper()
), f"K should not match for invalid tcId={tc['tcId']}"
False
), f"Unexpected result '{tc['result']}' for tcId={tc['tcId']}"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slightly misleading. "Unexpected result" to me suggests that something was wrong with the client's output -- but instead, we simply don't understand the test case?

out["K"].upper() != tc["K"].upper()
), f"K should not match for invalid tcId={tc['tcId']}"
False
), f"Unexpected result '{tc['result']}' for tcId={tc['tcId']}"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
), f"Unexpected result '{tc['result']}' for tcId={tc['tcId']}"
), f"Unsupported test result '{tc['result']}' for tcId={tc['tcId']}"

Copy link
Copy Markdown
Contributor

@hanno-becker hanno-becker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit, otherwise it looks good. Thank you very much for catching this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants