Skip to content

Include constant-time analysis framework#2449

Open
pablo-gf wants to merge 15 commits into
open-quantum-safe:mainfrom
pablo-gf:ct-tooling
Open

Include constant-time analysis framework#2449
pablo-gf wants to merge 15 commits into
open-quantum-safe:mainfrom
pablo-gf:ct-tooling

Conversation

@pablo-gf

Copy link
Copy Markdown
Contributor

This PR includes the constant-time testing framework developed throughout the course of the LFX Mentorship project supervised by @bhess. These are the main features of this framework:

  • It is based on two dynamic analysis tools: Valgrind's memcheck with the Kyberslash patch (renamed to Valgrind-Varlat), and Clang's MemSanitizer (MemSan) to detect possible sources (warnigns) of non-constant-time behavior.
  • Provides unified scripts for running constant-time checks both in the CI pipeline and locally.
  • Includes a script that aggregates analysis outputs and generates visual graphs to identify and track constant-time warning trends.
  • Includes several documentation files outlining the installation, setup, and usage process.

I would be happy to walk through the details of any of these features either here or during a status call. You can also find detailed guides in the documentation files I’ve added under tests/ct_tooling and its subdirectories. Feel free to throw any feedback or comments so that we can move this from draft to "ready for review"!

  • Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
  • Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged. Also, make sure to update the list of algorithms in the continuous benchmarking files: .github/workflows/kem-bench.yml and sig-bench.yml)

AI assistance was used to generate documentation and minor editing. The resulting code was reviewed, edited and verified for accuracy.

Comment thread .github/workflows/ct-tooling.yml Fixed
Comment thread .github/workflows/ct-tooling.yml Fixed
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@coveralls

coveralls commented May 29, 2026

Copy link
Copy Markdown

Coverage Status

Coverage is 82.185%pablo-gf:ct-tooling into open-quantum-safe:main. No base build found for open-quantum-safe:main.

@bhess bhess left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @pablo-gf for this work on improving the constant‑time tooling in liboqs, this is an important step towards more exhaustive ct-testing in liboqs.

I’ve left a few initial comments inline.

I would be happy to walk through the details of any of these features either here or during a status call.

That would be excellent. @dstebila / @RodriM11: could we allocate some time in an upcoming status call for @pablo-gf to walk us through this contribution? I think a walkthrough would be very valuable.

Comment thread .github/workflows/ct-tooling.yml Outdated
workflow_dispatch:

jobs:
interactive-inputs:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that this job requires interactive input to select the algorithms. Is that correct? If so, what is the intended usage, and can it be configured to run non‑interactively (e.g., for inclusion in the weekly CI runs)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is correct. The original idea was to execute these tests selectively—like when introducing a new algorithm or modifying the implementation of one. In a previous discussion with @dstebila, we noted the importance of filtering out false-positive warnings to prevent spreading false alarms. I went for the interactive-input setup so that we can execute CT tests only on those algorithms that have gone through false-positive exclusion. If we want to include this testing in CI, we can make it non-interactive, but that would require hard-coding the algorithms that went through false positive exclusion. Another option is simply running the tests on all algorithms even if that means retrieving many warnings that may be false-positives. However, this may require too many resources due to the wide range of variables handled by the framework, which was another reason why I went for the interactive-input setup. I think it comes down to what we want the ultimate purpose of this framework to be in liboqs. Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?

IMO both - properly configurable.

Comment thread .github/workflows/ct-tooling.yml Outdated
display: Select the algorithm(s) to execute valgrind-varlat constant-time testing on
type: multiselect
choices:
- "BIKE-L1"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern with this list is maintainability: if the set of algorithms in liboqs changes, the list would need to be updated manually. Is there a way to generate this list dynamically based on the algorithms currently available?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for pointing this out. We could fix this by retriving all available algorithms using the available_kems_by_name() and available_sigs_by_name() from helpers.py. I have included it in the latest commit, and the tests are working on my fork.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think about it, I could also create a separate PR to fix this same issue for the kem-bench.yaml and sig-bench.yaml workflows with the same solution, if you find that appropiate.

@@ -0,0 +1,396 @@
// SPDX-License-Identifier: MIT

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test_sig.c / test_kem.c files and the corresponding CMakeLists.txt appear to be largely copied from the versions in the /tests directory. Could we avoid duplicating these files and instead reuse the originals with additional macro definitions (e.g., to disable signature verification during ct-testing)? Or is there a specific reason why maintaining separate copies is necessary here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think that sounds like a better approach over the preliminary copy-pasting approach. I have updated it in the last commit and the tests are passing in my fork.

@@ -0,0 +1,15 @@
fun:PQCP_MLKEM_NATIVE_MLKEM*_C_poly_decompress_d*

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this approach, having an allowlist mechanism similar to the Valgrind one makes a lot of sense.

Comment thread .github/workflows/ct-tooling.yml Outdated
INSTALL_PREFIX="$PWD/valgrind_varlat"

echo "Cloning Valgrind's source code"
git clone git://sourceware.org/git/valgrind.git valgrind_varlat_src> /dev/null 2>&1 || true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statements with || true would silently fail.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Forgot to remove these after testing. Should I also remove > /dev/null 2>&1? I initially kept them so that the output in GitHub Actions looks cleaner and it is easier to spot the pass/fail of tests, but we can keep the output if preferred.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO if the output is helpful for debugging it should be kept, if not it could be sent to dev/null

@RodriM11

Copy link
Copy Markdown
Member

That would be excellent. @dstebila / @RodriM11: could we allocate some time in an upcoming status call for @pablo-gf to walk us through this contribution? I think a walkthrough would be very valuable.

For sure. Next week is TSC meeting, so we could do it the 9th of June, if @pablo-gf is available.

Comment thread tests/ct_tooling/ct_test.sh Outdated
done

# Create backup files of the original tests files
mv "$LIBOQS_DIR/tests/CMakeLists.txt" "$LIBOQS_DIR/tests/CMakeLists.txt.bak"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks brittle since it makes assumptions about the liboqs file structure. IMO avoiding replacing the original files and instead work CT-specific macros in the original files would be more robust.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed per the previous comment.

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@pablo-gf

pablo-gf commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

I’ve left a few initial comments inline.

Thank you for your comments, @bhess. Let me know if you have any further ideas/suggestions.

For sure. Next week is TSC meeting, so we could do it the 9th of June, if @pablo-gf is available.

Sounds good @RodriM11. See you next week then!

pablo-gf added 4 commits June 2, 2026 10:57
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Comment thread .github/workflows/ct-tooling.yml Fixed
Comment thread .github/workflows/ct-tooling.yml Fixed
Comment thread .github/workflows/ct-tooling.yml Fixed
@pablo-gf

pablo-gf commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

A couple of questions, @bhess:

With respect to the bot warnings: Is the "verified creator" an issue in this case? And would the pip command also require a pin by hash here?

Also, the failing job in CI seems to be a network issue during the docker pull. Any chance that could be re-executed?

@bhess

bhess commented Jun 3, 2026

Copy link
Copy Markdown
Member

With respect to the bot warnings: Is the "verified creator" an issue in this case? And would the pip command also require a pin by hash here?

I’m not entirely sure what to make of the “verified creator” warning, but it may have security implications. From what I can tell, this action is mainly meant for interactive use.
As you noted in the inline comment, this shouldn’t be necessary for regular CI runs if we run it non‑interactively, does that sound right?
For manual runs, could we instead trigger the action manually and pass the required parameters (the algorithm to test) explicitly?

Also, the failing job in CI seems to be a network issue during the docker pull. Any chance that could be re-executed?

Done

pablo-gf added 2 commits June 6, 2026 21:15
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@pablo-gf

pablo-gf commented Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

I’m not entirely sure what to make of the “verified creator” warning, but it may have security implications. From what I can tell, this action is mainly meant for interactive use. As you noted in the inline comment, this shouldn’t be necessary for regular CI runs if we run it non‑interactively, does that sound right? For manual runs, could we instead trigger the action manually and pass the required parameters (the algorithm to test) explicitly?

Sounds good, @bhess. I looked into it and we could also implement it using the functionality provided by Github Actions inputs, rather than this external party. Maybe this could simplify the whole thing for both CI tests and manual tests.
How would you like it to be included (weekly tests, release tests...)?
What algorithms should be included? I set the ML-KEM variants as default for now.

@pablo-gf pablo-gf force-pushed the ct-tooling branch 2 times, most recently from 5a2abdf to 86d566d Compare June 6, 2026 19:45
Comment thread .github/workflows/ct-tooling.yml Fixed
Comment thread .github/workflows/ct-tooling.yml Fixed
Comment thread .github/workflows/ct-tooling.yml Fixed
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
pablo-gf added 2 commits June 8, 2026 09:38
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@bhess

bhess commented Jun 8, 2026

Copy link
Copy Markdown
Member

Sounds good, @bhess. I looked into it and we could also implement it using the functionality provided by Github Actions inputs, rather than this external party. Maybe this could simplify the whole thing for both CI tests and manual tests.
How would you like it to be included (weekly tests, release tests...)?

Thanks for the updates @pablo-gf and for looking into this. I think it would be great to rely on the CI usage guidelines for this. I assume the CI tests will fall here:

If the limits (1 hour) are exceeded, consider:
Adding parallel jobs.
Moving longer-running jobs to weekly workflows (up to 2 hours).
Moving jobs running longer than 2 hours to tests only triggered manually.
All workflows, including those only triggered manually, should be run before a release.

Ideally, we would first try to add parallel jobs so that each job stays within the 2-hour limit and can be included in the weekly workflow.
If that's not feasible, then running the tests manually (on demand or before a release) would be the fallback.
It would be great to support both use cases (manual runs and regular CI), for example by using GitHub Actions inputs as you suggested.

What algorithms should be included? I set the ML-KEM variants as default for now.

I'd recommend adding those that are triaged.

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@pablo-gf

pablo-gf commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the comment @bhess. I have included the framework in the weekly tests, since the latest version executes in around 1 hour. Obviously, this would increase as more algorithms are triaged and included in the testing framework, so this is something to keep in mind for the future. At the moment, it is executing 5 jobs in parallel on all ML-KEM variants, which are those containing suppression files.

Any suggestions on the failing CI tests?

Comment thread .github/workflows/ct-tooling.yml Outdated
pablo-gf added 2 commits June 23, 2026 14:17
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@pablo-gf

pablo-gf commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

I have moved the Kyberslash patch to ci-containers so CI test should be much faster now. I also updated the documentation for false-positive suppression block handling, and merged the previous tooling with this new framework. I essentially combined everything into one single ct_tooling directory, moving the "passes" and "issues" that were already logged with the previous tooling into the new directory structure.

After talking with @bhess, we noticed that the previous tooling used a python script used for testing that is not currently used by the new framework. As a result, I have removed it. If anyone thinks differently, please let us know.

With this, I think it should be ready for review. Let me know if you have any comments or suggestions!

@pablo-gf pablo-gf marked this pull request as ready for review July 3, 2026 11:35

@baentsch baentsch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for the work @pablo-gf ! It seems this needs a do-over to catch up with "latest" developments (alg removal, etc.). Please also see my single comments. I did not look at the underlying logic, but just how this presents itself to users. A more general question I have is how this matches up to what is stated in https://github.com/open-quantum-safe/liboqs/blob/main/SECURITY.md? In particular, how are the results represented in the algorithm datasheets as claimed?

Comment thread tests/ct_tooling/analyze_results.py Outdated
for idx, m in enumerate(matches):
alg_name = m.group(1)
# Skip SPHINCS and SLH_DSA algorithms
if 'SPHINCS' in alg_name or 'SLH_DSA' in alg_name:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought Sphincs has been removed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I also removed its mention from ct-test.sh and the README files.


The `local_testing_example.sh` script demonstrates how to use `ct_test.sh` to run CT tests locally across a variety of compilers, compiler versions, liboqs target builds, and optimization flags.

The `ct-tooling-valgrind-varlat.yml` and `ct-tooling-memsan.yml` workflows also use `ct_test.sh` to execute CT tests in CI on user-selected algorithms (using [interactive-inputs](https://github.com/marketplace/actions/interactive-inputs)) when the workflows are manually triggered on GitHub Actions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm - that begs the question which algorithms are executed when done automatically in CI. Worthwhile stating here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI currently defaults to ML-KEM variants only because they have been thoroughly examined to exclude false-positives. I did not include it in the README because I thought the default algorithms used used for CI could potentially change over time, but let me know if you think we should include it anyways.

Something to keep in mind as we go over the warnings found is that while we can expand these defaults as we want, including a large number of default algorithms would also signifcantly increase the time and resources used during testing.

description: 'Algorithms to execute CT testing on (comma-separated)'
required: true
type: string
default: 'ML-KEM-512,ML-KEM-768,ML-KEM-1024'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, does this mean only these algs are CT-tested now?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For CI, yes. That is correct.

Comment thread tests/ct_tooling/README.md Outdated
- `tool`: `valgrind-varlat` or `memsan`
- `compiler_version`: clang, clang-20, gcc, gcc-14, ...
- `liboqs_build`: `generic` or `auto`
- `input`: `all`, `kems`, `sigs`, or a specific enabled algorithm variant

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's perfect -- is "all" enabled by default? But then how does "all" relate to the exclusion of SLH-DSA?

@pablo-gf pablo-gf Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For CI, the default option tests all ML-KEM variants as I mentioned earlier. The "all" option excludes SLH-DSA variants. I forgot to state that in the README. Thanks for noticing.

Comment thread tests/ct_tooling/README.md Outdated
2. **Test execution**
Then, `test()` is tasked with executing the tool's test on selected liboqs algorithms. Each tool has a different process through which it parses the tool's output to keep unique instances of the warnings, which are further detailed in their respective README files: [Valgrind-Varlat's README](tools/valgrind_varlat/README.md) and [MemSan's README](tools/memsan/README.md).

Both tools enforce a warning cap of 100,000 unique warnings per algorithm run. Once the cap is reached, further warnings are suppressed. All SPHINCS and SLH-DSA signature variants are currently skipped during SIG tests due to the excessive time they require to execute.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, does Sphincs warrant mentioning? What about stfl-sigs? They're even worse from a runtime perspective,no?

@pablo-gf pablo-gf Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for noticing, I removed SPHINCS mention. Currently, no tests are executed for stfl-sigs. That is something we would like to enable in the future.

Comment thread tests/CMakeLists.txt
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: MIT

option(OQS_ENABLE_TEST_CONSTANT_TIME "Build test suite with support for Valgrind-based detection of non-constant time behaviour." OFF)
option(OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN "Build test suite with support for MemorySanitizer-based detection of non-constant time behaviour." OFF)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we introduce a variable to enable memsan-only testing and define the previous variable to be Valgrind only testing, users get confused: Does setting "OQS_ENABLE_TEST_CONSTANT_TIME" enable all CT testing or just Valgrind-based testing? Wouldn't it be more sensible to add also a OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND option and re-define OQS_ENABLE_TEST_CONSTANT_TIME to enable both OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND and OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this is a better approach too. In this case, each tool requires a mutually exclusive testing environment since they rely on different compilation and execution frameworks to flag uninitialized secret data, so we would stay with OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND and OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN only. I have updated this in the latest version and documented it in CONFIGURE.md too.

Comment thread tests/test_kem.c Outdated
@@ -296,7 +300,7 @@ static OQS_STATUS kem_test_correctness(const char *method_name, bool derand) {
goto err;
}

#ifndef OQS_ENABLE_TEST_CONSTANT_TIME
#if defined(OQS_ENABLE_TEST_CONSTANT_TIME) || defined(OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shows the problem of bad option definitions. IMO there should only one define be necessary here...

@@ -0,0 +1,169 @@
# Valgrind-Varlat
This directory contains the necessary files to execute Valgrind's memcheck tool on liboqs with [Daniel Bernstein's Kyberslash patches](https://kyberslash.cr.yp.to/papers.html) (valgrind-try-patch-20250805.txt and valgrind-varlat-patch-20250805.txt) and another patch including variable latency warnings in the suppression block (valgrind_varlat_sup_block.txt).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why still this reference to a patch we now have in CI? If just for completeness, what about mentioning that? If no longer needed, why not remove this text?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Thanks for noticing! I rephrased it to simply mention that this repository uses such patches instead. I also included a link to the CI repo in case somebody wants to replicate it locally.

Comment thread .github/workflows/ct-tooling.yml Outdated
workflow_dispatch:

jobs:
interactive-inputs:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?

IMO both - properly configurable.

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
@pablo-gf

pablo-gf commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the comments and suggestions @baentsch. I have fixed and answered all, but let me know if you have any more.

A more general question I have is how this matches up to what is stated in https://github.com/open-quantum-safe/liboqs/blob/main/SECURITY.md? In particular, how are the results represented in the algorithm datasheets as claimed?

Regarding this, we could extend them to include a column for MemSan tests as well. What are your thoughts about this @bhess? How is this constant-time information reflected on the datasheet usually pipelined?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants