Include constant-time analysis framework#2449
Conversation
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
bhess
left a comment
There was a problem hiding this comment.
Thank you @pablo-gf for this work on improving the constant‑time tooling in liboqs, this is an important step towards more exhaustive ct-testing in liboqs.
I’ve left a few initial comments inline.
I would be happy to walk through the details of any of these features either here or during a status call.
That would be excellent. @dstebila / @RodriM11: could we allocate some time in an upcoming status call for @pablo-gf to walk us through this contribution? I think a walkthrough would be very valuable.
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| interactive-inputs: |
There was a problem hiding this comment.
My understanding is that this job requires interactive input to select the algorithms. Is that correct? If so, what is the intended usage, and can it be configured to run non‑interactively (e.g., for inclusion in the weekly CI runs)?
There was a problem hiding this comment.
Yes, that is correct. The original idea was to execute these tests selectively—like when introducing a new algorithm or modifying the implementation of one. In a previous discussion with @dstebila, we noted the importance of filtering out false-positive warnings to prevent spreading false alarms. I went for the interactive-input setup so that we can execute CT tests only on those algorithms that have gone through false-positive exclusion. If we want to include this testing in CI, we can make it non-interactive, but that would require hard-coding the algorithms that went through false positive exclusion. Another option is simply running the tests on all algorithms even if that means retrieving many warnings that may be false-positives. However, this may require too many resources due to the wide range of variables handled by the framework, which was another reason why I went for the interactive-input setup. I think it comes down to what we want the ultimate purpose of this framework to be in liboqs. Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?
There was a problem hiding this comment.
Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?
IMO both - properly configurable.
| display: Select the algorithm(s) to execute valgrind-varlat constant-time testing on | ||
| type: multiselect | ||
| choices: | ||
| - "BIKE-L1" |
There was a problem hiding this comment.
My concern with this list is maintainability: if the set of algorithms in liboqs changes, the list would need to be updated manually. Is there a way to generate this list dynamically based on the algorithms currently available?
There was a problem hiding this comment.
Thank you for pointing this out. We could fix this by retriving all available algorithms using the available_kems_by_name() and available_sigs_by_name() from helpers.py. I have included it in the latest commit, and the tests are working on my fork.
There was a problem hiding this comment.
Now that I think about it, I could also create a separate PR to fix this same issue for the kem-bench.yaml and sig-bench.yaml workflows with the same solution, if you find that appropiate.
| @@ -0,0 +1,396 @@ | |||
| // SPDX-License-Identifier: MIT | |||
There was a problem hiding this comment.
The test_sig.c / test_kem.c files and the corresponding CMakeLists.txt appear to be largely copied from the versions in the /tests directory. Could we avoid duplicating these files and instead reuse the originals with additional macro definitions (e.g., to disable signature verification during ct-testing)? Or is there a specific reason why maintaining separate copies is necessary here?
There was a problem hiding this comment.
Yes, I think that sounds like a better approach over the preliminary copy-pasting approach. I have updated it in the last commit and the tests are passing in my fork.
| @@ -0,0 +1,15 @@ | |||
| fun:PQCP_MLKEM_NATIVE_MLKEM*_C_poly_decompress_d* | |||
There was a problem hiding this comment.
I really like this approach, having an allowlist mechanism similar to the Valgrind one makes a lot of sense.
| INSTALL_PREFIX="$PWD/valgrind_varlat" | ||
|
|
||
| echo "Cloning Valgrind's source code" | ||
| git clone git://sourceware.org/git/valgrind.git valgrind_varlat_src> /dev/null 2>&1 || true |
There was a problem hiding this comment.
The statements with || true would silently fail.
There was a problem hiding this comment.
Thanks! Forgot to remove these after testing. Should I also remove > /dev/null 2>&1? I initially kept them so that the output in GitHub Actions looks cleaner and it is easier to spot the pass/fail of tests, but we can keep the output if preferred.
There was a problem hiding this comment.
IMO if the output is helpful for debugging it should be kept, if not it could be sent to dev/null
| done | ||
|
|
||
| # Create backup files of the original tests files | ||
| mv "$LIBOQS_DIR/tests/CMakeLists.txt" "$LIBOQS_DIR/tests/CMakeLists.txt.bak" |
There was a problem hiding this comment.
This looks brittle since it makes assumptions about the liboqs file structure. IMO avoiding replacing the original files and instead work CT-specific macros in the original files would be more robust.
There was a problem hiding this comment.
Fixed per the previous comment.
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
|
A couple of questions, @bhess: With respect to the bot warnings: Is the "verified creator" an issue in this case? And would the pip command also require a pin by hash here? Also, the failing job in CI seems to be a network issue during the docker pull. Any chance that could be re-executed? |
I’m not entirely sure what to make of the “verified creator” warning, but it may have security implications. From what I can tell, this action is mainly meant for interactive use.
Done |
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Sounds good, @bhess. I looked into it and we could also implement it using the functionality provided by Github Actions inputs, rather than this external party. Maybe this could simplify the whole thing for both CI tests and manual tests. |
5a2abdf to
86d566d
Compare
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Thanks for the updates @pablo-gf and for looking into this. I think it would be great to rely on the CI usage guidelines for this. I assume the CI tests will fall here:
Ideally, we would first try to add parallel jobs so that each job stays within the 2-hour limit and can be included in the weekly workflow.
I'd recommend adding those that are triaged. |
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
|
Thanks for the comment @bhess. I have included the framework in the weekly tests, since the latest version executes in around 1 hour. Obviously, this would increase as more algorithms are triaged and included in the testing framework, so this is something to keep in mind for the future. At the moment, it is executing 5 jobs in parallel on all ML-KEM variants, which are those containing suppression files. Any suggestions on the failing CI tests? |
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
|
I have moved the Kyberslash patch to ci-containers so CI test should be much faster now. I also updated the documentation for false-positive suppression block handling, and merged the previous tooling with this new framework. I essentially combined everything into one single ct_tooling directory, moving the "passes" and "issues" that were already logged with the previous tooling into the new directory structure. After talking with @bhess, we noticed that the previous tooling used a python script used for testing that is not currently used by the new framework. As a result, I have removed it. If anyone thinks differently, please let us know. With this, I think it should be ready for review. Let me know if you have any comments or suggestions! |
baentsch
left a comment
There was a problem hiding this comment.
Thanks again for the work @pablo-gf ! It seems this needs a do-over to catch up with "latest" developments (alg removal, etc.). Please also see my single comments. I did not look at the underlying logic, but just how this presents itself to users. A more general question I have is how this matches up to what is stated in https://github.com/open-quantum-safe/liboqs/blob/main/SECURITY.md? In particular, how are the results represented in the algorithm datasheets as claimed?
| for idx, m in enumerate(matches): | ||
| alg_name = m.group(1) | ||
| # Skip SPHINCS and SLH_DSA algorithms | ||
| if 'SPHINCS' in alg_name or 'SLH_DSA' in alg_name: |
There was a problem hiding this comment.
I thought Sphincs has been removed?
There was a problem hiding this comment.
True, I also removed its mention from ct-test.sh and the README files.
|
|
||
| The `local_testing_example.sh` script demonstrates how to use `ct_test.sh` to run CT tests locally across a variety of compilers, compiler versions, liboqs target builds, and optimization flags. | ||
|
|
||
| The `ct-tooling-valgrind-varlat.yml` and `ct-tooling-memsan.yml` workflows also use `ct_test.sh` to execute CT tests in CI on user-selected algorithms (using [interactive-inputs](https://github.com/marketplace/actions/interactive-inputs)) when the workflows are manually triggered on GitHub Actions. |
There was a problem hiding this comment.
Hmm - that begs the question which algorithms are executed when done automatically in CI. Worthwhile stating here?
There was a problem hiding this comment.
CI currently defaults to ML-KEM variants only because they have been thoroughly examined to exclude false-positives. I did not include it in the README because I thought the default algorithms used used for CI could potentially change over time, but let me know if you think we should include it anyways.
Something to keep in mind as we go over the warnings found is that while we can expand these defaults as we want, including a large number of default algorithms would also signifcantly increase the time and resources used during testing.
| description: 'Algorithms to execute CT testing on (comma-separated)' | ||
| required: true | ||
| type: string | ||
| default: 'ML-KEM-512,ML-KEM-768,ML-KEM-1024' |
There was a problem hiding this comment.
Hmm, does this mean only these algs are CT-tested now?
There was a problem hiding this comment.
For CI, yes. That is correct.
| - `tool`: `valgrind-varlat` or `memsan` | ||
| - `compiler_version`: clang, clang-20, gcc, gcc-14, ... | ||
| - `liboqs_build`: `generic` or `auto` | ||
| - `input`: `all`, `kems`, `sigs`, or a specific enabled algorithm variant |
There was a problem hiding this comment.
That's perfect -- is "all" enabled by default? But then how does "all" relate to the exclusion of SLH-DSA?
There was a problem hiding this comment.
For CI, the default option tests all ML-KEM variants as I mentioned earlier. The "all" option excludes SLH-DSA variants. I forgot to state that in the README. Thanks for noticing.
| 2. **Test execution** | ||
| Then, `test()` is tasked with executing the tool's test on selected liboqs algorithms. Each tool has a different process through which it parses the tool's output to keep unique instances of the warnings, which are further detailed in their respective README files: [Valgrind-Varlat's README](tools/valgrind_varlat/README.md) and [MemSan's README](tools/memsan/README.md). | ||
|
|
||
| Both tools enforce a warning cap of 100,000 unique warnings per algorithm run. Once the cap is reached, further warnings are suppressed. All SPHINCS and SLH-DSA signature variants are currently skipped during SIG tests due to the excessive time they require to execute. |
There was a problem hiding this comment.
Again, does Sphincs warrant mentioning? What about stfl-sigs? They're even worse from a runtime perspective,no?
There was a problem hiding this comment.
Thanks for noticing, I removed SPHINCS mention. Currently, no tests are executed for stfl-sigs. That is something we would like to enable in the future.
| @@ -1,6 +1,7 @@ | |||
| # SPDX-License-Identifier: MIT | |||
|
|
|||
| option(OQS_ENABLE_TEST_CONSTANT_TIME "Build test suite with support for Valgrind-based detection of non-constant time behaviour." OFF) | |||
| option(OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN "Build test suite with support for MemorySanitizer-based detection of non-constant time behaviour." OFF) | |||
There was a problem hiding this comment.
If we introduce a variable to enable memsan-only testing and define the previous variable to be Valgrind only testing, users get confused: Does setting "OQS_ENABLE_TEST_CONSTANT_TIME" enable all CT testing or just Valgrind-based testing? Wouldn't it be more sensible to add also a OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND option and re-define OQS_ENABLE_TEST_CONSTANT_TIME to enable both OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND and OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN?
There was a problem hiding this comment.
Yes, I think this is a better approach too. In this case, each tool requires a mutually exclusive testing environment since they rely on different compilation and execution frameworks to flag uninitialized secret data, so we would stay with OQS_ENABLE_TEST_CONSTANT_TIME_VALGRIND and OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN only. I have updated this in the latest version and documented it in CONFIGURE.md too.
| @@ -296,7 +300,7 @@ static OQS_STATUS kem_test_correctness(const char *method_name, bool derand) { | |||
| goto err; | |||
| } | |||
|
|
|||
| #ifndef OQS_ENABLE_TEST_CONSTANT_TIME | |||
| #if defined(OQS_ENABLE_TEST_CONSTANT_TIME) || defined(OQS_ENABLE_TEST_CONSTANT_TIME_MEMSAN) | |||
There was a problem hiding this comment.
This shows the problem of bad option definitions. IMO there should only one define be necessary here...
| @@ -0,0 +1,169 @@ | |||
| # Valgrind-Varlat | |||
| This directory contains the necessary files to execute Valgrind's memcheck tool on liboqs with [Daniel Bernstein's Kyberslash patches](https://kyberslash.cr.yp.to/papers.html) (valgrind-try-patch-20250805.txt and valgrind-varlat-patch-20250805.txt) and another patch including variable latency warnings in the suppression block (valgrind_varlat_sup_block.txt). | |||
There was a problem hiding this comment.
Why still this reference to a patch we now have in CI? If just for completeness, what about mentioning that? If no longer needed, why not remove this text?
There was a problem hiding this comment.
Oops. Thanks for noticing! I rephrased it to simply mention that this repository uses such patches instead. I also included a link to the CI repo in case somebody wants to replicate it locally.
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| interactive-inputs: |
There was a problem hiding this comment.
Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?
IMO both - properly configurable.
Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>
|
Thank you for the comments and suggestions @baentsch. I have fixed and answered all, but let me know if you have any more.
Regarding this, we could extend them to include a column for MemSan tests as well. What are your thoughts about this @bhess? How is this constant-time information reflected on the datasheet usually pipelined? |
This PR includes the constant-time testing framework developed throughout the course of the LFX Mentorship project supervised by @bhess. These are the main features of this framework:
I would be happy to walk through the details of any of these features either here or during a status call. You can also find detailed guides in the documentation files I’ve added under
tests/ct_toolingand its subdirectories. Feel free to throw any feedback or comments so that we can move this from draft to "ready for review"!AI assistance was used to generate documentation and minor editing. The resulting code was reviewed, edited and verified for accuracy.