CI: Re-enable benchmarking on Mac Mini/RPi4/A55/BPi#1641
Conversation
There was a problem hiding this comment.
Mac Mini (M1, 2020) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: 81a5e72 | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
12319 cycles |
12319 cycles |
1 |
ML-KEM-512 encaps |
14999 cycles |
14999 cycles |
1 |
ML-KEM-512 decaps |
19552 cycles |
19552 cycles |
1 |
ML-KEM-768 keypair |
21266 cycles |
21083 cycles |
1.01 |
ML-KEM-768 encaps |
23873 cycles |
23871 cycles |
1.00 |
ML-KEM-768 decaps |
30417 cycles |
30414 cycles |
1.00 |
ML-KEM-1024 keypair |
30329 cycles |
30329 cycles |
1 |
ML-KEM-1024 encaps |
34573 cycles |
34577 cycles |
1.00 |
ML-KEM-1024 decaps |
44190 cycles |
44184 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
ppc64le (POWER10) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
59868 cycles |
59900 cycles |
1.00 |
ML-KEM-512 encaps |
72806 cycles |
72669 cycles |
1.00 |
ML-KEM-512 decaps |
93080 cycles |
92737 cycles |
1.00 |
ML-KEM-768 keypair |
99886 cycles |
100517 cycles |
0.99 |
ML-KEM-768 encaps |
116411 cycles |
117320 cycles |
0.99 |
ML-KEM-768 decaps |
142507 cycles |
143329 cycles |
0.99 |
ML-KEM-1024 keypair |
151436 cycles |
152695 cycles |
0.99 |
ML-KEM-1024 encaps |
170338 cycles |
171817 cycles |
0.99 |
ML-KEM-1024 decaps |
203058 cycles |
203925 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
oqs-bot
left a comment
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'ppc64le (POWER10) benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.
| Benchmark suite | Current: d1f8a2b | Previous: ddbc21d | Ratio |
|---|---|---|---|
ML-KEM-1024 keypair |
153341 cycles |
148293 cycles |
1.03 |
ML-KEM-1024 decaps |
204407 cycles |
198126 cycles |
1.03 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
AMD EPYC 3rd gen (c6a)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
14239 cycles |
14380 cycles |
0.99 |
ML-KEM-512 encaps |
16033 cycles |
16061 cycles |
1.00 |
ML-KEM-512 decaps |
21516 cycles |
21569 cycles |
1.00 |
ML-KEM-768 keypair |
25215 cycles |
24790 cycles |
1.02 |
ML-KEM-768 encaps |
25743 cycles |
25549 cycles |
1.01 |
ML-KEM-768 decaps |
33548 cycles |
33362 cycles |
1.01 |
ML-KEM-1024 keypair |
37506 cycles |
37273 cycles |
1.01 |
ML-KEM-1024 encaps |
36160 cycles |
37239 cycles |
0.97 |
ML-KEM-1024 decaps |
47430 cycles |
46827 cycles |
1.01 |
This comment was automatically generated by workflow using github-action-benchmark.
oqs-bot
left a comment
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'AMD EPYC 3rd gen (c6a)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.
| Benchmark suite | Current: d1f8a2b | Previous: ddbc21d | Ratio |
|---|---|---|---|
ML-KEM-768 keypair |
25523 cycles |
24757 cycles |
1.03 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
AMD EPYC 4th gen (c7a)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
12873 cycles |
12760 cycles |
1.01 |
ML-KEM-512 encaps |
14269 cycles |
14242 cycles |
1.00 |
ML-KEM-512 decaps |
19138 cycles |
19119 cycles |
1.00 |
ML-KEM-768 keypair |
22512 cycles |
22416 cycles |
1.00 |
ML-KEM-768 encaps |
23051 cycles |
23106 cycles |
1.00 |
ML-KEM-768 decaps |
30057 cycles |
30082 cycles |
1.00 |
ML-KEM-1024 keypair |
34269 cycles |
33049 cycles |
1.04 |
ML-KEM-1024 encaps |
33042 cycles |
33012 cycles |
1.00 |
ML-KEM-1024 decaps |
42443 cycles |
42417 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'AMD EPYC 4th gen (c7a)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-1024 keypair |
34269 cycles |
33049 cycles |
1.04 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton4
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
17678 cycles |
17648 cycles |
1.00 |
ML-KEM-512 encaps |
20596 cycles |
20598 cycles |
1.00 |
ML-KEM-512 decaps |
27062 cycles |
27062 cycles |
1 |
ML-KEM-768 keypair |
29923 cycles |
29899 cycles |
1.00 |
ML-KEM-768 encaps |
32725 cycles |
32770 cycles |
1.00 |
ML-KEM-768 decaps |
41983 cycles |
41963 cycles |
1.00 |
ML-KEM-1024 keypair |
43721 cycles |
43739 cycles |
1.00 |
ML-KEM-1024 encaps |
48773 cycles |
48648 cycles |
1.00 |
ML-KEM-1024 decaps |
61383 cycles |
61385 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Intel Xeon 3rd gen (c6i)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
17586 cycles |
17533 cycles |
1.00 |
ML-KEM-512 encaps |
19855 cycles |
19866 cycles |
1.00 |
ML-KEM-512 decaps |
26392 cycles |
26401 cycles |
1.00 |
ML-KEM-768 keypair |
32478 cycles |
32496 cycles |
1.00 |
ML-KEM-768 encaps |
31063 cycles |
31064 cycles |
1.00 |
ML-KEM-768 decaps |
41529 cycles |
41498 cycles |
1.00 |
ML-KEM-1024 keypair |
44381 cycles |
43981 cycles |
1.01 |
ML-KEM-1024 encaps |
45691 cycles |
46953 cycles |
0.97 |
ML-KEM-1024 decaps |
58110 cycles |
58232 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
AMD EPYC 3rd gen (c6a) (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
40217 cycles |
40259 cycles |
1.00 |
ML-KEM-512 encaps |
48343 cycles |
48421 cycles |
1.00 |
ML-KEM-512 decaps |
62487 cycles |
62612 cycles |
1.00 |
ML-KEM-768 keypair |
64571 cycles |
63737 cycles |
1.01 |
ML-KEM-768 encaps |
74921 cycles |
74943 cycles |
1.00 |
ML-KEM-768 decaps |
93427 cycles |
93583 cycles |
1.00 |
ML-KEM-1024 keypair |
95273 cycles |
95240 cycles |
1.00 |
ML-KEM-1024 encaps |
109366 cycles |
109391 cycles |
1.00 |
ML-KEM-1024 decaps |
132099 cycles |
132163 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A76 (Raspberry Pi 5) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
28283 cycles |
28250 cycles |
1.00 |
ML-KEM-512 encaps |
34105 cycles |
34099 cycles |
1.00 |
ML-KEM-512 decaps |
44373 cycles |
44327 cycles |
1.00 |
ML-KEM-768 keypair |
47685 cycles |
47614 cycles |
1.00 |
ML-KEM-768 encaps |
53903 cycles |
53937 cycles |
1.00 |
ML-KEM-768 decaps |
68354 cycles |
68362 cycles |
1.00 |
ML-KEM-1024 keypair |
70279 cycles |
70211 cycles |
1.00 |
ML-KEM-1024 encaps |
78694 cycles |
78756 cycles |
1.00 |
ML-KEM-1024 decaps |
98456 cycles |
98438 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
AMD EPYC 4th gen (c7a) (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
36614 cycles |
36602 cycles |
1.00 |
ML-KEM-512 encaps |
43136 cycles |
43061 cycles |
1.00 |
ML-KEM-512 decaps |
55724 cycles |
55697 cycles |
1.00 |
ML-KEM-768 keypair |
58661 cycles |
58658 cycles |
1.00 |
ML-KEM-768 encaps |
67583 cycles |
67501 cycles |
1.00 |
ML-KEM-768 decaps |
84592 cycles |
84426 cycles |
1.00 |
ML-KEM-1024 keypair |
89042 cycles |
88986 cycles |
1.00 |
ML-KEM-1024 encaps |
99195 cycles |
99235 cycles |
1.00 |
ML-KEM-1024 decaps |
120638 cycles |
120547 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton4 (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
35448 cycles |
35410 cycles |
1.00 |
ML-KEM-512 encaps |
40092 cycles |
40115 cycles |
1.00 |
ML-KEM-512 decaps |
51097 cycles |
51137 cycles |
1.00 |
ML-KEM-768 keypair |
56742 cycles |
56672 cycles |
1.00 |
ML-KEM-768 encaps |
64537 cycles |
65151 cycles |
0.99 |
ML-KEM-768 decaps |
79370 cycles |
79301 cycles |
1.00 |
ML-KEM-1024 keypair |
87844 cycles |
87869 cycles |
1.00 |
ML-KEM-1024 encaps |
97111 cycles |
96878 cycles |
1.00 |
ML-KEM-1024 decaps |
115947 cycles |
115828 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton3
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
18659 cycles |
18637 cycles |
1.00 |
ML-KEM-512 encaps |
21882 cycles |
21873 cycles |
1.00 |
ML-KEM-512 decaps |
28878 cycles |
28862 cycles |
1.00 |
ML-KEM-768 keypair |
31587 cycles |
31539 cycles |
1.00 |
ML-KEM-768 encaps |
34742 cycles |
34775 cycles |
1.00 |
ML-KEM-768 decaps |
44810 cycles |
44775 cycles |
1.00 |
ML-KEM-1024 keypair |
46073 cycles |
46080 cycles |
1.00 |
ML-KEM-1024 encaps |
51509 cycles |
51496 cycles |
1.00 |
ML-KEM-1024 decaps |
65017 cycles |
65027 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Intel Xeon 3rd gen (c6i) (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
45688 cycles |
45672 cycles |
1.00 |
ML-KEM-512 encaps |
54399 cycles |
54295 cycles |
1.00 |
ML-KEM-512 decaps |
69751 cycles |
69725 cycles |
1.00 |
ML-KEM-768 keypair |
74146 cycles |
74093 cycles |
1.00 |
ML-KEM-768 encaps |
86019 cycles |
85991 cycles |
1.00 |
ML-KEM-768 decaps |
106602 cycles |
106591 cycles |
1.00 |
ML-KEM-1024 keypair |
112182 cycles |
112084 cycles |
1.00 |
ML-KEM-1024 encaps |
124671 cycles |
124581 cycles |
1.00 |
ML-KEM-1024 decaps |
150632 cycles |
150599 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton3 (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
38937 cycles |
38894 cycles |
1.00 |
ML-KEM-512 encaps |
44534 cycles |
44595 cycles |
1.00 |
ML-KEM-512 decaps |
56597 cycles |
56669 cycles |
1.00 |
ML-KEM-768 keypair |
62332 cycles |
62297 cycles |
1.00 |
ML-KEM-768 encaps |
71054 cycles |
72327 cycles |
0.98 |
ML-KEM-768 decaps |
87337 cycles |
87697 cycles |
1.00 |
ML-KEM-1024 keypair |
96219 cycles |
96159 cycles |
1.00 |
ML-KEM-1024 encaps |
106365 cycles |
106138 cycles |
1.00 |
ML-KEM-1024 decaps |
126801 cycles |
126578 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton2
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
28248 cycles |
28255 cycles |
1.00 |
ML-KEM-512 encaps |
34179 cycles |
34110 cycles |
1.00 |
ML-KEM-512 decaps |
44384 cycles |
44390 cycles |
1.00 |
ML-KEM-768 keypair |
47717 cycles |
47671 cycles |
1.00 |
ML-KEM-768 encaps |
54081 cycles |
53906 cycles |
1.00 |
ML-KEM-768 decaps |
68558 cycles |
68361 cycles |
1.00 |
ML-KEM-1024 keypair |
70346 cycles |
70281 cycles |
1.00 |
ML-KEM-1024 encaps |
78742 cycles |
78718 cycles |
1.00 |
ML-KEM-1024 decaps |
98529 cycles |
98476 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Graviton2 (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
59189 cycles |
59129 cycles |
1.00 |
ML-KEM-512 encaps |
68622 cycles |
68640 cycles |
1.00 |
ML-KEM-512 decaps |
87356 cycles |
87347 cycles |
1.00 |
ML-KEM-768 keypair |
95479 cycles |
95010 cycles |
1.00 |
ML-KEM-768 encaps |
110402 cycles |
109687 cycles |
1.01 |
ML-KEM-768 decaps |
134636 cycles |
134169 cycles |
1.00 |
ML-KEM-1024 keypair |
148046 cycles |
147998 cycles |
1.00 |
ML-KEM-1024 encaps |
163740 cycles |
163866 cycles |
1.00 |
ML-KEM-1024 decaps |
195486 cycles |
195539 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A55 (Snapdragon 888) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: 81a5e72 | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
59731 cycles |
59707 cycles |
1.00 |
ML-KEM-512 encaps |
67471 cycles |
67375 cycles |
1.00 |
ML-KEM-512 decaps |
86058 cycles |
86040 cycles |
1.00 |
ML-KEM-768 keypair |
97394 cycles |
97411 cycles |
1.00 |
ML-KEM-768 encaps |
110836 cycles |
110976 cycles |
1.00 |
ML-KEM-768 decaps |
137705 cycles |
138041 cycles |
1.00 |
ML-KEM-1024 keypair |
154757 cycles |
154703 cycles |
1.00 |
ML-KEM-1024 encaps |
171629 cycles |
171727 cycles |
1.00 |
ML-KEM-1024 decaps |
207366 cycles |
208007 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
CBMC Results (ML-KEM-512)
Full Results (190 proofs)
|
CBMC Results (ML-KEM-768)Full Results (190 proofs)
|
CBMC Results (ML-KEM-1024)Full Results (190 proofs)
|
There was a problem hiding this comment.
SpacemiT K1 8 (Banana Pi F3) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: 81a5e72 | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
155527 cycles |
155396 cycles |
1.00 |
ML-KEM-512 encaps |
163416 cycles |
163248 cycles |
1.00 |
ML-KEM-512 decaps |
206678 cycles |
206498 cycles |
1.00 |
ML-KEM-768 keypair |
249881 cycles |
249749 cycles |
1.00 |
ML-KEM-768 encaps |
270385 cycles |
270217 cycles |
1.00 |
ML-KEM-768 decaps |
332200 cycles |
332485 cycles |
1.00 |
ML-KEM-1024 keypair |
395810 cycles |
395420 cycles |
1.00 |
ML-KEM-1024 encaps |
423624 cycles |
423209 cycles |
1.00 |
ML-KEM-1024 decaps |
507309 cycles |
506804 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Arm Cortex-A72 (Raspberry Pi 4) benchmarks
Details
| Benchmark suite | Current: ae9162e | Previous: 81a5e72 | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
50673 cycles |
51363 cycles |
0.99 |
ML-KEM-512 encaps |
58542 cycles |
59494 cycles |
0.98 |
ML-KEM-512 decaps |
75315 cycles |
75347 cycles |
1.00 |
ML-KEM-768 keypair |
86171 cycles |
86863 cycles |
0.99 |
ML-KEM-768 encaps |
94320 cycles |
94915 cycles |
0.99 |
ML-KEM-768 decaps |
117702 cycles |
119808 cycles |
0.98 |
ML-KEM-1024 keypair |
129513 cycles |
131224 cycles |
0.99 |
ML-KEM-1024 encaps |
142127 cycles |
144261 cycles |
0.99 |
ML-KEM-1024 decaps |
174310 cycles |
176677 cycles |
0.99 |
This comment was automatically generated by workflow using github-action-benchmark.
d1f8a2b to
fb93b21
Compare
There was a problem hiding this comment.
Intel Xeon 4th gen (c7i)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
12056 cycles |
12037 cycles |
1.00 |
ML-KEM-512 encaps |
13593 cycles |
13756 cycles |
0.99 |
ML-KEM-512 decaps |
17746 cycles |
17776 cycles |
1.00 |
ML-KEM-768 keypair |
21251 cycles |
21044 cycles |
1.01 |
ML-KEM-768 encaps |
22080 cycles |
22040 cycles |
1.00 |
ML-KEM-768 decaps |
28477 cycles |
28416 cycles |
1.00 |
ML-KEM-1024 keypair |
29985 cycles |
29879 cycles |
1.00 |
ML-KEM-1024 encaps |
31703 cycles |
31716 cycles |
1.00 |
ML-KEM-1024 decaps |
39427 cycles |
39467 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Intel Xeon 4th gen (c7i) (no-opt)
Details
| Benchmark suite | Current: ae9162e | Previous: e2bc4ad | Ratio |
|---|---|---|---|
ML-KEM-512 keypair |
28235 cycles |
28191 cycles |
1.00 |
ML-KEM-512 encaps |
36632 cycles |
36609 cycles |
1.00 |
ML-KEM-512 decaps |
45151 cycles |
45197 cycles |
1.00 |
ML-KEM-768 keypair |
46233 cycles |
46321 cycles |
1.00 |
ML-KEM-768 encaps |
55779 cycles |
55810 cycles |
1.00 |
ML-KEM-768 decaps |
69830 cycles |
69876 cycles |
1.00 |
ML-KEM-1024 keypair |
70568 cycles |
70306 cycles |
1.00 |
ML-KEM-1024 encaps |
82482 cycles |
82479 cycles |
1.00 |
ML-KEM-1024 decaps |
99377 cycles |
98894 cycles |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
Our benchmarking boxes are back online. This renables benchmarking in CI. Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
fb93b21 to
ae9162e
Compare
Port of pq-code-package/mlkem-native#1641. Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
Port of pq-code-package/mlkem-native#1641. Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
Port of pq-code-package/mlkem-native#1641. Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
Port of pq-code-package/mlkem-native#1641. Signed-off-by: Matthias J. Kannwischer <matthias@zerorisc.com>
Our benchmarking boxes are back online. This renables benchmarking in CI.
One thing to note is this change: We used to have two runners - the RPi4 and RPi5. Both could run benchmarks locally and also execute benchmarks remotely on the A55 and BPi. Now only the RPi4 can connect to the A55 and BPi, so we can no longer run those in parallel.