Skip to content

METRICS_REPORT.md

Latest commit

 

History

History
281 lines (220 loc) · 13.5 KB

METRICS_REPORT.md

File metadata and controls

281 lines (220 loc) · 13.5 KB

AMA Cryptography — Verified Metrics Report

Version: 3.1.0 + Unreleased Measurement Date: 2026-05-16 (static LoC / test-function counts); benchmark figures last refreshed separately Last Reviewed: 2026-05-16 Repository snapshot: counts below were measured on this branch; re-run the reproduction commands after any rebase, squash, or large merge so the figures continue to match the shipped tree. If a documented count and this report disagree, the count is the bug. Generated by: Manual reproduction using standard POSIX utilities (find, wc, grep). Commands are listed in the Reproduction Instructions section.

Purpose

This report anchors all quantitative claims the project makes about itself (lines of code, test counts, NIST-vector counts, adversarial-test counts) to commands that any reviewer can run from a fresh clone. If a documented count and this report disagree, the reviewer should re-run the command and open an issue against the stale claim.

Previously circulated unanchored figures — "2,068 tests", "117 NIST", "180 adversarial", "11,246 LoC" — do not appear in this repository and are superseded by the counts below. If those figures originated elsewhere (presentations, draft PR descriptions), they should be corrected to match this report or explicitly define their counting methodology.

Lines of Code

Measured as non-empty-allowed wc -l over source files in each scope. wc -l counts newlines; trailing-newline-free files under-count by one.

Scope Files Lines
Library Python (ama_cryptography/*.py) 19 17,944
Native C (src/c/**/*.c, include/**/*.h) 110 39,091
Library total (Python + C + headers) 129 57,035
Top-level Python (monitors, benchmarks, demos) — 918
Tests (tests/**/*.py) 78 33,072
Cython (*.pyx, *.pxd) — 1,513
Whole project (source + docs + config) — 151,061

Library total (the figure that most closely tracks "library size"): 57,035 lines across 129 files under ama_cryptography/, src/c/, and include/. This supersedes any "11,246 LoC" claim that may have appeared externally.

Whole-project total (151,061 lines across Python, C, headers, Cython, Markdown, YAML/TOML/JSON config, CMake) is the broader figure some external claims may have been referencing. Report whichever scope is relevant; always state which scope you mean.

Scope Composition

The gap between "library" and "whole project" is informative: only 38% of the repository is library code. The rest is tests, docs, config, and scaffolding — which is a healthy ratio for a security library that takes verification seriously.

Scope Lines % of whole Paths
Library (Python + C + headers) 57,035 37.8% ama_cryptography/ + src/c/ + include/
Tests 33,072 21.9% tests/**/*.py
Top-level Python 918 0.6% *.py at repo root
Cython 1,513 1.0% *.pyx + *.pxd
Docs + config + build (remainder) 58,523 38.7% *.md, *.yml, *.toml, *.json, CMake, Makefile
Whole-project total 151,061 100% sum of the scopes above

Test code (21.9%) is over half the size of the library (37.8%) — i.e. the test-to-library ratio is roughly 0.58. Combined with docs+config (38.7%), non-library artifacts make up nearly two-thirds of the repo.

Reproduction

# Library Python
find ama_cryptography -name '*.py' -type f -print0 | xargs -0 wc -l | tail -1

# Native C + headers
find src/c include -type f \( -name '*.c' -o -name '*.h' \) -print0 | xargs -0 wc -l | tail -1

# Library total
find ama_cryptography src/c include -type f \
    \( -name '*.py' -o -name '*.c' -o -name '*.h' \) \
    -print0 | xargs -0 wc -l | tail -1

# Top-level Python (repo-root scripts — monitors, benchmarks, demos)
find . -maxdepth 1 -name '*.py' -type f -print0 | xargs -0 wc -l | tail -1

# Tests
find tests -name '*.py' -type f -print0 | xargs -0 wc -l | tail -1

# Cython
find . -type f \( -name '*.pyx' -o -name '*.pxd' \) ! -path './.git/*' -print0 \
    | xargs -0 wc -l | tail -1

# Whole project (source + docs + config)
find . -type f \
    \( -name '*.py' -o -name '*.c' -o -name '*.h' -o -name '*.pyx' \
    -o -name '*.pxd' -o -name '*.md' -o -name '*.yml' -o -name '*.yaml' \
    -o -name '*.toml' -o -name '*.json' -o -name '*.cmake' \
    -o -name 'CMakeLists.txt' -o -name 'Makefile' \) \
    ! -path './.git/*' ! -path './build/*' -print0 \
    | xargs -0 wc -l | tail -1

Test Counts

Static count of Python test functions (definitions matching ^\s*def test_):

Scope Count
Python test files under tests/ matching the static regex 78
Syntactic def test_ matches under tests/**/*.py 2,159
test_*.c files under tests/c/ (ctest-registered) 37
bench_*.c files under tests/c/ (standalone, not in ctest) 1
Fuzz harnesses under fuzz/ 14 C targets

Collected-via-pytest count is higher than the static function count because many tests are @pytest.mark.parametrized (the collection tool expands parametrized tests into individual test IDs). A full dynamic pytest --collect-only -q requires the native C library to be built (cmake -B build -DAMA_USE_NATIVE_PQC=ON && cmake --build build), because several tests import modules that enforce INVARIANT-7 (no cryptographic fallback) at import time. On a build-complete environment, record the count from:

pytest --collect-only -q | tail -1

Stderr is intentionally left unsuppressed so collection/import errors remain visible during reproduction.

The static count (2,159) and the dynamic collection count will differ. Any external claim ("N tests") must state which count it is reporting. This supersedes the earlier "866+ tests collected across 39 files" figure in ARCHITECTURE.md and any "2,068 tests" figure that may have circulated externally.

Reproduction

# Static function count
grep -rE "^\s*def test_" tests/ --include='*.py' | wc -l

# Static file count
grep -rlE "^\s*def test_" tests/ --include='*.py' | wc -l

# C test files registered in ctest (test_*.c)
find tests/c -maxdepth 1 -name 'test_*.c' -type f | wc -l

# Standalone C benchmark sources (bench_*.c, not registered in ctest)
find tests/c -maxdepth 1 -name 'bench_*.c' -type f | wc -l

# Fuzz harnesses
find fuzz -name 'fuzz_*.c' | wc -l

NIST ACVP Vector Counts

Authoritative anchor: CSRC_ALIGN_REPORT.md.

Count Value Anchor
Vectors tested 1,215 docs/compliance/CSRC_ALIGN_REPORT.md §Abstract
Vectors passed 1,215 docs/compliance/CSRC_ALIGN_REPORT.md §Abstract
Vectors failed 0 docs/compliance/CSRC_ALIGN_REPORT.md §Abstract
Vectors skipped 5,789 docs/compliance/CSRC_ALIGN_REPORT.md §Abstract (4,757 AFT-filtered + 1,032 non-AFT)
SHA-3 Monte Carlo Test (MCT) vectors 400 4 algos × 1 tcId × 100 outer iterations (FIPS-202 MCT spec)
Algorithm functions tested 12 docs/compliance/CSRC_ALIGN_REPORT.md §1.1
NIST standards covered 7 6 FIPS (180-4, 198-1, 202, 203, 204, 205) + SP 800-38D

If "117 NIST" has been claimed externally, it does not match any count maintained in this repository. If the intent was to count specific sub-vector categories, the definition must be published before the figure is reported. Otherwise the figure should be retracted and replaced with 1,215 vectors / 12 algorithm functions / 7 NIST standards (current; was 815 before SHA-3 MCT was added to the harness on the 2.1.5 line).

Reproduction

grep -E "^\*\*Summary:\*\*" docs/compliance/CSRC_ALIGN_REPORT.md
# Expected: **Summary:** 1,215 vectors tested, **1,215 passed**, **0 failed**, 5,789 skipped

# Standards covered
grep -E "FIPS |SP 800-" docs/compliance/CSRC_ALIGN_REPORT.md | head

Adversarial Test Counts

"Adversarial tests" is defined in this report as test functions in files whose names signal adversarial / penetration / fuzz / property-based testing intent. Inclusion is conservative; edge-case tests that merely exercise boundary conditions are excluded.

File Test functions Category
tests/test_adversarial_pqc_hardening.py 33 Adversarial PQC
tests/test_adversarial_security.py 34 Adversarial generic
tests/test_crypto_core_penetration.py 89 Penetration
tests/test_fuzzing.py 24 Fuzz harness orchestration
tests/test_hybrid_combiner_edge_cases.py 9 Edge case
tests/test_memory_security.py 12 Memory-safety attacks
tests/test_property_based_lyapunov.py 9 Property-based
tests/test_security_fixes.py 13 Regression for audit fixes
tests/test_gap_closure_review.py 35 Gap-closure security regressions
Total (adversarial umbrella) 258
Sub-total excluding fuzz/property/edge/regression 156 strict "adversarial" files only

If "180 adversarial" has been claimed externally, it does not match any fixed count in this repository; 156 (strict) or 258 (umbrella) is the accurate figure depending on the scope definition. External reports must state which scope they mean.

Reproduction

# Strict scope (files whose name includes 'adversarial' or 'penetration')
grep -cE "^\s*def test_" \
    tests/test_adversarial_pqc_hardening.py \
    tests/test_adversarial_security.py \
    tests/test_crypto_core_penetration.py \
    | awk -F: '{s+=$2} END{print s}'
# Expected: 156

# Umbrella scope (add fuzz / property / edge / regression)
grep -cE "^\s*def test_" \
    tests/test_adversarial_pqc_hardening.py \
    tests/test_adversarial_security.py \
    tests/test_crypto_core_penetration.py \
    tests/test_fuzzing.py \
    tests/test_hybrid_combiner_edge_cases.py \
    tests/test_memory_security.py \
    tests/test_property_based_lyapunov.py \
    tests/test_security_fixes.py \
    tests/test_gap_closure_review.py \
    | awk -F: '{s+=$2} END{print s}'
# Expected: 258

Reproduction Instructions

All commands above assume a fresh clone at the repository root on a POSIX shell with find, xargs, wc, and grep (GNU or BSD). No additional build steps are needed for the static counts. To reproduce the dynamic pytest-collection count you must first build the native C library:

cmake -B build -DAMA_USE_NATIVE_PQC=ON && cmake --build build
pip install -e ".[dev]"
pytest --collect-only -q | tail -1

Commit your reproduction output alongside this report when material figures change.

Change Log

Version Date Change
3.1.0 + Unreleased 2026-05-16 Re-measured static counts after PRs #301/#303/#305/#306/#307/#308/#309: 57,035 library LoC, 2,159 static Python test functions, 37 C test files, 14 fuzz harnesses, 156 / 258 adversarial tests. NIST ACVP vector count remains anchored separately at 1,215 / 1,215 / 0.
2.1.5 2026-04-18 Initial version; anchored 46,235 library LoC, 2,028 test functions, 815 NIST vectors, 156 / 223 adversarial tests for that branch snapshot. Superseded unanchored "2,068 tests / 117 NIST / 180 adversarial / 11,246 LoC" figures.
2.1.5 2026-04-20 Documentation alignment sweep; header date refresh across repository.
2.1.5 2026-04-21 Replaced unreachable commit-SHA anchor (d4806b9, no longer in history) with a branch-snapshot note; re-ran benchmarks/benchmark_runner.py and build/bin/benchmark_c_raw so README and wiki benchmark tables reflect the current tree (SHA3-256 ctypes 170,834 ops/s, Ed25519 sign 10,569, ML-DSA-65 verify 6,322, AES-256-GCM enc 278,298, ChaCha20-Poly1305 enc 271,362, X25519 22,918 — full table in benchmark-report.md).
2.1.5 2026-04-21 Added FIPS-202 Monte Carlo Test (MCT) coverage for SHA3-256/SHA3-512/SHAKE-128/SHAKE-256 (400 vectors = 4 algos × 1 tcId × 100 outer iterations). NIST vector count: 815 → 1,215; non-AFT skipped: 1,036 → 1,032; EXPECTED_VECTORS floor in .github/workflows/acvp_validation.yml: 815 → 1,215. Attestation JSON total_vectors_tested updated to match. Landed on the 2.1.5 line; first ships in the v3.0.0 release tag.
2.1.5 2026-04-25 Cherry-picked docs/AVX512_KECCAK_ADR.md (PR C closing plan) and re-ran the full validation + benchmark suite on a Linux x86-64 host with AVX-512F/BW/DQ/VL/VBMI + VAES + VPCLMULQDQ. NIST ACVP: 1,215 / 1,215 pass, 0 fail, 5,789 skipped (4,667 vectors_skipped + 1,122 mct_skipped) — matches the prior attestation count exactly. ctest: 20 / 20 pass. FIPS-140 self-test + KAT + SIMD KAT pytest lanes: 128 / 128 pass. Regression benchmark: 16 / 16 pass. Post-#261 + #265 + #266 throughput uplift is visible in the refreshed table — Ed25519 sign 10,569 → 51,206 ops/s (base-point comb table from #261 + verify SWE from #265), ML-DSA-65 sign 1,017 → 2,976 ops/s (rejection sampling + merged NTT from #260/#261), AES-256-GCM 1KB 278,298 → 271,449 ops/s (VAES YMM path from #266, within run-to-run noise at 1KB — larger blocks are where the VAES win shows up). README.md and benchmark-report.md measured-numbers tables refreshed.