Skip to content

Follow-up #461: Add Stryker mutation testing + per-file OSINT coverage thresholds + dedicated osint-qa CI jobΒ #482

@pethers

Description

@pethers

πŸ“‹ Task Overview

Follow-up to closed issue #461 / PR #474. The OSINT QA contract suite landed (tests/integration/osint/contract.test.ts β€” envelope, no-silent-zero, determinism), INTEGRATION_TESTING.md and CONTRIBUTING.md were updated, and ARCHITECTURE.md now links the suite. However, three deliverables from the original #461 acceptance criteria did not ship:

  1. Mutation testing baseline (Stryker, β‰₯70 % mutation score on the 15 OSINT tool files) β€” package.json has no test:mutation script; no stryker.config.json exists. Code search confirms Stryker is only mentioned aspirationally in FUTURE_WORKFLOWS.md.
  2. Per-file coverage thresholds (β‰₯90 % for OSINT tools, β‰₯95 % for DOCEO-touching files) β€” vitest.config.ts only enforces the global lines: 80; there is no coverage.thresholds.perFile entry.
  3. Dedicated CI job for the OSINT QA harness β€” neither osint-qa.yml nor an extension of test-and-report.yml runs the contract suite as a discrete job; it executes inside the general integration test pass without isolated reporting.

The contract test suite alone catches structural violations but cannot detect logic regressions in scoring/anomaly methodology β€” exactly the kind of regressions the seven companion correctness PRs (#467, #469, #472, #473, #476, #477, #479) introduced surface area for. Mutation testing + tight per-file thresholds close that gap.

🎯 Objectives

  • Add Stryker (@stryker-mutator/core, @stryker-mutator/vitest-runner) as a dev dependency.
  • stryker.config.json scoped to the 15 OSINT tool files + their shared utilities (src/utils/votingBaseline.ts, src/utils/graphAlgorithms.ts, src/utils/networkVotingSimilarity.ts, src/utils/effectivenessAggregator.ts, src/utils/lifecycleStatistics.ts, src/utils/doceoMepAggregator.ts, src/utils/politicalGroupNormalization.ts).
  • Initial mutation-score threshold informational (thresholds: { high: 80, low: 60, break: null }); promote break: 70 once one green run is observed on main.
  • npm run test:mutation script and matching osint-mutation job in CI.
  • Extend vitest.config.ts with coverage.thresholds.perFile entries:
    • src/tools/{the 14 non-DOCEO OSINT tools}.ts β†’ { lines: 90, branches: 85, functions: 90, statements: 90 }.
    • src/tools/assessMepInfluence.ts, src/tools/detectVotingAnomalies.ts, src/tools/sentimentTracker.ts, src/tools/networkAnalysis.ts, src/tools/analyzeCoalitionDynamics.ts (DOCEO-touching) β†’ { lines: 95, branches: 90, functions: 95, statements: 95 }.
  • Extract the contract suite into its own CI job (osint-qa) so failures are visibly attributed and golden-snapshot drift surfaces in PR checks.
  • Make the mutation job non-blocking initially (status check required: false), then promote in a follow-up PR after baseline stability is observed.

πŸ“¦ Deliverables

  • package.json β€” new devDependencies, test:mutation script, version bump in release-notes not required (CI-only change).
  • stryker.config.json at repo root scoped per above.
  • vitest.config.ts extended with per-file thresholds and an opt-in --changed-files mode for fast PR runs.
  • .github/workflows/osint-qa.yml (or extension of test-and-report.yml) with three sequential jobs: osint-contract, osint-coverage, osint-mutation (the last marked continue-on-error: true for the first 30 days).
  • CONTRIBUTING.md β€” section explaining how to run the mutation suite locally and what surviving-mutant categories are acceptable (e.g. log-string mutants).
  • INTEGRATION_TESTING.md β€” appendix on mutation-testing scope and exclusions.

πŸ”’ Security & Compliance

  • Supply Chain: Stryker packages must pass the existing npm run test:licenses allow-list (MIT/Apache-2.0/BSD/ISC) β€” verify before adding.
  • CI Resource Bound: scope must keep total mutation runtime under 15 minutes; use --concurrency 4 and exclude *.test.ts.
  • ISMS Reference: A.8.29 (Security testing in development & acceptance), A.8.34 (Protection during audit testing).
  • Compliance: ISO 27001, SLSA Level 3 (provenance covers test artefacts).

πŸ—οΈ Technical Approach

  • Run gh-advisory-database against the two Stryker packages before adding to lockfile.
  • Scope Stryker mutate glob to the 15 OSINT tool files and 7 shared utilities; ignorePatterns for tests and generated.
  • vitest-runner keeps the existing test infrastructure with no per-test fork overhead.
  • Per-file coverage uses Vitest 3's thresholds.perFile map syntax; CI prints offenders in a digestible table.
  • The new workflow uses actions/setup-node with the pinned SHA pattern from test-and-report.yml.

πŸ§ͺ Testing Requirements

  • Mutation run on main completes with score β‰₯60 % (baseline) before the threshold is bumped.
  • Per-file coverage gate fails CI when any OSINT tool drops below its assigned threshold.
  • osint-qa workflow visible as a discrete check on every PR touching src/tools/ or src/utils/votingBaseline.ts/graphAlgorithms.ts/etc.
  • Test runtime budget: contract ≀2 min, coverage ≀3 min, mutation ≀15 min.

πŸ“š Documentation Updates

  • CONTRIBUTING.md β€” mutation testing instructions.
  • INTEGRATION_TESTING.md β€” OSINT QA appendix.
  • WORKFLOWS.md β€” new osint-qa workflow entry.
  • ARCHITECTURE.md β€” reference to mutation testing as the test-quality enforcement point.

πŸ€– Recommended Agent

Agent: hack23-test-specialist (with hack23-devops-engineer for the CI workflow wiring)
Rationale: This is a pure QA-infrastructure task β€” mutation testing scope, Vitest threshold configuration, and CI job extraction map directly onto the test-specialist's remit; DevOps owns the workflow file.

πŸ”— Related Issues

πŸ“Œ ISMS References

  • Hack23/ISMS-PUBLIC Secure Development Policy Β§4.4 (Testing), Β§4.5 (Quality Assurance).
  • ISO 27001 Annex A.8.29 (Security testing), A.8.34 (Audit protection).
  • CIS Controls v8.1 Control 16 (Application Software Security).
  • EU CRA β€” verifiable testing evidence for product correctness.

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions