Add first-class adversarial fuzz runtime contracts#2020
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Architecture
runtime-coreowns backend-neutral contracts and algorithms only:adversarial-campaign.ts: deterministic round-based scheduling, stateful corpus mutation, novelty retention, resource budgets, minimization, deduplication, replay, and differential classificationtransport-faults.ts: request matching, deterministic outcome sequences/counters, capability negotiation, evidence redaction, and fault fingerprintsadversarial-browser.ts: descriptor-derived hostile journeys, generic UI oracles, and journey minimizationadversarial-artifacts.ts: bounded manifested findings/replays, secret/path redaction, file hashes, and a stable bundle digestruntime-playgroundowns Playwright mappings for browser routing and browser clock control.cliowns Docker-backed disposable service lifecycle and disruption controls. No component, plugin, product, transport vendor, or WordPress-specific attack policy enters the generic core.Public contracts
New discoverable schemas:
wp-codebox/adversarial-campaign/v1wp-codebox/adversarial-campaign-result/v1wp-codebox/adversarial-finding/v1wp-codebox/adversarial-replay/v1wp-codebox/adversarial-evidence-bundle/v1wp-codebox/transport-fault-model/v1wp-codebox/transport-fault-capabilities/v1wp-codebox/transport-fault-evidence/v1wp-codebox/adversarial-browser-plan/v1wp-codebox/adversarial-browser-oracle-result/v1wp-codebox/clock-control-capabilities/v1wp-codebox/differential-result/v1They are exported from the compatibility, curated public, and contracts facades and listed by
runtimeContractManifest().Determinism and concurrency
Cases are generated from a seeded SHA-256 schedule. Up to
workerscases execute concurrently in each round, then results are committed to the retained corpus in case-index order. Completion timing therefore cannot alter later mutations. Tests prove three workers overlap while two runs retain identical schedules, corpora, fingerprints, and replay metadata.Findings are minimized by deterministic action-subset replay plus input shrinking. Fingerprints intentionally exclude payload-specific noise and include oracle ids, normalized diagnostic codes, status, and matrix cell for stable deduplication.
Capability and fidelity model
Every transport adapter negotiates required semantics before execution:
Playwright payload truncation is never mislabeled as a transport disconnect. Browser clock freeze/advance/skew/resume is exact; server process, scheduler, and database clocks are explicitly unsupported by that adapter.
Services
inputs.servicesadditively accepts:All services use ephemeral loopback ports, tmpfs/no persistent volume, explicit reverse-order teardown, and runtime-created credentials where required. Typed controls support stop/start, pause/resume, restart, disconnect/reconnect, and provider-specific flush/read-only behavior. Recovery waits for readiness. Unsupported latency shaping points callers to the transport-fault adapter instead of pretending fidelity.
Security boundaries
Compatibility
fuzzRun.casesare unchangedArtifact and replay example
Each retained finding carries:
writeAdversarialEvidenceBundle()writesmanifest.json, the campaign result, fingerprint-addressed finding and replay files, andevidence/secret-scan.json, enforcing the declared artifact byte ceiling before publication.Test matrix
Passing:
npm run test:adversarial-runtime(11 contract/unit/integration tests)npm run test:runtime-servicesnpm run test:runtime-services-lifecyclenpm run test:fuzz-suite-runnernpm run test:playground-fuzz-suite-publicnpm run test:browser-scenariosnpm run test:schema-paritynpm run test:public-api-contractnpm run test:runtime-contract-manifestnpm run test:release-package-coveragenpm run buildnpm pack --dry-run --jsongit diff --checkThe repository-wide
npm run checkreaches the pre-existingcommand-registry-smokefailure tracked in #1745:wordpress.collect-workload-result outputShape should mention outputSchema id. This branch does not touch that command definition.Performance and resource limits
Defaults are 100 cases, one worker, 50 actions/case, 1 MiB input/case, 30 seconds/case, five minutes/campaign, and 100 MiB artifacts. Values are normalized to hard ceilings (100,000 cases, 128 workers, 10,000 actions, 64 MiB input, one hour/case, one day/campaign, and 4 GiB artifacts). Adapters remain responsible for lower backend limits.
Known residual gaps
This is the strongest coherent generic substrate slice, but it does not claim the complete #2014 acceptance criteria yet:
The current neutral tests prove deterministic discovery/minimization behavior without pretending they are live WordPress plugin/theme campaigns. For that reason this PR deliberately uses
Refs #2014, notCloses #2014.Refs #2014