Merge pull request #2020 from Automattic/feat/2014-adversarial-fuzzing

Add first-class adversarial fuzz runtime contracts

Author: chubes4

README.md

@@ -1,5 +1,9 @@
 # WP Codebox
 
+Adversarial campaign, transport fault, service disruption, browser oracle, and
+sealed replay contracts are documented in
+[`docs/adversarial-runtime.md`](docs/adversarial-runtime.md).
+
 **WP Codebox unlocks secure WordPress code execution from anywhere.** Run agents, accept untrusted patches, evaluate plugins, reproduce bugs, or experiment freely - every sandbox is a disposable contained WordPress runtime that can't touch its caller. Your host can be a CLI, CI job, mobile app, Node service, WordPress plugin, or anything else that can shell out or hit an API.
 
 WordPress has historically lacked a clean scratch space for code execution. Modern dev workflows assume one - Node has `npm install` per project, Python has venvs, containers have ephemeral filesystems. WP Codebox provides that primitive as a usable runtime contract: real WordPress, fully ephemeral, no host filesystem access except via declared mounts. Any product - WordPress or not - can offer code execution against a real WordPress instance without risking the caller.

docs/adversarial-runtime.md

@@ -0,0 +1,202 @@
+# Adversarial runtime
+
+WP Codebox exposes backend-neutral adversarial campaign and transport-fault
+contracts from `@automattic/wp-codebox-core/public`. The contracts compose with
+the existing fuzz suite, runtime episode, checkpoint, browser, service, artifact,
+and replay surfaces; they do not replace those surfaces.
+
+## Architecture
+
+- `adversarial-campaign.ts` owns deterministic corpus scheduling, generic value
+  and action-sequence mutation, novelty retention, bounded concurrent execution,
+  minimization, stable finding fingerprints, replay schedules, and differential
+  result classification.
+- `transport-faults.ts` owns request matching, deterministic stateful outcome
+  sequences, capability negotiation, redacted evidence, and deduplication. It
+  does not assume a browser, HTTP library, application, or transport.
+- `adversarial-browser.ts` owns DOM-derived journey planning, hostile generic
+  input generation, journey minimization, and user-facing oracle contracts.
+- `adversarial-artifacts.ts` writes bounded manifested finding and replay bundles,
+  removes declared secrets and machine-specific paths, and seals corpus/finding
+  identity with a deterministic content digest.
+- `runtime-playground` maps supported fault outcomes and browser clock controls
+  onto Playwright. Socket behavior that Playwright cannot faithfully produce is
+  reported as unsupported.
+- `runtime-services.ts` provisions loopback-only, ephemeral MySQL/MariaDB, Redis,
+  SMTP sink, and HTTP fixture services. Every service has idempotent teardown and
+  typed disruption controls.
+
+Generic layers contain no component, plugin, vendor integration, or product
+policy. WordPress grammars and attack packs remain extension responsibilities.
+
+## Campaign contract
+
+```ts
+const campaign = adversarialCampaign({
+  id: "component-campaign",
+  seed: "ci-seed-1",
+  corpus: [{
+    id: "create-update-read",
+    actions: [
+      { type: "create", input: { title: "seed" } },
+      { type: "update", input: { title: "candidate" } },
+      { type: "read" },
+    ],
+  }],
+  budgets: {
+    maxCases: 1000,
+    workers: 4,
+    maxCaseTimeMs: 30000,
+    maxWallTimeMs: 300000,
+  },
+  oracles: [{
+    schema: "wp-codebox/adversarial-oracle/v1",
+    id: "state-integrity",
+    severity: "high",
+  }],
+})
+
+const result = await runAdversarialCampaign(campaign, {
+  execute: async (plan, signal) => adapter.execute(plan, signal),
+  evaluate: async (plan, observation, oracles) =>
+    adapter.evaluate(plan, observation, oracles),
+})
+```
+
+Workers execute a deterministic round concurrently. Results are committed to the
+corpus in case-index order after every round, so completion timing cannot change
+the retained corpus or later mutations. Replay metadata records the seed,
+worker, iteration, matrix cell, minimized actions/input, fault schedule, exact
+provenance, normalized schedule, and a one-command replay instruction.
+
+## Mutation and novelty
+
+Core mutation supports generic scalar, structured, binary, and stateful sequence
+inputs. Mutation targets are selected from stable JSON paths using a seeded
+SHA-256 schedule. Extensions can represent multipart, serialized, markup, or
+domain-specific inputs as structured actions and register richer mutators at the
+adapter boundary without adding those grammars to runtime-core.
+
+Adapters return bounded novelty signals such as coverage edges, route ids, state
+digests, hook ids, or query fingerprints. New signals retain a case in the
+interesting corpus. Minimization removes action chunks and shrinks input values,
+replaying each candidate through the same adapter and oracle set.
+
+## Fault model
+
+```json
+{
+  "schema": "wp-codebox/transport-fault-model/v1",
+  "seed": "fault-seed-1",
+  "rules": [{
+    "id": "verification-outage",
+    "match": {
+      "host": "service.example",
+      "method": "POST",
+      "path": "/verify"
+    },
+    "sequence": [
+      { "status": 500 },
+      { "delayMs": 30000, "timeoutMs": 1000 },
+      { "status": 200, "body": "malformed", "truncateAfterBytes": 4 }
+    ],
+    "repeat": "last"
+  }]
+}
+```
+
+Each adapter publishes `wp-codebox/transport-fault-capabilities/v1`. Campaigns
+must negotiate before execution and fail closed when a required semantic is
+unsupported.
+
+### Playwright fidelity
+
+| Semantic | Fidelity |
+| --- | --- |
+| Response status/header/body substitution | Exact |
+| Delay, deterministic jitter, host remap | Exact |
+| Malformed/truncated payload, bandwidth, timeout, refusal/reset | Emulated and labeled |
+| Chunk framing, half-close, disconnect-after-N-bytes | Unsupported |
+
+Playwright owns HTTP framing and sockets, so payload truncation must not be
+reported as a transport disconnect. A lower-level proxy provider is required for
+those exact semantics.
+
+## Services
+
+Recipe `inputs.services` accepts:
+
+- `mysql`: MySQL 8.4 or MariaDB 11.4, preserving existing output and readiness
+  behavior.
+- `redis`: ephemeral Redis with persistence disabled.
+- `smtp`: SMTP message sink plus its loopback inspection port.
+- `http`: deterministic loopback response fixture.
+
+The provisioned service set exposes `control(serviceId, action, options)`.
+Container lifecycle actions (`stop`, `start`, `pause`, `resume`, `restart`,
+`disconnect`, and `reconnect`) have exact Docker fidelity. Provider-specific
+`flush` and read-only controls are exact where implemented. Network latency is
+explicitly unsupported by this provider and should use the transport-fault
+adapter instead.
+
+All containers bind ephemeral loopback ports, use no persistent volumes, and are
+removed in reverse order after success, failure, cancellation, or timeout.
+
+## Clock control
+
+Playwright browser time supports exact freeze, advance, skew, and resume through
+its clock API. The same capability response explicitly marks server process,
+scheduler, and database clocks unsupported. A WordPress runtime extension is
+required to control those surfaces without faking server behavior in browser
+JavaScript.
+
+## Browser oracles
+
+The generic browser planner derives actions from visible control descriptors and
+generates empty, oversized, hostile punctuation, bidi, and Unicode inputs. Submit
+controls are repeated to expose duplicate side effects. Generic oracles cover:
+
+- page errors, console errors, and unhandled rejections;
+- controls that accept input without an observable action;
+- loading indicators that exceed a declared threshold;
+- clipping, viewport overflow, and invisible focus;
+- adapter-provided accessibility violations;
+- duplicate observable effects.
+
+Journey minimization uses deterministic subset replay to retain the shortest
+sequence that preserves the oracle failure.
+
+## Evidence and safety
+
+`writeAdversarialEvidenceBundle()` writes a manifest, campaign result, one
+finding and replay document per stable fingerprint, and a secret-scan report.
+The writer enforces an artifact byte ceiling, redacts sensitive fields and
+caller-declared values, removes machine-specific paths, and records SHA-256 file
+digests plus a stable bundle content digest.
+
+Campaign budgets independently bound cases, actions, input bytes, per-case and
+wall time, worker count, and artifact bytes. Adapters remain responsible for the
+existing Codebox CPU, memory, disk, process, network-deny, mount, and disposable
+sandbox boundaries.
+
+## Compatibility
+
+Existing recipes and `fuzzRun` cases are unchanged. Existing MySQL defaults,
+outputs, readiness, and teardown behavior remain compatible. The new service
+kinds and public TypeScript contracts are additive.
+
+## Current boundaries
+
+The following capabilities are intentionally not claimed by this change:
+
+- exact socket framing faults require a lower-level proxy provider;
+- server-side WordPress HTTP fault interception requires a WordPress extension
+  adapter using the generic fault contract;
+- PHP/WordPress, cron, and database clock control require a WordPress extension;
+- WordPress-specific mutation grammars, security policies, and instrumentation
+  remain extension-owned;
+- live vulnerable plugin/theme discovery campaigns require disposable runtime
+  fixtures and are not replaced by the neutral contract tests.
+
+These are capability gaps, not silent skips. Consumers can inspect negotiation
+results before running a campaign.

package.json

@@ -230,6 +230,7 @@
     "test:recipe-run-provenance": "tsx tests/recipe-run-provenance.test.ts",
     "test:fuzz-run-recipe": "tsx tests/fuzz-run-recipe.test.ts",
     "test:fuzz-suite-runner": "tsx tests/fuzz-suite-runner.test.ts",
+    "test:adversarial-runtime": "tsx --test tests/transport-faults.test.ts tests/adversarial-campaign.test.ts tests/adversarial-browser.test.ts tests/browser-clock-control.test.ts",
     "test:playground-fuzz-suite-public": "tsx tests/playground-fuzz-suite-public.test.ts",
     "test:nested-fuzz-suite-recipe-command": "tsx tests/nested-fuzz-suite-recipe-command.test.ts",
     "test:wordpress-fuzz-suite-builders": "tsx tests/wordpress-fuzz-suite-builders.test.ts",

packages/cli/src/recipe-validation.ts

@@ -742,9 +742,15 @@ function validateRecipeRuntimeServices(recipe: WorkspaceRecipe, addIssue: (code:
     if (!/^[A-Za-z0-9][A-Za-z0-9_.-]*$/.test(service.id)) addIssue("invalid-runtime-service-id", `${path}.id`, "Runtime service ids must be stable identifiers.")
     if (ids.has(service.id)) addIssue("duplicate-runtime-service-id", `${path}.id`, `Runtime service ids must be unique: ${service.id}`)
     ids.add(service.id)
-    if (service.kind !== "mysql") addIssue("unsupported-runtime-service-kind", `${path}.kind`, `Unsupported managed runtime service kind: ${service.kind}`)
+    if (!["mysql", "redis", "smtp", "http"].includes(service.kind)) addIssue("unsupported-runtime-service-kind", `${path}.kind`, `Unsupported managed runtime service kind: ${service.kind}`)
+    const supportedOutputs: Record<string, RegExp> = {
+      mysql: /^(host|port|username|password|database)$/,
+      redis: /^(host|port|url)$/,
+      smtp: /^(host|port|httpPort|url)$/,
+      http: /^(host|port|url)$/,
+    }
     for (const [output, name] of Object.entries(service.outputs)) {
-      if (!/^(host|port|username|password|database)$/.test(output)) addIssue("unknown-runtime-service-output", `${path}.outputs.${output}`, `Unsupported ${service.kind} service output: ${output}`)
+      if (!(supportedOutputs[service.kind] ?? /^$/).test(output)) addIssue("unknown-runtime-service-output", `${path}.outputs.${output}`, `Unsupported ${service.kind} service output: ${output}`)
       if (!/^[A-Z_][A-Z0-9_]*$/.test(name)) addIssue("invalid-runtime-service-env", `${path}.outputs.${output}`, "Runtime service environment variable names must match /^[A-Z_][A-Z0-9_]*$/.")
       if (environment.has(name)) addIssue("duplicate-runtime-service-env", `${path}.outputs.${output}`, `Runtime service output environment variable is already declared: ${name}`)
       environment.add(name)