diff --git a/.changeset/aggregator-get-publisher.md b/.changeset/aggregator-get-publisher.md new file mode 100644 index 0000000000..8cef3b8b25 --- /dev/null +++ b/.changeset/aggregator-get-publisher.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-lexicons": minor +--- + +Adds an aggregator `getPublisher` query and `publisherView` definition, returning a publisher's signed profile (display name, description, and identity-level contact channels) by DID alongside any hydrated labels — the publisher counterpart to the existing `getPackage` read. diff --git a/.changeset/aggregator-publisher-verification-read.md b/.changeset/aggregator-publisher-verification-read.md new file mode 100644 index 0000000000..7b85fce193 --- /dev/null +++ b/.changeset/aggregator-publisher-verification-read.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-lexicons": minor +--- + +Adds the `com.emdashcms.experimental.aggregator.getPublisherVerification` query and its `publisherVerificationView` / `verificationClaimView` definitions: read the verification claims an aggregator has indexed for a publisher DID. diff --git a/.changeset/core-ssrf-doh-export.md b/.changeset/core-ssrf-doh-export.md new file mode 100644 index 0000000000..78d17abe28 --- /dev/null +++ b/.changeset/core-ssrf-doh-export.md @@ -0,0 +1,5 @@ +--- +"emdash": patch +--- + +Adds a `emdash/security/ssrf` subpath export exposing the `cloudflareDohResolver` DNS-over-HTTPS resolver and the SSRF URL/address validation helpers for reuse in Workers that fetch untrusted URLs. diff --git a/.changeset/cozy-worlds-smile.md b/.changeset/cozy-worlds-smile.md deleted file mode 100644 index a845151cc8..0000000000 --- a/.changeset/cozy-worlds-smile.md +++ /dev/null @@ -1,2 +0,0 @@ ---- ---- diff --git a/.changeset/discovery-empty-accept-labelers.md b/.changeset/discovery-empty-accept-labelers.md new file mode 100644 index 0000000000..ce9918563f --- /dev/null +++ b/.changeset/discovery-empty-accept-labelers.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-client": patch +--- + +Sends an explicitly empty `atproto-accept-labelers` header when `DiscoveryClient` is configured with `acceptLabelers: ""`, so "accept no labelers" is honored instead of being dropped and letting the aggregator apply its trusted defaults. Omitting the option entirely still sends no header. diff --git a/.changeset/graduate-registry-verification.md b/.changeset/graduate-registry-verification.md deleted file mode 100644 index 5dc6fa3b8d..0000000000 --- a/.changeset/graduate-registry-verification.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -"@emdash-cms/registry-verification": minor ---- - -Adds `@emdash-cms/registry-verification`: runtime-neutral primitives for verifying plugin registry release artifacts. Validates multihash checksums, fetches artifacts with size and redirect guards, checks canonical tarball and bundle structure, and verifies Sigstore build provenance. Runs on both Node and workerd. diff --git a/.changeset/loud-jobs-trade.md b/.changeset/loud-jobs-trade.md deleted file mode 100644 index a845151cc8..0000000000 --- a/.changeset/loud-jobs-trade.md +++ /dev/null @@ -1,2 +0,0 @@ ---- ---- diff --git a/.changeset/lucky-labelers-admin-moderation.md b/.changeset/lucky-labelers-admin-moderation.md new file mode 100644 index 0000000000..faa7a163d0 --- /dev/null +++ b/.changeset/lucky-labelers-admin-moderation.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/admin": patch +--- + +The registry browser now shows moderation state: blocked releases are visibly blocked with an explanation, warning-labelled releases ask for confirmation before install, and the plugin manager flags blocked or warned updates. diff --git a/.changeset/lucky-labelers-contract.md b/.changeset/lucky-labelers-contract.md new file mode 100644 index 0000000000..9baa130613 --- /dev/null +++ b/.changeset/lucky-labelers-contract.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-lexicons": patch +--- + +Adds experimental labeler assessment and policy query contracts. diff --git a/.changeset/lucky-labelers-search-description.md b/.changeset/lucky-labelers-search-description.md new file mode 100644 index 0000000000..f684e0a53f --- /dev/null +++ b/.changeset/lucky-labelers-search-description.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-lexicons": patch +--- + +Updates the `searchPackages` description to the canonical moderation vocabulary. diff --git a/.changeset/quiet-labels-moderate.md b/.changeset/quiet-labels-moderate.md new file mode 100644 index 0000000000..7160ec39cb --- /dev/null +++ b/.changeset/quiet-labels-moderate.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/registry-client": patch +--- + +Adds release moderation evaluation over hydrated aggregator labels and exposes the atproto-content-labelers response header. diff --git a/.changeset/registry-moderation-stack-initial.md b/.changeset/registry-moderation-stack-initial.md new file mode 100644 index 0000000000..bf7847fdb1 --- /dev/null +++ b/.changeset/registry-moderation-stack-initial.md @@ -0,0 +1,9 @@ +--- +"@emdash-cms/registry-moderation": minor +"@emdash-cms/registry-verification": minor +--- + +Initial release of the plugin-registry moderation stack. + +- `@emdash-cms/registry-moderation`: ATProto signed-label parsing, encoding, and public-key verification; release moderation evaluation over both signed and aggregator-hydrated labels (package and publisher cascades, CID-bound labels, negation and expiry); the shared hard-block and warning label vocabularies; the `atproto-accept-labelers` / `atproto-content-labelers` negotiation headers; and `isModerationBlocking`, the canonical blocking predicate for enforcement consumers. +- `@emdash-cms/registry-verification`: runtime-neutral primitives (Node and workerd) for verifying plugin release artifacts — multihash checksums, SSRF-guarded artifact fetch, canonical tarball and bundle structure validation with a validated file inventory, and Sigstore build-provenance verification. diff --git a/.changeset/tidy-labels-enforce.md b/.changeset/tidy-labels-enforce.md new file mode 100644 index 0000000000..329235eb61 --- /dev/null +++ b/.changeset/tidy-labels-enforce.md @@ -0,0 +1,5 @@ +--- +"emdash": patch +--- + +Registry install and update now evaluate the full moderation label set (package and publisher cascades, CID-bound labels, negation and expiry) instead of a single yanked-label string match. The update-check and artifact proxy endpoints also gate on this evaluation. diff --git a/.changeset/tidy-plugins-eligible.md b/.changeset/tidy-plugins-eligible.md new file mode 100644 index 0000000000..3701c125b3 --- /dev/null +++ b/.changeset/tidy-plugins-eligible.md @@ -0,0 +1,5 @@ +--- +"@emdash-cms/plugin-cli": patch +--- + +Search and info now show moderation eligibility for registry packages instead of raw label values. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fa2ba95fa8..ce5e1d9e93 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -124,6 +124,9 @@ jobs: - run: pnpm test:unit env: EMDASH_TEST_PG: postgres://postgres:test@localhost:5432/emdash_test + - run: pnpm --filter @emdash-cms/labeler build + - run: pnpm --filter @emdash-cms/labeler typecheck + - run: pnpm --filter @emdash-cms/labeler test # Render tests use the Astro Vite plugin (vitest.repro.config.ts); # they can't run under the plain-node config in test:unit. - run: pnpm --filter emdash exec vitest run --config vitest.repro.config.ts diff --git a/.gitignore b/.gitignore index 5e4e771b8b..c63bb485a6 100644 --- a/.gitignore +++ b/.gitignore @@ -75,6 +75,10 @@ examples/wp-theme-unit-test/ # pattern so we can re-include the agents subdirectory. .opencode/* !.opencode/agents/ +!.opencode/plans/ +.opencode/plans/* +!.opencode/plans/plugin-registry-labelling-service/ +!.opencode/plans/plugin-registry-labelling-service/** # .claude is local-only EXCEPT for the symlinks pointing at AGENTS.md and skills/ .claude/* diff --git a/.opencode/plans/plugin-registry-labelling-service/gate-0/contracts.md b/.opencode/plans/plugin-registry-labelling-service/gate-0/contracts.md new file mode 100644 index 0000000000..6eddba37e9 --- /dev/null +++ b/.opencode/plans/plugin-registry-labelling-service/gate-0/contracts.md @@ -0,0 +1,360 @@ +# Gate 0 Contract Ratification Proposal + +Status: ratified contracts for `W0.1` and `W0.2` as of 2026-07-10. This is not a production Lexicon or protocol implementation. + +Companions: [implementation spec](../spec.md), [implementation plan](../implementation-plan.md), [moderation policy fixture](../../../../apps/labeler/fixtures/moderation-policy.json), and [moderation cases](../../../../packages/registry-moderation/tests/fixtures/moderation-cases.json). + +## Amendments + +- 2026-07-10 (#1926): the unshipped surface was standardized on US `labeler` spelling. The ratified public NSIDs are `com.emdashcms.experimental.labeler.*`; identifiers in this document were updated to match. +- 2026-07-10 (#1917): the implemented evaluator added `applicableLabels: ModerationLabel[]` to `ReleaseModeration` for consumer display. +- The ratified fixtures moved from this folder into product paths: `apps/labeler/fixtures/moderation-policy.json` and `packages/registry-moderation/tests/fixtures/moderation-cases.json`. +- 2026-07-12 (W5 enforcement PR): the corpus's `oldPackageCid` value (`...moixb`, two occurrences in `eligible-package-mismatched-cid-label-ignored`) was not a decodable CID — a latent defect in the originally ratified corpus, inert until the hydrated evaluation path added structural CID validation. Corrected to a verified round-tripping CID (`...moixe`); the case's expected outcome is unchanged. + +## Scope + +This proposal freezes public identifiers, public API shapes, label issuance authority, current-label reduction, subject applicability, and official-client moderation precedence. Standard label distribution continues to use `com.atproto.label.*`; no EmDash label-distribution Lexicon is proposed. + +Subjects use the current experimental registry identifiers: + +- Package: `com.emdashcms.experimental.package.profile` +- Release: `com.emdashcms.experimental.package.release` +- Publisher: a DID, optionally described by `com.emdashcms.experimental.publisher.profile` +- Reference labeler: `did:web:labels.emdashcms.com` + +## Proposed Public Identifiers + +| NSID | Kind | Purpose | +| --------------------------------------------------------- | ----------- | -------------------------------------------------------------------------- | +| `com.emdashcms.experimental.labeler.defs` | definitions | Shared public assessment, policy, coverage, label, and manual-action views | +| `com.emdashcms.experimental.labeler.getAssessment` | query | Fetch one immutable historical assessment by public ID | +| `com.emdashcms.experimental.labeler.getCurrentAssessment` | query | Fetch effective assessment state for an exact release URI and CID | +| `com.emdashcms.experimental.labeler.listAssessments` | query | Page through public assessments using bounded filters | +| `com.emdashcms.experimental.labeler.getPolicy` | query | Fetch the current machine-readable policy document | + +The service also implements `com.atproto.label.queryLabels` and `com.atproto.label.subscribeLabels` without redefining them. + +### Decision notice + +No decision-notice repository record ships in v1. Labels provide the subscribable decision stream, public assessment queries provide explanations, and email remains the active publisher-notification channel. A future notice record can use a new experimental NSID without changing these queries. + +### Well-known policy path + +The permanent discovery URL is: + +```text +https://labels.emdashcms.com/.well-known/emdash-labeler-policy.json +``` + +It returns the same policy object as `com.emdashcms.experimental.labeler.getPolicy`, with `Content-Type: application/json`. Responses use a representation-derived ETag and may be publicly cached for at most five minutes. The policy document is informational; stale policy data never changes whether a signed label is current. + +## Shared Public Shapes + +These shapes are inputs to `W1.2`. They intentionally use only fixed object properties, arrays, and other Lexicon-expressible structures. All datetimes are validated RFC 3339 `datetime` values. + +### Assessment ID + +Assessment IDs are opaque, case-sensitive `asmt_<26-character canonical uppercase ULID>` strings. Example: `asmt_01J2Q5Y7V8N9M0K1H2G3F4E5D6`. The ULID timestamp is allocation metadata only; clients use response timestamps for ordering and display. + +### `publicAssessment` + +```ts +interface PublicAssessment { + id: string; + src: string; + subject: { uri: string; cid: string }; + artifact?: { id?: string; checksum: string }; + state: "pending" | "passed" | "warned" | "blocked" | "error" | "superseded"; + summary: string; + coverage: { + code: "complete" | "partial" | "unavailable"; + metadata: "complete" | "partial" | "unavailable"; + images: "complete" | "not-present" | "partial" | "unavailable"; + }; + labels: Array<{ + val: string; + active: boolean; + issuedAt: string; + expiresAt?: string; + }>; + policyVersion: string; + assessmentSchemaVersion: number; + model?: { + provider: "workers-ai"; + modelId: string; + promptVersion: string; + }; + createdAt: string; + completedAt?: string; + supersedesAssessmentId?: string; + reconsiderationUrl: string; +} +``` + +`artifact` and `model` are absent when unavailable. `state: "superseded"` is derived only when a newer completed assessment names this assessment and owns the current pointer. + +### `publicManualAction` + +```ts +interface PublicManualAction { + id: string; + src: string; + subject: { uri: string; cid?: string }; + type: "override" | "label-issue" | "label-retraction" | "emergency-takedown"; + summary: string; + labels: Array<{ val: string; active: boolean; issuedAt: string; expiresAt?: string }>; + createdAt: string; +} +``` + +This is a public summary, not the private operator reason or evidence record. + +### Policy document + +`apps/labeler/fixtures/moderation-policy.json` is the canonical example. Its Lexicon-expressible shape is: + +```ts +interface LabelerPolicyDocument { + schemaVersion: 1; + policyVersion: string; + effectiveAt: string; + labelerDid: string; + assessmentSchemaVersion: number; + supportedSubjects: { + publisher: { kind: "did" }; + packageCollections: string[]; + releaseCollections: string[]; + }; + reasonCodes: Array<{ code: string; description: string }>; + labels: Array<{ + value: string; + category: "eligibility" | "automated-block" | "warning" | "manual-system"; + officialEffect: "pass" | "pending" | "error" | "block" | "warn" | "redact"; + subjectRules: Array<{ + subject: "release" | "package" | "publisher"; + cidRule: "required" | "optional" | "forbidden"; + issuanceModes: Array<"automated" | "reviewer" | "admin">; + }>; + locales: Array<{ lang: string; name: string; description: string }>; + }>; + overrideRule: { + subject: "release"; + cidRule: "required"; + reviewerLabels: string[]; + requireSameSource: true; + requireAtomicIssuance: true; + }; + precedence: string[]; + publicApi: { + baseUrl: string; + policyUrl: string; + getAssessmentNsid: string; + getCurrentAssessmentNsid: string; + listAssessmentsNsid: string; + getPolicyNsid: string; + }; + contact: { reconsiderationUrl: string; reconsiderationEmail: string }; + transparency: { modelOutputIsAdvisoryEvidence: true }; +} +``` + +`issuanceModes` are subject-specific. An admin inherits reviewer capability, but the policy still lists `admin` where an action is admin-only. V1 advertises only deployed collections; clients do not infer wildcards. + +## Endpoint Contracts + +All four custom methods are unauthenticated `query` Lexicons served under `/xrpc/{NSID}`. + +### `getAssessment` + +Parameter: required `id`, maximum 64 bytes. Output: one `publicAssessment`. + +Errors: `InvalidRequest`, `NotFound`, `RateLimitExceeded`. + +`NotFound` does not distinguish never-existing, non-public, or retention-removed private data. A public summary referenced by an issued label remains resolvable. + +### `getCurrentAssessment` + +| Name | Required | Constraint | Meaning | +| ----- | -------- | --------------------------- | ---------------------------------------------- | +| `uri` | yes | exact `at-uri`, no wildcard | Release record URI | +| `cid` | yes | CID | Exact release record version being evaluated | +| `src` | no | DID | Labeler source; defaults to the endpoint's DID | + +```ts +interface CurrentAssessmentView { + src: string; + subject: { uri: string; cid: string }; + current?: PublicAssessment; + pending?: PublicAssessment; + activeLabels: Array<{ + src: string; + uri: string; + cid?: string; + val: string; + cts: string; + exp?: string; + }>; + override?: PublicManualAction; +} +``` + +`current`, `pending`, active labels, and override remain separate concepts. Active labels exclude a stream whose winner is negated, expired, or CID-inapplicable. Unknown URI/CID returns `NotFound`; a different `src` returns `UnsupportedSource` rather than an ambiguous empty response. + +Errors: `InvalidRequest`, `NotFound`, `UnsupportedSource`, `RateLimitExceeded`. + +### `listAssessments` + +| Name | Required | Constraint | +| -------- | -------- | --------------------------------- | +| `src` | no | endpoint DID only | +| `uri` | no | exact `at-uri`, no wildcard | +| `cid` | no | CID; requires `uri` | +| `state` | no | one public assessment state | +| `limit` | no | integer 1-100, default 50 | +| `cursor` | no | opaque string, maximum 1024 bytes | + +Output: `{ assessments: PublicAssessment[], cursor?: string }`. + +Ordering is `createdAt DESC, id DESC` with exclusive keyset pagination. The opaque base64url cursor encodes cursor version, final parsed `createdAt`, final `id`, and a hash of effective filters. A cursor is emitted only when another row exists. Changed filters, malformed data, or an unknown cursor version returns `InvalidCursor`, never page one. + +Errors: `InvalidRequest`, `InvalidCursor`, `UnsupportedSource`, `RateLimitExceeded`. + +### `getPolicy` + +No parameters. Output: current `LabelerPolicyDocument`. Historical policies remain operational artifacts in v1 rather than a public query surface. + +Error: `RateLimitExceeded`. + +### Error codes + +| Code | HTTP status | Meaning | +| --------------------- | ----------- | ----------------------------------------------------------------------- | +| `InvalidRequest` | 400 | Lexicon or cross-field validation failed | +| `InvalidCursor` | 400 | Cursor is malformed, unsupported, or bound to different filters | +| `UnsupportedSource` | 400 | `src` differs from this deployment's labeler DID | +| `NotFound` | 404 | Requested public assessment or exact subject is unavailable | +| `RateLimitExceeded` | 429 | Public rate limit exceeded; response should include `Retry-After` | +| `InternalServerError` | 500 | Generic XRPC error for an unhandled server failure, not method-declared | + +`com.atproto.moderation.createReport` returns `NotSupported` and persists nothing in v1. + +## Moderation Evaluation Contract + +`packages/registry-moderation/tests/fixtures/moderation-cases.json` is executable input for `W1.5`. + +### Inputs and accepted sources + +The evaluator receives a parsed evaluation instant, exact publisher/package/release subjects and current CIDs, accepted labeler DIDs with `redact` flags, and label history. Missing `atproto-accept-labelers` is resolved to deployment defaults before evaluation; explicitly empty means no accepted sources and therefore no qualifying positive assessment. + +Unaccepted sources are ignored. Every accepted source is reduced and evaluated independently. A source's valid override suppresses only that source's automated states. It never suppresses another accepted source's pending, error, automated block, warning, or manual action. + +The `redact` flag controls presentation only. An accepted `!takedown` always blocks installation and marks its subject redacted only when that source has `redact: true`. + +### Current label state + +Current state has exactly the ATProto key `(src, uri, val)`. CID is not part of the stream key. + +For each stream: + +1. Parse every validated `cts` into an instant; never compare raw RFC 3339 text. +2. Select the event with the greatest parsed `cts`. +3. Treat semantically identical events at that instant as duplicate delivery. Semantic identity compares `ver`, `src`, `uri`, `cid`, `val`, `neg`, `cts`, and `exp`; signatures and transport metadata do not break a tie. +4. If non-identical events share the greatest parsed `cts`, mark that source's stream ambiguous. The official evaluator returns fail-closed `error` with `label-state-collision`; neither event wins. +5. If the winner has `neg: true`, the stream is inactive. A negation replaces the stream regardless of its CID. +6. Otherwise parse `exp`; the stream is inactive when `exp <= evaluatedAt`. +7. Apply the winner's CID metadata to the current subject only after current-state selection. + +An expired winner does not reveal or reactivate an older event. An expired negation therefore leaves the stream inactive. Invalid `cts`/`exp` values are rejected before evaluation by label validation. + +Subscription sequence and query pagination order are cursor/replay bookkeeping only. They never select current moderation state or resolve equal-`cts` collisions. + +Subject issuance rules constrain what the reference issuer may sign; they do not alter the ATProto stream key. A consumer does not discard an accepted signed event before stream reduction merely because its CID would violate the current issuer policy. This lets a later CID-bearing negation safely retract an older URI-wide event and prevents legacy or buggy metadata from splitting one stream into two. + +### Subject and CID applicability + +- Release labels apply only to the exact release URI. A CID-bound winner applies only when its CID equals the current release CID; a URI-wide winner omits CID. +- Package labels apply only to the package URI. A CID-bound winner applies only to the current package CID; a URI-wide winner cascades to all package releases. +- Publisher labels target the publisher DID, omit CID, and cascade to all packages/releases from that DID. +- Automated eligibility, descriptive security, and warning labels are release-only and require release CID. +- `security-yanked` and release/package/publisher `!takedown` are URI-wide. +- `package-disputed` may be CID-bound to the current profile or URI-wide; this is a ratification proposal needed to exercise package-CID applicability without allowing manual issuance of automated descriptive values. +- CID mismatch makes the selected winner inapplicable; it does not restore an older same-stream event. + +### Issuance authority and provenance + +Standard labels do not prove whether an event came from automation or an operator. V1 therefore does not assign different consumer effects to the same descriptive value based on external action metadata. + +- Automated descriptive security and warning values are release-only and automated-only. +- Reviewers cannot manually issue those values. They use `security-yanked` for a release or `package-disputed` for a package. +- Admins use `!takedown` and `publisher-compromised` for emergency/broad actions. +- `assessment-passed` permits automated and reviewer issuance. Its reviewer meaning is recognized only when the same source also has active exact-CID `assessment-overridden`. +- `assessment-overridden` is reviewer-issued and exact-CID. +- The issuer permits a reviewer `assessment-passed` only in one atomic action with `assessment-overridden` for the same release URI and CID. It never offers a standalone reviewer-pass operation. +- The issuer enforces allowed subject/mode combinations before signing. Public action metadata explains an action but never changes evaluator semantics. + +### Per-source evaluation and aggregation + +For each accepted source: + +1. Apply manual blocks from that source. +2. Recognize an override only from active exact-CID `assessment-passed` plus `assessment-overridden` from that source. +3. Suppress that source's automated pending, error, and descriptive block values when its override is valid. +4. Otherwise classify active `assessment-error`, `assessment-pending`, automated blocks, pass, and warnings. + +Aggregate unsuppressed source results in this order: + +1. Any manual block: `blocked`. +2. Any label-state collision: `error`. +3. Any unsuppressed assessment error: `error`. +4. Any unsuppressed pending state: `pending`. +5. Any unsuppressed automated block: `blocked`. +6. No accepted source has an active pass or valid override: `blocked` with `missing-assessment-pass`. +7. Otherwise: `eligible`, with warnings from all accepted sources. + +A pass or override from source A cannot bypass pending, error, or block from source B. An accepted source that has no relevant state does not veto a pass from another accepted source. + +### Result fields + +```ts +interface ReleaseModeration { + eligibility: "eligible" | "pending" | "error" | "blocked"; + reasonCodes: string[]; + blockingLabels: string[]; + stateLabels: string[]; + warningLabels: string[]; + suppressedLabels: string[]; + applicableLabels: ModerationLabel[]; + redacted: boolean; +} +``` + +- `blockingLabels`: only active, applicable manual or automated labels whose policy effect is block/redact. +- `stateLabels`: active, applicable `assessment-pending` and `assessment-error` values. They never appear in `blockingLabels`. +- `warningLabels`: active, applicable warning values. +- `suppressedLabels`: same-source automated state/block values hidden by a valid override but retained for display. +- `applicableLabels`: the full active, applicable label objects across accepted sources, for consumer display. +- Label references in fixtures are ordered by source alias, subject, then value. Reasons follow aggregate precedence and are deduplicated. + +## Ratified Decisions + +1. Approve the five proposed NSIDs and no v1 decision-notice record. +2. Approve `asmt_` public assessment IDs. +3. Approve exact URI+CID current lookup and `createdAt DESC, id DESC` keyset pagination. +4. Approve `UnsupportedSource` for a foreign `src` on this single-DID deployment. +5. Approve the Lexicon-expressible policy shape, fixed API properties, array reason/scanner entries, and permanent well-known path. +6. Approve `(src, uri, val)` current-state keys, parsed-instant `cts` ordering, and sequence-independent reduction. +7. Approve fail-closed `error` for a non-identical equal-`cts` collision. +8. Approve reducing accepted signed events before applying issuer-policy CID restrictions. +9. Approve expired winners never revealing older events, including expired negations. +10. Approve subject-specific issuance rules and automated release-only descriptive labels. +11. Approve that automated descriptive security/warning values cannot be manually issued in v1; reviewers/admins use the dedicated manual values instead. +12. Approve consumer override recognition solely from the same-source exact-CID pass/override pair, because standard labels cannot prove action provenance. +13. Approve that reviewer-issued pass is available only as one atomic pass/override pair, never as a standalone action. +14. Approve independent per-source evaluation and aggregate error > pending > automated block > positive-assessment resolution after manual blocks/collisions. +15. Approve accepted `!takedown` always blocking while `redact` controls presentation. +16. Approve `package-disputed` as warning/non-recommendation, not a direct-install block. +17. Approve optional CID binding for `package-disputed`; URI-wide disputes persist, while CID-bound disputes apply only to one package-profile version. +18. Approve manual release/package/publisher blocks taking precedence over overrides. +19. Approve `stateLabels` as the result field for pending/error labels, separate from `blockingLabels`. + +No existing spec, plan, or production Lexicon was changed. The policy document and fixtures were ratified together on 2026-07-10; `W1.2` and `W1.5` may treat them as frozen once this PR merges into the integration branch. diff --git a/.opencode/plans/plugin-registry-labelling-service/gate-0/crypto-interop.md b/.opencode/plans/plugin-registry-labelling-service/gate-0/crypto-interop.md new file mode 100644 index 0000000000..4e020c04e1 --- /dev/null +++ b/.opencode/plans/plugin-registry-labelling-service/gate-0/crypto-interop.md @@ -0,0 +1,79 @@ +# Gate 0: Label crypto interoperability + +Status: P-256 interoperability and key-lifecycle contract ratified on 2026-07-10. + +## Decision + +Use P-256 (`secp256r1`) with `@atcute/cbor` 2.3.3 and `@atcute/crypto` 2.4.1. The production secret format is one unpadded base64url-encoded 32-byte P-256 private scalar. Decode it and pass the raw bytes to `P256PrivateKey.importRaw`. Publish the compressed public key as an ATProto P-256 multikey in the issuer DID document's exact `#atproto_label` verification method. + +P-256 is selected over k256 because the atcute implementation uses the WebCrypto `P-256`/`SHA-256` implementation available in workerd. Signatures are low-S, 64-byte IEEE-P1363 values. Do not accept DER signatures. A startup check must derive the public multikey from the secret and require it to equal the DID/configured key before issuance starts. + +The high-level atcute `P256PrivateKey.sign(data)` and `P256PublicKey.verify(sig, data)` APIs perform SHA-256 as part of WebCrypto ECDSA. They therefore receive canonical CBOR bytes, not an already computed digest. Passing that digest to these APIs would hash twice. + +## Signed shape + +The W0 spike reconstructed a label from this unsigned v1 allowlist only: + +`ver`, `src`, `uri`, `cid`, `val`, `neg`, `cts`, `exp` + +It requires `ver: 1`; validates `src` as a DID, `uri` as an ATProto generic URI, `cid` with the ATProto CID validator, and `cts`/`exp` as RFC3339 datetimes with valid calendar dates; requires `val` to contain 1 to 128 UTF-8 bytes; and rejects `sig`, `$type`, and every unknown field before DRISL encoding. An input `neg: false` is omitted from the reconstructed object. `sig` is attached only after signing. The signer must never accept an arbitrary object. + +## Key decoding and DID resolution + +Private scalar decoding is a signer boundary, not a permissive utility. It accepts only the unpadded base64url alphabet, requires canonical re-encoding to exactly the supplied string, requires exactly 32 decoded bytes, interprets them as an unsigned big-endian integer, and requires `1 <= d < n` for the P-256 order `n`. Padded, malformed, non-canonical, wrong-length, zero, and order-or-larger values fail before `P256PrivateKey.importRaw`. + +Resolve the issuer DID document and normalize relative `#atproto_label` and fully qualified `${issuerDid}#atproto_label` IDs to the same logical method. Either form is valid alone; any duplicate across relative or fully qualified forms fails independent of order. The DID document ID and method controller must equal the issuer DID. The method must use `type: "Multikey"`, include `publicKeyMultibase`, identify the P-256 codec, and contain an importable 33-byte compressed P-256 point. At startup, derive the signer public multikey from the configured private scalar and require exact equality with the resolved method's `publicKeyMultibase`; a valid but different P-256 key fails. A duplicate method, legacy key type, wrong controller, k256 key, malformed point, missing label key, or document containing only `#atproto` fails. There is no fallback to another verification method. + +## Independent source + +The independent implementation is Bluesky's `@atproto/crypto` 0.4.5 from , dual licensed MIT or Apache-2.0. It is independent of the Mary-ext atcute implementation and uses Noble Curves rather than WebCrypto for P-256. + +The spike used test-only private scalar `0x01`: + +- `@atcute/cbor` produced the canonical payload bytes. +- `@atproto/crypto` produced the deterministic `atprotoReferenceHex` signature; workerd/atcute verifies it. +- Actual workerd `@atcute/crypto` produced a signature that `@atproto/crypto.verifySignature` accepted. +- Both implementations derived the same compressed public key and P-256 `did:key` multikey. + +The spike vector was test material, not a usable deployment key. It established the selected API/key contract but is not retained as a product test fixture because no EmDash signer/verifier exists yet. When W1.4 implements `signLabel` and `verifyLabel`, it must add fresh external vectors and cross-implementation tests alongside that implementation. This Gate 0 document is explanatory only; retained tests and production code must not import or otherwise depend on files under `.opencode/plans`. + +## Spike result + +The feasibility spike ran both workerd atcute and independent `@atproto/crypto` verification. It confirmed canonical-CBOR input, P-256 compact signature interoperability, the field allowlist, scalar validation, normalized DID member IDs, signer/DID key equality, and rejection paths described below. The spike harness is intentionally not retained in the product test suite; W1.4 recreates equivalent coverage against EmDash's actual signer/verifier code. + +## Rejections proved + +- Unknown fields, including `sig`, `$type`, and a representative arbitrary field, are rejected before encoding. +- Unsupported versions and malformed required/optional field types are rejected before encoding. +- Invalid DID, URI, CID, RFC3339/calendar datetime, empty/oversized UTF-8 value, and explicit false negation handling are covered. +- Private scalar decoding rejects malformed, padded, non-canonical, wrong-length, zero, and out-of-range inputs. +- DID method resolution accepts either relative or fully qualified label-key IDs alone, rejects mixed-form and same-form duplicates, and rejects missing IDs, wrong type/controller/codec, absent multikey data, signer-key mismatch, and fallback to `#atproto`. +- Passing the recorded protocol digest to `sign` produces a double-hashed signature that does not verify against canonical CBOR. +- The correct signature fails under a different P-256 key. +- The correct signature fails for a changed label payload. +- The correct signature fails against malformed non-DRISL payload bytes. +- A bit-flipped signature fails. +- Truncated, oversized, high-S equivalent, and ASN.1 DER signatures fail; only low-S compact 64-byte signatures are accepted. + +## Key lifecycle contract + +### Routine rotation + +1. Pause the issuance boundary before creating any label sequence that would need a signature. Assessment decisions may queue, but no sequence is allocated, stored, or broadcast while paused. +2. Generate a fresh P-256 key through the approved ceremony, store its base64url private scalar as a new Secrets Store version, derive its public multikey, and record a non-secret key-version identifier. Keep the old private key available for rollback until rotation is confirmed. +3. Update the issuer DID document's `#atproto_label` method to the new public multikey and verify the resolved document from outside the deployment. +4. Activate the matching secret version only after the DID update is observable. The startup/runtime guard must refuse issuance if the derived key does not match the resolved/configured public key. +5. Resume issuance. Persist the key-version identifier with every signature. Monitor signing failures, DID mismatch, and downstream verification failures. +6. On query, lazily re-sign labels carrying an old key version with the current key without changing `cts`, then persist the replacement signature/key version. Event-stream backfill may retain old signatures as allowed by the ATProto label specification. +7. Subscribers retry one failed verification after re-resolving the DID. They recover missed labels from their last durably accepted cursor; rotation alone does not reset sequence history. + +### Suspected or confirmed compromise + +1. Pause issuance immediately at the same pre-sequence boundary. Do not use the compromised key for rollback or historical re-signing. +2. Replace or remove the compromised `#atproto_label` key in the DID document, publish an incident notice through the policy endpoint, establish the last trusted sequence/time where possible, and deploy a fresh key only after its DID publication is observable. +3. Preserve compromised signatures and their key-version mapping for forensics, but never treat them as currently valid merely because they once verified. +4. Reissue the current effective label set with new signed events. Do not rewrite compromised history into an apparently continuous trusted history. +5. Declare a safe replay cursor when one can be established. Otherwise require subscribers to clear derived state for this labeler and replay the retained stream from cursor `0` after trustworthy history/current-state recovery is available. +6. Subscribers re-resolve the DID on signature failure, reject labels that still fail, and alert rather than falling back to `#atproto` or any other DID key. + +W3.7 owns implementing this state machine and key-version persistence. W11.4 owns the operator runbook, ceremony/custody details, external DID verification, monitoring, and subscriber communications. diff --git a/.opencode/plans/plugin-registry-labelling-service/gate-0/vocabulary-audit.md b/.opencode/plans/plugin-registry-labelling-service/gate-0/vocabulary-audit.md new file mode 100644 index 0000000000..4905e63c80 --- /dev/null +++ b/.opencode/plans/plugin-registry-labelling-service/gate-0/vocabulary-audit.md @@ -0,0 +1,508 @@ +# Gate 0 Vocabulary Cutover Audit + +Status: W0.3 audit complete. Canonical `security-yanked`, no-new-legacy emission, collision-safe ingest identity, and conditional Branch A were ratified on 2026-07-10. The production preflight was confirmed by the maintainer on 2026-07-11: zero legacy rows exist, so Branch A applies and no compatibility path is added. Re-run the preflight immediately before canonical issuance is enabled. + +Scope: tracked source, migrations, tests, documentation, generated contracts, and deployment configuration as of branch `feat/labeller-02-vocabulary-audit`. This audit is read-only. It does not establish production database state. + +## Recommendation + +Ratify `security-yanked` as the only value emitted by new code. Use the no-compatibility branch if the production preflight returns no legacy rows. Repository evidence makes that the likely result, but only the read-only production query below can establish it. + +If legacy rows exist, do not rewrite signed `labels` history. First classify every row from its subject URI. Deploy a bounded consumer-only alias, reissue only active release-record rows as canonical `security-yanked`, and signed-negate their legacy values only after the compatibility floor is enforced. Quarantine package-profile, publisher-DID, malformed, and unknown-scope rows until an operator explicitly maps each action; never emit release-only `security-yanked` on those subjects. + +The spelling cutover alone is insufficient. Current consumers do not implement the complete blocking policy, active-label semantics, accepted-source policy, subject cascading, or CID applicability. The history primary key is also not collision-safe for distinct valid labels sharing `(src, uri, val, cts)`. W1.1 should centralize vocabulary, while a forward schema migration plus W1.5, W4, and W5 must land before label ingest or issuance is enabled. + +## Occurrence Inventory + +### Legacy `security:yanked` + +| Location | Reference | Current role | Required future action | +| ------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `apps/aggregator/migrations/0001_init.sql:187-230` | `labels`, `label_state`, and example at line 191 | Schema accepts arbitrary `val`; the comment names the legacy value. | Change the fresh-install comment. Do not edit signed persisted history. Add a mandatory forward collision-safe history migration before ingest. | +| `apps/aggregator/src/routes/xrpc/searchPackages.ts:148-162` | `ENFORCEMENT_FILTER_SQL` | Raw SQL blocks package-profile URIs with `!takedown` or the legacy value. | Replace with shared policy-driven subject evaluation. A temporary `IN (..., 'security:yanked')` alias is permitted only on the persisted-row branch. | +| `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/searchPackages.json:8` | Query description | Generated-contract source documents the legacy spelling. | Use `security-yanked` and describe policy rather than a partial hard-coded pair. Regenerate types. | +| `packages/registry-client/src/discovery/index.ts:1-13` | Module documentation | Says the aggregator enforces `!takedown` and the legacy value. | Correct spelling and document that the client evaluates typed moderation state rather than trusting endpoint filtering alone. | +| `packages/core/src/api/handlers/registry.ts:1-39` | Install-flow documentation | Describes only the legacy yank as the label gate. | Describe and call the shared release-moderation evaluator. | +| `packages/core/src/api/handlers/registry.ts:791-808` | `handleRegistryInstall` | Raw comparisons over package and release label arrays. | Replace both comparisons with one shared evaluation immediately before artifact download. | +| `packages/core/src/api/handlers/registry.ts:1462-1471` | `handleRegistryUpdate` | Raw comparison over release labels only. | Replace with the same shared evaluation used by install. | +| `packages/admin/src/components/RegistryPluginDetail.tsx:115-136` | Release filtering | Claims aggregator filtering plus local defense in depth. | Consume typed eligibility; do not locally classify strings. | +| `packages/admin/src/components/RegistryPluginDetail.tsx:833-850` | `YANKED_LABEL_VALUE`, `isYanked` | Defines and compares the legacy value and intentionally ignores `neg`. | Delete after the shared evaluator is wired into the UI. | + +No current runtime, migration, test, generated type, or public documentation occurrence of canonical `security-yanked` exists outside the labelling-service spec and implementation plan. The canonical value is therefore not currently recognized by shipped consumers. + +### `!takedown` and redaction + +| Location | Reference | Current behavior or gap | +| ------------------------------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `apps/aggregator/migrations/0001_init.sql:191` | Schema comment | `val` can store `!takedown`; no issuer or ingest path exists. | +| `apps/aggregator/src/routes/xrpc/searchPackages.ts:153-162` | `ENFORCEMENT_FILTER_SQL` | Filters only a package-profile URI, only when `trusted = 1`, and regardless of the request's accepted labelers or `redact` flag. | +| `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/searchPackages.json:8` | Query description | Describes takedown filtering but does not encode accepted-source or redaction semantics. | +| `packages/registry-client/src/discovery/index.ts:101-114` | `acceptLabelers` docs | Forwards an opaque header and documents `;redact`; it neither parses the header nor interprets the response policy. | +| `apps/aggregator/src/routes/xrpc/router.ts:43-64` | CORS contract | Allows `atproto-accept-labelers`, but incorrectly exposes that request header instead of `atproto-content-labelers`. The handler never parses either accepted DIDs or `redact`. | +| Core install/update handlers | No comparison | `!takedown` is not independently blocked. They rely on aggregator filtering that only exists on package search, not direct package/release reads. | +| Admin browse/detail | No rendering or comparison | No redacted/takedown state is shown or blocked locally. | + +### Verification and `verified` + +There are two unrelated mechanisms that current code partially conflates: + +1. `apps/aggregator/migrations/0001_init.sql:133-162` defines `publisher_verifications`, a table of signed `com.emdashcms.experimental.publisher.verification` records bound to subject handle and display name. +2. `apps/aggregator/src/records-consumer.ts:793-841` ingests those records. No read endpoint converts them into an ATProto label or hydrates a `verified` value. +3. `packages/admin/src/components/RegistryBrowse.tsx:168-188` and `RegistryPluginDetail.tsx:236-243` treat any raw label with `val === "verified"` as publisher verification. +4. `packages/admin/tests/components/RegistryPluginDetail.test.tsx:248-275` is the only focused verified-label fixture. It proves tooltip rendering, not source acceptance, negation, expiration, CID binding, or the verification-record validity rules. +5. `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/defs.json:51-59,113-121` permits generic standard ATProto labels but does not define a `verified` vocabulary value. +6. `docs/src/content/docs/plugins/registry.mdx:20-29,75-85` and `docs/src/content/docs/reference/configuration.mdx:400-405` describe verification labels even though the current aggregator does not hydrate them. + +`verified` must not become an eligibility shortcut. Coordinator confirmation is needed on whether publisher verification remains a separately evaluated registry record, becomes a typed non-moderation label under a distinct policy, or is removed from the admin label path. In every case, the current raw `verified` checks must be replaced. + +### Generated contracts and client types + +- `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/defs.json:6-123` is the source contract. Package and release views have optional arrays of standard `com.atproto.label.defs#label`; `labels` is not a required field. +- `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/listReleases.json:4-63` promises hard-enforcement filtering and hydrated labels that the current handler does not implement. +- `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/getLatestRelease.json:4-36` promises the highest non-yanked release and `NotFound` when no eligible release exists; current code excludes only publisher tombstones. +- `packages/registry-lexicons/src/generated/types/com/emdashcms/experimental/aggregator/defs.ts:5-145` preserves the complete standard label shape through generated validation, including fields such as `src`, `uri`, `cid`, `val`, `neg`, `cts`, and `exp`. It adds no vocabulary or applicability policy. +- Generated `listReleases.ts` and `getLatestRelease.ts` preserve method shapes but not source descriptions, so source-Lexicon behavior claims need direct contract tests in addition to generated-file freshness checks. +- `packages/registry-client/src/discovery/index.ts:36-55` preserves those arrays in `ValidatedPackageView` and `ValidatedReleaseView`, but provides no active-label or eligibility helper. +- `packages/registry-client/tests/discovery.test.ts:33-326` covers transport, header forwarding, envelope validation, and embedded-record validation. It has no moderation-state fixture. +- `packages/registry-client/README.md:27-32` documents only opaque server-side hard-takedown filtering. + +## Current Behavior and Gaps + +### Database state model + +`apps/aggregator/migrations/0001_init.sql:180-241` creates the following dormant label infrastructure: + +- `labels` is intended as append-only history and retains `cid`, `neg`, `exp`, signature, version, trust, and receive time. Its primary key `(src, uri, val, cts)` is not a safe event identity: two distinct valid signed labels can share those four fields while differing in CID, negation, expiry, or signature. The second insert would collide and history would be lost. +- `label_state` projects one latest row per `(src, uri, val)` and retains that row's `cid`, `neg`, `exp`, and trust bit. +- The schema comment defines active as `neg = 0` and unexpired. The only runtime query using this rule is package search. +- `trusted` is copied onto each row. It can record ingestion provenance or the deployment-default trust snapshot at receipt time, but it must never decide request applicability. Request evaluation uses the current parsed accepted-DID policy, so a trust configuration change takes effect without rewriting history. +- No tracked source contains `INSERT`, `UPDATE`, or `UPSERT` statements for `labels`, `label_state`, or `labelers`. No seed or deployment data populates them. + +Before any label ingest, add a forward migration to give each history event collision-safe identity. Preferred shape: a stable digest of the exact verified signed-label bytes as the primary/idempotency key, plus unique `(src, source_sequence, frame_index)` coordinates for subscription ingest. `subscribeLabels` sequence identifies a frame, and one frame may carry multiple labels, so `(src, source_sequence)` alone is not unique. A different event-coordinate primary key is acceptable only if every ingest and replay path proves the same per-label identity. Do not use `cts` as event identity. Projection ordering uses verified `(source_sequence, frame_index)` after `cts`; a replay lacking those coordinates must retain both same-`cts` events and quarantine an ambiguous state transition rather than silently choose by insertion order or digest. + +The state key chooses one current label for `(src, uri, val)` while retaining its CID. Consumers still have to reject a CID-bound state row when its CID does not equal the current subject CID. Current search does not perform that check. + +### Active, negated, expired, and CID-bound labels + +Current behavior is inconsistent: + +- Package search checks persisted `trusted = 1`, `neg = 0`, and a textual `exp > strftime(...)` comparison. +- Package search does not check `cid`, accepted source DID, or `redact`. +- RFC 3339 strings are not safely ordered as text, and SQLite date functions are permissive rather than RFC3339 validators. SQL inventories raw `exp` only. Application code must first validate strict RFC3339 syntax and then parse/compare the instant against one evaluation time. Equality is expired/inactive. New malformed-expiry labels are rejected before write. An already-persisted positive legacy row with malformed or ambiguous expiry is quarantined and held suppressive until disposition. +- Package search constructs only the package profile URI. Because future `security-yanked` is release-only, this filter cannot suppress a yanked release. A release label's URI does not equal the package profile URI. +- Core install/update raw-array checks use only `val`. A negated, expired, unaccepted, wrong-CID, or untrusted label would still block if hydrated. +- Admin `isYanked` deliberately mirrors that incorrect raw-value behavior. +- Admin `verified` checks likewise ignore source acceptance, negation, expiration, CID applicability, and the separate publisher-verification validity contract. +- Aggregator views currently return `labels: []` unconditionally, so the core/admin checks are normally inert. + +The future shared evaluator must define an applicable active label as one whose source DID is in the current request's accepted set, whose latest state is not negated, whose application-validated strict RFC3339 expiry is absent or strictly later than the evaluation instant, and whose CID is absent for a URI-wide action or exactly matches the current subject CID. Persisted `trusted` may explain ingestion/default provenance but cannot add or remove a source from that request. Unknown values remain visible as data but have no official effect unless policy assigns one. + +### Aggregator endpoint trace + +| Endpoint | Current implementation | Moderation consequence | +| ------------------ | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `searchPackages` | `searchPackages.ts:36-162` applies the partial package-URI SQL filter, then `packageView` emits `labels: []`. | Only dormant trusted package-level legacy yank/takedown state is filtered. Publisher scope, release scope, accepted sources, and redaction are absent. | +| `getPackage` | `getPackage.ts:17-38` reads `packages` and calls `packageView`. | Direct reads ignore all labels and takedowns. | +| `resolvePackage` | `resolvePackage.ts:50-84` resolves a handle, reads `packages`, and calls `packageView`. | Same as `getPackage`. | +| `listReleases` | `listReleases.ts:22-104` selects every non-tombstoned release and calls `releaseView`. | No moderation filter, tombstone, hydration, or explanation. | +| `getLatestRelease` | `getLatestRelease.ts:25-66` trusts `packages.latest_version`, then falls back to highest non-tombstoned semver. | "Eligible" currently means only non-tombstoned. Positive assessment and every blocking label are ignored. | +| View mapping | `views.ts:120-163` | Package and release labels are always empty arrays. | +| Request boundary | `router.ts:43-117` | The accepted-labeler header is allowed but not parsed; no `atproto-content-labelers` response is produced. | + +### Registry client trace + +`DiscoveryClient` forwards `acceptLabelers` on every request and validates response envelopes. It does not: + +- Parse accepted labelers or the aggregator's actual content-labeler response. +- Filter active state. +- Evaluate publisher, package, and release subjects together. +- Require `assessment-passed`. +- Classify pending, error, blocking, warning, manual override, or redacted states. +- Verify hydrated label signatures. + +Its `listReleases` and `getLatestRelease` comments promise yank-aware behavior that the aggregator does not implement. CLI `search` and `info` construct the client without accepted-labeler configuration (`packages/plugin-cli/src/commands/search.ts:48-58`, `info.ts:45-56`); `info` prints raw labels (`info.ts:98-102`) without explaining effect or active state. + +### Core install, update, and update-check trace + +- `handleRegistryInstall` resolves package and release views, then blocks only raw legacy yanks on package or release (`registry.ts:684-808`). It does not block `!takedown`, canonical yank, any descriptive security label, missing/pending/error assessment, or package/publisher cascades. +- Explicit-version installs page through `listReleases`, so they bypass any future latest-release filtering unless the handler performs the same shared evaluation. +- `handleRegistryUpdate` resolves a release and blocks only a raw legacy release yank (`registry.ts:1389-1471`). It does not load package/publisher state and has less defense than install. +- `handleRegistryUpdateCheck` trusts `getLatestRelease` and exposes an update without evaluating its moderation state (`registry.ts:1683-1744`). Failures are skipped, so an ineligible installed release also has no alert path. +- `packages/core/src/astro/routes/api/admin/plugins/registry/artifact.ts:293-353` independently resolves latest or explicit release media through `DiscoveryClient` and returns the declared URL without evaluating moderation. A stale or hand-built proxy URL could therefore retrieve media for a blocked/redacted explicit release unless this route shares the same policy or the aggregator returns a redacted tombstone. +- Focused core tests do not cover moderation. `packages/core/tests/unit/api/registry-handlers.test.ts:1-12` explicitly lacks update happy-path coverage; `registry-env-gate-handler.test.ts:47-67` supplies only `labels: []`. + +The shared evaluation must run after exact release selection and again immediately before artifact download for both install and update. Update-check and installed-release alert paths must consume the same result type rather than infer eligibility from endpoint selection. + +### Admin browse and detail trace + +- Browse trusts aggregator search filtering and shows only a raw `verified` shield (`RegistryBrowse.tsx:151-204`). Its React Query key at lines 51-62 omits `config.acceptLabelers`, so changing accepted policy can reuse stale results. +- Detail's package query key at lines 100-107 also omits `config.acceptLabelers`. The release-list key includes it at lines 126-130. +- Detail removes only raw legacy yanks and hides every release if all returned releases match that string (`RegistryPluginDetail.tsx:115-178,833-850`). It does not surface pending/error/block reasons, warnings, source, CID, expiry, overrides, or reconsideration. +- There is no `RegistryBrowse` component test and no yanked/moderation test in `RegistryPluginDetail.test.tsx`. +- Server errors document only `RELEASE_YANKED` (`packages/admin/src/lib/api/registry.ts:674-692`), not the future eligibility states. + +## Complete Official Blocking Policy + +W0.2 owns executable fixtures and is authoritative if it differs from this audit. The spec currently requires the following evaluation shape: + +1. Require an applicable active `assessment-passed` from an accepted labeler. Its absence is `pending`/unknown and blocks. +2. `assessment-pending` blocks temporarily. `assessment-error` blocks and explains an operational failure. +3. Block active automated security labels: `malware`, `data-exfiltration`, `credential-harvesting`, `supply-chain-compromise`, `critical-vulnerability`, `artifact-integrity-failure`, `invalid-bundle`, `undeclared-access`, and `impersonation`. +4. Block active manual `security-yanked` on a release. +5. Redact and block `!takedown` according to accepted-labeler `redact` policy. `!suspend` must follow the standard redaction contract where supported. +6. Block `publisher-compromised` across all packages and releases from the publisher. +7. Apply package and publisher labels at evaluation time without copying them onto release rows. `package-disputed` warns and prevents recommendation; direct-install blocking remains a ratification point. +8. Warning values (`suspicious-code`, `obfuscated-code`, `privacy-risk`, `misleading-metadata`, `low-quality`, `broken-release`) do not block alone, but official admin installation requires explicit warning consent. +9. An exact-CID action-backed `assessment-passed` plus `assessment-overridden` may override selected automated release blocks according to W0.2 fixtures. It never overrides `!takedown`, `security-yanked`, `publisher-compromised`, or broader manual package/publisher blocks. +10. A wrong-CID positive label never makes the current release eligible. A URI-wide manual action remains applicable across record edits until negated or expired. + +## Production Persistence Evidence + +Evidence that a database may exist: + +- `docs/src/content/docs/plugins/registry.mdx:12-18,37-49` says the reference aggregator runs at `registry.emdashcms.com`. +- `apps/aggregator/wrangler.jsonc:1-21` names Worker `emdash-aggregator` and D1 database `emdash-aggregator`; the database is auto-provisioned on first deploy. +- `apps/aggregator/package.json:7-16` provides production deploy and remote migration commands. +- Git history contains the aggregator scaffold, ingest, read API, install integration, and Workers Builds fixes, consistent with deployment work. + +Evidence against application-created label rows: + +- The only label tables are the initial migration's dormant infrastructure. +- `views.ts:15-18` and `searchPackages.ts:12-15,148-152` explicitly say label hydration/ingest has not landed and the table is empty in the current slice. +- Exhaustive tracked searches found no writer for `labels`, `label_state`, or `labelers` and no seed data containing any moderation value. +- There is no configured labeler subscription, label queue, or labeler DID in current deployment configuration. + +Evidence limitations: + +- `wrangler.jsonc` intentionally omits `database_id` and production routes are configured at deploy time, so this checkout cannot identify or inspect the deployed D1 instance. +- Operators could have inserted rows manually or through untracked deployment tooling. +- Documentation proving a deployed aggregator does not prove which migrations ran or what rows exist. + +Conclusion: persisted production label rows are unlikely, and application-produced legacy rows are especially unlikely, but production state is unknown until an authorized operator runs the read-only preflight. + +## Safe Production Preflight + +Run the following read-only SQL against each production/staging aggregator D1 database before W1.1 deploys. Do not run it against the labeler database. It returns counts first, followed by the matching rows without signature bytes. + +```sql +SELECT + 'labels_history' AS storage, + COUNT(*) AS total_rows, + COALESCE(SUM(CASE WHEN val = 'security:yanked' THEN 1 ELSE 0 END), 0) AS legacy_rows, + COALESCE(SUM(CASE WHEN val = 'security-yanked' THEN 1 ELSE 0 END), 0) AS canonical_rows +FROM labels +UNION ALL +SELECT + 'label_state' AS storage, + COUNT(*) AS total_rows, + COALESCE(SUM(CASE WHEN val = 'security:yanked' THEN 1 ELSE 0 END), 0) AS legacy_rows, + COALESCE(SUM(CASE WHEN val = 'security-yanked' THEN 1 ELSE 0 END), 0) AS canonical_rows +FROM label_state; + +WITH matching AS ( + SELECT + 'labels_history' AS storage, + src, + uri, + cid, + val, + neg, + cts, + exp, + trusted + FROM labels + WHERE val IN ('security:yanked', 'security-yanked') + UNION ALL + SELECT + 'label_state' AS storage, + src, + uri, + cid, + val, + neg, + cts, + exp, + trusted + FROM label_state + WHERE val IN ('security:yanked', 'security-yanked') +), uri_parts AS ( + SELECT + *, + CASE WHEN substr(uri, 1, 5) = 'at://' THEN 1 ELSE 0 END AS is_at_uri, + CASE + WHEN substr(uri, 1, 5) = 'at://' THEN instr(substr(uri, 6), '/') + ELSE 0 + END AS authority_slash + FROM matching +), authority_parts AS ( + SELECT + *, + CASE + WHEN is_at_uri = 1 AND authority_slash > 1 + THEN substr(uri, 6, authority_slash - 1) + WHEN is_at_uri = 0 + THEN uri + ELSE NULL + END AS candidate_did, + CASE + WHEN is_at_uri = 1 AND authority_slash > 1 + THEN substr(uri, 6 + authority_slash) + ELSE NULL + END AS path_after_authority + FROM uri_parts +), parsed_parts AS ( + SELECT + *, + substr(candidate_did, 5) AS did_body, + instr(substr(candidate_did, 5), ':') AS method_separator, + instr(path_after_authority, '/') AS collection_separator + FROM authority_parts +), fields AS ( + SELECT + *, + substr(did_body, 1, method_separator - 1) AS did_method, + substr(did_body, method_separator + 1) AS did_identifier, + substr(path_after_authority, 1, collection_separator - 1) AS subject_collection, + substr(path_after_authority, collection_separator + 1) AS subject_rkey + FROM parsed_parts +), screened AS ( + SELECT + *, + CASE + WHEN substr(candidate_did, 1, 4) = 'did:' + AND method_separator > 1 + AND did_method NOT GLOB '*[^a-z0-9]*' + AND length(did_identifier) > 0 + AND did_identifier NOT GLOB '*[^A-Za-z0-9._:-]*' + THEN 1 + ELSE 0 + END AS did_shape_candidate + FROM fields +), classified AS ( + SELECT + *, + CASE + WHEN is_at_uri = 0 AND did_shape_candidate = 1 THEN 'publisher_candidate' + WHEN is_at_uri = 1 + AND did_shape_candidate = 1 + AND collection_separator > 1 + AND length(subject_rkey) > 0 + AND subject_rkey NOT GLOB '*[/?#]*' + THEN CASE subject_collection + WHEN 'com.emdashcms.experimental.package.release' THEN 'release_candidate' + WHEN 'com.emdashcms.experimental.package.profile' THEN 'package_candidate' + ELSE 'unknown' + END + ELSE 'unknown' + END AS subject_scope + FROM screened +) +SELECT + storage, + subject_scope, + CASE WHEN is_at_uri = 1 THEN subject_collection ELSE NULL END AS subject_collection, + src, + uri, + cid, + val, + neg, + cts, + exp, + trusted +FROM classified +ORDER BY storage, subject_scope, src, uri, val, cts; +``` + +The SQL is only a conservative structural screen. It cannot validate full DID, AT URI, or ATProto record-key grammar. `release_candidate`, `package_candidate`, and `publisher_candidate` are not valid subjects yet. Before automatic canonical reissue, application code must use an ATProto parser/validator to require a fully valid DID authority, a plain publisher DID with no suffix, an AT URI with exactly authority/collection/rkey, the exact collection, and a record key satisfying the complete ATProto grammar, byte-length limit, and reserved `.`/`..` rejection. Any SQL candidate that application parsing does not strictly validate becomes `unknown` and remains quarantined/suppressive. + +The SQL intentionally rejects some potentially valid percent-encoded DID forms as `unknown` rather than risk a false positive. Conversely, strings such as a trailing-colon DID or invalid rkey may pass the SQL shape screen; candidate status never authorizes migration. + +Focused classifier fixtures: + +| Subject | SQL screen | Required application result | +| ------------------------------------------------------------------------- | --------------------- | -------------------------------------- | +| `did:plc:abc` | `publisher_candidate` | `publisher` | +| `did:plc:abc/path` or `did:plc:abc#fragment` | `unknown` | `unknown` | +| `did::abc` or `did:PLC:abc` | `unknown` | `unknown` | +| `did:web:example.com:` | `publisher_candidate` | `unknown` (invalid trailing-colon DID) | +| `at://did:plc:abc/com.emdashcms.experimental.package.release/pkg:1.0.0` | `release_candidate` | `release` | +| `at://did:plc:abc/com.emdashcms.experimental.package.profile/pkg` | `package_candidate` | `package` | +| Missing authority, empty collection/rkey, extra path, or wrong collection | `unknown` | `unknown` | +| Release rkey containing a space or `%` | `release_candidate` | `unknown` | +| Release rkey longer than the ATProto byte limit | `release_candidate` | `unknown` | +| Release rkey `.` or `..` | `release_candidate` | `unknown` (reserved) | + +Interpretation: + +- If both `legacy_rows` values are zero, choose Branch A. +- If either is nonzero, choose Branch B even when every projected legacy row is negated or expired. History replay and old projections still need an explicit compatibility decision. +- Only an application-validated `release_candidate` is eligible for automatic canonical reissue. Package/publisher candidates, failed application validation, and SQL `unknown` rows require explicit operator mapping and remain quarantined. +- SQL does not determine activity from `exp`. Application code treats a latest positive row as non-suppressive only when `exp` is strict RFC3339 and its parsed instant is at or before the single evaluation instant. Missing, malformed, or ambiguous expiry remains an active hold; a latest negation is inactive regardless of expiry. +- Any canonical row before the official signer exists is unexpected and requires provenance review before proceeding. + +## Migration Branches + +### Branch A: no persisted legacy rows + +Recommended branch, contingent on the preflight: + +1. Add no vocabulary compatibility alias and no legacy-row data migration. +2. Change all tracked examples and raw references to canonical `security-yanked` while centralizing the complete vocabulary. +3. Add the collision-safe forward history migration even on an empty database; it is a prerequisite for ingest, not legacy-data compatibility. +4. Add the shared evaluator and endpoint/client tests before issuing any production label. +5. Deploy the compatibility-floor component versions before enabling canonical label issuance. +6. Re-run the preflight immediately before issuance. A newly appeared legacy row switches the rollout to Branch B. + +This follows the spec's explicit instruction not to carry compatibility without persisted data. + +### Branch B: persisted legacy rows exist + +1. Export/backup the affected D1 database and retain the preflight output as rollout evidence. +2. Apply the collision-safe forward history migration before enabling any subscriber, query backfill, or replay writer. +3. Inventory SQL candidates, then application-parse each raw subject and expiry. Record strict subject validity/scope, issuer, CID, current projected state, strict RFC3339 result/instant, and signature provenance. +4. Mark only application-validated `at://.../com.emdashcms.experimental.package.release/` rows as release-scoped. Quarantine package-profile URIs, publisher DIDs, malformed/ambiguous URIs or rkeys, SQL-only candidates, unknown collections, and unexplained/manual rows. Do not reinterpret them as release yanks and do not negate them until an operator approves a policy-valid replacement or explicit retirement. +5. Before changing vocabulary state, materialize fail-closed compatibility holds for every active quarantined row. A package hold suppresses the package and all of its releases; a publisher hold suppresses every indexed package/release for that DID. For malformed/unknown rows, preserve every subject suppressed by the pre-cutover query and suppress any indexed subject resolved by exact stored URI; an unresolvable row blocks exact-URI access and alias removal but must not create an unrelated ecosystem-wide block. +6. Deploy consumers that normalize only release-scoped `security:yanked` to the canonical release-yank policy class on read. Non-release/malformed rows use compatibility holds, not canonical normalization. The alias is consumer-only; no producer may emit the legacy value. +7. Enforce the compatibility floor below across the reference aggregator and supported official clients. Publishing packages does not upgrade deployed EmDash sites, CLIs, third-party aggregators, or cached admin bundles. +8. For each active, verified release-scoped legacy action, issue canonical `security-yanked` with the same release URI and intended CID scope. Keep the legacy positive active during the compatibility interval so old consumers retain their existing signal. +9. Only after the compatibility floor is enforced, issue a signed negation of that release-scoped legacy value. Canonical blocking must already be effective. Never emit canonical `security-yanked` on a package URI or publisher DID. +10. Let normal verified ingestion update `label_state`; do not hand-edit its projection unless a separately reviewed repair runbook is required. +11. Replay into a clean collision-safe history/projection and confirm identical blocked release subjects under the dual-read evaluator, including same-`cts` events. +12. Keep the dual-read rollout as the minimum rollback version. Rolling back below it after any legacy negation can fail open. +13. Remove the alias and compatibility holds only after all configured aggregators report zero active legacy state, every quarantined scope has an explicit replacement/retirement, full replay reproduces canonical state, and monitoring confirms no producer emitted the legacy value for the agreed retention window. + +Quarantined-state behavior is explicit: + +- Positive rows with no expiry or a strictly parsed future RFC3339 instant are active holds. A positive row with malformed or ambiguous expiry is also an active hold because expiry cannot be proved. +- Latest negated rows and positive rows whose strict RFC3339 instant is at or before the single evaluation instant remain quarantined for audit/mapping but do not suppress subjects. Exact equality is non-suppressive. +- A later valid positive can reactivate a hold according to collision-safe sequence/frame ordering. +- Manual retirement removes a hold only through an audited disposition; it does not rewrite signed history or emit release-only canonical vocabulary on a broader/malformed subject. + +## Component Compatibility Floor + +The current versions below are evidence from tracked manifests, not claims about every deployed installation. The implementation PR must replace each `TBD` with the first released version or deployed commit satisfying that class. + +| Component class | Current/target version marker | Existing behavior | Required compatibility-floor behavior | Safe after legacy negation? | +| ------------------------------------------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `A0` current/reference-old aggregator | Private `@emdash-cms/aggregator@0.0.0`; current branch commit | Legacy package-search string filter only; direct package/release reads, latest selection, explicit release lists, and labels are not policy-complete. | Upgrade to `A1` before canonical issuance is relied upon. | No. | +| `A1` reference aggregator | First deployed compatibility-floor commit: `TBD` | New class. | Verify before write; collision-safe history; current accepted-DID evaluation; canonical+legacy dual-read; application-validated subject/expiry; publisher/package/release expansion; block/redact search, package, resolve, list, latest, explicit-version, and media/artifact resolution consistently. For clients without typed tombstone support, omit blocked releases rather than returning an installable-looking view. | Yes for acquisition paths. | +| `S0` already-installed EmDash/core/admin | `emdash@0.28.1`, `@emdash-cms/admin@0.28.1`, client `0.3.2` | Embeds its registry-client and raw legacy checks at build/install time. Updating npm packages or the reference Worker does not upgrade it. No complete installed-release alert path. | Either upgrade the installation to `S1` or require it to use an `A1` reference endpoint that fail-closes every latest/explicit/artifact path. Publish the minimum supported EmDash version. | Acquisition can be safe through `A1`; installed-plugin alert/disable coverage remains absent. | +| `S1` policy-aware EmDash/core/admin | First compatible EmDash/admin/client releases: `TBD` | New class. | Shared evaluator immediately before install/update artifact fetch, explicit-version parity, update-check/installed-release alerts, policy-keyed admin queries, and blocked/redacted artifact proxy. | Yes. | +| `C0` installed CLI/registry-client | `@emdash-cms/plugin-cli@0.6.0`, registry client `0.3.2` | No accepted-labeler option in current search/info commands; displays unclassified raw labels; no install/update command. | Upgrade to `C1` for accurate moderation display. | No install bypass exists, but output is not trustworthy as policy explanation. | +| `C1` policy-aware CLI/client | First compatible CLI/client releases: `TBD` | New class. | Current accepted-DID policy, typed moderation result, canonical+bounded-legacy handling, and content-labeler reporting. | Yes for its supported read-only paths. | +| Third-party/self-hosted old aggregator or client | Deployment-specific; no repository-controlled version floor | Version and rollout are outside the reference deployment's control. | Operator attestation of `A1`/`S1` equivalent behavior or explicit exclusion from the supported compatibility set. | Unknown until attested. | + +Safe rollout and rollback protections: + +1. Land and deploy `A1` before using canonical labels for official enforcement. Exercise latest, explicit-version, list pagination, direct package/resolve, and artifact/media paths against both vocabulary values. +2. Release `S1` and `C1`, publish their exact minimum versions, and inventory managed installations. Self-hosted installations cannot be assumed upgraded; their operators must upgrade or accept unsupported status explicitly. +3. Under Branch B, issue canonical release labels while retaining active legacy positives. This overlap is the only interval in which both old legacy-aware and new canonical-aware consumers can remain blocked. +4. Do not signed-negate any legacy positive until the operator has enforced the declared floor: all reference endpoints are `A1`, supported managed EmDash installations are `S1` or pinned to `A1` fail-closed endpoints, and third-party compatibility has an explicit disposition. +5. Before negation, rollback to `A0`/`S0` remains possible only because legacy positives remain active, though their pre-existing policy gaps still exist. After the first negation, pin rollback to `A1`/`S1`; restore by replaying immutable history/projection, never by downgrading consumers. +6. Old installed sites have no automatic installed-plugin alert/disable upgrade path. Do not claim fleet-wide post-install protection until those sites run `S1`; `A1` only protects future acquisition requests that reach it. + +Removal inventory (application parsing decides whether each positive row is suppressive): + +```sql +SELECT + src, + uri, + cid, + neg, + cts, + exp, + trusted +FROM label_state +WHERE val = 'security:yanked' + AND neg = 0 +ORDER BY src, uri, cts; +``` + +The removal procedure passes every raw `exp` through the same strict application validator. A valid RFC3339 instant exactly equal to or before the evaluation instant is inactive; a future instant or absent expiry is active. Invalid, date-only, timezone-less, space-separated, or otherwise ambiguous values remain active fail-closed holds and produce diagnostics. Historical legacy rows remain valid audit evidence after the compatibility alias is removed. + +## Future PR Patch List + +### W1.1: vocabulary cutover + +- Add the ratified vocabulary to the shared moderation package selected by W1.5. No endpoint-local constants. +- `apps/aggregator/migrations/0001_init.sql`: correct the fresh-install example; preserve already-applied databases. +- Add a forward aggregator migration replacing history identity `(src, uri, val, cts)` with collision-safe event identity. Prefer signed-label digest/idempotency plus unique verified `(src, source_sequence, frame_index)` subscription coordinates; define deterministic duplicate, multi-label-frame, and same-`cts` conflict handling before ingest starts. +- `apps/aggregator/src/routes/xrpc/searchPackages.ts`: remove the legacy raw pair in favor of shared policy inputs, or add the bounded alias only under Branch B. +- `packages/registry-lexicons/lexicons/com/emdashcms/experimental/aggregator/defs.json`, `searchPackages.json`, `listReleases.json`, and `getLatestRelease.json`: correct source contract claims and describe typed moderation/redaction behavior. Regenerate `defs.ts`, `searchPackages.ts`, `listReleases.ts`, `getLatestRelease.ts`, generated indexes, and ambient registrations. +- `packages/registry-lexicons/tests/types.test.ts` plus aggregator read-contract tests: assert the regenerated package/release label shapes and the observable list/latest eligibility contract rather than relying on descriptions that generated TypeScript omits. +- `packages/registry-client/src/discovery/index.ts`, `README.md`, and `tests/discovery.test.ts`: correct vocabulary and add typed moderation-state fixtures. +- `packages/core/src/api/handlers/registry.ts`: remove all three legacy comparisons and route install/update through the shared evaluator. +- `packages/admin/src/components/RegistryPluginDetail.tsx`: remove `YANKED_LABEL_VALUE` and `isYanked`. +- Add changesets for affected published packages when implementation lands. + +### W4: aggregator policy and hydration + +- `apps/aggregator/src/routes/xrpc/router.ts`: parse accepted labelers once, set/CORS-expose `atproto-content-labelers`, and preserve missing versus empty policy. +- Add label ingest/projection code that validates payload shape and source, verifies signatures before any history/projection write, and preserves exact `src`, `uri`, `cid`, `val`, `neg`, `cts`, `exp`, signed bytes/digest, verified source sequence, and frame index. +- Treat persisted `trusted` only as receipt-time ingestion/default-policy metadata. Every request resolves current accepted DIDs independently; changing deployment trust must immediately affect old rows without rewriting them. +- Add one application-layer ATProto subject validator used by preflight migration and runtime policy. It must fully validate DID grammar, AT URI segmentation/collection, and record-key grammar, byte length, and reserved `.`/`..`; SQL candidates confer no validity. +- Add one strict application-layer RFC3339 syntax/instant parser. SQL stores/inventories raw expiry only. New malformed expiry is rejected; persisted malformed/ambiguous positive legacy expiry stays suppressive; strict instant equality is inactive. +- Under Branch B, add temporary compatibility holds for active quarantined package, publisher, malformed, and unknown-scope legacy rows. Hydration/filtering must apply their fail-closed suppression without converting them to canonical release vocabulary; negated/validly expired rows retain audit state without an active hold. +- `apps/aggregator/src/routes/xrpc/views.ts`: replace unconditional empty arrays with batched publisher/package/release hydration. +- `searchPackages.ts`, `getPackage.ts`, `resolvePackage.ts`, `listReleases.ts`, and `getLatestRelease.ts`: use shared subject expansion and evaluation. Search/redaction, direct tombstones, and latest eligible release must agree. +- `apps/aggregator/test/read-api.test.ts`: clear label tables between tests and add source, active/negated/expired, URI-wide/CID-bound, subject-cascade, redaction, and latest-selection coverage. +- Add request-header/CORS contract tests for absent, empty, malformed, repeated, unavailable, and `redact` accepted-labeler forms. +- Add workerd/D1 ingest security tests for invalid signatures, wrong DID/key, malformed/extra-field payloads, routine key rotation, one DID/key refresh on first verification failure, persistent failure after refresh, and unverifiable replay. Every rejection must prove zero history rows, zero projection writes, and no durable cursor advancement. +- Add collision/replay tests proving every label in one multi-label subscription frame receives distinct `(src, source_sequence, frame_index)` coordinates, exact redelivery deduplicates by digest, distinct valid same-`cts` labels both survive history, verified frame coordinates resolve projection order, and coordinate-less ambiguous replay is quarantined rather than overwritten. + +### W5: client, core, CLI, and admin enforcement + +- Add `evaluateReleaseModeration` and `ReleaseModeration` to the shared package and re-export/wrap it from registry-client without duplicating policy. +- Registry client must preserve response `atproto-content-labelers`, all label applicability fields, and assessment references; add full-label verification access separately. +- `handleRegistryInstall`, `handleRegistryUpdate`, and `handleRegistryUpdateCheck` must share evaluation over publisher, package, and exact release subjects. Explicit-version paths must not bypass it. +- `packages/core/src/astro/routes/api/admin/plugins/registry/artifact.ts` must fail closed for redacted/blocked exact releases so media retrieval cannot bypass direct-read policy. +- Add focused core handler tests for every eligibility outcome and prove rejection occurs before artifact fetch. +- `packages/admin/src/components/RegistryBrowse.tsx`: add accepted-labeler policy to the query key and render typed eligibility, not raw `verified`. +- `RegistryPluginDetail.tsx`: add accepted-labeler policy to the package query key; consume typed eligibility, warnings, issuer, summary, override, and reconsideration state. +- `packages/admin/tests/components/RegistryPluginDetail.test.tsx`: replace raw-yank assumptions with policy fixtures; add negation/expiry/CID/warning/override tests. Add browse tests. +- `packages/admin/src/lib/api/registry.ts`: expose localized server error mapping for pending, error, blocked, redacted, and warning-consent outcomes. +- `packages/plugin-cli/src/commands/search.ts` and `info.ts`: use configured official accepted-labeler policy and display the shared moderation result rather than unclassified raw labels. +- `docs/src/content/docs/plugins/registry.mdx`, `registry-client.mdx`, and `reference/configuration.mdx`: document canonical vocabulary, positive-assessment requirement, warning behavior, and actual accepted/content-labeler semantics after implementation. +- Add installed-release refresh/alert coverage for canonical yank, takedown, and newly blocking labels. + +## Test Matrix + +| Layer | Required cases | +| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Vocabulary | Canonical value accepted; legacy rejected for issuance; legacy read alias present only in Branch B. | +| History identity | Multi-label frame indices; exact-redelivery digest idempotency; distinct valid same-`cts` labels retained; verified `(sequence,index)` tie-break; coordinate-less conflict quarantine; replay. | +| State projection | Newer positive, newer negation, out-of-order older positive, duplicate delivery, collision-safe replay from history. | +| Expiry | SQL raw preservation; strict app parsing of `Z`/offset and fractional/no-fraction forms; reject date-only, timezone-less, space-separated, malformed; exact-instant equality inactive. | +| Source policy | Missing header defaults, empty accepts none, multiple/repeated DIDs, unavailable DID, `redact` merge, correct `atproto-content-labelers`, and trust changes after ingest without row rewrites. | +| Ingest security | Invalid signature, wrong key/DID, malformed or extra-field payload, rotated key after one refresh, failure after refresh, unverifiable replay, zero writes/cursor advance on rejection. | +| CID applicability | Exact current CID, wrong CID, URI-wide action, package profile CID change, release CID change. | +| Subject scope | SQL candidate only; app DID/AT/rkey validation; trailing colon, suffix, spaces, percent, overlength, `.`/`..`, empty/extra path; quarantine holds; no broad canonical conversion. | +| Eligibility | Missing pass, passed, pending, error, every blocking security label, canonical yank, takedown, publisher compromise, warning-only, unknown label. | +| Overrides | Exact-CID automated-block override, newer assessment while override remains active, override retraction, inability to bypass manual release/package/publisher blocks. | +| Aggregator reads | Search, package, resolve, release list, latest release, direct blocked tombstone, all-blocked package, cache `private, no-store`. | +| Registry client | Envelope preservation, content-labeler response, shared evaluator parity, signed full-label retrieval, malformed label fail-closed behavior. | +| Core | Latest and explicit install, latest and explicit update, update check, no artifact request before eligibility, stable localized error codes. | +| Admin | Browse/detail query-key isolation, pending/error/block/warn UI, warning consent, issuer display, reconsideration, installed alert, Arabic RTL. | +| Compatibility | Zero-row no-alias path; active/negated/expired legacy rows; `A0/A1`, `S0/S1`, and `C0/C1`; latest/explicit/artifact paths; canonical overlap before negation; rollback floor; alias removal. | +| Verification | Verification-record valid/expired/tombstoned/identity-drift states; raw `verified` label cannot grant eligibility or bypass moderation. | + +## Rollback and Compatibility Removal + +- Before legacy negation, canonical and legacy positives overlap. Rollback may retain legacy-aware consumers, but current `A0`/`S0` policy gaps remain and must not be described as complete enforcement. +- After the first legacy negation, rollback must not go below recorded `A1`/`S1` compatibility-floor versions. Package deployment alone is not evidence that installed sites upgraded. +- Under Branch B, retain the legacy read alias until the removal condition is met across every production projection and replay target. +- If migration or verification validation fails, pause issuance, retain dual-read and active legacy positives, restore/rebuild projections from immutable collision-safe signed history, and do not mutate historical `val` or signatures. +- Remove compatibility when active legacy state is zero, every quarantined non-release row has an explicit disposition, clean replay yields the same effective blocks, reference and supported consumer versions meet the floor, no producer emits legacy values during the ratified retention window, and coordinator evidence records those checks. + +## Ratification Items + +Coordinator/Matt confirmation is required for: + +1. An authorized operator to identify every deployed aggregator environment and run the read-only preflight there. +2. Branch A versus Branch B based on that output. This audit recommends Branch A only conditionally. +3. Explicit replacement or retirement for each quarantined package, publisher, malformed, or unknown-scope legacy row. Automatic conversion is prohibited. +4. The W0.2 executable matrix, especially `package-disputed` direct-install behavior and exact manual-override interaction with pending/error states. +5. Whether publisher verification remains a separate signed record or becomes a defined non-moderation label. The current undocumented raw `verified` value should not be retained by accident. +6. The exact released versions/commits defining `A1`, `S1`, and `C1`, the supported old-installation floor, third-party attestation policy, and Branch B retention window before any legacy negation. + +Ratification recommendation: approve the canonical vocabulary and no-new-legacy-emission rule now; approve the no-compatibility implementation only after the production preflight is attached to the coordinating PR. diff --git a/.opencode/plans/plugin-registry-labelling-service/implementation-plan.md b/.opencode/plans/plugin-registry-labelling-service/implementation-plan.md new file mode 100644 index 0000000000..938291393d --- /dev/null +++ b/.opencode/plans/plugin-registry-labelling-service/implementation-plan.md @@ -0,0 +1,1681 @@ +# Plugin Registry Labelling Service Implementation Plan + +Companion: [Implementation spec](./spec.md) + +Status: execution plan pending Gate 0 contract and platform decisions + +This plan turns the plugin registry labelling-service spec into independently deliverable workstreams. It defines dependencies, integration gates, file ownership, merge boundaries, and completion criteria. It intentionally contains no time estimates. + +## Outcomes + +The implementation is complete when: + +1. Every valid new registry release is independently observed and receives a CID-bound assessment state. +2. The EmDash labeler issues standard DRISL-signed labels through public `queryLabels` and replayable `subscribeLabels` endpoints. +3. A fresh aggregator verifies, ingests, hydrates, and enforces labels from configured labelers. +4. Official clients require an active positive assessment and consistently block pending, errored, yanked, taken-down, or high-risk releases. +5. Deterministic validation, capability analysis, code/metadata AI, image analysis, and publisher-history context run under one versioned policy. +6. AI can hard-block only critical security or impersonation findings; quality findings warn or downrank. +7. Reviewers can inspect evidence, rerun assessments, and override false positives; admins alone control emergency `!takedown` and publisher-wide compromise actions. +8. Public assessment summaries explain decisions without exposing private evidence or exploit details. +9. Publishers receive actionable notices and have a documented manual reconsideration route. +10. Signing-key rotation, full label replay, assessment outages, artifact failures, backups, and disaster recovery are exercised. +11. The hosted service and a fresh third-party aggregator pass the same public protocol conformance suite. + +## Execution Rules + +- The [spec](./spec.md) is the source of truth for policy and behavior. This plan may sequence work but must not weaken its invariants. +- Label vocabulary, subject/CID rules, and official-client effects are frozen before production labels are issued. +- Standard label serialization/signature logic and consumer policy logic have one shared implementation each. Do not duplicate them across the labeler, aggregator, registry client, core handlers, and admin. +- Every asynchronous producer/consumer is idempotent before it is connected to a real Queue, Workflow, Durable Object, or Jetstream source. +- A Jetstream event is discovery only. No public label, including `assessment-pending`, is signed before source-record verification. +- Model output is normalized evidence. It never calls signing primitives or selects arbitrary label values. +- Automated labels target an exact release URI + CID. Package and publisher labels remain manual in v1. +- No positive assessment requirement is enabled in production until first-party releases have been assessed and rollback is tested. +- Incomplete integration remains unreachable by default. Feature flags may expose staging paths, but production clients must not depend on half-built label state. +- Tests land with each workstream. Protocol, security, and failure tests are not deferred to the final gate. +- `.opencode/plans` is ephemeral coordination material. Retained runtime code and tests must never import it. A feasibility spike does not become a product test merely because it was useful; vectors and fixtures move into normal package/test directories only when they test an implemented EmDash component. +- The operator console uses Kumo and RTL-safe logical layout from its first UI change. Decision (2026-07-13): no Lingui/i18n in the console — it is internal infrastructure with a handful of operators, and the catalog/build machinery isn't worth carrying. Plain English strings; `packages/admin` localization rules are unaffected. +- Existing unrelated worktree changes are left untouched. +- Published-package changes receive changesets when their workstream lands. + +## Dependency Model + +### Workstream IDs + +| ID | Workstream | Primary output | +| ----- | -------------------------------------- | ------------------------------------------------------------------------------ | +| `W0` | Decisions and feasibility | Ratified contracts and proved platform assumptions | +| `W1` | Shared protocol and policy | Lexicons, label crypto, policy schema, and moderation semantics | +| `W2` | Service foundation and persistence | `apps/labeler`, D1 state, bindings, queues, and workflow skeleton | +| `W3` | Label issuance and distribution | Typed signer, label history, query API, and replayable subscription | +| `W4` | Aggregator label consumption | Verified subscription, current state, hydration, redaction, and cascades | +| `W5` | Client eligibility enforcement | One moderation helper used by registry client, core, CLI, and admin | +| `W6` | Discovery and assessment orchestration | Independent Jetstream ingest, verified subjects, run lifecycle, reconciliation | +| `W7` | Artifact and deterministic analysis | Safe artifact acquisition, bundle checks, capability analysis | +| `W8` | AI, image, and policy resolution | Versioned model pipeline, normalized findings, automated decisions | +| `W9` | Operator authentication and console | Access-protected review UI and manual label actions | +| `W10` | Transparency and notifications | Public assessment API, policy document, email, reconsideration flow | +| `W11` | Operations and self-hosting | Deployment, secrets, observability, backups, rotation, runbooks | +| `W12` | Conformance and security | Cross-component, adversarial, browser, and production-launch verification | + +### High-Level Graph + +```mermaid +flowchart TD + W0[W0 Decisions and feasibility] + W1[W1 Shared protocol and policy] + W2[W2 Service foundation] + W3[W3 Issuance and distribution] + W4[W4 Aggregator consumption] + W5[W5 Client enforcement] + W6[W6 Discovery and orchestration] + W7[W7 Artifact and deterministic analysis] + W8[W8 AI and policy resolution] + W9[W9 Operator console] + W10[W10 Transparency and notifications] + W11[W11 Operations] + W12[W12 Conformance and security] + + W0 --> W1 + W0 --> W2 + W0 --> W7 + W0 --> W11 + W1 --> W2 + W1 --> W3 + W1 --> W4 + W1 --> W5 + W1 --> W8 + W2 --> W3 + W2 --> W6 + W2 --> W8 + W2 --> W9 + W2 --> W10 + W2 --> W11 + W3 --> W4 + W3 --> W6 + W3 --> W9 + W3 --> W10 + W4 --> W5 + W4 --> W12 + W5 --> W9 + W5 --> W12 + W7 --> W6 + W7 --> W8 + W8 --> W9 + W8 --> W10 + W8 --> W12 + W9 --> W12 + W10 --> W12 + W11 --> W12 +``` + +### Critical Paths + +Protocol and enforcement path: + +```text +W0 contract decisions +-> W1 shared labels and policy +-> W2 service state foundation +-> W3 signed distribution +-> W4 aggregator consumption +-> W5 install/update enforcement +-> W12 conformance +``` + +Automated assessment path: + +```text +W0 model/artifact/capability feasibility +-> W7 deterministic analysis +-> W8 AI and policy resolution + \ +W2 service foundation -> W6 verified discovery/orchestration + / + -> W12 conformance +``` + +Operational launch path: + +```text +W2 service foundation +-> W3 signer +-> W9 operator controls +-> W11 key/backup/incident operations +-> W12 production drill +``` + +`W4` can begin against signed fixtures before the hosted labeler is deployed. `W7` can build shared artifact primitives in parallel with `W2` after Gate 0. `W9` can build read-only assessment views after the state/API contracts freeze, but mutation UI waits for `W3` and `W5`. + +## Integration Gates + +### Gate 0: Contract and Platform Decisions + +Required before production implementation is treated as stable: + +- Experimental labeler NSIDs and public API shapes are ratified. +- Label vocabulary, subject/CID rules, blocking matrix, and override precedence are represented as executable policy fixtures. +- The `security:yanked` to `security-yanked` production preflight is complete. +- The labeler-declaration strategy is decided: minimal `app.bsky.labeler.service/self` or base label service plus EmDash policy document. +- Signing-key format, Secrets Store binding, generation ceremony, and recovery custody are selected. +- DRISL labels signed in workerd verify in an independent ATProto implementation, and vice versa. +- The aggregator mirror contract can supply artifact bytes and metadata required by the assessor. +- The deterministic capability-analysis rule set (declared-vs-actual access) and the exact critical block rule for code/capability findings are confirmed. +- `@cf/moonshotai/kimi-k2.7-code` and its structured-output path can process representative plugin code and image inputs through the Workers AI binding (`env.AI`). + +Gate owner: `W0`. + +### Gate 1: Shared Contract Foundation + +- Labeler lexicons generate and round-trip through checked-in types. +- Shared policy fixtures produce identical outcomes in Node and workerd. +- DRISL sign/verify vectors pass in the shared package. +- Every current registry consumer recognizes `security-yanked` and the complete blocking vocabulary. +- No production label has been issued under the legacy colon value. + +Gate owners: `W1`. + +### Gate 2: Signed Label Distribution + +- An authorized test action produces one signed label and one immutable audit action. +- `queryLabels` returns it with valid signature and pagination. +- `subscribeLabels` replays it from cursor `0`, resumes after disconnect, and streams a later negation. +- A fresh aggregator verifies the source DID/key, ingests history, and projects correct current state. +- Key rotation test vectors and replay behavior pass. + +Gate owners: `W2`, `W3`, minimum `W4`. + +### Gate 3: Consumer Eligibility Contract + +- Aggregator request-header parsing and `atproto-content-labelers` behavior pass contract tests. +- Search, package, release, and latest-release endpoints expand publisher/package/release subjects correctly. +- One registry-client helper returns the same eligibility result for browser, server install, update, and CLI paths. +- Missing positive assessment, pending, error, blocking labels, package/publisher cascades, yanks, and takedowns cannot be installed through official paths. +- Manual override precedence works for an exact release CID and cannot override broader manual blocks. + +Gate owners: `W4`, `W5`. + +### Gate 4: Automated Assessment + +- A verified release event creates one idempotent run and no label is signed before record verification. +- Verified mirror bytes are checksum-checked and safely extracted; declared-URL fallback passes SSRF/adversarial tests. +- Permanent validation failures become deterministic findings; transient failures retry and become `assessment-error` only after exhaustion. +- Capability-analysis, code/metadata, image, and publisher-history stages produce normalized findings. +- Policy emits blocking labels only for critical security/impersonation findings and warning labels for quality findings. +- A model outage never creates a malicious-content label. + +Gate owners: `W6`, `W7`, `W8`. + +### Gate 5: Operator and Transparency Readiness + +- Reviewers/admins authenticate through verified Access JWTs and see only role-allowed actions. +- False-positive unblock atomically negates selected automated labels and issues `assessment-passed` plus `assessment-overridden`. +- Only admins can issue/retract `!takedown` and `publisher-compromised`. +- Public current/historical assessment APIs distinguish automated runs, pending runs, and manual actions. +- Notification outbox delivers block/warn/retraction notices without exposing private evidence. +- Arabic RTL, keyboard, localization, and confirmation-ceremony tests pass. + +Gate owners: `W9`, `W10`. + +### Gate 6: Production Launch + +- Every first-party release has a current assessment under the production policy. +- Positive-assessment enforcement is enabled only after the assessed catalog and rollback path are verified. +- Full label replay into a clean aggregator reproduces current state. +- Signing-key rotation, compromised-key response, D1 restore, Jetstream cursor recovery, Queue/DLQ recovery, and notification recovery drills pass. +- External security review has no unresolved critical or high findings. +- Production smoke succeeds from release publication through assessment, label ingest, discovery, and clean-site install decision. +- No source/test imports `.opencode/plans`, every retained fixture tests an implemented EmDash component, and this planning folder is removed before the umbrella PR merges. + +Gate owners: `W11`, `W12`, with all preceding gates complete. + +## Workstream W0: Decisions and Feasibility + +This workstream converts the remaining ratification points into executable decisions and platform proofs. + +### `W0.1` Freeze public identifiers and API shapes + +Decide: + +- Experimental labeler query/procedure NSIDs. +- Whether a decision-notice repository record ships in v1. +- Stable path for `/.well-known/emdash-labeler-policy.json`. +- Public assessment ID format and current-assessment lookup parameters. +- Error codes and pagination/cursor shapes. + +Output: approved lexicon examples and endpoint table used by `W1.2`. + +Dependencies: none. + +### `W0.2` Freeze label policy fixtures + +Encode table-driven fixtures for: + +- Eligibility labels and precedence. +- Blocking security labels. +- Warning/quality labels. +- Manual release/package/publisher labels. +- CID-bound versus URI-wide applicability. +- Manual override behavior. +- Negation and expiration. +- Accepted-labeler and `redact` policy. + +Output: machine-readable fixture set with expected `eligible | pending | error | blocked` outcomes. + +Dependencies: none. + +### `W0.3` Complete the vocabulary cutover audit + +Inspect and enumerate every current use of `security:yanked`, including: + +- `apps/aggregator` migrations and SQL filters. +- Aggregator read endpoint descriptions/tests. +- `packages/registry-client` docs/types/tests. +- `packages/core/src/api/handlers/registry.ts` install/update checks. +- Admin rendering/tests. +- Seed/deployment data, if any. + +Define the preflight query for persisted legacy labels and the temporary dual-read procedure only if production data exists. + +Output: exact patch list and migration decision for `W1.1`. + +Dependencies: none. + +### `W0.4` Prove label crypto interoperability + +Build a focused workerd prototype using `@atcute/cbor` and `@atcute/crypto`: + +- Construct only allowed v1 fields. +- DRISL encode, hash, and sign with selected curve/key format. +- Verify through an independent implementation or reference vectors. +- Verify an independently signed label in the prototype. +- Reject extra fields, wrong keys, malformed bytes, and invalid signatures. +- Define the minimal routine-rotation and compromise procedure: issuance pause boundary, DID update ordering, old-key identification, historical re-signing, and subscriber recovery. + +Output: selected crypto API and key-lifecycle contract consumed by `W3.7` and documented by `W11.4`. W1.4 adds vectors only alongside the implemented signer/verifier. + +Dependencies: signing-key format draft. + +### `W0.5` Decide identity and declaration hosting + +Decision: v1 uses the base `#atproto_labeler` service plus the EmDash policy document without an `app.bsky.labeler.service/self` declaration. The declaration is application-specific and its report-scope metadata would misrepresent a service that rejects standard moderation reports. Clients obtain localized label definitions from the versioned policy document. + +The labeler deployment owns the static DID document and policy endpoint. W3.1 publishes the DID fixture with `#atproto_label` and `#atproto_labeler`; the ratified moderation-policy fixture is the policy source of truth. + +Output: DID document fixture, declaration/policy fixture, and deployment ownership. + +Dependencies: `W0.1` draft vocabulary. + +### `W0.6` Prove artifact mirror consumption + +Against the current/planned aggregator mirror: + +- Resolve release coordinates to mirror URL/object metadata. +- Fetch the exact bundle without privileged database coupling. +- Verify release record, CID, artifact ID, and checksum. +- Distinguish mirror miss, transient failure, and permanent mismatch. +- Confirm declared-URL fallback inputs. + +Output: artifact acquisition contract consumed by `W7`. + +Dependencies: none. + +Decision (2026-07-16): v1 acquisition is DECLARED-URL-FIRST — the aggregator mirror is unbuilt (`records-consumer.ts` TODO; `mirrored_artifacts` never populated), and the release record's checksum/CID pin makes a declared-URL fetch exactly as verifiable as a mirror fetch, so the mirror is an availability/resilience layer, not an integrity requirement. The labeler fetches from the publisher's declared artifact URL under the ratified SSRF controls (HTTPS, DNS/redirect hardening, byte/time budgets, no ambient credentials) with checksum/CID verification; dead or changed URLs classify as the existing transient/permanent acquisition failures. W7.2's acquisition module is written mirror-ready with a source-preference switch: the future mirror interface is a deterministic R2 object key derived from release coordinates, resolved through the aggregator service binding (same read-path pattern ratified for W8.4 slice 3), and mirror-first ordering activates when the aggregator mirror lands on its own timetable (delegated release-service branch) without changing the labeler contract. W7 is no longer gated on aggregator mirror work. + +### `W0.7` Confirm AI model and capability analysis + +Evaluate representative fixtures through: + +- Confirm `@cf/moonshotai/kimi-k2.7-code` via the Workers AI binding (`env.AI`) handles full-bundle code analysis and image analysis with strict structured output. `@cf/zai-org/glm-5.2` (text-only, same binding) is an available alternative; v1 ships single-model. +- Confirm the release artifact size cap fits within the model's context window; otherwise define the coverage-degradation fallback (`partial`/`unavailable` rather than silent truncation). +- Define the deterministic capability-analysis rule set: API-surface extraction (network/fetch, storage, `eval`/`Function`, filesystem, crypto, env/process, DOM) versus declared access. +- Record model input/output limits, supported file/image types, and prompt version. + +Output: confirmed model + capability-analysis baseline. + +Dependencies: policy fixtures from `W0.2`. + +### W0 Completion + +Gate 0 passes. Failed crypto, identity, mirror, or model assumptions change the spec before downstream production work continues. + +## Workstream W1: Shared Protocol and Policy + +Create or extend shared packages so the labeler and every consumer use the same contracts. + +### `W1.1` Apply the label vocabulary cutover + +Update all current references from `security:yanked` to `security-yanked` and centralize the complete vocabulary. + +Files include: + +- `apps/aggregator/migrations/0001_init.sql` +- `apps/aggregator/src/routes/xrpc/searchPackages.ts` +- `packages/registry-client/src/discovery/index.ts` +- `packages/core/src/api/handlers/registry.ts` +- Registry admin components and tests. + +If `W0.3` finds persisted legacy values, add the bounded compatibility read and explicit removal condition. Do not emit new colon-valued labels. + +Dependencies: `W0.2`, `W0.3`. + +### `W1.2` Add labeler lexicons + +Under `packages/registry-lexicons/lexicons/com/emdashcms/experimental/labeler/`, define: + +- Assessment/public finding definitions. +- `getAssessment`. +- `getCurrentAssessment`. +- `listAssessments`. +- `getPolicy`. +- Optional decision-notice record if ratified. + +Use standard `com.atproto.label.*` lexicons for label distribution rather than redefining labels. + +Dependencies: `W0.1`. + +### `W1.3` Generate and export typed contracts + +- Regenerate atcute types. +- Export values and types from `packages/registry-lexicons/src/index.ts`. +- Extend `NSID`, `QUERY_NSIDS`, and procedure/record constants as appropriate. +- Add valid/invalid/current/manual-override/superseded fixtures. +- Add a changeset for the published lexicon package. + +Dependencies: `W1.2`. + +### `W1.4` Create shared label crypto primitives + +Create `packages/registry-moderation` (or a ratified equivalent) with Workers/Node-compatible exports: + +- Strict `LabelV1` construction. +- `signLabel` and `verifyLabel`. +- Public-key extraction for `#atproto_label`. +- Signature and key-rotation error codes. +- External test vectors from `W0.4`. + +No generic `signObject` API is exported. + +Dependencies: `W0.4`. + +### `W1.5` Implement policy and subject applicability + +In the shared moderation package: + +- Export label vocabulary and definitions. +- Parse versioned policy fixtures. +- Classify labels as eligibility, automated blocking, warning, or manual system labels. +- Evaluate source DID, negation, expiration, CID binding, and subject scope. +- Expand publisher/package/release subject sets from typed inputs. +- Enforce manual override precedence without bypassing broader manual blocks. +- Export the single pure `evaluateReleaseModeration` function and `ReleaseModeration` result type used by every consumer. +- Drive evaluator behavior from the `W0.2` fixture corpus. + +Dependencies: `W0.2`. + +### `W1.6` Implement accepted-labeler header parsing + +Add a shared RFC-8941-compatible parser/serializer for: + +- `atproto-accept-labelers`. +- `atproto-content-labelers`. +- DID deduplication. +- Boolean `redact` merge behavior. +- Missing versus empty distinction. +- Invalid syntax errors. + +Dependencies: none after Gate 0. + +### `W1.7` Publish policy schema and fixtures + +Define the machine-readable policy document schema, versioning rules, localized label definitions, official-client effects, contact fields, and assessment schema version. + +Fixtures cover: + +- Current production policy. +- Unknown future labels. +- Invalid block mappings. +- AI critical quality normalization to warning. +- Manual action restrictions. + +Dependencies: `W1.3`, `W1.5`. + +### W1 Completion + +Gate 1 passes. The signer, aggregator, registry client, core handlers, CLI, and admin can consume one typed policy and fixture corpus. + +## Workstream W2: Service Foundation and Persistence + +### `W2.1` Scaffold `apps/labeler` + +Follow `apps/aggregator` deployment conventions: + +- Cloudflare Vite plugin. +- Wrangler config and generated Worker types. +- D1 migrations. +- Queues and DLQs. +- Workflows binding. +- Durable Object migrations. +- R2 evidence binding. +- Workers AI binding (`AI`), no AI Gateway. +- Static assets for the later console. +- Real workerd/D1 Vitest configuration. + +Add root/CI scripts so `apps/labeler` build, typecheck, tests, and migrations are not skipped by package-only filters. + +Dependencies: Gate 0, `W1.3` package shape. + +### `W2.2` Implement D1 migrations + +Create tables from the spec for: + +- `subjects`. +- `assessments` and `current_assessments`. +- `findings` and `evidence_objects`. +- `issued_labels` and `label_sequence`. +- `actions` and `notifications`. +- `ingest_state`. +- Assessment stage attempts and dead-letter metadata where needed. + +Add constraints/indexes for run-key uniqueness, subject lookup, current assessment, label sequence, and outbox delivery. + +Dependencies: `W1.5`, schema ratification. + +### `W2.3` Implement typed repositories + +Create narrow repositories for: + +- Subject observation/deletion. +- Assessment run creation and state transition. +- Current-assessment projection. +- Findings/evidence metadata. +- Action audit. +- Notification outbox. +- Ingest cursors. + +Repositories expose transaction-scoped operations needed by finalization. Do not spread raw SQL across handlers and workflow steps. + +Dependencies: `W2.2`. + +### `W2.4` Implement assessment lifecycle state machine + +Define legal transitions for: + +```text +observed -> verifying -> pending -> running -> passed | warned | blocked | error + -> stale | cancelled +``` + +Include: + +- Deterministic `runKey` and stable trigger IDs. +- Redelivery idempotency. +- Workflow lease/ownership. +- Immutable operator-trigger/action reference shape used by rerun backends and later UI. +- Typed assessment-stage adapter interfaces and policy-proposal/finalization boundary, with deterministic stub implementations for orchestration tests. +- Supersession links. +- Preserve immutable completed row state; derive public `superseded` when a newer completed current assessment links to it and replaces the current pointer. +- Current pointer update only on successful completed finalization. +- Pending/error labels represented as effects, not database state shortcuts. + +Dependencies: `W2.3`. + +### `W2.5` Implement evidence storage boundary + +- Hash every evidence object. +- Store large private evidence in R2 and references in D1. +- Separate public summaries from private detail at the type boundary. +- Support lifecycle metadata without deciding an organization retention policy. +- Prevent public serializers from accepting private evidence types. + +Dependencies: `W2.3`. + +### `W2.6` Add health and internal diagnostics + +Provide unauthenticated liveness and protected readiness/detail endpoints covering D1, Queue, Workflow, DO, AI, R2, signing configuration, and policy version without exposing secrets. + +Dependencies: `W2.1`, `W2.3`. + +### W2 Completion + +The service can persist and recover every lifecycle state without model or label-distribution code. All migrations and repositories pass real workerd/D1 tests. + +## Workstream W3: Label Issuance and Distribution + +### `W3.1` Provision service identity and key material + +Staging and production both use real custom domains on the emdashcms.com zone from the start; no workers.dev DID identity. Production key generation is a maintainer action performed later: the maintainer generates the scalar and keeps the offline copy in Keeper. Code work uses a disposable staging key until then and must not block on the production ceremony. + +- Add the minimal staging custom-domain, D1, Secrets Store, and static DID/policy bindings required to exercise the signer. Full production environment hardening remains in `W11.1`. +- Deploy staging `did:web` document. +- Publish `#atproto_label` and `#atproto_labeler` entries. +- Configure Secrets Store binding for the private key. +- Publish the selected declaration/policy discovery path. +- Add startup validation that public/private key IDs agree. + +Dependencies: `W0.5`, `W2.1` app scaffold. + +### `W3.2` Implement typed issuance proposals + +Define separate proposal types for: + +- Automated assessment labels. +- Reviewer release actions, including `security-yanked`. +- Reviewer package actions limited to `package-disputed`. +- Admin release/package/publisher/system actions, including `!takedown` and `publisher-compromised`. +- Negations. + +The issuer validates vocabulary, actor role, subject type, CID rules, finding category/severity, reason, action ID, and idempotency key before signing. + +Dependencies: `W1.4`, `W1.5`, `W2.3`. + +### `W3.3` Implement atomic sequence and issuance + +In one D1 transaction: + +- Allocate the next sequence without `MAX(seq)`. +- Store immutable action/assessment linkage. +- Store the signed label. +- Update any effective local projection. +- Commit outbox/broadcast work. + +Concurrent issuance tests prove monotonic unique sequence and idempotent retries. + +Dependencies: `W3.2`. + +### `W3.4` Implement `queryLabels` + +Support: + +- URI patterns. +- Sources. +- Limit and cursor. +- Full signatures. +- Negation/history semantics. +- Stable ordering and pagination. +- Public rate limits. + +Dependencies: `W3.3`, standard lexicon types. + +### `W3.5` Implement Subscriber Durable Object + +One named DO: + +- Accepts inbound WebSockets with Hibernation API. +- Replays D1 rows after cursor. +- Joins live broadcast only after replay boundary is safe. +- Handles cursor `0` full history. +- Isolates slow/broken subscribers. +- Recovers entirely from D1 after eviction/restart. + +Dependencies: `W3.3`. + +### `W3.6` Implement `subscribeLabels` + +Wire the standard XRPC subscription endpoint to the Subscriber DO and cover framing, info/error messages, cursor validation, disconnect/reconnect, replay/live race, and backpressure. + +Dependencies: `W3.5`. + +### `W3.7` Implement routine key rotation + +- Pause issuance. +- Publish new DID key. +- Resume with new key ID. +- Lazily re-sign queried historical labels without changing `cts`. +- Expose rotation status and alerts. + +Dependencies: `W0.4` key-lifecycle contract, `W3.4`. + +### `W3.8` Reject unsupported standard reports + +Implement `com.atproto.moderation.createReport` as an explicit `NotSupported` XRPC response while v1 report intake is deferred. The handler never writes a report row or enqueues work. Add a conformance test and keep the identity/declaration metadata consistent with this behavior. + +Dependencies: `W0.5`, XRPC router from `W3.4`. + +### W3 Completion + +The labeler passes Gate 2's signer/query/subscription checks independently of automated assessment. + +## Workstream W4: Aggregator Label Consumption + +### `W4.1` Implement labeler configuration and DID resolution + +Use the existing `labelers` table for bounded trusted/discoverable sources. Resolve endpoint/key, cache with expiry, refresh periodically, and retry once on signature failure. + +Dependencies: `W1.4`, `W3.1` fixtures. + +### `W4.2` Implement outbound subscription Durable Object + +One DO per configured labeler: + +- Connect to `subscribeLabels`. +- Resume from `ingest_state` cursor. +- Verify signatures before Queue acceptance. +- Refresh DID key on first verification failure. +- Persist cursor only after durable acceptance. +- Reconnect with jittered backoff. +- Support cursor `0` rebuild. + +Dependencies: `W4.1`, `W3.6` or protocol fixtures. + +### `W4.3` Implement Labels Queue consumer + +Before the first ingest write, add the forward aggregator migration from the `W0.3` audit: replace the `labels` history identity `(src, uri, val, cts)` with a collision-safe event identity (signed-label digest plus unique verified `(src, source_sequence, frame_index)` coordinates), with deterministic duplicate, multi-label-frame, and same-`cts` conflict handling. + +In a D1 transaction: + +- Insert append-only label history. +- Upsert `label_state` only for newer `cts`. +- Preserve negation, expiry, CID, trusted source, signature, and received metadata. +- Ignore duplicate and out-of-order state changes correctly. + +Dependencies: `W4.2`. + +### `W4.4` Parse request labeler policy + +At the aggregator request boundary: + +- Parse `atproto-accept-labelers` once using `W1.6`. +- Apply deployment defaults only when absent. +- Treat empty as no labelers. +- Reject malformed syntax. +- Resolve unavailable sources. +- Pass typed policy to all handlers. +- Set and CORS-expose `atproto-content-labelers`. + +Dependencies: `W1.6`. + +### `W4.5` Implement batched subject expansion and hydration + +Refactor `apps/aggregator/src/routes/xrpc/views.ts` and handlers to batch labels for: + +- Publisher DID. +- Package profile URI/current CID. +- Release URI/current CID. + +Apply accepted source, negation, expiry, URI-wide, and CID-bound rules. Avoid mapper-level N+1 queries. + +Dependencies: `W1.5`, `W4.3`, `W4.4`. + +### `W4.6` Implement endpoint filtering/redaction + +Update: + +- `searchPackages`. +- `getPackage`. +- `resolvePackage`. +- `listReleases`. +- `getLatestRelease`. + +Requirements: + +- `redact` handles `!takedown`/`!suspend`. +- Direct reads return explainable tombstone/moderation state. +- Latest release is selected dynamically from eligible releases, not blindly from `packages.latest_version`. +- Package/publisher labels cascade at policy evaluation, not by copying labels. + +Dependencies: `W1.5` shared evaluator, `W4.5`. + +### `W4.7` Add replay and reconciliation + +- Rebuild one labeler from cursor `0` into an empty projection. +- Compare local sequence/high-water state with subscription progress. +- Re-resolve signing keys on cron. +- Emit component metrics/events for lag and verification failure; `W11.3` turns them into deployment dashboards and alerts. + +Dependencies: `W4.3`. + +### W4 Completion + +The aggregator consumes any conforming configured labeler and produces deterministic hydrated/filtering behavior from the shared policy. + +## Workstream W5: Client Eligibility Enforcement + +### `W5.1` Integrate the shared moderation evaluator + +Re-export or wrap `W1.5` from `packages/registry-client` without reimplementing policy. The shared evaluator accepts: + +- Accepted labeler policy. +- Publisher labels. +- Package labels and current package CID. +- Release labels and current release CID. +- Optional public assessment reference. + +Return: + +```ts +interface ReleaseModeration { + eligibility: "eligible" | "pending" | "error" | "blocked"; + reasonCodes: string[]; + blockingLabels: string[]; + stateLabels: string[]; + warningLabels: string[]; + suppressedLabels: string[]; + applicableLabels: ModerationLabel[]; + redacted: boolean; +} +``` + +Run the `W0.2` fixture corpus against the shared package and the registry-client export to prove they are identical. + +Dependencies: `W1.5`. + +### `W5.2` Extend discovery-client response handling + +- Interpret `atproto-content-labelers`. +- Preserve source/CID/expiry needed by the evaluator. +- Fetch `getCurrentAssessment` on demand. +- Allow per-request accepted-labeler overrides if needed. +- Keep signature verification available through a full-label fetch path. + +Dependencies: `W1.3`, `W4.4`, `W5.1`. + +### `W5.3` Enforce in core install and update handlers + +Replace direct `security:yanked` comparisons in `packages/core/src/api/handlers/registry.ts` with `W5.1` immediately before artifact download. + +Both install and update must reject: + +- Missing positive assessment. +- Pending/error. +- Automated blocking labels. +- `security-yanked`. +- `!takedown`/redacted state. +- Package/publisher blocking cascades. + +Warning-only releases require explicit consent data already validated by the server. + +Dependencies: `W4.6`, `W5.1`. + +### `W5.4` Surface eligibility in existing CLI discovery paths + +Use the same evaluator for the current CLI registry search/info surfaces. Do not invent install/update commands in this workstream. If CLI install/update is added later, it must consume the same evaluator; pinned installs may override warnings only, never missing assessment or blocking/manual system labels under official policy. + +Dependencies: `W5.1`, existing CLI registry discovery path. + +### `W5.5` Update admin registry UI + +Update registry browse/detail/install surfaces to show: + +- Eligibility state. +- Localized label definitions. +- Issuing labeler. +- Public summary/coverage. +- Automated versus manual override. +- Warning confirmation. +- Block/reconsideration details. + +Include accepted-labeler config in every React Query key. + +Dependencies: `W5.2`, `W10.2` public API contract; static fixtures may unblock UI earlier. + +### `W5.6` Add installed-release alerts + +When an installed release becomes `security-yanked`, taken down, or newly blocked, surface it in the admin and prevent updating to another ineligible version. Define periodic/on-admin-load refresh without adding unsafe background assumptions. + +Dependencies: `W5.1`, existing plugin-state storage. + +### W5 Completion + +Gate 3 passes. No official install/update path contains its own label-string policy. + +## Workstream W6: Discovery and Assessment Orchestration + +### `W6.1` Reuse/extract Jetstream client primitives + +Share or adapt the tested patterns from: + +- `apps/aggregator/src/jetstream-client.ts`. +- `apps/aggregator/src/jetstream-ingestor.ts`. +- `apps/aggregator/src/records-do.ts`. + +Preserve close semantics, structural validation, cursor-after-acceptance, jittered reconnect, and test injection. + +Dependencies: `W2.1`. + +### `W6.2` Implement Discovery Durable Object + +- Subscribe to stable and experimental release collections. +- Create deterministic verification job keys. +- Enqueue before persisting cursor. +- Handle create/update/delete. +- Expose health/cursor state. +- Never issue labels directly. + +Dependencies: `W6.1`, `W2.3`. + +### `W6.3` Verify source records + +For each job: + +- Fetch exact URI/CID through aggregator `sync.getRecord` cache or publisher PDS fallback. +- Verify MST proof/signature and collection/rkey. +- Reject mismatch/forgery/deletion. +- Persist subject only after verification. +- Issue `assessment-pending` only after verification through `W3`. + +Dependencies: `W3.3`, existing aggregator/PDS verification helpers, `W2.4`. + +Decision (2026-07-16): the aggregator's `sync.getRecord` passthrough is RATIFIED as excluded from label enforcement. It serves the publisher's own signed record CAR — metadata, not artifact bytes — which exists identically on the publisher's PDS and the relay firehose, so filtering our cache removes nothing from the network while breaking the sync layer's verifiable-passthrough contract and its public cacheability. Label enforcement lives in the application views, install eligibility, and — bindingly — the future artifact mirror: when the aggregator mirror lands, a `!takedown` MUST stop mirror serving of the artifact bytes (that is where our infrastructure would distribute the actual content). Unlawful content is handled by actual out-of-band removal per the CSAM decision, and a deleted row 404s from `sync.getRecord` automatically — deletion, not label redaction, is the mechanism that binds this route. + +### `W6.4` Implement Workflow stage orchestration + +Workflow stages: + +1. Verify subject. +2. Acquire artifact inputs. +3. Deterministic bundle validation. +4. Capability analysis. +5. Code/metadata AI. +6. Image AI where applicable. +7. Policy resolution. +8. Atomic finalization/label issuance. +9. Notification outbox enqueue. + +Each stage stores bounded serializable output and has explicit retry/permanent-failure classification. + +Decision (2026-07-16, resolves the #1978 queue-vs-Workflow question): production wiring executes each assessment run as a Cloudflare WORKFLOW instance whose id derives from the subject (URI+CID) — a thin shell over the tested `AssessmentOrchestrator`: steps call the same stage adapters and atomic finalization, gaining per-stage durable resume, and the instance-per-subject serialization IS the §14.1 per-subject lock, closing both races #1978 deferred by construction. The queue consumer's role narrows to verified discovery + Workflow dispatch. The shell stays thin enough that workerd integration tests plus the existing orchestrator suite carry coverage (Workflows' local-test story is the accepted weak point). + +Dependencies: `W2.4` stage interfaces, `W6.3`. `W7` and `W8` later supply production adapters without changing orchestration. + +### `W6.5` Implement supersession and current projection + +- Serialize finalization per URI + CID. +- Under the same subject lock used for finalization, re-read deletion and current-CID state immediately before label issuance. +- If the subject was deleted or its CID changed, retain the completed evidence as stale/forensic-only, issue no new positive/current labels, and do not move the current pointer. +- Keep exactly one completed automated current assessment. +- Leave current pointer unchanged while a newer run is pending. +- Let pending/error labels affect eligibility as specified. +- Never let stale/cancelled runs become current. +- Preserve active manual overrides across automated runs. +- Set `supersedes_assessment_id` on a successful replacement; public serializers derive the old run's `superseded` state from that link/current pointer without mutating its original completed result. + +Dependencies: `W2.4` policy-proposal contract, `W3.3`. + +### `W6.6` Implement initial and operator rerun triggers + +- `initial:` trigger. +- Immutable operator action ID trigger. +- Deterministic run key including policy/model/prompt. + +Dependencies: `W2.4` operator-trigger contract. + +### `W6.7` Implement model/policy-revision rerun triggers + +- Accept a new model ID or policy version as an immutable trigger ID. +- Select only releases whose current assessment used an older model/policy version. +- Reuse the same deterministic run creation path as initial/operator triggers. +- Deduplicate repeated revision notifications. + +Dependencies: `W6.6`. + +### `W6.8` Implement reconciliation + +Scheduled reconciliation finds: + +- Aggregator releases absent from subjects. +- Verified subjects without assessment state. +- Stuck pending/running runs. +- Completed current assessments with inconsistent effective labels. +- Deleted subjects with queued/running work. + +Dependencies: `W6.5`, aggregator read API. + +### W6 Completion + +Every verified event creates the correct idempotent lifecycle, and replay/reconciliation repair missed work without duplicate public decisions. + +## Workstream W7: Artifact and Deterministic Analysis + +Coordinate this work with the delegated release service plan's proposed `packages/registry-verification`; do not create competing fetch/bundle implementations. + +### `W7.1` Establish shared verification ownership + +Decision (2026-07-11): the delegated release service work owns `packages/registry-verification`, already built on its integration branch (checksum, safe fetch, canonical bundle validation, Sigstore provenance, workerd tests) and covering the required shared surface. `W7.2`/`W7.3` extend that package after it reaches main with the labeler's additions: mirror-first acquisition, the SSRF-hardened declared-URL fallback, and mirror-miss/transient/permanent-mismatch classification. If sequencing inverts and W7 starts first, graduate the package from that branch to main in a standalone PR; do not cherry-pick across integration branches or build a competing implementation. Ratified (2026-07-16, sequencing has inverted): the graduation unit is a coordinated THREE-package promotion — registry-verification hard-depends on the delegated branch's additive, main-absent slices of `@emdash-cms/plugin-types` (manifest-schema, declared-access) and `@emdash-cms/registry-lexicons` (profileExtension) — lifted verbatim in one main PR with minor changesets for the two published packages; content-identical code reconciles into the delegated branch on its next main sync. + +Dependencies: `W0.6`, `packages/registry-verification` on main. + +### `W7.2` Implement verified artifact acquisition + +- Prefer validated aggregator mirrors. +- Verify release URI/CID/artifact ID/checksum. +- Fall back to declared URL with HTTPS, DNS/redirect SSRF controls, byte/time budgets, and no ambient credentials. +- Distinguish mirror miss, transient fetch failure, permanent checksum mismatch, and policy rejection. + +Dependencies: `W7.1`. + +### `W7.3` Implement canonical bundle validation + +Reject: + +- Compressed/decompressed/per-file/file-count excess. +- Traversal and duplicate normalized paths. +- Symlinks, devices, and special files. +- Malformed gzip/tar. +- Missing/duplicate manifest or entrypoint. +- Unknown access categories/operations. +- Manifest/release declared-access mismatch. + +Produce validated file inventory and safe full-bundle/image inputs for later stages. + +Dependencies: `W7.1`. + +### `W7.4` Implement deterministic security checks + +Decision (2026-07-17, ratified): **the deterministic-analysis surface is cut to integrity/bundle validation only; capability analysis (W7.5) and all SBOM/dependency/malware/advisory scanning are NOT built, and the `deterministic` + `dependency` orchestrator stages are removed.** Rationale: (1) **plugins are self-contained with no external dependencies** — SBOM, dependency-graph, and advisory scanning have nothing to analyze, and a dependency-CVE is not a moderation signal anyway (over-moderation / Dependabot's job). (2) **The plugin runtime ENFORCES declared access**, so static undeclared-access / capability extraction is redundant: plugins execute in a dynamically-loaded workerd sandbox whose `globalOutbound` routes ALL outbound `fetch()` (raw global fetch included) through the backing service (`packages/workerd/src/sandbox/capnp.ts`, `globalOutbound = "emdash-backing"`), and `packages/cloudflare/src/sandbox/bridge-http.ts` checks the `network:request` capability + the `allowedHosts` allowlist on every request and every redirect hop (SSRF-hardened), throwing on an undeclared host; every other host resource (content/media/users/email) reaches the plugin only through the capability-gated backing-service API, never a raw binding. A plugin declaring "no network" but calling `fetch('evil.com')` is blocked at runtime regardless of its source, so statically detecting the attempt catches nothing the sandbox does not already stop — and the code-AI adapter is the better judge of malicious _intent_ on top of that. Signature-based AV is additionally a poor fit (novel obfuscated source-level JS, infeasible in workerd). The only residual "intelligence" idea with distinct value — a curated known-bad artifact-checksum denylist — mostly overlaps the existing cross-publisher checksum-reuse detection in the history stage and is deferred (do-it-now-or-not-at-all: not now). **Consequence:** the deterministic half of the pipeline is just integrity/bundle validation, which already emits real `artifact-integrity-failure` / `invalid-bundle` findings from the acquire stage (#2069); the W7.4 forbidden-archive/executable-payload and metadata-consistency checks fold into that acquire-stage validation rather than a separate stage. W7.5 and the capability-mismatch portion of W7.6 are struck. This dissolves the "Select dependency/malware scanners, SBOM tooling" gate — there is nothing to select. + +Adapters/findings for: + +- Declared-vs-actual capability analysis (feeding `undeclared-access`). +- Forbidden archive entries and executable payload classes. +- Unambiguous forbidden runtime patterns (local rules, not a signature feed). +- Metadata/identity/URL consistency. +- Image dimensions/types. + +Map permanent conditions to `artifact-integrity-failure`, `invalid-bundle`, `undeclared-access`, or other ratified deterministic findings. Never map transport errors to malicious findings. + +Dependencies: `W0.2`, `W7.3`. + +### `W7.5` Implement full-bundle capability extraction + +- Statically extract the bundle's real API/capability surface (network/fetch, storage, `eval`/`Function`, filesystem, crypto, env/process, DOM). +- Diff the extracted surface against the signed release's declared access for the `W7.4` declared-vs-actual check. +- Produce the normalized capability evidence consumed by the `W8.2` AI adapter. + +No dependency, SBOM, or advisory content is produced by this task. + +Dependencies: `W7.3`. + +### `W7.6` Port and expand deterministic fixtures + +Reuse legacy marketplace fixtures and add archive, SSRF, capability declaration mismatch, checksum, and identity cases. Tests run without any live external service. + +Dependencies: `W7.2` through `W7.5`. + +### W7 Completion + +The service can produce complete deterministic and capability findings and safely prepare the full bundle as AI input without calling a model. + +## Workstream W8: AI, Image, and Policy Resolution + +### `W8.1` Define normalized finding contracts + +Implement strict schemas for deterministic, model, image, and history findings with: + +- Allowed category. +- Severity. +- Confidence where applicable. +- Public summary. +- Private detail. +- Evidence references. +- Affected files/images. +- Source/version metadata. + +Unknown categories and unresolved evidence references fail validation. + +Dependencies: `W1.5`, `W2.5`. + +### `W8.2` Implement code/metadata AI adapter + +- Invoke `@cf/moonshotai/kimi-k2.7-code` through the Workers AI binding (`env.AI`), not AI Gateway. +- Never cache moderation inference. +- Attach assessment/policy/model metadata without secrets. +- Delimit all plugin-controlled text as untrusted input. +- Use strict structured output. +- Analyze the full bundle (size-capped, fits the model context; degrade coverage rather than truncate if a bundle ever exceeds context). +- Store model/prompt hashes and an internal per-call inference request ID. +- Treat model/parse/transport failures as retryable operational failures. + +Dependencies: `W0.7`, `W7.3`, `W8.1`. + +### `W8.3` Implement image AI adapter + +- Invoke the same `@cf/moonshotai/kimi-k2.7-code` model (vision) through the Workers AI binding (`env.AI`). +- Preserve true MIME/hash/dimensions/source path. +- Analyze supported icons/screenshots for impersonation, phishing UI, misleading content, and policy imagery. +- Bound image count/size and model concurrency. +- Keep image quality findings non-blocking unless they establish critical impersonation/credential harvesting. + +Dependencies: `W0.7`, `W7.3`, `W8.1`. + +### `W8.4` Implement publisher-history context + +Provide bounded factual context: + +- Prior releases from the DID. +- Recent handle/profile changes. +- Existing active manual labels. +- Repeated exact artifact checksums or policy findings. +- Verification state as display context only. + +History may recommend operator review but cannot automatically produce package/publisher labels. + +Dependencies: `W2.3`, registry publisher/profile data. + +Decisions (2026-07-14): history is not a new contract surface — `FindingSource` already includes `"history"` and the resolver already drops it (`policy-resolver.ts` excludes `source: "history"` before any category→label mapping; `isBlockingFinding` returns false for it), so "never auto-label" is structural, not a new guard. "Recommend review" is passive: a history finding surfaces through the existing assessment-detail projection an operator already opens; no `operational_events` push and no new "flag-for-review" operator action (D3). Two of the five inputs — recent handle/profile changes and verification state — live only in the aggregator's D1, which the labeler has no binding to reach (and `constants.ts` deliberately avoids ingesting profiles); they are deferred to a later slice gated on a ratified read-only labeler→aggregator read path (service-binding preferred over a second D1 binding), not built now (D1). Ratified (2026-07-16): slice 3 uses a read-only SERVICE BINDING to the aggregator — the aggregator exposes the publisher/profile/verification reads as an internal API and the labeler binds to it; no second D1 binding (coupling to the aggregator's physical schema was rejected). Repeated-checksum detection is global (cross-publisher): a checksum reused under a different DID is the sharper signal, needing a new `idx_assessments_artifact_checksum` and a cross-DID lookup (D2). Prior releases, active manual labels (reuse `getActiveLabelState`, filter `!automated && active`), and repeated checksums/findings come from the labeler's own D1. The finding contract is amended (touches W8.1): `source: "history"` findings cite a dedicated history/review category set exempt from the automated-block ∪ warning constraint that `validateFinding` enforces, rather than borrowing a warn category (D4). Neutral display facts (prior-release count, verification badge) are assembled at console read-time in the existing subject-history endpoint, not persisted as a pipeline `ContextBundle` (D5). The stage is DB-bound (`analyzeHistory(db, assessment, opts) → NormalizedFinding[]`), wired into `OrchestratorStages`/`STAGE_ORDER`/`stubStages` and run last. Because history is operator-only context that never becomes a label, the stage is best-effort — on any error it logs and returns no findings rather than throwing, so it can never fail the run and discard the decision-relevant findings from the other stages (adversary-flagged 2026-07-14). History findings keep sensitive specifics (cross-publisher checksum reuse, active manual-label identities/existence including redactions) in `privateDetail` only, never in the public-facing `title`/`publicSummary`; slice (2)'s surfacing must additionally render history findings in the operator projection only and exclude them from any public serializer. Sliced: (1) own-D1 history findings + the W8.1 amendment + the checksum index — unblocked; (2) console neutral-context surfacing — unblocked, optional; (3) the two aggregator-sourced inputs — blocked on D1's read path. + +Ratified (2026-07-16, implementation shape for the read path): the service binding is FETCH-OVER-BINDING, not RPC — the aggregator is an HTTP/XRPC-only module Worker (`apps/aggregator/src/index.ts` exports a plain `{fetch,queue,scheduled}`, no `WorkerEntrypoint`), and the repo's sole precedent (`infra/perf-monitor`) consumes a bound Worker via `env.BINDING.fetch(...)`. Adding a WorkerEntrypoint/RPC class was rejected as a new pattern for marginal typing gain when the shared `@emdash-cms/registry-lexicons` view types already give the response contract. The FOUNDATION (labeler→aggregator binding + typed `aggregator-client.ts` over the EXISTING XRPC reads `getPackage`/`getLatestRelease`/`listReleases`) reuses routes that already serve what most consumers need: `releaseView.release.artifacts` supplies the acquire stage's URL+checksum (the assessment row has cid/checksum but no URL), and `packageView.profile.authors[]`/`security[]` supply W10.4 tiers 1–2. Internal reads omit the `atproto-accept-labelers` header so the view is unfiltered. The client does exactly one fetch per call (no caching — honors W10.4's fresh-read-at-notification-time rule); NotFound→null, transport/5xx→throw (caller treats as transient). Slice 3's own two inputs (verification state, recent handle/profile changes) are NOT in any current view and require NEW aggregator internal reads added on top of the foundation — that is slice 3's remaining work, not the foundation's. + +Ratified (2026-07-16, slice-3 build outcome + adversary pass): of slice 3's two inputs, only VERIFICATION STATE is buildable now — it has a real `publisher_verifications` table (issuer/subject/expiry/tombstone) fed by `ingestPublisherVerification`. RECENT HANDLE/PROFILE CHANGES is genuinely un-ingested: `publishers` is upsert-only current-state with no change log, and the jetstream client does not subscribe to the atproto identity events that carry handle changes, so this input is blocked on W4-era ingestion work (a new ticket), not a read path. It ships as a documented seam with no code path — verified NOT half-wired (no empty-changes return that could read downstream as "verified: no suspicious changes"). Verification state is exposed via a NEW public aggregator read `com.emdashcms.experimental.aggregator.getPublisherVerification` (subject_did → non-tombstoned claims, newest-first). Issuer-redaction decision: because each verification claim is authored by a DIFFERENT party (`issuer_did`, its own records), the read applies redaction PER CLAIM under the request's accepted-labeler policy — a claim whose issuer carries an accepted redact label is filtered from the view (not a 404 on the whole view, so one redacted issuer never hides a publisher's whole verification state). The internal labeler reader sends a blank `atproto-accept-labelers` header → accepted set empty → nothing filtered → it sees every claim including redacted-issuer ones and weighs trust itself (D5's read-time neutral badge is separate from the pipeline finding, which carries issuer DIDs in `privateDetail` only). Expiry is NOT filtered server-side: expired-but-non-tombstoned claims are returned with `expiresAt` so the labeler consumer re-evaluates in-force state itself (the read intentionally surfaces expired claims; the migration index's "unexpired" hot-path scope is an index concern, not a read filter). + +### `W8.5` Implement versioned policy resolver + +Resolve normalized findings under an immutable policy version: + +- Permanent deterministic critical findings -> blocking descriptive labels. +- AI critical security/impersonation -> blocking descriptive labels. +- Quality findings of any severity -> warning/downranking labels. +- No blocking findings -> `assessment-passed`. +- Transient exhaustion -> `assessment-error`. +- Negate stale automated labels, but never action-backed manual labels. + +Return a typed proposal accepted by `W3.2`, not a signed label. + +Dependencies: `W1.7`, `W8.1` through `W8.4`. + +Decisions (2026-07-15, from the first calibration run): the "AI critical" blocking rule is amended — a model/image finding on ANY automated-block category blocks at `high` or `critical` severity, not `critical` only, and a block-category finding below `high` is never dropped: it degrades to a warning label, so any malicious-code finding always produces a visible label (the ratified principle: flag ALL suspected-malicious code; blocking is reserved for high-confidence findings). The first calibration sweep showed two independent models rating a genuine data-exfiltration sample `high`, which the critical-only gate reduced to no label at all (the finding was neither blocked nor warned — dropped entirely); prompt guidance to rate real threats `critical` is added as well, but the resolver gate is the defense because the resolver was built to distrust model severity calibration. Warning-category behavior is unchanged. The issuer's `validateAutomatedProposal` is aligned to accept `high` for automated-block values; because proposals carry no `source`, it gates uniformly at ≥`high`, so a deterministic sub-`high` block proposal — which the resolver would block — would be rejected there. Latent today (`finalize` bypasses the validator and deterministic block findings are high/critical in practice); when the issuer-hardening routes issuance through the validator, plumb `source` through the proposal to close the gap. The moderation-policy vocabulary gains three image-content automated-block categories — `hateful-imagery`, `explicit-imagery`, and `graphic-violence` (three-way split ratified over the legacy two-axis offensive/nsfw grouping) — each release-subject, cid-required, `automated`+`reviewer` issuance, with `en` locales and a `policyVersion` bump; the change is additive and safe for `queryLabels` consumers, and the image adapter's severity-guidance sentence (which steered models away from rating content-policy violations critical) is rewritten to name them as block-worthy. CSAM and other unlawful content stays OUT of the label vocabulary: no automated classification (unreliable and legally fraught); suspected cases route through operator escalation to `!takedown`, and an out-of-band legal reporting process (NCMEC-style) is a separate pre-launch work item owned by the maintainer. + +Decision (2026-07-17, ratified from the severity-amended calibration sweep): **`undeclared-access` moves from an automated-BLOCK category to a WARNING** (`officialEffect` block→warn, `policyVersion` bump). Grounding: the sweep's only flagged code-lane FP was glm-5.2 blocking `clean-with-images` on a real `undeclared-access` finding (the fixture declares `storage: {}` but its code calls `ctx.storage.gallery.put`) — glm was correct, but the RUNTIME already neutralizes undeclared access: the sandbox throws `Storage collection not declared` (`packages/cloudflare/src/sandbox/bridge.ts:300`) and the equivalent network `allowedHosts` / capability gates reject undeclared network/content/media/users access, so an undeclared capability cannot execute — it is a manifest/behavior MISMATCH (the plugin's undeclared feature silently fails), not an executable threat. Genuinely malicious over-reach is already caught by the SPECIFIC malicious categories (`data-exfiltration`, `credential-harvesting`, etc.) which block on their own merits; `undeclared-access` is the residual "mismatch without other malicious signal", whose honest severity is a warning (surface the inaccuracy) not a hard block (hide from install surfaces). Hard-blocking a merely-sloppy manifest is over-enforcement. Additive-safe for `queryLabels` consumers (the label value is unchanged; only its effect softens). Consequence: the code-AI system prompt (data-driven from the policy's block∪warn category sets) now presents `undeclared-access` as a warning category, so the change must be re-validated by a confirming calibration sweep. Corpus fix folded in: `clean-with-images` genuinely declares its `gallery` storage so it is a valid clean baseline (matching its name); a dedicated `undeclared-access` fixture is a noted corpus gap for a follow-up. + +### `W8.6` Build calibration harness + +- Port legacy code/image audit fixtures. +- Add false-positive-sensitive legitimate security/network plugins. +- Record model outputs outside normal deterministic CI. +- Compare policy outcomes between model/prompt versions. +- Require explicit review of newly blocked/newly allowed fixtures. +- Produce a calibration report artifact for policy/model changes. + +Dependencies: `W8.2`, `W8.3`, `W8.5`. + +Decisions (2026-07-15): the harness is runnable locally now — not gated on staging infra. It drives the REAL adapters (`analyzeCode`/`analyzeImages` with their production prompts, response schemas, and the live policy) via a `RestAiBinding` satisfying the adapters' `AiBinding` interface against the Workers AI REST endpoint, authenticated with `wrangler auth token`; the model+prompt+schema combination is the unit under evaluation, never a hand-rolled eval prompt. Model matrix (all verified enabled on the account): code lane `@cf/moonshotai/kimi-k2.7-code` (current default) vs `@cf/moonshotai/kimi-k2.6` vs `@cf/zai-org/glm-5.2` (text-only — excluded from the image lane); image lane kimi-k2.7-code vs kimi-k2.6 vs `@cf/meta/llama-3.2-11b-vision-instruct`, with `@cf/moondream/moondream3.1-9B-A2B` as an experimental caption-only lens (Image-to-Text input shape — cannot flow through the real adapter, results not comparable). The legacy corpus is `packages/marketplace/tests/fixtures/audit/` (13 fixtures), ported into `apps/labeler/calibration/`; its expected.json files were baselined on retired models and are treated as priors for human review, not ground truth. Runs execute via a `calibrate` script outside the `test` CI gate; recorded run artifacts are gitignored review material, and the diff report (fixture × model outcome matrix, newly-blocked/newly-allowed sections, per-model agreement) is the calibration report artifact. Reasoning models spend output budget on `reasoning_content` before `content` (observed live on kimi-k2.6), so runs record finish_reason and token usage to make truncation diagnosable. + +Outcomes ratified from the first run (2026-07-15): the code-lane default model switches from `kimi-k2.7-code` (weakest measured: two under-blocked threats, 2× glm's latency) to `@cf/zai-org/glm-5.2`, to be re-validated by a follow-up sweep after the W8.5 severity amendment lands; `kimi-k2.6` is excluded from serving (72s mean latency, timeout, invalid structured output on 3 of 16 calls) but retained in the matrix as a detection-quality reference. The production adapters' `parseModelOutput` gains support for the OpenAI-compatible `{choices[].message.content}` envelope the live models actually return (as written it could parse none of them — the run's headline finding), folded into the harness PR; once production parses both envelopes the harness's transport unwrap is REMOVED so the eval exercises the true production parse path (adversary-flagged: the unwrap silently repaired, and never disclosed, the exact defect the harness exists to catch). The report additionally tallies over-warning on clean fixtures (passed→warned), which the FP count alone concealed — the run showed every model warning on every clean fixture, a prompt/threshold problem tracked for the same follow-up sweep. The Meta license for `llama-3.2-11b-vision-instruct` was accepted on the account (2026-07-15), unblocking the image lane. + +Prompt-calibration outcomes (2026-07-15, adversary-reviewed): the adapter system prompts are rewritten around a deception/hidden-behavior/unexpected-data-flow framing (empty findings is the stated common case; declared-capability-used-as-declared, ordinary code, and terse metadata are explicitly non-findings; declarations are anchored to the manifest section only — a host or capability named in file content, a comment, or a string literal is plugin data, not a declaration, so forged in-file declarations cannot suppress `undeclared-access`). Empirically validated with the harness: over-warn on the clean corpus dropped from every-model-every-fixture to 0–1 (the residual being a correct `undeclared-access` detection the fixture's legacy expectation missed), kimi false-blocks 3→0, all threat fixtures still blocked by all models, and two warn-positive tripwire fixtures (obfuscated-but-benign, undisclosed analytics beacon) guard against warning-suppression — v2 of the prompt was caught under-warning obfuscation by exactly this tripwire; the calibration expectation schema scores warn-expected fixtures (warned=agree, passed=under-warn, blocked=FP) with under-warns surfaced as their own tally. **Accepted residual (do not close in-prompt):** purpose-justified data flow is an irreducible content-analysis blind spot — a plugin whose stated purpose is "send content to our API" can exfiltrate by design and no content-only analyzer can distinguish it from a legitimate integration; deceptive variants remain caught (undisclosed destinations → misleading-metadata, over-collection → privacy-risk), and the honest variant must be defended out-of-band (host reputation, publisher trust/review — pre-launch gates). Known corpus gaps for future work: no split-across-files exfiltration fixture, no borderline-subtle malware, and warn-expected scoring compares outcome state only (category drift invisible). + +Gate 4 passes. A complete automated run can propose and issue the expected labels while preserving the signer boundary. + +Production-stage wiring (2026-07-17): the built analysis adapters are connected into the production orchestrator. **SLICED — the deploy gate does NOT lift yet.** Scoping found the `acquire` stage (the first stage; produces the bundle every later stage reads) has NO production consumer: `createAcquireStage` has zero non-test callers, and its SSRF-hardened egress deps (`resolveHostname`, `fetch`) and its `resolveTarget` (assessment → verified release → `AcquisitionTarget`) are never constructed in production. Without acquire wired, `holder.result` is never set, the AI stages no-op, and a prod run would sign `assessment-passed` for UNSCANNED subjects — exactly what the `import.meta.env.PROD` gate in `buildStages()` prevents. A second missing producer surfaced with the same shape: the IMAGE stage needs an image-metadata extractor (raw `bundle.files` bytes → `ImageAnalysisImage` with mime/sha256/base64/width/height/kind) that does not exist — symmetric to how acquire produces the decoded code `.files` (AcquiredArtifact documents "binary files (images) remain available on bundle.files"). It is bundle-production, so it folds into slice B with acquire; building `createImageAiStage` over a non-existent extractor would be a stub-in-disguise. The fault line is **inputs-that-exist-today vs bundle-production-completion**. So: **slice A (this)** wires only the two stages whose inputs already exist — `createCodeAiStage` (code `.files` + `declaredAccessToCapabilities`) and `createHistoryStage` (own D1 + the AGGREGATOR verification read) — plus the code-stage coverage seam, removal of the dead `deterministic`/`dependency` stages, the finalization-comment fix, and tests (with a test-injected acquire); `imageAi` stays a stub (deferred, not dead), and `buildStages`/the gate/`executeAssessmentInstance` are UNTOUCHED. **Declared-access mapping decision:** `createCodeAiStage` passes `declaredAccessToCapabilities(bundle.declaredAccess).capabilities` — CAPABILITIES-ONLY, allowedHosts excluded — matching the calibration input shape (the fixture loader passes `manifest.capabilities` and drops `allowedHosts`; the model avoids `undeclared-access` via the declared capability, not the host, and the runtime enforces allowedHosts so host-level access is not a moderation signal). **Calibration-fidelity flag:** the production declared vocabulary from `declaredAccessToCapabilities` (`content:read`, `network:request`) differs from the retired fixtures' legacy `manifest.capabilities` vocabulary (`read:content`, `network:fetch`); the planned post-severity-amendment calibration sweep MUST re-validate the model against the production vocabulary + the capabilities-only input before enforcement launch. **slice B (next, security-critical, needs its own design + ratification)** completes bundle production: the acquire consumer — release-resolution + the SSRF-hardened egress (`resolveHostname`/`fetch` implementation; approach TBD: DoH resolver vs egress-proxy binding) — AND the deterministic image-metadata extractor, then `createImageAiStage` (thin, over the extended holder), assembles all four stages in `buildStages(env, holder, config, policy)`, wires `executeAssessmentInstance`, and lifts the gate. Only when a real acquire runs is it safe to lift. Adversary items carried into slice B: (1) **thread the release record's description into `CodeAiStageOptions`** — the bundle manifest has no name/description, so the code stage currently passes `description: ""`, leaving misleading-metadata/privacy-risk (both rooted in stated purpose) near-inert; the release record is in hand during acquire, so slice B threads its description through, and the calibration sweep must run against the real-description input (flagged as the second calibration-input divergence in the stage's comment); (2) once a post-codeAi stage can transient-exhaust (the image stage), an `error` finalization can carry `coverage.code: "complete"` from the earlier successful code pass — decide then whether that's the intended "describes what WAS analyzed" semantics or should be cleared on error. + +Slice-B design (2026-07-17, SSRF approach ratified): **`resolveHostname` = the existing `cloudflareDohResolver`** (`packages/core/src/security/ssrf.ts` — DoH against `cloudflare-dns.com/dns-query`, A+AAAA, fails closed, structurally assignable to registry-verification's `HostnameResolver`), injected into the already-hardened `fetchVerifiedResource` (which enforces HTTPS-only, credential-free URLs, per-hop redirect re-validation with per-hop re-resolution, private/reserved-IP rejection, and byte/time budgets — `ACQUISITION_FETCH_LIMITS`). The **egress-proxy Worker option was considered and rejected** for v1: it is the only complete TTL-0-rebinding defense, but it is a net-new security-critical deploy surface, and the residual is doubly bounded here — the labeler is a top-level Worker on Cloudflare's edge with no private network behind it, and fetched bytes are checksum/CID-verified against the signed release before any use, so the worst case is a blind GET (the same accepted residual `ssrf.ts` documents). Lexical-only resolution (the sandbox bridge's approach) fails the ratified "DNS hardening" control and is prohibited — the sandbox is only safe lexically because of `allowedHosts` + `globalOutbound`, neither of which applies to the labeler's arbitrary declared hosts. **Release resolution (`resolveTarget`):** the declared artifact URL is NOT stored labeler-side (the assessment row keeps only pinned `artifact_id`/`artifact_checksum`; discovery-consumer does not persist the URL) — it lives in the aggregator `ReleaseView.release` artifacts blob, so `resolveTarget` reads over the existing `AGGREGATOR` service binding (the ratified W8.4 slice-3 read-path pattern): `getLatestRelease` fast path, falling back to a `listReleases` scan for the release matching the assessment's pinned `(uri, cid)` since no by-cid ReleaseView endpoint exists; the existing `crossCheckCoordinates` guards declared-vs-pinned drift, a release absent from the aggregator classifies as the existing transient acquisition failure (aggregator lag; reconciliation owns true deletions). Slice B also builds the deterministic image-metadata extractor (bundle bytes → `ImageAnalysisImage`: mime sniff, sha256, base64, PNG/JPEG/WebP/GIF dimensions, icon-vs-screenshot classification) + `createImageAiStage`, threads the release description into the code stage (adversary item 1; sourced from the package PROFILE via `getPackage`, not the ReleaseView — best-effort, never fails acquisition), assembles all four stages in `buildStages`, wires `executeAssessmentInstance`, and lifts the PROD gate — the gate-lift is valid ONLY with a real acquire running. Adversary pass (2026-07-17) confirmed the gate-lift core invariant HOLDS (no path to a false `passed`) and the SSRF main path HOLDS; two MED fixes folded in: **(M1)** a later stage's transient-exhaustion must not discard a block a prior stage already found — the orchestrator, on transient exhaustion, resolves the accumulated findings and finalizes `blocked` if they already block (a confirmed block is monotonic; no later stage can lift it), else `error` (incomplete analysis never concludes `passed`/`warned`); without this, a plugin pairing block-worthy code with a vision-model-breaking image escapes the block into `error` and stays installable across reconciliation reruns. **(M2)** `registry-verification`'s `fetch.ts isForbiddenAddress` is extended to reject IPv4-compatible (`::7f00:1`) and NAT64 (`64:ff9b::`) private/metadata encodings it previously missed (parity with core's `isPrivateIp`); strictly more-restrictive, additive, changeset'd — closes an SSRF gap the labeler's DoH-resolver composition exposes and that self-hosted Node egress (no platform firewall backstop) needs. L1: automated-label provenance (`model_id`/`prompt_hash` provisional) is on the calibration-launch checklist, not a gate blocker. Three `create*Stage` wrappers follow the `createAcquireStage(options)` closure idiom — each captures its deps + the shared `AcquisitionHolder` at construction and returns a `StageAdapter` that reads `holder.result` (the acquired, decoded `{path,content}` file map + parsed `declaredAccess`) and `ctx.assessment`: `createCodeAiStage` builds `CodeAnalysisInput` and calls `analyzeCode(input, {ai: env.AI, policy, promptVersion})`; `createImageAiStage` builds `ImageAnalysisInput` from the bundle's binary files and calls `analyzeImages(input, {ai: env.AI, ...})`; `createHistoryStage` closes over `env.DB` + a `PublisherVerificationReader` over the `AGGREGATOR` service binding and calls `analyzeHistory(db, assessment, opts)`. `buildStages(env, holder, policy)` assembles them; all required bindings (`AI`, `AGGREGATOR`, `DB`, `EVIDENCE`) already exist in `wrangler.jsonc` — no new binding. The AI adapters' `coverage` (`complete`/`partial`) + `droppedFiles` are merged into the assessment's `coverage_json` (already an updatable field via `updateAssessment.coverageJson`). The dead `deterministic` and `dependency` stages are removed from `OrchestratorStages`/`STAGE_ORDER`/`stubStages` (per the analysis-scope cut above); the W7.4 integrity/metadata checks remain in the acquire stage. **The finalization label-leak window the orchestrator comment flagged for "the real-stage wiring" is ALREADY closed** — #2072 gates every issuance statement on `requireAssessmentState: toState` with the whole batch all-or-nothing against the run→toState CAS and a signing-free currency re-check; the stale comment is updated, no new guard needed. Best-effort contract preserved: `analyzeHistory` swallows its own errors (context-only, never a label); a code/image adapter `ModelTransientError` propagates as a `StageTransientError` so the run retries rather than finalizing on a flaky model call. + +## Workstream W9: Operator Authentication and Console + +### `W9.1` Implement Access JWT verification + +- Verify `Cf-Access-Jwt-Assertion` using remote JWKS, exact issuer/audience, expiry, and not-before. +- Cache JWKS safely. +- Map ratified reviewer/admin claims. +- Distinguish human and service-token identities. +- Reject spoofed raw identity headers. + +Dependencies: `W2.1`. + +### `W9.2` Implement mutation protections + +Decision (2026-07-13, ratified with spec §12/§14.4 updates): a single composable guard (`guardMutation` -> `commitMutation`) every W9.4-W9.6 endpoint routes through. CSRF is a required `X-EmDash-Request: 1` header plus JSON content-type plus origin checks — no token/double-submit. The operator audit table is `operator_actions` (distinct from the signing-layer `issuance_actions`), append-only via triggers, `subject_uri` nullable for subject-less admin actions, action vocabulary enforced as a typed union in the guard rather than a SQL `CHECK`. Idempotency keys are globally unique; replay with an identical request fingerprint returns the stored result, a different fingerprint returns 409. The audit row commits in the same `db.batch` as the effect. + +- Same-origin validation (`Origin`/`Sec-Fetch-Site` when present). +- Required custom header (`X-EmDash-Request: 1`). +- JSON content type. +- Idempotency key with fingerprint-based replay/conflict semantics. +- Fresh role evaluation (per-request `verifyAccessRequest` + `hasRole`). +- Required reason. +- Immutable action audit (`operator_actions`). + +Dependencies: `W9.1`, `W2.3`. + +### `W9.3` Build console shell and read-only views + +Decision (2026-07-13): the console is a Vite React SPA colocated in `apps/labeler`, served as Workers static assets from the labeler Worker under `/admin`, calling same-origin `/admin/api/*`. It mirrors `packages/admin`'s conventions (Kumo, RTL-safe classes, TanStack Router where routing is needed) without depending on that package; copy thin pieces rather than extracting shared packages preemptively. No Lingui — plain English strings (internal-infrastructure decision, see Constraints). + +Routes: + +- Dashboard. +- Assessment list/detail. +- Subject history. +- Audit log. +- System status. + +Use Kumo, Lingui, RTL-safe classes, accessible tables/dialogs, and server APIs that never serialize private evidence to public routes. + +Dependencies: `W2.3`, `W9.1`, static `W8.1` fixtures. + +### `W9.4` Implement reviewer label actions + +- Issue/retract descriptive release labels. +- Issue/retract `security-yanked`. +- Issue/retract `package-disputed`. +- Require exact CID/URI-wide confirmation as appropriate. +- Show resulting official-client effect before submit. + +Decisions (2026-07-13): the moderation policy grants `reviewer` issuance on descriptive release labels (policy version bump; a reviewer-issued label is human-retract-only per §10). The typed confirmation is server-validated (exact CID for CID-bound actions, rkey for URI-wide) and participates in the idempotency fingerprint. Caller identity/roles for cosmetic UI gating come from a new `/admin/api/whoami` read. Retraction is manual negation through the same issuance path. The effect preview is a server-derived read grounded in the aggregator evaluator — the console holds no policy logic. + +Dependencies: `W3.3`, `W5.1`, `W9.2`. + +### `W9.5` Implement rerun and false-positive override + +- Rerun creates an immutable operator trigger ID. +- Unblock is one transaction/action that negates selected automated blocking labels and issues `assessment-passed` plus `assessment-overridden` for exact URI + CID. +- New automated findings remain visible but cannot remove the override. +- Reviewer/admin can explicitly retract the override. +- Carried from W9.4: manually-issued labels (`assessment_id` NULL, keyed to subject URI+CID) aren't visible in the console — `getAllLabelsForAssessment` is assessment-scoped. Add a subject+CID label read (`GET /admin/api/subjects/:uri/labels?cid=`, wrapping `getActiveLabelState`) and surface manual labels on the subject view and merged into the assessment detail page. Override/rerun needs the same subject-label read, so it lands here. + +Decisions (2026-07-13): rerun mints the operator trigger + fresh run + re-issues `assessment-pending` (terminal verdict waits on the W7/W8 orchestrator wiring — not pulled forward). Override negates ALL live automated blocks for the exact URI+CID (client submits the observed set, server validates against live state), never an operator-chosen subset. The multi-label override commits as one `commitMutation` batch (audit row + N negations + `assessment-passed` + `assessment-overridden`) under one operator action, each issuance piece keyed `${actionId}:${val}:${neg}`; `assertIssuancePersisted` verifies all N+2 landed. Override-retract negates only `assessment-passed`/`assessment-overridden` and leaves the blocks negated → evaluator resolves to blocked/`missing-assessment-pass` (safe default); re-surfacing real findings is "retract then rerun" (re-exposing blocks would corrupt automated provenance under §10). Override-permanence needs no evaluator change — `evaluateReleaseModerationCore` already suppresses current and future automated blocks when the pass+override pair is active. + +Dependencies: `W6.6`, `W9.4`. + +### `W9.6` Implement admin-only emergency actions + +- `!takedown` on release/package/publisher. +- `publisher-compromised` on DID. +- High-friction typed confirmation. +- Emit an immediate high-severity operational event and notification outbox; `W11.3` configures deployment alerts from that event. +- Pause/resume automated issuance. +- Dead-letter retry/quarantine controls. + +Decisions (2026-07-13): `!takedown` is admin-retractable via the same guarded path (negation + full ceremony); the resting state after retraction is the subject's pre-takedown computed state — the never-negated automated labels re-expose, so a still-dangerous release falls back to its blocks, never to eligible. Publisher/package takedown does NOT cascade to per-release labels: one dominant URI-wide/DID label the evaluator honors globally (eliminates the partial-application surface). The automation kill-switch is global and gates Jetstream ingestion only — enforcement lives in the discovery consumer, never the signing layer, so reruns/takedowns/manual labels stay available mid-incident; the pause read fails closed (unreadable switch → retry, never issue). Enforcement mechanism is consumer-`retry()`: a long pause drains the discovery queue to the DLQ, recovered by resume + the DLQ-retry control (accepted trade at expected volume). Because the switch is checked in the consumer rather than the signing layer, any future automated issuance path (notably the W7/W8 orchestrator finalization) does NOT inherit the gate and must add its own `isAutomationPaused` check. Sub-decisions: publisher typed-confirmation identifier is the DID's final segment (no handle resolution); intent phrases are server constants; `operational_events` (append-only) is the alert source of truth and `notification_outbox` the delivery queue W11.3 drains; event/outbox INSERTs are gated on the label's in-batch existence so no alert can fire for a label suppression `assertIssuancePersisted` will 503. + +Post-review decisions (2026-07-14, after PR-1..PR-3 merged): an emergency retract with no matching active label is rejected (404/409), not accepted as a no-op that emits a spurious high-severity event — the retract endpoints pre-check that the positive label is live before signing. Left as-is: `operator_actions.action` stays shared across issue/retract with direction in `metadata.neg` (no new union members); a pause records its high-severity operational event only, no notification-outbox delivery row (a pause is alarm-visible via the event but is not itself a page-worthy incident); the concurrent two-admin same-dead-letter race keeps its current behavior (at-most-once enqueue holds; the loser's misleading 200 + info event is an accepted rare wrinkle, not worth a post-commit read or a frozen-file gate change). + +W9.6 delivered across PR-0 (#2024 operational-event/outbox subsystem + kill-switch mechanism), PR-1 (#2026 takedown/publisher-compromised + ceremony), PR-2 (#2027 pause/resume), PR-3 (#2028 DLQ retry/quarantine). + +Dependencies: `W9.2`, `W3.3`. + +### `W9.7` Add browser/accessibility/localization tests + +- Reviewer versus admin controls. +- Expired/revoked Access session. +- CSRF/idempotency. +- Confirmation ceremony. +- Keyboard navigation. +- Arabic RTL. +- Localized labels/operator strings. +- Public/private evidence separation. + +Dependencies: `W9.3` through `W9.6`. + +### W9 Completion + +Operators can handle the expected low-volume queue without direct database edits or unsafe generic signing APIs. + +## Workstream W10: Transparency and Notifications + +### `W10.1` Publish machine-readable policy + +Serve the versioned policy document with: + +- Labeler DID. +- Policy/schema versions. +- Supported subjects. +- Localized label definitions. +- Official effects. +- Public API/contact/reconsideration links. +- Model transparency statement. + +Dependencies: `W1.7`, `W3.1` identity. + +### `W10.2` Implement public assessment APIs + +- Immutable `getAssessment(id)`. +- Materialized `getCurrentAssessment(uri, cid, src?)`. +- Paginated `listAssessments` with bounded public filters. +- `getPolicy`. + +Current response includes completed automated assessment, newer pending run, active labels, and active manual override as separate concepts. + +Historical `getAssessment(id)` derives `state: "superseded"` when a newer completed assessment names it as `supersedesAssessmentId` and owns the current pointer. Pending/stale/cancelled runs never supersede the previous completed assessment. + +Dependencies: `W1.3`, `W2.3`, `W6.5`. + +### `W10.3` Implement public/private serializers + +Explicitly exclude source excerpts, raw prompts/responses, contacts, raw inference payloads, operator notes, and dangerous exploit detail. Add compile-time types and snapshot tests proving no private fields leak. + +Dependencies: `W2.5`, `W10.2`. + +### `W10.4` Implement contact resolution + +Resolve in order: + +1. Package security contact. +2. Package author/contact metadata. +3. Publisher profile contact. + +Do not access private PDS account email. Store delivery addresses only as long as needed and retain keyed hashes for audit/deduplication. + +Dependencies: registry profile/package reads. + +Decisions (2026-07-16): all three contact sources already exist as lexicon fields (package `security[]` — required; package `authors[]` — contact optional; publisher profile `contact[]` with security/general kinds) — no lexicon changes. Resolution walks the tiers for the first entry carrying a non-empty EMAIL (url-only contacts are skipped, not "resolved"), preferring `kind: security` within tier 3, with a fresh read at notification time via the W8.4 slice-3 aggregator service binding (no caching; no self-fetch/PDS path). Anti-abuse is DOUBLE OPT-IN: unverified publishers receive one content-neutral confirmation mail and substantive notices only after confirming (the public assessment API and the monitored reconsideration inbox remain the channel for unconfirmed publishers); verified publishers skip confirmation; universal suppression honors unsubscribe/"not me"/hard-failure, with per-address and per-DID rate limits on confirmation sends — a third-party victim named in hostile metadata receives at most one neutral mail, ever. No resolvable contact is an expected non-error terminal state (delivery marked undeliverable, kept for audit), with an operator `operational_event` added only for emergency-takedown notices that cannot be delivered. Addresses are keyed as HMAC-SHA256 over the lowercased/trimmed email with a secret pepper binding; plaintext lives only on the delivery row, cleared on successful send, undelivered rows swept after 30 days (versioned constant); audit/dedup query hashes only. The publisher-facing delivery table keeps the spec's `notifications` naming — the existing `notification_outbox` is the OPERATOR alert subsystem and stays untouched; W10.4 owns `notification_contacts` (confirm state, token hash, rate-limit timestamps) and `notification_suppressions`, plus the confirm/unsubscribe/not-me token endpoints. Noted for W10.5: spec §14.4 keys `notifications.action_id` to `operator_actions`, but automated blocks live in `issuance_actions` — the notification trigger needs a source covering automated issuance. + +Observed (2026-07-16, slice-A adversary): the aggregator's `isRedacted` gate on both `getPackage` and the new `getPublisher` reacts only to `!takedown`, so a subject carrying `publisher-compromised` (but not taken down) still serves its contact emails publicly. This is pre-existing parity with `getPackage`, not a slice-A regression, and is arguably intended (a compromised publisher's security contact should stay reachable). Recorded here as a confirm item for whenever aggregator redaction policy is next revisited — not blocking, no change made. + +### `W10.5` Implement notification outbox and email adapter + +Delivery uses Cloudflare Email Sending through the Workers `send_email` binding; the sending domain is onboarded on the emdashcms.com zone (`wrangler email sending enable`). No provider API keys. Messages include both `html` and `text` bodies. + +Verified mechanics (2026-07-16, against live Cloudflare docs — Email Sending is a distinct public-beta product, Workers Paid only, separate from Email Routing's verified-destination-only sends): ARBITRARY external recipients are supported once the domain is onboarded (the load-bearing feasibility fact; the bare `{"send_email":[{"name":"EMAIL"}]}` binding form with no destination restriction is the correct one — `destination_address`/`allowed_destination_addresses` are the old verified-only model). The structured `env.EMAIL.send({to, from, subject, html, text, headers})` API supersedes raw-MIME `EmailMessage`; multipart html+text is automatic; `from` must be on the onboarded domain; success returns `response.messageId` → our `provider_id`. Failures THROW typed errors with `.code`: retryable (`E_RATE_LIMIT_EXCEEDED`, `E_DELIVERY_FAILED`, `E_INTERNAL_SERVER_ERROR`), quota (`E_DAILY_LIMIT_EXCEEDED`), and terminal — notably `E_RECIPIENT_SUPPRESSED` (Cloudflare account-level hard-bounce/complaint suppression): the adapter MUST map that to our own `suppress(reason:'bounce')` + undeliverable, never retry it. Set `List-Unsubscribe: ` + `List-Unsubscribe-Post: List-Unsubscribe=One-Click` headers on every send (allowlisted + validated; Gmail/Yahoo bulk-sender requirement). Limits: 5 MiB/message, 50 recipients, 3k mails/mo included then $0.35/1k, daily quota auto-scales. Transactional-only ToS — our action-triggered notices qualify. Testing: miniflare/vitest-pool-workers natively simulates the binding (no stub needed; verify with a quick spike — binary attachments don't serialize locally, irrelevant here). OPERATOR-OWNED go-live prereq: one-time Email Sending onboarding of emdashcms.com (dashboard or `wrangler email sending enable`; auto-managed SPF/DKIM/DMARC on a cf-bounce subdomain, additive alongside any existing Email Routing config, 5-15 min propagation). + +Notify on block, warning, override, retraction, prolonged error, and emergency takedown. Include public summary/effect/reconsideration URL without private details. Delivery retries independently and never rolls back a label. + +Hard requirement (2026-07-16, from the W10.4 slice-C adversary pass): the confirm token this slice generates MUST come from a CSPRNG with ≥128 bits of entropy. The confirm endpoint stores `SHA-256(token)` without a pepper (unlike the recipient hash, which is HMAC-peppered because email is low-entropy) and applies no in-worker rate limit, so the token's unguessability is the sole protection against a brute-force confirm of a contact whose recipient hash leaked. A weak or predictable token would make double opt-in bypassable through the public endpoint. Do not reuse a ULID or any monotonic/timestamped id as the token. + +Send-path contracts (2026-07-16, from the two W10.4 slice-A adversary passes): the slice-A `seedPublisherContact` returns `{seeded:true}` even when the contact already exists in a `confirmed` or `declined` state (`ensureContact` is insert-if-absent, so it no-ops), and it does not — cannot — apply send rate limits because there is no send path yet. So this slice MUST: (1) gate every send on the confirm-state machine (`canSendConfirm`/`confirmState`), NEVER on `seeded:true` — a send keyed off `seeded:true` would re-mail confirmed recipients and, worse, recipients who explicitly said "not me" (`declined`); (2) enforce per-address AND per-DID rate limits on confirmation sends, the only thing preventing hostile metadata naming `victim@x` from turning double opt-in into an email-bomb amplifier; (3) re-check suppression atomically at send time rather than trusting the seeded row, since slice-A's seed-time `isSuppressed`→`ensureContact` is check-then-act and a suppression can land in the gap. + +Decisions (2026-07-16, ratified): **Source reference** — the new `notifications` delivery table references its trigger via a POLYMORPHIC `source_type` ('issuance' | 'operator') + `source_id`, with NO cross-table SQL FK (SQLite can't FK-to-one-of-two, and the writer enforces integrity). This deviates from spec §14.4's literal `action_id → operator_actions(id)`, which predates the ratified issuance/operator audit split: automated block/warning/retraction live in `issuance_actions`, operator override/emergency in `operator_actions`. Synthesizing system-attributed `operator_actions` rows was rejected — it pollutes the operator audit log with non-operator events. **Prolonged-error deferred**: W10.5 wires the five events with clear hook points (new block, new warning, override, retraction, emergency takedown); the sixth spec event — "assessment error unresolved past the operational target" — has no code representation (no stuck-run detection; reconciliation doesn't surface it), so it waits on a follow-up ticket that builds stuck-run detection into the reconciliation cron. Blocked publishers retain the public assessment API + reconsideration inbox meanwhile. + +Sliced (2026-07-16): **slice 1** — migration `0008` (`notifications` delivery table: polymorphic source, `kind` confirmation|notice, `state` pending/sent/failed/undeliverable, `attempts`/`last_error`/`provider_id`, nullable `plaintext_email` cleared on successful send, `recipient_hash`, timestamps) + CSPRNG confirm-token generation (base64url ≥128 bits from `crypto.getRandomValues`, hashed via the existing `hashConfirmToken`, persisted via the existing `recordConfirmSent`) + the gated send-orchestration CORE against an injected `NotificationSender` interface (no real email adapter yet, tested with a fake sender). The core is the security surface: fresh contact resolve → `recipientHash` → ATOMIC suppression re-check at send → confirm-state gate (`getContactState`/`canSendConfirm`, NEVER `seeded:true`): `confirmed` → substantive notice; `unconfirmed` → only if per-address (`last_confirm_sent_at_epoch_ms`) AND per-DID rate limits both pass → generate+record token + send confirmation mail (never a substantive notice); `declined`/`suppressed`/no-contact → mark undeliverable, no send. Per-DID confirmation rate limit needs its own queryable ledger keyed by publisher DID (count distinct-recipient confirmation sends in a window) — a malicious DID naming many `victim@x` is the threat per-address limits don't cover. STRICT double opt-in for ALL publishers in slice 1 (verified-publisher-skip-confirmation deferred to slice 2 — sending a content-neutral confirmation mail to a verified publisher is safe/more conservative, just higher friction). **slice 2** — the Cloudflare Email Sending adapter (`send_email` binding + wrangler config + vitest stub + `EmailMessage` with html+text bodies + confirmation-mail and substantive-notice templates carrying subject/label/public summary/assessment URL/effect/reconsideration, no private detail) implementing `NotificationSender`, + trigger wiring at the five issuance points (in-batch-guarded via the `gateOnIssuedLabelActionKey`/`requireAssessmentState` idiom so a signing-suppressed batch queues no dangling notification row), + verified-publisher-skip-confirmation via the W8.4 slice-3 `getPublisherVerification` read, + the cron retry sweep and 30-day undeliverable-row cleanup as a third `ctx.waitUntil` branch in `scheduled()`. + +Prolonged-error follow-up (2026-07-17, ratified): the deferred sixth event is now designed as a TWO-STAGE escalation ladder driven by the reconciliation cron. **Threshold + audience (Matt's decision): 24h → operator alert; 72h → publisher notice if still unresolved.** An assessment in the terminal `error` state (retries already exhausted; `completed_at` stamped; `assessment-error` label issued; `error` has no exit transition and does not move the current-assessment pointer) that is NOT superseded — no newer run exists for the same `(uri, cid)` — escalates: at `PROLONGED_ERROR_OPERATOR_THRESHOLD_MS` (24h past `completed_at_epoch_ms`) a `assessment-prolonged-error` operational_event (severity `high`, `subject_uri` = assessment uri, `assessmentId` in payload, null `action_id`) is raised so operators can triage an infra-vs-publisher cause before the publisher is emailed; at `PROLONGED_ERROR_PUBLISHER_THRESHOLD_MS` (72h) — only if the error is still the live unsuperseded run — the publisher notice fires. Rationale for the ladder: an `error` is the labeler FAILING to assess (often our infra, sometimes a publisher-fixable artifact URL), so operators get a chance to self-heal before we email the publisher about our own failure. **Fire-once mechanism:** a small tracking table `assessment_error_escalations` (migration `0010`; `assessment_id` PK, `subject_uri`, `subject_cid`, nullable `operator_alerted_at_epoch_ms` / `publisher_notified_at_epoch_ms`, `created_at_epoch_ms`) makes each stage idempotent across the 5-minute cron ticks — the cron upserts the row, alerts once when `operator_alerted_at` is null, notifies once when past 72h and `publisher_notified_at` is null. **Source reuse (avoids a `notifications.source_type` CHECK rebuild):** the publisher notice reuses `source_type = 'issuance'`, `source_id = ` — an errored assessment never produces a block/warn notice, so the `(issuance, id)` key never collides with a finalization notice; a SEPARATE `prolongedErrorNoticeContent` builder is added, and `resolveNoticeForSource`'s issuance branch routes by loaded assessment state (`blocked`/`warned` → `assessmentNoticeContent`; `error` → `prolongedErrorNoticeContent`; else null). The finalization path's `assessmentNoticeContent` KEEPS returning null for `error` so `notifyAssessmentOutcome` never fires on error — only the cron's 72h path notifies. `assessment-prolonged-error` is an additive `OperationalEventType` member. Double-opt-in + suppression gating and `contactTargetFromUri(assessment.uri)` are reused unchanged, so an unconfirmed publisher gets only the neutral confirmation mail. Migration-order dependency: `0010` must land after the reconsideration `0009` (#2083), so this PR merges after #2083. Adversary pass (2026-07-17): two MED fixes landed — (1) the escalation scan LEFT JOINs `assessment_error_escalations` and keeps only rows with work left, so a fully-escalated row drops out of the `LIMIT 200` window (otherwise a >200-error backlog starves newer errors indefinitely); (2) the operator-alert event insert is gated on the escalation still being un-alerted (Cloudflare does not serialize a cron trigger against its own next tick, so an unconditional insert could double-alert under overlap). **Finding 2 accepted (Matt's decision): the backlog/rollout head-start collapse stands** — an error first seen already past 72h fires the operator alert and the publisher notice in the same tick (no 24h→72h triage window); steady-state fresh errors keep the full head-start. Not enforcing a since-alert head-start keeps the two thresholds as the literal spec ("72h from error") and avoids rows lingering in the scan window. + +Slice-1 outcome (2026-07-16, ratified after the send-core adversary pass): **the confirmation mail is a true LIFETIME cap — at most one per address, EVER**, not one-per-interval (Matt's decision; the earlier "at most one, ever" wording is now literally enforced, superseding the per-address interval gate). Mechanism: the confirmation-path claim is a single atomic `INSERT ... SELECT ... WHERE NOT EXISTS (suppression) AND NOT EXISTS (prior non-undeliverable kind='confirmation' row for this recipient_hash)`, which enforces not-suppressed + lifetime-cap + concurrency (a racing second send's guard sees the first's row and inserts nothing) in one statement. `state != 'undeliverable'` is deliberate so a prior that never actually mailed (race-loser/suppressed-at-claim) does not block a later legitimate confirmation. Consequence: a persistently-named victim receives at most one content-neutral confirmation mail total, and one click of "not me" (→ declined+suppressed) stops everything permanently; an unconfirmed legit publisher who ignores that one mail is never re-nudged and keeps the public assessment API + reconsideration inbox as their channel (the double-opt-in bargain). The per-DID confirmation ledger cap is BEST-EFFORT under concurrency (count→check→write is not atomic — bounded overshoot admits more distinct victims but never more than one mail per victim, since the per-address lifetime cap is the hard bound); exact per-DID serialization needs a Durable Object and is deferred to slice 2. Slice-2 contracts recorded from the adversary pass: (a) the `NotificationSender` implementation MUST NOT put the confirm URL or raw token into its returned `error` string — it is persisted verbatim in `notifications.last_error` and never cleared, so a token there would leak to the DB; (b) a `failed` confirmation row retains `plaintext_email` but only the token HASH (not the raw token), so the slice-2 retry sweep must REGENERATE the token when re-sending a failed confirmation, not attempt to reuse stored data; (c) because `pending` and `failed` confirmation rows HOLD the lifetime cap, a crash-stuck `pending` row or an exhausted-retry `failed` row permanently forecloses the email channel for that address unless the sweep handles them — the slice-2 sweep MUST re-drive stuck `pending` rows (not just `failed`), and when it terminally abandons a row it MUST flip it to `undeliverable` (which re-opens the cap for a later legitimate send) rather than leaving a dead `failed` row as a silent permanent lockout. + +Dependencies: `W2.3`, `W10.3`, `W10.4`. + +### `W10.6` Implement reconsideration workflow + +- Publish monitored contact instructions. +- Accept opaque assessment ID in communication. +- Let operator attach private reconsideration notes. +- Record rerun/override/outcome as auditable actions. +- Send outcome notice. + +No authenticated appeals portal or SLA is introduced. + +Dependencies: `W9.5`, `W10.5`. + +Decisions (2026-07-16, ratified): **Requirements 1 and 2 are already satisfied** — the "monitored contact instructions" are published machine-readably today (`/.well-known/emdash-labeler-policy.json` serializes the whole `moderation-policy.json`, including the `contact` block with `reconsiderationUrl` + `reconsiderationEmail`), and the "opaque assessment ID" a publisher quotes is the existing public `asmt_` (the `assessments.id` PK, emitted by `toPublicAssessment` and in every notice's `assessmentUrl`). No new public/XRPC surface is built; the prose instructions page lives externally on emdashcms.com and is operator-owned. W10.6's new code is purely the operator-side CASE MANAGEMENT plus the outcome notice. + +**Case model.** New mutable table `reconsiderations` keyed on subject `uri` + `cid` (STABLE across reruns — a reconsideration is about "this release," and reruns mint fresh `asmt_` ids), with a `triggering_assessment_id` reference for context. State `open` → `resolved`. At most one `open` case per (`uri`,`cid`) via a partial unique index; after resolution a fresh case may be opened. Records `opened_by`/`opened_at` and `resolved_by`/`resolved_at` actor+timestamp provenance and the resolution `outcome`. Private operator notes go in an append-only, immutable child table `reconsideration_notes` (id, `reconsideration_id`, actor fields, note text, `created_at`) — NOT on `operator_actions` (immutable but semi-public: its `reason` leaks into notices/events) — matching the codebase's append-only-audit ethos. Next migration is `0009_reconsiderations.sql`. + +**Outcome vocabulary.** `granted` (assessment changed in the publisher's favour — the operator separately runs the existing rerun/override/override-retract actions, which carry their own audit + notices; resolving records that the request was granted), `denied` (assessment upheld, no label change — the ONLY outcome for which nothing else notifies), `withdrawn` (requester withdrew / duplicate / moot — no notice). + +**Operator actions** (all via `guardMutation`/`commitMutation`, adding purely-additive `OperatorActionType` union members — no hand-rolled auth/CSRF/idempotency): `reconsideration-open` (reviewer; creates the case + an initial private note, emits a `reconsideration-opened` `operational_event`), `reconsideration-note` (reviewer; appends a private note), `reconsideration-resolve` (reviewer; sets `outcome`+state, emits `reconsideration-resolved` event, and — per the ratified notice decision — fires the outcome notice). Resolve issues NO label (closest existing analog is `runDeadLetterAction`: a status UPDATE guarded by current-state so a double-resolve is a 409, plus an ungated `operational_event` in the same batch). Reruns/overrides are NOT re-implemented — the operator uses the existing actions and the case's private notes/outcome tie them together for audit. + +**Outcome notice — "Pipeline notice on every resolve" (Matt's decision).** A `granted` or `denied` resolve fires a new `reconsideration-outcome` notice through the existing W10.5 double-opt-in pipeline, `source_type: 'operator'`, `source_id`: the resolve `operator_action` id — so dedup/replay-safety key on the resolve action and the sweep can re-render it. Requires: a `notifyReconsiderationOutcome` trigger + `reconsiderationOutcomeNoticeContent` builder in `notification-triggers.ts` (public summary + assessment URL + outcome + reconsideration URL, NO private detail — the private notes NEVER enter notice copy), a `deferReconsiderationNotify` helper called post-commit, and a `reconsideration-resolve` case in `resolveNoticeForSource` (else the retry sweep abandons the row). `withdrawn` fires no notice. On a `granted` outcome this notice layers on top of any override/retract notice — accepted: they carry distinct messages ("your block was lifted" vs "your reconsideration request was granted"), and both are double-opt-in + suppression gated like every substantive notice. New `operational_event` types `reconsideration-opened`/`reconsideration-resolved` (additive `OperationalEventType` members, severity `info`). + +**Console UI.** Case surface attaches to `AssessmentDetail.tsx` (the action hub) following the existing dialog-per-action + `invalidateKeys` pattern: an "Open reconsideration" / "Add note" / "Resolve" block showing the open case, its private notes, and the outcome. Localized + RTL-safe. (Console is not yet localized per W10.3 status; match the surrounding console conventions.) + +### `W10.7` Decide/implement optional ATProto decision notice + +Decision (2026-07-17, ratified — DROPPED): the optional ATProto decision-notice record is NOT built. It was always conditional ("only if `W0.1` ratifies it"), and it is declined: it has no known consumer, and everything it would offer already exists — labels distribute over the standard `com.atproto.label` subscription (built), publisher notification is the W10.5 double-opt-in email pipeline plus the reconsideration inbox, and public lookup is the current/historical assessment API. Publishing human-decision-level records as durable, replicated, subscribable repo records would add a broadcast surface (amplifying every block into a network event, with retraction/reach and right-to-be-forgotten concerns) for no demand. If a concrete transparency-feed consumer emerges later, revisit via a Discussion. Email remains the active notification channel; the label stream remains the transparency channel. + +Superseded requirements (not built): + +- ~~Publish a repository record referencing subject and public assessment.~~ +- ~~Make clear it is subscribable transparency, not guaranteed delivery.~~ +- ~~Keep email as the active notification channel.~~ (Email is the active channel regardless.) + +Dependencies: `W1.2`, `W3.1`, `W10.2`. + +### W10 Completion + +Gate 5 transparency/notification requirements pass without exposing private evidence. + +## Workstream W11: Operations and Self-Hosting + +### `W11.1` Harden production deployment environments + +Extend the minimal staging signer environment from `W3.1` into complete staging/production Wrangler configuration for: + +- Custom domain and routes. +- D1. +- R2. +- Queues/DLQs. +- Workflow. +- Durable Objects and migrations. +- Workers AI (`AI` binding). +- Secrets Store. +- Access application/audience. +- Email sending (`send_email`) binding and onboarded sending domain. +- Cron triggers. + +Generate Worker types; do not hand-maintain `Env`. + +Dependencies: `W0` platform choices, `W2.1`. + +### `W11.2` Add CI and deployment safety + +- Build/typecheck/test `apps/labeler` explicitly. +- Run real workerd/D1 migrations in CI. +- Validate generated lexicons/types are current. +- Block deploy on missing required bindings/secrets/policy version. +- Add staging smoke before production promotion. +- Keep signing-key provisioning outside repository/workflow logs. + +Dependencies: `W2.1`, `W3.1`. + +### `W11.3` Implement metrics, logs, and alerts + +Each component workstream emits its own typed counters/events without depending on `W11`. Aggregate those signals into deployment logs, dashboards, and alerts covering every signal in spec section 22, including Jetstream cursor age, stage latency, model failures/cost, issuance sequence, subscriber lag, overrides, notification failures, and DLQ age. + +Alerts include emergency actions, signing/DID mismatch, block spikes, stale pending work, and reconciliation gaps. + +Dependencies: component metrics from `W2` through `W10`. + +### `W11.4` Write key-management runbooks + +Decision (2026-07-17, ratified): the runbooks (#2100) surfaced that the signing-key rotation state machine (`signing-rotation.ts`: `beginRoutineKeyRotation`/`activateRoutineKeyRotation`/`abortRoutineKeyRotation`/`initializeSigningState`) is implemented + tested but has ZERO production callers. A CLI break-glass tool was scoped and then **DESCOPED (Matt, 2026-07-17)** after a feasibility pass proved `activate` fundamentally cannot run from a CLI: (1) `getPlatformProxy` remote bindings proxy resource _operations_, not secret _values_, and `LABEL_SIGNING_PRIVATE_KEY` is a write-only Worker secret that cannot be read back — so the only way to feed the CLI the key is local pasting (forbidden); (2) even with the key, a CLI-local signer check verifies the WRONG principal (it proves the CLI holds a key, not that the deployed Worker does), defeating activation's entire safety gate. Since `activate` must run inside the deployed Worker, a CLI could drive only begin/abort/status — it could never _complete_ a rotation — so a bespoke CLI for a rare event is not worth it. **Launch answer: rotation stays a documented MANUAL procedure** (runbooks.md § Routine rotation / Compromise response describe the begin → deploy key → activate-in-Worker → resume ceremony). Consistent with the step-back framing that the labeler is a conservative trust/transparency layer over the runtime sandbox, not a primary security control whose every operational edge needs bespoke tooling pre-launch. If rotation frequency ever warrants self-serve, the only correct `activate` interface is a machine-authenticated Worker endpoint (option (a) from the feasibility pass) that runs `activateRoutineKeyRotation` with the in-Worker signer — revisit then. + +- Begin from the ratified `W0.4` key-lifecycle contract and the implemented `W3.7` behavior. +- Initial generation and custody: the offline copy of the production scalar lives in the maintainer's Keeper vault. +- Routine rotation. +- Compromise response. +- Issuance pause/resume. +- Historical re-signing. +- Aggregator replay instructions. +- Audit evidence and incident communications. + +Dependencies: `W3.7`. + +### `W11.5` Implement backup, restore, and retention jobs + +- D1 backup/export. +- Evidence metadata and retained R2 objects. +- Restore validation. +- Retention deletion with active-block exceptions. +- Contact-data minimization. +- Inference log minimization (metadata only, no raw payloads retained). + +Dependencies: `W2.5`. + +### `W11.6` Write operational runbooks + +- Jetstream outage/cursor loss. +- Queue/DLQ recovery. +- Workflow stuck runs. +- AI outage. +- Mirror/fallback outage. +- Subscriber lag. +- Publisher reconsideration. +- Emergency takedown/retraction. +- Full aggregator replay. + +Dependencies: all owning workstreams provide failure semantics. + +### `W11.7` Document self-hosting and third-party consumption + +Document: + +- One-deployment-per-DID model. +- Required bindings and costs. +- DID/key/domain setup. +- Policy/model configuration boundaries. +- How a third-party aggregator subscribes and verifies. +- How clients choose accepted labelers. +- Which defaults are EmDash policy rather than protocol rules. + +Dependencies: public contracts stable through Gate 5. + +### W11 Completion + +The service can be deployed, observed, rotated, restored, and consumed without undocumented operator knowledge. + +## Workstream W12: Conformance and Security + +### `W12.1` Build protocol conformance suite + +Black-box tests for: + +- DID/key discovery. +- `queryLabels` pagination/filtering/signatures. +- `subscribeLabels` replay/live/negation. +- Cursor recovery. +- Key rotation. +- Policy and assessment APIs. +- Fresh third-party aggregator subscription. + +Dependencies: `W3`, `W4`, `W10`. + +### `W12.2` Build full registry integration suite + +Exercise: + +```text +publish release +-> Jetstream discovery +-> source verification +-> artifact acquisition +-> assessment +-> signed labels +-> aggregator ingest/hydration +-> admin/CLI eligibility +-> install or block +``` + +Cover pass, warning, block, pending, error, override, yank, takedown, new CID, deletion, and package/publisher cascade. + +Dependencies: `W4` through `W10`. + +### `W12.3` Build adversarial suite + +Cover every spec section 24.4 case plus: + +- Forged pending-label attempt from unverified Jetstream event. +- Model output selecting arbitrary labels. +- Manual pass without `assessment-overridden`. +- Override attempting to bypass publisher/package manual block. +- Concurrent assessment finalization/current-pointer races. +- Reassessment under changed model/policy version. +- Legacy `security:yanked` data preflight behavior. +- Malformed RFC-8941 accepted-labeler headers. + +Dependencies: relevant implementation workstreams. + +### `W12.4` Add query-count and performance checks + +- Prove label hydration is batched. +- Snapshot aggregator query counts for search/package/release/latest paths. +- Measure full-history replay and current-state rebuild. +- Exercise Queue bursts above expected launch volume. +- Confirm model/source limits avoid Worker resource failures. + +Dependencies: `W4.5`, `W6`, `W8`. + +### `W12.5` Run calibration and false-positive review + +- Run the production model/policy on historical/fixture corpus. +- Manually inspect every automated blocking result. +- Record false-positive/false-negative and override expectations. +- Freeze production model, prompt hash, and policy version. + +Dependencies: `W8.6`, `W9` review UI optional but preferred. + +### `W12.6` Run browser and accessibility conformance + +- Desktop/mobile operator console. +- Arabic RTL. +- Keyboard-only actions. +- Screen-reader labels/dialogs. +- Access role changes during session. +- Registry consumer warning/block/override displays. + +Dependencies: `W5.5`, `W9.7`. + +### `W12.7` Run operational drills + +- Routine key rotation. +- Compromised key. +- D1 restore. +- Lost Jetstream cursor. +- Full subscriber replay. +- Queue/DLQ recovery. +- AI outage. +- Mirror outage and safe fallback. +- Emergency takedown and retraction. +- Notification provider outage. + +Dependencies: `W11` runbooks and complete staging deployment. + +### `W12.8` External security review and closure + +Review focus: + +- Signing-key boundary. +- Label signing/verification. +- Artifact fetch/extraction. +- Prompt injection/model authority. +- Access authentication/authorization. +- Operator action and override semantics. +- Public/private evidence separation. +- Aggregator/client enforcement consistency. + +All critical/high findings are fixed and retested before Gate 6. + +Dependencies: feature-complete staging system. + +### W12 Completion + +Gate 6 passes and production positive-assessment enforcement may be enabled. + +## Parallelization Guide + +After Gate 0: + +| Parallel lane | Can proceed | Must wait for | +| ----------------- | ---------------------------------------------------------------- | ------------------------------------------------------------ | +| Protocol | `W1.2` lexicons, `W1.4` crypto, `W1.6` headers | Ratified identifiers/vectors | +| Service core | `W2.1` scaffold, migration drafting | Stable schema types for final migration | +| Artifact security | `W7.1` ownership, `W7.2`/`W7.3` primitives | Mirror contract and shared-package decision | +| Operations | Minimal staging signer bindings in `W3.1`, CI skeleton in `W2.1` | Full production hardening waits for final secrets/key format | +| Aggregator | Subscription/hydration design and fixture tests | `W1` contracts; live connection waits for `W3` | +| Client/UI | Static eligibility fixtures and UI states | Runtime integration waits for `W4`/`W10` | + +After Gate 2: + +- `W4` aggregator ingestion, `W6` verified discovery, `W9` auth/shell, and `W10` policy/public API can proceed concurrently. +- `W5` registry-client/core/CLI adapters can integrate the `W1.5` evaluator before live hydration; server enforcement waits for `W4.6`. +- `W8` adapters can develop against recorded validated-bundle fixtures while `W6` orchestration is built. + +After Gate 4: + +- `W9` mutation flows, `W10` notifications, `W11` complete operations, and `W12` integration/adversarial suites proceed concurrently. + +## Recommended Merge Sequence + +Each item should remain independently reviewable and leave production behavior safe by default. + +1. Gate 0 fixtures and decision records. +2. Vocabulary cutover, shared policy fixture corpus, and the single shared moderation evaluator. +3. Labeler lexicons/generated types. +4. Shared label crypto and header parsing. +5. `apps/labeler` scaffold, migrations, repositories, and CI coverage. +6. Typed signer, `queryLabels`, and signing vectors. +7. Subscriber DO and `subscribeLabels`. +8. Aggregator subscription, verification, and current-state projection. +9. Aggregator header parsing, batched hydration, and endpoint filtering using the shared evaluator. +10. Registry-client evaluator adapter and server install/update enforcement behind a disabled production flag. +11. Admin/CLI consumer UI states against fixtures. +12. Jetstream discovery and source-record verification. +13. Shared artifact acquisition and canonical bundle validation. +14. Deterministic and capability analysis. +15. AI/image adapters, policy resolver, and calibration harness. +16. Automated assessment finalization and current-assessment projection. +17. Access authentication and read-only operator console. +18. Reviewer/admin mutations, overrides, and emergency actions. +19. Public assessment API, policy document, email, and reconsideration flow. +20. Observability, backups, rotation, self-hosting docs, and runbooks. +21. Conformance/adversarial/security closure. +22. First-party catalog assessment and staged production enforcement enablement. + +## Cross-Workstream Ownership Boundaries + +| Concern | Owner | Consumers | +| --------------------------------------- | ----- | --------------------------------------------- | +| Label vocabulary/effects | `W1` | All workstreams | +| DRISL signing/verification | `W1` | `W3`, `W4`, conformance | +| D1 state transitions | `W2` | `W3`, `W6`, `W9`, `W10` | +| Public label history/sequence | `W3` | `W4`, third parties | +| Aggregator current label state | `W4` | `W5`, directory/admin | +| Release eligibility semantics/evaluator | `W1` | `W4`, `W5`, core, CLI, admin | +| Consumer eligibility adapters | `W5` | Core, CLI, admin | +| Run lifecycle/supersession | `W6` | `W8`, `W9`, `W10` | +| Artifact safety | `W7` | Labeler, installer, delegated release service | +| Finding-to-label mapping | `W8` | `W3`, `W9`, transparency | +| Human authorization/actions | `W9` | `W3`, audit, notifications | +| Public explanation/contact | `W10` | Registry client/admin, publishers | +| Production configuration/runbooks | `W11` | Operators, self-hosters | +| Cross-component acceptance | `W12` | Gate 6 | + +If a change crosses an ownership boundary, the owning shared contract changes first and all consumers update in the same integration gate. Do not fix policy divergence with local string checks. + +## Launch Sequencing + +### Staging-only protocol phase + +- Deploy DID, signer, query, and subscription endpoints. +- Issue only test labels against test subjects. +- Connect staging aggregator. +- Keep official production client policy unchanged. + +Exit: Gate 2. + +### Consumer shadow phase + +- Production aggregator may ingest/hydrate labels without enforcing positive assessment. +- Admin may display shadow moderation state to operators only. +- Compare shared evaluator decisions against current install behavior. +- Assess first-party catalog and resolve unexpected blocks. + +Exit: Gates 3 and 4 plus complete first-party assessment. + +### Warning enforcement phase + +- Surface warnings and assessment state. +- Continue allowing current production behavior where the positive gate is not yet enabled. +- Exercise operator override, notification, and incident paths on staging/controlled subjects. + +Exit: Gate 5 and operational drills. + +### Positive-assessment enforcement phase + +- Enable the shared eligibility gate for official admin/core/CLI paths. +- Monitor pending/error rates, blocks, overrides, subscriber lag, and install failures. +- Keep deployment-level rollback capable of restoring prior client policy without deleting labels or corrupting assessment history. + +Exit: Gate 6 production smoke and sustained healthy operation. + +## Final Acceptance Checklist + +### Contracts + +- [ ] Stable policy/vocabulary fixtures are ratified. +- [ ] Labeler lexicons and policy schema are published. +- [ ] Shared crypto/header/moderation packages are the only implementations. +- [ ] Legacy colon vocabulary is absent or bounded by an explicit migration. +- [ ] Retained source/tests have no `.opencode/plans` dependency, all durable fixtures have moved to normal package/test paths, and the ephemeral planning folder is removed. + +### Service + +- [ ] D1 migrations and real workerd tests pass. +- [ ] Signer, query, subscription, replay, negation, and rotation pass. +- [ ] Discovery never labels unverified subjects. +- [ ] Assessment reruns/supersession/current pointer are deterministic. +- [ ] Artifact, capability-analysis, AI, image, and history stages pass their fixtures. + +### Consumers + +- [ ] Aggregator verifies, hydrates, filters, redacts, and cascades without N+1 queries. +- [ ] `atproto-content-labelers` contract is correct. +- [ ] Core install/update and CLI/admin use the same evaluator. +- [ ] Manual override cannot bypass package/publisher/takedown actions. +- [ ] Installed blocked/yanked release alerts work. + +### Operators and Publishers + +- [ ] Access JWT and roles are verified server-side. +- [ ] Reviewer/admin action boundaries pass tests. +- [ ] Public/private evidence separation is proven. +- [ ] Notifications and reconsideration flow work without DB edits. +- [ ] Kumo/RTL/accessibility requirements pass. + +### Operations + +- [ ] Metrics and alerts cover every critical dependency. +- [ ] Backup/restore and retention jobs pass. +- [ ] Key rotation and compromise drills pass. +- [ ] Full replay into a fresh aggregator passes. +- [ ] Self-hosting/third-party consumption docs are complete. +- [ ] External security review critical/high findings are closed. diff --git a/.opencode/plans/plugin-registry-labelling-service/spec.md b/.opencode/plans/plugin-registry-labelling-service/spec.md new file mode 100644 index 0000000000..42eda80c01 --- /dev/null +++ b/.opencode/plans/plugin-registry-labelling-service/spec.md @@ -0,0 +1,1372 @@ +# EmDash Plugin Registry Labelling Service + +Companion: [Implementation plan](./implementation-plan.md) + +Status: draft implementation spec for the follow-on trust and moderation work described by RFC #694. + +This document specifies the EmDash-operated reference labeler for the decentralized plugin registry. It covers automated release assessment, signed ATProto labels, operator actions, the public explanation API, aggregator and client integration, and the small operator console. + +It does not amend RFC #694. The RFC establishes the registry and identifies a labeler as the trust and moderation layer; this document defines that service in enough detail to implement and review it. + +## 1. Decisions + +The following decisions were made before drafting this spec. + +| Area | Decision | +| ------------------------- | --------------------------------------------------------------------------------------------------------------- | +| Launch role | Safety and quality labeler, with narrow takedown authority | +| Assessment coverage | Assess every release, including verified and first-party publishers | +| Automated inputs | Bundle code, manifest and metadata, images, and publisher history | +| Automated authority | Deterministic critical findings and a single AI critical finding may hard-block | +| AI hard-block scope | Security and impersonation findings only; quality findings never hard-block | +| Hard-block representation | Descriptive labels, not `!takedown` | +| Official client policy | Require a positive assessment label; hard-block configured high-risk labels | +| Manual scope | Review/override automated decisions and issue emergency takedowns | +| Broad labels | Package and publisher labels require manual action | +| Emergency takedown | Admin-only `!takedown` issuance and retraction | +| Operator auth | Cloudflare Access | +| Operator roles | Reviewer and admin | +| Deployable shape | One Worker application for assessment, issuance, distribution, and console | +| Discovery | Independent Jetstream consumer | +| Artifact source | Verified aggregator mirror, with a declared-URL fallback | +| AI | Workers AI (`AI` runtime binding), `@cf/moonshotai/kimi-k2.7-code` | +| Public evidence | Public assessment summary; private detailed evidence | +| Publisher notice | Package security contact, then publisher profile fallback; email plus an ATProto-visible notice where practical | +| Appeals | Email/manual reconsideration, not a first-class appeals workflow in v1 | +| Policy changes | Versioned in code and deployed through normal review | +| Reassessment | New releases, operator request, and model/policy-revision changes | +| Assessment failure | Release remains in a temporary pending/unknown state and official clients block it | +| Reports | Defer `com.atproto.moderation.createReport` and report triage in v1 | + +## 2. Goals + +1. Assess every newly observed EmDash plugin release without making publication depend on a central upload service. +2. Produce standard, signed ATProto labels that any aggregator or client can consume and verify. +3. Let the official aggregator and EmDash admin make conservative install decisions without treating the aggregator as the source of truth. +4. Automate the common path at the expected initial volume of a handful of releases per day. +5. Give operators enough interface to understand a decision, override it, re-run an assessment, or issue an emergency takedown. +6. Give publishers a useful public explanation and a direct notification when a release is blocked or warned. +7. Preserve the distinction between publication and reach: publishers control their records; labelers publish opinions; consumers choose policy. +8. Keep policy, model, and operator decisions auditable and reproducible. + +## 3. Non-goals + +- Preventing a publisher from writing a release record to their own repository. +- Acting as the only valid labeler for the ecosystem. +- Replacing install-time provenance, checksum, manifest consistency, or sandbox enforcement. +- Guaranteeing that an AI assessment proves code is safe. +- Accepting user reports through `com.atproto.moderation.createReport` in v1. +- A full case-management, appeals, or customer-support system. +- Automated package-wide or publisher-wide punishment. +- Publisher verification issuance. Verification remains a separate trust process even if a future console combines the views. +- Ratings, reviews, install counts, popularity ranking, or recommendation scoring. +- General ATProto social-content moderation. +- Native npm plugin assessment. V1 covers sandboxed registry releases only. +- Multi-tenant hosted labelling. One deployment represents one labeler DID. + +## 4. Trust Model + +There are four distinct authorities. The implementation must not collapse them. + +1. **Publisher repository:** authoritative for package and release records. +2. **Artifact checksum in the signed release:** authoritative for the bytes the publisher released. +3. **Labeler:** authoritative only for labels issued by its DID. +4. **Consumer policy:** authoritative for whether a label warns, hides, blocks, or is ignored. + +The official EmDash deployment trusts `did:web:labels.emdashcms.com` by default. Other aggregators and clients may ignore it, consume it without enforcing it, or combine it with other labelers. + +An automated hard block is therefore not deletion and is not a network takedown. It is a signed descriptive claim, such as `malware`, which the official EmDash policy treats as ineligible for installation. + +`!takedown` is stronger. It requests protocol-level redaction when the labeler is accepted with the `redact` flag. V1 reserves it for an admin's emergency manual action. + +## 5. Architecture + +The reference service is a single Cloudflare Worker application in `apps/labeler`. It contains separate internal capabilities even though they deploy together. + +```mermaid +flowchart LR + JS[Jetstream] --> JD[Discovery Durable Object] + JD --> Q[Assessment Queue] + Q --> WF[Assessment Workflow] + AGG[Aggregator mirror and sync.getRecord] --> WF + SRC[Publisher artifact URL fallback] --> WF + WF --> DET[Deterministic checks] + WF --> AI[Workers AI binding] + DET --> PE[Versioned policy engine] + AI --> PE + PE --> D1[(Labeler D1)] + PE --> ISSUER[Label issuer] + UI[Access-protected operator console] --> D1 + UI --> ISSUER + ISSUER --> D1 + ISSUER --> SD[Subscriber Durable Object] + SD --> SUB[com.atproto.label.subscribeLabels] + D1 --> QUERY[com.atproto.label.queryLabels] + D1 --> PUBLIC[Public assessment API] + SUB --> AGGI[Aggregator label ingest] + QUERY --> AGGI +``` + +### 5.1 Components + +| Component | Responsibility | +| ------------------------- | ----------------------------------------------------------------------------------- | +| Discovery Durable Object | Maintains the outbound Jetstream subscription and persists the cursor | +| Assessment Queue | Buffers and deduplicates immutable release assessment jobs | +| Assessment Workflow | Fetches verified inputs, runs checks, stores evidence, and invokes policy | +| Policy engine | Maps normalized findings to labels and client effects under a versioned policy | +| Label issuer | Allocates sequence numbers, signs labels, stores history, and broadcasts them | +| Subscriber Durable Object | Serves replayable `subscribeLabels` WebSocket connections using hibernation | +| Query API | Implements standard `queryLabels` | +| Public assessment API | Exposes case-specific summaries without private evidence | +| Operator console | Provides queue, assessment detail, override, re-run, and emergency takedown actions | +| Notification worker step | Sends publisher notices after externally visible decisions | + +### 5.2 Why one deployable + +At a handful of releases per day, separate services would add deployment, authentication, and failure modes without useful scale isolation. Internal modules must still preserve least privilege: + +- Only the issuer module can read the label-signing key. +- Assessment model output is data, never a direct call to signing primitives. +- The policy engine constructs a bounded decision proposal. +- The issuer validates every proposal against the policy's allowed label vocabulary and subject rules. +- Operator routes are protected by Access JWT verification and role checks. +- Public label and assessment routes cannot reach operator mutations. + +If model execution later becomes high-volume or accepts broader untrusted inputs, the assessor can split into another Worker without changing the public label protocol. + +## 6. Identity and Declaration + +The service identity is `did:web:labels.emdashcms.com`. + +Its DID document must include: + +- `#atproto_label`, the public key used to verify labels. +- `#atproto_labeler`, an `AtprotoLabeler` service pointing to `https://labels.emdashcms.com`. + +V1 does not publish an `app.bsky.labeler.service/self` declaration. That record is application-specific and its subject and reason fields describe accepted moderation reports, which this service does not support. Labeler discovery uses the base ATProto DID service, while label definitions, localized descriptions, subject restrictions, and contact details are published in the EmDash policy document. A call to `com.atproto.moderation.createReport` returns `NotSupported`; it is never silently accepted into an unmonitored queue. + +The versioned, machine-readable policy document is available at: + +```text +GET /.well-known/emdash-labeler-policy.json +``` + +It includes: + +- Labeler DID. +- Policy version and effective timestamp. +- Supported subject collections. +- Label definitions and default official-client effects. +- Current assessment schema version. +- Public explanation API base URL. +- Contact and reconsideration URL/email. +- A statement that model output is advisory evidence interpreted by versioned policy. + +This document is informational. Signed labels remain the interoperable source of claims. + +## 7. Label Vocabulary + +Label values use lowercase kebab case as recommended by the ATProto label specification. Confidence, scores, model names, and structured evidence do not go in `val`. + +### 7.1 Eligibility labels + +| Label | Subject | Meaning | Official policy | +| ----------------------- | ----------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | +| `assessment-passed` | Release URI + CID | Required checks completed and either policy or an authorized manual override found no blocking condition | Eligible, subject to all other labels and install checks | +| `assessment-overridden` | Release URI + CID | A reviewer/admin explicitly superseded the automated outcome for this exact release CID | Use the accompanying manual pass until explicitly retracted | +| `assessment-pending` | Release URI + CID | Assessment is queued, running, retrying, or awaiting an artifact | Block temporarily | +| `assessment-error` | Release URI + CID | Assessment could not complete after bounded retries | Block and surface operational failure | + +The official client requires an active `assessment-passed` label from an accepted labeler. Absence of that label means unknown, not safe. + +`assessment-pending` improves UX but is not the security gate. If the pending label is delayed, missing, expired, or retracted incorrectly, the missing positive label still blocks. + +When assessment completes, the issuer negates `assessment-pending` and issues exactly one of: + +- `assessment-passed`, possibly alongside warning labels. +- One or more blocking descriptive labels, without `assessment-passed`. +- `assessment-error` if the pipeline exhausted retries without a judgment. + +A reviewer/admin may manually unblock a false positive by issuing negations for the automated blocking labels plus action-backed `assessment-passed` and `assessment-overridden` labels for the exact URI + CID. The public assessment response links the public action summary and retains the superseded automated assessment. Merely retracting a blocking label is not enough to make a release eligible. + +### 7.2 Blocking security labels + +| Label | Meaning | Automatic issuance | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| `malware` | Code or payload intentionally compromises, persists, mines, destroys, or executes an unauthorized workload | Deterministic critical or AI critical | +| `data-exfiltration` | Plugin intentionally sends protected site/user data outside its declared and expected purpose | Deterministic critical or AI critical | +| `credential-harvesting` | Plugin solicits, captures, or transmits credentials deceptively | Deterministic critical or AI critical | +| `supply-chain-compromise` | Bundled code evidence indicates a known malicious or substituted component inlined at build time | AI critical | +| `critical-vulnerability` | AI identifies a critical, applicable vulnerability in the bundled code under the policy's blocking rule | AI critical only; v1 has no dependency/vulnerability scanner | +| `artifact-integrity-failure` | Available artifact bytes do not match the checksum in the signed release | Deterministic critical | +| `invalid-bundle` | The archive is malformed, unsafe, or missing required installable content | Deterministic critical | +| `undeclared-access` | Bundle behavior or manifest inconsistency attempts access outside the signed declaration | Deterministic critical; AI may propose but not establish manifest inequality | +| `impersonation` | Package or publisher materially impersonates another identity, product, or trusted project | Deterministic identity evidence or AI critical | + +Official EmDash clients block these labels. Other consumers choose their own behavior. + +The policy must distinguish a routine code weakness from an actually blocking critical vulnerability. AI confidence alone is insufficient. The versioned rule should require reachable critical behavior, corroborating capability-analysis evidence, or another explicit rule. + +### 7.3 Warning labels + +| Label | Meaning | Official policy | +| --------------------- | ---------------------------------------------------------------------------------------------- | ------------------- | +| `suspicious-code` | Concerning behavior lacks enough evidence for a blocking security label | Warn before install | +| `obfuscated-code` | Material code is intentionally difficult to inspect | Warn before install | +| `privacy-risk` | Collection, tracking, or transmission creates a non-blocking privacy concern | Warn before install | +| `misleading-metadata` | Description, screenshots, identity claims, or declared purpose materially differ from behavior | Warn and downrank | +| `low-quality` | Release is placeholder, generated without substance, or lacks meaningful functionality | Warn and downrank | +| `broken-release` | Bundle cannot perform its stated function despite passing structural registry validation | Warn and downrank | + +Quality labels never prevent direct installation on their own. The official admin presents the reason and requires confirmation when warning labels are active. + +### 7.4 Manual system labels + +| Label | Subject | Authority | Effect | +| ----------------------- | ------------------------------ | ----------------- | ---------------------------------------------------------------------- | +| `!takedown` | Release, package, or publisher | Admin only | Redact for consumers accepting this labeler with `redact` | +| `security-yanked` | Release | Reviewer or admin | Block install/update; retain visible tombstone and explanation | +| `publisher-compromised` | Publisher DID | Admin only | Block all publisher packages/releases until retracted | +| `package-disputed` | Package URI | Reviewer or admin | Warn and prevent recommendation; direct install policy is configurable | + +`security-yanked` replaces the colon-containing `security:yanked` placeholder currently present in parts of the aggregator and admin. Colon-structured values conflict with ATProto's recommended label syntax. + +The vocabulary cutover is required before the labeler issues production labels. Aggregator SQL, registry-client helpers, admin rendering, and install/update handlers must use `security-yanked` and the complete blocking-label set. A production preflight checks for persisted legacy `security:yanked` labels. If any exist, consumers temporarily recognize both values while operators reissue/negate them; otherwise no compatibility path is added. + +### 7.5 Severity is assessment data + +Labels describe conditions, not model severity. Each assessment finding has an internal severity of `critical`, `high`, `medium`, `low`, or `info`. + +Policy maps findings to labels. Only a `critical` security or impersonation finding may produce an automated blocking label. A `critical` quality finding is normalized to a warning because quality labels are non-blocking by policy. + +## 8. Subject and CID Rules + +### 8.1 Automated labels + +Automation labels only an individual release record and always supplies both: + +- `uri`: immutable release AT URI. +- `cid`: the exact release record CID assessed. + +The assessment also binds the artifact checksum, artifact ID, and observed bytes. This prevents an assessment of one release-record version or artifact from being replayed as an assessment of another. + +### 8.2 Package labels + +Package/profile labels are manual in v1. + +- A correction-specific label such as `misleading-metadata` should be CID-bound so editing the profile removes its applicability pending reassessment. +- A policy action such as `package-disputed` or `!takedown` is URI-wide and remains active across profile edits until explicitly retracted. + +### 8.3 Publisher labels + +Publisher DID labels are manual in v1 and apply to all packages from that identity according to consumer policy. A release finding never automatically escalates to a publisher label. + +### 8.4 Deletion + +Deleting a release record does not erase assessment history or labels. Aggregators may stop listing the release because of the registry tombstone, while the public assessment remains available for transparency and incident response. + +## 9. Release Assessment Workflow + +### 9.1 Discovery + +The Discovery Durable Object maintains an independent Jetstream subscription for experimental and stable EmDash release collections. + +For each create or update event: + +1. Validate the event shape. +2. Derive a verification job key from `(release URI, release CID)`. +3. Enqueue the verification/assessment job durably. +4. Persist the Jetstream cursor only after queue acceptance. +5. In the workflow, fetch and cryptographically verify the source record for the exact URI + CID. +6. Only after verification, insert the subject and issue `assessment-pending`. +7. Continue with artifact acquisition and assessment. + +Deletes record a tombstone and cancel queued work where possible. A running assessment may finish for forensic purposes but must not issue a new positive label for a deleted subject. + +Jetstream is discovery only. Raw event JSON is never trusted as an assessment input or as sufficient authority to sign even a pending label. A forged or unverifiable event is retained as an operational dead letter and produces no public label. + +### 9.2 Idempotency + +Every assessment run has a deterministic `runKey`: + +```text +sha256( + release-uri + "\n" + + release-cid + "\n" + + policy-version + "\n" + + model-id + "\n" + + prompt-hash + "\n" + + trigger-id +) +``` + +`trigger-id` is stable for a logical trigger: `initial:` for first observation, the immutable operator action ID for a manual rerun, or the new model/policy version for a model-or-policy-revision rerun. At most one active workflow exists for a `runKey`. Redelivery wakes or observes the same run; a new model/policy revision or operator action creates a new immutable run that may supersede the previous effective assessment. + +A new policy version does not automatically reassess every release. A model-or-policy-revision trigger selects affected releases and enqueues a run with the new model ID or policy version and trigger ID. + +### 9.3 Acquire verified inputs + +The workflow obtains: + +1. Signed release record and CID. +2. Associated package profile and publisher profile. +3. EmDash release extension and declared access. +4. Bundle from a validated aggregator mirror. +5. Bundle from the publisher-declared URL only if the mirror is unavailable. +6. Images referenced by the package/release where needed. +7. Relevant publisher history already observed by the labeler. + +Every artifact source is checksum-verified against the signed release before extraction. The mirror is a transport optimization, not a trust root. + +Fallback URL fetching must use the same class of controls as the registry installer: + +- HTTPS by default. +- DNS-aware SSRF checks. +- Redirect revalidation and bounded redirect count. +- Compressed, per-file, decompressed, and file-count caps. +- Streaming abort at limits. +- Safe tar path handling and rejection of links/special files. +- Global time and byte budgets. +- Content-type sniffing rather than trusting headers. +- No ambient credentials or cookies. + +### 9.4 Deterministic validation + +Run deterministic checks before any AI call: + +- Release and extension lexicon validity. +- Release rkey/package/version consistency. +- Artifact checksum and archive validity. +- Required bundle files and manifest validity. +- Canonical manifest `declaredAccess` equality with the signed release extension. +- Unknown access category/operation rejection. +- Forbidden archive entries and executable payload classes. +- Static forbidden-runtime patterns where they are unambiguous. +- Metadata URL and identity consistency. +- Image dimensions/types and archive policy. + +A registry-invalid release may already have been rejected by the aggregator. The labeler still stores the observation if independently discovered, but only issues labels for a resolvable, valid subject. Invalid/unresolvable records go to operational dead letters rather than creating unverifiable public labels. + +Permanent failures after the source record is verified are findings, not operational errors. Checksum mismatch maps to `artifact-integrity-failure`; malformed/unsafe/missing bundle content maps to `invalid-bundle`; manifest declaration mismatch maps to `undeclared-access`. Transient mirror/network/model unavailability retries and may become `assessment-error`. + +### 9.5 Capability analysis + +EmDash registry plugins ship as a single bundled artifact: dependencies are inlined at build time, so there is no dependency graph, lockfile, or SBOM in the shipped bundle, and no dependency, SBOM, or advisory/CVE analysis is performed. + +Instead the labeler statically extracts the bundle's real API/capability surface (network/fetch, storage, `eval`/`Function`, filesystem, crypto, env/process, DOM) and diffs it against the `releaseExtension` declared access categories. A mismatch produces an `undeclared-access` finding. This deterministic declared-vs-actual diff is the highest-signal, EmDash-specific check in the pipeline and also feeds the normalized capability evidence given to the AI code adapter. + +Missing or incomplete capability extraction is not itself a blocking condition in v1. It may lower assessment coverage and be shown in the public summary. + +A model-or-policy-revision trigger enqueues affected releases for reassessment; see section 10. + +### 9.6 Code and metadata AI assessment + +`@cf/moonshotai/kimi-k2.7-code` is invoked directly through the runtime `AI` binding (`env.AI`), not AI Gateway; moderation inference is never cached. An internal inference request ID generated per call is retained with private evidence for correlation. + +The model receives explicitly delimited inputs: + +- Manifest and signed declared access. +- Package/release metadata. +- File inventory and deterministic check and capability-analysis summary. +- The full bundle source, size-capped by the existing release artifact size cap. If a bundle ever exceeds the model's context window, coverage degrades (`partial`/`unavailable`) rather than silently truncating. +- Publisher-history facts, not an unbounded reputation narrative. +- A policy-generated task and output schema. + +All plugin-controlled text and code is untrusted data. The prompt states that instructions inside those inputs must never alter the audit task. Model output is parsed through a strict schema and cannot name arbitrary labels. + +The model returns findings, not labels: + +```ts +interface ModelFinding { + category: AllowedFindingCategory; + severity: "critical" | "high" | "medium" | "low" | "info"; + confidence: number; + title: string; + summary: string; + evidenceRefs: string[]; + affectedFiles: string[]; +} +``` + +`AllowedFindingCategory` is generated from the versioned policy. Unknown categories fail validation. + +The model may produce a critical security or impersonation finding that the policy maps directly to a hard-blocking descriptive label. This is an explicit product decision. Guardrails are therefore mandatory: + +- Structured output with strict category allowlist. +- Evidence references must resolve to supplied input spans or deterministic facts. +- No hard block from free-form summary text. +- No hard block from quality categories. +- Fixture-based calibration before model/prompt changes deploy. +- Model ID, prompt hash, policy version, and input manifest retained. +- Immediate operator retraction path. + +### 9.7 Image assessment + +Images are assessed for: + +- Brand impersonation. +- Misleading screenshots. +- Credential phishing or fake system UI. +- Hateful or explicit imagery that violates the published directory policy. +- Mismatch between screenshots and actual plugin behavior where code/metadata evidence supports it. + +Image-only quality concerns generate warning labels. Critical impersonation or credential-harvesting evidence may hard-block. + +Every image retains its original MIME type, hash, dimensions, and source path in private evidence. The implementation must not coerce arbitrary image bytes to `image/png` as the legacy marketplace auditor did. + +### 9.8 Publisher history + +History is context, not guilt by association. The model/policy may use: + +- Prior releases from the same DID. +- Recent identity/handle/profile changes. +- Existing active manual labels. +- Repeated exact artifact checksums previously found violating, or repeated policy violations. +- Verification state as display context, never as an assessment exemption. + +Automation cannot issue package- or publisher-wide labels from history. It can create an operator review recommendation. + +### 9.9 Policy resolution + +The policy engine consumes normalized deterministic, image, and model findings. + +Resolution order: + +1. If the subject disappeared or CID changed, store the result as stale and issue no current label. +2. Map permanent artifact, archive, manifest, and other validation failures to deterministic findings. Do not retry them as infrastructure failures. +3. If transient infrastructure failure prevented complete required inputs, retry; after exhaustion issue `assessment-error` and negate `assessment-pending`. +4. Map blocking deterministic findings to descriptive labels. +5. Map AI critical security/impersonation findings to descriptive blocking labels. +6. Map non-blocking findings to warning labels. +7. If no blocking label exists, issue `assessment-passed`. +8. Negate prior active automated labels for the same URI + CID that are no longer supported by this assessment. +9. Negate `assessment-pending`. +10. Store the public summary and private evidence. +11. Notify the publisher for newly issued/retracted blocking or warning labels. + +An assessment may pass and warn at the same time. `assessment-passed` means no hard-blocking condition, not "the labeler found nothing concerning." + +## 10. Reassessment and Supersession + +V1 reassesses when: + +- A new release CID is observed. +- The assessment model or policy version changes (a model/policy revision). +- An operator requests a re-run. + +Policy/model changes do not silently rewrite historical outcomes. A deployment may mark old assessments as using an older policy, and an admin may enqueue a bounded migration if a change requires it. + +When a new assessment supersedes an old assessment for the same URI + CID: + +- The old assessment remains immutable and public as historical metadata. +- Current labels are changed only through newly signed positive or negation labels. +- The new assessment records `supersedesAssessmentId`. +- Notifications explain whether the effective install policy changed. + +Manual overrides remain in force across automated reassessment until an operator explicitly removes them. Automation cannot negate action-backed `assessment-passed`/`assessment-overridden` labels or undo a human `!takedown`, `security-yanked`, package label, or publisher label. + +Exactly one completed automated assessment is current for each `(labeler DID, release URI, release CID)`. Starting a newer run does not move that pointer, but its active `assessment-pending` label temporarily makes the release ineligible even if the prior run passed, unless `assessment-overridden` remains active. When the newer run completes, label changes and the current-assessment pointer update in the same transaction. A stale/cancelled run never becomes current. A manual pass is a separate active override layered over the current automated assessment, not a fabricated automated assessment row. + +## 11. Manual Operator Workflow + +### 11.1 Console pages + +The purpose-built Kumo console is served by the labeler Worker under `/admin` and protected by Cloudflare Access. + +V1 pages: + +| Route | Purpose | +| ----------------------------- | ------------------------------------------------------------------------------------------------------ | +| `/admin` | Queue health, pending/error counts, recent blocking decisions, subscription health | +| `/admin/assessments` | Filterable assessment list by state, label, publisher, package, model, and policy version | +| `/admin/assessments/:id` | Public summary, private findings, evidence excerpts, model provenance, labels, and timeline | +| `/admin/subjects/:encodedUri` | All assessments and active/historical labels for a release/package/publisher | +| `/admin/audit-log` | Immutable operator and system action log | +| `/admin/system` | Read-only policy/model versions, signing key ID, Jetstream cursor, subscriber health, queue/DLQ health | + +All layout uses RTL-safe logical classes and the console uses Kumo components rather than hand-rolled controls. Console strings are plain English — decision (2026-07-13): the console is internal infrastructure and is not localized; `packages/admin` localization rules are unaffected. + +### 11.2 Reviewer actions + +A reviewer may: + +- Issue or retract a descriptive release warning/block label. +- Override an automated decision with a required reason. +- Re-run an assessment. +- Issue/retract `security-yanked` on a release. +- Issue/retract `package-disputed`. +- View private evidence and notification status. + +An unblock override is one atomic action: negate the selected automated blocking labels and issue manual `assessment-passed` plus `assessment-overridden` labels for the exact URI + CID. The console does not offer a standalone "retract and allow" shortcut. New automated findings remain visible and alert reviewers, but cannot negate the override; a reviewer/admin must retract it explicitly. + +A reviewer may not: + +- Issue or retract `!takedown`. +- Issue or retract `publisher-compromised`. +- Change policy, model, signing keys, Access role mappings, or system configuration. + +### 11.3 Admin actions + +An admin may perform reviewer actions and: + +- Issue/retract `!takedown` on a release, package, or publisher. +- Issue/retract `publisher-compromised`. +- Retry or quarantine dead-letter work. +- Pause automated issuance while leaving public query/subscription endpoints available. +- Start and complete signing-key rotation. + +Policy remains versioned in code. Admin status pages show configuration but do not provide live prompt/threshold editing. + +### 11.4 Action ceremony + +Every mutation uses a confirmation dialog showing: + +- Subject URI/DID and current resolved display identity. +- Label value and resulting official-client effect. +- Whether the action is CID-bound or subject-wide. +- Existing active labels that will be superseded. +- Required operator reason. + +`!takedown` additionally requires typing the package slug, publisher handle, or final URI segment. At current scale, two-person approval is not required, but the audit event and alert are immediate. + +## 12. Operator Authentication and Authorization + +Cloudflare Access protects `/admin/*` and internal mutation APIs. + +The Worker must validate `Cf-Access-Jwt-Assertion` itself using: + +- The Access team domain JWKS. +- Expected issuer. +- Exact Access application audience. +- Signature, expiry, and not-before claims. + +Presence of an Access header is not authentication. + +Role mapping is deployment configuration: + +```json +{ + "admins": ["emdash-labeler-admins"], + "reviewers": ["emdash-labeler-reviewers"] +} +``` + +The exact claim used for groups must be verified against the configured Access identity provider. Email allowlists may be used initially, but roles must not be inferred from an unverified request header. + +Mutation requests also require: + +- Same-origin checks (`Origin` and `Sec-Fetch-Site` when present; absence passes for non-browser clients such as Access service tokens). +- A required custom header (`X-EmDash-Request: 1`). Decision (2026-07-13): replaces the earlier CSRF token/double-submit wording — combined with the JSON content-type and origin checks it blocks every browser forgery vector via CORS preflight, and matches emdash core's convention. The Access assertion alone is not a CSRF defense: the edge injects it from the `CF_Authorization` cookie. Console API routes must never emit cross-origin CORS allow headers. +- JSON content type (`application/json`, optional `utf-8` charset only). +- Idempotency key. +- Freshly verified Access identity. +- Role check in the handler. + +Service-to-service automation does not reuse human Access sessions. Internal queue/workflow calls use bindings. Any external administrative automation uses an Access service token plus a separate scoped service credential checked by the Worker. + +## 13. Public and Internal Interfaces + +### 13.1 Standard ATProto endpoints + +The service implements: + +- `GET /xrpc/com.atproto.label.queryLabels` +- `GET /xrpc/com.atproto.label.subscribeLabels` with WebSocket upgrade + +Semantics follow the ATProto label specification, including cursors, signatures, negation, expiration, source filtering, URI patterns, and replay from cursor `0` for retained history. + +The service does not require authentication for public label distribution. + +### 13.2 Public assessment API + +The public API supplies the case-specific context labels cannot carry. + +Proposed experimental XRPC endpoints: + +```text +com.emdashcms.experimental.labeler.getAssessment +com.emdashcms.experimental.labeler.getCurrentAssessment +com.emdashcms.experimental.labeler.listAssessments +com.emdashcms.experimental.labeler.getPolicy +``` + +`getAssessment` accepts `id` and returns: + +```ts +interface PublicAssessment { + id: string; + src: string; + subject: { uri: string; cid: string }; + artifact?: { id?: string; checksum: string }; + state: "pending" | "passed" | "warned" | "blocked" | "error" | "superseded"; + summary: string; + coverage: { + code: "complete" | "partial" | "unavailable"; + metadata: "complete" | "partial" | "unavailable"; + images: "complete" | "not-present" | "partial" | "unavailable"; + }; + labels: Array<{ val: string; active: boolean; issuedAt: string; expiresAt?: string }>; + policyVersion: string; + assessmentSchemaVersion: number; + model?: { provider: "workers-ai"; modelId: string; promptVersion: string }; + createdAt: string; + completedAt?: string; + supersedesAssessmentId?: string; + reconsiderationUrl: string; +} +``` + +Public output excludes: + +- Source excerpts not already public through the artifact. +- Full model prompts/responses. +- Operator-only notes. +- Publisher contact data. +- Raw inference request/response payloads and tokens. +- Exploit details that would increase harm before remediation. + +Standard label objects cannot carry an assessment ID, so clients do not infer one from timestamps. `getCurrentAssessment` accepts exact `uri`, `cid`, and optional `src`, then reads the materialized current pointer and returns the current completed automated assessment, any newer pending run, active public labels, and any active manual override. Manual labels/actions have public action summaries and no fabricated assessment association. `getAssessment(id)` remains the immutable historical lookup. + +### 13.3 Operator API + +Purpose-built JSON endpoints under `/admin/api` are acceptable because they are not ecosystem protocol. They include: + +- `GET /assessments` +- `GET /assessments/:id` +- `POST /assessments/:id/retry` +- `POST /labels/issue` +- `POST /labels/retract` +- `GET /subjects` +- `GET /audit-log` +- `GET /system/health` + +Mutation bodies identify a subject, label, optional CID/expiry, reason, and idempotency key. The server derives `src`, `cts`, sequence, operator identity, and signing key ID. + +There is no generic "sign arbitrary label object" endpoint. + +## 14. Storage Model + +The labeler uses an independent D1 database. + +### 14.1 Subjects and assessments + +```sql +CREATE TABLE subjects ( + uri TEXT NOT NULL, + cid TEXT NOT NULL, + did TEXT NOT NULL, + collection TEXT NOT NULL, + rkey TEXT NOT NULL, + observed_at TEXT NOT NULL, + deleted_at TEXT, + PRIMARY KEY (uri, cid) +); + +CREATE TABLE assessments ( + id TEXT PRIMARY KEY, + run_key TEXT NOT NULL UNIQUE, + uri TEXT NOT NULL, + cid TEXT NOT NULL, + artifact_id TEXT, + artifact_checksum TEXT, + state TEXT NOT NULL, + trigger TEXT NOT NULL, + trigger_id TEXT NOT NULL, + policy_version TEXT NOT NULL, + model_id TEXT, + prompt_hash TEXT, + public_summary TEXT, + coverage_json TEXT NOT NULL, + supersedes_assessment_id TEXT, + started_at TEXT, + completed_at TEXT, + created_at TEXT NOT NULL, + FOREIGN KEY (uri, cid) REFERENCES subjects(uri, cid) +); + +CREATE INDEX idx_assessments_state_created +ON assessments(state, created_at DESC); + +CREATE INDEX idx_assessments_subject +ON assessments(uri, cid, created_at DESC); + +CREATE TABLE current_assessments ( + src TEXT NOT NULL, + uri TEXT NOT NULL, + cid TEXT NOT NULL, + assessment_id TEXT NOT NULL REFERENCES assessments(id), + updated_at TEXT NOT NULL, + PRIMARY KEY (src, uri, cid) +); +``` + +The policy-finalization transaction updates `current_assessments` only after storing the completed result and issuing its label changes. It replaces the pointer only for the run being finalized under the subject's workflow lock. Pending, stale, cancelled, and failed-before-decision runs do not replace it; an `assessment-error` remains visible through labels and run history while the last completed pointer is retained for explanation. + +### 14.2 Findings and evidence + +```sql +CREATE TABLE findings ( + id TEXT PRIMARY KEY, + assessment_id TEXT NOT NULL REFERENCES assessments(id), + source TEXT NOT NULL, + category TEXT NOT NULL, + severity TEXT NOT NULL, + confidence REAL, + title TEXT NOT NULL, + public_summary TEXT NOT NULL, + private_detail TEXT NOT NULL, + evidence_refs_json TEXT NOT NULL, + created_at TEXT NOT NULL +); + +CREATE INDEX idx_findings_assessment +ON findings(assessment_id); + +CREATE TABLE evidence_objects ( + id TEXT PRIMARY KEY, + assessment_id TEXT NOT NULL REFERENCES assessments(id), + kind TEXT NOT NULL, + sha256 TEXT NOT NULL, + r2_key TEXT, + metadata_json TEXT NOT NULL, + created_at TEXT NOT NULL +); +``` + +Large private evidence lives encrypted at rest in R2; D1 stores references and hashes. Raw bundles need not be retained by the labeler when the aggregator mirror is durable. Selected evidence required to explain a decision is retained according to the policy below. + +### 14.3 Issued labels + +```sql +CREATE TABLE issued_labels ( + seq INTEGER NOT NULL UNIQUE, + src TEXT NOT NULL, + uri TEXT NOT NULL, + cid TEXT, + val TEXT NOT NULL, + neg INTEGER NOT NULL DEFAULT 0, + cts TEXT NOT NULL, + exp TEXT, + sig BLOB NOT NULL, + ver INTEGER NOT NULL DEFAULT 1, + signing_key_id TEXT NOT NULL, + assessment_id TEXT, + action_id TEXT, + PRIMARY KEY (src, uri, val, cts) +); + +CREATE INDEX idx_issued_labels_subject +ON issued_labels(uri, cid, seq); + +CREATE TABLE label_sequence ( + id INTEGER PRIMARY KEY CHECK (id = 1), + next_seq INTEGER NOT NULL +); +``` + +Sequence allocation and label insertion occur atomically. The service never derives sequence from `MAX(seq)`. + +### 14.4 Actions, notifications, and cursors + +```sql +CREATE TABLE operator_actions ( + id TEXT PRIMARY KEY, + actor_type TEXT NOT NULL, + actor_id TEXT NOT NULL, + role TEXT, + action TEXT NOT NULL, + subject_uri TEXT, + subject_cid TEXT, + label_value TEXT, + reason TEXT NOT NULL, + idempotency_key TEXT NOT NULL UNIQUE, + metadata_json TEXT NOT NULL, + created_at TEXT NOT NULL +); +``` + +Decisions (2026-07-13): the table is named `operator_actions` (this spec previously said `actions`) to disambiguate from the signing-layer `issuance_actions` audit. `subject_uri` is nullable because admin actions such as pause/resume issuance and DLQ controls have no subject. The `action` vocabulary is enforced as a typed union in the guard module (the sole writer), not a SQL `CHECK`, so later workstreams extend it without table rebuilds. The table is append-only, enforced with `BEFORE UPDATE`/`BEFORE DELETE` triggers. + +```sql +CREATE TABLE notifications ( + id TEXT PRIMARY KEY, + action_id TEXT NOT NULL REFERENCES operator_actions(id), + channel TEXT NOT NULL, + recipient_hash TEXT NOT NULL, + state TEXT NOT NULL, + attempts INTEGER NOT NULL DEFAULT 0, + provider_id TEXT, + last_error TEXT, + created_at TEXT NOT NULL, + sent_at TEXT +); + +CREATE TABLE ingest_state ( + source TEXT PRIMARY KEY, + cursor TEXT NOT NULL, + updated_at TEXT NOT NULL +); +``` + +## 15. Label Signing and Distribution + +Signing follows the ATProto label specification exactly: + +1. Construct only the allowed v1 label fields, excluding `sig` and `$type`. +2. Encode with deterministic DRISL CBOR. +3. At the protocol level, SHA-256 the canonical bytes and sign that digest. +4. With the selected high-level `@atcute/crypto` API, pass the canonical CBOR bytes directly to `P256PrivateKey.sign`, and pass those same bytes to `P256PublicKey.verify`. WebCrypto ECDSA performs the protocol SHA-256 operation internally. Never pass the precomputed digest to either high-level API, because that hashes the digest again and produces a non-interoperable signature. +5. Store signature bytes and signing key ID. + +No generic object-signing helper accepts arbitrary fields. The issuer takes a typed internal proposal and reconstructs the complete label object itself. + +The issuer normalizes verification method IDs before selection: relative `#atproto_label` and fully qualified `${issuerDid}#atproto_label` identify the same logical method, and either form is valid alone. Exactly one logical label key must exist; duplicates across either or both forms are invalid regardless of order. The method must have `type: "Multikey"`, `controller` equal to the issuer DID, and a `publicKeyMultibase` using the P-256 multikey codec. At startup, the public multikey derived from the configured private scalar must exactly equal the resolved method's `publicKeyMultibase`; a structurally valid different P-256 key is a fatal mismatch. Missing, duplicate, differently controlled, legacy-typed, or non-P-256 methods are invalid. The issuer and subscribers never fall back to `#atproto` or another DID key. + +The private secret is an unpadded, canonically encoded base64url 32-byte P-256 scalar. Decoding rejects non-base64url characters, padding, non-canonical encodings, lengths other than 32 bytes, zero, and values greater than or equal to the P-256 group order. + +The private key is stored in Cloudflare Secrets Store. A plain Worker secret is acceptable only for initial local/staging deployment, not production launch. + +### 15.1 Subscriber Durable Object + +One named instance, `main`, owns inbound subscribers. + +- Uses WebSocket Hibernation. +- Replays D1 labels with `seq > cursor` before joining live delivery. +- Persists no correctness-critical state only in memory. +- Broadcast failures affect one subscriber and do not roll back an issued label. +- Supports replay from `cursor=0` for retained history. + +### 15.2 Key rotation + +1. Pause new issuance while assessments continue to queue decisions. +2. Generate and store a new private key. +3. Update the DID document's `#atproto_label` verification method. +4. Resume issuance with the new key ID. +5. Re-sign old labels lazily when queried, preserving `cts`. +6. Monitor downstream signature failures. + +The operator console exposes rotation state but never private key material. + +### 15.3 Key compromise + +Key compromise is different from routine rotation. The incident procedure must: + +- Remove/replace the compromised key in the DID document. +- Pause issuance. +- Publish an incident notice in the policy endpoint. +- Establish the last trusted sequence/time where possible. +- Reissue current effective labels with the new key. +- Tell aggregators to replay from a declared safe cursor or full history. +- Preserve compromised history for forensics without treating signatures as currently valid. + +## 16. Aggregator Integration + +The existing aggregator schema already has `labels`, `label_state`, `labelers`, and `ingest_state`, but no working subscription or hydration path. + +### 16.1 Ingest + +For each configured labeler: + +1. Resolve `#atproto_labeler` and `#atproto_label` from its DID document. +2. Maintain an outbound `subscribeLabels` connection in a per-labeler Durable Object. +3. Resume from the persisted cursor. +4. Verify each signature, refreshing the DID document once on failure. +5. Enqueue verified labels. +6. Insert append-only history and update current state only when the incoming `cts` is newer. +7. Persist cursor after durable acceptance. + +Negated and expired labels remain in history but are not hydrated as active labels. + +### 16.2 Request policy + +The aggregator parses `atproto-accept-labelers` once at the request boundary and passes a typed policy to every handler. + +- Missing header: apply deployment defaults. +- Empty header: hydrate/enforce no labelers. +- Invalid syntax: return a client error. +- Repeated DIDs: merge flags. +- Unknown/unavailable DID: omit it from `atproto-content-labelers`. +- `redact`: apply `!takedown` and `!suspend` redaction semantics. + +The response sets `atproto-content-labelers` to the sources actually considered. + +The current router's response-header contract must be corrected as part of label ingest: allow and parse `atproto-accept-labelers`, set `atproto-content-labelers`, and expose `atproto-content-labelers` through CORS. It must not echo/expose the request header as though it were the interpreted response policy. Contract tests cover missing, empty, malformed, repeated, unavailable, and `redact` forms. + +### 16.3 Hydration + +Hydrate labels in batches for all package/release URIs in a response. Do not query once per item. + +The package and release views include active labels from requested/accepted sources. The aggregator may omit signatures in hydrated views as allowed by the label spec; clients can retrieve full signed labels from the labeler. + +### 16.4 Eligibility and filtering + +For the official default policy: + +- Search excludes package/publisher `!takedown`, `publisher-compromised`, and package-wide blocking labels. +- Release lists omit redacted releases and expose blocked releases as tombstones where useful. +- `getLatestRelease` selects the highest semver with active `assessment-passed` and no active blocking labels. +- Direct release resolution returns enough status to explain pending/error/blocked outcomes rather than pretending the release does not exist. +- Package/publisher labels cascade at query-policy time; they are not copied onto every release. + +Subject expansion is explicit: + +- Package search/detail loads labels for the publisher DID and package profile URI. A CID-bound package label applies only when its CID matches the current profile CID; a URI-wide label has no CID. +- Release list/detail/latest loads labels for the publisher DID, package profile URI, and release URI. A CID-bound package or release label applies only to the corresponding current record CID. +- Publisher DID labels are URI-wide because an account label has no record CID. +- Every lookup also filters accepted source DID, `neg = 0`, and unexpired state before policy evaluation. + +Queries batch these subject sets and use indexes covering `(uri, src, val)`/current-state access. Tests exercise all three subject scopes and both CID-bound and URI-wide labels. + +`packages.latest_version` cannot be trusted for label-aware selection. `getLatestRelease` must query eligible releases dynamically or maintain a separate policy-specific projection updated by label ingestion. At current scale, the dynamic query is simpler and safer. + +### 16.5 Cache behavior + +Label-dependent responses remain `private, no-store` and read D1 primary state so a newly ingested takedown affects the next response globally. Static policy and DID documents may be cached. + +## 17. Registry Client and EmDash Admin Behavior + +The registry client exposes one policy helper rather than making every UI or handler compare raw strings. + +```ts +interface ReleaseModeration { + eligibility: "eligible" | "pending" | "error" | "blocked"; + reasonCodes: string[]; + blockingLabels: string[]; + stateLabels: string[]; + warningLabels: string[]; + suppressedLabels: string[]; + applicableLabels: ModerationLabel[]; + redacted: boolean; +} +``` + +This is the shape shipped by `packages/registry-moderation` (`evaluateReleaseModeration`). Label fields carry values, not label objects; `applicableLabels` carries the full applicable label set for display. Assessment context is fetched separately through `getCurrentAssessment`, not returned by the evaluator. + +The helper takes release labels, package labels, publisher labels, accepted-labeler policy, and the current package/release CIDs. It is the only supported server-side eligibility decision path. Both install and update handlers call it immediately before artifact download; they must not retain an independent `security-yanked` string check. + +Official behavior: + +- Manual `!takedown`, `security-yanked`, package, or publisher blocking labels always block and cannot be bypassed by a release override. +- Active `assessment-overridden` plus `assessment-passed`: treat the exact release CID as eligible despite automated pending/error/blocking labels, while displaying the newer findings to the operator. +- Otherwise, active `assessment-pending` or `assessment-error`: do not recommend or install, even if an older automated `assessment-passed` remains active. +- No active `assessment-passed`: do not recommend or install. +- Active blocking label: do not install. +- Warning labels only: show a warning and require explicit confirmation. +- `!takedown`: redact according to accepted labeler flags; no install override. +- `security-yanked`: block fresh installs and updates; alert sites with the installed version. +- Package/publisher blocking label: apply to all releases. + +The server-side install/update handler repeats the eligibility check immediately before downloading. Browser UI state is not an authorization or safety boundary. + +The admin displays: + +- Label name and localized definition. +- Issuing labeler. +- Public assessment summary and coverage. +- Policy/model version. +- Whether the decision is automated or manually overridden. +- Reconsideration contact for blocked publishers. + +Query/cache keys must include accepted labeler configuration. A policy change must not leave stale search/detail data in React Query caches. + +Pinned CLI installs may offer a deliberate override for warning labels, but never for missing assessment, blocking labels, or `!takedown` under the official policy unless a site operator explicitly changes trusted-labeler policy at deployment level. + +## 18. Publisher Notifications and Reconsideration + +### 18.1 Contact resolution + +Resolve contacts in order: + +1. Package `security[]` contact. +2. Package author/contact metadata. +3. Publisher profile contact. + +The service does not attempt to read private PDS account email. + +### 18.2 Notification events + +Notify on: + +- New blocking label. +- New warning label. +- Manual override. +- Retraction of a blocking/warning label. +- Assessment error that remains unresolved past the operational target. +- Emergency takedown. + +Do not send a separate email for every internal retry or unchanged reassessment. + +Email includes subject, label, public summary, public assessment URL, effect in official clients, and reconsideration instructions. It does not include private exploit detail. + +### 18.3 ATProto-visible notice + +There is no generic guaranteed ATProto inbox for arbitrary plugin publishers. V1 should publish an EmDash notification/decision record in the labeler's repository referencing the subject and public assessment, if a small interoperable record shape is justified. Publishers and tools may subscribe to those records. + +This is supplementary. Email is the active notification channel. The spec must not claim that publishing a record guarantees delivery. + +### 18.4 Reconsideration + +V1 documents a monitored email address and includes an opaque assessment ID. An operator locates the assessment, records the publisher's reason in private notes, and may re-run or override. + +There is no formal SLA, reporter portal, or authenticated appeals state machine in v1. Every resulting label action is still audited and the publisher receives the outcome. + +## 19. Evidence, Privacy, and Retention + +### 19.1 Public transparency + +Public summaries explain: + +- What class of issue was found. +- Whether it came from automation or manual action. +- Which assessment coverage completed. +- Which policy/model versions were used. +- Which labels are active. +- How to request reconsideration. + +They do not expose secrets, contact addresses, exploit recipes, or irrelevant source excerpts. + +### 19.2 Private evidence + +Private evidence may include: + +- Model request/response after secret redaction. +- Code excerpts and file hashes. +- Capability-analysis raw results. +- Image-analysis details. +- Operator notes. +- Notification delivery failures. + +Access is limited to reviewer/admin roles and every view of highly sensitive evidence may be audit-logged if practical. + +### 19.3 Retention + +Initial policy: + +- Signed labels and public assessment summaries: indefinite while the service operates. +- Operator audit log: indefinite. +- Normal private assessment evidence: 180 days. +- Evidence supporting an active block/takedown: retain while active plus 365 days. +- Raw fetched bundles: do not retain after assessment unless quarantined as incident evidence. +- Contact addresses: store only for delivery duration; retain a keyed hash for deduplication/audit. +- Inference logs: the `AI` binding runs in-account with no third-party gateway log surface; retain only assessment/policy/model metadata (never raw payloads), consistent with private evidence policy. + +Retention values are policy constants versioned in code and may change after legal/privacy review. + +## 20. Security Requirements + +### 20.1 Threats and mitigations + +| Threat | Mitigation | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| Prompt injection in source/metadata | Delimit untrusted input, strict system task, structured schema, category allowlist, evidence-reference validation | +| Model false positive blocks legitimate release | Narrow blockable categories, fixture calibration, public explanation, immediate reviewer retraction, immutable audit trail | +| Model false negative | Defense in depth: deterministic checks, capability analysis, sandbox enforcement, reports in future, reassessment on model/policy revision | +| Malicious artifact attacks fetcher/extractor | Mirror preference, checksum verification, SSRF controls, streaming caps, safe archive parsing, time budgets | +| Aggregator serves wrong artifact | Verify signed release record, CID, and artifact checksum independently | +| Publisher mutates artifact URL bytes | Checksum binds accepted bytes; mismatch blocks assessment and installation | +| Forged Jetstream event | Re-fetch and verify signed source record before assessment | +| Forged label | DRISL signature verification against `#atproto_label` | +| Signing-key exfiltration through assessor | Typed policy proposal boundary; key accessible only to issuer module; Secrets Store | +| Access header spoofing | Verify Access JWT signature, issuer, audience, expiry; do not trust raw identity headers | +| Reviewer account compromise | Role limits, required reasons, confirmation ceremony, alerts, immutable action log | +| Admin incorrectly issues `!takedown` | Explicit high-friction confirmation, immediate alert, signed negation rollback | +| Replay/out-of-order labels | Monotonic sequence for stream and latest-`cts` current-state projection | +| Stale labels after CID update | Automated labels are CID-bound; clients check CID applicability | +| Package/publisher overreach | Broad labels are manual only; subject-wide action shown explicitly | +| Assessment absence mistaken for safety | Official clients require positive `assessment-passed` | +| Labeler outage blocks ecosystem indefinitely | Clear pending/error state, alerts, retry/DLQ, site operators can choose a different trust policy/labeler | +| Policy/model drift changes outcomes silently | Version all policy/model/prompt inputs; no automatic whole-catalog rewrite | +| Evidence leaks vulnerability details | Public/private split and redacted summaries | + +### 20.2 Signing proposal validation + +Before signing, the issuer verifies: + +- `val` is in the deployed policy vocabulary or a protocol-reserved value allowed for the actor role. +- Automated actions target release records only and include CID. +- Automated blocking labels correspond to a critical finding in an allowed security/impersonation category. +- Quality findings cannot map to blocking labels. +- `!takedown` and publisher-wide labels come from an admin action. +- Manual `assessment-passed` and `assessment-overridden` labels come from the same reviewer/admin unblock action, target an exact release URI + CID, and atomically negate the blocking labels named by that action. +- Manual action has a verified actor, role, reason, and idempotency key. +- Subject URI/DID parses and belongs to an allowed collection/type. + +This enforcement is code, not prompt text. + +## 21. Reliability and Recovery + +### 21.1 Targets + +Initial operational targets, not protocol guarantees: + +- New release observed from Jetstream: under 30 seconds p95. +- Pending label issued: under 60 seconds p95. +- Normal assessment completed: under 10 minutes p95. +- Emergency manual label visible to connected subscribers: under 10 seconds p95. +- Public label/query availability: 99.9% monthly target. + +At a handful of releases per day, correctness and debuggability matter more than throughput. + +### 21.2 Retry policy + +- Record/artifact fetch: bounded exponential retry; distinguish permanent checksum/validation failure from transient network failure. +- Model calls: bounded retry on transport/rate/availability errors; never reinterpret an execution error as a malicious finding. +- Notifications: independent retry and DLQ; notification failure does not roll back a label. +- Label broadcast: D1 commit is authoritative; disconnected subscribers replay by cursor. + +### 21.3 Reconciliation + +A scheduled reconciliation job: + +- Compares recently observed aggregator releases with labeler subjects. +- Finds releases lacking any assessment state. +- Finds pending assessments older than the target. +- Validates that each completed effective assessment has the expected current labels. +- Verifies subscriber high-water sequence against D1. +- Re-resolves the labeler DID document and signing key publication. + +The independent Jetstream consumer remains the primary discovery path. Reconciliation may use the aggregator as a completeness cross-check without making it the authority for source records. + +### 21.4 Disaster recovery + +Back up D1 and private evidence metadata regularly. Recovery order: + +1. Restore D1 and signing key version mapping. +2. Restore public query endpoints. +3. Restore subscriber stream and verify sequence continuity. +4. Resume Jetstream discovery from persisted cursor. +5. Reconcile missing releases. +6. Resume assessment workflows and notifications. + +If sequence continuity cannot be preserved, publish an operational incident and require subscribers to replay from cursor `0` after the restored history is available. + +## 22. Observability + +Emit metrics for: + +- Jetstream connection state and cursor age. +- Releases observed, deduplicated, queued, running, completed, blocked, warned, errored. +- Assessment duration by stage. +- Artifact mirror hit/fallback/failure and checksum mismatch. +- Model error rate, latency, token use, and cost. +- Finding and label counts by category and policy/model version. +- Operator overrides and automated-label retraction rate. +- Issued label sequence and Subscriber DO connection count. +- Aggregator subscriber lag where observable. +- Notification delivery state. +- DLQ depth and oldest age. + +Alert on: + +- Jetstream disconnected or cursor stale for more than five minutes. +- Pending assessment older than 20 minutes. +- Any sustained model failure causing assessment errors. +- Signing failure or DID/key mismatch. +- Subscriber endpoint failure. +- Emergency `!takedown` issuance. +- Spike in automated blocks or operator override rate. +- Queue/DLQ growth. +- Reconciliation finds an unassessed release. + +Inference logging should include assessment ID, policy version, stage, and model route, but not publisher contact or secrets. + +## 23. Policy and Model Change Process + +Policy, prompts, schemas, label definitions, model routes, and thresholds are versioned source files reviewed like code. + +A change requires: + +1. New immutable policy version. +2. Fixture corpus evaluation. +3. Comparison against the prior production version. +4. Explicit review of newly blocked and newly allowed fixtures. +5. Staging run against a representative historical sample. +6. Deployment notes stating whether historical reassessment is required. +7. Monitoring of block and override rates after rollout. + +The fixture corpus includes the legacy marketplace cases plus new cases for: + +- Legitimate network use. +- Data exfiltration. +- Credential harvesting. +- Crypto mining/resource abuse. +- Obfuscation. +- Prompt injection. +- Dynamic URL construction. +- Misleading screenshots. +- Brand impersonation. +- Declared-vs-actual capability mismatch. +- Bundle exceeding the model context window (coverage degrades). +- Generated but functional plugin. +- Low-effort placeholder. +- False-positive-sensitive security tooling. + +Tests cannot call a nondeterministic live model in normal CI. CI validates parsers, policy mapping, fixtures with recorded model outputs, and optional scheduled calibration runs. Model upgrades require a recorded calibration report. + +## 24. Testing Strategy + +### 24.1 Unit tests + +- Label DRISL signing and verification against external vectors. +- Strict signing-field allowlist. +- Label vocabulary and role/subject restrictions. +- Finding normalization and policy mapping. +- Security/impersonation critical block behavior. +- Quality critical warning-only behavior. +- CID applicability, negation, expiry, and supersession. +- Access JWT verification and role mapping. +- CSRF/idempotency enforcement. +- Public/private assessment serialization. +- Contact resolution and redaction. + +### 24.2 Real D1/workerd tests + +- Migrations and constraints. +- Atomic sequence allocation under concurrency. +- Idempotent discovery/assessment enqueue. +- Assessment state transitions. +- Issuance plus action audit transaction. +- Subscriber replay and reconnect from cursor. +- Key rotation query re-signing. +- Notification outbox retries. + +### 24.3 Integration tests + +- Publish release record -> Jetstream event -> verified record fetch -> mirrored bundle -> assessment -> labels -> aggregator ingest -> admin eligibility. +- No positive assessment -> install blocked. +- Pass plus warning -> install confirmation required. +- Blocking descriptive label -> search/detail visible with explanation, install blocked. +- Admin `!takedown` -> redacted response for `redact` consumer. +- Negation -> release becomes eligible on next primary read where `assessment-passed` is active. +- New release CID -> old CID labels do not apply. +- Package/publisher manual label cascades through aggregator policy. +- Artifact mirror unavailable -> safe declared-URL fallback. +- Checksum mismatch -> no AI call and blocking deterministic result. +- Model outage -> retries then `assessment-error`, never a malicious label. + +### 24.4 Adversarial tests + +- Source code instructs the model to output pass/critical/arbitrary labels. +- Findings reference nonexistent evidence spans. +- Archive bomb, path traversal, symlink, huge sparse file, malformed gzip/tar. +- Redirect chain to private/link-local/metadata addresses. +- DNS rebinding between validation and fetch. +- Image parser bombs and MIME confusion. +- Forged release event and forged record proof. +- Forged label, stale signing key, and malicious extra signed fields. +- Out-of-order positive/negation delivery. +- Compromised reviewer attempts admin-only action. +- Spoofed Access identity header without valid JWT. +- Duplicate mutation/idempotency replay. +- Publisher edits profile during assessment. +- Release deletion during assessment. + +### 24.5 UI tests + +- Reviewer and admin action availability. +- Confirmation ceremony and required reason. +- Public/private evidence separation. +- Pending/error/blocked/warned states. +- Arabic RTL layout and keyboard navigation. +- Localized label definitions and operator strings. + +## 25. Delivery Plan + +### Phase 0: Contract freeze + +- Ratify label vocabulary, subject rules, and official-client effects. +- Cut current aggregator/admin/core constants and SQL from `security:yanked` to `security-yanked`, with a production preflight for persisted legacy labels. +- Define the complete blocking-label set and the single registry-client moderation helper contract. +- Add experimental labeler lexicons and policy schema. +- Publish signing/verification test vectors. +- Decide the standard labeler declaration/PDS hosting detail. + +### Phase 1: Issuer and distribution + +- Create `apps/labeler` Worker, D1, DID document, signing key, query endpoint, and Subscriber DO. +- Implement typed internal issuance, negation, sequence allocation, key IDs, and audit actions. +- Add a minimal admin-only CLI/API for test issuance before the console. +- Implement aggregator subscription, signature verification, label state, subject expansion, and RFC-8941 request-header parsing. +- Set and CORS-expose `atproto-content-labelers`; add missing/empty/malformed/repeated/unavailable/redact contract tests. +- Prove a manual label reaches aggregator and client end to end. + +### Phase 2: Eligibility contract + +- Add `assessment-pending`, `assessment-passed`, and `assessment-error` handling. +- Make aggregator latest-release selection label-aware. +- Add registry-client moderation helpers. +- Make the shared helper the only install/update eligibility gate and enforce positive assessment in both handlers. +- Update admin search/detail/install UI. + +This phase may initially use deterministic test assessments. It freezes the safety contract before AI is involved. + +### Phase 3: Automated assessment + +- Add independent Jetstream discovery and reconciliation. +- Add verified mirror artifact fetch and safe fallback. +- Port/harden deterministic checks, capability analysis, code, metadata, and image audit inputs. +- Add Workers AI (`@cf/moonshotai/kimi-k2.7-code`) through the `AI` binding (`env.AI`) and versioned policy resolution. +- Port and extend the fixture corpus. + +### Phase 4: Operator product + +- Add Access JWT validation and reviewer/admin mapping. +- Build Kumo console. +- Add override, re-run, `security-yanked`, package action, and admin emergency takedown flows. +- Add notifications and reconsideration handling. +- Add operational dashboards and alerts. + +### Phase 5: Production launch + +- Assess all first-party releases with no exemption. +- Run historical staging calibration. +- Complete key backup/rotation and disaster-recovery drill. +- Verify full replay from cursor `0` into a fresh aggregator. +- Verify a fresh self-hosted aggregator can subscribe and enforce labels. +- Publish policy, contact, privacy, and reconsideration documentation. +- Enable positive-assessment requirement in official production clients. + +## 26. Definition of Done + +The v1 labelling service is complete when: + +- Every new release is independently observed and receives a CID-bound assessment state. +- Official clients require a signed positive assessment and block configured descriptive security labels. +- Deterministic, capability-analysis, code/metadata AI, image, and publisher-history inputs run under a versioned policy. +- AI critical findings can hard-block only security/impersonation categories. +- Quality findings remain warning/downranking signals. +- Labels are spec-compliant, signed, queryable, replayable, negatable, and verified by the aggregator. +- The aggregator correctly parses accepted labelers, hydrates labels, handles redaction, and selects eligible latest releases. +- Reviewers can inspect and override assessments; admins can issue emergency takedowns. +- Cloudflare Access JWTs are verified in the Worker and roles are enforced server-side. +- Public assessment summaries explain decisions without exposing private evidence. +- Publishers receive notices through security/profile contacts and have a documented reconsideration route. +- Signing-key rotation, subscriber replay, model outage, artifact failure, and disaster recovery have been exercised. +- A third-party fresh aggregator can consume the public labeler without private EmDash infrastructure. + +## 27. Deferred Follow-ups + +- Standard `com.atproto.moderation.createReport` intake and a report-review queue. +- Authenticated publisher appeals with status tracking. +- Two-person approval or expiring emergency holds for high-risk manual actions. +- Automated package/publisher escalation for repeated offenses. +- Additional independent/community labelers and site-level trust-policy UI. +- Publisher self-requested security yanks through authenticated repository records. +- Formal ATProto notification record if the ecosystem converges on one. +- Automated whole-catalog reassessment on major policy/model versions. +- Native plugin/npm assessment. +- Multi-tenant labelling SaaS. +- Hardware-backed or remote signing service if Secrets Store custody is insufficient. + +## 28. Remaining Ratification Points + +These are implementation-specific decisions that should be settled before Phase 0 closes, not product questions requiring another broad design round. + +1. The measured calibration threshold used to assign model severity/confidence for `@cf/moonshotai/kimi-k2.7-code`. +2. The initial model and the exact critical block rule for code/capability findings. +3. The production key-generation ceremony, performed by the maintainer before launch. The scalar format is ratified in the Gate 0 crypto contract; the offline copy lives in the maintainer's Keeper vault. +4. The monitored reconsideration address. Email delivery is decided: Cloudflare Email Sending through the Workers `send_email` binding, sending from an onboarded emdashcms.com address. +5. Final retention values after legal/privacy review. + +The public NSIDs and the no-decision-notice decision were ratified in the Gate 0 contracts. diff --git a/apps/aggregator/migrations/0003_label_history_identity.sql b/apps/aggregator/migrations/0003_label_history_identity.sql new file mode 100644 index 0000000000..93436f3ee8 --- /dev/null +++ b/apps/aggregator/migrations/0003_label_history_identity.sql @@ -0,0 +1,79 @@ +-- Collision-safe label history identity + integer-comparable timestamps. +-- +-- `labels`/`label_state` are empty in every deployment (no writer exists yet; +-- production preflight confirmed zero rows), so this drops and recreates both +-- rather than migrating data. Primary key becomes the SHA-256 digest of the +-- canonical signed-label bytes (`encodeSignedLabel`), so exact redelivery is a +-- silent no-op; `UNIQUE (src, source_sequence, frame_index)` catches a +-- different label landing at coordinates the ingestor already used. `cts`/`exp` +-- gain epoch-millisecond columns because RFC 3339 strings compare incorrectly +-- across timezone offsets in SQL. +-- Guard: refuse to run against a deployment that has label rows. The +-- reference deployment was preflight-confirmed empty, but a self-hosted +-- aggregator may not be. The CHECK is unsatisfiable, so the INSERTs only +-- succeed when their SELECTs return no rows. +CREATE TABLE _label_migration_guard (never INTEGER CHECK (never IS NULL AND never IS NOT NULL)); +INSERT INTO _label_migration_guard SELECT 1 FROM labels LIMIT 1; +INSERT INTO _label_migration_guard SELECT 1 FROM label_state LIMIT 1; +DROP TABLE _label_migration_guard; + +DROP TABLE labels; +DROP TABLE label_state; + +-- Append-only label history. Every label received is written here, including +-- negations. Current state is derived from latest cts per (src, uri, val) and +-- projected into label_state below for hot-path lookups. +CREATE TABLE labels ( + digest TEXT PRIMARY KEY, -- SHA-256 hex of encodeSignedLabel(label) + src TEXT NOT NULL, -- labeler DID + uri TEXT NOT NULL, -- AT URI of subject + cid TEXT, -- optional version-specific CID + val TEXT NOT NULL, -- e.g. 'security-yanked', '!takedown' + neg INTEGER NOT NULL DEFAULT 0, + cts TEXT NOT NULL, + cts_epoch_ms INTEGER NOT NULL, + exp TEXT, -- optional expiry (RFC 3339) + exp_epoch_ms INTEGER, + sig BLOB NOT NULL, -- raw signature for client re-verification + ver INTEGER NOT NULL DEFAULT 1, + source_sequence INTEGER NOT NULL, -- subscribeLabels frame seq + frame_index INTEGER NOT NULL, -- index of this label within the frame's labels array + trusted INTEGER NOT NULL DEFAULT 0, + received_at TEXT NOT NULL, + UNIQUE (src, source_sequence, frame_index) +); + +CREATE INDEX idx_labels_subject ON labels(uri); +CREATE INDEX idx_labels_latest ON labels(src, uri, val, cts_epoch_ms DESC); + +-- Latest-state projection: one row per (src, uri, val) holding the most recent +-- cts seen, including the neg flag and exp timestamp. Updated in the same +-- batch as the labels insert above. Query-time filters apply +-- `neg = 0 AND (exp_epoch_ms IS NULL OR exp_epoch_ms > now())` to determine +-- whether a label is currently in force. +-- +-- Why retain rows for negated/expired labels rather than deleting them: an +-- out-of-order delivery (a positive label arriving after its negation) could +-- otherwise reinsert a row we'd already retracted. Keeping the row with its +-- `cts_epoch_ms` lets the upsert reject the older positive. +CREATE TABLE label_state ( + src TEXT NOT NULL, + uri TEXT NOT NULL, + val TEXT NOT NULL, + cid TEXT, + neg INTEGER NOT NULL DEFAULT 0, + cts TEXT NOT NULL, + cts_epoch_ms INTEGER NOT NULL, + exp TEXT, + exp_epoch_ms INTEGER, + digest TEXT NOT NULL, + source_sequence INTEGER NOT NULL, + frame_index INTEGER NOT NULL, + trusted INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (src, uri, val) +); + +-- Hot path for hard filters (yanked, takedown, etc.) from trusted issuers. +-- Partial index keeps the index small by storing only currently-active rows. +CREATE INDEX idx_label_state_enforce ON label_state(uri, val, trusted) + WHERE neg = 0 AND trusted = 1; diff --git a/apps/aggregator/migrations/0004_rename_labellers_to_labelers.sql b/apps/aggregator/migrations/0004_rename_labellers_to_labelers.sql new file mode 100644 index 0000000000..9dfa867549 --- /dev/null +++ b/apps/aggregator/migrations/0004_rename_labellers_to_labelers.sql @@ -0,0 +1,12 @@ +-- Rename `labellers` -> `labelers` to match the spelling every query in +-- `src/` uses (`labeler-resolver.ts`, `labels-consumer.ts`, +-- `request-policy.ts`, `index.ts`). `0001_init.sql` shipped the table as +-- `labellers` and is immutable once applied, so the correction has to land as +-- a forward migration: databases provisioned by `0001` keep `labellers` and +-- only this migration brings them to `labelers`. +-- +-- No indexes, triggers, or views reference the table, so SQLite's +-- `ALTER TABLE ... RENAME TO` is sufficient — there are no explicitly-named +-- objects embedding the old spelling to recreate. + +ALTER TABLE labellers RENAME TO labelers; diff --git a/apps/aggregator/package.json b/apps/aggregator/package.json index aa5cc00397..7137b0efbf 100644 --- a/apps/aggregator/package.json +++ b/apps/aggregator/package.json @@ -31,7 +31,9 @@ "@atcute/repo": "catalog:", "@atcute/xrpc-server": "catalog:", "@atcute/xrpc-server-cloudflare": "catalog:", - "@emdash-cms/registry-lexicons": "workspace:*" + "@emdash-cms/registry-lexicons": "workspace:*", + "@emdash-cms/registry-moderation": "workspace:*", + "emdash": "workspace:*" }, "devDependencies": { "@cloudflare/vite-plugin": "catalog:", diff --git a/apps/aggregator/src/index.ts b/apps/aggregator/src/index.ts index 854bcd6519..1c34f10fad 100644 --- a/apps/aggregator/src/index.ts +++ b/apps/aggregator/src/index.ts @@ -20,6 +20,8 @@ import { isDid } from "@atcute/lexicons/syntax"; import { drainBackfillDeadLetterBatch, processBackfillBatch } from "./backfill-consumer.js"; import { discoverDids, enqueueBackfillJobs } from "./backfill.js"; import type { BackfillJob, RecordsJob } from "./env.js"; +import type { LabelIngestJob } from "./label-ingest-types.js"; +import { drainLabelsDeadLetterBatch, processLabelsBatch } from "./labels-consumer.js"; import { drainDeadLetterBatch, processBatch } from "./records-consumer.js"; import { RECORDS_DO_NAME } from "./records-do.js"; import { handleXrpc } from "./routes/xrpc/router.js"; @@ -29,7 +31,10 @@ const RECORDS_QUEUE_NAME = "emdash-aggregator-records"; const RECORDS_DLQ_NAME = "emdash-aggregator-records-dlq"; const BACKFILL_QUEUE_NAME = "emdash-aggregator-backfill"; const BACKFILL_DLQ_NAME = "emdash-aggregator-backfill-dlq"; +const LABELS_QUEUE_NAME = "emdash-aggregator-labels"; +const LABELS_DLQ_NAME = "emdash-aggregator-labels-dlq"; +export { LabelIngestDO } from "./label-ingest-do.js"; export { RecordsJetstreamDO } from "./records-do.js"; /** @@ -286,6 +291,14 @@ export default { // eslint-disable-next-line typescript/no-unsafe-type-assertion -- narrowed by queue name drainBackfillDeadLetterBatch(batch as MessageBatch, env); return; + case LABELS_QUEUE_NAME: + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- narrowed by queue name + await processLabelsBatch(batch as MessageBatch, env); + return; + case LABELS_DLQ_NAME: + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- narrowed by queue name + await drainLabelsDeadLetterBatch(batch as MessageBatch, env); + return; default: console.error("[aggregator] unknown queue, acking batch", { queue: batch.queue }); for (const m of batch.messages) m.ack(); @@ -302,9 +315,41 @@ export default { const id = env.RECORDS_DO.idFromName(RECORDS_DO_NAME); const stub = env.RECORDS_DO.get(id); ctx.waitUntil(stub.fetch("https://do.internal/liveness")); + + // Same liveness pump for every configured labeler's ingest DO. One + // instance per DID (`getByName`); `trusted` gates enforcement + // elsewhere, so every configured labeler gets a subscription + // regardless of that flag. + ctx.waitUntil(wakeLabelIngestDOs(env)); }, }; +async function wakeLabelIngestDOs(env: Env): Promise { + try { + const rows = await env.DB.prepare(`SELECT did FROM labelers`).all<{ did: string }>(); + // Awaited collectively so waitUntil keeps the invocation alive until + // every wake lands; the per-fetch catch keeps one unreachable DO from + // failing the rest. + await Promise.all( + (rows.results ?? []).map((row) => + env.LABEL_INGEST_DO.getByName(row.did) + .fetch(`https://do.internal/wake?did=${encodeURIComponent(row.did)}`) + .then(() => undefined) + .catch((err: unknown) => { + console.error("[aggregator] label ingest DO wake failed", { + did: row.did, + error: err instanceof Error ? err.message : String(err), + }); + }), + ), + ); + } catch (err) { + console.error("[aggregator] label ingest DO wake pump failed", { + error: err instanceof Error ? err.message : String(err), + }); + } +} + type BackfillRequestParsed = { mode: "explicit"; dids: string[] } | { mode: "discover" }; /** diff --git a/apps/aggregator/src/label-ingest-do.ts b/apps/aggregator/src/label-ingest-do.ts new file mode 100644 index 0000000000..00d25a6d72 --- /dev/null +++ b/apps/aggregator/src/label-ingest-do.ts @@ -0,0 +1,154 @@ +/** + * Label ingest DO: holds a long-lived outbound WebSocket to one labeler's + * `subscribeLabels` stream. One instance per labeler, named by DID + * (`env.LABEL_INGEST_DO.getByName(did)`) — see `index.ts`'s `scheduled()`. + * + * Unlike the records DO (a fixed singleton), this DO doesn't know its DID + * until something tells it. The `did` isn't part of the binding name lookup + * key material available inside the DO itself, so the first `fetch()` (the + * cron wake, `?did=...`) supplies it; the DO persists it in its own storage + * so later wakes/restarts don't need the query param. A DO instance that has + * never been woken with a `did` refuses to run. + * + * The DO is thin by design — all the loop / verification / cursor / backoff + * logic lives in `LabelIngestor`. The DO just wires real bindings in and + * exposes a health check. + */ + +import { + AtprotoWebDidDocumentResolver, + CompositeDidDocumentResolver, + PlcDidDocumentResolver, +} from "@atcute/identity-resolver"; +import { DurableObject } from "cloudflare:workers"; + +import { LabelIngestor, type LabelCursorStore } from "./label-ingestor.js"; +import { RealLabelStreamClient } from "./label-stream-client.js"; +import { createD1LabelerIdentityCache, LabelerResolver } from "./labeler-resolver.js"; +import { boundFetch } from "./utils.js"; + +const DID_STORAGE_KEY = "labelIngest:did"; + +/** D1-backed cursor store for one labeler, keyed `labeler:` in the + * shared `ingest_state` table (also used by the Jetstream cursor). Cursor is + * stored as TEXT — `ingest_state.cursor` holds both Jetstream's microsecond + * `time_us` and this frame `seq`, so the column stays a string for either + * shape. */ +function createD1LabelCursorStore(db: D1Database, did: string): LabelCursorStore { + const source = `labeler:${did}`; + return { + async get(): Promise { + const row = await db + .prepare(`SELECT cursor FROM ingest_state WHERE source = ?`) + .bind(source) + .first<{ cursor: string }>(); + if (!row) return undefined; + const cursor = Number(row.cursor); + return Number.isSafeInteger(cursor) ? cursor : undefined; + }, + async put(cursor: number): Promise { + await db + .prepare( + `INSERT INTO ingest_state (source, cursor, updated_at) + VALUES (?, ?, datetime('now')) + ON CONFLICT(source) DO UPDATE SET + cursor = excluded.cursor, + updated_at = excluded.updated_at`, + ) + .bind(source, String(cursor)) + .run(); + }, + }; +} + +export class LabelIngestDO extends DurableObject { + private did: string | null = null; + private ingestor: LabelIngestor | null = null; + /** Held so the run loop isn't garbage-collected. */ + private runPromise: Promise | null = null; + + constructor(state: DurableObjectState, env: Env) { + super(state, env); + // Blocks fetch() until the persisted DID (if any) is loaded, so a + // restarted DO resumes its labeler without needing the wake request + // to repeat `?did=`. + state + .blockConcurrencyWhile(async () => { + const stored = await state.storage.get(DID_STORAGE_KEY); + if (stored) this.start(stored); + }) + .catch((err: unknown) => { + console.error("[aggregator] label ingest DO storage bootstrap failed", { + error: err instanceof Error ? err.message : String(err), + }); + }); + } + + /** + * Health surface for the cron wake pump. `consecutiveFailures: 0` means + * the most recent connection attempt fully processed at least one frame; + * non-zero means the labeler is unreachable, its signing key can't be + * resolved, or every label in the current frame is failing verification. + */ + override async fetch(request: Request): Promise { + const url = new URL(request.url); + const requestedDid = url.searchParams.get("did"); + if (!this.did) { + if (!requestedDid) { + return Response.json({ error: "missing did" }, { status: 400 }); + } + await this.ctx.storage.put(DID_STORAGE_KEY, requestedDid); + this.start(requestedDid); + } else if (requestedDid && requestedDid !== this.did) { + // The DO's name is derived from `did` (`getByName(did)`), so this + // should be unreachable in production. Log rather than crash — an + // operational script hitting the wrong DO by hand shouldn't panic + // an otherwise-healthy ingestor. + console.error("[aggregator] label ingest DO wake did mismatch", { + did: this.did, + requestedDid, + }); + } + return Response.json({ + did: this.did, + cursor: this.ingestor?.currentCursor ?? null, + consecutiveFailures: this.ingestor?.consecutiveFailures ?? 0, + }); + } + + private start(did: string): void { + if (this.ingestor) return; + this.did = did; + const resolver = new LabelerResolver({ + cache: createD1LabelerIdentityCache(this.env.DB), + resolver: new CompositeDidDocumentResolver({ + methods: { + plc: new PlcDidDocumentResolver({ fetch: boundFetch }), + web: new AtprotoWebDidDocumentResolver({ fetch: boundFetch }), + }, + }), + }); + this.ingestor = new LabelIngestor({ + did, + client: new RealLabelStreamClient(), + queue: this.env.LABELS_QUEUE, + cursorStore: createD1LabelCursorStore(this.env.DB, did), + resolver, + logger: { + warn: (msg, ctx) => console.warn(`[aggregator] ${msg}`, { did, ...ctx }), + error: (msg, ctx) => console.error(`[aggregator] ${msg}`, { did, ...ctx }), + }, + }); + // Fire-and-forget. The run loop absorbs every recoverable error path + // internally (transient queue failures, connection drops, verification + // failures all retry with backoff). The catch is here defensively — if + // a future change introduces a non-recoverable rejection, we want it in + // the logs rather than as an unhandled promise. + this.runPromise = this.ingestor.run().catch((err) => { + console.error("[aggregator] label ingestor crashed", { + did, + error: err instanceof Error ? err.message : String(err), + }); + }); + } +} diff --git a/apps/aggregator/src/label-ingest-types.ts b/apps/aggregator/src/label-ingest-types.ts new file mode 100644 index 0000000000..b6484a3979 --- /dev/null +++ b/apps/aggregator/src/label-ingest-types.ts @@ -0,0 +1,30 @@ +import { fromBase64Url, toBase64Url } from "@atcute/multibase"; +import type { SignedLabel } from "@emdash-cms/registry-moderation"; + +/** A `SignedLabel` with `sig` as an unpadded base64url string for queue serialization. */ +export interface SignedLabelWire extends Omit { + sig: string; +} + +/** + * One label queued for durable history + projection writes. Enqueued only + * after `label-ingestor.ts` verifies the signature; `sourceSequence` and + * `frameIndex` pin the label's coordinates in the source labeler's stream for + * the `(src, source_sequence, frame_index)` collision check. + */ +export interface LabelIngestJob { + src: string; + sourceSequence: number; + frameIndex: number; + label: SignedLabelWire; +} + +export function toWire(label: SignedLabel): SignedLabelWire { + const { sig, ...rest } = label; + return { ...rest, sig: toBase64Url(sig) }; +} + +export function fromWire(wire: SignedLabelWire): SignedLabel { + const { sig, ...rest } = wire; + return { ...rest, sig: fromBase64Url(sig) }; +} diff --git a/apps/aggregator/src/label-ingestor.ts b/apps/aggregator/src/label-ingestor.ts new file mode 100644 index 0000000000..a9e6eca0ea --- /dev/null +++ b/apps/aggregator/src/label-ingestor.ts @@ -0,0 +1,308 @@ +/** + * Signed-label ingestor. Subscribes to one labeler's `subscribeLabels` + * stream, verifies every label against the labeler's resolved `#atproto_label` + * key, enqueues verified labels, and persists the cursor so reconnects resume + * cleanly. + * + * Owns: + * - Connection lifecycle (connect → consume → disconnect → backoff → + * reconnect, indefinitely until `stop()` is called). Same shape as + * `JetstreamIngestor`. + * - Fail-closed verification: a label is enqueued only after its signature + * verifies against the configured labeler's key. One resolver-refresh + * retry per connection absorbs a key rotation landing mid-stream; a + * second failure closes the connection rather than skip the label — + * the labeler is untrusted from that point until it reconnects. + * - Cursor persistence after every label in a frame has been enqueued. + * Persisting only after all sends resolve means a crash mid-frame + * replays the whole frame on reconnect; the consumer's digest-keyed + * `ON CONFLICT DO NOTHING` absorbs the duplicate. + * - Exponential backoff with jitter, capped, reset on each connection + * attempt that made progress. + * + * Pure constructor injection — no DO/D1/Queue/cloudflare:workers imports — + * so unit tests instantiate it directly with a fake `LabelStreamClient` + an + * in-memory queue + a `Map`-backed cursor store + a fake resolver. + */ + +import { + parseSignedLabel, + verifyLabelWithPublicKey, + type SignedLabel, +} from "@emdash-cms/registry-moderation"; + +import { toWire } from "./label-ingest-types.js"; +import type { LabelIngestJob } from "./label-ingest-types.js"; +import type { LabelStreamClient, LabelStreamHandle } from "./label-stream-client.js"; +import type { ResolvedLabelerIdentity } from "./labeler-resolver.js"; + +/** Subset of `Queue.send` we use; loose return type because workerd's + * `Queue.send` resolves to `QueueSendResponse` while a hand-rolled in-memory + * test queue resolves to `void` — neither matters to the ingestor. */ +export interface LabelJobQueue { + send(job: LabelIngestJob): Promise; +} + +/** Persisted cursor is the last fully processed frame `seq`. `get()` returns + * `undefined` when nothing has been persisted yet (fresh labeler, DO + * restart with empty D1 row) — the ingestor subscribes with `cursor: 0`. */ +export interface LabelCursorStore { + get(): Promise; + put(cursor: number): Promise; +} + +/** Narrow slice of `LabelerResolver` the ingestor needs. */ +export interface LabelResolver { + resolve(did: string): Promise; + /** Bypasses the cache for the single per-connection retry after a + * verification failure. */ + resolveFresh(did: string): Promise; +} + +export interface LabelIngestorBackoffConfig { + /** Initial delay after the first disconnect (ms). Default 1s. */ + initialDelayMs?: number; + /** Cap (ms). Default 60s. */ + maxDelayMs?: number; + /** Multiplier per retry. Default 2. */ + multiplier?: number; + /** ±jitter as a fraction of the delay. Default 0.2 (±20%). 0 disables. */ + jitter?: number; +} + +export interface LabelIngestorLogger { + info?(msg: string, ctx?: Record): void; + warn?(msg: string, ctx?: Record): void; + error?(msg: string, ctx?: Record): void; +} + +export interface LabelIngestorOptions { + /** The labeler DID this ingestor serves. Every label's `src` must equal + * this exactly; a mismatch is treated as a verification failure. */ + did: string; + client: LabelStreamClient; + queue: LabelJobQueue; + cursorStore: LabelCursorStore; + resolver: LabelResolver; + backoff?: LabelIngestorBackoffConfig; + logger?: LabelIngestorLogger; + /** Sleep impl, swap in tests to skip real backoff waits. */ + sleep?: (ms: number) => Promise; + /** Random source for jitter, swap in tests for determinism. */ + random?: () => number; +} + +const DEFAULT_BACKOFF: Required = { + initialDelayMs: 1_000, + maxDelayMs: 60_000, + multiplier: 2, + jitter: 0.2, +}; + +export class LabelIngestor { + private readonly did: string; + private readonly client: LabelStreamClient; + private readonly queue: LabelJobQueue; + private readonly cursorStore: LabelCursorStore; + private readonly resolver: LabelResolver; + private readonly backoff: Required; + private readonly logger: LabelIngestorLogger; + private readonly sleep: (ms: number) => Promise; + private readonly random: () => number; + + private stopped = false; + private currentSub: LabelStreamHandle | null = null; + private cursor = 0; + /** Set once a frame's labels are fully enqueued and its cursor persisted. + * Read by the run loop to decide whether to reset the backoff counter. */ + private madeProgress = false; + + private _consecutiveFailures = 0; + + constructor(opts: LabelIngestorOptions) { + this.did = opts.did; + this.client = opts.client; + this.queue = opts.queue; + this.cursorStore = opts.cursorStore; + this.resolver = opts.resolver; + this.backoff = { ...DEFAULT_BACKOFF, ...opts.backoff }; + this.logger = opts.logger ?? {}; + this.sleep = opts.sleep ?? defaultSleep; + this.random = opts.random ?? Math.random; + } + + /** The frame `seq` most recently enqueued + persisted. `0` until the + * first frame or if nothing was ever persisted. */ + get currentCursor(): number { + return this.cursor; + } + + /** Number of consecutive failed/empty connection attempts. `0` means the + * most recent attempt produced at least one fully processed frame. */ + get consecutiveFailures(): number { + return this._consecutiveFailures; + } + + /** + * Run the connect-consume-reconnect loop until `stop()` is called. + * + * Resolves when `stop()` is called and the current subscription drains. + * Does NOT reject for transient failures — connection drops, decode + * errors, verification failures, and queue.send rejections all increment + * the backoff counter and retry. The DO observes liveness via + * `currentCursor` and `consecutiveFailures`. + */ + async run(): Promise { + // Loaded inside the retry loop: a transient D1 failure at DO cold start + // must count as a failed attempt and retry, not kill the loop for the + // lifetime of the DO instance. + let cursorLoaded = false; + + while (!this.stopped) { + try { + if (!cursorLoaded) { + this.cursor = (await this.cursorStore.get()) ?? 0; + cursorLoaded = true; + } + await this.connectAndConsume(); + if (this.madeProgress) this._consecutiveFailures = 0; + else this._consecutiveFailures += 1; + } catch (err) { + if (this.madeProgress) this._consecutiveFailures = 0; + else this._consecutiveFailures += 1; + this.logger.warn?.("label stream subscription failed", { + did: this.did, + error: err instanceof Error ? err.message : String(err), + consecutiveFailures: this._consecutiveFailures, + }); + } + if (this.stopped) break; + await this.sleep(this.computeBackoff(this._consecutiveFailures)); + } + } + + stop(): void { + this.stopped = true; + this.currentSub?.close(); + } + + private async connectAndConsume(): Promise { + this.madeProgress = false; + + const identity = await this.resolver.resolve(this.did); + let publicKey = identity.publicKey; + // One resolver-refresh retry per connection, spent on the first label + // that fails verification. A second failure — either the retry itself, + // or any later label once the retry is spent — closes the connection. + let retryUsed = false; + + const sub = this.client.subscribe({ endpoint: identity.endpoint, cursor: this.cursor }); + this.currentSub = sub; + try { + for await (const event of sub) { + if (this.stopped) break; + + const verified: SignedLabel[] = []; + let verificationClosed = false; + + for (let frameIndex = 0; frameIndex < event.labels.length; frameIndex++) { + const raw = event.labels[frameIndex]; + try { + verified.push(await this.verifyOne(raw, publicKey)); + continue; + } catch (err) { + if (retryUsed) { + this.logger.error?.("label verification failed", { + did: this.did, + seq: event.seq, + frameIndex, + error: err instanceof Error ? err.message : String(err), + }); + verificationClosed = true; + break; + } + retryUsed = true; + try { + const fresh = await this.resolver.resolveFresh(this.did); + publicKey = fresh.publicKey; + verified.push(await this.verifyOne(raw, publicKey)); + continue; + } catch (retryErr) { + this.logger.error?.("label verification failed after key refresh", { + did: this.did, + seq: event.seq, + frameIndex, + error: retryErr instanceof Error ? retryErr.message : String(retryErr), + }); + verificationClosed = true; + break; + } + } + } + + if (verificationClosed) { + // Fail closed: nothing from this frame is enqueued, the + // cursor doesn't move, and the connection ends here (the + // `finally` below closes `sub`). Earlier frames in this + // connection don't count as progress — otherwise a stream + // alternating good and unverifiable frames would reset the + // backoff counter every connection and never escalate. + this.madeProgress = false; + break; + } + + for (let frameIndex = 0; frameIndex < verified.length; frameIndex++) { + await this.queue.send({ + src: this.did, + sourceSequence: event.seq, + frameIndex, + label: toWire(verified[frameIndex]!), + }); + } + // Persist only after every send in the frame has resolved. + this.cursor = event.seq; + await this.cursorStore.put(event.seq); + this.madeProgress = true; + } + } finally { + sub.close(); + if (this.currentSub === sub) this.currentSub = null; + } + } + + private async verifyOne( + raw: unknown, + publicKey: ResolvedLabelerIdentity["publicKey"], + ): Promise { + const label = parseSignedLabel(raw); + if (label.src !== this.did) { + throw new TypeError( + `label.src '${label.src}' does not match configured labeler '${this.did}'`, + ); + } + await verifyLabelWithPublicKey({ label, expectedSource: this.did, publicKey }); + return label; + } + + private computeBackoff(failures: number): number { + // Defensive: `failures` is always >= 1 when called from the run loop + // (the increment happens before computeBackoff), but a future caller + // passing 0 would give `initialDelayMs / multiplier`, which is below + // the floor. Clamp explicitly. + const exp = Math.min( + Math.max( + this.backoff.initialDelayMs, + this.backoff.initialDelayMs * this.backoff.multiplier ** (failures - 1), + ), + this.backoff.maxDelayMs, + ); + if (this.backoff.jitter <= 0) return exp; + const range = exp * this.backoff.jitter; + const offset = (this.random() * 2 - 1) * range; + return Math.max(0, Math.round(exp + offset)); + } +} + +function defaultSleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/apps/aggregator/src/label-stream-client.ts b/apps/aggregator/src/label-stream-client.ts new file mode 100644 index 0000000000..efd7f722e2 --- /dev/null +++ b/apps/aggregator/src/label-stream-client.ts @@ -0,0 +1,280 @@ +/** + * Signed-label subscription transport (`com.atproto.label.subscribeLabels`). + * + * Mirrors the `JetstreamClient`/`RealJetstreamClient` split in + * `jetstream-client.ts`: a narrow interface the ingestor depends on, plus a + * production implementation. Unlike Jetstream (wrapped from `@atcute/jetstream`), + * there's no library for this XRPC subscription — we hand-roll the WebSocket + * over a `fetch` upgrade and the two-CBOR-value frame format the labeler + * writes in `apps/labeler/src/subscribe-labels.ts` (`encodeEvent`). + * + * Frame shape: a CBOR-encoded header object concatenated with a CBOR-encoded + * payload object in one binary WebSocket message — `decodeFirst` twice, no + * length prefix. Two message kinds: + * - `{ op: 1, t: "#labels" }` header, `{ seq, labels }` payload — one + * labeler event. Structurally validated here; label contents stay + * `unknown[]` because parsing/verification is the ingestor's job. + * - `{ op: -1 }` header, `{ error, message }` payload — the labeler is + * refusing the subscription (e.g. `FutureCursor`). Thrown as + * `LabelStreamError`; the ingestor logs and backs off rather than + * resetting its cursor. + * + * An unknown `t` under `op: 1` is ignored for forward compatibility. Anything + * else — bad CBOR, wrong shape, unrecognised `op` — throws, because a label + * stream we can't parse is a stream we must not silently skip past. + */ + +import { decodeFirst } from "@atcute/cbor"; + +import { isPlainObject } from "./utils.js"; + +const MAX_LABELS_PER_FRAME = 200; +/** Cap on decoded frames buffered ahead of the consumer. The labeler bounds + * its own outbound buffer, but that says nothing about our inbound side: a + * fast replay against a consumer stalled on verification or queue sends + * would otherwise grow the buffer without limit. */ +const MAX_BUFFERED_FRAMES = 256; + +export interface LabelStreamEvent { + seq: number; + labels: unknown[]; +} + +export interface LabelStreamSubscribeOptions { + endpoint: string; + /** Always sent explicitly — the labeler's default cursor is "now", which + * would silently skip history. */ + cursor: number; +} + +export interface LabelStreamHandle extends AsyncIterable { + close(): void; +} + +export interface LabelStreamClient { + subscribe(opts: LabelStreamSubscribeOptions): LabelStreamHandle; +} + +/** Carries the labeler's `{ error, message }` payload from an `op: -1` frame. */ +export class LabelStreamError extends Error { + override readonly name = "LabelStreamError"; + constructor( + readonly error: string, + message: string, + ) { + super(message); + } +} + +/** + * Decodes and structurally validates one binary subscribeLabels message. + * Exported as the test seam for `label-stream-client.test.ts` — same role as + * `wrapAtcuteSubscription` in `jetstream-client.ts`: exercise the decode path + * directly instead of standing up a real socket. + * + * Returns `null` for a forward-compatible `op: 1` frame with an unrecognised + * `t` (ignore and keep reading). Throws `LabelStreamError` for `op: -1`. + * Throws a plain `TypeError` for anything malformed or structurally invalid. + */ +export function decodeLabelStreamFrame(bytes: Uint8Array): LabelStreamEvent | null { + let header: unknown; + let remainder: Uint8Array; + try { + [header, remainder] = decodeFirst(bytes); + } catch { + throw new TypeError("subscribeLabels frame header failed to decode as CBOR"); + } + if (!isPlainObject(header) || typeof header["op"] !== "number") { + throw new TypeError("subscribeLabels frame header must be an object with a numeric op"); + } + + let payload: unknown; + try { + [payload] = decodeFirst(remainder); + } catch { + throw new TypeError("subscribeLabels frame payload failed to decode as CBOR"); + } + + const op = header["op"]; + if (op === -1) { + if ( + !isPlainObject(payload) || + typeof payload["error"] !== "string" || + typeof payload["message"] !== "string" + ) { + throw new TypeError("subscribeLabels error frame must carry string error and message fields"); + } + throw new LabelStreamError(payload["error"], payload["message"]); + } + if (op !== 1) { + throw new TypeError(`subscribeLabels frame has unsupported op: ${op}`); + } + if (header["t"] !== "#labels") return null; + return validateLabelsPayload(payload); +} + +function validateLabelsPayload(payload: unknown): LabelStreamEvent { + if (!isPlainObject(payload)) { + throw new TypeError("#labels frame payload must be an object"); + } + const seq = payload["seq"]; + if (typeof seq !== "number" || !Number.isSafeInteger(seq) || seq <= 0) { + throw new TypeError("#labels frame seq must be a positive safe integer"); + } + const labels = payload["labels"]; + if (!Array.isArray(labels) || labels.length === 0) { + throw new TypeError("#labels frame labels must be a non-empty array"); + } + if (labels.length > MAX_LABELS_PER_FRAME) { + throw new TypeError( + `#labels frame labels must contain at most ${MAX_LABELS_PER_FRAME} entries`, + ); + } + return { seq, labels }; +} + +type QueueEntry = { kind: "value"; value: LabelStreamEvent } | { kind: "error"; error: unknown }; + +interface PendingNext { + resolve(result: IteratorResult): void; + reject(error: unknown): void; +} + +/** + * Production client: opens the WebSocket via a `fetch` upgrade (workerd's + * outbound WebSocket idiom — there's no browser `new WebSocket()` here) and + * adapts the event-driven socket into a pull-based async iterator. + * + * Buffering scheme: decoded frames (or decode errors) land in a small FIFO + * queue as they arrive; a pending `next()` call is handed the head of the + * queue directly instead of buffering it, so a fast consumer never pays for + * the queue at all. `close()` unblocks a pending `next()` immediately + * (`done: true`) rather than waiting on the socket's own close event, which + * may never come if the remote end is unresponsive. + */ +export class RealLabelStreamClient implements LabelStreamClient { + subscribe(opts: LabelStreamSubscribeOptions): LabelStreamHandle { + return createHandle(opts); + } +} + +function createHandle(opts: LabelStreamSubscribeOptions): LabelStreamHandle { + const queue: QueueEntry[] = []; + let pending: PendingNext | null = null; + let ended = false; + let socket: WebSocket | null = null; + + const deliver = (entry: QueueEntry): void => { + if (ended) return; + if (pending) { + const waiter = pending; + pending = null; + if (entry.kind === "error") waiter.reject(entry.error); + else waiter.resolve({ value: entry.value, done: false }); + return; + } + queue.push(entry); + if (queue.length > MAX_BUFFERED_FRAMES) { + // Fail closed like every other unrecoverable stream condition. The + // dropped frames were never cursor-persisted, so the reconnect + // replays them from the labeler's retained history. + queue.length = 0; + queue.push({ + kind: "error", + error: new Error("subscribeLabels inbound buffer overflow: consumer too slow"), + }); + socket?.close(); + } + }; + + const finish = (): void => { + if (ended) return; + ended = true; + if (pending) { + const waiter = pending; + pending = null; + waiter.resolve({ value: undefined, done: true }); + } + }; + + const handleMessage = (data: ArrayBuffer): void => { + let decoded: LabelStreamEvent | null; + try { + decoded = decodeLabelStreamFrame(new Uint8Array(data)); + } catch (err) { + deliver({ kind: "error", error: err }); + // Fail closed: don't keep reading a stream we can't parse. + socket?.close(); + return; + } + if (decoded !== null) deliver({ kind: "value", value: decoded }); + }; + + void (async () => { + try { + const url = `${opts.endpoint}/xrpc/com.atproto.label.subscribeLabels?cursor=${opts.cursor}`; + const response = await fetch(url, { headers: { upgrade: "websocket" } }); + if (response.status !== 101 || !response.webSocket) { + throw new Error(`subscribeLabels upgrade failed with status ${response.status}`); + } + socket = response.webSocket; + // Listeners attach before accept() so a frame or close arriving + // immediately after the upgrade can't slip past unhandled. + socket.addEventListener("message", (event: MessageEvent) => { + if (!(event.data instanceof ArrayBuffer)) { + deliver({ + kind: "error", + error: new TypeError("subscribeLabels message was not binary"), + }); + socket?.close(); + return; + } + handleMessage(event.data); + }); + // Code 1013 is the labeler's backpressure signal (see + // `apps/labeler/src/subscribe-labels.ts`'s `send`): the subscriber + // must reconnect with a cursor. Every other close code also just + // ends the iterator — the run loop's reconnect-with-backoff handles + // both cases identically. + socket.addEventListener("close", () => { + finish(); + }); + socket.accept(); + if (ended) { + // close() won the race against the upgrade: without this the + // accepted socket would stay open with nobody reading it. + socket.close(); + return; + } + } catch (err) { + deliver({ kind: "error", error: err }); + finish(); + } + })(); + + return { + close(): void { + socket?.close(); + finish(); + }, + [Symbol.asyncIterator](): AsyncIterator { + return { + next(): Promise> { + const entry = queue.shift(); + if (entry) { + if (entry.kind === "error") return Promise.reject(entry.error); + return Promise.resolve({ value: entry.value, done: false }); + } + if (ended) return Promise.resolve({ value: undefined, done: true }); + return new Promise((resolve, reject) => { + pending = { resolve, reject }; + }); + }, + return(): Promise> { + finish(); + return Promise.resolve({ value: undefined, done: true }); + }, + }; + }, + }; +} diff --git a/apps/aggregator/src/labeler-resolver.ts b/apps/aggregator/src/labeler-resolver.ts new file mode 100644 index 0000000000..17763c8241 --- /dev/null +++ b/apps/aggregator/src/labeler-resolver.ts @@ -0,0 +1,274 @@ +import { P256PublicKey, parsePublicMultikey } from "@atcute/crypto"; +import { type Did, isDid } from "@atcute/lexicons/syntax"; + +const DEFAULT_TTL_MS = 24 * 60 * 60 * 1000; +const EXPIRED_AT = "1970-01-01T00:00:00.000Z"; + +export interface CachedLabelerIdentity { + endpoint: string; + signingKey: string; + signingKeyId: string; + resolvedAt: Date; +} + +export interface LabelerIdentityCache { + read(did: string): Promise; + refresh( + did: string, + identity: Omit, + now: Date, + ): Promise; + expire(did: string): Promise; +} + +export interface LabelerDidResolverLike { + resolve(did: Did): Promise; +} + +export interface LabelerResolverOptions { + cache: LabelerIdentityCache; + resolver: LabelerDidResolverLike; + ttlMs?: number; + now?: () => Date; +} + +export interface ResolvedLabelerIdentity { + endpoint: string; + publicKey: P256PublicKey; + signingKeyId: string; +} + +export class LabelerResolver { + private readonly cache: LabelerIdentityCache; + private readonly resolver: LabelerDidResolverLike; + private readonly ttlMs: number; + private readonly now: () => Date; + + constructor(options: LabelerResolverOptions) { + this.cache = options.cache; + this.resolver = options.resolver; + this.ttlMs = options.ttlMs ?? DEFAULT_TTL_MS; + this.now = options.now ?? (() => new Date()); + } + + async resolve(did: string): Promise { + return this.resolveConfigured(asDid(did), false); + } + + /** Bypasses a fresh cache entry for W4.2's single retry after signature failure. */ + async resolveFresh(did: string): Promise { + return this.resolveConfigured(asDid(did), true); + } + + /** Expires only resolver freshness; cached identity and operator config remain intact. */ + async invalidate(did: string): Promise { + await this.cache.expire(asDid(did)); + } + + private async resolveConfigured(did: Did, forceFresh: boolean): Promise { + const cached = await this.cache.read(did); + if (!cached) throw new Error(`labeler is not configured: ${did}`); + + const now = this.now(); + if (!forceFresh && now.getTime() - cached.resolvedAt.getTime() < this.ttlMs) { + return await materialise(cached, did); + } + + const document = await this.resolver.resolve(did); + const fresh = await extractIdentity(document, did); + await this.cache.refresh(did, fresh, now); + return await materialise({ ...fresh, resolvedAt: now }, did); + } +} + +function asDid(value: string): Did { + if (!isDid(value)) throw new Error(`invalid DID: ${value}`); + return value; +} + +function isObject(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function normalizeId(did: string, id: string): string { + return id.startsWith("#") ? `${did}${id}` : id; +} + +async function extractIdentity( + document: unknown, + did: string, +): Promise> { + if (!isObject(document)) throw new TypeError("DID document must be an object"); + if (typeof document.id !== "string" || !isDid(document.id)) { + throw new TypeError("DID document id must be a DID"); + } + if (document.id !== did) throw new TypeError("DID document id does not match labeler DID"); + + const service = findLogicalEntry( + document.service, + did, + `${did}#atproto_labeler`, + "service", + "#atproto_labeler", + ); + if (service.type !== "AtprotoLabeler") { + throw new TypeError("#atproto_labeler service must have type AtprotoLabeler"); + } + const endpoint = validateEndpoint(service.serviceEndpoint); + + const method = findLogicalEntry( + document.verificationMethod, + did, + `${did}#atproto_label`, + "verification method", + "#atproto_label", + ); + if (method.type !== "Multikey") { + throw new TypeError("#atproto_label verification method must have type Multikey"); + } + if (method.controller !== did) { + throw new TypeError("#atproto_label verification method controller must equal the labeler DID"); + } + if (typeof method.publicKeyMultibase !== "string") { + throw new TypeError("#atproto_label verification method has an invalid Multikey"); + } + await importCanonicalP256(method.publicKeyMultibase); + + return { + endpoint, + signingKey: method.publicKeyMultibase, + signingKeyId: `${did}#atproto_label`, + }; +} + +function findLogicalEntry( + value: unknown, + did: string, + targetId: string, + kind: string, + fragment: string, +): Record { + if (!Array.isArray(value)) + throw new TypeError(`DID document must contain exactly one ${fragment} ${kind}`); + const matches: Record[] = []; + for (const entry of value) { + if (!isObject(entry) || typeof entry.id !== "string") { + throw new TypeError(`DID document contains a malformed ${kind}`); + } + if (normalizeId(did, entry.id) === targetId) matches.push(entry); + } + if (matches.length !== 1) { + throw new TypeError(`DID document must contain exactly one logical ${fragment} ${kind}`); + } + return matches[0]!; +} + +function validateEndpoint(value: unknown): string { + if (typeof value !== "string") { + throw new TypeError("#atproto_labeler service endpoint must be an HTTPS URL"); + } + let url: URL; + try { + url = new URL(value); + } catch { + throw new TypeError("#atproto_labeler service endpoint must be an HTTPS URL"); + } + if (url.protocol !== "https:") { + throw new TypeError("#atproto_labeler service endpoint must use HTTPS"); + } + if (url.username || url.password) { + throw new TypeError("#atproto_labeler service endpoint must not contain credentials"); + } + if (url.hash || value.includes("#")) + throw new TypeError("#atproto_labeler service endpoint must not contain a fragment"); + return value; +} + +async function importCanonicalP256(multikey: string): Promise { + let parsed: ReturnType; + try { + parsed = parsePublicMultikey(multikey); + } catch { + throw new TypeError("#atproto_label verification method has an invalid Multikey"); + } + if (parsed.type !== "p256") { + throw new TypeError("#atproto_label verification method must contain a P-256 Multikey"); + } + if ( + parsed.publicKeyBytes.length !== 33 || + (parsed.publicKeyBytes[0] !== 0x02 && parsed.publicKeyBytes[0] !== 0x03) + ) { + throw new TypeError("#atproto_label verification method must use a canonical P-256 Multikey"); + } + let publicKey: P256PublicKey; + try { + publicKey = await P256PublicKey.importRaw(parsed.publicKeyBytes); + } catch { + throw new TypeError("#atproto_label verification method has an invalid P-256 Multikey"); + } + if ((await publicKey.exportPublicKey("multikey")) !== multikey) { + throw new TypeError("#atproto_label verification method must use a canonical P-256 Multikey"); + } + return publicKey; +} + +async function materialise( + cached: CachedLabelerIdentity, + did: string, +): Promise { + validateEndpoint(cached.endpoint); + if (cached.signingKeyId !== `${did}#atproto_label`) { + throw new TypeError("cached labeler signing key id is not canonical"); + } + return { + endpoint: cached.endpoint, + publicKey: await importCanonicalP256(cached.signingKey), + signingKeyId: cached.signingKeyId, + }; +} + +export function createD1LabelerIdentityCache(db: D1Database): LabelerIdentityCache { + return { + async read(did): Promise { + const row = await db + .prepare( + `SELECT endpoint, signing_key, signing_key_id, last_resolved_at + FROM labelers + WHERE did = ?`, + ) + .bind(did) + .first<{ + endpoint: string; + signing_key: string; + signing_key_id: string; + last_resolved_at: string; + }>(); + if (!row) return null; + return { + endpoint: row.endpoint, + signingKey: row.signing_key, + signingKeyId: row.signing_key_id, + resolvedAt: new Date(row.last_resolved_at), + }; + }, + + async refresh(did, identity, now): Promise { + const result = await db + .prepare( + `UPDATE labelers + SET endpoint = ?, signing_key = ?, signing_key_id = ?, last_resolved_at = ? + WHERE did = ?`, + ) + .bind(identity.endpoint, identity.signingKey, identity.signingKeyId, now.toISOString(), did) + .run(); + if (result.meta.changes !== 1) throw new Error(`labeler is not configured: ${did}`); + }, + + async expire(did): Promise { + await db + .prepare("UPDATE labelers SET last_resolved_at = ? WHERE did = ?") + .bind(EXPIRED_AT, did) + .run(); + }, + }; +} diff --git a/apps/aggregator/src/labels-consumer.ts b/apps/aggregator/src/labels-consumer.ts new file mode 100644 index 0000000000..3de3cdda63 --- /dev/null +++ b/apps/aggregator/src/labels-consumer.ts @@ -0,0 +1,387 @@ +/** + * Labels queue consumer. Writes append-only history (`labels`) plus a + * current-state projection (`label_state`) for every verified signed label + * enqueued by `LabelIngestor` (`label-ingestor.ts`). + * + * For each `LabelIngestJob`: + * + * 1. `fromWire` + `parseSignedLabel` revalidate the payload (defense in + * depth — the queue payload is internal, but the parse is cheap). + * 2. Compute the SHA-256 digest of `encodeSignedLabel(label)`. This is the + * `labels` primary key, so exact redelivery (same signed bytes) is a + * silent no-op via `ON CONFLICT(digest) DO NOTHING`. + * 3. Look up `labelers.trusted` for `job.src` (cached per batch) and snapshot + * it onto the written rows as receipt-time provenance. + * 4. One `db.batch()` writes the history row and conditionally upserts + * `label_state`, keyed by `(src, uri, val)`, keeping only the winner by + * `(cts_epoch_ms, source_sequence, frame_index)`. + * + * Error policy: + * - Structurally invalid wire payload: `dead_letters` row (`LABEL_INVALID`), + * ack. Never retry — the bytes won't parse differently next time. + * - `src` absent from `labelers`: `dead_letters` row + * (`LABEL_UNKNOWN_SOURCE`), ack. The ingestor only subscribes to + * configured labelers, so this indicates the row was removed between + * subscribe and delivery. + * - A different signed label landing at coordinates + * `(src, source_sequence, frame_index)` already occupied by another + * label: `dead_letters` row (`LABEL_COORDINATE_CONFLICT`), ack. This is a + * permanent conflict — the labeler is replaying its stream inconsistently. + * - Any other D1 failure: `retry()`. + */ + +import { + encodeSignedLabel, + parseSignedLabel, + type SignedLabel, +} from "@emdash-cms/registry-moderation"; + +import { fromWire, type LabelIngestJob } from "./label-ingest-types.js"; + +/** Deps the consumer needs at runtime. Tests inject their own `db` (and + * optionally `now`) to run against a real D1 instance without a live queue. */ +export interface ConsumerDeps { + db: D1Database; + now?: () => Date; +} + +/** Subset of `cloudflare:workers` `Message` we use; defining inline so tests + * don't need to import workerd types. */ +export interface MessageController { + ack(): void; + retry(): void; +} + +/** Subset of a `MessageBatch`. Workers' real batch object satisfies this. */ +export interface MessageBatchLike { + readonly messages: ReadonlyArray; +} + +/** Reason codes written to `dead_letters.reason` for labels jobs. */ +export type LabelDeadLetterReason = + | "LABEL_INVALID" + | "LABEL_UNKNOWN_SOURCE" + | "LABEL_COORDINATE_CONFLICT" + | "LABEL_UNEXPECTED_ERROR"; + +/** `dead_letters.collection` value for labels jobs — the table is shared with + * the records consumer, which stores an actual NSID there; labels jobs have + * no NSID, so this is a fixed tag identifying the ingest pipeline. */ +const LABEL_DEAD_LETTER_COLLECTION = "com.atproto.label.subscribeLabels"; + +export async function processLabelsBatch( + batch: MessageBatchLike, + env: Env, + depsOverride?: ConsumerDeps, +): Promise { + const deps = depsOverride ?? { db: env.DB }; + // Cached across the whole batch: labelers rarely change mid-batch, and a + // lookup per message would otherwise cost one D1 round trip per label. + const trustedCache = new Map(); + // Process jobs independently — see records-consumer.ts's processBatch for + // why the try/catch has to wrap each message rather than the loop. + for (const message of batch.messages) { + try { + await processLabelMessage(message.body, message, deps, trustedCache); + } catch (err) { + console.error("[aggregator] processLabelMessage threw unexpectedly", { + src: message.body.src, + sourceSequence: message.body.sourceSequence, + frameIndex: message.body.frameIndex, + error: err instanceof Error ? err.message : String(err), + }); + message.retry(); + } + } +} + +/** + * Drain the labels DLQ. Same "log + ack" policy as `drainDeadLetterBatch` in + * records-consumer.ts: record the job to Workers logs and `dead_letters`, ack + * so the DLQ doesn't grow unbounded. + */ +export async function drainLabelsDeadLetterBatch( + batch: MessageBatchLike, + env: Env, +): Promise { + const now = new Date(); + for (const message of batch.messages) { + const job = message.body; + console.warn("[aggregator] labels DLQ drain: acking job", { + src: job.src, + sourceSequence: job.sourceSequence, + frameIndex: job.frameIndex, + }); + try { + await writeDeadLetter(env.DB, job, "LABEL_UNEXPECTED_ERROR", "drained from DLQ", now); + message.ack(); + } catch (err) { + console.error("[aggregator] labels DLQ drain: failed to write forensics row, retrying", { + src: job.src, + error: err instanceof Error ? err.message : String(err), + }); + message.retry(); + } + } +} + +async function processLabelMessage( + job: LabelIngestJob, + controller: MessageController, + deps: ConsumerDeps, + trustedCache: Map, +): Promise { + const now = deps.now ?? (() => new Date()); + + let label: SignedLabel; + try { + label = parseSignedLabel(fromWire(job.label)); + } catch (err) { + await writeDeadLetter( + deps.db, + job, + "LABEL_INVALID", + err instanceof Error ? err.message : String(err), + now(), + ); + controller.ack(); + return; + } + + if (label.src !== job.src) { + // The ingestor enqueues job.src = verified label.src, so a mismatch means + // the job did not come from our producer. The digest and the (src, ...) + // row keys would disagree about which labeler signed these bytes. + await writeDeadLetter( + deps.db, + job, + "LABEL_INVALID", + `label src '${label.src}' does not match job src '${job.src}'`, + now(), + ); + controller.ack(); + return; + } + + const trusted = await lookupTrusted(deps.db, job.src, trustedCache); + if (trusted === null) { + await writeDeadLetter( + deps.db, + job, + "LABEL_UNKNOWN_SOURCE", + `no labelers row for src '${job.src}'`, + now(), + ); + controller.ack(); + return; + } + + const digest = await digestSignedLabel(label); + const ctsEpochMs = Date.parse(label.cts); + const expEpochMs = label.exp === undefined ? null : Date.parse(label.exp); + if (Number.isNaN(ctsEpochMs) || (expEpochMs !== null && Number.isNaN(expEpochMs))) { + // parseSignedLabel validates both datetimes, so this shouldn't fire — + // but a NaN committed into the epoch columns would silently invert + // every projection and enforcement comparison. Fail closed instead. + await writeDeadLetter( + deps.db, + job, + "LABEL_INVALID", + `cts/exp did not parse to an instant (cts=${label.cts}, exp=${label.exp ?? "absent"})`, + now(), + ); + controller.ack(); + return; + } + + try { + await deps.db.batch([ + insertLabelStmt(deps.db, job, label, digest, ctsEpochMs, expEpochMs, trusted, now()), + upsertLabelStateStmt(deps.db, job, label, digest, ctsEpochMs, expEpochMs, trusted), + ]); + } catch (err) { + if (isCoordinateConflict(err)) { + await writeDeadLetter( + deps.db, + job, + "LABEL_COORDINATE_CONFLICT", + `a different label already occupies (src=${job.src}, source_sequence=${job.sourceSequence}, frame_index=${job.frameIndex})`, + now(), + ); + controller.ack(); + return; + } + console.error("[aggregator] labels batch write failed", { + src: job.src, + sourceSequence: job.sourceSequence, + frameIndex: job.frameIndex, + error: err instanceof Error ? err.message : String(err), + }); + controller.retry(); + return; + } + + controller.ack(); +} + +async function lookupTrusted( + db: D1Database, + src: string, + cache: Map, +): Promise { + const cached = cache.get(src); + if (cached !== undefined) return cached; + const row = await db + .prepare(`SELECT trusted FROM labelers WHERE did = ?`) + .bind(src) + .first<{ trusted: number }>(); + const trusted = row ? row.trusted === 1 : null; + cache.set(src, trusted); + return trusted; +} + +async function digestSignedLabel(label: SignedLabel): Promise { + const bytes = encodeSignedLabel(label); + const hash = await crypto.subtle.digest("SHA-256", bytes); + return toHex(new Uint8Array(hash)); +} + +const HEX_ALPHABET = "0123456789abcdef"; +function toHex(bytes: Uint8Array): string { + let out = ""; + for (const byte of bytes) { + out += HEX_ALPHABET[byte >> 4]; + out += HEX_ALPHABET[byte & 0x0f]; + } + return out; +} + +function insertLabelStmt( + db: D1Database, + job: LabelIngestJob, + label: SignedLabel, + digest: string, + ctsEpochMs: number, + expEpochMs: number | null, + trusted: boolean, + receivedAt: Date, +): D1PreparedStatement { + return db + .prepare( + `INSERT INTO labels + (digest, src, uri, cid, val, neg, cts, cts_epoch_ms, exp, exp_epoch_ms, + sig, ver, source_sequence, frame_index, trusted, received_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(digest) DO NOTHING`, + ) + .bind( + digest, + job.src, + label.uri, + label.cid ?? null, + label.val, + label.neg === true ? 1 : 0, + label.cts, + ctsEpochMs, + label.exp ?? null, + expEpochMs, + label.sig, + label.ver, + job.sourceSequence, + job.frameIndex, + trusted ? 1 : 0, + receivedAt.toISOString(), + ); +} + +function upsertLabelStateStmt( + db: D1Database, + job: LabelIngestJob, + label: SignedLabel, + digest: string, + ctsEpochMs: number, + expEpochMs: number | null, + trusted: boolean, +): D1PreparedStatement { + return db + .prepare( + `INSERT INTO label_state + (src, uri, val, cid, neg, cts, cts_epoch_ms, exp, exp_epoch_ms, + digest, source_sequence, frame_index, trusted) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(src, uri, val) DO UPDATE SET + cid = excluded.cid, + neg = excluded.neg, + cts = excluded.cts, + cts_epoch_ms = excluded.cts_epoch_ms, + exp = excluded.exp, + exp_epoch_ms = excluded.exp_epoch_ms, + digest = excluded.digest, + source_sequence = excluded.source_sequence, + frame_index = excluded.frame_index, + trusted = excluded.trusted + WHERE excluded.cts_epoch_ms > label_state.cts_epoch_ms + OR (excluded.cts_epoch_ms = label_state.cts_epoch_ms + AND (excluded.source_sequence > label_state.source_sequence + OR (excluded.source_sequence = label_state.source_sequence + AND excluded.frame_index > label_state.frame_index)))`, + ) + .bind( + job.src, + label.uri, + label.val, + label.cid ?? null, + label.neg === true ? 1 : 0, + label.cts, + ctsEpochMs, + label.exp ?? null, + expEpochMs, + digest, + job.sourceSequence, + job.frameIndex, + trusted ? 1 : 0, + ); +} + +/** + * Distinguishes the `UNIQUE (src, source_sequence, frame_index)` conflict + * from any other D1 failure. The `labels` INSERT already resolves a digest + * (primary key) conflict via `ON CONFLICT(digest) DO NOTHING`, so the only + * constraint violation that can still reach here is the coordinate index — + * D1 doesn't surface a structured error code, so the column names in + * SQLite's message text are the only reliable signal. + */ +function isCoordinateConflict(err: unknown): boolean { + if (!(err instanceof Error)) return false; + return ( + err.message.includes("UNIQUE constraint failed") && + err.message.includes("labels.source_sequence") && + err.message.includes("labels.frame_index") + ); +} + +async function writeDeadLetter( + db: D1Database, + job: LabelIngestJob, + reason: LabelDeadLetterReason, + detail: string | null, + now: Date, +): Promise { + const payloadBytes = new TextEncoder().encode(JSON.stringify(job.label)); + await db + .prepare( + `INSERT INTO dead_letters + (did, collection, rkey, reason, detail, payload, received_at) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + job.src, + LABEL_DEAD_LETTER_COLLECTION, + `${job.sourceSequence}:${job.frameIndex}`, + reason, + detail, + payloadBytes, + now.toISOString(), + ) + .run(); +} diff --git a/apps/aggregator/src/pds-verify.ts b/apps/aggregator/src/pds-verify.ts index 4b31bb0b0a..1c8965bd19 100644 --- a/apps/aggregator/src/pds-verify.ts +++ b/apps/aggregator/src/pds-verify.ts @@ -20,18 +20,40 @@ import type { PublicKey } from "@atcute/crypto"; import { type AtprotoDid, isDid } from "@atcute/lexicons/syntax"; import { verifyRecord } from "@atcute/repo"; +import { type DnsResolver, resolveAndValidateExternalUrl, SsrfError } from "emdash/security/ssrf"; const DEFAULT_TIMEOUT_MS = 15_000; /** 5 MB ceiling. Records and their proofs are tiny (sub-KB typical); this is * a defence against a hostile or broken PDS streaming an unbounded body. */ const DEFAULT_MAX_RESPONSE_BYTES = 5 * 1024 * 1024; +/** Redirect hops we'll follow before giving up. The PDS endpoint is + * publisher-controlled, so redirects are an SSRF pivot: every hop is + * re-validated, and this caps the chain. */ +const MAX_PDS_REDIRECTS = 3; +/** Message prefix `resolveAndValidateExternalUrl` uses when the injected DNS + * resolver itself throws (DoH network error / SERVFAIL / timeout) — a transient + * infrastructure failure, distinct from a disallowed address. Keyed on here to + * classify those as retryable; see `assertFetchableUrl`. */ +const RESOLVER_FAILURE_PREFIX = "Could not resolve hostname:"; +/** Message `resolveAndValidateExternalUrl` uses for an empty DNS answer + * (NXDOMAIN / NOERROR-NODATA / CNAME-only). Transient: a host mid-propagation + * would otherwise be permanently dead-lettered; a genuinely-gone host instead + * dead-letters via retry exhaustion. Not a disallowed address. */ +const EMPTY_ANSWER_MESSAGE = "Hostname resolved to no addresses"; +/** Delay, in seconds, before the consumer re-delivers a message that failed on a + * transient DNS resolution (empty answer / resolver outage). The records queue + * re-delivers immediately by default, so without this a host mid-propagation + * burns all `max_retries` in seconds and lands in the DLQ before DNS settles. + * A fixed pause spreads the retries across a propagation window. */ +const RESOLUTION_RETRY_DELAY_SECONDS = 60; export type VerificationFailureReason = | "PDS_NETWORK_ERROR" | "PDS_HTTP_ERROR" | "RECORD_NOT_FOUND" | "RESPONSE_TOO_LARGE" - | "INVALID_PROOF"; + | "INVALID_PROOF" + | "PDS_ADDRESS_BLOCKED"; export class PdsVerificationError extends Error { override readonly name = "PdsVerificationError"; @@ -40,6 +62,10 @@ export class PdsVerificationError extends Error { message: string, readonly status?: number, override readonly cause?: unknown, + /** When set, the consumer re-delivers the message after this many seconds + * instead of immediately. Set only for transient DNS-resolution failures + * that need time to propagate; other transient retries stay immediate. */ + readonly retryAfterSeconds?: number, ) { super(message); } @@ -51,12 +77,20 @@ export interface FetchAndVerifyOptions { collection: string; rkey: string; publicKey: PublicKey; - /** Default 15s. Aborts the fetch if the PDS is slow. */ + /** Default 15s wall-clock budget covering every redirect hop AND the body + * read — a slow-drip body is bounded by this, not just the initial fetch. */ timeoutMs?: number; /** Default 5 MB. Rejects with `RESPONSE_TOO_LARGE` if exceeded. */ maxResponseBytes?: number; /** Inject for tests; defaults to `globalThis.fetch`. */ fetch?: typeof fetch; + /** + * SSRF-safe hostname resolver applied to the PDS endpoint and every redirect + * target before we connect. Production injects `cloudflareDohResolver`; tests + * stub it. Absent means fail closed — the publisher-controlled fetch is + * refused rather than issued unprotected. + */ + resolveHostname?: DnsResolver; } export interface VerifiedPdsRecord { @@ -74,6 +108,16 @@ export async function fetchAndVerifyRecord( const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS; const maxResponseBytes = opts.maxResponseBytes ?? DEFAULT_MAX_RESPONSE_BYTES; + // Fail closed: without a resolver we can't validate the publisher-controlled + // endpoint against the SSRF blocklist, so we refuse rather than fetch blind. + const resolveHostname = opts.resolveHostname; + if (!resolveHostname) { + throw new PdsVerificationError( + "PDS_ADDRESS_BLOCKED", + "refusing PDS fetch: no SSRF-safe hostname resolver was provided", + ); + } + if (!isAtprotoDid(opts.did)) { // Caller is expected to have validated this upstream (the resolver // rejects non-DID strings before reaching here), but the verifier's @@ -84,7 +128,7 @@ export async function fetchAndVerifyRecord( ); } const url = buildGetRecordUrl(opts.pds, opts.did, opts.collection, opts.rkey); - const carBytes = await fetchCar(fetchImpl, url, timeoutMs, maxResponseBytes); + const carBytes = await fetchCar(fetchImpl, url, timeoutMs, maxResponseBytes, resolveHostname); try { const result = await verifyRecord({ @@ -117,54 +161,179 @@ function buildGetRecordUrl(pds: string, did: string, collection: string, rkey: s return url.toString(); } +function isRedirectStatus(status: number): boolean { + return status === 301 || status === 302 || status === 303 || status === 307 || status === 308; +} + +/** + * True when an `SsrfError` message denotes a transient resolution failure (the + * resolver threw, or the DNS answer was empty) rather than a disallowed address. + * Keyed on the shared helper's messages; guarded by tests so a wording change + * there fails loudly instead of silently re-classifying. + */ +function isTransientResolutionFailure(message: string): boolean { + return message.startsWith(RESOLVER_FAILURE_PREFIX) || message === EMPTY_ANSWER_MESSAGE; +} + +/** + * Reject the URL unless it's HTTPS and resolves to a public address. The PDS + * endpoint (and every redirect target) is publisher-controlled, so this is the + * SSRF boundary: HTTP is refused (no plaintext downgrade to an internal + * listener the resolver can't see), and `resolveAndValidateExternalUrl` resolves + * the hostname and rejects any private/reserved address. IP/address logic lives + * in the shared helper — never duplicated here. + */ +async function assertFetchableUrl(url: string, resolver: DnsResolver): Promise { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + throw new PdsVerificationError("PDS_ADDRESS_BLOCKED", `invalid PDS URL: ${url}`); + } + if (parsed.protocol !== "https:") { + throw new PdsVerificationError("PDS_ADDRESS_BLOCKED", `PDS URL must be https: ${url}`); + } + try { + await resolveAndValidateExternalUrl(url, { resolver }); + } catch (err) { + if (err instanceof SsrfError) { + // `resolveAndValidateExternalUrl` throws `SsrfError` for two very + // different classes of situation: a disallowed address (permanent — + // dead-letter) and a resolution failure (transient — retry). The + // transient class is a DoH network error / SERVFAIL / timeout (the + // resolver threw, prefixed `RESOLVER_FAILURE_PREFIX`) or an empty DNS + // answer (`EMPTY_ANSWER_MESSAGE` — a host mid-propagation). `SsrfError.code` + // is the same constant for all, so the only discriminator is the message. + // Tests assert both transient mappings so a future wording change in the + // shared helper breaks loudly here rather than silently dead-lettering + // legitimate records during a DoH blip or DNS propagation. + if (isTransientResolutionFailure(err.message)) { + throw new PdsVerificationError( + "PDS_NETWORK_ERROR", + `PDS host resolution failed: ${err.message}`, + undefined, + err, + RESOLUTION_RETRY_DELAY_SECONDS, + ); + } + throw new PdsVerificationError( + "PDS_ADDRESS_BLOCKED", + `PDS address rejected: ${err.message}`, + undefined, + err, + ); + } + throw err; + } +} + async function fetchCar( fetchImpl: typeof fetch, - url: string, + initialUrl: string, timeoutMs: number, maxBytes: number, + resolver: DnsResolver, ): Promise { + // One wall-clock budget spanning every redirect hop and the body read, so a + // PDS that dribbles headers-then-body can't hold the request open past it. const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), timeoutMs); - let response: Response; try { - response = await fetchImpl(url, { - signal: controller.signal, - headers: { accept: "application/vnd.ipld.car" }, - }); - } catch (err) { - // Whether the fetch threw because we aborted (timeout) or because the - // network failed at the OS layer, the right caller behaviour is the - // same: retry. Lump them under PDS_NETWORK_ERROR. - throw new PdsVerificationError( - "PDS_NETWORK_ERROR", - err instanceof Error && err.name === "AbortError" - ? `PDS fetch aborted after ${timeoutMs}ms` - : `PDS fetch failed: ${err instanceof Error ? err.message : String(err)}`, - undefined, - err, - ); + let currentUrl = initialUrl; + for (let hop = 0; ; hop++) { + // Re-resolve and re-validate every hop: an allowed public host can + // 30x to a private one, so validating only the initial URL is a hole. + await assertFetchableUrl(currentUrl, resolver); + + let response: Response; + try { + response = await fetchImpl(currentUrl, { + signal: controller.signal, + // Intercept redirects so each target is re-validated before we + // follow it, rather than letting fetch chase them unchecked. + redirect: "manual", + headers: { accept: "application/vnd.ipld.car" }, + }); + } catch (err) { + // Whether the fetch threw because we aborted (timeout) or because + // the network failed at the OS layer, the right caller behaviour + // is the same: retry. Lump them under PDS_NETWORK_ERROR. + throw new PdsVerificationError( + "PDS_NETWORK_ERROR", + err instanceof Error && err.name === "AbortError" + ? `PDS fetch aborted after ${timeoutMs}ms` + : `PDS fetch failed: ${err instanceof Error ? err.message : String(err)}`, + undefined, + err, + ); + } + + if (isRedirectStatus(response.status)) { + // Free the redirect response's socket before following. + await response.body?.cancel().catch(() => {}); + if (hop >= MAX_PDS_REDIRECTS) { + throw new PdsVerificationError( + "PDS_ADDRESS_BLOCKED", + `PDS exceeded ${MAX_PDS_REDIRECTS} redirects`, + ); + } + const location = response.headers.get("location"); + if (!location) { + throw new PdsVerificationError( + "PDS_HTTP_ERROR", + `PDS redirect without a Location header`, + response.status, + ); + } + // The publisher fully controls this header; a malformed value + // (e.g. `http://` with an empty authority) makes `new URL` throw a + // raw TypeError. Classify it as a blocked redirect rather than + // letting it escape as an UNEXPECTED_ERROR the publisher can force. + try { + currentUrl = new URL(location, currentUrl).toString(); + } catch { + throw new PdsVerificationError( + "PDS_ADDRESS_BLOCKED", + `PDS redirect Location is not a valid URL: ${location}`, + ); + } + continue; + } + + if (response.status === 404) { + // Distinct from a generic 4xx. The publisher may have deleted the + // record between Jetstream emitting and us fetching, which is the + // common cause; other 4xx (auth, bad request) suggest programming + // errors. Both end up dead-lettered by the consumer (the audit + // trail is useful even for legitimate races so operators can spot + // systematic Jetstream-vs-PDS skew); the distinct reason code keeps + // them queryable separately. + throw new PdsVerificationError( + "RECORD_NOT_FOUND", + `PDS returned 404 for ${currentUrl}`, + 404, + ); + } + if (!response.ok) { + throw new PdsVerificationError( + "PDS_HTTP_ERROR", + `PDS returned ${response.status} for ${currentUrl}`, + response.status, + ); + } + + return await readCarBody(response, maxBytes, timeoutMs); + } } finally { clearTimeout(timer); } +} - if (response.status === 404) { - // Distinct from a generic 4xx. The publisher may have deleted the - // record between Jetstream emitting and us fetching, which is the - // common cause; other 4xx (auth, bad request) suggest programming - // errors. Both end up dead-lettered by the consumer (the audit trail - // is useful even for legitimate races so operators can spot - // systematic Jetstream-vs-PDS skew); the distinct reason code keeps - // them queryable separately. - throw new PdsVerificationError("RECORD_NOT_FOUND", `PDS returned 404 for ${url}`, 404); - } - if (!response.ok) { - throw new PdsVerificationError( - "PDS_HTTP_ERROR", - `PDS returned ${response.status} for ${url}`, - response.status, - ); - } - +async function readCarBody( + response: Response, + maxBytes: number, + timeoutMs: number, +): Promise { // Buffer the body up to the size limit. Don't trust Content-Length; a // hostile or buggy PDS could under-report and stream more bytes than // advertised. @@ -175,8 +344,7 @@ async function fetchCar( const chunks: Uint8Array[] = []; let total = 0; try { - // biome-ignore lint/correctness/noConstantCondition: drains the stream - while (true) { + for (;;) { const { done, value } = await reader.read(); if (done) break; total += value.byteLength; @@ -193,7 +361,18 @@ async function fetchCar( await reader.cancel().catch(() => { /* swallow — we already have a primary error to surface */ }); - throw err; + // Our own errors (too-large) carry their classification; pass them + // through. Anything else — a dropped socket, or the wall-clock deadline + // aborting a slow-drip body mid-download — is transient: retry. + if (err instanceof PdsVerificationError) throw err; + throw new PdsVerificationError( + "PDS_NETWORK_ERROR", + err instanceof Error && err.name === "AbortError" + ? `PDS body read aborted after ${timeoutMs}ms` + : `PDS stream failed mid-download: ${err instanceof Error ? err.message : String(err)}`, + undefined, + err, + ); } finally { reader.releaseLock(); } diff --git a/apps/aggregator/src/records-consumer.ts b/apps/aggregator/src/records-consumer.ts index eb2c0af03e..e44fe5e23d 100644 --- a/apps/aggregator/src/records-consumer.ts +++ b/apps/aggregator/src/records-consumer.ts @@ -43,6 +43,7 @@ import { PublisherProfile, PublisherVerification, } from "@emdash-cms/registry-lexicons"; +import { cloudflareDohResolver, type DnsResolver } from "emdash/security/ssrf"; import { createD1DidDocCache, DidResolver } from "./did-resolver.js"; import type { RecordsJob } from "./env.js"; @@ -63,6 +64,12 @@ export interface ConsumerDeps { db: D1Database; resolver: DidResolver; fetch?: typeof fetch; + /** + * SSRF-safe hostname resolver handed to `fetchAndVerifyRecord` so it can + * validate the publisher-controlled PDS endpoint (and any redirect) before + * connecting. Production wires `cloudflareDohResolver`. + */ + resolveHostname?: DnsResolver; now?: () => Date; /** * Optional override for the PDS-verification step. Used by tests to inject @@ -78,6 +85,7 @@ export interface ConsumerDeps { rkey: string; publicKey: import("@atcute/crypto").PublicKey; fetch?: typeof fetch; + resolveHostname?: DnsResolver; }) => Promise; } @@ -85,7 +93,7 @@ export interface ConsumerDeps { * don't need to import workerd types. */ export interface MessageController { ack(): void; - retry(): void; + retry(options?: { delaySeconds?: number }): void; } /** Subset of a `MessageBatch`. Workers' real batch object satisfies this. */ @@ -102,6 +110,7 @@ export type DeadLetterReason = | "RESPONSE_TOO_LARGE" | "INVALID_PROOF" | "PDS_HTTP_ERROR" + | "PDS_ADDRESS_BLOCKED" // structural checks (consumer-enforced) | "LEXICON_VALIDATION_FAILED" | "RKEY_MISMATCH" @@ -241,7 +250,12 @@ export async function processMessage( } catch (err) { if (err instanceof PdsVerificationError) { if (isTransient(err.reason, err.status)) { - controller.retry(); + // A transient DNS-resolution failure carries a re-delivery delay so + // retries span the DNS-propagation window instead of firing back to + // back; other transient failures (network, 5xx, timeout) retry now. + controller.retry( + err.retryAfterSeconds !== undefined ? { delaySeconds: err.retryAfterSeconds } : undefined, + ); return; } // Compute the mapped reason in its own try/catch — `mapPdsReason` @@ -311,6 +325,7 @@ async function verifyAndIngest(job: RecordsJob, deps: ConsumerDeps): Promise { const session = env.DB.withSession("first-primary"); + const { accepted } = getRequestLabelerPolicy(request); - // Fast path: pull the latest_version pointer + matching release in one - // query. The tombstoned_at filter is the integrity gate — if the - // pointer is stale (release tombstoned, refresh pending), this misses - // and we fall through. - const fast = await session - .prepare( - `SELECT ${releaseColumns("r.")} - FROM packages p - JOIN releases r ON r.did = p.did AND r.package = p.slug AND r.version = p.latest_version - WHERE p.did = ? AND p.slug = ? AND r.tombstoned_at IS NULL`, - ) - .bind(params.did, params.package) - .first(); - if (fast) return json(releaseView(fast)); + if (accepted.length === 0) { + // Fast path: pull the latest_version pointer + matching release in one + // query. The tombstoned_at filter is the integrity gate — if the + // pointer is stale (release tombstoned, refresh pending), this misses + // and we fall through. + const fast = await session + .prepare( + `SELECT ${releaseColumns("r.")} + FROM packages p + JOIN releases r ON r.did = p.did AND r.package = p.slug AND r.version = p.latest_version + WHERE p.did = ? AND p.slug = ? AND r.tombstoned_at IS NULL`, + ) + .bind(params.did, params.package) + .first(); + if (fast) return json(releaseView(fast)); + + // Slow-path fallback: the authoritative ORDER BY. Costs an extra D1 + // round-trip on the rare miss but guarantees we don't 404 on a package + // that has live releases. Tiebreakers (version, rkey) keep the result + // deterministic if version_sort ties, matching listReleases' ordering. + const slow = await session + .prepare( + `SELECT ${releaseColumns()} + FROM releases + WHERE did = ? AND package = ? AND tombstoned_at IS NULL + ORDER BY version_sort DESC, version DESC, rkey DESC + LIMIT 1`, + ) + .bind(params.did, params.package) + .first(); + if (slow) return json(releaseView(slow)); + + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No eligible release for (${params.did}, ${params.package}).`, + }); + } - // Slow-path fallback: the authoritative ORDER BY. Costs an extra D1 - // round-trip on the rare miss but guarantees we don't 404 on a package - // that has live releases. Tiebreakers (version, rkey) keep the result - // deterministic if version_sort ties, matching listReleases' ordering. - const slow = await session + const nowMs = Date.now(); + const enforcement = buildReleaseEnforcementSql(accepted, nowMs); + const winner = await session .prepare( - `SELECT ${releaseColumns()} - FROM releases - WHERE did = ? AND package = ? AND tombstoned_at IS NULL - ORDER BY version_sort DESC, version DESC, rkey DESC + `SELECT ${releaseColumns("r.")}, p.signature_metadata AS package_signature_metadata + FROM releases r + JOIN packages p ON p.did = r.did AND p.slug = r.package + WHERE r.did = ? AND r.package = ? AND r.tombstoned_at IS NULL + ${enforcement.sql} + ORDER BY r.version_sort DESC, r.version DESC, r.rkey DESC LIMIT 1`, ) - .bind(params.did, params.package) - .first(); - if (slow) return json(releaseView(slow)); + .bind(params.did, params.package, ...enforcement.bindings) + .first(); + if (!winner) { + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No eligible release for (${params.did}, ${params.package}).`, + }); + } - throw new XRPCError({ - status: 404, - error: "NotFound", - message: `No eligible release for (${params.did}, ${params.package}).`, - }); + const releaseSubjectUri = releaseUri(winner); + const packageSubjectUri = packageUri({ did: winner.did, slug: winner.package }); + const labelsByUri = await hydrateLabels( + session, + accepted, + [ + { + uri: releaseSubjectUri, + currentCid: parseSignatureMetadataCid(winner.signature_metadata) ?? undefined, + }, + { + uri: packageSubjectUri, + currentCid: parseSignatureMetadataCid(winner.package_signature_metadata) ?? undefined, + }, + { uri: winner.did }, + ], + nowMs, + ); + const labels = [ + ...(labelsByUri.get(releaseSubjectUri) ?? []), + ...(labelsByUri.get(packageSubjectUri) ?? []), + ...(labelsByUri.get(winner.did) ?? []), + ]; + return json(releaseView(winner, labels)); } diff --git a/apps/aggregator/src/routes/xrpc/getPackage.ts b/apps/aggregator/src/routes/xrpc/getPackage.ts index d9efb1793a..2f14099e01 100644 --- a/apps/aggregator/src/routes/xrpc/getPackage.ts +++ b/apps/aggregator/src/routes/xrpc/getPackage.ts @@ -12,14 +12,18 @@ import { json, XRPCError } from "@atcute/xrpc-server"; import { type AggregatorDefs, type AggregatorGetPackage } from "@emdash-cms/registry-lexicons"; -import { type PackageRow, packageColumns, packageView } from "./views.js"; +import { parseSignatureMetadataCid } from "../../utils.js"; +import { hydrateLabels, isRedacted } from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { type PackageRow, packageColumns, packageUri, packageView } from "./views.js"; export async function getPackage( env: Env, params: AggregatorGetPackage.$params, + request: Request, ): Promise { // `first-primary` because the same row could become subject to a takedown - // label between two reads; once the labeller (Slice 2) writes, the next + // label between two reads; once the labeler (Slice 2) writes, the next // read everywhere should reflect it. Per plan §XRPC endpoints. const session = env.DB.withSession("first-primary"); const row = await session @@ -33,6 +37,28 @@ export async function getPackage( message: `No package indexed under (${params.did}, ${params.slug}).`, }); } - const view: AggregatorDefs.PackageView = packageView(row); + + const { accepted } = getRequestLabelerPolicy(request); + const uri = packageUri(row); + const labelsByUri = await hydrateLabels( + session, + accepted, + [ + { uri, currentCid: parseSignatureMetadataCid(row.signature_metadata) ?? undefined }, + { uri: row.did }, + ], + Date.now(), + ); + const labels = [...(labelsByUri.get(uri) ?? []), ...(labelsByUri.get(row.did) ?? [])]; + if (isRedacted(labels, accepted)) { + // Indistinguishable from absence — that is what redaction means. + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No package indexed under (${params.did}, ${params.slug}).`, + }); + } + + const view: AggregatorDefs.PackageView = packageView(row, labels); return json(view); } diff --git a/apps/aggregator/src/routes/xrpc/getPublisher.ts b/apps/aggregator/src/routes/xrpc/getPublisher.ts new file mode 100644 index 0000000000..8b0ef8c9c9 --- /dev/null +++ b/apps/aggregator/src/routes/xrpc/getPublisher.ts @@ -0,0 +1,62 @@ +/** + * `com.emdashcms.experimental.aggregator.getPublisher` — single publisher + * profile by DID. Returns the lexicon's `publisherView` envelope (decoded + * record + cid + indexedAt + empty labels unless hydrated). + * + * Throws `XRPCError("NotFound")` when no row matches. Publisher deletes + * hard-delete the `publishers` row, so a NotFound covers both the + * never-indexed and the deleted cases — clients treat them equivalently. + */ + +import { json, XRPCError } from "@atcute/xrpc-server"; +import { type AggregatorDefs, type AggregatorGetPublisher } from "@emdash-cms/registry-lexicons"; + +import { parseSignatureMetadataCid } from "../../utils.js"; +import { hydrateLabels, isRedacted } from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { type PublisherRow, publisherColumns, publisherUri, publisherView } from "./views.js"; + +export async function getPublisher( + env: Env, + params: AggregatorGetPublisher.$params, + request: Request, +): Promise { + // `first-primary` for the same reason as getPackage: a label written by + // the labeler between two reads should be reflected on the next read. + const session = env.DB.withSession("first-primary"); + const row = await session + .prepare(`SELECT ${publisherColumns()} FROM publishers WHERE did = ?`) + .bind(params.did) + .first(); + if (!row) { + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No publisher indexed under ${params.did}.`, + }); + } + + const { accepted } = getRequestLabelerPolicy(request); + const uri = publisherUri(row); + const labelsByUri = await hydrateLabels( + session, + accepted, + [ + { uri, currentCid: parseSignatureMetadataCid(row.signature_metadata) ?? undefined }, + { uri: row.did }, + ], + Date.now(), + ); + const labels = [...(labelsByUri.get(uri) ?? []), ...(labelsByUri.get(row.did) ?? [])]; + if (isRedacted(labels, accepted)) { + // Indistinguishable from absence — that is what redaction means. + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No publisher indexed under ${params.did}.`, + }); + } + + const view: AggregatorDefs.PublisherView = publisherView(row, labels); + return json(view); +} diff --git a/apps/aggregator/src/routes/xrpc/getPublisherVerification.ts b/apps/aggregator/src/routes/xrpc/getPublisherVerification.ts new file mode 100644 index 0000000000..3b832bb229 --- /dev/null +++ b/apps/aggregator/src/routes/xrpc/getPublisherVerification.ts @@ -0,0 +1,98 @@ +/** + * `com.emdashcms.experimental.aggregator.getPublisherVerification` — the + * verification state indexed for a publisher DID: the non-tombstoned + * `publisher_verifications` claims naming it as subject, as operator/display + * context. + * + * Redaction runs at two scopes, because a claim is authored by its ISSUER's + * repo, not the subject's: + * - the subject DID gates the whole view (redacted → NotFound), the same rule + * `getPackage` applies to a publisher subject; + * - each claim's issuer DID gates only that claim (redacted issuer → the claim + * is dropped from `verifications`), so one taken-down issuer can't hide a + * publisher's other, still-valid claims. + * An internal caller opting out with a blank `atproto-accept-labelers` header + * has an empty accepted set, so nothing is redacted — it reads every claim, + * including redacted-issuer ones, and weighs trust itself. + * + * An empty `verifications` array is a valid 200 (an unverified publisher, or one + * whose every claim's issuer is redacted for this caller) — a NON-redacted + * subject is never a 404, so 404 (redacted subject) is distinguishable from an + * empty result, unlike `getPackage` where absence is also a 404. + * + * The companion input in plan W8.4 slice 3 — recent handle/profile *changes* — + * has no read here because the aggregator does not ingest a history of them: + * `publishers` is upserted to current state only and handle changes arrive as + * atproto identity events the ingestor does not subscribe to + * (`jetstream-client.ts`). Exposing them is blocked on that ingestion work, + * not on a read path. + */ + +import { json, XRPCError } from "@atcute/xrpc-server"; +import { + type AggregatorDefs, + type AggregatorGetPublisherVerification, +} from "@emdash-cms/registry-lexicons"; + +import { hydrateLabels, isRedacted } from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { + type PublisherVerificationRow, + publisherVerificationColumns, + publisherVerificationView, +} from "./views.js"; + +/** Upper bound on claims returned in one view. A publisher with more than this + * many verification claims is pathological; the view's lexicon `maxLength` + * matches. */ +const MAX_VERIFICATION_CLAIMS = 100; + +export async function getPublisherVerification( + env: Env, + params: AggregatorGetPublisherVerification.$params, + request: Request, +): Promise { + // `first-primary` for read-after-write consistency with a takedown label + // that could land between two reads, mirroring `getPackage`. + const session = env.DB.withSession("first-primary"); + const result = await session + .prepare( + `SELECT ${publisherVerificationColumns()} FROM publisher_verifications + WHERE subject_did = ? AND tombstoned_at IS NULL + ORDER BY created_at DESC, issuer_did ASC + LIMIT ?`, + ) + .bind(params.did, MAX_VERIFICATION_CLAIMS) + .all(); + const rows = result.results ?? []; + + const { accepted } = getRequestLabelerPolicy(request); + // One batched hydration for the subject DID plus every distinct issuer DID + // in the result. `hydrateLabels` chunks its `uri IN (...)` at 50, so this + // stays a single query unless a publisher has 50+ distinct issuers. + const issuerDids = [...new Set(rows.map((row) => row.issuer_did))]; + const subjects = [params.did, ...issuerDids].map((uri) => ({ uri })); + const labelsByUri = await hydrateLabels(session, accepted, subjects, Date.now()); + + const subjectLabels = labelsByUri.get(params.did) ?? []; + if (isRedacted(subjectLabels, accepted)) { + // A redacted publisher DID hides the whole view — the caller can't tell it + // apart from an empty result, which is the point of redaction. + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No publisher indexed under ${params.did}.`, + }); + } + + const visibleRows = rows.filter( + (row) => !isRedacted(labelsByUri.get(row.issuer_did) ?? [], accepted), + ); + + const view: AggregatorDefs.PublisherVerificationView = publisherVerificationView( + params.did, + visibleRows, + subjectLabels, + ); + return json(view); +} diff --git a/apps/aggregator/src/routes/xrpc/label-enforcement.ts b/apps/aggregator/src/routes/xrpc/label-enforcement.ts new file mode 100644 index 0000000000..be1b870dc1 --- /dev/null +++ b/apps/aggregator/src/routes/xrpc/label-enforcement.ts @@ -0,0 +1,325 @@ +/** + * Shared label enforcement (SQL exclusion) and hydration (view `labels`) + * for the Read API, keyed off the request's accepted-labeler policy + * (W4.4, `request-policy.ts`). + * + * Two separate concerns live here: + * - `build*EnforcementSql` appends a `NOT EXISTS` clause that excludes + * subjects carrying a hard-block label from an accepted source. Used + * for search (page-level filtering) and latest-release selection. + * - `hydrateLabels` fetches the actual label rows for a set of subjects + * so views can carry them on the wire; `isRedacted` then decides + * whether a `redact: true` source's `!takedown` should turn a 200 + * into a 404 (`getPackage`/`resolvePackage`/`listReleases`). + * + * Both builders take `nowMs` explicitly (rather than calling `Date.now()` + * internally) so a single request's SQL-side exclusion and hydration pass + * agree on the same instant. + */ + +import { NSID } from "@emdash-cms/registry-lexicons"; +import { + AUTOMATED_BLOCKS, + PACKAGE_SCOPE_BLOCK_VALUES, + RELEASE_BLOCK_VALUES, + type AcceptedLabelerPolicy, +} from "@emdash-cms/registry-moderation"; + +/** Keeps a hydration query's `uri IN (...)` clause within D1's + * bound-parameter limit when a page carries many subjects. */ +const HYDRATION_CHUNK_SIZE = 50; +/** Matches the lexicon's `labels` maxLength on both `packageView` and + * `releaseView`. Applied only at the view boundary (`capLabels`), never to + * hydration results — redaction decisions must see the full label set. */ +export const LABELS_MAX_LENGTH = 64; + +/** Release-scope automated-block values (`RELEASE_BLOCK_VALUES` minus the + * manual `security-yanked` / `!takedown`). The override exception below + * applies only to these — the manual values are never suppressed. */ +const AUTOMATED_BLOCK_VALUES: readonly string[] = [...AUTOMATED_BLOCKS]; +/** The manual release blocks the reviewer override pair never suppresses. */ +const MANUAL_RELEASE_BLOCK_VALUES: readonly string[] = RELEASE_BLOCK_VALUES.filter( + (value) => !AUTOMATED_BLOCKS.has(value), +); +/** Assessment-state label values: `assessment-pending` / `assessment-error` + * steer eligibility, and `assessment-passed` / `assessment-overridden` form + * the reviewer override pair. Enforcement-relevant for view truncation. */ +const ASSESSMENT_STATE_VALUES: readonly string[] = [ + "assessment-pending", + "assessment-passed", + "assessment-overridden", + "assessment-error", +]; + +const ENFORCEMENT_HARD_BLOCK_VALUES: ReadonlySet = new Set([ + ...RELEASE_BLOCK_VALUES, + ...PACKAGE_SCOPE_BLOCK_VALUES, +]); +const ENFORCEMENT_STATE_VALUES: ReadonlySet = new Set(ASSESSMENT_STATE_VALUES); + +/** Truncation priority for a view's `labels`: hard blocks (0) rank ahead of + * assessment states (1) ahead of informational labels (2). Blocks decide a + * client's install/serve refusal, so the lexicon `maxLength` cap must never + * drop one in favour of a display-only label; states change eligibility and + * complete the override pair. Used by `capLabels`. */ +export function labelTruncationPriority(val: string): 0 | 1 | 2 { + if (ENFORCEMENT_HARD_BLOCK_VALUES.has(val)) return 0; + if (ENFORCEMENT_STATE_VALUES.has(val)) return 1; + return 2; +} + +export interface EnforcementSql { + sql: string; + bindings: unknown[]; +} + +/** Matches `com.atproto.label.defs#label`'s optional fields; hydrated + * views never carry `sig` or `ver` (nor `neg` — only active labels are + * hydrated in the first place). */ +export interface LabelView { + src: string; + uri: string; + cid?: string; + val: string; + cts: string; + exp?: string; +} + +export interface HydrationSubject { + uri: string; + /** Omit for a subject with no single current version (publisher DIDs). + * A CID-bound label never applies to such a subject. */ + currentCid?: string; +} + +function inClause(values: readonly string[]): string { + return values.map(() => "?").join(", "); +} + +function chunk(items: readonly T[], size: number): T[][] { + const out: T[][] = []; + for (let index = 0; index < items.length; index += size) + out.push(items.slice(index, index + size)); + return out; +} + +/** + * `NOT EXISTS` clause excluding a package whose profile URI or publisher + * DID carries an active, unexpired `PACKAGE_SCOPE_BLOCK_VALUES` label from + * an accepted source. Empty `accepted` is a no-op (no enforcement without + * an accepted policy to enforce). All values — source DIDs, label values, + * `nowMs` — are bound parameters; only the compile-time NSID constant is + * interpolated into the SQL text. + * + * Deliberately BROADER than `evaluateReleaseModeration`'s subject rules, + * always in the fail-closed (over-hide) direction: the shared CID predicate + * lets a CID-bound label match at DID scope, and value/subject pairings the + * evaluator would reject (e.g. `publisher-compromised` on a package URI) + * still exclude here. Read enforcement hides; the evaluator remains the + * only eligibility engine. + */ +export function buildPackageEnforcementSql( + accepted: AcceptedLabelerPolicy[], + nowMs: number, + alias = "p.", +): EnforcementSql { + if (accepted.length === 0) return { sql: "", bindings: [] }; + assertAlias(alias); + const srcs = accepted.map((policy) => policy.did); + const sql = ` + AND NOT EXISTS ( + SELECT 1 FROM label_state ls + WHERE ls.src IN (${inClause(srcs)}) + AND ls.val IN (${inClause(PACKAGE_SCOPE_BLOCK_VALUES)}) + AND (ls.uri = 'at://' || ${alias}did || '/${NSID.packageProfile}/' || ${alias}slug OR ls.uri = ${alias}did) + AND ls.neg = 0 + AND (ls.exp_epoch_ms IS NULL OR ls.exp_epoch_ms > ?) + AND (ls.cid IS NULL OR ls.cid = json_extract(${alias}signature_metadata, '$.cid')) + ) + `; + return { sql, bindings: [...srcs, ...PACKAGE_SCOPE_BLOCK_VALUES, nowMs] }; +} + +export interface ReleaseEnforcementAliases { + /** Alias (with trailing dot) of the `releases` row. Default `"r."`. */ + release?: string; + /** Alias (with trailing dot) of the joined `packages` row, needed for + * the package-scope cascade's CID comparison. Default `"p."`. */ + package?: string; +} + +/** A SQL row alias with its trailing dot (e.g. `"r."`). */ +const ALIAS_PATTERN = /^[a-zA-Z][a-zA-Z0-9_]*\.$/; + +/** Guards a caller-supplied alias before it is interpolated into raw SQL. + * Callers pass compile-time constants today, but the builders are exported. */ +function assertAlias(alias: string): void { + if (!ALIAS_PATTERN.test(alias)) + throw new TypeError("SQL alias must be an identifier followed by '.'"); +} + +/** + * `NOT EXISTS` clause excluding a release whose own URI carries a + * `RELEASE_BLOCK_VALUES` label, or whose parent package URI / publisher DID + * carries a `PACKAGE_SCOPE_BLOCK_VALUES` label (cascade), from an accepted + * source. Requires the `packages` row joined under `aliases.package` — the + * package-scope branch's CID comparison reads its `signature_metadata`. + * + * Mirrors the hydrated evaluator's §10 override rule (see + * `evaluateReleaseModerationCore` in `@emdash-cms/registry-moderation`): a + * source's exact-CID `assessment-passed` + `assessment-overridden` pair + * suppresses that same source's automated blocks for the release. Without + * this, a post-override re-assessment that re-issues a live automated block + * would keep the release out of latest-selection even though the evaluator + * treats it as eligible. The exception is scoped exactly as the evaluator + * scopes it: automated blocks on the release URI only — the manual + * `security-yanked` / `!takedown` release blocks and the package/publisher + * cascade are never suppressed by the pair. + */ +export function buildReleaseEnforcementSql( + accepted: AcceptedLabelerPolicy[], + nowMs: number, + aliases: ReleaseEnforcementAliases = {}, +): EnforcementSql { + if (accepted.length === 0) return { sql: "", bindings: [] }; + const releaseAlias = aliases.release ?? "r."; + const packageAlias = aliases.package ?? "p."; + assertAlias(releaseAlias); + assertAlias(packageAlias); + const srcs = accepted.map((policy) => policy.did); + const releaseUriExpr = `'at://' || ${releaseAlias}did || '/${NSID.packageRelease}/' || ${releaseAlias}rkey`; + const releaseCidExpr = `json_extract(${releaseAlias}signature_metadata, '$.cid')`; + const sql = ` + AND NOT EXISTS ( + SELECT 1 FROM label_state ls + WHERE ls.src IN (${inClause(srcs)}) + AND ls.neg = 0 + AND (ls.exp_epoch_ms IS NULL OR ls.exp_epoch_ms > ?) + AND ( + ( + ls.uri = ${releaseUriExpr} + AND ls.val IN (${inClause(AUTOMATED_BLOCK_VALUES)}) + AND (ls.cid IS NULL OR ls.cid = ${releaseCidExpr}) + AND NOT ( + EXISTS ( + SELECT 1 FROM label_state pass + WHERE pass.src = ls.src + AND pass.uri = ${releaseUriExpr} + AND pass.val = 'assessment-passed' + AND pass.cid = ${releaseCidExpr} + AND pass.neg = 0 + AND (pass.exp_epoch_ms IS NULL OR pass.exp_epoch_ms > ?) + ) + AND EXISTS ( + SELECT 1 FROM label_state ovr + WHERE ovr.src = ls.src + AND ovr.uri = ${releaseUriExpr} + AND ovr.val = 'assessment-overridden' + AND ovr.cid = ${releaseCidExpr} + AND ovr.neg = 0 + AND (ovr.exp_epoch_ms IS NULL OR ovr.exp_epoch_ms > ?) + ) + ) + ) + OR + ( + ls.uri = ${releaseUriExpr} + AND ls.val IN (${inClause(MANUAL_RELEASE_BLOCK_VALUES)}) + AND (ls.cid IS NULL OR ls.cid = ${releaseCidExpr}) + ) + OR + ( + (ls.uri = 'at://' || ${packageAlias}did || '/${NSID.packageProfile}/' || ${packageAlias}slug OR ls.uri = ${releaseAlias}did) + AND ls.val IN (${inClause(PACKAGE_SCOPE_BLOCK_VALUES)}) + AND (ls.cid IS NULL OR ls.cid = json_extract(${packageAlias}signature_metadata, '$.cid')) + ) + ) + ) + `; + return { + sql, + bindings: [ + ...srcs, + nowMs, + ...AUTOMATED_BLOCK_VALUES, + nowMs, + nowMs, + ...MANUAL_RELEASE_BLOCK_VALUES, + ...PACKAGE_SCOPE_BLOCK_VALUES, + ], + }; +} + +interface LabelStateRow { + src: string; + uri: string; + cid: string | null; + val: string; + cts: string; + exp: string | null; +} + +/** + * Fetches active, unexpired label rows from accepted sources for a set of + * subjects, applying CID applicability app-side (a CID-bound label applies + * only when it matches the subject's current CID; `cid IS NULL` is + * URI-wide). Returns a `Map` keyed by subject URI — callers combine the + * entries relevant to one view (e.g. release + package + publisher URIs). + * + * Returns UNTRUNCATED label sets: `isRedacted` must see every applicable + * label, or a `!takedown` past a truncation boundary silently un-redacts + * the subject. The lexicon's 64-label wire cap is applied by the views + * (`capLabels`), after redaction has been decided. + */ +export async function hydrateLabels( + db: D1DatabaseSession, + accepted: AcceptedLabelerPolicy[], + subjects: HydrationSubject[], + nowMs: number, +): Promise> { + const out = new Map(); + if (accepted.length === 0 || subjects.length === 0) return out; + + const currentCidByUri = new Map(); + const uris: string[] = []; + for (const subject of subjects) { + if (!currentCidByUri.has(subject.uri)) uris.push(subject.uri); + currentCidByUri.set(subject.uri, subject.currentCid); + } + const srcs = accepted.map((policy) => policy.did); + + for (const batch of chunk(uris, HYDRATION_CHUNK_SIZE)) { + const result = await db + .prepare( + `SELECT src, uri, cid, val, cts, exp + FROM label_state + WHERE uri IN (${inClause(batch)}) + AND src IN (${inClause(srcs)}) + AND neg = 0 + AND (exp_epoch_ms IS NULL OR exp_epoch_ms > ?)`, + ) + .bind(...batch, ...srcs, nowMs) + .all(); + + for (const row of result.results ?? []) { + const currentCid = currentCidByUri.get(row.uri); + if (row.cid !== null && row.cid !== currentCid) continue; + const label: LabelView = { src: row.src, uri: row.uri, val: row.val, cts: row.cts }; + if (row.cid !== null) label.cid = row.cid; + if (row.exp !== null) label.exp = row.exp; + const existing = out.get(row.uri); + if (existing) existing.push(label); + else out.set(row.uri, [label]); + } + } + + return out; +} + +/** True iff any hydrated label is an active `!takedown` from a source + * whose accepted policy has `redact: true`. Applicability (CID scoping, + * expiry, negation) was already established by `hydrateLabels`. */ +export function isRedacted(labels: LabelView[], accepted: AcceptedLabelerPolicy[]): boolean { + const redactBySrc = new Map(accepted.map((policy) => [policy.did, policy.redact])); + return labels.some((label) => label.val === "!takedown" && redactBySrc.get(label.src) === true); +} diff --git a/apps/aggregator/src/routes/xrpc/listReleases.ts b/apps/aggregator/src/routes/xrpc/listReleases.ts index 9a44b74f41..133c7bfa29 100644 --- a/apps/aggregator/src/routes/xrpc/listReleases.ts +++ b/apps/aggregator/src/routes/xrpc/listReleases.ts @@ -13,8 +13,11 @@ import { InvalidRequestError, json, XRPCError } from "@atcute/xrpc-server"; import { type AggregatorDefs, type AggregatorListReleases } from "@emdash-cms/registry-lexicons"; +import { parseSignatureMetadataCid } from "../../utils.js"; import { decodeListCursor, encodeListCursor, InvalidCursorError } from "./cursor.js"; -import { type ReleaseRow, releaseColumns, releaseView } from "./views.js"; +import { type HydrationSubject, hydrateLabels, isRedacted } from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { type ReleaseRow, packageUri, releaseColumns, releaseUri, releaseView } from "./views.js"; const DEFAULT_LIMIT = 25; const MAX_LIMIT = 100; @@ -22,20 +25,24 @@ const MAX_LIMIT = 100; export async function listReleases( env: Env, params: AggregatorListReleases.$params, + request: Request, ): Promise { const limit = clampLimit(params.limit); const session = env.DB.withSession("first-primary"); + const { accepted } = getRequestLabelerPolicy(request); + const nowMs = Date.now(); - // Confirm parent package exists. One extra D1 read per request — could be - // folded into a JOIN, but the explicit existence check keeps the NotFound - // signal cheap and unambiguous (the empty-list response shape would - // otherwise mean "package exists, no releases" or "package doesn't exist" - // indistinguishably). - const parentExists = await session - .prepare(`SELECT 1 AS hit FROM packages WHERE did = ? AND slug = ?`) + // Confirm parent package exists, fetching its `signature_metadata` too — + // package-scope hydration below needs the current CID. One extra D1 read + // per request — could be folded into a JOIN, but the explicit existence + // check keeps the NotFound signal cheap and unambiguous (the empty-list + // response shape would otherwise mean "package exists, no releases" or + // "package doesn't exist" indistinguishably). + const parent = await session + .prepare(`SELECT signature_metadata FROM packages WHERE did = ? AND slug = ?`) .bind(params.did, params.package) - .first<{ hit: number }>(); - if (!parentExists) { + .first<{ signature_metadata: string | null }>(); + if (!parent) { throw new XRPCError({ status: 404, error: "NotFound", @@ -84,14 +91,57 @@ export async function listReleases( // Read limit+1 to detect a next page without a trailing COUNT query. const hasMore = items.length > limit; const page = hasMore ? items.slice(0, limit) : items; + // Cursor derives from the last FETCHED row, before label-redaction + // filtering below — a page shortened by redacted omissions must still + // advance the cursor past every row the caller saw, not just the ones + // it returned. const last = page.at(-1); + const parentPackageUri = packageUri({ did: params.did, slug: params.package }); + const parentPackageCid = parseSignatureMetadataCid(parent.signature_metadata) ?? undefined; + const subjects: HydrationSubject[] = [ + { uri: parentPackageUri, currentCid: parentPackageCid }, + { uri: params.did }, + ]; + for (const row of page) { + subjects.push({ + uri: releaseUri(row), + currentCid: parseSignatureMetadataCid(row.signature_metadata) ?? undefined, + }); + } + const labelsByUri = await hydrateLabels(session, accepted, subjects, nowMs); + + // A redacted parent package must be indistinguishable from an absent one, + // matching getPackage — a 200 with an empty list would leak that the + // package exists but was taken down. + const parentLabels = [ + ...(labelsByUri.get(parentPackageUri) ?? []), + ...(labelsByUri.get(params.did) ?? []), + ]; + if (isRedacted(parentLabels, accepted)) { + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No package indexed under (${params.did}, ${params.package}).`, + }); + } + + const releases: AggregatorDefs.ReleaseView[] = []; + for (const row of page) { + const uri = releaseUri(row); + const labels = [ + ...(labelsByUri.get(uri) ?? []), + ...(labelsByUri.get(parentPackageUri) ?? []), + ...(labelsByUri.get(params.did) ?? []), + ]; + if (isRedacted(labels, accepted)) continue; + releases.push(releaseView(row, labels)); + } + const response: { releases: AggregatorDefs.ReleaseView[]; cursor?: string; - } = { - releases: page.map(releaseView), - }; + } = { releases }; if (hasMore && last) { // Cursor encodes the internal `version_sort` format. If the // `computeVersionSort` encoding ever changes, in-flight cursors @@ -99,6 +149,12 @@ export async function listReleases( // 400 (per the strict-cursor policy) and fall back to fetching // page 1. Acceptable for the experimental NSID; revisit if/when // we stabilise. + // + // An all-redacted page returns an empty list WITH a cursor, which + // lets a caller infer hidden rows exist. Accepted trade-off: the + // alternatives either stop pagination early (cursor only when a row + // survived) or re-page server-side until the page fills. Package- + // level redaction — the sensitive case — 404s above instead. response.cursor = encodeListCursor({ versionSort: last.version_sort, version: last.version }); } return json(response); diff --git a/apps/aggregator/src/routes/xrpc/request-policy.ts b/apps/aggregator/src/routes/xrpc/request-policy.ts new file mode 100644 index 0000000000..8463249b40 --- /dev/null +++ b/apps/aggregator/src/routes/xrpc/request-policy.ts @@ -0,0 +1,109 @@ +/** + * Resolves the accepted-labelers policy for one aggregator request from the + * `atproto-accept-labelers` request header (W4.4). `handleXrpc` resolves this + * once per request before dispatch; handlers read the result back via + * `getRequestLabelerPolicy`. + */ + +import { InvalidRequestError } from "@atcute/xrpc-server"; +import { + InvalidAcceptLabelersHeaderError, + parseAcceptLabelersHeader, + serializeContentLabelersHeader, + type AcceptedLabelerPolicy, +} from "@emdash-cms/registry-moderation"; + +const ACCEPT_LABELERS_HEADER = "atproto-accept-labelers"; +// Keeps `IN (...)` clauses within D1's bound-parameter limit. +const AVAILABILITY_CHUNK_SIZE = 50; + +export interface RequestLabelerPolicy { + /** Effective, deduped, availability-filtered accepted labelers. */ + accepted: AcceptedLabelerPolicy[]; + /** `atproto-content-labelers` header value; "" means the header is omitted. */ + contentLabelersHeader: string; +} + +function chunk(items: T[], size: number): T[][] { + const out: T[][] = []; + for (let index = 0; index < items.length; index += size) + out.push(items.slice(index, index + size)); + return out; +} + +async function defaultAcceptedPolicy(env: Env): Promise { + const session = env.DB.withSession("first-primary"); + const result = await session + .prepare(`SELECT did FROM labelers WHERE trusted = 1 ORDER BY did ASC`) + .all<{ did: string }>(); + return (result.results ?? []).map((row) => ({ did: row.did, redact: true })); +} + +/** Drops requested DIDs the deployment doesn't ingest, preserving request order. */ +async function filterAvailable( + env: Env, + requested: AcceptedLabelerPolicy[], +): Promise { + if (requested.length === 0) return []; + const session = env.DB.withSession("first-primary"); + const available = new Set(); + for (const batch of chunk( + requested.map((policy) => policy.did), + AVAILABILITY_CHUNK_SIZE, + )) { + const placeholders = batch.map(() => "?").join(", "); + const result = await session + .prepare(`SELECT did FROM labelers WHERE did IN (${placeholders})`) + .bind(...batch) + .all<{ did: string }>(); + for (const row of result.results ?? []) available.add(row.did); + } + return requested.filter((policy) => available.has(policy.did)); +} + +const requestPolicies = new WeakMap(); + +/** + * Resolves and stashes the request's labeler policy. Call exactly once per + * request, before dispatching to a handler. Throws `InvalidRequestError` for + * malformed header syntax; any other rejection (e.g. a D1 error) propagates + * as-is so the caller 500s rather than failing open to an empty policy. + */ +export async function resolveRequestLabelerPolicy( + env: Env, + request: Request, +): Promise { + const header = request.headers.get(ACCEPT_LABELERS_HEADER); + let accepted: AcceptedLabelerPolicy[]; + if (header === null) { + accepted = await defaultAcceptedPolicy(env); + } else { + let requested: AcceptedLabelerPolicy[]; + try { + requested = parseAcceptLabelersHeader(header); + } catch (err) { + if (err instanceof InvalidAcceptLabelersHeaderError) { + throw new InvalidRequestError({ + message: `invalid ${ACCEPT_LABELERS_HEADER} header: ${err.message}`, + }); + } + throw err; + } + accepted = await filterAvailable(env, requested); + } + const policy: RequestLabelerPolicy = { + accepted, + contentLabelersHeader: serializeContentLabelersHeader(accepted), + }; + requestPolicies.set(request, policy); + return policy; +} + +/** Reads back the policy `resolveRequestLabelerPolicy` stashed for this request. */ +export function getRequestLabelerPolicy(request: Request): RequestLabelerPolicy { + const policy = requestPolicies.get(request); + if (!policy) { + throw new Error("request labeler policy was not resolved before handler dispatch"); + } + return policy; +} diff --git a/apps/aggregator/src/routes/xrpc/resolvePackage.ts b/apps/aggregator/src/routes/xrpc/resolvePackage.ts index 3ae1275339..fddd58942e 100644 --- a/apps/aggregator/src/routes/xrpc/resolvePackage.ts +++ b/apps/aggregator/src/routes/xrpc/resolvePackage.ts @@ -25,8 +25,10 @@ import { import { json, XRPCError } from "@atcute/xrpc-server"; import { type AggregatorResolvePackage } from "@emdash-cms/registry-lexicons"; -import { boundFetch } from "../../utils.js"; -import { type PackageRow, packageColumns, packageView } from "./views.js"; +import { boundFetch, parseSignatureMetadataCid } from "../../utils.js"; +import { hydrateLabels, isRedacted } from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { type PackageRow, packageColumns, packageUri, packageView } from "./views.js"; /** Cache the resolver per worker isolate. Construction is allocation-only * (no I/O), but reusing a single instance avoids per-request setup. */ @@ -50,6 +52,7 @@ function getHandleResolver(): CompositeHandleResolver { export async function resolvePackage( env: Env, params: AggregatorResolvePackage.$params, + request: Request, ): Promise { let did: string; try { @@ -77,7 +80,28 @@ export async function resolvePackage( message: `No package indexed under resolved (${did}, ${params.slug}).`, }); } - const view = packageView(row); + + const { accepted } = getRequestLabelerPolicy(request); + const uri = packageUri(row); + const labelsByUri = await hydrateLabels( + session, + accepted, + [ + { uri, currentCid: parseSignatureMetadataCid(row.signature_metadata) ?? undefined }, + { uri: row.did }, + ], + Date.now(), + ); + const labels = [...(labelsByUri.get(uri) ?? []), ...(labelsByUri.get(row.did) ?? [])]; + if (isRedacted(labels, accepted)) { + throw new XRPCError({ + status: 404, + error: "NotFound", + message: `No package indexed under resolved (${did}, ${params.slug}).`, + }); + } + + const view = packageView(row, labels); // Surface the handle we resolved — the lexicon's view has an optional // `handle` field for exactly this case (best-effort current handle). view.handle = params.handle; diff --git a/apps/aggregator/src/routes/xrpc/router.ts b/apps/aggregator/src/routes/xrpc/router.ts index a38cb78be4..4c3f219fe7 100644 --- a/apps/aggregator/src/routes/xrpc/router.ts +++ b/apps/aggregator/src/routes/xrpc/router.ts @@ -21,10 +21,12 @@ * (immutable bytes). */ -import { XRPCRouter } from "@atcute/xrpc-server"; +import { InternalServerError, XRPCError, XRPCRouter } from "@atcute/xrpc-server"; import { AggregatorGetLatestRelease, AggregatorGetPackage, + AggregatorGetPublisher, + AggregatorGetPublisherVerification, AggregatorListReleases, AggregatorResolvePackage, AggregatorSearchPackages, @@ -32,7 +34,10 @@ import { import { getLatestRelease } from "./getLatestRelease.js"; import { getPackage } from "./getPackage.js"; +import { getPublisher } from "./getPublisher.js"; +import { getPublisherVerification } from "./getPublisherVerification.js"; import { listReleases } from "./listReleases.js"; +import { resolveRequestLabelerPolicy } from "./request-policy.js"; import { resolvePackage } from "./resolvePackage.js"; import { searchPackages } from "./searchPackages.js"; import { syncGetRecord } from "./sync-get-record.js"; @@ -51,15 +56,16 @@ const SYNC_GET_RECORD_PATH = "/xrpc/com.atproto.sync.getRecord"; * `*` is correct here because nothing in our responses depends on the * caller's origin or credentials -- there are no cookies, no auth, no * per-origin policy. We allow `atproto-accept-labelers` and - * `content-type` as request headers (the only two clients send), echo - * back the labellers header for symmetry with atproto's labeller-aware - * clients, and cap preflight cache at 24h. + * `content-type` as request headers (the only two clients send), expose + * the response headers a labeler-aware client needs to read + * (`atproto-content-labelers`, `content-language`), and cap preflight + * cache at 24h. */ const CORS_HEADERS: Record = { "access-control-allow-origin": "*", "access-control-allow-methods": "GET, POST, OPTIONS", "access-control-allow-headers": "content-type, atproto-accept-labelers", - "access-control-expose-headers": "atproto-accept-labelers, content-language", + "access-control-expose-headers": "atproto-content-labelers, content-language", "access-control-max-age": "86400", }; @@ -69,6 +75,37 @@ function applyCorsHeaders(headers: Headers): void { } } +/** + * Generic 500 for an unexpected failure (e.g. a D1 error). Logs the internal + * error under `context` for operators but returns the router's own opaque + * envelope, so no internal detail (SQL text, stack) reaches the client. + * Matches the body the router itself produces for a handler throw. + */ +function internalErrorResponse(err: unknown, context: string): Response { + console.error(`[aggregator] xrpc ${context} failed`, { + error: err instanceof Error ? (err.stack ?? err.message) : String(err), + }); + return new InternalServerError({ + message: "an exception happened whilst processing this request", + }).toResponse(); +} + +/** + * Wrap an unexpected dispatch failure as an error Response carrying CORS + + * `no-store`. An unwrapped throw escapes to workerd's bare 500, dropping both + * — and on the cacheable `sync.getRecord` path it would leave a takedown- + * relevant error under a public cache header. `XRPCError` keeps its typed + * envelope (e.g. a malformed accept-labelers header); anything else is opaque. + */ +function wrapDispatchError(err: unknown, context: string): Response { + const errorResponse = + err instanceof XRPCError ? err.toResponse() : internalErrorResponse(err, context); + const headers = new Headers(errorResponse.headers); + headers.set("cache-control", NO_STORE); + applyCorsHeaders(headers); + return new Response(errorResponse.body, { status: errorResponse.status, headers }); +} + /** * Dispatch any `/xrpc/*` request. Returns null when the path isn't an * XRPC route (caller falls through to other route matching). @@ -87,7 +124,12 @@ export async function handleXrpc(env: Env, request: Request): Promise>; + try { + policy = await resolveRequestLabelerPolicy(env, request); + } catch (err) { + return wrapDispatchError(err, "policy resolution"); + } + const router = getRouter(env); const response = await router.fetch(request); // Override Cache-Control unconditionally on aggregator endpoints — the @@ -109,6 +164,9 @@ export async function handleXrpc(env: Env, request: Request): Promise getPackage(env, params), + handler: ({ params, request }) => getPackage(env, params, request), }); router.addQuery(AggregatorListReleases.mainSchema, { - handler: ({ params }) => listReleases(env, params), + handler: ({ params, request }) => listReleases(env, params, request), }); router.addQuery(AggregatorGetLatestRelease.mainSchema, { - handler: ({ params }) => getLatestRelease(env, params), + handler: ({ params, request }) => getLatestRelease(env, params, request), }); router.addQuery(AggregatorSearchPackages.mainSchema, { - handler: ({ params }) => searchPackages(env, params), + handler: ({ params, request }) => searchPackages(env, params, request), }); router.addQuery(AggregatorResolvePackage.mainSchema, { - handler: ({ params }) => resolvePackage(env, params), + handler: ({ params, request }) => resolvePackage(env, params, request), + }); + router.addQuery(AggregatorGetPublisher.mainSchema, { + handler: ({ params, request }) => getPublisher(env, params, request), + }); + router.addQuery(AggregatorGetPublisherVerification.mainSchema, { + handler: ({ params, request }) => getPublisherVerification(env, params, request), }); return router; } diff --git a/apps/aggregator/src/routes/xrpc/searchPackages.ts b/apps/aggregator/src/routes/xrpc/searchPackages.ts index ee377e4370..09b4af7257 100644 --- a/apps/aggregator/src/routes/xrpc/searchPackages.ts +++ b/apps/aggregator/src/routes/xrpc/searchPackages.ts @@ -9,10 +9,10 @@ * pagination scans more rows. At Slice 1 scale (hundreds of packages) it's * a non-issue. * - * Takedown filter joins `label_state` even though that table is empty in - * Slice 1 — keeps the contract honest for Slice 2 (when the labeller starts - * writing), and the optimiser short-circuits the NOT EXISTS subquery - * cheaply when no rows match. See plan §Search. + * Enforcement excludes packages carrying an active `PACKAGE_SCOPE_BLOCK_VALUES` + * label from a source the request's accepted-labeler policy (W4.4) covers — + * see `buildPackageEnforcementSql`. Other label state is hydrated on each + * page result so clients can make their own eligibility call. * * `q` is passed directly to FTS5 MATCH. Special characters in user input * are escaped via `quoteFtsQuery` so a stray `"`/`*`/`(` doesn't blow up @@ -21,14 +21,18 @@ */ import { InvalidRequestError, json } from "@atcute/xrpc-server"; -import { - type AggregatorDefs, - NSID, - type AggregatorSearchPackages, -} from "@emdash-cms/registry-lexicons"; +import { type AggregatorDefs, type AggregatorSearchPackages } from "@emdash-cms/registry-lexicons"; +import { parseSignatureMetadataCid } from "../../utils.js"; import { decodeOffsetCursor, encodeOffsetCursor, InvalidCursorError } from "./cursor.js"; -import { type PackageRow, packageColumns, packageView } from "./views.js"; +import { + buildPackageEnforcementSql, + type EnforcementSql, + type HydrationSubject, + hydrateLabels, +} from "./label-enforcement.js"; +import { getRequestLabelerPolicy } from "./request-policy.js"; +import { type PackageRow, packageColumns, packageUri, packageView } from "./views.js"; const DEFAULT_LIMIT = 25; const MAX_LIMIT = 100; @@ -36,6 +40,7 @@ const MAX_LIMIT = 100; export async function searchPackages( env: Env, params: AggregatorSearchPackages.$params, + request: Request, ): Promise { const limit = clampLimit(params.limit); let offset: number; @@ -48,6 +53,9 @@ export async function searchPackages( throw err; } const session = env.DB.withSession("first-primary"); + const { accepted } = getRequestLabelerPolicy(request); + const nowMs = Date.now(); + const enforcement = buildPackageEnforcementSql(accepted, nowMs); const hasQuery = typeof params.q === "string" && params.q.trim().length > 0; const hasCapability = typeof params.capability === "string" && params.capability.length > 0; @@ -56,10 +64,11 @@ export async function searchPackages( if (hasQuery) { const ftsQuery = quoteFtsQuery(params.q!); const result = await session - .prepare(buildFtsSearchSql(hasCapability)) + .prepare(buildFtsSearchSql(hasCapability, enforcement.sql)) .bind( ...buildFtsBindings( ftsQuery, + enforcement.bindings, hasCapability ? params.capability : undefined, limit + 1, offset, @@ -71,9 +80,14 @@ export async function searchPackages( // No query → ordered list of all packages, label-filtered. last_updated // DESC keeps the "what's new" view sensible for an empty search box. const result = await session - .prepare(buildBrowseSql(hasCapability)) + .prepare(buildBrowseSql(hasCapability, enforcement.sql)) .bind( - ...buildBrowseBindings(hasCapability ? params.capability : undefined, limit + 1, offset), + ...buildBrowseBindings( + enforcement.bindings, + hasCapability ? params.capability : undefined, + limit + 1, + offset, + ), ) .all(); rows = result.results ?? []; @@ -82,23 +96,41 @@ export async function searchPackages( const hasMore = rows.length > limit; const page = hasMore ? rows.slice(0, limit) : rows; + const subjects: HydrationSubject[] = []; + const seenDids = new Set(); + for (const row of page) { + subjects.push({ + uri: packageUri(row), + currentCid: parseSignatureMetadataCid(row.signature_metadata) ?? undefined, + }); + if (!seenDids.has(row.did)) { + seenDids.add(row.did); + subjects.push({ uri: row.did }); + } + } + const labelsByUri = await hydrateLabels(session, accepted, subjects, nowMs); + const response: { packages: AggregatorDefs.PackageView[]; cursor?: string; } = { - packages: page.map(packageView), + packages: page.map((row) => { + const uri = packageUri(row); + const labels = [...(labelsByUri.get(uri) ?? []), ...(labelsByUri.get(row.did) ?? [])]; + return packageView(row, labels); + }), }; if (hasMore) response.cursor = encodeOffsetCursor({ offset: offset + limit }); return json(response); } -function buildFtsSearchSql(hasCapability: boolean): string { +function buildFtsSearchSql(hasCapability: boolean, enforcementSql: string): string { return ` SELECT ${packageColumns("p.")} FROM packages_fts JOIN packages p ON p.rowid = packages_fts.rowid WHERE packages_fts MATCH ? - ${ENFORCEMENT_FILTER_SQL} + ${enforcementSql} ${hasCapability ? CAPABILITY_FILTER_SQL : ""} ORDER BY bm25(packages_fts), p.last_updated DESC, p.did ASC, p.slug ASC LIMIT ? OFFSET ? @@ -107,17 +139,18 @@ function buildFtsSearchSql(hasCapability: boolean): string { function buildFtsBindings( ftsQuery: string, + enforcementBindings: EnforcementSql["bindings"], capability: string | undefined, limit: number, offset: number, ): unknown[] { - const out: unknown[] = [ftsQuery]; + const out: unknown[] = [ftsQuery, ...enforcementBindings]; if (capability !== undefined) out.push(capability); out.push(limit, offset); return out; } -function buildBrowseSql(hasCapability: boolean): string { +function buildBrowseSql(hasCapability: boolean, enforcementSql: string): string { // Stable tiebreakers (did, slug) so offset pagination doesn't shuffle // rows across pages when many packages share `last_updated` (or it's // NULL — `last_updated` comes from the optional record.lastUpdated @@ -127,7 +160,7 @@ function buildBrowseSql(hasCapability: boolean): string { SELECT ${packageColumns("p.")} FROM packages p WHERE 1=1 - ${ENFORCEMENT_FILTER_SQL} + ${enforcementSql} ${hasCapability ? CAPABILITY_FILTER_SQL : ""} ORDER BY p.last_updated IS NULL, p.last_updated DESC, p.did ASC, p.slug ASC LIMIT ? OFFSET ? @@ -135,32 +168,17 @@ function buildBrowseSql(hasCapability: boolean): string { } function buildBrowseBindings( + enforcementBindings: EnforcementSql["bindings"], capability: string | undefined, limit: number, offset: number, ): unknown[] { - const out: unknown[] = []; + const out: unknown[] = [...enforcementBindings]; if (capability !== undefined) out.push(capability); out.push(limit, offset); return out; } -/** Hard-enforcement label filter. The Slice 2 labeller writes to - * `label_state`; until then the table is empty so this clause is a no-op - * (the NOT EXISTS short-circuits). The plan calls for shipping the filter - * now to lock the contract — adding it later would be a behaviour change - * for cached clients. */ -const ENFORCEMENT_FILTER_SQL = ` - AND NOT EXISTS ( - SELECT 1 FROM label_state ls - WHERE ls.uri = 'at://' || p.did || '/${NSID.packageProfile}/' || p.slug - AND ls.val IN ('!takedown', 'security:yanked') - AND ls.trusted = 1 - AND ls.neg = 0 - AND (ls.exp IS NULL OR ls.exp > strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) - ) -`; - const CAPABILITY_FILTER_SQL = ` AND p.capabilities IS NOT NULL AND EXISTS (SELECT 1 FROM json_each(p.capabilities) WHERE value = ?) diff --git a/apps/aggregator/src/routes/xrpc/views.ts b/apps/aggregator/src/routes/xrpc/views.ts index 97703b70b1..87be393bad 100644 --- a/apps/aggregator/src/routes/xrpc/views.ts +++ b/apps/aggregator/src/routes/xrpc/views.ts @@ -21,6 +21,7 @@ import { type AggregatorDefs, NSID } from "@emdash-cms/registry-lexicons"; import { isPlainObject, parseSignatureMetadataCid } from "../../utils.js"; +import { LABELS_MAX_LENGTH, labelTruncationPriority, type LabelView } from "./label-enforcement.js"; /** Subset of columns from `packages` we read for `packageView`. Selecting * exactly these columns keeps the SQL query auditable and cheap. */ @@ -42,6 +43,19 @@ export interface PackageRow { indexed_at: string | null; } +/** Subset of columns from `publishers` we read for `publisherView`. */ +export interface PublisherRow { + did: string; + display_name: string; + description: string | null; + url: string | null; + contact: string | null; // JSON array of { kind, url?, email? } + updated_at: string | null; + signature_metadata: string | null; + verified_at: string; + indexed_at: string | null; +} + /** Subset of columns from `releases` we read for `releaseView`. */ export interface ReleaseRow { did: string; @@ -78,6 +92,19 @@ const PACKAGE_VIEW_COLUMN_NAMES = [ "indexed_at", ] as const; +/** Column list backing `PublisherRow`. */ +const PUBLISHER_VIEW_COLUMN_NAMES = [ + "did", + "display_name", + "description", + "url", + "contact", + "updated_at", + "signature_metadata", + "verified_at", + "indexed_at", +] as const; + /** Column list backing `ReleaseRow`. */ const RELEASE_VIEW_COLUMN_NAMES = [ "did", @@ -107,6 +134,84 @@ export function releaseColumns(prefix = ""): string { return RELEASE_VIEW_COLUMN_NAMES.map((c) => `${prefix}${c}`).join(", "); } +/** SELECT-clause string for `PublisherRow`, optionally prefixed for JOINs. */ +export function publisherColumns(prefix = ""): string { + return PUBLISHER_VIEW_COLUMN_NAMES.map((c) => `${prefix}${c}`).join(", "); +} + +/** AT URI of a package's profile record — the `packageView.uri` and the + * hydration subject handlers hydrate labels for before calling + * `packageView`. Single source of truth so the two never drift. No return + * type annotation, and `as const` on the template literal: together these + * keep the inferred type as a template-literal pattern rather than + * widening to plain `string`, which `packageView.uri`'s `ResourceUri` + * type needs. */ +export function packageUri(row: Pick) { + return `at://${row.did}/${NSID.packageProfile}/${row.slug}` as const; +} + +/** AT URI of a release record — the `releaseView.uri` and the hydration + * subject handlers hydrate labels for before calling `releaseView`. */ +export function releaseUri(row: Pick) { + return `at://${row.did}/${NSID.packageRelease}/${row.rkey}` as const; +} + +/** AT URI of a publisher's profile record — the `publisherView.uri` and a + * label-hydration subject. The rkey is always `self` (enforced at ingest). */ +export function publisherUri(row: Pick) { + return `at://${row.did}/${NSID.publisherProfile}/self` as const; +} + +/** Caps a combined labels array at the lexicon's maxLength. A view's + * labels are the union of multiple hydrated subjects (e.g. a release's own + * URI plus its parent package and publisher DID). Hydration returns + * untruncated per-subject sets so redaction decisions see every label; + * this boundary cap trims only the final view array. + * + * The trim is enforcement-preserving: a plain slice can drop the only hard + * block when a subject carries more than `LABELS_MAX_LENGTH` labels, and a + * client evaluating the truncated view would then treat the release as + * installable. Labels are ordered by `labelTruncationPriority` (blocks, then + * assessment states, then informational) before slicing, so blocks survive. + * Hydrated labels never carry negations (`hydrateLabels` returns only + * `neg = 0` rows), so ordering can't split a label from its negation. In the + * pathological case of more than `LABELS_MAX_LENGTH` hard blocks the kept + * slice is still all blocks — the release stays blocked, only the exact set + * is lossy — and we log an error. */ +function capLabels(labels: LabelView[], uri: string): LabelView[] { + if (labels.length <= LABELS_MAX_LENGTH) return labels; + const ordered = labels.toSorted( + (left, right) => labelTruncationPriority(left.val) - labelTruncationPriority(right.val), + ); + const kept = ordered.slice(0, LABELS_MAX_LENGTH); + const droppedBlocks = ordered + .slice(LABELS_MAX_LENGTH) + .filter((label) => labelTruncationPriority(label.val) === 0).length; + if (droppedBlocks > 0) { + console.error("[aggregator] view label cap dropped hard-block labels", { + uri, + count: labels.length, + cap: LABELS_MAX_LENGTH, + droppedBlocks, + }); + } else { + console.warn("[aggregator] view labels truncated to maxLength", { + uri, + count: labels.length, + cap: LABELS_MAX_LENGTH, + }); + } + return kept; +} + +/** `LabelView.src` is `string`; the lexicon's `Label.src` is a branded DID + * template type. Safe to assert — `src` is a labeler DID validated at + * ingest time, same trust boundary as the `did` cast below. */ +function toLexiconLabels(labels: LabelView[]): AggregatorDefs.PackageView["labels"] { + // eslint-disable-next-line typescript/no-unsafe-type-assertion + return labels as AggregatorDefs.PackageView["labels"]; +} + /** * Map a `packages` row to the lexicon's `packageView`. The synthesized * `profile` field reconstructs the package.profile record JSON from the @@ -116,9 +221,13 @@ export function releaseColumns(prefix = ""): string { * `indexedAt` falls back to `verified_at` for any historical row that * predates migration 0002 (`indexed_at` is nullable at the schema level — * see migration comment). + * + * `labels` are hydrated by the caller (package URI + publisher DID + * subjects) and passed in; defaulting to `[]` covers the accepted-policy- + * empty case where there's nothing to hydrate. */ -export function packageView(row: PackageRow): AggregatorDefs.PackageView { - const uri = `at://${row.did}/${NSID.packageProfile}/${row.slug}` as const; +export function packageView(row: PackageRow, labels: LabelView[] = []): AggregatorDefs.PackageView { + const uri = packageUri(row); const cid = parseSignatureMetadataCid(row.signature_metadata) ?? ""; // `mirrors` is on releaseView, not packageView — packages aren't // mirrored, releases are. Don't add it here even though they share the @@ -131,7 +240,7 @@ export function packageView(row: PackageRow): AggregatorDefs.PackageView { slug: row.slug, profile: synthesizePackageProfile(row, uri), indexedAt: row.indexed_at ?? row.verified_at, - labels: [], + labels: toLexiconLabels(capLabels(labels, uri)), }; if (row.latest_version !== null) { view.latestVersion = row.latest_version; @@ -145,9 +254,12 @@ export function packageView(row: PackageRow): AggregatorDefs.PackageView { * normalised columns. `mirrors: []` is intentional — the artifact mirror * worker (Slice 3) is what populates real mirror URLs; until then the * field is the empty contract. + * + * `labels` are hydrated by the caller (release URI + package URI + + * publisher DID subjects, carrying cascade context) and passed in. */ -export function releaseView(row: ReleaseRow): AggregatorDefs.ReleaseView { - const uri = `at://${row.did}/${NSID.packageRelease}/${row.rkey}` as const; +export function releaseView(row: ReleaseRow, labels: LabelView[] = []): AggregatorDefs.ReleaseView { + const uri = releaseUri(row); const cid = parseSignatureMetadataCid(row.signature_metadata) ?? ""; return { uri, @@ -159,10 +271,54 @@ export function releaseView(row: ReleaseRow): AggregatorDefs.ReleaseView { release: synthesizePackageRelease(row), mirrors: [], indexedAt: row.indexed_at ?? row.verified_at, - labels: [], + labels: toLexiconLabels(capLabels(labels, uri)), }; } +/** + * Map a `publishers` row to the lexicon's `publisherView`. The synthesized + * `profile` field reconstructs the publisher.profile record JSON from the + * normalised columns — same field values the publisher signed. For + * byte-identical bytes, clients call `sync.getRecord` and re-verify. + * + * `labels` are hydrated by the caller (profile URI + publisher DID subjects) + * and passed in; defaulting to `[]` covers the accepted-policy-empty case. + */ +export function publisherView( + row: PublisherRow, + labels: LabelView[] = [], +): AggregatorDefs.PublisherView { + const uri = publisherUri(row); + const cid = parseSignatureMetadataCid(row.signature_metadata) ?? ""; + return { + uri, + cid, + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- `did` is consumer-validated at write time + did: row.did as `did:${string}:${string}`, + profile: synthesizePublisherProfile(row), + indexedAt: row.indexed_at ?? row.verified_at, + labels: toLexiconLabels(capLabels(labels, uri)), + }; +} + +/** Reconstruct the `com.emdashcms.experimental.publisher.profile` record + * JSON from the row's columns. Optional fields are omitted (rather than + * emitted as null) so the JSON shape matches what a publisher would have + * written. Same passthrough-shape caveat as `synthesizePackageProfile`: + * typed as `Record` because clients re-validate against + * the published lexicon. */ +function synthesizePublisherProfile(row: PublisherRow): Record { + const profile: Record = { + $type: NSID.publisherProfile, + displayName: row.display_name, + }; + if (row.description !== null) profile["description"] = row.description; + if (row.url !== null) profile["url"] = row.url; + if (row.contact !== null) profile["contact"] = parseJsonArray(row.contact); + if (row.updated_at !== null) profile["updatedAt"] = row.updated_at; + return profile; +} + /** Reconstruct the `com.emdashcms.experimental.package.profile` record * JSON from the row's columns. Field set matches what the consumer's * `ingestPackageProfile` writer accepts; optional fields are omitted @@ -228,6 +384,77 @@ function synthesizePackageRelease(row: ReleaseRow): Record { return release; } +/** Subset of columns from `publisher_verifications` we read for + * `verificationClaimView`. */ +export interface PublisherVerificationRow { + issuer_did: string; + subject_handle: string; + subject_display_name: string; + created_at: string; + expires_at: string | null; + indexed_at: string | null; + verified_at: string; +} + +const PUBLISHER_VERIFICATION_VIEW_COLUMN_NAMES = [ + "issuer_did", + "subject_handle", + "subject_display_name", + "created_at", + "expires_at", + "indexed_at", + "verified_at", +] as const; + +/** SELECT-clause string for `PublisherVerificationRow`, optionally prefixed + * for JOINs. */ +export function publisherVerificationColumns(prefix = ""): string { + return PUBLISHER_VERIFICATION_VIEW_COLUMN_NAMES.map((c) => `${prefix}${c}`).join(", "); +} + +/** + * Map the `publisher_verifications` rows naming `did` as subject to the + * lexicon's `publisherVerificationView`. `labels` are hydrated by the caller + * (the publisher DID subject) and passed in; an empty `rows` is a valid view + * (an unverified publisher), distinct from the caller throwing NotFound for a + * redacted DID. + */ +export function publisherVerificationView( + did: string, + rows: PublisherVerificationRow[], + labels: LabelView[] = [], +): AggregatorDefs.PublisherVerificationView { + return { + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- `did` is the route param, validated as a DID by the lexicon + did: did as `did:${string}:${string}`, + verifications: rows.map(verificationClaimView), + labels: toLexiconVerificationLabels(capLabels(labels, did)), + }; +} + +function verificationClaimView( + row: PublisherVerificationRow, +): AggregatorDefs.VerificationClaimView { + const claim: AggregatorDefs.VerificationClaimView = { + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- issuer_did is consumer-validated at ingest time + issuer: row.issuer_did as `did:${string}:${string}`, + // eslint-disable-next-line typescript/no-unsafe-type-assertion -- subject_handle is consumer-validated at ingest time + handle: row.subject_handle as AggregatorDefs.VerificationClaimView["handle"], + displayName: row.subject_display_name, + createdAt: row.created_at, + indexedAt: row.indexed_at ?? row.verified_at, + }; + if (row.expires_at !== null) claim.expiresAt = row.expires_at; + return claim; +} + +function toLexiconVerificationLabels( + labels: LabelView[], +): AggregatorDefs.PublisherVerificationView["labels"] { + // eslint-disable-next-line typescript/no-unsafe-type-assertion + return labels as AggregatorDefs.PublisherVerificationView["labels"]; +} + function parseJsonArray(json: string): unknown[] { try { const parsed: unknown = JSON.parse(json); diff --git a/apps/aggregator/test/fixtures/main-migrations/0001_init.sql b/apps/aggregator/test/fixtures/main-migrations/0001_init.sql new file mode 100644 index 0000000000..b8ebc193ad --- /dev/null +++ b/apps/aggregator/test/fixtures/main-migrations/0001_init.sql @@ -0,0 +1,355 @@ +-- Frozen snapshot of `migrations/0001_init.sql` as it shipped on `main` +-- (with the `labellers` table spelling). The upgrade test in +-- `migration-upgrade.test.ts` applies this directory (main's shipped migration +-- set) to reproduce a database provisioned before the `labellers` -> +-- `labelers` rename, then applies the live migration set on top. Do not "fix" +-- the spelling here: this file must keep reproducing the historical schema. +-- +-- EmDash plugin registry aggregator: initial schema. +-- +-- Lands every table that the v1 read API + ingest pipeline + label hydration +-- + mirror tracking needs, at once on purpose: features that read these +-- tables don't need to add new ones, so this is the only DDL we expect to +-- ship while NSIDs remain experimental. + +------------------------------------------------------------------------------ +-- Records: package profiles + releases +------------------------------------------------------------------------------ + +CREATE TABLE packages ( + did TEXT NOT NULL, + slug TEXT NOT NULL, + type TEXT NOT NULL, -- 'emdash-plugin' + name TEXT, + description TEXT, + license TEXT NOT NULL, + authors TEXT NOT NULL, -- JSON array + security TEXT NOT NULL, -- JSON array + keywords TEXT, -- JSON array + sections TEXT, -- JSON map + last_updated TEXT, + -- Denormalised from latest release for query convenience. Updated on every + -- new release insert; readers never compute "latest" by sorting. + latest_version TEXT, + capabilities TEXT, -- JSON array + -- Raw signed record bytes for verification + envelope passthrough. Clients + -- re-verify the MST signature against the publisher's DID document at + -- install time. + record_blob BLOB NOT NULL, + signature_metadata TEXT, -- JSON: head CID, signing key id + verified_at TEXT NOT NULL, + PRIMARY KEY (did, slug) +); + +CREATE TABLE releases ( + did TEXT NOT NULL, + package TEXT NOT NULL, -- matches the parent profile's rkey/slug (record.package field) + version TEXT NOT NULL, -- canonical (un-percent-encoded) semver from record.version + rkey TEXT NOT NULL, -- exact rkey of the form `:` + -- Pre-computed semver-precedence-ordered string for ORDER BY. Application + -- code writes this; SQLite cannot compute semver order natively. Format + -- packs zero-padded major.minor.patch with prerelease tags compared per + -- semver precedence rules. + version_sort TEXT NOT NULL, + artifacts TEXT NOT NULL, -- JSON + requires TEXT, -- JSON + suggests TEXT, -- JSON + -- com.emdashcms.experimental.package.releaseExtension contents: + -- { declaredAccess }. The capabilities-shaped projection lives in + -- packages.capabilities for query convenience. + emdash_extension TEXT NOT NULL, + repo_url TEXT, + cts TEXT NOT NULL, -- creation timestamp from the record + record_blob BLOB NOT NULL, + signature_metadata TEXT, + verified_at TEXT NOT NULL, + tombstoned_at TEXT, -- soft delete (publisher deleted record) + PRIMARY KEY (did, package, version), + -- ON DELETE CASCADE because Jetstream events for a publisher can arrive + -- in arbitrary order under network reorder. A publisher who deletes their + -- profile (and all releases) emits the events in author-order, but the + -- profile-delete might land at the consumer before the release-deletes. + -- Without cascade, the consumer would have to either skip the profile + -- delete (leaving stale rows) or sequence retries, neither of which is + -- worth the complexity. Releases are version-immutable from a publishing + -- perspective, but a publisher is still entitled to remove their entire + -- package; cascade mirrors that intent. + FOREIGN KEY (did, package) REFERENCES packages(did, slug) ON DELETE CASCADE +); + +CREATE INDEX idx_releases_latest ON releases(did, package, version_sort DESC) WHERE tombstoned_at IS NULL; +CREATE INDEX idx_releases_cts ON releases(cts); + +-- Audit trail for rejected duplicate-version attempts. FAIR PR #77 makes +-- versions immutable: a second record at the same (did, package, version) is +-- rejected at the SQL layer and logged here for forensics. +-- +-- The UNIQUE constraint dedupes attempts by content (CID), not by raw bytes. +-- CAR bytes include the publisher's commit + MST proof which churns whenever +-- the publisher writes any other record in the same repo, so byte-equality +-- would misclassify benign retries as new attempts and bloat the table. +-- The CID is content-addressed and stable for an unchanged record. +-- +-- The consumer's INSERT uses `ON CONFLICT … DO UPDATE SET rejected_at, +-- attempted_record_blob = excluded.{rejected_at, attempted_record_blob}` so +-- the row tracks the latest attempt timestamp + the latest envelope bytes +-- (newer proofs supersede older ones in the forensics column). +CREATE TABLE release_duplicate_attempts ( + did TEXT NOT NULL, + package TEXT NOT NULL, + version TEXT NOT NULL, + -- CID of the verified record (stable for content; changes only when the + -- record itself changes). Used as the dedup key. + attempted_cid TEXT NOT NULL, + rejected_at TEXT NOT NULL, + reason TEXT NOT NULL, + -- Raw CAR bytes from the most recent attempt. Kept for forensics so + -- operators can inspect what was actually attempted even if the + -- publisher has since deleted the offending record. + attempted_record_blob BLOB NOT NULL, + UNIQUE (did, package, version, attempted_cid) +); + +-- The UNIQUE constraint creates an implicit index on +-- (did, package, version, attempted_record_blob); a separate index on the +-- (did, package, version) prefix is redundant for both lookups (the implicit +-- index handles prefix seeks) and inserts (one fewer index to maintain). + +------------------------------------------------------------------------------ +-- Publishers: identity-level publisher profiles + verification claims +------------------------------------------------------------------------------ + +-- One row per publisher DID (rkey is always literal `self`). Optional: a DID +-- may publish packages without ever publishing a publisher.profile, in which +-- case the row is absent and clients fall back to the handle. This table is +-- the canonical source for "who is publishing these packages?" — distinct from +-- `packages.authors`, which is per-package and remains authoritative for that +-- package. +CREATE TABLE publishers ( + did TEXT PRIMARY KEY, + display_name TEXT NOT NULL, -- bound by verification records — see publisher_verifications + description TEXT, + url TEXT, + contact TEXT, -- JSON array of { kind, url?, email? } + updated_at TEXT, + record_blob BLOB NOT NULL, + signature_metadata TEXT, -- JSON: head CID, signing key id + verified_at TEXT NOT NULL +); + +-- Verification claims: issuer DID vouches for subject DID as a trusted +-- publisher. The rkey is a TID, so an issuer can issue multiple claims (e.g. +-- delegated + official) and we store each as its own row. Validity is bound to +-- the subject's handle + publisher.profile.displayName at issuance time: +-- clients re-resolve those at read time and treat the claim as not in force if +-- either has changed. Ingest stores the facts; the validity check is a +-- query-time concern. +CREATE TABLE publisher_verifications ( + issuer_did TEXT NOT NULL, -- DID of the repo that wrote the record + rkey TEXT NOT NULL, -- TID + subject_did TEXT NOT NULL, + subject_handle TEXT NOT NULL, -- bound at issuance; query-time validity check compares against current + subject_display_name TEXT NOT NULL, -- bound at issuance; query-time validity check compares against current + created_at TEXT NOT NULL, + expires_at TEXT, + record_blob BLOB NOT NULL, + signature_metadata TEXT, + verified_at TEXT NOT NULL, + tombstoned_at TEXT, + PRIMARY KEY (issuer_did, rkey) +); + +-- Hot path: "show me all unexpired, non-tombstoned verifications for subject X". +-- Partial index keeps the index small by excluding tombstoned rows. +CREATE INDEX idx_publisher_verifications_subject ON publisher_verifications(subject_did) + WHERE tombstoned_at IS NULL; + +-- For periodic expiry sweeps. +CREATE INDEX idx_publisher_verifications_expires ON publisher_verifications(expires_at) + WHERE expires_at IS NOT NULL AND tombstoned_at IS NULL; + +------------------------------------------------------------------------------ +-- Mirror tracking (populated when the artifact mirror lands) +------------------------------------------------------------------------------ + +CREATE TABLE mirrored_artifacts ( + did TEXT NOT NULL, + slug TEXT NOT NULL, + version TEXT NOT NULL, + artifact_id TEXT NOT NULL, -- 'package', 'icon', etc. + r2_key TEXT NOT NULL, + bytes INTEGER NOT NULL, + content_type TEXT NOT NULL, + mirrored_at TEXT NOT NULL, + PRIMARY KEY (did, slug, version, artifact_id) +); + +------------------------------------------------------------------------------ +-- Labels (populated when the labeller integration lands) +------------------------------------------------------------------------------ + +-- Append-only label history. Every label received is written here, including +-- negations. Current state is derived from latest cts per (src, uri, val) and +-- projected into label_state below for hot-path lookups. +CREATE TABLE labels ( + src TEXT NOT NULL, -- labeller DID + uri TEXT NOT NULL, -- AT URI of subject + cid TEXT, -- optional version-specific CID + val TEXT NOT NULL, -- e.g. 'security:yanked', '!takedown' + neg INTEGER NOT NULL DEFAULT 0, + cts TEXT NOT NULL, + exp TEXT, -- optional expiry (RFC 3339) + sig BLOB NOT NULL, -- raw signature for client re-verification + ver INTEGER NOT NULL DEFAULT 1, + trusted INTEGER NOT NULL DEFAULT 0, + received_at TEXT NOT NULL, + PRIMARY KEY (src, uri, val, cts) +); + +CREATE INDEX idx_labels_subject ON labels(uri); +CREATE INDEX idx_labels_latest ON labels(src, uri, val, cts DESC); + +-- Latest-state projection: one row per (src, uri, val) holding the most recent +-- cts seen, including the neg flag and exp timestamp. Updated on every label +-- write within the same transaction. Query-time filters apply +-- `neg = 0 AND (exp IS NULL OR exp > now())` to determine whether a label is +-- currently in force. +-- +-- Why retain rows for negated/expired labels rather than deleting them: an +-- out-of-order delivery (a positive label arriving after its negation) could +-- otherwise reinsert a row we'd already retracted. Keeping the row with its +-- `cts` lets the upsert reject the older positive. +CREATE TABLE label_state ( + src TEXT NOT NULL, + uri TEXT NOT NULL, + val TEXT NOT NULL, + cid TEXT, + neg INTEGER NOT NULL DEFAULT 0, + cts TEXT NOT NULL, + exp TEXT, + trusted INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (src, uri, val) +); + +-- Hot path for hard filters (yanked, takedown, etc.) from trusted issuers. +-- Partial index keeps the index small by storing only currently-active rows. +CREATE INDEX idx_label_state_enforce ON label_state(uri, val, trusted) + WHERE neg = 0 AND trusted = 1; + +-- Trusted/known labellers (operator config, edited via deployment). +CREATE TABLE labellers ( + did TEXT PRIMARY KEY, + endpoint TEXT NOT NULL, -- subscribeLabels URL + signing_key TEXT NOT NULL, -- cached #atproto_label key + signing_key_id TEXT NOT NULL, + trusted INTEGER NOT NULL DEFAULT 0, + added_at TEXT NOT NULL, + last_resolved_at TEXT NOT NULL, + notes TEXT +); + +------------------------------------------------------------------------------ +-- Search: FTS5 over packages +------------------------------------------------------------------------------ + +CREATE VIRTUAL TABLE packages_fts USING fts5( + name, + description, + keywords, + authors, + sections, + content='packages', + content_rowid='rowid', + tokenize='porter unicode61 remove_diacritics 2' +); + +CREATE TRIGGER packages_ai AFTER INSERT ON packages BEGIN + INSERT INTO packages_fts(rowid, name, description, keywords, authors, sections) + VALUES (new.rowid, new.name, new.description, new.keywords, new.authors, new.sections); +END; + +CREATE TRIGGER packages_au AFTER UPDATE ON packages BEGIN + INSERT INTO packages_fts(packages_fts, rowid, name, description, keywords, authors, sections) + VALUES ('delete', old.rowid, old.name, old.description, old.keywords, old.authors, old.sections); + INSERT INTO packages_fts(rowid, name, description, keywords, authors, sections) + VALUES (new.rowid, new.name, new.description, new.keywords, new.authors, new.sections); +END; + +CREATE TRIGGER packages_ad AFTER DELETE ON packages BEGIN + INSERT INTO packages_fts(packages_fts, rowid, name, description, keywords, authors, sections) + VALUES ('delete', old.rowid, old.name, old.description, old.keywords, old.authors, old.sections); +END; + +------------------------------------------------------------------------------ +-- Ingest cursor state +------------------------------------------------------------------------------ + +-- Cursor state for ingest sources (Jetstream microsecond timestamp, +-- subscribeLabels seq cursors per labeller, etc.). +CREATE TABLE ingest_state ( + source TEXT PRIMARY KEY, -- 'jetstream', 'labeller:did:web:labels.example.com', etc. + cursor TEXT NOT NULL, + updated_at TEXT NOT NULL +); + +-- Known publisher DIDs we've seen via Jetstream or Constellation. Reconciliation +-- iterates this table; cold-start backfill seeds it from Constellation. +-- +-- Doubles as the DID-document resolution cache: `pds`, `signing_key`, +-- `signing_key_id` are populated by the records consumer on first verification +-- and refreshed when `pds_resolved_at` is older than the consumer's TTL +-- (currently 24h, applied at query time as +-- `pds_resolved_at > datetime('now', '-1 day')`). Backfill may insert a row +-- with these fields null; the consumer's first event for that DID forces a +-- resolution and UPDATE. +CREATE TABLE known_publishers ( + did TEXT PRIMARY KEY, + pds TEXT, -- cached PDS endpoint from DID document + signing_key TEXT, -- cached #atproto signing key (multibase) + signing_key_id TEXT, -- e.g. 'did:plc:xxx#atproto' + pds_resolved_at TEXT, -- last successful DID-doc resolution + first_seen_at TEXT NOT NULL, + last_seen_at TEXT NOT NULL +); + +------------------------------------------------------------------------------ +-- Verification-failure forensics +------------------------------------------------------------------------------ + +-- Records that failed PDS-verified ingest (signature, MST proof, AT-URI, +-- lexicon, content-mismatch). Written instead of retrying, because these +-- failures indicate malicious or broken upstream — retrying would just burn +-- PDS round trips. Operators query this table to investigate suspected attacks +-- or upstream regressions; it is NOT used as a retry queue. +-- +-- Distinct from the configured Cloudflare DLQ (`emdash-aggregator-records-dlq`, +-- see wrangler.jsonc), which receives messages after `max_retries` exhausted — +-- that is for transient failures (PDS down, profile-not-yet-arrived). Two +-- distinct failure modes, two distinct destinations. +-- +-- `payload` holds the unverified record bytes from the Jetstream event so an +-- operator can inspect what was attempted without going back to the source PDS. +CREATE TABLE dead_letters ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + did TEXT NOT NULL, + collection TEXT NOT NULL, + rkey TEXT NOT NULL, + -- Reason code; matches the `DeadLetterReason` union in records-consumer.ts. + -- Current values: 'RECORD_NOT_FOUND', 'RESPONSE_TOO_LARGE', 'INVALID_PROOF', + -- 'PDS_HTTP_ERROR', 'LEXICON_VALIDATION_FAILED', 'RKEY_MISMATCH', + -- 'CONTACT_VALIDATION_FAILED', 'INVALID_VERSION', 'UNKNOWN_COLLECTION', + -- 'UNEXPECTED_ERROR'. + reason TEXT NOT NULL, + -- Free-form context (which field, expected vs got, library error message, etc.). + detail TEXT, + -- UTF-8 encoded JSON bytes of `RecordsJob.jetstreamRecord` when present, or a + -- fallback envelope `{operation, cid}` for delete events that don't carry one. + -- Stored as BLOB so future formats (CBOR, raw record bytes) can land here + -- without a schema change; today operators must `CAST(payload AS TEXT)` to + -- read. + payload BLOB NOT NULL, + received_at TEXT NOT NULL DEFAULT (datetime('now')) +); + +CREATE INDEX idx_dead_letters_did ON dead_letters(did); +CREATE INDEX idx_dead_letters_received ON dead_letters(received_at); diff --git a/apps/aggregator/test/fixtures/main-migrations/0002_indexed_at.sql b/apps/aggregator/test/fixtures/main-migrations/0002_indexed_at.sql new file mode 100644 index 0000000000..385d7e35d9 --- /dev/null +++ b/apps/aggregator/test/fixtures/main-migrations/0002_indexed_at.sql @@ -0,0 +1,35 @@ +-- Add `indexed_at` to the four content tables that the read API exposes via +-- the `packageView` / `releaseView` lexicon shapes. +-- +-- Why a separate column from `verified_at`: `verified_at` tracks "when the +-- aggregator last verified this record", and is bumped on every upsert +-- (including no-op re-verifications). The read API needs "when the aggregator +-- first observed this record" so the lexicon's `indexedAt` field is stable +-- across re-ingest. They diverge whenever a record is re-fetched (e.g. +-- backfill catching up on a record we already had via Jetstream). +-- +-- Defaulting to `verified_at` for any rows that pre-date this migration: the +-- closest approximation we can make for already-indexed content. Going +-- forward, the consumer's INSERTs set `indexed_at` to `now()` on first write +-- and COALESCE the existing value on conflict (see records-consumer.ts). +-- +-- NOT NULL after backfill from `verified_at`. SQLite's `ALTER TABLE` doesn't +-- support DEFAULT-with-NOT-NULL in one step, so we add the column nullable, +-- backfill, then move the constraint via the new-table dance — except SQLite +-- also can't add NOT NULL via ALTER, so we keep the column nullable at the +-- schema level and have the writer (consumer) treat it as required. The +-- read-API code defends against NULL by falling back to verified_at if +-- indexed_at is somehow missing on a row, which won't happen for new writes +-- but covers the historical-data corner case without a table rebuild. + +ALTER TABLE packages ADD COLUMN indexed_at TEXT; +UPDATE packages SET indexed_at = verified_at WHERE indexed_at IS NULL; + +ALTER TABLE releases ADD COLUMN indexed_at TEXT; +UPDATE releases SET indexed_at = verified_at WHERE indexed_at IS NULL; + +ALTER TABLE publishers ADD COLUMN indexed_at TEXT; +UPDATE publishers SET indexed_at = verified_at WHERE indexed_at IS NULL; + +ALTER TABLE publisher_verifications ADD COLUMN indexed_at TEXT; +UPDATE publisher_verifications SET indexed_at = verified_at WHERE indexed_at IS NULL; diff --git a/apps/aggregator/test/label-enforcement.test.ts b/apps/aggregator/test/label-enforcement.test.ts new file mode 100644 index 0000000000..1d92a748f4 --- /dev/null +++ b/apps/aggregator/test/label-enforcement.test.ts @@ -0,0 +1,43 @@ +/** + * Unit tests for the enforcement SQL builders' input guards. The builders are + * exported and interpolate the caller-supplied row alias into raw SQL, so a + * malformed alias must be rejected rather than reach the query text. + */ + +import { type AcceptedLabelerPolicy } from "@emdash-cms/registry-moderation"; +import { describe, expect, it } from "vitest"; + +import { + buildPackageEnforcementSql, + buildReleaseEnforcementSql, +} from "../src/routes/xrpc/label-enforcement.js"; + +const accepted: AcceptedLabelerPolicy[] = [{ did: "did:web:labels.example", redact: false }]; + +describe("buildReleaseEnforcementSql alias validation", () => { + it("rejects an alias that is not an identifier followed by a dot", () => { + expect(() => + buildReleaseEnforcementSql(accepted, 0, { release: "r; DROP TABLE releases;--" }), + ).toThrow(TypeError); + expect(() => buildReleaseEnforcementSql(accepted, 0, { package: "p" })).toThrow(TypeError); + }); + + it("accepts the default aliases", () => { + const { sql } = buildReleaseEnforcementSql(accepted, 0); + expect(sql).toContain("NOT EXISTS"); + }); +}); + +describe("buildPackageEnforcementSql alias validation", () => { + it("rejects an alias that is not an identifier followed by a dot", () => { + expect(() => buildPackageEnforcementSql(accepted, 0, "p; DROP TABLE packages;--")).toThrow( + TypeError, + ); + expect(() => buildPackageEnforcementSql(accepted, 0, "p")).toThrow(TypeError); + }); + + it("accepts the default alias", () => { + const { sql } = buildPackageEnforcementSql(accepted, 0); + expect(sql).toContain("NOT EXISTS"); + }); +}); diff --git a/apps/aggregator/test/label-ingestor.test.ts b/apps/aggregator/test/label-ingestor.test.ts new file mode 100644 index 0000000000..3ae3464a1f --- /dev/null +++ b/apps/aggregator/test/label-ingestor.test.ts @@ -0,0 +1,741 @@ +/** + * LabelIngestor unit tests. + * + * Drives the ingestor against a fake `LabelStreamClient`, an in-memory queue, + * a `Map`-backed cursor store, and a fake resolver — no DO/D1/Queue runtime + * needed, mirroring `jetstream-ingestor.test.ts`'s injection style. + * + * The happy-path and verification-retry tests sign labels with a real P-256 + * keypair via `createLabelSigner` (`@emdash-cms/registry-moderation`), so + * `verifyLabelWithPublicKey` runs for real rather than through a stub — + * exactly the pipeline the labeler and the ingestor both use in production. + */ + +import { P256PrivateKeyExportable } from "@atcute/crypto"; +import { toBase64Url } from "@atcute/multibase"; +import { + createLabelSigner, + type LabelSigner, + type UnsignedLabel, +} from "@emdash-cms/registry-moderation"; +import { describe, expect, it } from "vitest"; + +import { toWire, type LabelIngestJob } from "../src/label-ingest-types.js"; +import { + LabelIngestor, + type LabelCursorStore, + type LabelIngestorLogger, + type LabelJobQueue, + type LabelResolver, +} from "../src/label-ingestor.js"; +import type { + LabelStreamClient, + LabelStreamEvent, + LabelStreamSubscribeOptions, +} from "../src/label-stream-client.js"; +import type { ResolvedLabelerIdentity } from "../src/labeler-resolver.js"; + +const LABELER_DID = "did:web:labeler.example"; +const ENDPOINT = "https://labeler.example"; + +interface TestKey { + signer: LabelSigner; + identity: ResolvedLabelerIdentity; +} + +/** Generates a real P-256 keypair, wraps it in a `createLabelSigner` whose + * `resolveDid` is a closure returning a self-consistent DID document (no + * network, no fixtures) and an `ResolvedLabelerIdentity` pointing at the same + * key — everything the ingestor and `verifyLabelWithPublicKey` need. */ +async function makeKey(issuerDid: string = LABELER_DID): Promise { + const keypair = await P256PrivateKeyExportable.createKeypair(); + const privateKey = toBase64Url(await keypair.exportPrivateKey("raw")); + const multikey = await keypair.exportPublicKey("multikey"); + const document = { + id: issuerDid, + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: issuerDid, + publicKeyMultibase: multikey, + }, + ], + }; + const signer = await createLabelSigner({ + issuerDid, + privateKey, + resolveDid: async () => document, + }); + return { + signer, + identity: { + endpoint: ENDPOINT, + publicKey: keypair, + signingKeyId: `${issuerDid}#atproto_label`, + }, + }; +} + +function labelInput(uri: string): Omit { + return { ver: 1, uri, val: "test-value", cts: "2026-07-10T12:00:00.000Z" }; +} + +class FakeHandle { + closed = false; + private queue: LabelStreamEvent[] = []; + private pending: ((result: IteratorResult) => void) | null = null; + private ended = false; + + push(event: LabelStreamEvent): void { + if (this.pending) { + const resolve = this.pending; + this.pending = null; + resolve({ value: event, done: false }); + return; + } + this.queue.push(event); + } + + endNormally(): void { + if (this.ended) return; + this.ended = true; + if (this.pending) { + const resolve = this.pending; + this.pending = null; + resolve({ value: undefined, done: true }); + } + } + + close(): void { + this.closed = true; + this.endNormally(); + } + + [Symbol.asyncIterator](): AsyncIterator { + return { + next: (): Promise> => { + const item = this.queue.shift(); + if (item) return Promise.resolve({ value: item, done: false }); + if (this.ended) return Promise.resolve({ value: undefined, done: true }); + return new Promise((resolve) => { + this.pending = resolve; + }); + }, + }; + } +} + +/** Fake transport. `emit`/`disconnect` act on the most recently subscribed + * handle, mirroring `MockJetstream`'s driveable style. */ +class FakeLabelStreamClient implements LabelStreamClient { + readonly subscribeCalls: LabelStreamSubscribeOptions[] = []; + private readonly handles: FakeHandle[] = []; + + subscribe(opts: LabelStreamSubscribeOptions): FakeHandle { + this.subscribeCalls.push(opts); + const handle = new FakeHandle(); + this.handles.push(handle); + return handle; + } + + private current(): FakeHandle { + const handle = this.handles.at(-1); + if (!handle) throw new Error("subscribe() has not been called yet"); + return handle; + } + + emit(event: LabelStreamEvent): void { + this.current().push(event); + } + + /** Simulates a clean server-side disconnect uninvolved with verification + * — the ingestor should reconnect with backoff. */ + disconnect(): void { + this.current().endNormally(); + } + + get closedHandleCount(): number { + return this.handles.filter((h) => h.closed).length; + } +} + +class FakeLabelResolver implements LabelResolver { + resolveCalls = 0; + resolveFreshCalls = 0; + current: ResolvedLabelerIdentity; + /** What `resolveFresh` returns. Defaults to `current`; tests override to + * simulate a key rotation (or a still-wrong key, for the persistent-failure + * path). */ + fresh: ResolvedLabelerIdentity; + + constructor(identity: ResolvedLabelerIdentity) { + this.current = identity; + this.fresh = identity; + } + + resolve(): Promise { + this.resolveCalls += 1; + return Promise.resolve(this.current); + } + + resolveFresh(): Promise { + this.resolveFreshCalls += 1; + return Promise.resolve(this.fresh); + } +} + +class InMemoryQueue implements LabelJobQueue { + readonly jobs: LabelIngestJob[] = []; + send(job: LabelIngestJob): Promise { + this.jobs.push(job); + return Promise.resolve(); + } +} + +class MapCursorStore implements LabelCursorStore { + private cursor: number | undefined; + readonly puts: number[] = []; + constructor(initial?: number) { + this.cursor = initial; + } + get(): Promise { + return Promise.resolve(this.cursor); + } + put(cursor: number): Promise { + this.cursor = cursor; + this.puts.push(cursor); + return Promise.resolve(); + } +} + +/** Records send-resolution and cursor-put order into one shared log so + * ordering assertions don't depend on wall-clock timing. */ +class DeferredQueue implements LabelJobQueue { + readonly jobs: LabelIngestJob[] = []; + private readonly resolvers: Array<() => void> = []; + constructor(private readonly log: string[]) {} + + send(job: LabelIngestJob): Promise { + this.jobs.push(job); + return new Promise((resolve) => { + this.resolvers.push(() => { + this.log.push(`send-resolved:${job.frameIndex}`); + resolve(); + }); + }); + } + + resolveNext(): void { + const next = this.resolvers.shift(); + if (!next) throw new Error("no pending send to resolve"); + next(); + } +} + +class LoggingCursorStore implements LabelCursorStore { + private cursor: number | undefined; + constructor(private readonly log: string[]) {} + get(): Promise { + return Promise.resolve(this.cursor); + } + put(cursor: number): Promise { + this.log.push(`cursor-put:${cursor}`); + this.cursor = cursor; + return Promise.resolve(); + } +} + +const TIGHT_BACKOFF = { initialDelayMs: 1, maxDelayMs: 5, multiplier: 2, jitter: 0 }; + +/** Wait until the predicate returns true or the test times out. Polls the + * microtask + macrotask queue rather than wall-clock. */ +async function waitFor(predicate: () => boolean, label: string, attempts = 200): Promise { + for (let i = 0; i < attempts; i++) { + if (predicate()) return; + await Promise.resolve(); + await new Promise((r) => setTimeout(r, 0)); + } + throw new Error(`waitFor timed out: ${label}`); +} + +describe("LabelIngestor", () => { + it("verifies a real signed label and enqueues it, advancing the cursor", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const cursorStore = new MapCursorStore(); + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore, + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const signed = await key.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 1, labels: [signed] }); + + await waitFor(() => queue.jobs.length === 1, "label enqueued"); + expect(queue.jobs[0]).toEqual({ + src: LABELER_DID, + sourceSequence: 1, + frameIndex: 0, + label: toWire(signed), + }); + expect(ingestor.currentCursor).toBe(1); + await waitFor(() => cursorStore.puts.includes(1), "cursor persisted"); + + ingestor.stop(); + await runPromise; + }); + + it("enqueues every label in a multi-label frame with the correct frameIndex", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const a = await key.signer.sign(labelInput("at://did:example:pub/a/1")); + const b = await key.signer.sign(labelInput("at://did:example:pub/b/1")); + client.emit({ seq: 7, labels: [a, b] }); + + await waitFor(() => queue.jobs.length === 2, "both labels enqueued"); + expect(queue.jobs[0]?.frameIndex).toBe(0); + expect(queue.jobs[1]?.frameIndex).toBe(1); + expect(queue.jobs[0]?.sourceSequence).toBe(7); + expect(queue.jobs[1]?.sourceSequence).toBe(7); + + ingestor.stop(); + await runPromise; + }); + + it("subscribes with cursor 0 when no cursor is persisted", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue: new InMemoryQueue(), + cursorStore: new MapCursorStore(), + resolver: new FakeLabelResolver(key.identity), + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + await waitFor(() => client.subscribeCalls.length === 1, "subscribed"); + expect(client.subscribeCalls[0]).toEqual({ endpoint: ENDPOINT, cursor: 0 }); + + ingestor.stop(); + await runPromise; + }); + + it("subscribes with the persisted cursor when one exists", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue: new InMemoryQueue(), + cursorStore: new MapCursorStore(42), + resolver: new FakeLabelResolver(key.identity), + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + await waitFor(() => client.subscribeCalls.length === 1, "subscribed"); + expect(client.subscribeCalls[0]).toEqual({ endpoint: ENDPOINT, cursor: 42 }); + + ingestor.stop(); + await runPromise; + }); + + it("retries a failed initial cursor load instead of dying at cold start", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const store = new MapCursorStore(42); + let getCalls = 0; + const flakyStore: LabelCursorStore = { + get(): Promise { + getCalls += 1; + if (getCalls === 1) return Promise.reject(new Error("transient D1 failure")); + return store.get(); + }, + put: (cursor) => store.put(cursor), + }; + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue: new InMemoryQueue(), + cursorStore: flakyStore, + resolver: new FakeLabelResolver(key.identity), + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + await waitFor(() => client.subscribeCalls.length === 1, "subscribed after cursor retry"); + expect(getCalls).toBe(2); + expect(client.subscribeCalls[0]).toEqual({ endpoint: ENDPOINT, cursor: 42 }); + + ingestor.stop(); + await runPromise; + }); + + it("retries once via resolveFresh after a verification failure, then enqueues", async () => { + const staleKey = await makeKey(); + const rotatedKey = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const cursorStore = new MapCursorStore(); + const resolver = new FakeLabelResolver(staleKey.identity); + resolver.fresh = rotatedKey.identity; + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore, + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + // Signed with the rotated key, but the resolver's cache still points + // at the stale one — first verification attempt must fail. + const signed = await rotatedKey.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 5, labels: [signed] }); + + await waitFor(() => queue.jobs.length === 1, "label enqueued after retry"); + expect(resolver.resolveFreshCalls).toBe(1); + expect(ingestor.currentCursor).toBe(5); + + ingestor.stop(); + await runPromise; + }); + + it("closes the connection on a second verification failure, enqueueing nothing and leaving the cursor unchanged", async () => { + const configuredKey = await makeKey(); + const unknownKey = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const cursorStore = new MapCursorStore(); + const resolver = new FakeLabelResolver(configuredKey.identity); + // The refresh doesn't help — still the wrong key. + resolver.fresh = configuredKey.identity; + const logger: LabelIngestorLogger = { error: () => {}, warn: () => {} }; + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore, + resolver, + logger, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const signed = await unknownKey.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 9, labels: [signed] }); + + await waitFor(() => ingestor.consecutiveFailures >= 1, "failure counted"); + expect(queue.jobs).toHaveLength(0); + expect(ingestor.currentCursor).toBe(0); + expect(resolver.resolveFreshCalls).toBe(1); + expect(client.closedHandleCount).toBeGreaterThanOrEqual(1); + + ingestor.stop(); + await runPromise; + }); + + it("treats a src mismatch as a verification failure", async () => { + const configuredKey = await makeKey(LABELER_DID); + const otherIssuer = await makeKey("did:web:other.example"); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const resolver = new FakeLabelResolver(configuredKey.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + // Validly signed by its own issuer, but that issuer isn't the DID this + // ingestor is configured for. + const signed = await otherIssuer.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 3, labels: [signed] }); + + await waitFor(() => ingestor.consecutiveFailures >= 1, "failure counted"); + expect(queue.jobs).toHaveLength(0); + expect(ingestor.currentCursor).toBe(0); + + ingestor.stop(); + await runPromise; + }); + + it("fails closed on a malformed label (parseSignedLabel throws)", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + await waitFor(() => client.subscribeCalls.length === 1, "subscribed"); + client.emit({ seq: 4, labels: [{ not: "a valid label" }] }); + + await waitFor(() => ingestor.consecutiveFailures >= 1, "failure counted"); + expect(queue.jobs).toHaveLength(0); + expect(ingestor.currentCursor).toBe(0); + + ingestor.stop(); + await runPromise; + }); + + it("fails closed on a tampered signature", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const signed = await key.signer.sign(labelInput("at://did:example:pub/x/1")); + const tampered = { ...signed, sig: signed.sig.map((b, i) => (i === 0 ? b ^ 0xff : b)) }; + client.emit({ seq: 6, labels: [tampered] }); + + await waitFor(() => ingestor.consecutiveFailures >= 1, "failure counted"); + expect(queue.jobs).toHaveLength(0); + + ingestor.stop(); + await runPromise; + }); + + it("counts a failure when a connection ends on a verification failure after earlier progress", async () => { + const key = await makeKey(); + const wrongKey = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver: new FakeLabelResolver(key.identity), + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const good = await key.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 1, labels: [good] }); + await waitFor(() => queue.jobs.length === 1, "good frame enqueued"); + + // Same connection: a frame that cannot verify. The good frame must not + // count as progress that resets the backoff counter. + const bad = await wrongKey.signer.sign(labelInput("at://did:example:pub/x/2")); + client.emit({ seq: 2, labels: [bad] }); + await waitFor( + () => ingestor.consecutiveFailures >= 1, + "failure counted despite earlier progress", + ); + expect(queue.jobs).toHaveLength(1); + expect(ingestor.currentCursor).toBe(1); + + ingestor.stop(); + await runPromise; + }); + + it("persists the cursor only after every send in the frame resolves", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const log: string[] = []; + const queue = new DeferredQueue(log); + const cursorStore = new LoggingCursorStore(log); + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore, + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const a = await key.signer.sign(labelInput("at://did:example:pub/a/1")); + const b = await key.signer.sign(labelInput("at://did:example:pub/b/1")); + client.emit({ seq: 3, labels: [a, b] }); + + await waitFor(() => queue.jobs.length === 1, "first send started"); + expect(log).toEqual([]); + + queue.resolveNext(); + await waitFor(() => queue.jobs.length === 2, "second send started"); + expect(log).toEqual(["send-resolved:0"]); + + queue.resolveNext(); + await waitFor(() => log.includes("cursor-put:3"), "cursor persisted"); + expect(log).toEqual(["send-resolved:0", "send-resolved:1", "cursor-put:3"]); + + ingestor.stop(); + await runPromise; + }); + + it("does not persist the cursor when a queue.send call rejects", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const cursorStore = new MapCursorStore(); + const queue: LabelJobQueue = { + send: () => Promise.reject(new Error("queue unavailable")), + }; + const resolver = new FakeLabelResolver(key.identity); + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore, + resolver, + backoff: TIGHT_BACKOFF, + sleep: () => Promise.resolve(), + }); + const runPromise = ingestor.run(); + + const signed = await key.signer.sign(labelInput("at://did:example:pub/x/1")); + client.emit({ seq: 1, labels: [signed] }); + + await waitFor(() => ingestor.consecutiveFailures >= 1, "failure counted after send rejection"); + expect(cursorStore.puts).toHaveLength(0); + expect(ingestor.currentCursor).toBe(0); + + ingestor.stop(); + await runPromise; + }); + + it("resets backoff after a frame is fully processed, even across reconnects", async () => { + const key = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + const resolver = new FakeLabelResolver(key.identity); + const sleeps: number[] = []; + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: { initialDelayMs: 10, maxDelayMs: 1000, multiplier: 10, jitter: 0 }, + sleep: (ms) => { + sleeps.push(ms); + return Promise.resolve(); + }, + }); + const runPromise = ingestor.run(); + + const a = await key.signer.sign(labelInput("at://did:example:pub/a/1")); + client.emit({ seq: 1, labels: [a] }); + await waitFor(() => queue.jobs.length === 1, "first job"); + client.disconnect(); + await waitFor(() => sleeps.length >= 1, "first backoff"); + await waitFor(() => client.subscribeCalls.length === 2, "reconnected"); + + const b = await key.signer.sign(labelInput("at://did:example:pub/b/1")); + client.emit({ seq: 2, labels: [b] }); + await waitFor(() => queue.jobs.length === 2, "second job", 500); + client.disconnect(); + await waitFor(() => sleeps.length >= 2, "second backoff", 500); + + // Both backoffs are the initial delay (10ms), not 100ms (10×10) — + // progress in between resets the counter each time. + expect(sleeps[0]).toBe(10); + expect(sleeps[1]).toBe(10); + + ingestor.stop(); + await runPromise; + }); + + it("escalates backoff across repeated verification failures and resets once one succeeds", async () => { + const configuredKey = await makeKey(); + const unknownKey = await makeKey(); + const client = new FakeLabelStreamClient(); + const queue = new InMemoryQueue(); + // The refresh never helps — every attempt is signed by a key the + // resolver never returns, so every connection attempt fails. + const resolver = new FakeLabelResolver(configuredKey.identity); + const sleeps: number[] = []; + const ingestor = new LabelIngestor({ + did: LABELER_DID, + client, + queue, + cursorStore: new MapCursorStore(), + resolver, + backoff: { initialDelayMs: 10, maxDelayMs: 80, multiplier: 2, jitter: 0 }, + sleep: (ms) => { + sleeps.push(ms); + return Promise.resolve(); + }, + }); + const runPromise = ingestor.run(); + + for (let i = 0; i < 2; i++) { + await waitFor(() => client.subscribeCalls.length === i + 1, `attempt ${i} connected`); + const signed = await unknownKey.signer.sign(labelInput(`at://did:example:pub/x/${i}`)); + client.emit({ seq: i + 1, labels: [signed] }); + await waitFor(() => sleeps.length === i + 1, `backoff ${i}`); + } + expect(sleeps[0]).toBe(10); + expect(sleeps[1]).toBe(20); + + // Now let the same connection succeed with the correctly configured key. + await waitFor(() => client.subscribeCalls.length === 3, "third attempt connected"); + const good = await configuredKey.signer.sign(labelInput("at://did:example:pub/good/1")); + client.emit({ seq: 100, labels: [good] }); + await waitFor(() => queue.jobs.length === 1, "succeeded after escalation"); + // consecutiveFailures resets only once connectAndConsume() returns, which + // requires the subscription to end — disconnect to observe the reset. + client.disconnect(); + await waitFor(() => ingestor.consecutiveFailures === 0, "backoff reset after progress"); + + ingestor.stop(); + await runPromise; + }); +}); diff --git a/apps/aggregator/test/label-stream-client.test.ts b/apps/aggregator/test/label-stream-client.test.ts new file mode 100644 index 0000000000..66f73d775e --- /dev/null +++ b/apps/aggregator/test/label-stream-client.test.ts @@ -0,0 +1,145 @@ +/** + * `decodeLabelStreamFrame` unit tests. + * + * Frames are built the same way `apps/labeler/src/subscribe-labels.ts`'s + * `encodeEvent` builds them: two CBOR-encoded values (header, payload) + * concatenated with no length prefix. Testing the decode function directly + * (rather than driving a real WebSocket) mirrors `jetstream-client.ts`'s + * `wrapAtcuteSubscription` test seam — the transport plumbing around it is + * exercised by the ingestor/DO integration paths, not here. + */ + +import { encode, toBytes } from "@atcute/cbor"; +import { describe, expect, it } from "vitest"; + +import { decodeLabelStreamFrame, LabelStreamError } from "../src/label-stream-client.js"; + +function frame(header: Record, payload: Record): Uint8Array { + const encodedHeader = encode(header); + const encodedPayload = encode(payload); + const bytes = new Uint8Array(encodedHeader.length + encodedPayload.length); + bytes.set(encodedHeader); + bytes.set(encodedPayload, encodedHeader.length); + return bytes; +} + +function sampleLabel(uri: string): Record { + return { + ver: 1, + src: "did:web:labeler.example", + uri, + val: "test-value", + cts: "2026-07-10T12:00:00.000Z", + sig: toBytes(new Uint8Array(64).fill(7)), + }; +} + +describe("decodeLabelStreamFrame", () => { + it("decodes a valid #labels frame", () => { + const bytes = frame( + { op: 1, t: "#labels" }, + { seq: 1, labels: [sampleLabel("at://did:example:pub/x/1")] }, + ); + const event = decodeLabelStreamFrame(bytes); + expect(event).not.toBeNull(); + expect(event?.seq).toBe(1); + expect(event?.labels).toHaveLength(1); + }); + + it("decodes a multi-label frame", () => { + const bytes = frame( + { op: 1, t: "#labels" }, + { + seq: 42, + labels: [sampleLabel("at://did:example:pub/a/1"), sampleLabel("at://did:example:pub/b/1")], + }, + ); + const event = decodeLabelStreamFrame(bytes); + expect(event?.seq).toBe(42); + expect(event?.labels).toHaveLength(2); + }); + + it("ignores an op:1 frame with an unrecognised t (forward compatibility)", () => { + const bytes = frame({ op: 1, t: "#futureEventKind" }, { anything: true }); + expect(decodeLabelStreamFrame(bytes)).toBeNull(); + }); + + it("throws LabelStreamError carrying error and message for an op:-1 frame", () => { + const bytes = frame( + { op: -1 }, + { error: "FutureCursor", message: "cursor is ahead of the stream" }, + ); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(LabelStreamError); + try { + decodeLabelStreamFrame(bytes); + expect.unreachable(); + } catch (err) { + expect(err).toBeInstanceOf(LabelStreamError); + const streamErr = err as LabelStreamError; + expect(streamErr.error).toBe("FutureCursor"); + expect(streamErr.message).toBe("cursor is ahead of the stream"); + } + }); + + it("throws on an op:-1 frame missing error/message fields", () => { + const bytes = frame({ op: -1 }, { unrelated: true }); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("throws on an unsupported op", () => { + const bytes = frame({ op: 2 }, { seq: 1, labels: [sampleLabel("at://did:example:pub/x/1")] }); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("throws on malformed CBOR", () => { + const garbage = new Uint8Array([0xff, 0xff, 0xff, 0xff, 0xff]); + expect(() => decodeLabelStreamFrame(garbage)).toThrow(TypeError); + }); + + it("throws when the header isn't a CBOR object", () => { + const encodedHeader = encode("not-an-object"); + const encodedPayload = encode({ seq: 1, labels: [sampleLabel("at://did:example:pub/x/1")] }); + const bytes = new Uint8Array(encodedHeader.length + encodedPayload.length); + bytes.set(encodedHeader); + bytes.set(encodedPayload, encodedHeader.length); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("rejects a non-positive seq", () => { + const bytes = frame( + { op: 1, t: "#labels" }, + { seq: 0, labels: [sampleLabel("at://did:example:pub/x/1")] }, + ); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("rejects a non-integer seq", () => { + const bytes = frame( + { op: 1, t: "#labels" }, + { seq: 1.5, labels: [sampleLabel("at://did:example:pub/x/1")] }, + ); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("rejects an empty labels array", () => { + const bytes = frame({ op: 1, t: "#labels" }, { seq: 1, labels: [] }); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("rejects a labels array over the 200-entry cap", () => { + const labels = Array.from({ length: 201 }, (_, i) => + sampleLabel(`at://did:example:pub/x/${i}`), + ); + const bytes = frame({ op: 1, t: "#labels" }, { seq: 1, labels }); + expect(() => decodeLabelStreamFrame(bytes)).toThrow(TypeError); + }); + + it("accepts exactly 200 labels", () => { + const labels = Array.from({ length: 200 }, (_, i) => + sampleLabel(`at://did:example:pub/x/${i}`), + ); + const bytes = frame({ op: 1, t: "#labels" }, { seq: 1, labels }); + const event = decodeLabelStreamFrame(bytes); + expect(event?.labels).toHaveLength(200); + }); +}); diff --git a/apps/aggregator/test/labeler-resolver.test.ts b/apps/aggregator/test/labeler-resolver.test.ts new file mode 100644 index 0000000000..eb2e9c5051 --- /dev/null +++ b/apps/aggregator/test/labeler-resolver.test.ts @@ -0,0 +1,609 @@ +import { P256PrivateKeyExportable, Secp256k1PrivateKeyExportable } from "@atcute/crypto"; +import { fromBase64Url, toBase58Btc } from "@atcute/multibase"; +import { applyD1Migrations, env } from "cloudflare:test"; +import { beforeAll, beforeEach, describe, expect, it } from "vitest"; + +import { + type CachedLabelerIdentity, + createD1LabelerIdentityCache, + type LabelerDidResolverLike, + type LabelerIdentityCache, + LabelerResolver, +} from "../src/labeler-resolver.js"; + +interface TestEnv { + DB: D1Database; + TEST_MIGRATIONS: Parameters[1]; +} + +const testEnv = env as unknown as TestEnv; +const DID = "did:web:labeler.example"; +const ENDPOINT = "https://labeler.example"; +const NOW = new Date("2026-07-11T12:00:00.000Z"); + +let signingKey: string; +let otherSigningKey: string; +let wrongCurveKey: string; +let nonCanonicalKey: string; + +beforeAll(async () => { + const keypair = await P256PrivateKeyExportable.createKeypair(); + const otherKeypair = await P256PrivateKeyExportable.createKeypair(); + const secp256k1 = await Secp256k1PrivateKeyExportable.createKeypair(); + signingKey = await keypair.exportPublicKey("multikey"); + otherSigningKey = await otherKeypair.exportPublicKey("multikey"); + wrongCurveKey = await secp256k1.exportPublicKey("multikey"); + + const jwk = await keypair.exportPublicKey("jwk"); + const uncompressed = new Uint8Array([ + 0x80, + 0x24, + 0x04, + ...fromBase64Url(jwk.x!), + ...fromBase64Url(jwk.y!), + ]); + nonCanonicalKey = `z${toBase58Btc(uncompressed)}`; +}); + +function document(overrides: Record = {}): unknown { + return { + "@context": ["https://www.w3.org/ns/did/v1", "https://w3id.org/security/multikey/v1"], + id: DID, + verificationMethod: [ + { + id: `${DID}#atproto_label`, + type: "Multikey", + controller: DID, + publicKeyMultibase: signingKey, + }, + ], + service: [ + { + id: `${DID}#atproto_labeler`, + type: "AtprotoLabeler", + serviceEndpoint: ENDPOINT, + }, + ], + ...overrides, + }; +} + +class MapCache implements LabelerIdentityCache { + readonly entries = new Map(); + readonly refreshes: Array<{ + did: string; + identity: Omit; + now: Date; + }> = []; + readonly expires: string[] = []; + + read(did: string): Promise { + return Promise.resolve(this.entries.get(did) ?? null); + } + + refresh( + did: string, + identity: Omit, + now: Date, + ): Promise { + if (!this.entries.has(did)) throw new Error("labeler is not configured"); + this.refreshes.push({ did, identity, now }); + this.entries.set(did, { ...identity, resolvedAt: now }); + return Promise.resolve(); + } + + expire(did: string): Promise { + this.expires.push(did); + const cached = this.entries.get(did); + if (cached) this.entries.set(did, { ...cached, resolvedAt: new Date(0) }); + return Promise.resolve(); + } +} + +class StubResolver implements LabelerDidResolverLike { + readonly calls: string[] = []; + response: unknown = document(); + error: Error | null = null; + + resolve(did: string): Promise { + this.calls.push(did); + if (this.error) return Promise.reject(this.error); + return Promise.resolve(this.response); + } +} + +function configured( + cache: MapCache, + overrides: Partial = {}, +): CachedLabelerIdentity { + const identity = { + endpoint: "https://cached.example", + signingKey, + signingKeyId: `${DID}#atproto_label`, + resolvedAt: NOW, + ...overrides, + }; + cache.entries.set(DID, identity); + return identity; +} + +describe("LabelerResolver", () => { + it("returns a fresh configured cache entry without fetching", async () => { + const cache = new MapCache(); + const cached = configured(cache); + const didResolver = new StubResolver(); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + const result = await subject.resolve(DID); + + expect(result.endpoint).toBe(cached.endpoint); + expect(result.signingKeyId).toBe(`${DID}#atproto_label`); + expect(typeof result.publicKey.verify).toBe("function"); + expect(didResolver.calls).toEqual([]); + }); + + it("rejects a noncanonical signing key ID in a fresh cache entry without fetching", async () => { + const cache = new MapCache(); + configured(cache, { signingKeyId: `${DID}#other` }); + const didResolver = new StubResolver(); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await expect(subject.resolve(DID)).rejects.toThrow(/signing key id/i); + expect(didResolver.calls).toEqual([]); + }); + + it("re-resolves a stale configured row and accepts relative canonical ids", async () => { + const cache = new MapCache(); + configured(cache, { resolvedAt: new Date(NOW.getTime() - 24 * 60 * 60 * 1000) }); + const didResolver = new StubResolver(); + didResolver.response = document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: otherSigningKey, + }, + ], + service: [{ id: "#atproto_labeler", type: "AtprotoLabeler", serviceEndpoint: ENDPOINT }], + }); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + const result = await subject.resolve(DID); + + expect(result.endpoint).toBe(ENDPOINT); + expect(result.signingKeyId).toBe(`${DID}#atproto_label`); + expect(didResolver.calls).toEqual([DID]); + expect(cache.refreshes[0]).toMatchObject({ + did: DID, + identity: { + endpoint: ENDPOINT, + signingKey: otherSigningKey, + signingKeyId: `${DID}#atproto_label`, + }, + now: NOW, + }); + }); + + it("rejects an unconfigured DID without fetching", async () => { + const cache = new MapCache(); + const didResolver = new StubResolver(); + const subject = new LabelerResolver({ cache, resolver: didResolver }); + + await expect(subject.resolve(DID)).rejects.toThrow(/not configured/i); + expect(didResolver.calls).toEqual([]); + }); + + it("resolveFresh bypasses TTL but still requires a configured DID", async () => { + const cache = new MapCache(); + configured(cache); + const didResolver = new StubResolver(); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await subject.resolveFresh(DID); + expect(didResolver.calls).toEqual([DID]); + + const unconfigured = new LabelerResolver({ cache: new MapCache(), resolver: didResolver }); + await expect(unconfigured.resolveFresh("did:web:other.example")).rejects.toThrow( + /not configured/i, + ); + expect(didResolver.calls).toEqual([DID]); + }); + + it("failed stale resolution preserves the prior cache and timestamp", async () => { + const cache = new MapCache(); + const prior = configured(cache, { resolvedAt: new Date(0) }); + const didResolver = new StubResolver(); + didResolver.response = document({ service: [] }); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await expect(subject.resolve(DID)).rejects.toThrow(/atproto_labeler/i); + expect(cache.entries.get(DID)).toEqual(prior); + expect(cache.refreshes).toEqual([]); + }); + + it("resolver failure preserves the prior cache and timestamp", async () => { + const cache = new MapCache(); + const prior = configured(cache, { resolvedAt: new Date(0) }); + const didResolver = new StubResolver(); + didResolver.error = new Error("DID resolver unavailable"); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await expect(subject.resolve(DID)).rejects.toThrow("DID resolver unavailable"); + expect(cache.entries.get(DID)).toEqual(prior); + expect(cache.refreshes).toEqual([]); + }); + + it("invalidate expires only the configured row's freshness", async () => { + const cache = new MapCache(); + const prior = configured(cache); + const didResolver = new StubResolver(); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await subject.invalidate(DID); + + expect(cache.entries.get(DID)).toEqual({ ...prior, resolvedAt: new Date(0) }); + expect(cache.expires).toEqual([DID]); + }); + + it("invalidate makes the next resolve fetch and refresh", async () => { + const cache = new MapCache(); + configured(cache); + const didResolver = new StubResolver(); + didResolver.response = document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: otherSigningKey, + }, + ], + }); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await subject.invalidate(DID); + const result = await subject.resolve(DID); + + expect(didResolver.calls).toEqual([DID]); + expect(cache.refreshes).toHaveLength(1); + expect(cache.refreshes[0]).toMatchObject({ + did: DID, + identity: { + signingKey: otherSigningKey, + signingKeyId: `${DID}#atproto_label`, + }, + now: NOW, + }); + expect(result.signingKeyId).toBe(`${DID}#atproto_label`); + }); + + const invalidDocuments = [ + ["a non-object document", () => null, /DID document must be an object/i], + [ + "a mismatched document id", + () => document({ id: "did:web:other.example" }), + /id does not match/i, + ], + ["missing services", () => document({ service: [] }), /exactly one.*atproto_labeler/i], + [ + "duplicate relative and absolute services", + () => + document({ + service: [ + { id: "#atproto_labeler", type: "AtprotoLabeler", serviceEndpoint: ENDPOINT }, + { + id: `${DID}#atproto_labeler`, + type: "AtprotoLabeler", + serviceEndpoint: ENDPOINT, + }, + ], + }), + /exactly one.*atproto_labeler/i, + ], + [ + "a wrong service type", + () => + document({ + service: [{ id: "#atproto_labeler", type: "Other", serviceEndpoint: ENDPOINT }], + }), + /type AtprotoLabeler/i, + ], + [ + "a service endpoint with credentials", + () => + document({ + service: [ + { + id: "#atproto_labeler", + type: "AtprotoLabeler", + serviceEndpoint: "https://user:pass@labeler.example", + }, + ], + }), + /credentials/i, + ], + [ + "a service endpoint with a fragment", + () => + document({ + service: [ + { + id: "#atproto_labeler", + type: "AtprotoLabeler", + serviceEndpoint: "https://labeler.example/#fragment", + }, + ], + }), + /fragment/i, + ], + [ + "a service endpoint with an empty fragment", + () => + document({ + service: [ + { + id: "#atproto_labeler", + type: "AtprotoLabeler", + serviceEndpoint: "https://labeler.example/#", + }, + ], + }), + /fragment/i, + ], + [ + "a non-HTTPS service endpoint", + () => + document({ + service: [ + { + id: "#atproto_labeler", + type: "AtprotoLabeler", + serviceEndpoint: "http://labeler.example", + }, + ], + }), + /HTTPS/i, + ], + [ + "missing verification methods", + () => document({ verificationMethod: [] }), + /exactly one.*atproto_label/i, + ], + [ + "duplicate relative and absolute methods", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: signingKey, + }, + { + id: `${DID}#atproto_label`, + type: "Multikey", + controller: DID, + publicKeyMultibase: signingKey, + }, + ], + }), + /exactly one.*atproto_label/i, + ], + [ + "a wrong verification method type", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "JsonWebKey2020", + controller: DID, + publicKeyMultibase: signingKey, + }, + ], + }), + /type Multikey/i, + ], + [ + "a wrong verification method controller", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: "did:web:other.example", + publicKeyMultibase: signingKey, + }, + ], + }), + /controller/i, + ], + [ + "a malformed multikey", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: "not-a-key", + }, + ], + }), + /invalid.*Multikey/i, + ], + [ + "a non-P256 multikey", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: wrongCurveKey, + }, + ], + }), + /P-256/i, + ], + [ + "a non-canonical P256 multikey", + () => + document({ + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: DID, + publicKeyMultibase: nonCanonicalKey, + }, + ], + }), + /canonical P-256/i, + ], + ] satisfies Array unknown, RegExp]>; + + it.each(invalidDocuments)("rejects %s", async (_case, makeResponse, message) => { + const cache = new MapCache(); + configured(cache, { resolvedAt: new Date(0) }); + const didResolver = new StubResolver(); + didResolver.response = makeResponse(); + const subject = new LabelerResolver({ cache, resolver: didResolver, now: () => NOW }); + + await expect(subject.resolve(DID)).rejects.toThrow(message); + expect(cache.refreshes).toEqual([]); + }); +}); + +describe("createD1LabelerIdentityCache", () => { + beforeAll(async () => { + await applyD1Migrations(testEnv.DB, testEnv.TEST_MIGRATIONS); + }); + + beforeEach(async () => { + await testEnv.DB.exec("DELETE FROM labelers"); + }); + + async function insertLabeler(overrides: Record = {}): Promise { + const row = { + did: DID, + endpoint: "https://cached.example", + signingKey, + signingKeyId: `${DID}#atproto_label`, + trusted: 1, + addedAt: "2026-07-01T00:00:00.000Z", + resolvedAt: "2026-07-01T00:00:00.000Z", + notes: "operator note", + ...overrides, + }; + await testEnv.DB.prepare( + `INSERT INTO labelers + (did, endpoint, signing_key, signing_key_id, trusted, added_at, last_resolved_at, notes) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + row.did, + row.endpoint, + row.signingKey, + row.signingKeyId, + row.trusted, + row.addedAt, + row.resolvedAt, + row.notes, + ) + .run(); + } + + it("uses labelers as an allowlist and round-trips cached identity", async () => { + const cache = createD1LabelerIdentityCache(testEnv.DB); + expect(await cache.read(DID)).toBeNull(); + + await insertLabeler(); + expect(await cache.read(DID)).toEqual({ + endpoint: "https://cached.example", + signingKey, + signingKeyId: `${DID}#atproto_label`, + resolvedAt: new Date("2026-07-01T00:00:00.000Z"), + }); + }); + + it("refresh updates only cache fields and preserves operator metadata", async () => { + await insertLabeler(); + const cache = createD1LabelerIdentityCache(testEnv.DB); + + await cache.refresh( + DID, + { + endpoint: ENDPOINT, + signingKey: otherSigningKey, + signingKeyId: `${DID}#atproto_label`, + }, + NOW, + ); + + const row = await testEnv.DB.prepare("SELECT * FROM labelers WHERE did = ?").bind(DID).first(); + expect(row).toMatchObject({ + endpoint: ENDPOINT, + signing_key: otherSigningKey, + signing_key_id: `${DID}#atproto_label`, + last_resolved_at: NOW.toISOString(), + trusted: 1, + added_at: "2026-07-01T00:00:00.000Z", + notes: "operator note", + }); + }); + + it("refresh cannot insert an unconfigured DID", async () => { + const cache = createD1LabelerIdentityCache(testEnv.DB); + + await expect( + cache.refresh( + DID, + { endpoint: ENDPOINT, signingKey, signingKeyId: `${DID}#atproto_label` }, + NOW, + ), + ).rejects.toThrow(/not configured/i); + expect(await testEnv.DB.prepare("SELECT did FROM labelers").first()).toBeNull(); + }); + + it("expire changes only last_resolved_at", async () => { + await insertLabeler(); + const before = await testEnv.DB.prepare("SELECT * FROM labelers WHERE did = ?") + .bind(DID) + .first(); + const cache = createD1LabelerIdentityCache(testEnv.DB); + + await cache.expire(DID); + + const after = await testEnv.DB.prepare("SELECT * FROM labelers WHERE did = ?") + .bind(DID) + .first(); + expect(after).toEqual({ ...before, last_resolved_at: "1970-01-01T00:00:00.000Z" }); + }); + + it("failed real-D1 refresh preserves the prior cache and timestamp", async () => { + await insertLabeler(); + const didResolver = new StubResolver(); + didResolver.response = document({ service: [] }); + const subject = new LabelerResolver({ + cache: createD1LabelerIdentityCache(testEnv.DB), + resolver: didResolver, + now: () => NOW, + }); + const before = await testEnv.DB.prepare("SELECT * FROM labelers WHERE did = ?") + .bind(DID) + .first(); + + await expect(subject.resolve(DID)).rejects.toThrow(/atproto_labeler/i); + + const after = await testEnv.DB.prepare("SELECT * FROM labelers WHERE did = ?") + .bind(DID) + .first(); + expect(after).toEqual(before); + }); +}); diff --git a/apps/aggregator/test/labels-consumer.test.ts b/apps/aggregator/test/labels-consumer.test.ts new file mode 100644 index 0000000000..4bddb5aca5 --- /dev/null +++ b/apps/aggregator/test/labels-consumer.test.ts @@ -0,0 +1,598 @@ +/** + * Labels consumer tests. + * + * Runs against real D1 (workerd test pool) with real signed labels: a P-256 + * keypair via `P256PrivateKeyExportable.createKeypair()` and + * `createLabelSigner` from `@emdash-cms/registry-moderation`, with a closure + * `resolveDid` returning a self-consistent DID document — the same helper + * shape as `label-ingestor.test.ts`. This exercises `parseSignedLabel` + + * `encodeSignedLabel` for real rather than through a stub, since digest + * identity (the whole point of this consumer) depends on canonical encoding. + */ + +import { create as createCid, toString as cidToString, CODEC_RAW } from "@atcute/cid"; +import { P256PrivateKeyExportable } from "@atcute/crypto"; +import { toBase64Url } from "@atcute/multibase"; +import { + createLabelSigner, + encodeSignedLabel, + type LabelSigner, + type SignedLabel, + type UnsignedLabel, +} from "@emdash-cms/registry-moderation"; +import { applyD1Migrations, env } from "cloudflare:test"; +import { beforeAll, beforeEach, describe, expect, it } from "vitest"; + +import { toWire, type LabelIngestJob } from "../src/label-ingest-types.js"; +import { + type ConsumerDeps, + drainLabelsDeadLetterBatch, + type MessageController, + processLabelsBatch, +} from "../src/labels-consumer.js"; + +interface TestEnv { + DB: D1Database; + TEST_MIGRATIONS: Parameters[1]; +} +const testEnv = env as unknown as TestEnv; + +const LABELER_DID = "did:web:labeler.example"; +const UNTRUSTED_LABELER_DID = "did:web:untrusted-labeler.example"; +const UNKNOWN_LABELER_DID = "did:web:unknown-labeler.example"; +const SUBJECT_URI = "at://did:plc:subject00000000000000000/com.example.thing/x"; +const NOW = new Date("2026-07-11T12:00:00.000Z"); + +let signer: LabelSigner; +let untrustedSigner: LabelSigner; + +async function makeSigner(issuerDid: string): Promise { + const keypair = await P256PrivateKeyExportable.createKeypair(); + const privateKey = toBase64Url(await keypair.exportPrivateKey("raw")); + const multikey = await keypair.exportPublicKey("multikey"); + const document = { + id: issuerDid, + verificationMethod: [ + { + id: "#atproto_label", + type: "Multikey", + controller: issuerDid, + publicKeyMultibase: multikey, + }, + ], + }; + return createLabelSigner({ issuerDid, privateKey, resolveDid: async () => document }); +} + +async function makeCid(seed: string): Promise { + const encoded = new TextEncoder().encode(seed); + const bytes = new Uint8Array(new ArrayBuffer(encoded.length)); + bytes.set(encoded); + const cid = await createCid(CODEC_RAW, bytes); + return cidToString(cid); +} + +async function digestOf(label: SignedLabel): Promise { + const bytes = encodeSignedLabel(label); + const hash = await crypto.subtle.digest("SHA-256", bytes); + return Array.from(new Uint8Array(hash), (b) => b.toString(16).padStart(2, "0")).join(""); +} + +function labelInput( + overrides: Partial> = {}, +): Omit { + return { + ver: 1, + uri: SUBJECT_URI, + val: "test-value", + cts: "2026-07-10T12:00:00.000Z", + ...overrides, + }; +} + +function jobFor( + src: string, + sourceSequence: number, + frameIndex: number, + label: SignedLabel, +): LabelIngestJob { + return { src, sourceSequence, frameIndex, label: toWire(label) }; +} + +class FakeMessage implements MessageController { + acked = 0; + retried = 0; + constructor(readonly body: LabelIngestJob) {} + ack(): void { + this.acked += 1; + } + retry(): void { + this.retried += 1; + } +} + +async function runBatch(jobs: LabelIngestJob[], deps?: ConsumerDeps): Promise { + const messages = jobs.map((job) => new FakeMessage(job)); + await processLabelsBatch({ messages }, { DB: testEnv.DB } as unknown as Env, deps); + return messages; +} + +async function seedLabeler(did: string, trusted: boolean): Promise { + await testEnv.DB.prepare( + `INSERT INTO labelers (did, endpoint, signing_key, signing_key_id, trusted, added_at, last_resolved_at) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + did, + "https://labeler.example", + "unused-in-tests", + `${did}#atproto_label`, + trusted ? 1 : 0, + NOW.toISOString(), + NOW.toISOString(), + ) + .run(); +} + +async function countRows(table: string): Promise { + const row = await testEnv.DB.prepare(`SELECT COUNT(*) as n FROM ${table}`).first<{ n: number }>(); + return row?.n ?? 0; +} + +interface LabelRow { + digest: string; + src: string; + uri: string; + cid: string | null; + val: string; + neg: number; + cts: string; + cts_epoch_ms: number; + exp: string | null; + exp_epoch_ms: number | null; + sig: ArrayBuffer; + ver: number; + source_sequence: number; + frame_index: number; + trusted: number; + received_at: string; +} + +interface LabelStateRow { + src: string; + uri: string; + val: string; + cid: string | null; + neg: number; + cts: string; + cts_epoch_ms: number; + exp: string | null; + exp_epoch_ms: number | null; + digest: string; + source_sequence: number; + frame_index: number; + trusted: number; +} + +async function getLabelState(src: string, uri: string, val: string): Promise { + return ( + (await testEnv.DB.prepare(`SELECT * FROM label_state WHERE src = ? AND uri = ? AND val = ?`) + .bind(src, uri, val) + .first()) ?? null + ); +} + +const DEPS: ConsumerDeps = { db: testEnv.DB, now: () => NOW }; + +beforeAll(async () => { + await applyD1Migrations(testEnv.DB, testEnv.TEST_MIGRATIONS); + signer = await makeSigner(LABELER_DID); + untrustedSigner = await makeSigner(UNTRUSTED_LABELER_DID); +}); + +beforeEach(async () => { + for (const table of ["labels", "label_state", "dead_letters", "labelers"]) { + await testEnv.DB.prepare(`DELETE FROM ${table}`).run(); + } + await seedLabeler(LABELER_DID, true); +}); + +describe("processLabelsBatch: happy path", () => { + it("writes one labels row and one label_state row for a single verified label", async () => { + const label = await signer.sign(labelInput()); + const job = jobFor(LABELER_DID, 1, 0, label); + + const [msg] = await runBatch([job], DEPS); + expect(msg?.acked).toBe(1); + expect(msg?.retried).toBe(0); + + const digest = await digestOf(label); + const row = await testEnv.DB.prepare(`SELECT * FROM labels WHERE digest = ?`) + .bind(digest) + .first(); + expect(row).toMatchObject({ + digest, + src: LABELER_DID, + uri: SUBJECT_URI, + cid: null, + val: "test-value", + neg: 0, + cts: "2026-07-10T12:00:00.000Z", + cts_epoch_ms: Date.parse("2026-07-10T12:00:00.000Z"), + exp: null, + exp_epoch_ms: null, + ver: 1, + source_sequence: 1, + frame_index: 0, + trusted: 1, + received_at: NOW.toISOString(), + }); + expect(new Uint8Array(row!.sig)).toEqual(label.sig); + + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state).toMatchObject({ + digest, + neg: 0, + cts_epoch_ms: Date.parse("2026-07-10T12:00:00.000Z"), + source_sequence: 1, + frame_index: 0, + trusted: 1, + }); + }); +}); + +describe("processLabelsBatch: exact redelivery", () => { + it("is a silent no-op — acked both times, one history row, state unchanged", async () => { + const label = await signer.sign(labelInput()); + const job = jobFor(LABELER_DID, 1, 0, label); + + const [first] = await runBatch([job], DEPS); + expect(first?.acked).toBe(1); + const [second] = await runBatch([job], DEPS); + expect(second?.acked).toBe(1); + expect(second?.retried).toBe(0); + + expect(await countRows("labels")).toBe(1); + expect(await countRows("dead_letters")).toBe(0); + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state).toMatchObject({ source_sequence: 1, frame_index: 0 }); + }); +}); + +describe("processLabelsBatch: same-cts survival", () => { + it("keeps both history rows when two different labels share (src, uri, val, cts)", async () => { + const cts = "2026-07-10T12:00:00.000Z"; + const cidA = await makeCid("label-a"); + const cidB = await makeCid("label-b"); + const labelA = await signer.sign(labelInput({ cts, cid: cidA })); + const labelB = await signer.sign(labelInput({ cts, cid: cidB })); + const jobA = jobFor(LABELER_DID, 1, 0, labelA); + const jobB = jobFor(LABELER_DID, 2, 0, labelB); + + const [msgA] = await runBatch([jobA], DEPS); + const [msgB] = await runBatch([jobB], DEPS); + expect(msgA?.acked).toBe(1); + expect(msgB?.acked).toBe(1); + + expect(await countRows("labels")).toBe(2); + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state?.digest).toBe(await digestOf(labelB)); + expect(state).toMatchObject({ source_sequence: 2, frame_index: 0, cid: cidB }); + }); +}); + +describe("processLabelsBatch: projection ordering", () => { + it("a newer cts replaces state", async () => { + const older = await signer.sign(labelInput({ cts: "2026-07-10T12:00:00.000Z" })); + const newer = await signer.sign(labelInput({ cts: "2026-07-10T13:00:00.000Z" })); + await runBatch([jobFor(LABELER_DID, 1, 0, older)], DEPS); + await runBatch([jobFor(LABELER_DID, 2, 0, newer)], DEPS); + + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state?.digest).toBe(await digestOf(newer)); + }); + + it("an older-cts job arriving after (out-of-order delivery) leaves state at the newer event", async () => { + const older = await signer.sign(labelInput({ cts: "2026-07-10T12:00:00.000Z" })); + const newer = await signer.sign(labelInput({ cts: "2026-07-10T13:00:00.000Z" })); + await runBatch([jobFor(LABELER_DID, 2, 0, newer)], DEPS); + await runBatch([jobFor(LABELER_DID, 1, 0, older)], DEPS); + + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state?.digest).toBe(await digestOf(newer)); + expect(await countRows("labels")).toBe(2); + }); + + it("equal cts with lower coordinates does not replace state", async () => { + const cts = "2026-07-10T12:00:00.000Z"; + const cidA = await makeCid("equal-cts-a"); + const cidB = await makeCid("equal-cts-b"); + const winner = await signer.sign(labelInput({ cts, cid: cidA })); + const loser = await signer.sign(labelInput({ cts, cid: cidB })); + await runBatch([jobFor(LABELER_DID, 5, 2, winner)], DEPS); + await runBatch([jobFor(LABELER_DID, 5, 1, loser)], DEPS); + + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state?.digest).toBe(await digestOf(winner)); + }); +}); + +describe("processLabelsBatch: negation", () => { + it("a later neg=1 event wins state; history keeps both", async () => { + const positive = await signer.sign(labelInput({ cts: "2026-07-10T12:00:00.000Z" })); + const negation = await signer.sign(labelInput({ cts: "2026-07-10T13:00:00.000Z", neg: true })); + await runBatch([jobFor(LABELER_DID, 1, 0, positive)], DEPS); + await runBatch([jobFor(LABELER_DID, 2, 0, negation)], DEPS); + + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state).toMatchObject({ neg: 1, digest: await digestOf(negation) }); + expect(await countRows("labels")).toBe(2); + }); +}); + +describe("processLabelsBatch: expiry metadata", () => { + it("stores exp/exp_epoch_ms on both tables when present", async () => { + const label = await signer.sign(labelInput({ exp: "2027-01-01T00:00:00.000Z" })); + await runBatch([jobFor(LABELER_DID, 1, 0, label)], DEPS); + + const digest = await digestOf(label); + const row = await testEnv.DB.prepare(`SELECT exp, exp_epoch_ms FROM labels WHERE digest = ?`) + .bind(digest) + .first<{ exp: string; exp_epoch_ms: number }>(); + expect(row).toMatchObject({ + exp: "2027-01-01T00:00:00.000Z", + exp_epoch_ms: Date.parse("2027-01-01T00:00:00.000Z"), + }); + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state).toMatchObject({ + exp: "2027-01-01T00:00:00.000Z", + exp_epoch_ms: Date.parse("2027-01-01T00:00:00.000Z"), + }); + }); + + it("leaves exp/exp_epoch_ms NULL when absent", async () => { + const label = await signer.sign(labelInput()); + await runBatch([jobFor(LABELER_DID, 1, 0, label)], DEPS); + + const digest = await digestOf(label); + const row = await testEnv.DB.prepare(`SELECT exp, exp_epoch_ms FROM labels WHERE digest = ?`) + .bind(digest) + .first<{ exp: string | null; exp_epoch_ms: number | null }>(); + expect(row).toMatchObject({ exp: null, exp_epoch_ms: null }); + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "test-value"); + expect(state).toMatchObject({ exp: null, exp_epoch_ms: null }); + }); +}); + +describe("processLabelsBatch: trusted snapshot", () => { + it("a job from an untrusted source writes trusted=0, unaffected by later flips", async () => { + await seedLabeler(UNTRUSTED_LABELER_DID, false); + const label = await untrustedSigner.sign(labelInput()); + await runBatch([jobFor(UNTRUSTED_LABELER_DID, 1, 0, label)], DEPS); + + const digest = await digestOf(label); + const before = await testEnv.DB.prepare(`SELECT trusted FROM labels WHERE digest = ?`) + .bind(digest) + .first<{ trusted: number }>(); + expect(before?.trusted).toBe(0); + const stateBefore = await getLabelState(UNTRUSTED_LABELER_DID, SUBJECT_URI, "test-value"); + expect(stateBefore?.trusted).toBe(0); + + await testEnv.DB.prepare(`UPDATE labelers SET trusted = 1 WHERE did = ?`) + .bind(UNTRUSTED_LABELER_DID) + .run(); + + const after = await testEnv.DB.prepare(`SELECT trusted FROM labels WHERE digest = ?`) + .bind(digest) + .first<{ trusted: number }>(); + expect(after?.trusted).toBe(0); + const stateAfter = await getLabelState(UNTRUSTED_LABELER_DID, SUBJECT_URI, "test-value"); + expect(stateAfter?.trusted).toBe(0); + }); +}); + +describe("processLabelsBatch: LABEL_UNKNOWN_SOURCE", () => { + it("dead-letters a job whose src has no labelers row, acks, writes nothing else", async () => { + const orphanSigner = await makeSigner(UNKNOWN_LABELER_DID); + const label = await orphanSigner.sign(labelInput()); + const [msg] = await runBatch([jobFor(UNKNOWN_LABELER_DID, 1, 0, label)], DEPS); + + expect(msg?.acked).toBe(1); + expect(msg?.retried).toBe(0); + expect(await countRows("labels")).toBe(0); + expect(await countRows("label_state")).toBe(0); + const dl = await testEnv.DB.prepare(`SELECT reason, did FROM dead_letters`).first<{ + reason: string; + did: string; + }>(); + expect(dl).toMatchObject({ reason: "LABEL_UNKNOWN_SOURCE", did: UNKNOWN_LABELER_DID }); + }); +}); + +describe("processLabelsBatch: LABEL_INVALID", () => { + it("dead-letters a structurally broken wire label, acks, writes nothing else", async () => { + const label = await signer.sign(labelInput()); + const wire = toWire(label); + const job: LabelIngestJob = { + src: LABELER_DID, + sourceSequence: 1, + frameIndex: 0, + label: { ...wire, val: "" }, // empty val fails validateLabelValue + }; + + const [msg] = await runBatch([job], DEPS); + expect(msg?.acked).toBe(1); + expect(msg?.retried).toBe(0); + expect(await countRows("labels")).toBe(0); + expect(await countRows("label_state")).toBe(0); + const dl = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(dl?.reason).toBe("LABEL_INVALID"); + }); +}); + +describe("processLabelsBatch: src mismatch", () => { + it("dead-letters a job whose src differs from the signed label's src", async () => { + await seedLabeler(UNTRUSTED_LABELER_DID, false); + const label = await untrustedSigner.sign(labelInput()); + const [msg] = await runBatch([jobFor(LABELER_DID, 1, 0, label)], DEPS); + + expect(msg?.acked).toBe(1); + expect(msg?.retried).toBe(0); + expect(await countRows("labels")).toBe(0); + expect(await countRows("label_state")).toBe(0); + const dl = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(dl?.reason).toBe("LABEL_INVALID"); + }); +}); + +describe("processLabelsBatch: LABEL_COORDINATE_CONFLICT", () => { + it("a second, different label at the same coordinates dead-letters; first survives unaffected", async () => { + const first = await signer.sign(labelInput({ val: "value-a" })); + const second = await signer.sign(labelInput({ val: "value-b" })); + const jobFirst = jobFor(LABELER_DID, 9, 3, first); + const jobSecond = jobFor(LABELER_DID, 9, 3, second); + + const [msg1] = await runBatch([jobFirst], DEPS); + expect(msg1?.acked).toBe(1); + const [msg2] = await runBatch([jobSecond], DEPS); + expect(msg2?.acked).toBe(1); + expect(msg2?.retried).toBe(0); + + expect(await countRows("labels")).toBe(1); + const digestFirst = await digestOf(first); + const row = await testEnv.DB.prepare(`SELECT digest FROM labels`).first<{ digest: string }>(); + expect(row?.digest).toBe(digestFirst); + + const dl = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(dl?.reason).toBe("LABEL_COORDINATE_CONFLICT"); + + // No state row for the second label's val — its batch rolled back entirely. + const stateSecond = await getLabelState(LABELER_DID, SUBJECT_URI, "value-b"); + expect(stateSecond).toBeNull(); + const stateFirst = await getLabelState(LABELER_DID, SUBJECT_URI, "value-a"); + expect(stateFirst?.digest).toBe(digestFirst); + }); +}); + +describe("processLabelsBatch: cursor-0 replay idempotency", () => { + it("replaying an identical mixed batch leaves history count and state identical", async () => { + const cidC = await makeCid("replay-c"); + const labels = await Promise.all([ + signer.sign(labelInput({ val: "v1", cts: "2026-07-10T12:00:00.000Z" })), + signer.sign(labelInput({ val: "v2", cts: "2026-07-10T12:05:00.000Z" })), + signer.sign(labelInput({ val: "v3", cts: "2026-07-10T12:10:00.000Z", neg: true })), + signer.sign( + labelInput({ val: "v4", cts: "2026-07-10T12:15:00.000Z", exp: "2027-01-01T00:00:00.000Z" }), + ), + signer.sign(labelInput({ val: "v5", cts: "2026-07-10T12:20:00.000Z", cid: cidC })), + ]); + const jobs = labels.map((label, i) => jobFor(LABELER_DID, 100 + i, 0, label)); + + await runBatch(jobs, DEPS); + const historyBefore = await countRows("labels"); + const statesBefore = await testEnv.DB.prepare( + `SELECT src, uri, val, digest, cts_epoch_ms, source_sequence, frame_index, trusted + FROM label_state ORDER BY val`, + ).all(); + + // Replay the identical set in a fresh batch (simulates cursor-0 rebuild). + await runBatch(jobs, DEPS); + const historyAfter = await countRows("labels"); + const statesAfter = await testEnv.DB.prepare( + `SELECT src, uri, val, digest, cts_epoch_ms, source_sequence, frame_index, trusted + FROM label_state ORDER BY val`, + ).all(); + + expect(historyAfter).toBe(historyBefore); + expect(historyBefore).toBe(5); + expect(statesAfter.results).toEqual(statesBefore.results); + }); +}); + +describe("processLabelsBatch: per-message isolation", () => { + it("a valid job lands while an invalid job in the same batch dead-letters; both ack", async () => { + const valid = await signer.sign(labelInput({ val: "valid-value" })); + const validJob = jobFor(LABELER_DID, 1, 0, valid); + const invalidWire = toWire(await signer.sign(labelInput({ val: "invalid-value" }))); + const invalidJob: LabelIngestJob = { + src: LABELER_DID, + sourceSequence: 1, + frameIndex: 1, + label: { ...invalidWire, val: "" }, + }; + + const [msgValid, msgInvalid] = await runBatch([validJob, invalidJob], DEPS); + expect(msgValid?.acked).toBe(1); + expect(msgInvalid?.acked).toBe(1); + + expect(await countRows("labels")).toBe(1); + const state = await getLabelState(LABELER_DID, SUBJECT_URI, "valid-value"); + expect(state).not.toBeNull(); + const dl = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(dl?.reason).toBe("LABEL_INVALID"); + }); +}); + +describe("processLabelsBatch: transient D1 failure", () => { + it("retries when db.batch() throws a non-constraint error", async () => { + const label = await signer.sign(labelInput()); + const job = jobFor(LABELER_DID, 1, 0, label); + + const failingDb = { + prepare: () => ({ + bind: () => ({ + first: () => Promise.resolve({ trusted: 1 }), + run: () => Promise.resolve({ meta: { changes: 1 } }), + }), + }), + batch: () => Promise.reject(new Error("D1 unavailable")), + } as unknown as D1Database; + + const [msg] = await runBatch([job], { db: failingDb, now: () => NOW }); + expect(msg?.retried).toBe(1); + expect(msg?.acked).toBe(0); + }); +}); + +describe("drainLabelsDeadLetterBatch", () => { + it("acks each message and writes a forensics row", async () => { + const label = await signer.sign(labelInput()); + const job = jobFor(LABELER_DID, 1, 0, label); + const message = new FakeMessage(job); + + await drainLabelsDeadLetterBatch({ messages: [message] }, { DB: testEnv.DB } as unknown as Env); + + expect(message.acked).toBe(1); + expect(await countRows("dead_letters")).toBe(1); + const dl = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(dl?.reason).toBe("LABEL_UNEXPECTED_ERROR"); + }); + + it("retries the message when writeDeadLetter throws", async () => { + const label = await signer.sign(labelInput()); + const job = jobFor(LABELER_DID, 1, 0, label); + const message = new FakeMessage(job); + const failingDb = { + prepare: () => ({ + bind: () => ({ + run: () => Promise.reject(new Error("D1 unavailable")), + }), + }), + } as unknown as D1Database; + + await drainLabelsDeadLetterBatch({ messages: [message] }, { DB: failingDb } as unknown as Env); + + expect(message.retried).toBe(1); + expect(message.acked).toBe(0); + }); +}); diff --git a/apps/aggregator/test/migration-upgrade.test.ts b/apps/aggregator/test/migration-upgrade.test.ts new file mode 100644 index 0000000000..cc17ce2b67 --- /dev/null +++ b/apps/aggregator/test/migration-upgrade.test.ts @@ -0,0 +1,61 @@ +/** + * Upgrade-path regression for the `labellers` -> `labelers` rename. + * + * `0001_init.sql` shipped on `main` creating a `labellers` table, then a later + * revision renamed it in place. Editing an applied migration is invisible to + * databases that already recorded it, so instances deployed off `main` kept + * `labellers` while `src/` moved to querying `labelers` -> every such query + * failed with `no such table`. The rename now lands as a forward migration. + * + * This test provisions the schema the way a `main` deployment had it (via the + * `MAIN_MIGRATIONS` fixture, a frozen copy of `main`'s shipped migration set), + * then applies the live migration set on top. `applyD1Migrations` skips + * migrations already recorded by name, so neither frozen file is re-run — + * exactly what happens on a real upgrade. It lives in its own file because the workers pool isolates + * D1 storage per test file, not per test, and this scenario needs a database + * that starts from `main`'s schema rather than the live one. + */ + +import { applyD1Migrations, env } from "cloudflare:test"; +import { describe, expect, it } from "vitest"; + +interface TestEnv { + DB: D1Database; + TEST_MIGRATIONS: Parameters[1]; + MAIN_MIGRATIONS: Parameters[1]; +} + +const testEnv = env as unknown as TestEnv; + +describe("labellers -> labelers upgrade", () => { + it("converges an already-migrated main database on the labelers table", async () => { + await applyD1Migrations(testEnv.DB, testEnv.MAIN_MIGRATIONS); + expect((await testEnv.DB.prepare(`SELECT did FROM labellers`).all()).success).toBe(true); + + const did = "did:web:labels.example.test"; + await testEnv.DB.prepare( + `INSERT INTO labellers (did, endpoint, signing_key, signing_key_id, trusted, added_at, last_resolved_at) + VALUES (?, ?, ?, ?, 1, ?, ?)`, + ) + .bind( + did, + "https://labels.example.test/subscribe", + "zKey", + `${did}#atproto_label`, + "2026-01-01T00:00:00.000Z", + "2026-01-01T00:00:00.000Z", + ) + .run(); + + await applyD1Migrations(testEnv.DB, testEnv.TEST_MIGRATIONS); + + // The RENAME must carry the existing row across, not drop/recreate the table. + const result = await testEnv.DB.prepare( + `SELECT did FROM labelers WHERE trusted = 1 ORDER BY did ASC`, + ).all<{ did: string }>(); + expect(result.success).toBe(true); + expect(result.results?.map((row) => row.did)).toContain(did); + + await expect(testEnv.DB.prepare(`SELECT did FROM labellers`).all()).rejects.toThrow(); + }); +}); diff --git a/apps/aggregator/test/migrations.test.ts b/apps/aggregator/test/migrations.test.ts new file mode 100644 index 0000000000..0f17d72b50 --- /dev/null +++ b/apps/aggregator/test/migrations.test.ts @@ -0,0 +1,35 @@ +import { applyD1Migrations, env } from "cloudflare:test"; +import { describe, expect, it } from "vitest"; + +interface TestEnv { + DB: D1Database; + TEST_MIGRATIONS: Parameters[1]; +} +const testEnv = env as unknown as TestEnv; + +describe("0003_label_history_identity", () => { + it("refuses to run when label rows already exist", async () => { + const migrations = testEnv.TEST_MIGRATIONS; + const guarded = migrations.findIndex((m) => m.name.includes("0003_label_history_identity")); + expect(guarded).toBeGreaterThan(0); + + await applyD1Migrations(testEnv.DB, migrations.slice(0, guarded)); + await testEnv.DB.prepare( + `INSERT INTO labels (src, uri, val, cts, sig, received_at) + VALUES (?, ?, ?, ?, ?, ?)`, + ) + .bind( + "did:web:labels.example", + "at://did:plc:x/com.emdashcms.experimental.package.release/pkg:1.0.0", + "security-yanked", + "2026-01-01T00:00:00.000Z", + new Uint8Array([1, 2, 3]), + "2026-01-01T00:00:00.000Z", + ) + .run(); + + await expect( + applyD1Migrations(testEnv.DB, migrations.slice(guarded, guarded + 1)), + ).rejects.toThrow(); + }); +}); diff --git a/apps/aggregator/test/pds-verify.test.ts b/apps/aggregator/test/pds-verify.test.ts index 68c2055d10..f3d62b2330 100644 --- a/apps/aggregator/test/pds-verify.test.ts +++ b/apps/aggregator/test/pds-verify.test.ts @@ -15,12 +15,21 @@ */ import { P256PublicKey, P256PrivateKeyExportable } from "@atcute/crypto"; +import type { DnsResolver } from "emdash/security/ssrf"; import { beforeAll, describe, expect, it } from "vitest"; import { fetchAndVerifyRecord, isTransient, PdsVerificationError } from "../src/pds-verify.js"; const TEST_DID = "did:plc:test00000000000000000000"; const TEST_PDS = "https://pds.test.example"; +const TEST_PDS_HOST = "pds.test.example"; +/** A public, non-reserved address so the default resolver stub passes the + * SSRF blocklist. Real production uses `cloudflareDohResolver`. */ +const PUBLIC_IP = "93.184.216.34"; + +/** Default resolver stub: every hostname maps to one public address. Tests that + * exercise the SSRF boundary pass their own. */ +const publicResolver: DnsResolver = async () => [PUBLIC_IP]; let publicKey: P256PublicKey; @@ -44,6 +53,8 @@ function buildOpts(overrides: { fetch: typeof fetch; timeoutMs?: number; maxResponseBytes?: number; + pds?: string; + resolveHostname?: DnsResolver; }) { return { pds: TEST_PDS, @@ -51,6 +62,7 @@ function buildOpts(overrides: { collection: "com.emdashcms.experimental.package.profile", rkey: "demo", publicKey, + resolveHostname: publicResolver, ...overrides, }; } @@ -149,6 +161,181 @@ describe("fetchAndVerifyRecord — HTTP path", () => { }); }); +describe("fetchAndVerifyRecord — SSRF egress hardening", () => { + const rejectFetch: typeof fetch = () => { + throw new Error("fetch must not be reached when the URL is blocked"); + }; + + it("fails closed with PDS_ADDRESS_BLOCKED when no resolver is provided", async () => { + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: rejectFetch, resolveHostname: undefined })), + ); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + }); + + it("rejects a non-HTTPS PDS endpoint with PDS_ADDRESS_BLOCKED", async () => { + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: rejectFetch, pds: "http://pds.test.example" })), + ); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + }); + + it("maps an empty DNS answer to the transient PDS_NETWORK_ERROR", async () => { + // An empty resolver answer (NXDOMAIN / NOERROR-NODATA / CNAME-only) is a + // host mid-propagation, not a disallowed address — it must retry, not + // dead-letter. A genuinely-gone host dead-letters via retry exhaustion. + // Keyed on the shared helper's `Hostname resolved to no addresses` wording; + // this breaks if that wording changes rather than silently mis-classifying. + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: rejectFetch, resolveHostname: async () => [] })), + ); + expect(err.reason).toBe("PDS_NETWORK_ERROR"); + expect(isTransient(err.reason, err.status)).toBe(true); + // Carries a re-delivery delay so retries span the DNS-propagation window. + expect(err.retryAfterSeconds).toBeGreaterThan(0); + }); + + it("maps a resolver infrastructure failure to the transient PDS_NETWORK_ERROR", async () => { + // A throwing resolver models a DoH network error / SERVFAIL / timeout — + // transient infrastructure, not a disallowed address. It must NOT dead-letter. + // If the shared helper's `Could not resolve hostname:` wording ever changes, + // this assertion breaks instead of silently mis-classifying as permanent. + const throwingResolver: DnsResolver = () => Promise.reject(new Error("DoH 503")); + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: rejectFetch, resolveHostname: throwingResolver })), + ); + expect(err.reason).toBe("PDS_NETWORK_ERROR"); + expect(isTransient(err.reason, err.status)).toBe(true); + expect(err.retryAfterSeconds).toBeGreaterThan(0); + }); + + it("rejects when the endpoint resolves to a private address", async () => { + const err = await captureRejection( + fetchAndVerifyRecord( + buildOpts({ fetch: rejectFetch, resolveHostname: async () => ["10.0.0.5"] }), + ), + ); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + }); + + it("re-validates a redirect target and blocks one resolving to a reserved address", async () => { + // Initial host resolves public; the redirect target resolves to the + // cloud-metadata address. The private target must never be fetched. + const resolver: DnsResolver = async (host) => + host === TEST_PDS_HOST ? [PUBLIC_IP] : ["169.254.169.254"]; + let evilFetched = false; + const redirectModes: RequestInit["redirect"][] = []; + const fetchImpl: typeof fetch = (input, init) => { + redirectModes.push(init?.redirect); + const href = + typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (href.includes(TEST_PDS_HOST)) { + return Promise.resolve( + new Response(null, { + status: 302, + headers: { location: "https://metadata.evil.example/x" }, + }), + ); + } + evilFetched = true; + return Promise.resolve(new Response(new Uint8Array([0]), { status: 200 })); + }; + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: fetchImpl, resolveHostname: resolver })), + ); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + expect(evilFetched).toBe(false); + // `redirect: "manual"` is load-bearing: it's what forces our own per-hop + // re-validation instead of fetch auto-following unchecked. + expect(redirectModes).toEqual(["manual"]); + }); + + it("follows a redirect to a public target under redirect:manual on every hop", async () => { + // Both hosts resolve public; the redirect is followed to the second host, + // which returns garbage bytes (verifyRecord then rejects). The point is + // that each hop was issued with `redirect: "manual"`. + const redirectModes: RequestInit["redirect"][] = []; + const fetchImpl: typeof fetch = (input, init) => { + redirectModes.push(init?.redirect); + const href = + typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (href.includes(TEST_PDS_HOST)) { + return Promise.resolve( + new Response(null, { + status: 302, + headers: { location: "https://mirror.test.example/x" }, + }), + ); + } + return Promise.resolve(new Response(new Uint8Array([1, 2, 3]), { status: 200 })); + }; + await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: fetchImpl, resolveHostname: publicResolver })), + ); + expect(redirectModes).toEqual(["manual", "manual"]); + }); + + it("rejects a malformed redirect Location with PDS_ADDRESS_BLOCKED", async () => { + // `http://` has an empty authority — `new URL` throws. A hostile publisher + // controls this header, so the parse failure must be a blocked redirect, + // not an escaping UNEXPECTED_ERROR, and the next hop must not be fetched. + let hops = 0; + const fetchImpl: typeof fetch = () => { + hops += 1; + return Promise.resolve(new Response(null, { status: 302, headers: { location: "http://" } })); + }; + const err = await captureRejection(fetchAndVerifyRecord(buildOpts({ fetch: fetchImpl }))); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + expect(hops).toBe(1); + }); + + it("rejects with PDS_ADDRESS_BLOCKED when redirects exceed the hop limit", async () => { + // Every hop redirects to a fresh public host. After MAX_PDS_REDIRECTS + // followed hops the next redirect is refused (permanent — dead-letter). + let hops = 0; + const fetchImpl: typeof fetch = () => { + hops += 1; + return Promise.resolve( + new Response(null, { + status: 302, + headers: { location: `https://hop-${hops}.test.example/x` }, + }), + ); + }; + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: fetchImpl, resolveHostname: publicResolver })), + ); + expect(err.reason).toBe("PDS_ADDRESS_BLOCKED"); + expect(err.message).toMatch(/exceeded 3 redirects/); + // Initial hop + 3 followed redirects are fetched; the 4th is refused + // before a fetch is issued. + expect(hops).toBe(4); + }); + + it("bounds a slow-drip body by the wall-clock deadline", async () => { + // One byte then silence; the read only unblocks when the deadline aborts + // the fetch signal, which errors the stream. + const fetchImpl: typeof fetch = (_input, init) => { + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(new Uint8Array([1])); + init?.signal?.addEventListener("abort", () => { + controller.error(new DOMException("aborted", "AbortError")); + }); + }, + }); + return Promise.resolve(new Response(stream, { status: 200 })); + }; + const err = await captureRejection( + fetchAndVerifyRecord(buildOpts({ fetch: fetchImpl, timeoutMs: 30 })), + ); + expect(err.reason).toBe("PDS_NETWORK_ERROR"); + expect(err.message).toMatch(/aborted after 30ms/); + // A non-resolution transient retries immediately — no propagation delay. + expect(err.retryAfterSeconds).toBeUndefined(); + }); +}); + describe("isTransient policy", () => { it("network errors retry", () => { expect(isTransient("PDS_NETWORK_ERROR", undefined)).toBe(true); diff --git a/apps/aggregator/test/read-api.test.ts b/apps/aggregator/test/read-api.test.ts index 063a6474f4..6a12c755cc 100644 --- a/apps/aggregator/test/read-api.test.ts +++ b/apps/aggregator/test/read-api.test.ts @@ -14,7 +14,7 @@ import { NSID } from "@emdash-cms/registry-lexicons"; import { applyD1Migrations, env, SELF } from "cloudflare:test"; -import { beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; interface TestEnv { DB: D1Database; @@ -26,6 +26,13 @@ const DID_A = "did:plc:read000000000000000000aa"; const DID_B = "did:plc:read000000000000000000bb"; const NOW = new Date("2026-05-10T12:00:00.000Z"); +/** Default-policy labeler: seeded `trusted = 1` so the missing-header + * default (`SELECT did FROM labelers WHERE trusted = 1`) picks it up. */ +const LABELER_DID = "did:web:labels.example"; +/** Configured but untrusted — absent from the default policy, but + * available (and echoed) when a request explicitly names it. */ +const UNTRUSTED_LABELER_DID = "did:web:untrusted-labels.example"; + beforeAll(async () => { await applyD1Migrations(testEnv.DB, testEnv.TEST_MIGRATIONS); }); @@ -37,6 +44,8 @@ beforeEach(async () => { await testEnv.DB.prepare("DELETE FROM packages").run(); await testEnv.DB.prepare("DELETE FROM publishers").run(); await testEnv.DB.prepare("DELETE FROM publisher_verifications").run(); + await testEnv.DB.prepare("DELETE FROM label_state").run(); + await testEnv.DB.prepare("DELETE FROM labelers").run(); }); interface SeedPackageOpts { @@ -138,6 +147,155 @@ function defaultVersionSort(version: string): string { return `${pad(major)}.${pad(minor)}.${pad(patch)}.~`; } +function packageUri(slug: string, did: string = DID_A): string { + return `at://${did}/${NSID.packageProfile}/${slug}`; +} + +function releaseUri(pkg: string, version: string, did: string = DID_A): string { + return `at://${did}/${NSID.packageRelease}/${pkg}:${version}`; +} + +function publisherUri(did: string = DID_A): string { + return `at://${did}/${NSID.publisherProfile}/self`; +} + +interface SeedPublisherOpts { + did?: string; + displayName?: string; + description?: string | null; + url?: string | null; + contact?: unknown[] | null; + updatedAt?: string | null; + cid?: string; + indexedAt?: string; + verifiedAt?: string; +} + +async function seedPublisher(opts: SeedPublisherOpts = {}): Promise { + const did = opts.did ?? DID_A; + const indexedAt = opts.indexedAt ?? NOW.toISOString(); + await testEnv.DB.prepare( + `INSERT INTO publishers + (did, display_name, description, url, contact, updated_at, + record_blob, signature_metadata, verified_at, indexed_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + did, + opts.displayName ?? "Demo Publisher", + opts.description ?? null, + opts.url ?? null, + opts.contact === null + ? null + : JSON.stringify(opts.contact ?? [{ kind: "security", email: "sec@pub.test" }]), + opts.updatedAt ?? null, + new Uint8Array([0xb1, 0xb2, 0xb3]), + JSON.stringify({ cid: opts.cid ?? "bafypub" }), + opts.verifiedAt ?? NOW.toISOString(), + indexedAt, + ) + .run(); +} + +/** Seeds a `labelers` row so a DID is "available" per W4.4's resolution: + * `trusted = 1` rows feed the missing-header default set; any row (trusted + * or not) makes a DID acceptable when explicitly requested. */ +async function seedLabeler(did: string, trusted: boolean): Promise { + await testEnv.DB.prepare( + `INSERT INTO labelers (did, endpoint, signing_key, signing_key_id, trusted, added_at, last_resolved_at) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + did, + "https://labeler.example", + "unused-in-tests", + `${did}#atproto_label`, + trusted ? 1 : 0, + NOW.toISOString(), + NOW.toISOString(), + ) + .run(); +} + +async function seedLabelState(opts: { + uri: string; + val: string; + src?: string; + cid?: string | null; + neg?: boolean; + exp?: string; +}): Promise { + const src = opts.src ?? LABELER_DID; + await testEnv.DB.prepare( + `INSERT INTO label_state + (src, uri, val, cid, neg, cts, cts_epoch_ms, exp, exp_epoch_ms, + digest, source_sequence, frame_index, trusted) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + src, + opts.uri, + opts.val, + opts.cid ?? null, + opts.neg ? 1 : 0, + NOW.toISOString(), + NOW.getTime(), + opts.exp ?? null, + opts.exp === undefined ? null : Date.parse(opts.exp), + `digest-${src}-${opts.uri}-${opts.val}`, + 1, + 0, + 1, + ) + .run(); +} + +interface SeedVerificationOpts { + issuerDid?: string; + rkey?: string; + subjectDid?: string; + handle?: string; + displayName?: string; + createdAt?: string; + expiresAt?: string | null; + indexedAt?: string; + tombstoned?: boolean; +} + +async function seedVerification(opts: SeedVerificationOpts = {}): Promise { + await testEnv.DB.prepare( + `INSERT INTO publisher_verifications + (issuer_did, rkey, subject_did, subject_handle, subject_display_name, + created_at, expires_at, record_blob, signature_metadata, verified_at, + indexed_at, tombstoned_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + opts.issuerDid ?? DID_B, + opts.rkey ?? "3kaverifyrkey000", + opts.subjectDid ?? DID_A, + opts.handle ?? "publisher.example", + opts.displayName ?? "Publisher", + opts.createdAt ?? NOW.toISOString(), + opts.expiresAt === undefined ? null : opts.expiresAt, + new Uint8Array([0xc1, 0xc2, 0xc3]), + JSON.stringify({ cid: "bafyverif" }), + NOW.toISOString(), + opts.indexedAt ?? NOW.toISOString(), + opts.tombstoned ? NOW.toISOString() : null, + ) + .run(); +} + +/** An expiry that is an hour in the past as an instant, rendered with a + * +14:00 offset so its raw string compares lexically AFTER the current UTC + * timestamp — the case that text comparison gets wrong. */ +function offsetExpiredExp(): string { + const instant = Date.now() - 60 * 60 * 1000; + const local = new Date(instant + 14 * 60 * 60 * 1000); + return local.toISOString().replace(/\.\d{3}Z$/, "+14:00"); +} + describe("getPackage", () => { it("returns the packageView envelope for an indexed package", async () => { await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); @@ -195,6 +353,225 @@ describe("getPackage", () => { const body = (await res.json()) as Record; expect(body).not.toHaveProperty("latestVersion"); }); + + it("404s (redacted) when a default-accepted source's !takedown is active", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(404); + const body = (await res.json()) as { error: string }; + expect(body.error).toBe("NotFound"); + }); + + it("redacts even when the takedown sits beyond the 64-label wire cap", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + for (let i = 0; i < 64; i++) { + await seedLabelState({ uri: packageUri("demo"), val: `note-${i}` }); + } + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(404); + }); + + it("caps wire labels at 64 without affecting the response status", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + for (let i = 0; i < 70; i++) { + await seedLabelState({ uri: packageUri("demo"), val: `note-${i}` }); + } + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { labels: unknown[] }; + expect(body.labels).toHaveLength(64); + }); + + it("returns 200 with the label present when the same takedown comes from a non-redact accepted source", async () => { + await seedLabeler(UNTRUSTED_LABELER_DID, false); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown", src: UNTRUSTED_LABELER_DID }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + { headers: { "atproto-accept-labelers": UNTRUSTED_LABELER_DID } }, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { labels: Array<{ src: string; val: string }> }; + expect(body.labels).toContainEqual( + expect.objectContaining({ src: UNTRUSTED_LABELER_DID, val: "!takedown" }), + ); + }); + + it("hydrates publisher-DID labels alongside package-URI labels", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: DID_A, val: "low-quality" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { labels: Array<{ src: string; uri: string; val: string }> }; + expect(body.labels).toContainEqual( + expect.objectContaining({ src: LABELER_DID, uri: DID_A, val: "low-quality" }), + ); + }); + + it("only hydrates unexpired, non-negated labels", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "low-quality" }); + await seedLabelState({ + uri: packageUri("demo"), + val: "broken-release", + exp: new Date(NOW.getTime() - 60_000).toISOString(), + }); + await seedLabelState({ uri: packageUri("demo"), val: "obfuscated-code", neg: true }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { labels: Array<{ val: string }> }; + expect(body.labels.map((l) => l.val)).toEqual(["low-quality"]); + }); + + it("sets atproto-content-labelers when the request used the default policy", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.headers.get("atproto-content-labelers")).toBe(`${LABELER_DID};redact`); + }); + + it("hydrated labels carry only the label spec's optional fields (no sig, no neg)", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "low-quality", cid: null }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + const body = (await res.json()) as { labels: Array> }; + expect(body.labels).toHaveLength(1); + const label = body.labels[0]!; + expect(label).toMatchObject({ src: LABELER_DID, uri: packageUri("demo"), val: "low-quality" }); + expect(label).toHaveProperty("cts"); + expect(label).not.toHaveProperty("cid"); + expect(label).not.toHaveProperty("exp"); + expect(label).not.toHaveProperty("sig"); + expect(label).not.toHaveProperty("neg"); + expect(label).not.toHaveProperty("ver"); + }); +}); + +describe("getPublisher", () => { + it("returns the publisherView envelope for an indexed publisher", async () => { + await seedPublisher({ + displayName: "Acme Plugin Co.", + description: "We make plugins", + url: "https://acme.test", + contact: [{ kind: "security", email: "security@acme.test" }], + updatedAt: NOW.toISOString(), + }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body).toMatchObject({ + uri: publisherUri(), + cid: "bafypub", + did: DID_A, + indexedAt: NOW.toISOString(), + labels: [], + }); + expect(body).not.toHaveProperty("slug"); + const profile = body["profile"] as Record; + expect(profile["$type"]).toBe(NSID.publisherProfile); + expect(profile["displayName"]).toBe("Acme Plugin Co."); + expect(profile["contact"]).toEqual([{ kind: "security", email: "security@acme.test" }]); + }); + + it("omits optional profile fields the publisher did not set", async () => { + await seedPublisher({ + displayName: "Minimal", + description: null, + url: null, + contact: null, + updatedAt: null, + }); + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + const body = (await res.json()) as { profile: Record }; + expect(body.profile).not.toHaveProperty("description"); + expect(body.profile).not.toHaveProperty("url"); + expect(body.profile).not.toHaveProperty("contact"); + expect(body.profile).not.toHaveProperty("updatedAt"); + }); + + it("returns 404 NotFound when no publisher is indexed", async () => { + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_B}`); + expect(res.status).toBe(404); + const body = (await res.json()) as { error: string }; + expect(body.error).toBe("NotFound"); + }); + + it("returns 400 InvalidRequest when did is missing", async () => { + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}`); + expect(res.status).toBe(400); + }); + + it("sets Cache-Control: private, no-store on success", async () => { + await seedPublisher(); + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + expect(res.headers.get("cache-control")).toBe("private, no-store"); + }); + + it("404s (redacted) when a default-accepted source's !takedown is active on the publisher DID", async () => { + await seedLabeler(LABELER_DID, true); + await seedPublisher(); + await seedLabelState({ uri: DID_A, val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + expect(res.status).toBe(404); + }); + + it("404s (redacted) when the !takedown is active on the profile record URI, not the DID", async () => { + await seedLabeler(LABELER_DID, true); + await seedPublisher(); + await seedLabelState({ uri: publisherUri(), val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + expect(res.status).toBe(404); + }); + + it("hydrates labels on the publisher profile record URI", async () => { + await seedLabeler(LABELER_DID, true); + await seedPublisher(); + await seedLabelState({ uri: publisherUri(), val: "unverified-publisher" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisher}?did=${DID_A}`); + expect(res.status).toBe(200); + const body = (await res.json()) as { labels: Array<{ src: string; uri: string; val: string }> }; + expect(body.labels).toContainEqual( + expect.objectContaining({ + src: LABELER_DID, + uri: publisherUri(), + val: "unverified-publisher", + }), + ); + }); }); describe("listReleases", () => { @@ -265,6 +642,88 @@ describe("listReleases", () => { ); expect(res.status).toBe(400); }); + + it("omits a release redacted by a default-accepted source's !takedown", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedRelease({ version: "1.0.0" }); + await seedRelease({ version: "2.0.0" }); + await seedLabelState({ uri: releaseUri("demo", "2.0.0"), val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorListReleases}?did=${DID_A}&package=demo`, + ); + const body = (await res.json()) as { releases: Array<{ version: string }> }; + expect(body.releases.map((r) => r.version)).toEqual(["1.0.0"]); + }); + + it("returns a blocked-but-not-redacted release with its labels intact", async () => { + await seedLabeler(UNTRUSTED_LABELER_DID, false); + await seedPackage({ slug: "demo" }); + await seedRelease({ version: "1.0.0" }); + await seedLabelState({ + uri: releaseUri("demo", "1.0.0"), + val: "malware", + src: UNTRUSTED_LABELER_DID, + }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorListReleases}?did=${DID_A}&package=demo`, + { headers: { "atproto-accept-labelers": UNTRUSTED_LABELER_DID } }, + ); + const body = (await res.json()) as { + releases: Array<{ version: string; labels: Array<{ val: string }> }>; + }; + expect(body.releases.map((r) => r.version)).toEqual(["1.0.0"]); + expect(body.releases[0]!.labels).toContainEqual( + expect.objectContaining({ src: UNTRUSTED_LABELER_DID, val: "malware" }), + ); + }); + + it("404s when the parent package carries a redacted takedown, matching getPackage", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedRelease({ version: "1.0.0" }); + await seedRelease({ version: "2.0.0" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorListReleases}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + const body = (await res.json()) as { error: string }; + expect(body.error).toBe("NotFound"); + }); + + it("pages correctly when a page has redacted omissions", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedRelease({ version: "1.1.0" }); + await seedRelease({ version: "1.2.0" }); + await seedRelease({ version: "1.3.0" }); + await seedRelease({ version: "1.4.0" }); + // Redact the last item of what would otherwise be the first raw page + // ([1.4.0, 1.3.0] at limit=2) — the cursor must still derive from + // 1.3.0 (the last row actually fetched), not 1.4.0 (the last row + // that survived the redaction filter). + await seedLabelState({ uri: releaseUri("demo", "1.3.0"), val: "!takedown" }); + + const first = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorListReleases}?did=${DID_A}&package=demo&limit=2`, + ); + const firstBody = (await first.json()) as { + releases: Array<{ version: string }>; + cursor: string; + }; + expect(firstBody.releases.map((r) => r.version)).toEqual(["1.4.0"]); + expect(firstBody.cursor).toBeTruthy(); + + const second = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorListReleases}?did=${DID_A}&package=demo&limit=2&cursor=${encodeURIComponent(firstBody.cursor)}`, + ); + const secondBody = (await second.json()) as { releases: Array<{ version: string }> }; + expect(secondBody.releases.map((r) => r.version)).toEqual(["1.2.0", "1.1.0"]); + }); }); describe("getLatestRelease", () => { @@ -315,6 +774,191 @@ describe("getLatestRelease", () => { ); expect(res.status).toBe(404); }); + + it("skips a hard-blocked highest release in favour of the next-best one", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "2.0.0" }); + await seedRelease({ version: "1.0.0" }); + await seedRelease({ version: "2.0.0" }); + await seedLabelState({ uri: releaseUri("demo", "2.0.0"), val: "malware" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body["version"]).toBe("1.0.0"); + }); + + it("returns 404 when every release is hard-blocked", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0" }); + await seedLabelState({ uri: releaseUri("demo", "1.0.0"), val: "malware" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); + + it("respects packages.latest_version via the fast path when the accepted policy is empty, even with block labels elsewhere", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0" }); + await seedRelease({ version: "2.0.0" }); + // A hard-block label exists, and would exclude 2.0.0 under the + // authoritative path — but the empty explicit header disables + // enforcement entirely, so this should never be consulted. + await seedLabelState({ uri: releaseUri("demo", "2.0.0"), val: "malware" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + { headers: { "atproto-accept-labelers": "" } }, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body["version"]).toBe("1.0.0"); + }); + + it("returns 404 on a package-cascade block even when the release itself carries no label", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); + + it("ignores a release-scope label whose CID no longer matches the current release", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: "bafcurrent" }); + await seedLabelState({ + uri: releaseUri("demo", "1.0.0"), + val: "malware", + cid: "bafstale", + }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body["version"]).toBe("1.0.0"); + }); + + // §10 override rule: a source's exact-CID assessment-passed + + // assessment-overridden pair suppresses that source's automated blocks for + // the release, mirroring `evaluateReleaseModerationCore`. The enforcement + // SQL must not exclude a release whose only live block is so suppressed. + const OVERRIDE_CID = "bafoverridecid"; + + it("includes a release whose re-issued automated block is suppressed by a same-source exact-CID override pair", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + const uri = releaseUri("demo", "1.0.0"); + await seedLabelState({ uri, val: "assessment-passed", cid: OVERRIDE_CID }); + await seedLabelState({ uri, val: "assessment-overridden", cid: OVERRIDE_CID }); + await seedLabelState({ uri, val: "malware", cid: OVERRIDE_CID }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body["version"]).toBe("1.0.0"); + }); + + it("includes a release carrying an override pair and no block", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + const uri = releaseUri("demo", "1.0.0"); + await seedLabelState({ uri, val: "assessment-passed", cid: OVERRIDE_CID }); + await seedLabelState({ uri, val: "assessment-overridden", cid: OVERRIDE_CID }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as Record; + expect(body["version"]).toBe("1.0.0"); + }); + + it("excludes a release with an automated block and no override pair", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + await seedLabelState({ uri: releaseUri("demo", "1.0.0"), val: "malware", cid: OVERRIDE_CID }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); + + it("still excludes when the override pair is from a different source than the block", async () => { + const OTHER_LABELER_DID = "did:web:other-labels.example"; + await seedLabeler(LABELER_DID, true); + await seedLabeler(OTHER_LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + const uri = releaseUri("demo", "1.0.0"); + await seedLabelState({ uri, val: "malware", cid: OVERRIDE_CID }); + await seedLabelState({ + uri, + val: "assessment-passed", + cid: OVERRIDE_CID, + src: OTHER_LABELER_DID, + }); + await seedLabelState({ + uri, + val: "assessment-overridden", + cid: OVERRIDE_CID, + src: OTHER_LABELER_DID, + }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); + + it("still excludes when the override pair CID does not match the release", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + const uri = releaseUri("demo", "1.0.0"); + await seedLabelState({ uri, val: "malware", cid: OVERRIDE_CID }); + await seedLabelState({ uri, val: "assessment-passed", cid: "bafstalecid" }); + await seedLabelState({ uri, val: "assessment-overridden", cid: "bafstalecid" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); + + it("never suppresses a security-yanked release block, even with an override pair", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", latestVersion: "1.0.0" }); + await seedRelease({ version: "1.0.0", cid: OVERRIDE_CID }); + const uri = releaseUri("demo", "1.0.0"); + await seedLabelState({ uri, val: "assessment-passed", cid: OVERRIDE_CID }); + await seedLabelState({ uri, val: "assessment-overridden", cid: OVERRIDE_CID }); + // `security-yanked` is issued CID-less (policy cidRule: forbidden). + await seedLabelState({ uri, val: "security-yanked" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetLatestRelease}?did=${DID_A}&package=demo`, + ); + expect(res.status).toBe(404); + }); }); describe("searchPackages", () => { @@ -360,6 +1004,104 @@ describe("searchPackages", () => { expect(overlap).toEqual([]); }); + it("hides a package with an active default-accepted !takedown label", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "risky" }); + await seedPackage({ slug: "safe" }); + await seedLabelState({ uri: packageUri("risky"), val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["safe"]); + }); + + it("does not hide a package whose takedown expired, even when the raw exp string compares lexically after now", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "recovered" }); + await seedLabelState({ + uri: packageUri("recovered"), + val: "!takedown", + exp: offsetExpiredExp(), + }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["recovered"]); + }); + + it("ignores blocking labels from a source outside the accepted policy", async () => { + // Configured but untrusted — the missing-header default only picks up + // `trusted = 1` labelers, so this source's label doesn't enforce. + await seedLabeler(LABELER_DID, false); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["demo"]); + }); + + it("enforces a takedown from an explicitly accepted untrusted source", async () => { + await seedLabeler(UNTRUSTED_LABELER_DID, false); + await seedPackage({ slug: "demo" }); + await seedLabelState({ + uri: packageUri("demo"), + val: "!takedown", + src: UNTRUSTED_LABELER_DID, + }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`, { + headers: { "atproto-accept-labelers": UNTRUSTED_LABELER_DID }, + }); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual([]); + }); + + it("disables enforcement and skips hydration with an explicit empty header", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`, { + headers: { "atproto-accept-labelers": "" }, + }); + const body = (await res.json()) as { packages: Array<{ slug: string; labels: unknown[] }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["demo"]); + expect(body.packages[0]!.labels).toEqual([]); + }); + + it("does not hide a package whose label is CID-bound to a stale CID", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo", cid: "bafcurrent" }); + await seedLabelState({ uri: packageUri("demo"), val: "!takedown", cid: "bafstale" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["demo"]); + }); + + it("hides every package under a publisher DID with an active takedown", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ did: DID_A, slug: "a1" }); + await seedPackage({ did: DID_A, slug: "a2" }); + await seedPackage({ did: DID_B, slug: "b1" }); + await seedLabelState({ uri: DID_A, val: "!takedown" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual(["b1"]); + }); + + it("hides a package with an active publisher-compromised label", async () => { + await seedLabeler(LABELER_DID, true); + await seedPackage({ slug: "demo" }); + await seedLabelState({ uri: packageUri("demo"), val: "publisher-compromised" }); + + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorSearchPackages}`); + const body = (await res.json()) as { packages: Array<{ slug: string }> }; + expect(body.packages.map((p) => p.slug)).toEqual([]); + }); + it("doesn't blow up on FTS-unsafe query chars (defensive quoting)", async () => { await seedPackage({ slug: "demo", name: "Demo" }); const res = await SELF.fetch( @@ -512,3 +1254,225 @@ describe("XRPC dispatcher", () => { expect(res.status).toBe(404); }); }); + +describe("XRPC dispatcher — unexpected policy-resolution failure", () => { + // A non-XRPC throw before dispatch (here: a D1 error because `labelers` + // is gone) must still take the CORS + `no-store` wrapper, not escape to + // workerd's bare 500. Capture the table's schema so the shared beforeEach + // (`DELETE FROM labelers`) keeps working after a test drops it. + let labelersSchema: string; + beforeAll(async () => { + const row = await testEnv.DB.prepare( + "SELECT sql FROM sqlite_master WHERE type='table' AND name='labelers'", + ).first<{ sql: string }>(); + labelersSchema = row!.sql; + }); + afterEach(async () => { + const exists = await testEnv.DB.prepare( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='labelers'", + ).first(); + if (!exists) await testEnv.DB.prepare(labelersSchema).run(); + }); + + it("wraps a non-XRPC failure in a 500 carrying CORS + no-store and no leaked internals", async () => { + await seedPackage({ slug: "demo" }); + // Resolving the default policy runs `SELECT did FROM labelers`; dropping + // the table makes that throw a non-XRPC D1 error before dispatch. + await testEnv.DB.prepare("DROP TABLE labelers").run(); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPackage}?did=${DID_A}&slug=demo`, + ); + expect(res.status).toBe(500); + expect(res.headers.get("cache-control")).toBe("private, no-store"); + expect(res.headers.get("access-control-allow-origin")).toBe("*"); + const body = (await res.json()) as { error: string; message?: string }; + expect(body.error).toBe("InternalServerError"); + // No internal detail (SQL text, table name, stack) leaks to the client. + expect(JSON.stringify(body)).not.toMatch(/labelers|no such table|SQL|SELECT/i); + }); +}); + +describe("sync.getRecord — unexpected blob-fetch failure", () => { + // A D1 error inside the CAR passthrough (here: `publishers` is gone) must + // take the CORS + `no-store` wrapper rather than escape to workerd's bare + // 500 — and must NOT inherit the cacheable success header. Capture the + // table's schema so the shared beforeEach (`DELETE FROM publishers`) keeps + // working after a test drops it. + let publishersSchema: string; + beforeAll(async () => { + const row = await testEnv.DB.prepare( + "SELECT sql FROM sqlite_master WHERE type='table' AND name='publishers'", + ).first<{ sql: string }>(); + publishersSchema = row!.sql; + }); + afterEach(async () => { + const exists = await testEnv.DB.prepare( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='publishers'", + ).first(); + if (!exists) await testEnv.DB.prepare(publishersSchema).run(); + }); + + it("wraps a D1 failure in a 500 carrying CORS + no-store and no leaked internals", async () => { + // The publisher-profile blob read runs `SELECT record_blob FROM publishers`; + // dropping the table makes that throw a non-XRPC D1 error. + await testEnv.DB.prepare("DROP TABLE publishers").run(); + + const res = await SELF.fetch( + `https://test/xrpc/com.atproto.sync.getRecord?did=${DID_A}&collection=${NSID.publisherProfile}&rkey=self`, + ); + expect(res.status).toBe(500); + expect(res.headers.get("cache-control")).toBe("private, no-store"); + expect(res.headers.get("access-control-allow-origin")).toBe("*"); + const body = (await res.json()) as { error: string; message?: string }; + expect(body.error).toBe("InternalServerError"); + // No internal detail (SQL text, table name, stack) leaks to the client. + expect(JSON.stringify(body)).not.toMatch(/publishers|no such table|SQL|SELECT/i); + }); +}); + +describe("getPublisherVerification", () => { + it("returns the verification claims naming a DID as subject, newest first", async () => { + await seedVerification({ + issuerDid: DID_B, + rkey: "3kolder000000000", + createdAt: "2026-01-01T00:00:00.000Z", + handle: "pub.example", + displayName: "Pub", + }); + await seedVerification({ + issuerDid: DID_B, + rkey: "3knewer000000000", + createdAt: "2026-03-01T00:00:00.000Z", + handle: "pub.example", + displayName: "Pub", + expiresAt: "2027-01-01T00:00:00.000Z", + }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { + did: string; + labels: unknown[]; + verifications: Array>; + }; + expect(body.did).toBe(DID_A); + expect(body.labels).toEqual([]); + expect(body.verifications).toHaveLength(2); + // created_at DESC — the March claim comes first. + expect(body.verifications[0]).toMatchObject({ + issuer: DID_B, + handle: "pub.example", + displayName: "Pub", + createdAt: "2026-03-01T00:00:00.000Z", + expiresAt: "2027-01-01T00:00:00.000Z", + indexedAt: NOW.toISOString(), + }); + // The older claim has no expiry — expiresAt is omitted, not null. + expect(body.verifications[1]).not.toHaveProperty("expiresAt"); + }); + + it("returns an empty verifications array for a DID with no claims (not a 404)", async () => { + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { did: string; verifications: unknown[] }; + expect(body.verifications).toEqual([]); + }); + + it("excludes tombstoned claims", async () => { + await seedVerification({ rkey: "3klive0000000000" }); + await seedVerification({ rkey: "3kdead0000000000", tombstoned: true }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + const body = (await res.json()) as { verifications: unknown[] }; + expect(body.verifications).toHaveLength(1); + }); + + it("returns 400 InvalidRequest when the did param is missing", async () => { + const res = await SELF.fetch(`https://test/xrpc/${NSID.aggregatorGetPublisherVerification}`); + expect(res.status).toBe(400); + }); + + it("sets Cache-Control: private, no-store on success", async () => { + await seedVerification(); + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + expect(res.headers.get("cache-control")).toBe("private, no-store"); + }); + + it("404s (redacted) when a default-accepted source's !takedown covers the DID", async () => { + await seedLabeler(LABELER_DID, true); + await seedVerification(); + await seedLabelState({ uri: DID_A, val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + expect(res.status).toBe(404); + const body = (await res.json()) as { error: string }; + expect(body.error).toBe("NotFound"); + }); + + it("serves the view unfiltered for a blank accept-labelers header despite an active takedown", async () => { + await seedLabeler(LABELER_DID, true); + await seedVerification(); + await seedLabelState({ uri: DID_A, val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + { headers: { "atproto-accept-labelers": "" } }, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { verifications: unknown[]; labels: unknown[] }; + expect(body.verifications).toHaveLength(1); + expect(body.labels).toEqual([]); + }); + + it("drops a claim whose issuer is redacted for a default-policy caller, keeping the others", async () => { + const redactedIssuer = "did:plc:issuertakedown00000000000"; + const okIssuer = "did:plc:issuerok0000000000000000"; + await seedLabeler(LABELER_DID, true); + await seedVerification({ issuerDid: redactedIssuer, rkey: "3kredacted000000" }); + await seedVerification({ issuerDid: okIssuer, rkey: "3kok000000000000" }); + // Takedown is on the ISSUER's DID, not the subject's. + await seedLabelState({ uri: redactedIssuer, val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { verifications: Array<{ issuer: string }> }; + // The subject is not redacted, so the view is served; only the + // redacted-issuer claim is filtered out. + expect(body.verifications).toHaveLength(1); + expect(body.verifications[0]?.issuer).toBe(okIssuer); + }); + + it("keeps a redacted-issuer claim for a blank accept-labelers caller (internal unfiltered read)", async () => { + const redactedIssuer = "did:plc:issuertakedown00000000000"; + const okIssuer = "did:plc:issuerok0000000000000000"; + await seedLabeler(LABELER_DID, true); + await seedVerification({ issuerDid: redactedIssuer, rkey: "3kredacted000000" }); + await seedVerification({ issuerDid: okIssuer, rkey: "3kok000000000000" }); + await seedLabelState({ uri: redactedIssuer, val: "!takedown" }); + + const res = await SELF.fetch( + `https://test/xrpc/${NSID.aggregatorGetPublisherVerification}?did=${DID_A}`, + { headers: { "atproto-accept-labelers": "" } }, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { verifications: Array<{ issuer: string }> }; + // Empty accepted set → nothing redacted → the labeler sees both claims. + expect(body.verifications).toHaveLength(2); + expect(body.verifications.map((v) => v.issuer).toSorted()).toEqual( + [okIssuer, redactedIssuer].toSorted(), + ); + }); +}); diff --git a/apps/aggregator/test/records-consumer.test.ts b/apps/aggregator/test/records-consumer.test.ts index e5609426c4..f5980d3b31 100644 --- a/apps/aggregator/test/records-consumer.test.ts +++ b/apps/aggregator/test/records-consumer.test.ts @@ -543,15 +543,20 @@ class MapDidDocCache implements DidDocCache { class FakeMessage implements MessageController { acked = 0; retried = 0; + retryDelaySeconds: number | undefined = undefined; ack() { this.acked += 1; } - retry() { + retry(options?: { delaySeconds?: number }) { this.retried += 1; + this.retryDelaySeconds = options?.delaySeconds; } } -function buildDeps(opts: { fetch: typeof fetch }): { +function buildDeps(opts: { + fetch: typeof fetch; + resolveHostname?: ConsumerDeps["resolveHostname"]; +}): { deps: ConsumerDeps; cache: MapDidDocCache; } { @@ -564,7 +569,15 @@ function buildDeps(opts: { fetch: typeof fetch }): { now: () => NOW, }); return { - deps: { db: testEnv.DB, resolver, fetch: opts.fetch, now: () => NOW }, + deps: { + db: testEnv.DB, + resolver, + fetch: opts.fetch, + // pds.test.example resolves public so the SSRF egress guard in + // `fetchAndVerifyRecord` passes and these tests reach the fetch stub. + resolveHostname: opts.resolveHostname ?? (async () => ["93.184.216.34"]), + now: () => NOW, + }, cache, }; } @@ -608,6 +621,8 @@ describe("processMessage dispatcher", () => { expect(msg.retried).toBe(1); expect(msg.acked).toBe(0); expect(await deadLetterCount()).toBe(0); + // A 5xx retries immediately — the propagation delay is DNS-only. + expect(msg.retryDelaySeconds).toBeUndefined(); }); it("retries on a network error", async () => { @@ -621,6 +636,45 @@ describe("processMessage dispatcher", () => { expect(msg.retried).toBe(1); expect(await deadLetterCount()).toBe(0); + expect(msg.retryDelaySeconds).toBeUndefined(); + }); + + it("retries when the SSRF host resolver fails (transient DoH outage, not dead-lettered)", async () => { + const { deps, cache } = buildDeps({ + // Reached only if the resolver somehow passed — it must not. + fetch: () => Promise.reject(new Error("fetch must not run when resolution fails")), + resolveHostname: () => Promise.reject(new Error("DoH 503")), + }); + cache.seed(DID_A); + const msg = new FakeMessage(); + + await processMessage(jobFor(DID_A, NSID.packageProfile, "demo"), msg, deps); + + expect(msg.retried).toBe(1); + expect(msg.acked).toBe(0); + expect(await deadLetterCount()).toBe(0); + // Re-delivery is delayed so retries span the DNS-propagation window + // instead of burning max_retries before the record's host resolves. + expect(msg.retryDelaySeconds).toBeGreaterThan(0); + }); + + it("acks and dead-letters PDS_ADDRESS_BLOCKED when the endpoint resolves to a private address", async () => { + const { deps, cache } = buildDeps({ + fetch: () => Promise.reject(new Error("fetch must not run for a blocked address")), + resolveHostname: async () => ["10.0.0.5"], + }); + cache.seed(DID_A); + const msg = new FakeMessage(); + + await processMessage(jobFor(DID_A, NSID.packageProfile, "demo"), msg, deps); + + expect(msg.acked).toBe(1); + expect(msg.retried).toBe(0); + expect(await deadLetterCount()).toBe(1); + const row = await testEnv.DB.prepare(`SELECT reason FROM dead_letters`).first<{ + reason: string; + }>(); + expect(row?.reason).toBe("PDS_ADDRESS_BLOCKED"); }); it("forensics + acks on garbage CAR bytes (verifyRecord rejects → INVALID_PROOF)", async () => { diff --git a/apps/aggregator/test/request-policy.test.ts b/apps/aggregator/test/request-policy.test.ts new file mode 100644 index 0000000000..b791e2567a --- /dev/null +++ b/apps/aggregator/test/request-policy.test.ts @@ -0,0 +1,118 @@ +/** + * Request-scoped labeler-policy resolution (W4.4): the deployment default + * when `atproto-accept-labelers` is absent, explicit-header parsing and + * availability filtering, and the response-side `atproto-content-labelers` / + * CORS surface `router.ts` builds around it. Handlers don't yet act on the + * resolved policy (W4.5/W4.6) — these tests only exercise resolution and + * the header contract, via `getPackage` as a cheap endpoint. + */ + +import { NSID } from "@emdash-cms/registry-lexicons"; +import { applyD1Migrations, env, SELF } from "cloudflare:test"; +import { beforeAll, beforeEach, describe, expect, it } from "vitest"; + +interface TestEnv { + DB: D1Database; + TEST_MIGRATIONS: Parameters[1]; +} +const testEnv = env as unknown as TestEnv; + +const NOW = new Date("2026-07-11T12:00:00.000Z"); +const TRUSTED_DID = "did:web:trusted-labeler.example"; +const UNTRUSTED_DID = "did:web:untrusted-labeler.example"; +const UNKNOWN_DID = "did:web:unknown-labeler.example"; +// No package is seeded for this DID/slug; the handler 404s, but resolution +// and header-setting happen in `handleXrpc` regardless of the response. +const GET_PACKAGE_URL = `https://test/xrpc/${NSID.aggregatorGetPackage}?did=did:plc:read000000000000000000aa&slug=missing`; + +beforeAll(async () => { + await applyD1Migrations(testEnv.DB, testEnv.TEST_MIGRATIONS); +}); + +beforeEach(async () => { + await testEnv.DB.prepare("DELETE FROM labelers").run(); +}); + +async function seedLabeler(did: string, trusted: boolean): Promise { + await testEnv.DB.prepare( + `INSERT INTO labelers (did, endpoint, signing_key, signing_key_id, trusted, added_at, last_resolved_at) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .bind( + did, + "https://labeler.example", + "unused-in-tests", + `${did}#atproto_label`, + trusted ? 1 : 0, + NOW.toISOString(), + NOW.toISOString(), + ) + .run(); +} + +describe("request labeler policy", () => { + it("defaults to every trusted labeler with redact:true when the header is missing", async () => { + await seedLabeler(TRUSTED_DID, true); + await seedLabeler(UNTRUSTED_DID, false); + + const res = await SELF.fetch(GET_PACKAGE_URL); + expect(res.headers.get("atproto-content-labelers")).toBe(`${TRUSTED_DID};redact`); + }); + + it("disables the default set and omits the response header when the header is present but empty", async () => { + await seedLabeler(TRUSTED_DID, true); + + const res = await SELF.fetch(GET_PACKAGE_URL, { + headers: { "atproto-accept-labelers": "" }, + }); + expect(res.headers.has("atproto-content-labelers")).toBe(false); + }); + + it("400s with an InvalidRequest envelope on malformed header syntax", async () => { + const res = await SELF.fetch(GET_PACKAGE_URL, { + headers: { "atproto-accept-labelers": "not-a-valid-did" }, + }); + expect(res.status).toBe(400); + const body = (await res.json()) as { error: string; message?: string }; + expect(body.error).toBe("InvalidRequest"); + expect(typeof body.message).toBe("string"); + }); + + it("merges repeated DIDs in the explicit header, OR-ing redact", async () => { + await seedLabeler(TRUSTED_DID, true); + + const res = await SELF.fetch(GET_PACKAGE_URL, { + headers: { "atproto-accept-labelers": `${TRUSTED_DID}, ${TRUSTED_DID};redact` }, + }); + expect(res.headers.get("atproto-content-labelers")).toBe(`${TRUSTED_DID};redact`); + }); + + it("omits an unavailable DID from the explicit header", async () => { + await seedLabeler(TRUSTED_DID, true); + + const res = await SELF.fetch(GET_PACKAGE_URL, { + headers: { "atproto-accept-labelers": `${UNKNOWN_DID}, ${TRUSTED_DID}` }, + }); + expect(res.headers.get("atproto-content-labelers")).toBe(TRUSTED_DID); + }); + + it("accepts and echoes an explicit untrusted-but-configured DID", async () => { + await seedLabeler(UNTRUSTED_DID, false); + + const res = await SELF.fetch(GET_PACKAGE_URL, { + headers: { "atproto-accept-labelers": UNTRUSTED_DID }, + }); + expect(res.headers.get("atproto-content-labelers")).toBe(UNTRUSTED_DID); + }); + + it("lists the right allow/expose headers on the OPTIONS preflight", async () => { + const res = await SELF.fetch(GET_PACKAGE_URL, { method: "OPTIONS" }); + expect(res.status).toBe(204); + expect(res.headers.get("access-control-allow-headers")).toBe( + "content-type, atproto-accept-labelers", + ); + expect(res.headers.get("access-control-expose-headers")).toBe( + "atproto-content-labelers, content-language", + ); + }); +}); diff --git a/apps/aggregator/test/smoke.test.ts b/apps/aggregator/test/smoke.test.ts index c02ab33165..5ef081cea0 100644 --- a/apps/aggregator/test/smoke.test.ts +++ b/apps/aggregator/test/smoke.test.ts @@ -84,6 +84,14 @@ describe("aggregator scaffold smoke test", () => { expect(result?.slug).toBe("searchable"); }); + it("exposes a queryable labelers table under the name src queries", async () => { + // `0001_init.sql` creates the table as `labellers`; `0004` renames it to + // `labelers`, the spelling every query in `src/` uses. A fresh install + // must land on `labelers`. + const result = await testEnv.DB.prepare(`SELECT did FROM labelers`).all(); + expect(result.success).toBe(true); + }); + it("rejects a release whose did/package does not match an existing profile (FK)", async () => { // Releases reference packages via composite FK; SQLite enforces this when // FK checks are enabled. workers-pool's miniflare D1 has FK checks on by diff --git a/apps/aggregator/test/views.test.ts b/apps/aggregator/test/views.test.ts new file mode 100644 index 0000000000..dcebe992bd --- /dev/null +++ b/apps/aggregator/test/views.test.ts @@ -0,0 +1,66 @@ +/** + * View mapper unit tests. + * + * `capLabels` is exercised through `releaseView` (its only caller shape): + * the lexicon caps a view's `labels` at `LABELS_MAX_LENGTH`, and the trim + * must be enforcement-preserving — a plain slice in hydration order can drop + * the only hard block, and a client evaluating the truncated view would then + * treat the release as installable. + */ + +import { describe, expect, it } from "vitest"; + +import { LABELS_MAX_LENGTH, type LabelView } from "../src/routes/xrpc/label-enforcement.js"; +import { type ReleaseRow, releaseView } from "../src/routes/xrpc/views.js"; + +const NOW = "2026-05-10T12:00:00.000Z"; + +function makeReleaseRow(): ReleaseRow { + return { + did: "did:plc:abc", + package: "demo", + version: "1.0.0", + rkey: "demo:1.0.0", + artifacts: JSON.stringify({ package: { url: "https://x.test/d.tgz", checksum: "bsha256" } }), + requires: null, + suggests: null, + emdash_extension: JSON.stringify({ declaredAccess: {} }), + repo_url: null, + signature_metadata: JSON.stringify({ cid: "bafrel" }), + verified_at: NOW, + indexed_at: NOW, + }; +} + +function label(val: string, index: number): LabelView { + return { src: "did:web:labeler.example", uri: `at://did:plc:abc/x/${index}`, val, cts: NOW }; +} + +describe("releaseView label cap (enforcement-preserving)", () => { + it("returns labels unchanged when at or under the cap", () => { + const labels = [label("low-quality", 0), label("malware", 1), label("suspicious-code", 2)]; + const view = releaseView(makeReleaseRow(), labels); + expect(view.labels).toEqual(labels); + }); + + it("keeps the sole hard block even when it sorts last in hydration order", () => { + const labels: LabelView[] = []; + for (let i = 0; i < LABELS_MAX_LENGTH; i++) labels.push(label("low-quality", i)); + labels.push(label("malware", LABELS_MAX_LENGTH)); + expect(labels).toHaveLength(LABELS_MAX_LENGTH + 1); + + const view = releaseView(makeReleaseRow(), labels); + expect(view.labels).toHaveLength(LABELS_MAX_LENGTH); + expect(view.labels?.some((l) => l.val === "malware")).toBe(true); + }); + + it("keeps an assessment-state label above informational labels when over the cap", () => { + const labels: LabelView[] = []; + for (let i = 0; i < LABELS_MAX_LENGTH; i++) labels.push(label("low-quality", i)); + labels.push(label("assessment-pending", LABELS_MAX_LENGTH)); + + const view = releaseView(makeReleaseRow(), labels); + expect(view.labels).toHaveLength(LABELS_MAX_LENGTH); + expect(view.labels?.some((l) => l.val === "assessment-pending")).toBe(true); + }); +}); diff --git a/apps/aggregator/vitest.config.ts b/apps/aggregator/vitest.config.ts index c44bb02004..7eb2e844d3 100644 --- a/apps/aggregator/vitest.config.ts +++ b/apps/aggregator/vitest.config.ts @@ -28,6 +28,14 @@ import { defineConfig } from "vitest/config"; const migrationsPath = fileURLToPath(new URL("./migrations", import.meta.url)); const migrations = await readD1Migrations(migrationsPath); +// Frozen copy of the schema `main` shipped (pre `labellers` -> `labelers` +// rename), exposed so the upgrade test can provision a database the way an +// already-deployed instance was before applying the live migration set. +const mainMigrationsPath = fileURLToPath( + new URL("./test/fixtures/main-migrations", import.meta.url), +); +const mainMigrations = await readD1Migrations(mainMigrationsPath); + export default defineConfig({ plugins: [ cloudflareTest({ @@ -35,6 +43,7 @@ export default defineConfig({ miniflare: { bindings: { TEST_MIGRATIONS: migrations, + MAIN_MIGRATIONS: mainMigrations, // Stub admin auth token so tests can exercise the auth-gated // admin routes without needing a real secret in the test // environment. Production deploys pull from diff --git a/apps/aggregator/worker-configuration.d.ts b/apps/aggregator/worker-configuration.d.ts index b16346f3b6..9052a722c4 100644 --- a/apps/aggregator/worker-configuration.d.ts +++ b/apps/aggregator/worker-configuration.d.ts @@ -4,16 +4,18 @@ declare namespace Cloudflare { interface GlobalProps { mainModule: typeof import("./src/index"); - durableNamespaces: "RecordsJetstreamDO"; + durableNamespaces: "RecordsJetstreamDO" | "LabelIngestDO"; } interface Env { DB: D1Database; RECORDS_QUEUE: Queue; BACKFILL_QUEUE: Queue; + LABELS_QUEUE: Queue; JETSTREAM_URL: "wss://jetstream2.us-east.bsky.network/subscribe"; RELAY_URL: "https://bsky.network"; ADMIN_TOKEN: string; RECORDS_DO: DurableObjectNamespace; + LABEL_INGEST_DO: DurableObjectNamespace; } } interface Env extends Cloudflare.Env {} diff --git a/apps/aggregator/wrangler.jsonc b/apps/aggregator/wrangler.jsonc index 568db1c79c..58df33f5ab 100644 --- a/apps/aggregator/wrangler.jsonc +++ b/apps/aggregator/wrangler.jsonc @@ -35,6 +35,13 @@ "binding": "BACKFILL_QUEUE", "queue": "emdash-aggregator-backfill", }, + { + // Verified signed labels, one per labeler-stream frame + // element. Producer is LabelIngestDO; consumed by + // labels-consumer.ts into the labels/label_state tables. + "binding": "LABELS_QUEUE", + "queue": "emdash-aggregator-labels", + }, ], "consumers": [ { @@ -86,6 +93,26 @@ "max_batch_timeout": 30, "max_retries": 3, }, + { + // The digest primary key and cts-ordered projection upsert + // make same-batch message order irrelevant, so this mirrors + // the records queue's steady-state tuning. + "queue": "emdash-aggregator-labels", + "max_batch_size": 25, + "max_batch_timeout": 5, + "max_retries": 5, + "dead_letter_queue": "emdash-aggregator-labels-dlq", + }, + { + // Drains the labels DLQ. Same posture as the records DLQ: + // log + write a `dead_letters` row, then ack. No DLQ-of-DLQ; + // workerd drops the message if the D1 write also fails after + // max_retries. + "queue": "emdash-aggregator-labels-dlq", + "max_batch_size": 25, + "max_batch_timeout": 30, + "max_retries": 3, + }, ], }, "durable_objects": { @@ -94,6 +121,10 @@ "name": "RECORDS_DO", "class_name": "RecordsJetstreamDO", }, + { + "name": "LABEL_INGEST_DO", + "class_name": "LabelIngestDO", + }, ], }, "migrations": [ @@ -101,6 +132,10 @@ "tag": "v1", "new_sqlite_classes": ["RecordsJetstreamDO"], }, + { + "tag": "v2", + "new_sqlite_classes": ["LabelIngestDO"], + }, ], "triggers": { // Liveness ping for the records DO. The DO holds Jetstream open and diff --git a/apps/labeler/README.md b/apps/labeler/README.md new file mode 100644 index 0000000000..1b08d4eac3 --- /dev/null +++ b/apps/labeler/README.md @@ -0,0 +1,83 @@ +# @emdash-cms/labeler + +An ATProto labeler service for the emdash plugin registry, deployed as a single Cloudflare Worker. It ingests package-registry records from the ATProto firehose, runs automated moderation assessments on package releases, and emits signed ATProto labels describing each release's eligibility (passed, blocked, warned, pending, error). Consumers such as the emdash aggregator subscribe to those labels and decide what to surface and serve. A React operator console under `/admin` lets human reviewers and admins inspect assessments and act on them. + +## How it fits together + +``` +registry records ──▶ Jetstream firehose ──▶ discovery queue ──▶ automated assessment + │ + ▼ +consumers ◀── subscribeLabels / queryLabels ◀── signed labels ◀── label store +``` + +The Worker consumes the registry's records from the ATProto firehose (Jetstream), enqueues discovered packages and releases, and runs an assessment pipeline over each release. Assessments produce signed labels. Consumers resolve the labeler's DID, discover its endpoint, subscribe to the label stream, verify each label's signature against the public key published in the DID document, and apply the resulting eligibility decisions. + +The concepts behind all of this (DIDs, `did:web`, the DID document, labels, signing, and the XRPC surface) are explained in [docs/atproto.md](docs/atproto.md). The label vocabulary and how labels overlay into an eligibility state are in [docs/moderation-model.md](docs/moderation-model.md). Running the operator console day to day is covered in [docs/operating.md](docs/operating.md). + +## Local development + +Scripts are run through pnpm, filtered to this package: + +```bash +pnpm --filter @emdash-cms/labeler dev # Worker + console dev server (vite dev) +pnpm --filter @emdash-cms/labeler console:dev # console SPA on its own vite config +pnpm --filter @emdash-cms/labeler test # Worker + console test suites +pnpm --filter @emdash-cms/labeler typecheck # tsgo over Worker and console +pnpm --filter @emdash-cms/labeler db:migrate:local # apply D1 migrations to the local database +``` + +The console is not localized (plain English) and there is no dev auth bypass, so operator flows are exercised through real Cloudflare Access in a deployed environment. + +## Deploying + +The Worker checks that its signing keypair matches the public key published in its DID document the first time it signs a label — not at deploy time. `wrangler deploy` only enforces that the `LABEL_SIGNING_PRIVATE_KEY` secret exists, so a mismatched pair deploys cleanly and then fails on the first signing operation. After setting or rotating the key, exercise a signing path (a console mutation, an assessment, or a `queryLabels` re-sign) to confirm the pair is consistent. Work through the checklist in order. + +1. **Create a Cloudflare Access application** covering `/admin/*` on the labeler's domain, with two Access groups (one for admins, one for reviewers) mapped to your identity provider. This produces the application's **AUD tag**. + +2. **Set the operator Access config.** Put the AUD tag, your team domain, and the group names into `OPERATOR_ACCESS_CONFIG` in `wrangler.jsonc` (it ships with a `REPLACE_WITH_ACCESS_APP_AUD_TAG` placeholder): + + ```jsonc + "OPERATOR_ACCESS_CONFIG": "{\"teamDomain\":\"https://your-team.cloudflareaccess.com\",\"audience\":\"\",\"admins\":[\"emdash-labeler-admins\"],\"reviewers\":[\"emdash-labeler-reviewers\"]}" + ``` + +3. **Generate the signing keypair** and install it. The keygen script prints both values in the exact formats the Worker expects; nothing is written to disk. + + ```bash + pnpm --filter @emdash-cms/labeler keygen + echo -n '' | wrangler secret put LABEL_SIGNING_PRIVATE_KEY + # then set LABEL_SIGNING_PUBLIC_KEY in wrangler.jsonc to the printed public key + ``` + + Bump `LABEL_SIGNING_KEY_VERSION` if you are rotating an existing key. + +4. **Run migrations** against the remote D1 database: + + ```bash + pnpm --filter @emdash-cms/labeler db:migrate + ``` + +5. **Deploy.** This builds the console and the Worker, then publishes: + + ```bash + pnpm --filter @emdash-cms/labeler deploy + ``` + +## Configuration reference + +| Key | Kind | Description | +| --------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------- | +| `LABELER_DID` | var | The labeler's DID, `did:web:labels.emdashcms.com`. | +| `LABELER_SERVICE_URL` | var | Service endpoint published in the DID document, `https://labels.emdashcms.com`. | +| `LABEL_SIGNING_PUBLIC_KEY` | var | P-256 public key as a Multikey; published in the DID document's `#atproto_label` method for signature verification. | +| `LABEL_SIGNING_PRIVATE_KEY` | secret | P-256 private key (unpadded base64url of the raw 32-byte scalar) used to sign labels server-side. | +| `LABEL_SIGNING_KEY_VERSION` | var | Signing key version (`v1`); bump when rotating. | +| `OPERATOR_ACCESS_CONFIG` | var | JSON: `teamDomain`, `audience` (Access AUD tag), and `admins` / `reviewers` group names. | + +The private key is declared as a required secret in `wrangler.jsonc`, so `wrangler deploy` refuses to publish if it has not been set on the target Worker. + +## Documents + +- [docs/atproto.md](docs/atproto.md) — ATProto foundations: DIDs, `did:web`, the DID document, labels, signing, and the XRPC surface. +- [docs/operating.md](docs/operating.md) — operator console guide: sign-in, roles, and the review and emergency workflows. +- [docs/moderation-model.md](docs/moderation-model.md) — label vocabulary and how labels evaluate into an eligibility state. diff --git a/apps/labeler/calibration/.gitignore b/apps/labeler/calibration/.gitignore new file mode 100644 index 0000000000..a1e03960fb --- /dev/null +++ b/apps/labeler/calibration/.gitignore @@ -0,0 +1 @@ +runs/ diff --git a/apps/labeler/calibration/README.md b/apps/labeler/calibration/README.md new file mode 100644 index 0000000000..223bc84929 --- /dev/null +++ b/apps/labeler/calibration/README.md @@ -0,0 +1,62 @@ +# Labeler calibration harness (W8.6) + +Runs the **real** moderation adapters (`analyzeCode`, `analyzeImages`) and the +real policy resolver over a fixture corpus against live Workers AI models, so +false positives/negatives can be reviewed before a policy or model change ships. +The model + prompt + schema + policy combination is the unit under evaluation — +nothing here re-implements the eval prompt. + +This is not part of `pnpm test`. It calls real models and costs tokens. + +## Layout + +- `fixtures//` — ported audit fixtures: `manifest.json`, `backend.js`, + optional `icon.png`, and `expected.json` re-expressed in labeler policy terms. +- `models.ts` — the data-driven model matrix (add/remove models here). +- `rest-ai-binding.ts` — `AiBinding` backed by the Workers AI REST endpoint. +- `fixture-loader.ts` — pure conversion (PNG dims, sha256, legacy→labeler map). +- `run.ts` / `run.calibrate.ts` — the sweep; writes artifacts to `runs/`. +- `report.ts` / `report.calibrate.ts` — markdown report + run-vs-run diff. +- `runs/` — recorded artifacts (gitignored; review outputs, not source). + +## Run a sweep + +```bash +CLOUDFLARE_API_TOKEN=$(pnpm --filter @emdash-cms/labeler exec wrangler auth token | tail -1) \ +CLOUDFLARE_ACCOUNT_ID= \ +CALIBRATE_LABEL=baseline \ + pnpm --filter @emdash-cms/labeler calibrate +``` + +The token is read from the environment and never logged or persisted. Artifacts +land in `calibration/runs/-