Skip to content

feat(pds): emit sync 1.1 firehose events#171

Merged
ascorbic merged 1 commit into
mainfrom
fix/sync-1.1-compliance
May 24, 2026
Merged

feat(pds): emit sync 1.1 firehose events#171
ascorbic merged 1 commit into
mainfrom
fix/sync-1.1-compliance

Conversation

@ascorbic
Copy link
Copy Markdown
Owner

Summary

Brings Cirrus's firehose up to AT Protocol sync 1.1 so the bsky.network relay's strict-validation warnings (notably missing prevData field, logged against wisp.mk.gg and other Cirrus hosts) stop firing, and so account/sync state propagates to relays without polling. Diagnosed from public relay log charts shared by @calabro.io on this thread; cross-referenced against the upstream PDS reference impl at bluesky-social/atproto/packages/pds/src/sequencer/events.ts.

What changed on the wire

#commit

  • Adds prevData (the prior commit's MST root CID) so consumers can run the inductive MST-inversion verification.
  • CAR slice now contains both newBlocks and relevantBlocks (the covering proof). Previously Cirrus shipped only the new-this-rev blocks via SELECT bytes FROM blocks WHERE rev = ?.
  • Each ops[] entry gets prev on update/delete (prior record CID), omitted on create.
  • tooBig pinned to false. It was previously carBytes.length > 1_000_000, which is meaningless under the modern spec — tooBig is deprecated.
  • rebase: false retained for back-compat (unchanged).

New event types

  • #sync{did, rev, blocks: CAR with the commit block, time}. Emitted on activation so relays pick up the current state (e.g. after migration import) without diffing.
  • #account{did, active, status?} with status ∈ {takendown, suspended, deleted, deactivated}. Emitted on rpcActivateAccount (active=true) and rpcDeactivateAccount (active=false, status='deactivated'). Both previously just flipped a SQLite flag.

#identity

  • handle is now optional (per spec — presence does not signal a change).
  • rpcEmitIdentityEvent now routes through sequencer.sequenceIdentity instead of writing empty payloads to the DB.

Cursor handling

  • A #info frame with name: "OutdatedCursor" is sent when a client connects with a cursor before the retention window; the stream continues from the oldest available event instead of disconnecting.

Spec caps

  • applyWrites rejects >200 operations.

Implementation notes

  • All four write paths (rpcCreateRecord, rpcDeleteRecord, rpcPutRecord, rpcApplyWrites) refactored from repo.applyWrites(ops, keypair) to repo.formatCommit(ops, keypair) + repo.applyCommit(commit) so the CommitData (with newBlocks, relevantBlocks, etc.) is exposed to the sequencer.
  • Frame encoding generalized to {op: 1, t: '#${event.type}'} — single dispatch path for all four event kinds.
  • broadcastCommitbroadcastEvent (the function was already generic over SeqEvent).
  • prevData is captured as repo.commit.data before applyCommit. The signed commit object's prev field (prior commit CID) is unchanged — that's a different concept, handled by @atproto/repo.

Test plan

  • All 279 pre-existing PDS tests still pass.
  • 10 new tests in firehose.test.ts covering: prevData presence + linkage to prior commit's data CID, ops[].prev for update/delete and absence for create, tooBig=false, the 200-op rejection, #account + #identity + #sync emitted on activate, #account on deactivate, idempotent activate not double-emitting, frame t-tag dispatch for #sync/#account, OutdatedCursor info frame, optional handle on #identity.
  • pnpm exec tsc --noEmit introduces zero new TypeScript errors (16 pre-existing unrelated errors remain).
  • pnpm build succeeds.
  • After merge + deploy, monitor the relay dashboards in the referenced threadwisp.mk.gg's missing prevData field count should drop to zero.

Brings the firehose payload up to the current spec so the bsky.network
relay's strict-validation warnings (notably "missing prevData field")
stop firing against Cirrus hosts, and so account/sync state actually
propagates without polling.

- #commit gets prevData + ops[].prev, tooBig pinned to false, and the
  CAR slice now carries the MST covering proof (newBlocks +
  relevantBlocks) needed for inductive verification.
- Writes go through repo.formatCommit + repo.applyCommit instead of
  applyWrites so the CommitData is available; drops the SQL-by-rev
  block reconstruction.
- New #sync and #account events; activate/deactivate now emit them
  (plus #identity on activate).
- #identity.handle is optional; rpcEmitIdentityEvent routes through
  the sequencer instead of writing empty payloads.
- #info OutdatedCursor sent when a cursor predates the retention
  window; the stream continues from the earliest available event.
- applyWrites caps at 200 ops per spec.
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
atproto-pds 1c8824a May 24 2026, 06:11 AM

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 24, 2026

Open in StackBlitz

npm i https://pkg.pr.new/create-pds@171
npm i https://pkg.pr.new/@getcirrus/oauth-provider@171
npm i https://pkg.pr.new/@getcirrus/pds@171

commit: 1c8824a

@ascorbic ascorbic merged commit bf2f857 into main May 24, 2026
5 checks passed
@ascorbic ascorbic deleted the fix/sync-1.1-compliance branch May 24, 2026 06:25
@mixie-bot mixie-bot Bot mentioned this pull request May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant