feat(tap): add durable recompute and URL enrichment workers

[Kzoeps]

Summary

• move Tap record handling onto fast durable SQLite snapshot/job writes
• add a coalesced recompute worker for scoring, label writes, retries, stale-job recovery, and delete cleanup
• add detachable URL enrichment cache/worker with bounded checks, retry/backoff, metrics, and provisional scoring behavior
• document Railway/app Tap env and queue/enrichment behavior

Verification

• npm run build
• npx tsc --noEmit --pretty false
• local DB/job queue smoke test for coalescing, race preservation, stale running recovery, and URL cache scoring
• local Tap smoke run with 3 org repos: 3 activities, 3 done recompute jobs, 0 failed jobs, URL checks queued/processed

Out of scope

• Phase 3 durable HuggingFace classification worker
• Phase 4 dashboard dead-letter/enrichment status views

Summary by CodeRabbit

• New Features

  • Background URL enrichment worker with cached URL-resolution and automated recompute enqueueing.
  • Prometheus metrics for recompute jobs, URL checks, and handler durations.
  • Caddy-based reverse-proxy setup and updated service start scripts.

• Improvements

  • Durable SQLite-backed job queue and URL-check / actor-PDS caches; scoring now uses cached URL state.
  • Actor-PDS test detection, gating, and test-signal overrides.
  • Enhanced authenticity checks (placeholder domains, repeated-character detection) and startup/health handling.

• Documentation

  • New architecture/plan/design pages and updated deployment/README guidance.

• Tests

  • New suites for pending deletes, recompute jobs, URL checks, PDS gating, and authenticity scoring.

[linear-code]

HYPER-406 Fix subscribeLabels function

Currently subscribeLabels returns 502 on wss and doesnt work

Review in Linear

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• ✅ Review completed - (🔄 Check again to review again)

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: ea79ce0d-b939-4d34-b840-59758490f60a

📥 Commits

Reviewing files that changed from the base of the PR and between 11e720a and 55faeca.

📒 Files selected for processing (15)

• .env.example
• DEPLOYMENT.md
• README.md
• docs/test-pds-option-2-design.html
• src/labeler/tap-consumer.ts
• src/labeler/url-enrichment-worker.ts
• src/lib/actor-pds-policy.ts
• src/lib/config.ts
• src/lib/db.ts
• src/lib/pds-test-override.ts
• src/lib/pds-utils.ts
• src/lib/resolve-pds.ts
• src/lib/scorer.ts
• tests/pds-test-override.test.ts
• tests/url-enrichment-pds-gate.test.ts

✅ Files skipped from review due to trivial changes (1)

• README.md

🚧 Files skipped from review as they are similar to previous changes (3)

• .env.example
• src/labeler/url-enrichment-worker.ts
• src/labeler/tap-consumer.ts

---

📝 Walkthrough

Walkthrough

Adds Caddy as the public reverse proxy; moves label recomputation to a durable SQLite job queue with a background recompute worker; adds a URL enrichment worker and URL-check cache; scoring now reads precomputed urlResolution; updates startup, metrics, Docker/scripts, docs, and tests.

Changes

Reverse Proxy, Worker Orchestration, and Job Infrastructure

Layer / File(s)	Summary
Caddy reverse proxy and env examples .env.example, Caddyfile, Dockerfile	Adds PORT and NEXT_PORT, changes HOST to 127.0.0.1, replaces LABELER_ENDPOINT with NEXT_PUBLIC_LABELER_ENDPOINT, and adds a Caddyfile that proxies XRPC label endpoints to the labeler and other traffic to Next.js.
Service scripts and Docker adjustments package.json, Dockerfile, railway.toml	Adds start:next/start:proxy, updates start:service to run proxy/next/labeler via concurrently --kill-others-on-fail, installs/exposes Caddy/NEXT_PORT, and includes Caddyfile in Railway watch patterns.
Metrics and startup lifecycle src/labeler/metrics.ts, src/labeler/start.ts	Prometheus gauges for recompute-job and url-check counts and a TAP handler duration histogram added; metrics gauges refreshed on /metrics; recompute and URL enrichment workers start before Tap consumer and are destroyed on shutdown; metrics server logs bind errors.

Recompute Worker and Tap Handler Refactoring

Layer / File(s)	Summary
Tap handler enqueues durable jobs src/labeler/tap-consumer.ts	Tap record handler enqueues recompute-org jobs and queues pending deletes instead of immediate recompute/HF refresh; startup reconciliation drains pending deletes first.
Recompute job processing and worker loop src/lib/db.ts, src/labeler/tap-consumer.ts	Adds recompute job queue APIs (enqueue/claim/complete/fail/recover), pending-delete retry semantics, and startRecomputeWorker() which claims due jobs, recovers stale running jobs, performs scoring (with urlResolution), applies labels/HF refresh, and schedules retries.
Recompute error handling and propagation src/labeler/tap-consumer.ts	Recompute bails when pending deletes exist, attaches urlResolution to scoring input, throws on unsupported tiers, logs and rethrows apply errors to trigger job failure paths.

URL Enrichment Worker and Deferred Resolution

Layer / File(s)	Summary
URL enrichment worker with DNS and HTTP resolution src/labeler/url-enrichment-worker.ts, src/lib/db.ts	Background worker discovers due URL checks from snapshots, validates DNS resolution to public addresses, follows redirects with HEAD/GET and timeout, persists outcomes with attempt/backoff and hard-failure cutoff, and enqueues recompute jobs when resolution state changes.
Public network address validation & placeholder handling src/lib/website-utils.ts, src/lib/constants.ts	Adds isPublicNetworkAddress() and isPlaceholderWebsiteUrl(), and updates placeholder-domain/TLD lists and display-name placeholder patterns.
Synchronous scoring with precomputed URL resolution src/lib/scorer.ts, src/lib/scoring-input.ts	Scoring now reads urlResolution from MergedScoringInput synchronously (no network calls), uses getUrlResolutionState, and limits organization URL checks to MAX_ORGANIZATION_URLS_TO_CHECK.

Architecture, Deployment, and Tests

Layer / File(s)	Summary
Architecture docs and plan docs/tap-worker-architecture.html, docs/tap-worker-plan.md	Adds a static architecture page and a detailed plan describing Tap fast-path, durable job queue, debounced recompute worker, URL enrichment, SQL schemas, scoring contract, and rollout phases.
Deployment and README updates DEPLOYMENT.md, README.md, RAILWAY.md, .env.example	Docs updated for Caddy+Next+labeler runtime, new env vars and URL enrichment tuning, TAP admin guidance, and Caddy routing notes (websocket requirement for subscribeLabels).
DB and URL-check tests tests/*	Adds tests for pending-organization-delete completion semantics, recompute-job coalescing and stale recovery, URL-check cache behaviors (rediscovery, expiration), PDS override behavior, URL enrichment gating, and scoring-authenticity unit tests.

Sequence Diagram(s)

sequenceDiagram
  participant TapStream as Tap Stream
  participant TapHandler as Tap Handler
  participant DB as SQLite (recompute_jobs)
  participant RecomputeWorker as Recompute Worker
  participant LabelsDB as SQLite (labels)
  participant UrlWorker as URL Enrichment Worker
  participant UrlDB as SQLite (url_checks)
  participant DNS as DNS resolver
  participant HTTP as HTTP fetch

  TapStream->>TapHandler: record event (org/profile upsert or delete)
  TapHandler->>DB: enqueueRecomputeJob('recompute-org', did) or recordPendingDelete()
  TapHandler-->>TapStream: ack

  RecomputeWorker->>DB: claimDueRecomputeJob()
  DB-->>RecomputeWorker: RecomputeJob
  RecomputeWorker->>UrlDB: getUrlResolutionMapForDid(did)
  UrlDB-->>RecomputeWorker: urlResolution map
  RecomputeWorker->>LabelsDB: scoreActivity() with urlResolution
  LabelsDB-->>RecomputeWorker: computed labels / HF refresh
  RecomputeWorker->>LabelsDB: updateLabels()
  RecomputeWorker->>DB: completeRecomputeJob()

  UrlWorker->>UrlDB: getDueUrlCheck()
  UrlDB-->>UrlWorker: URL to resolve
  UrlWorker->>DNS: lookupHostAddresses(host)
  DNS-->>UrlWorker: addresses
  UrlWorker->>HTTP: fetch(url) (HEAD/GET, follow redirects)
  HTTP-->>UrlWorker: response or error
  UrlWorker->>UrlDB: recordUrlCheckOk() or recordUrlCheckFailure()
  alt resolution state changed
    UrlWorker->>DB: enqueueRecomputeJob('recompute-org', did)
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐇 I hop through Caddy gates with joy so spry,

Jobs queue, workers hum, URLs I spy.
DNS checked, redirects chased, caches kept neat,
Scoring quick, Tap acks fast — the flow's complete.
Little rabbit cheers: async wins the feat.

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}