feat(tap): add durable recompute and URL enrichment workers

.env.example

@@ -8,11 +8,16 @@ SIGNING_KEY=your-signing-key
 BSKY_IDENTIFIER=your-handle.bsky.social
 BSKY_PASSWORD=your-app-password
 
-# Host to bind the labeler server to
-# Use 0.0.0.0 in production (Railway) to bind to all interfaces
-HOST=0.0.0.0
+# Public HTTP port for Caddy. Hosted platforms usually set PORT automatically.
+PORT=8080
 
-# Port for the labeler server (AT Protocol labeler endpoint)
+# Internal Next.js dashboard port behind Caddy
+NEXT_PORT=3000
+
+# Host to bind the labeler server to. 127.0.0.1 is enough when Caddy runs in the same container.
+HOST=127.0.0.1
+
+# Internal port for the labeler server (AT Protocol labeler endpoint behind Caddy)
 LABELER_PORT=4100
 
 # Port for the Prometheus metrics endpoint
@@ -24,21 +29,22 @@ ACTIVITY_DB_PATH=activity-log.db
 # Path to the labels database (used by @skyware/labeler LabelerServer)
 LABELS_DB_PATH=labels.db
 
-# Public URL of the labeler endpoint (e.g. https://labeler.yourdomain.com)
-# On Railway, set this to your service's public URL
-LABELER_ENDPOINT=https://labeler.yourdomain.com
+# Public URL of the Caddy-fronted app/labeler endpoint (e.g. https://labeler.yourdomain.com)
+# /xrpc/* is routed to the labeler and dashboard/API routes are routed to Next.js.
+NEXT_PUBLIC_LABELER_ENDPOINT=https://labeler.yourdomain.com
 
 # PDS URL for the labeler account (auto-detected during setup, override if needed)
 PDS_URL=https://climateai.org
 
-# Tap sidecar configuration
-TAP_URL=http://localhost:2480
+# External Tap service URL. Tap runs as a separate service, not inside this app container.
+TAP_URL=https://your-tap-service.example.com
 TAP_ADMIN_PASSWORD=
-TAP_BIND=:2480
-TAP_DB_PATH=tap.db
+
+# Comma-separated PDS hosts whose actors should always be labeled likely-test.
+# When set, URL enrichment is deferred until each actor's PDS is known and skipped for matching test PDS hosts.
+TEST_PDS_HOSTS=
 
 # --- Railway deployment ---
 # Mount a persistent volume at /app/data and set:
 # ACTIVITY_DB_PATH=/app/data/activity-log.db
 # LABELS_DB_PATH=/app/data/labels.db
-# TAP_DB_PATH=/app/data/tap.db

Caddyfile

@@ -0,0 +1,40 @@
+{
+  auto_https off
+  admin off
+}
+
+:{$PORT:8080} {
+  encode zstd gzip
+
+  @xrpcPreflight {
+    path /xrpc/*
+    method OPTIONS
+  }
+
+  handle @xrpcPreflight {
+    header Access-Control-Allow-Origin "*"
+    header Access-Control-Allow-Headers "authorization, content-type"
+    header Access-Control-Allow-Methods "GET, POST, OPTIONS"
+    header Access-Control-Max-Age "86400"
+    respond "" 204
+  }
+
+  @publicLabelerXrpc {
+    path /xrpc/_health /xrpc/com.atproto.label.queryLabels /xrpc/com.atproto.label.subscribeLabels
+  }
+
+  handle @publicLabelerXrpc {
+    header {
+      Access-Control-Allow-Origin "*"
+      Cache-Control "no-store"
+    }
+
+    reverse_proxy 127.0.0.1:{$LABELER_PORT:4100} {
+      flush_interval -1
+    }
+  }
+
+  handle {
+    reverse_proxy 127.0.0.1:{$NEXT_PORT:3000}
+  }
+}

DEPLOYMENT.md

@@ -15,7 +15,9 @@ Current scripts:
 - `npm run start` → `next start`
 - `npm run dev:service` → runs `npm run dev` and `npm run labeler` together
 - `npm run labeler` → `tsx src/labeler/start.ts`
-- `npm run start:service` → runs `npm run start` and `npm run labeler` together
+- `npm run start:service` → runs Caddy, `npm run start:next`, and `npm run labeler` together
+- `npm run start:next` → starts Next.js on `NEXT_PORT` (default `3000`)
+- `npm run start:proxy` → starts Caddy using `Caddyfile`
 
 For hosted deployment, the image build uses `scripts/build.sh`, which sets `NEXT_PUBLIC_DEPLOY_TIME` and then runs `npm run build`.
 
@@ -29,11 +31,13 @@ At runtime, the labeler process starts:
 
 Tap is a separate service. The labeler process connects to it via `TAP_URL` instead of launching it locally, and startup should fail if `TAP_URL` is missing.
 
-The dashboard proxies only the public label distribution XRPC methods under `/xrpc/*` to the local labeler server at `http://127.0.0.1:LABELER_PORT`:
+Caddy is the public front door for the app service. It routes dashboard and API traffic to Next.js, and routes public label distribution XRPC methods directly to the local labeler server at `http://127.0.0.1:LABELER_PORT`:
 
 - `com.atproto.label.queryLabels`
 - `com.atproto.label.subscribeLabels`
 
+The direct Caddy route is required for `subscribeLabels`, because it is a WebSocket endpoint.
+
 ## Environment variables
 
 ### Required for the running service
@@ -72,8 +76,11 @@ If your Tap deployment uses admin auth, configure `TAP_ADMIN_PASSWORD` on the Ta
 
 - `HF_TOKEN` — enables HuggingFace classification; if unset, HF scoring stays disabled
 - `PDS_URL` — overrides PDS discovery from the DID document
-- `HOST` — bind host for the labeler server (`0.0.0.0` by default)
-- `LABELER_PORT` — labeler HTTP port (`4100` by default)
+- `TEST_PDS_HOSTS` — comma-separated PDS hosts whose actors should always be labeled `likely-test`; URL enrichment waits for actor PDS resolution and skips matching test PDS hosts
+- `PORT` — public Caddy HTTP port (hosted platforms usually set this; default fallback `8080`)
+- `NEXT_PORT` — internal Next.js HTTP port (`3000` by default)
+- `HOST` — bind host for the labeler server (`127.0.0.1` is recommended when Caddy runs in the same container)
+- `LABELER_PORT` — internal labeler HTTP port (`4100` by default)
 - `METRICS_PORT` — metrics port (`4101` by default)
 - `RESET_DB=true` — deletes the SQLite files before startup
 
@@ -103,6 +110,6 @@ Also persist the `-wal` and `-shm` companions that SQLite may create beside each
 ## Hosted environment notes
 
 - Run a single replica only for each service; SQLite state is not safe to share across multiple hosts.
-- Expose the dashboard on the host-assigned HTTP port and make the labeler endpoint publicly reachable at `NEXT_PUBLIC_LABELER_ENDPOINT`.
+- Expose Caddy on the host-assigned HTTP port and set `NEXT_PUBLIC_LABELER_ENDPOINT` to that public URL. Caddy will route `/xrpc/*` label methods to the labeler and dashboard/API routes to Next.js.
 - Keep Tap reachable from the app service at `TAP_URL`; do not rely on a localhost default.
 - If you need to reset state on a hosted platform, set `RESET_DB=true` for one restart, then remove it.

Dockerfile

@@ -1,8 +1,8 @@
 # Build and run the app service
 FROM node:22-slim
 
-# better-sqlite3 needs build tools
-RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*
+# better-sqlite3 needs build tools; Caddy fronts Next and the labeler XRPC server
+RUN apt-get update && apt-get install -y python3 make g++ caddy && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 
@@ -20,8 +20,9 @@ ENV NEXT_PUBLIC_COMMIT_SHA=$RAILWAY_GIT_COMMIT_SHA
 ENV NEXT_PUBLIC_LABELER_ENDPOINT=$NEXT_PUBLIC_LABELER_ENDPOINT
 RUN chmod +x scripts/build.sh && ./scripts/build.sh
 
-# Expose ports for the app service only; Tap runs separately via TAP_URL
-EXPOSE 3000 4100 4101
+# Caddy listens on $PORT/8080; Next and labeler stay internal; Tap runs separately via TAP_URL
+ENV NEXT_PORT=3000
+EXPOSE 8080 3000 4100 4101
 
 # Run the app service (dashboard + labeler)
 CMD ["npm", "run", "start:service"]

RAILWAY.md

@@ -37,6 +37,7 @@ Required:
 | `SIGNING_KEY` | Labeler private key |
 | `NEXT_PUBLIC_LABELER_ENDPOINT` | Public HTTPS URL of the app service |
 | `TAP_URL` | Required URL of the separate Tap service; no localhost default |
+| `TAP_ADMIN_PASSWORD` | App-side Tap admin password; required when the Tap service enables admin auth |
 
 Usually needed for setup/sync:
 
@@ -57,10 +58,15 @@ Optional:
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `HF_TOKEN` | Enables HuggingFace scoring | _(empty)_ |
-| `HOST` | Labeler bind host | `0.0.0.0` |
-| `LABELER_PORT` | Labeler HTTP port | `4100` |
+| `NEXT_PORT` | Internal Next.js port behind Caddy | `3000` |
+| `HOST` | Labeler bind host | `127.0.0.1` |
+| `LABELER_PORT` | Internal labeler HTTP port behind Caddy | `4100` |
 | `METRICS_PORT` | Metrics port | `4101` |
 | `RESET_DB` | Clear app DB files on startup | _(empty)_ |
+| `URL_ENRICHMENT_ENABLED` | Enable detachable async URL checks through `url_checks` | `true` |
+| `URL_CHECK_TIMEOUT_MS` | Timeout for one URL resolution attempt | `4000` |
+| `URL_CHECK_INTERVAL_MS` | URL worker polling interval | `1000` |
+| `URL_CHECK_MAX_URLS_PER_DID` | Maximum profile/organization URLs cached and checked per DID | `5` |
 
 Start the app with:
 
@@ -92,8 +98,9 @@ Do not point Tap at a custom database path unless the service itself exposes tha
 
 | Port | Service | Notes |
 |------|---------|-------|
-| `$PORT` | App service | Railway routes web traffic here |
-| `4100` | App service | Public labeler endpoint |
+| `$PORT` | App service | Caddy listens here and routes traffic |
+| `3000` | App service | Internal Next.js dashboard behind Caddy |
+| `4100` | App service | Internal labeler endpoint behind Caddy |
 | `4101` | App service | Internal metrics only |
 | `2480` | Tap service | Tap HTTP endpoint |
 
@@ -107,7 +114,7 @@ The app service must be able to reach `TAP_URL`; the Tap service does not need t
 2. Set the app service `TAP_URL` to the Tap service URL.
 3. Set the app service database paths to the mounted app volume.
 4. Run `npm run setup` locally once to create the labeler credentials if needed.
-5. Deploy the app service.
+5. Deploy the app service. Caddy will route `/xrpc/com.atproto.label.queryLabels` and `/xrpc/com.atproto.label.subscribeLabels` directly to the labeler, and all dashboard/API routes to Next.js.
 
 ---
 
@@ -127,7 +134,8 @@ The app service must be able to reach `TAP_URL`; the Tap service does not need t
 ### App service health check fails
 
 - Check `DID`, `SIGNING_KEY`, and `NEXT_PUBLIC_LABELER_ENDPOINT`
-- Check the app logs for the labeler process error
+- Check that Caddy is listening on `$PORT`
+- Check the app logs for Next.js or labeler process errors
 
 ### Emergency reset
 

README.md

@@ -43,12 +43,15 @@ The setup script will:
 ### Run
 
 ```bash
-# Start the app service (dashboard + labeler)
+# Start the app service locally (dashboard + labeler, without Caddy)
 npm run dev:service
 
 # Or start separately:
 npm run dev            # Dashboard on http://localhost:3000
 npm run labeler        # Labeler backend on port 4100 + metrics on 4101
+
+# Production start runs Caddy + Next + labeler
+npm run start:service   # Caddy on $PORT, Next on NEXT_PORT, labeler on LABELER_PORT
 ```
 
 Tap runs as a separate service. Point `TAP_URL` at that service's URL; there is no localhost fallback and the app will not start without it.
@@ -87,7 +90,13 @@ Scores `app.certified.actor.organization` records on 13 completeness signals (10
 | Avatar | 10 | Has an avatar image |
 | Banner | 10 | Has a banner image |
 
-Test detection: regex patterns catch common placeholder strings (`test`, `asdf`, `lorem ipsum`, etc.) and override the score to force ⚠ Likely Test.
+Test detection: authenticity checks catch common placeholder strings (`test`, `asdf`, `lorem ipsum`, etc.) and override the score to force ⚠ Likely Test. Operators can also set `TEST_PDS_HOSTS` to force actors from known development PDS hosts into ⚠ Likely Test. Actor PDS lookup runs through the durable recompute queue; the first record from an uncached actor may be labeled by content score first, then corrected once the actor DID document is resolved.
+
+### URL enrichment
+
+Tap handlers never fetch URLs. New records are scored immediately with optimistic provisional URL resolve points for valid-looking public URLs. A detachable in-process URL enrichment worker checks those URLs later, stores results in the independent `url_checks` cache table, and queues a recompute only when cached URL state changes.
+
+When `TEST_PDS_HOSTS` is configured, URL enrichment is PDS-aware: it defers URL checks until the actor PDS cache is fresh, skips actors on configured test PDS hosts, and only checks URLs for actors on non-test PDS hosts. Set `URL_ENRICHMENT_ENABLED=false` to disable URL checks completely. When disabled, scoring keeps the provisional URL behavior and does not depend on the `url_checks` table.
 
 ## Scripts
 
@@ -96,7 +105,7 @@ Test detection: regex patterns catch common placeholder strings (`test`, `asdf`,
 | `npm run dev` | Start Next.js dashboard |
 | `npm run labeler` | Start labeler backend |
 | `npm run dev:service` | Start dashboard + labeler concurrently |
-| `npm run start:service` | Start the production app service processes |
+| `npm run start:service` | Start Caddy reverse proxy + production dashboard + labeler process |
 | `npm run setup` | Initialize labeler account |
 | `npm run set-labels` | Push/update label definitions |
 | `npm run build` | Production build |
@@ -112,32 +121,36 @@ Test detection: regex patterns catch common placeholder strings (`test`, `asdf`,
 | `BSKY_PASSWORD` | (set by setup) | Account password or app password |
 | `PDS_URL` | (auto-detected) | PDS endpoint URL |
 | `NEXT_PUBLIC_LABELER_ENDPOINT` | https://labeler.<handle> | Public HTTPS URL for the labeler |
-| `HOST` | 127.0.0.1 | Labeler server bind address |
-| `LABELER_PORT` | 4100 | Labeler server port |
+| `PORT` | 8080 | Public HTTP port listened to by Caddy; hosted platforms usually set this |
+| `NEXT_PORT` | 3000 | Internal Next.js port behind Caddy |
+| `HOST` | 127.0.0.1 | Labeler server bind address; keep local when Caddy is in the same container |
+| `LABELER_PORT` | 4100 | Internal labeler server port behind Caddy |
 | `METRICS_PORT` | 4101 | Prometheus metrics port |
 | `TAP_URL` | required | URL of the separate Tap service (no localhost default) |
+| `TAP_ADMIN_PASSWORD` | empty | App-side password for Tap admin auth; must match the Tap service when auth is enabled |
+| `TEST_PDS_HOSTS` | empty | Comma-separated PDS hosts whose actors should always be labeled `likely-test`; when set, URL enrichment waits for actor PDS resolution and skips matching test PDS hosts |
 | `ACTIVITY_DB_PATH` | `activity-log.db` | Activity log database path |
+| `URL_ENRICHMENT_ENABLED` | `true` | Enables async URL checks through the detachable `url_checks` cache |
+| `URL_CHECK_TIMEOUT_MS` | `4000` | Timeout for one URL resolution attempt |
+| `URL_CHECK_INTERVAL_MS` | `1000` | Poll interval for the URL enrichment worker |
+| `URL_CHECK_MAX_URLS_PER_DID` | `5` | Maximum profile/organization URLs cached and checked per DID |
 
-Tap-specific settings belong on the Tap service, not the app service. This repo only needs `TAP_URL` here; set `TAP_ADMIN_PASSWORD` on the Tap service if you use Tap admin auth.
+Tap runtime settings belong on the Tap service. If the Tap service sets `TAP_ADMIN_PASSWORD`, set the same value on the app service so health checks and the Tap WebSocket can authenticate.
 
 ## Production Deployment
 
 Deploy the app service and Tap as separate services. The app service runs the dashboard plus labeler backend and connects to Tap over `TAP_URL`; Tap owns its own database, volume, lifecycle, and any Tap-specific auth settings.
 
-The labeler backend (port 4100) must be accessible via HTTPS for AT Protocol clients. Use a reverse proxy:
-
-```nginx
-server {
-    listen 443 ssl;
-    server_name labeler.yourdomain.com;
-
-    location / {
-        proxy_pass http://127.0.0.1:4100;
-        proxy_set_header Host $host;
-    }
-}
+The production app uses Caddy as the front door. Caddy routes public AT Protocol XRPC label methods directly to the labeler process and everything else to Next.js:
+
+```txt
+/xrpc/com.atproto.label.queryLabels     -> 127.0.0.1:4100
+/xrpc/com.atproto.label.subscribeLabels -> 127.0.0.1:4100
+/*                                      -> 127.0.0.1:3000
 ```
 
+This is important because `subscribeLabels` uses WebSockets, which need a real reverse proxy rather than the Next.js `fetch()` proxy fallback.
+
 ## Tech Stack
 
 - **Runtime:** Node.js 22