diff --git a/README.md b/README.md index c9d4130..2522c39 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > Automated quality scoring for merged `app.certified.actor.profile` + `app.certified.actor.organization` data on AT Protocol -This fork monitors both `app.certified.actor.profile` and `app.certified.actor.organization` records, merges them by DID for display and classification context, and labels organizations based on how complete, consistent, and non-placeholder the organization record looks. +This fork monitors both `app.certified.actor.profile` and `app.certified.actor.organization` records, merges them by DID for display and classification context, and labels actor DIDs based on how complete, consistent, and non-placeholder the organization profile looks. ## Labels @@ -64,7 +64,7 @@ The runtime is split into three pieces: - Tap service → labeler process over `TAP_URL` - Labeler process → `labels.db` + `activity-log.db` -The Tap sidecar listens to both `app.certified.actor.profile` and `app.certified.actor.organization`, merges them by DID for actor context, and still applies labels to the organization record URI. +The Tap sidecar listens to both `app.certified.actor.profile` and `app.certified.actor.organization`, merges them by DID for actor context, and applies quality labels to the actor DID. Fresh deployments should wipe the app and Tap volumes together so old local labels, activity rows, and Tap cursor state are not reused. The Next.js dashboard reads from `activity-log.db`. diff --git a/docs/tap-worker-architecture.html b/docs/tap-worker-architecture.html deleted file mode 100644 index 75cb3cc..0000000 --- a/docs/tap-worker-architecture.html +++ /dev/null @@ -1,587 +0,0 @@ - - - - - - Orglabeler Tap Worker Architecture - - - -
-
-
-

Tap worker architecture

-

Fast ingest, durable jobs, best-known labels.

-
-
-

Tap is only responsible for delivering events. The app persists snapshots, acks quickly, then workers score, enrich, classify, and relabel from local durable state.

-
- No network I/O before ack - One org per DID: rkey self - Crash-safe job claims - Durable delete negation - Relabel only on tier changes -
-
-
- -
-
-
-
-

Tap stream

- external service -
- -
-

Tap event

-

Profile or organization create/update/delete events arrive from Tap.

-
    -
  • app.certified.actor.profile
  • -
  • app.certified.actor.organization
  • -
  • org record invariant: rkey = self
  • -
-
- -
handoff
- -
-

Fast handler

-

Only does cheap local work.

-
    -
  • minimal parse/validation
  • -
  • persist latest snapshot/tombstone
  • -
  • upsert durable job
  • -
  • return so Tap can ack
  • -
-
- -
-

Delete branch

-

Deletes do not negate labels inline. They persist a durable negate-org-label job with the record URI.

-
-
- -
-
-

SQLite state

- durable local source -
- -
-

Snapshots

-

Latest actor state, keyed by DID.

-
    -
  • profile_snapshots[did]
  • -
  • organization_snapshots[did]
  • -
  • canonical org URI:
    at://{did}/app.certified.actor.organization/self
  • -
-
- -
-

Typed jobs

-

One crash-safe queue with typed work.

-
- recompute-org{did} - negate-org-label{did} - enrich-url{normalized_url} - classify-hf{did}:{model}:{hash} -
-
- -
-

Worker claims

-

Jobs cannot be stranded by process restarts.

-
    -
  • pending → running → done
  • -
  • locked_until recovers stale running jobs
  • -
  • attempts + backoff + jitter
  • -
  • failed only after max attempts
  • -
-
-
- -
-
-

Workers

- after ack -
- -
-

Debounced recompute

-

recompute-org:{did} waits 1–3 seconds so profile and org backfill can land together.

-
    -
  • reads latest snapshots
  • -
  • reads URL/HF caches
  • -
  • runs pure local scoring
  • -
  • applies best-known label quickly
  • -
-
- -
-

URL enrichment

-

Network URL checks happen in a bounded worker, never inside Tap ack path.

-
    -
  • low concurrency
  • -
  • short timeout
  • -
  • cache by normalized URL
  • -
  • enqueue recompute after result
  • -
-
- -
-

HF/BERT classification

-

Classifier worker writes cache rows and asks recompute to decide labels.

-
    -
  • key by model + input hash
  • -
  • pending has no effect
  • -
  • high-confidence low-quality can penalize/cap
  • -
  • recompute owns relabel decisions
  • -
-
-
- -
-
-

Outputs

- public + dashboard -
- -
-

Pure scoring contract

-

Scoring uses local inputs only.

-
    -
  • profile + org snapshots
  • -
  • URL check cache
  • -
  • HF classification cache
  • -
  • returns score, tier, breakdown, enrichment status
  • -
-
- -
-

Provisional label

-

Apply a label from best-known state immediately. The dashboard can mark enrichment as pending.

-
- -
-

Later relabel

-

When URL/HF results arrive, recompute. If tier is unchanged, update the dashboard only. If tier changes, negate old quality label and apply the new one.

-
- -
-

Delete correctness

-

negate-org-label retries remote label negation until success or max attempts. Local tombstone stays until public labels are cleaned up.

-
-
-
-
- -
-
- URL scoring -

Valid-looking URLs can receive provisional points. Confirmed failures can remove resolve points later.

-
-
- HF scoring -

Pending classifier results have no effect. High-confidence low-quality results can add a penalty or tier cap.

-
-
- DID keying -

Orglabeler has one organization record per DID, so recompute and org state are keyed by DID.

-
-
- No churn -

Workers recompute often, but public label writes happen only when the effective tier changes.

-
-
- -
-
-
0
-

Emergency fix

-

Keep URL network checks out of scoreActivity(); cap org URL checks.

-
-
-
1
-

Jobs + recompute

-

Add typed jobs, claim locks, debounce, and provisional best-known labels.

-
-
-
2
-

Durable deletes

-

Move label negation into retryable negate-org-label jobs.

-
-
-
3
-

URL + HF workers

-

Workers write enrichment caches and enqueue recompute; they never apply labels directly.

-
-
-
4
-

Observability

-

Track handler duration, queue depth, dead letters, and enrichment status in the dashboard.

-
-
- - -
- - diff --git a/docs/tap-worker-plan.md b/docs/tap-worker-plan.md deleted file mode 100644 index 3d43a7b..0000000 --- a/docs/tap-worker-plan.md +++ /dev/null @@ -1,205 +0,0 @@ -# Tap Worker Plan - -## Goal - -Keep Tap ingestion fast and reliable by making the Tap handler do only cheap, durable local work. Anything that can block — URL checks, HuggingFace classification, label application retries, expensive recomputes — should run after Tap has been acknowledged. - -## Problem - -Tap only receives an ack after the event handler returns. When the handler performs network work, one slow URL or classifier request can hold the ack path long enough for Tap to retry events and grow the outbox buffer. - -Confirmed example: website resolution inside `scoreActivity()` blocked Tap acks. Removing network URL checks let the outbox drain. - -## Target flow - -```txt -Tap event - ↓ -Fast handler - - validate collection/action - - persist latest profile/org snapshot - - enqueue debounced recompute job for the DID - - return so Tap can ack - ↓ -Debounced recompute worker - - wait a short window so profile/org events can land together - - recompute from best-known local state - - apply/update a provisional label immediately - - update dashboard DB - ↓ -Enrichment workers - - resolve URLs with cache + timeout - - classify with HuggingFace/BERT - - enqueue recompute after results land - - relabel only if the effective tier changes -``` - -## Rules - -1. **No network I/O in the Tap handler** - - no URL fetches - - no HuggingFace calls - - no remote label reads/writes if avoidable - -2. **Tap handler must persist before returning** - - write snapshots/job rows to SQLite - - after that, the event can be safely acked - -3. **Jobs must be durable and coalesced** - - if profile and org events arrive close together, they should update one recompute job for the DID - - workers should process the latest state, not every intermediate event - -4. **Apply best-known labels immediately** - - do not wait for URL or HF enrichment before applying a label - - dashboard can show `pending enrichment`, but public labels should use the current best-known score - - later enrichment can trigger a recompute and relabel if the tier changes - -5. **Debounce actor recomputes briefly** - - profile and organization events for the same DID often arrive close together - - upsert one recompute job per DID with `run_after = now + 1–3 seconds` - - repeated events update the same job instead of producing multiple labels - -## Scoring model - -Scoring should be pure and local-only. It can read snapshots and cached enrichment rows, but it must never perform network I/O. - -Initial labels are **best-known provisional labels**. They are applied before URL checks or HuggingFace/BERT classification finish. - -### URL scoring before enrichment - -URL fields should be optimistic while enrichment is pending: - -| URL state | Scoring behavior | -| --- | --- | -| Missing URL | `0` URL points | -| Invalid URL syntax | `0` URL points | -| Valid-looking URL, not checked yet | award provisional URL points | -| Resolved OK later | keep/confirm URL points | -| Temporary check failure | keep provisional/unknown state and retry | -| Repeated hard failure | remove resolve points and enqueue recompute | - -This avoids delaying the first public label on slow or flaky URL checks. URL enrichment can still downgrade later if a URL repeatedly fails. - -### HF/BERT scoring before enrichment - -Classifier results should be conservative while pending: - -| HF/BERT state | Scoring behavior | -| --- | --- | -| Missing or pending result | no effect | -| Failed or timed out | no effect, retry later | -| Positive/meaningful result | no effect or small confirmation only | -| Low-quality result with high confidence | add penalty, authenticity signal, or tier cap | - -HF/BERT should not give optimistic points while pending. It should mainly act as a high-confidence negative signal because classifier false positives can otherwise cause label churn. - -### Recompute and relabel rule - -Any enrichment result writes to its cache table and then enqueues `recompute-org:{did}`. The recompute worker: - -1. reads latest profile/org snapshots, -2. reads cached URL and HF/BERT results, -3. computes the current best-known tier, -4. updates the dashboard row every time, and -5. writes public ATProto labels only when the effective tier changes. - -## Lightweight data model - -### `recompute_jobs` - -Actor-level work queue for scoring and label updates. `run_after` provides the short debounce window for profile/org events. - -```sql -id INTEGER PRIMARY KEY, -kind TEXT NOT NULL, -- recompute-org -key TEXT NOT NULL, -- DID or DID+rkey -status TEXT NOT NULL, -- pending, running, done, failed -attempts INTEGER NOT NULL, -run_after TEXT NOT NULL, -payload TEXT, -last_error TEXT, -created_at TEXT NOT NULL, -updated_at TEXT NOT NULL, -UNIQUE(kind, key) -``` - -### `url_checks` - -Cache URL resolution so the same URL is not fetched on every record update. - -```sql -normalized_url TEXT PRIMARY KEY, -status TEXT NOT NULL, -- pending, ok, failed -resolvable INTEGER, -status_code INTEGER, -error TEXT, -checked_at TEXT, -expires_at TEXT -``` - -### `hf_classifications` - -Stores classifier results keyed by model and input hash. - -```sql -did TEXT NOT NULL, -rkey TEXT NOT NULL, -model TEXT NOT NULL, -input_hash TEXT NOT NULL, -status TEXT NOT NULL, -- pending, done, failed -label TEXT, -score REAL, -error TEXT, -classified_at TEXT, -PRIMARY KEY (did, rkey, model, input_hash) -``` - -## Rollout phases - -### Phase 0 — current emergency fix - -- Keep URL network checks out of `scoreActivity()`. -- Keep org URL scoring bounded. -- Confirm Tap outbox drains after deploy. - -### Phase 1 — durable recompute worker - -Status: implemented. - -- `src/lib/db.ts` creates and manages `recompute_jobs` in the dashboard SQLite database. -- Tap handlers persist profile/org snapshots or pending delete rows, upsert one `recompute-org` job per DID, then return without scoring, HuggingFace calls, or label writes. -- Jobs use a 2 second debounce window so profile/org events can coalesce. -- `startRecomputeWorker()` drains due jobs, recomputes from best-known local state, writes dashboard rows, applies labels, and refreshes HuggingFace classification outside the Tap ack path. -- Organization deletes are persisted in `pending_organization_deletes` and negated/cleaned up by the worker or startup reconciliation. -- `/metrics` exposes `orglabeler_recompute_jobs{status="..."}` for durable job row status and `orglabeler_tap_handler_duration_ms` for Tap handler latency. - -### Phase 2 — URL enrichment worker - -Status: implemented. - -- `url_checks` is a detachable cache table. It stores only normalized URL resolution state and retry metadata; snapshots and activity rows do not depend on it. -- `src/labeler/url-enrichment-worker.ts` runs as an in-process polling worker with low concurrency, short fetch timeouts, manual redirect validation, bounded URLs per DID, and bounded retry/backoff. -- Scoring reads cached URL states as optional input. Missing or pending cache rows stay optimistic and keep provisional URL resolve points. -- `ok` URL checks confirm the provisional score. Repeated hard failures mark the URL `failed`, remove resolve points on the next recompute, and enqueue `recompute-org` for affected DIDs. -- The whole feature can be disabled with `URL_ENRICHMENT_ENABLED=false`; scoring then falls back to current provisional behavior. -- `/metrics` exposes `orglabeler_url_checks{status="..."}` for cache state counts. - -### Phase 3 — HuggingFace/BERT worker - -- Add `hf_classifications`. -- Hash classifier input so unchanged text reuses the same result. -- Process with low concurrency and retries/backoff. -- Recompute labels after classification lands. -- Relabel only if the effective tier changes. - -### Phase 4 — cleanup and observability - -- Show enrichment status in dashboard. -- Add dead-letter view for failed jobs. -- Add structured logs around job attempts, duration, and final label changes. - -## Open questions - -- What debounce window is enough for profile + organization backfill: 1s, 2s, or 3s? -- What retry policy should be used for URL checks and HF failures? -- How long should URL cache entries stay valid? diff --git a/docs/test-pds-option-2-design.html b/docs/test-pds-option-2-design.html deleted file mode 100644 index 2cbae07..0000000 --- a/docs/test-pds-option-2-design.html +++ /dev/null @@ -1,1167 +0,0 @@ - - - - - - TEST_PDS Option 2 — Orglabeler Design Doc - - - -
-
-
-

Orglabeler design doc · option 2

-

TEST_PDS through the existing recompute loop.

-
- -
- -
- - -
- - -
-
- -
-
-

Cached test PDS path

-

The cached actor PDS host matches TEST_PDS_HOSTS, so the recompute result is rewritten to likely-test before DB logging and label sync.

-
- override active -
- -
-
-
Tap event writes snapshots1
-

Profile and organization records are persisted. The Tap handler still avoids slow external work.

- snapshot saved -
- -
-
Durable recompute job2
-

enqueueRecomputeJob('recompute-org', did) coalesces actor-level scoring work.

- debounced -
- -
-
Completeness scoring3
-

Build merged profile + organization input, score authenticity/completeness, and keep validation notes.

- pure score -
- -
-
Actor PDS cache check4
-

Read local actor_pds_cache. Missing or stale rows queue a PDS-resolution job.

- cache hit -
- -
-
PDS test override5
-

Add actor-pds-test-host: host to testSignals when the normalized host is configured. The numeric score stays unchanged.

- tier forced to likely-test -
- -
-
Log activity row6
-

Store score, tier, breakdown, test signals, validation notes, and HF fields as today.

- DB source of truth -
- -
-
Sync ATProto label7
-

Apply the DB tier to the organization record URI. Existing quality labels are negated if needed.

- label applied -
- -
-
Resolve DID → PDS8
-

For cache misses, the same worker handles resolve-actor-pds, fetches the DID document, and caches host state.

- async only -
- -
-
Correction recompute9
-

After resolving PDS, enqueue recompute-org again so DB, labels, and URL-enrichment eligibility converge on the cached host truth.

- eventual correction -
-
-
-
- -
-
-
-
-

Implementation seam

-

The key difference from hyperlabel: no standalone actor-PDS worker. Add one new recompute job kind and run it inside startRecomputeWorker().

-
// recompute worker loop
-if (job.kind === 'resolve-actor-pds') {
-  resolveAndCacheActorPds(job.key)
-  enqueueRecomputeJob('recompute-org', job.key)
-  completeRecomputeJob(job.id)
-}

-if (job.kind === 'recompute-org') {
-  await recomputeLabeledOrganizationRow(job.key)
-}
-
- -
-

Why this shape

-

It preserves Tap ack speed, avoids a second interval worker, and lets the existing retry/backoff/recovery path handle PDS failures.

-
-
- -
-

Files to touch

-
-
src/lib/config.tsAdd TEST_PDS_HOSTS parsed from env.
-
src/lib/pds-utils.tsNormalize hostnames, parse comma-separated values, exact-match configured hosts.
-
src/lib/pds-test-override.tsManage the stable test signal and derive the override tier.
-
src/lib/actor-pds-policy.tsCentralize cache-miss enqueueing and URL-enrichment gating.
-
src/lib/resolve-pds.tsAdd resolvePdsForDid() for did:plc and ideally did:web.
-
src/lib/db.tsAdd actor_pds_cache helpers and extend RecomputeJobKind.
-
src/labeler/tap-consumer.tsApply cached PDS override during recompute and process resolution jobs.
-
src/labeler/url-enrichment-worker.tsSkip URL checks for test-PDS actors and defer checks while PDS is unknown or stale.
-
README.md / DEPLOYMENT.mdDocument TEST_PDS_HOSTS, URL gating, and first-pass correction behavior.
-
-
-
- -
- - - - - - - - - - -
StepChangeDeveloper-readable behavior
1Parse envTEST_PDS_HOSTS="epds1.test.certified.app,dev.example" becomes normalized exact host list.
2Persist cacheactor_pds_cache stores pending | ok | failed, host, URL, expiry, and last error.
3Score normallyNormal authenticity/completeness score is produced first.
4Override tierIf cached host matches configured test hosts, add a test signal and force likely-test.
5Resolve laterMissing/stale cache rows queue resolve-actor-pds; resolved cache triggers another org recompute.
6Gate URL checksURL enrichment is deferred until PDS cache is fresh, skipped for configured test PDS hosts, and allowed for fresh non-test hosts.
-
-
- -
-
-
-

State contracts

-
-
A
Config is optional.If TEST_PDS_HOSTS is empty, PDS cache checks and resolution jobs are skipped.
-
B
The score stays honest.PDS override does not mutate totalScore or completeness breakdown. It only changes testSignals and derived tier.
-
C
Signals are stable.Use a prefix like actor-pds-test-host: so stale PDS signals can be removed when config/cache changes.
-
D
Labels follow DB.The recompute result is logged first; then ATProto label sync uses the DB tier.
-
E
URL checks wait for PDS truth.With TEST_PDS_HOSTS set, URL enrichment only runs after a fresh non-test PDS cache hit.
-
-
- -
-

Data model sketch

-
CREATE TABLE actor_pds_cache (
-  did TEXT PRIMARY KEY,
-  status TEXT NOT NULL CHECK(status IN ('pending', 'ok', 'failed')),
-  pds_url TEXT,
-  pds_host TEXT,
-  checked_at TEXT,
-  expires_at TEXT NOT NULL,
-  last_error TEXT,
-  created_at TEXT NOT NULL DEFAULT (datetime('now')),
-  updated_at TEXT NOT NULL DEFAULT (datetime('now'))
-);
-
Exact host matching only. Start with normalized exact host equality. Wildcards look convenient, but they create surprising label scope.
-
-
-
- -
-
-
-
-

Failure and edge-case behavior

-

The easy path should be correct, but the failure path must be boring: retry, keep prior data if safe, and tell operators what happened.

-
- operator-visible -
- -
- - - - - - - - - - -
CaseBehaviorWhat to log / expose
Cache missQueue resolve-actor-pds; first recompute uses normal tier and URL enrichment is deferred.pending correction reason: cache miss.
Cache staleUse stale host if present for current recompute, refresh in background, and defer URL enrichment until fresh.refresh queued reason: stale cache.
DID document fetch failsRecord failed cache attempt and retry with bounded backoff.retry include HTTP status/error message.
Configured host removedRecompute removes stale actor-pds-test-host: signals and lets score tier win.self-healing count of rows updated.
Actor moves PDSTTL expiry refreshes cache; next recompute corrects labels and URL-enrichment eligibility.cache updated old host → new host.
Strict no-temporary-label requirementThis design is eventually correct. If temporary standard/high-quality labels are unacceptable, switch to blocking lookup or pre-seed cache.product decision document explicitly.
-
-
-
-
-
- - -
- - - - diff --git a/src/app/docs/page.tsx b/src/app/docs/page.tsx index 3fbff8b..731ade2 100644 --- a/src/app/docs/page.tsx +++ b/src/app/docs/page.tsx @@ -110,7 +110,7 @@ export default function DocsPage() { Documentation

- How the certified organization labeler merges profile and organization records, then scores and labels the organization record on AT Protocol. + How the certified organization labeler merges profile and organization records, then scores the organization context and labels the actor DID on AT Protocol.

@@ -125,7 +125,7 @@ export default function DocsPage() { { step: 'Profile + Organization Records', sub: 'collected together by Tap' }, { step: 'Merged by DID', sub: 'for actor context' }, { step: 'Scored', sub: 'merged profile + organization context' }, - { step: 'Labeled', sub: 'signed label applied to the organization record URI' }, + { step: 'Labeled', sub: 'signed label applied to the actor DID' }, ].map(({ step, sub }, i, arr) => (
@@ -174,8 +174,8 @@ export default function DocsPage() {
  • 4. - A signed AT Protocol label is applied to the organization record URI based on the score tier. - When a record is rescored, the previous label is negated and replaced with the new one. + A signed AT Protocol label is applied to the actor DID based on the score tier. + When the actor is rescored, the previous DID label is negated and replaced with the new one.
  • @@ -190,7 +190,7 @@ export default function DocsPage() { points, plus a configurable actor-PDS trust bonus. The rubric gives less credit to basic identity fields and more weight to harder-to-fake credibility signals like website resolution, location, and visual/profile completeness. Hard test signals override the score to likely-test; validation notes are informational only, - and labels stay attached to the organization record URI. + and labels stay attached to the actor DID.

    @@ -302,7 +302,7 @@ export default function DocsPage() {

    The certified organization labeler exposes a small REST API for the dashboard as well as the standard AT Protocol - labeler XRPC endpoint. Labels are still published against the organization record URI. + labeler XRPC endpoint. Quality labels are published against actor DIDs.

    {API_ENDPOINTS.map(({ method, path, description, curl }) => ( @@ -360,7 +360,7 @@ export default function DocsPage() {
  • - Each label is signed with ed25519 and includes: source DID, record URI, label value, + Each label is signed with ed25519 and includes: source DID, subject DID, label value, timestamp, and a cryptographic signature.
  • @@ -368,15 +368,15 @@ export default function DocsPage() { Apps can subscribe to the labeler to automatically receive organization quality signals for - app.certified.actor.organization records and filter or sort them by tier. The related profile record is - used for merged actor context, not for the label target. + actor DIDs and filter or sort them by tier. The related profile and organization records are + used for merged actor context, not as the label target.
  • - Only one quality label is active per record URI at a time. When a record is updated and - re-scored, the previous label is negated before the new one is applied. + Only one quality label is active per actor DID at a time. When actor context changes and + is re-scored, the previous label is negated before the new one is applied.
  • diff --git a/src/labeler/server.ts b/src/labeler/server.ts index e4ff986..01f2ae7 100644 --- a/src/labeler/server.ts +++ b/src/labeler/server.ts @@ -5,11 +5,16 @@ import logger from './logger' export const labelerServer = new LabelerServer({ did: DID, signingKey: SIGNING_KEY, dbPath: LABELS_DB_PATH }) -export async function fetchCurrentLabels(uri: string): Promise> { +/** + * Returns the active labels for one AT Protocol label subject. + * Subjects are normally actor DIDs; negated labels remove earlier active + * values from the returned set. + */ +export async function fetchCurrentLabels(subjectUri: string): Promise> { await labelerServer.db.execute('SELECT 1') // ensure db is ready const result = await labelerServer.db.execute({ sql: 'SELECT val, neg FROM labels WHERE uri = ? ORDER BY id ASC', - args: [uri], + args: [subjectUri], }) const active = new Set() @@ -25,94 +30,79 @@ export async function fetchCurrentLabels(uri: string): Promise> { return active } -export async function negateAllDIDLabels(): Promise { - await labelerServer.db.execute('SELECT 1') // ensure db is ready - const result = await labelerServer.db.execute({ - sql: 'SELECT DISTINCT uri FROM labels WHERE uri LIKE \'did:%\'', - args: [], - }) - - let negatedCount = 0 - for (const row of result.rows) { - const did = row['uri'] as string - try { - const activeLabels = await fetchCurrentLabels(did) - const activeQualityLabels = [...activeLabels].filter(l => QUALITY_LABEL_IDENTIFIERS.includes(l)) - if (activeQualityLabels.length > 0) { - logger.info({ did, negating: activeQualityLabels }, 'Negating stale DID-level quality labels') - await labelerServer.createLabels({ uri: did }, { negate: activeQualityLabels }) - negatedCount++ - } - } catch (err) { - logger.error({ err, did }, 'Failed to negate DID-level labels, continuing') - } - } - return negatedCount -} - -export async function negateQualityLabels(recordUri: string): Promise { - const existing = uriLocks.get(recordUri) +/** + * Negates all active quality labels on one label subject. + * Use this when an organization actor is deleted and its DID should no longer + * carry an active quality tier. + */ +export async function negateQualityLabels(subjectUri: string): Promise { + const existing = uriLocks.get(subjectUri) const operation = (existing ?? Promise.resolve()).then(async () => { await labelerServer.db.execute('SELECT 1') // ensure db is ready - const currentLabels = await fetchCurrentLabels(recordUri) + const currentLabels = await fetchCurrentLabels(subjectUri) const activeQualityLabels = [...currentLabels].filter(l => QUALITY_LABEL_IDENTIFIERS.includes(l)) if (activeQualityLabels.length === 0) { return 0 } - logger.info({ uri: recordUri, negating: activeQualityLabels }, 'Negating quality labels for deleted record') - await labelerServer.createLabels({ uri: recordUri }, { negate: activeQualityLabels }) + logger.info({ uri: subjectUri, negating: activeQualityLabels }, 'Negating quality labels') + await labelerServer.createLabels({ uri: subjectUri }, { negate: activeQualityLabels }) return activeQualityLabels.length }).finally(() => { - if (uriLocks.get(recordUri) === operation) { - uriLocks.delete(recordUri) + if (uriLocks.get(subjectUri) === operation) { + uriLocks.delete(subjectUri) } }) - uriLocks.set(recordUri, operation) + uriLocks.set(subjectUri, operation) return operation as Promise } const uriLocks = new Map>() -export async function applyQualityLabel(recordUri: string, labelIdentifier: string): Promise { +/** + * Ensures one quality label is active on a label subject. + * Existing quality labels on the same DID are negated before the new value is + * written, so each actor has at most one active quality tier. + */ +export async function applyQualityLabel(subjectUri: string, labelIdentifier: string): Promise { // Validate labelIdentifier before any DB operations if (!QUALITY_LABEL_IDENTIFIERS.includes(labelIdentifier)) { - logger.error({ uri: recordUri, label: labelIdentifier }, 'Rejected unknown label identifier') + logger.error({ uri: subjectUri, label: labelIdentifier }, 'Rejected unknown label identifier') return } // Wait for any in-flight operation on this URI - const existing = uriLocks.get(recordUri) + const existing = uriLocks.get(subjectUri) const operation = (existing ?? Promise.resolve()).then(async () => { - // 1. Fetch current labels for the record URI - const currentLabels = await fetchCurrentLabels(recordUri) + // 1. Fetch current labels for the subject URI + const currentLabels = await fetchCurrentLabels(subjectUri) // 2. Filter to only QUALITY_LABEL_IDENTIFIERS const currentQualityLabels = [...currentLabels].filter(l => QUALITY_LABEL_IDENTIFIERS.includes(l)) - // 3. If record already has the same quality label → return (no change needed) + // 3. If subject already has the same quality label → return (no change needed) if (currentQualityLabels.includes(labelIdentifier)) { - logger.info({ uri: recordUri, label: labelIdentifier }, 'Record already has label, skipping') + logger.info({ uri: subjectUri, label: labelIdentifier }, 'Subject already has label, skipping') return } - // 4. If record has a different quality label → negate it first + // 4. If subject has a different quality label → negate it first if (currentQualityLabels.length > 0) { - logger.info({ uri: recordUri, negating: currentQualityLabels }, 'Negating existing quality labels') - await labelerServer.createLabels({ uri: recordUri }, { negate: currentQualityLabels }) + logger.info({ uri: subjectUri, negating: currentQualityLabels }, 'Negating existing quality labels') + await labelerServer.createLabels({ uri: subjectUri }, { negate: currentQualityLabels }) } // 5. Create the new label - logger.info({ uri: recordUri, label: labelIdentifier }, 'Applying quality label') - await labelerServer.createLabel({ uri: recordUri, val: labelIdentifier }) + logger.info({ uri: subjectUri, label: labelIdentifier }, 'Applying quality label') + await labelerServer.createLabel({ uri: subjectUri, val: labelIdentifier }) }).finally(() => { // Clean up lock if we're still the latest - if (uriLocks.get(recordUri) === operation) { - uriLocks.delete(recordUri) + if (uriLocks.get(subjectUri) === operation) { + uriLocks.delete(subjectUri) } }) - uriLocks.set(recordUri, operation) + uriLocks.set(subjectUri, operation) return operation as Promise } diff --git a/src/labeler/start.ts b/src/labeler/start.ts index 4c73c0d..f27e900 100644 --- a/src/labeler/start.ts +++ b/src/labeler/start.ts @@ -2,7 +2,7 @@ import 'dotenv/config' import fs from 'node:fs' import { HOST, LABELER_PORT, METRICS_PORT, TAP_URL, APP_DB_PATHS, TAP_ADMIN_PASSWORD } from '../lib/config' import { getPendingActivities, deleteActivity } from '../lib/db' -import { labelerServer, negateAllDIDLabels, applyQualityLabel } from './server' +import { labelerServer, applyQualityLabel } from './server' import { startTapConsumer, backfillHfClassification, @@ -103,24 +103,15 @@ async function main() { }) }) - // Wire HF reclassification to also update ATProto labels - setReclassifyCallback(applyQualityLabel) + // Wire HF reclassification to update DID labels. + setReclassifyCallback(async (did, newTier) => { + await applyQualityLabel(did, newTier) + }) // 2. Start metrics server startMetricsServer(METRICS_PORT) logger.info({ port: METRICS_PORT }, 'Metrics server started') - // 2b. Negate any stale DID-level labels from previous deployments - // Fix 1: wrap in try/catch so a failure doesn't crash startup - try { - const negatedCount = await negateAllDIDLabels() - if (negatedCount > 0) { - logger.info({ count: negatedCount }, 'Negated stale DID-level labels') - } - } catch (err) { - logger.error({ err }, 'Failed to negate stale DID-level labels — continuing startup') - } - // Fix 7: clean up stale pending records BEFORE starting tap consumer // so a backfill event can't re-score a record that we're about to delete const pendingRecords = getPendingActivities() @@ -149,7 +140,7 @@ async function main() { logger.info('Tap consumer started — receiving backfill + live events') backfillHfClassification() - // One-time sync: fix any records where DB tier disagrees with ATProto label + // One-time sync: fix any actors where DB tier disagrees with ATProto label // (caused by pre-fix HF reclassifications that only updated the DB) syncLabelsWithDb().catch(err => { logger.warn({ err }, 'Label sync failed — will retry on next restart') diff --git a/src/labeler/tap-consumer.ts b/src/labeler/tap-consumer.ts index 2c9b6be..4ac493b 100644 --- a/src/labeler/tap-consumer.ts +++ b/src/labeler/tap-consumer.ts @@ -16,6 +16,7 @@ import { failRecomputeJob, getRecomputeJobCounts, hasPendingOrganizationDelete, + hasMatchingPendingOrganizationDelete, recoverStaleRunningRecomputeJobs, getProfileSnapshot, getOrganizationSnapshot, @@ -247,7 +248,7 @@ export async function recomputeLabeledOrganizationRow(did: string): Promise { for (const activity of activities) { try { - const currentLabels = await fetchCurrentLabels(activity.uri) - const currentQuality = [...currentLabels].filter(isRuntimeLabelTier) if (!isRuntimeLabelTier(activity.tier)) { continue } + const currentLabels = await fetchCurrentLabels(activity.did) + const currentQuality = [...currentLabels].filter(isRuntimeLabelTier) const dbTier = activity.tier as RuntimeLabelTier - // If the ATProto label doesn't match the DB tier, update it + // If the ATProto DID label doesn't match the DB tier, update it if (!currentQuality.includes(dbTier)) { - logger.info({ uri: activity.uri, dbTier, atprotoLabels: currentQuality }, 'Syncing mismatched label') - await applyQualityLabel(activity.uri, dbTier) + logger.info({ did: activity.did, dbTier, atprotoLabels: currentQuality }, 'Syncing mismatched DID label') + await applyQualityLabel(activity.did, dbTier) synced++ } + } catch (err) { - logger.warn({ err, uri: activity.uri }, 'Failed to sync label, continuing') + logger.warn({ err, did: activity.did, uri: activity.uri }, 'Failed to sync label, continuing') } } if (synced > 0) { - logger.info({ count: synced }, 'Synced mismatched ATProto labels with DB tiers') + logger.info({ count: synced }, 'Synced mismatched DID labels with DB tiers') } else { - logger.info('All ATProto labels match DB tiers') + logger.info('All DID labels match DB tiers') } } diff --git a/src/lib/db.ts b/src/lib/db.ts index f8bd339..ee086bf 100644 --- a/src/lib/db.ts +++ b/src/lib/db.ts @@ -589,6 +589,22 @@ export function hasPendingOrganizationDelete(did: string): boolean { return Boolean(row) } +/** + * Checks whether a queued organization delete still targets the same record. + * Use this after async label cleanup to avoid applying actor-level cleanup for + * a delete that was superseded by a newer organization upsert. + */ +export function hasMatchingPendingOrganizationDelete(did: string, rkey: string, recordUri: string): boolean { + const db = getDb() + const row = db.prepare(` + SELECT 1 + FROM pending_organization_deletes + WHERE did = @did AND rkey = @rkey AND record_uri = @recordUri + LIMIT 1 + `).get({ did, rkey, recordUri }) + return Boolean(row) +} + export function deletePendingOrganizationDelete(did: string): void { const db = getDb() db.prepare('DELETE FROM pending_organization_deletes WHERE did = ?').run(did) diff --git a/src/lib/hf-classifier.ts b/src/lib/hf-classifier.ts index ee3c6aa..8660ba2 100644 --- a/src/lib/hf-classifier.ts +++ b/src/lib/hf-classifier.ts @@ -30,9 +30,13 @@ interface QueueItem { const queue: QueueItem[] = [] let processing = false -type ReclassifyCallback = (uri: string, newTier: string) => Promise +type ReclassifyCallback = (did: string, newTier: string) => Promise let _onReclassify: ReclassifyCallback | null = null +/** + * Registers the labeler-side hook used when HF signals move an actor to a new + * quality tier after the main scoring pass has already stored an activity row. + */ export function setReclassifyCallback(cb: ReclassifyCallback): void { _onReclassify = cb } @@ -105,7 +109,7 @@ function reclassifyWithHfSignal(did: string, rkey: string, classification: Conte }) if (tier !== row.tier && _onReclassify) { - void _onReclassify(row.uri, tier).catch(err => { + void _onReclassify(row.did, tier).catch(err => { console.warn('[hf-classifier] label update failed:', err instanceof Error ? err.message : err) }) } diff --git a/src/lib/types.ts b/src/lib/types.ts index fbad899..fc299ce 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -122,7 +122,7 @@ export interface ActivityLogEntry { id?: number did: string rkey: string - uri: string + uri: string // Organization record URI retained for dashboard links; labels target did. displayName: string score: number tier: LabelTier diff --git a/tests/did-label-subjects.test.ts b/tests/did-label-subjects.test.ts new file mode 100644 index 0000000..083e954 --- /dev/null +++ b/tests/did-label-subjects.test.ts @@ -0,0 +1,106 @@ +import assert from 'node:assert/strict' +import { after, beforeEach, test } from 'node:test' +import { mkdtempSync, rmSync } from 'node:fs' +import { join } from 'node:path' +import { tmpdir } from 'node:os' +import type { OrganizationSnapshot } from '../src/lib/types' + +const testDir = mkdtempSync(join(tmpdir(), 'orglabeler-did-label-subjects-')) +process.env.ACTIVITY_DB_PATH = join(testDir, 'activity-log.db') +process.env.LABELS_DB_PATH = join(testDir, 'labels.db') +process.env.DID = 'did:example:labeler' +process.env.SIGNING_KEY = '01'.repeat(32) +process.env.TAP_URL = 'http://127.0.0.1:65535' +process.env.HF_TOKEN = '' + +const { + closeDb, + getDb, + logActivity, + upsertOrganizationSnapshot, +} = await import('../src/lib/db') +const { fetchCurrentLabels, labelerServer } = await import('../src/labeler/server') +const { recomputeLabeledOrganizationRow, syncLabelsWithDb } = await import('../src/labeler/tap-consumer') + +async function resetLabelDb(): Promise { + await (labelerServer as unknown as { dbInitLock?: Promise }).dbInitLock + await labelerServer.db.execute('DELETE FROM labels') +} + +beforeEach(async () => { + getDb().exec(` + DELETE FROM activities; + DELETE FROM organization_snapshots; + DELETE FROM profile_snapshots; + DELETE FROM pending_organization_deletes; + DELETE FROM recompute_jobs; + DELETE FROM actor_pds_cache; + DELETE FROM url_checks; + `) + await resetLabelDb() +}) + +after(() => { + labelerServer.db.close() + closeDb() + rmSync(testDir, { recursive: true, force: true }) +}) + +function organizationPayload(): OrganizationSnapshot['payload'] { + return { + $type: 'app.certified.actor.organization', + organizationType: ['ngo'], + createdAt: '2024-01-01T00:00:00.000Z', + } +} + +async function storedLabelSubjects(): Promise { + const result = await labelerServer.db.execute('SELECT uri FROM labels ORDER BY id ASC') + return result.rows.map(row => String(row.uri)) +} + +test('recompute applies quality labels to the actor DID, not the organization record URI', async () => { + const did = 'did:example:org-recompute' + const rkey = 'self' + const recordUri = `at://${did}/app.certified.actor.organization/${rkey}` + + upsertOrganizationSnapshot({ + did, + rkey, + recordUri, + payload: organizationPayload(), + }) + + const outcome = await recomputeLabeledOrganizationRow(did) + + assert.ok(outcome) + assert.equal(outcome.tier, 'standard') + assert.deepEqual([...await fetchCurrentLabels(did)], ['standard']) + assert.deepEqual([...await fetchCurrentLabels(recordUri)], []) + assert.deepEqual(await storedLabelSubjects(), [did]) +}) + +test('DB label sync repairs missing actor DID labels without labeling record URIs', async () => { + const did = 'did:example:org-sync' + const rkey = 'self' + const recordUri = `at://${did}/app.certified.actor.organization/${rkey}` + + logActivity({ + did, + rkey, + uri: recordUri, + title: 'Sync Example Organization', + score: 80, + tier: 'high-quality', + breakdown: '{}', + testSignals: '[]', + validationNotes: [], + labeledAt: new Date().toISOString(), + }) + + await syncLabelsWithDb() + + assert.deepEqual([...await fetchCurrentLabels(did)], ['high-quality']) + assert.deepEqual([...await fetchCurrentLabels(recordUri)], []) + assert.deepEqual(await storedLabelSubjects(), [did]) +}) diff --git a/tests/pending-organization-deletes.test.ts b/tests/pending-organization-deletes.test.ts index a0fcfd5..6186e1d 100644 --- a/tests/pending-organization-deletes.test.ts +++ b/tests/pending-organization-deletes.test.ts @@ -15,6 +15,7 @@ const { deletePendingOrganizationDelete, getDb, getOrganizationSnapshot, + hasMatchingPendingOrganizationDelete, hasPendingOrganizationDelete, upsertOrganizationSnapshot, upsertPendingOrganizationDelete, @@ -59,6 +60,18 @@ test('completing a pending delete removes only the matching local organization s assert.equal(hasPendingOrganizationDelete(did), false) }) +test('matching pending delete check requires the same record URI and rkey', () => { + const did = 'did:example:delete-match' + const rkey = 'self' + const recordUri = `at://${did}/app.certified.actor.organization/${rkey}` + + upsertPendingOrganizationDelete(did, rkey, recordUri) + + assert.equal(hasMatchingPendingOrganizationDelete(did, rkey, recordUri), true) + assert.equal(hasMatchingPendingOrganizationDelete(did, 'other', recordUri), false) + assert.equal(hasMatchingPendingOrganizationDelete(did, rkey, `${recordUri}-old`), false) +}) + test('stale pending delete cleanup does not remove a newer replacement snapshot', () => { const did = 'did:example:delete-race' const staleRkey = 'old'