docs(adr): ADR-0061 record search architecture + liveness audit

docs/adr/0061-record-search-architecture.md

@@ -0,0 +1,90 @@
+# ADR-0061: Record Search Architecture — metadata-driven `$search` resolution
+
+**Status**: Proposed — recommended for acceptance; nothing implemented yet (2026-06-21)
+**Deciders**: ObjectStack Protocol Architects
+**Builds on**: ADR-0045 (additive materialization & visibility gate — search must be visibility-aware), ADR-0049 (enforce-or-remove / no unenforced declaration), ADR-0054 (runtime proof per enforced surface), ADR-0060 (conformance ledger as a platform pattern)
+**Consumers**: spec, objectql, driver-sql, driver-memory, driver-mongodb, rest, objectui (`fields`, `plugin-list`, `app-shell`), verify, dogfood
+**References**: `docs/audits/2026-06-record-search-liveness.md`; objectui #1860 / #1863 / framework #2125 / #2128 (lookup picker UX that exposed the gap)
+
+## TL;DR
+
+`$search` is a textbook **declared-but-unenforced** surface: `FullTextSearchSchema` is defined and **every** client (lookup picker, list quick-search, ⌘K) sends `$search`, yet **no engine or driver executes it** — a silent no-op. Only the public-form picker actually filters; `$searchFields` is already sent by the client but undocumented and unhonored; three object/view search-metadata shapes have zero data-layer consumers.
+
+Decision: **enforce** record search as one **metadata-driven, server-resolved** capability. The client sends only the query text; the **server resolves which fields to search from object metadata**; a **single resolver feeds all surfaces**, behind a **two-tier executor** (driver `contains` now; native FTS / external engine / relevance later). Migration is **additive** — it turns a no-op into a filter. Ship **Tier 1 + the contract + unify lookup**; defer Tier 2 (index / relevance / server `searchAll`); keep knowledge/vector a separate subsystem.
+
+> **North star**: the client says *what to search for*; the server decides *which fields, how to rank, and what may be seen*. One contract for every surface; the executor is swappable.
+
+## Context
+
+### The declared-but-unenforced failure, again
+ObjectStack's defining risk is not a crash but a primitive that is *declarable* yet *inert* (ADR-0049, ADR-0060). Record search is a clean instance: the `docs/audits/2026-06-record-search-liveness.md` evidence shows `$search` (`spec/.../query.zod.ts:454`) parsed but executed by no driver (`driver-sql`/`memory` both `fullTextSearch: false`), so it returns rows unfiltered. AI authorship amplifies this — a model emits `searchableFields` that *looks* wired and nothing reads it.
+
+### The requirement is platform-wide, not "lookup multi-field search"
+Search surfaces, all sending the same `$search` today:
+- **Lookup picker** (`fields/LookupField`, `RecordPickerDialog`) — type-ahead + dialog.
+- **List quick-search** (`plugin-list/ListView`) — already sends `$searchFields` (drift).
+- **Global / ⌘K** (`app-shell/CommandPalette` → `react/useRecordSearch`) — client fan-out + client ranking.
+- **Public-form picker** (`rest-server.ts:3751`) — the only working, scoped path.
+- **Structured filters** (FilterUI, grid/report filter bars) — `$filter`/`$contains`, *not* full-text; out of scope here.
+- **Knowledge / RAG / vector** — separate semantic subsystem; out of scope here.
+
+Three object/view metadata shapes overlap (`object.searchable`, `object.search`, `view.searchableFields`) with no data-layer consumer — so "collapse to one source of truth" is itself a decision.
+
+## Decision
+
+### D1 — Contract: client sends the query; server resolves fields; `$searchFields` is a *validated override*
+The default contract is `$search: string` (plus existing `$filter`/scope). **Field resolution is server-side from metadata.** `$searchFields` is **promoted into the formal `QueryParams`/`FullTextSearchSchema` contract** as an *optional* override and **the server intersects it with the object's allowed searchable set**, silently dropping disallowed fields. Client-decided fields are never the primary path and never widen beyond what metadata permits.
+
+### D2 — Single source of truth for searchable fields
+- `object.searchableFields: string[]` is the **canonical** list (shorthand that feeds `object.search.fields`).
+- `object.searchable: boolean` is retained as the **on/off gate** (does this object participate in search at all) — orthogonal to *which* fields.
+- `view.searchableFields` is a **narrowing override** for a specific surface.
+- **Resolution order**: validated `$searchFields` → `view.searchableFields` → `object.searchableFields` → **auto-default** (the name/title field + short-text fields: `text`/`email`/`phone`/`url`; excluding long-text/json/secret/system).
+- Collapse the duplicate field list inside `SearchConfigSchema` (it keeps only Tier-2 knobs: analyzer/boost/provider). Field-level `searchable` stays pruned.
+
+### D3 — Two-tier executor behind one `$search` API
+- **Tier 1 (this phase)**: the engine expands `$search` into a cross-field predicate pushed to the driver — `driver-sql` reuses existing `$or` + `$contains` (ILIKE, wildcard-escaped); `driver-memory` does in-memory `contains`. New driver capability `textSearch: true`. Matching is **case-insensitive; terms AND-ed, fields OR-ed** (each term must hit some field) — i.e. `FullTextSearchSchema.operator` default. Default is **contains** (not prefix). When a driver lacks `textSearch`, the **engine applies a bounded in-memory fallback** (degraded scale, logged) so search works everywhere.
+- **Tier 2 (deferred)**: `pg_trgm` (makes `contains` indexable) / `tsvector` ranked full-text and the external providers in `system/search-engine.zod`; relevance / `boost` / `minScore` / `fuzzy` live here.
+
+### D4 — label↔value and ranking resolve server-side
+- For `select`/`status` fields the resolver maps the query to **option values via their (i18n) labels** before matching (Tier 1 — metadata is in-object and cheap). For `lookup`/`reference` fields, searching the related display name needs a join → **Tier 2** (or a denormalized display column).
+- **Per-object relevance ranking is server-side (Tier 2)**; the client retains only the **cross-object merge sort** for global fan-out (unavoidable without a unified index).
+
+### D5 — Security: search is a thin layer over the gated `find()`
+Search runs through the **same query pipeline as `find()`** — RLS, field-level security, and row filters apply automatically; **no separate unguarded search path**. Results respect the **ADR-0045 materialization/visibility gate** (draft vs published). The `$searchFields` override is subset-validated (D1). **Secret / encrypted / PII fields are never searchable** (excluded from default, rejected from override). Public/anonymous search keeps the existing `publicPicker` model (projection + `maxResults` ≤ 50, no enumeration). LIKE wildcards are escaped (driver-sql already does).
+
+### D6 — Global search: fan-out now, unified `searchAll` is Tier 2
+Global search this phase = **client fan-out over the per-object Tier-1 resolver** (CommandPalette already fans out; it only needs per-object search to actually filter) — zero new server surface. The unified server `searchAll` with cross-object relevance is **Tier 2** (it needs a unified index to rank well). **Knowledge/vector stays a separate subsystem**; "blended" record+knowledge search is future and out of scope, but API-compatible.
+
+### D7 — Migration: one resolver, additive rollout
+All surfaces converge on `$search` (+ validated `$searchFields`); client-side field-guessing and the lookup's hardcoded `label+description` local filter are removed; CommandPalette keeps only its cross-object merge. Because every surface **already sends `$search` (today a no-op)**, enabling server execution is **backward-compatible** — it converts a no-op into a filter.
+
+### Default behaviour (normative)
+| Dimension | Default |
+|---|---|
+| Client sends | `$search` string (+ optional validated `$searchFields`) |
+| Field resolution order | `$searchFields` (validated) → view → `object.searchableFields` → auto (name + short-text) |
+| Matching | case-insensitive; terms AND-ed, fields OR-ed; `contains` |
+| `select`/`status` | query mapped to option values via labels (Tier 1) |
+| Ranking | per-object server-side (Tier 2); cross-object merge client-side |
+| Security | over RLS + ADR-0045 visibility; secret/PII never searchable; override subset-validated |
+| Execution | Tier 1 driver `contains` + engine in-memory fallback; Tier 2 trigram/FTS/external |
+
+### Conformance (ADR-0060 alignment)
+`$search` is **"landed" iff** a driver executes it **and** a dogfood proof asserts a multi-field match over the real HTTP API — e.g. searching `"retail"` returns an account matched by `industry`, not `name`. Add a `search-conformance` ledger entry; CI fails if `$search` is declared but no proof resolves (closing the declared-but-unenforced loop).
+
+## Phasing
+- **P1 — contract + liveness**: add `object.searchableFields`, formalize `$searchFields` in `QueryParams`/`FullTextSearchSchema`, collapse the duplicate metadata, add the conformance ledger entry.
+- **P2 — executor**: objectql `$search` resolver + `driver-sql` `contains` + `driver-memory` fallback + `select` label→value; dogfood proof.
+- **P3 — unify**: point lookup / list / ⌘K at the one resolver; strip client field-guessing & local filters; auto-default fields; configure flagship `searchableFields` in showcase.
+- **P4 — scale (deferred)**: `pg_trgm`/`tsvector` indexes, relevance/`boost`/`fuzzy`, external engines, server `searchAll`, blended knowledge search.
+
+## Rejected alternatives
+- **Renderer-side `$or`/`$contains` across the picker's columns** — duplicates "what's searchable" in the client, leaks value↔label and storage semantics to the UI, doesn't help list/global, can't be indexed, and becomes drift the moment Tier 2 lands. Acceptable *only* as a throwaway P0 stopgap, deleted by P2.
+- **Client-decided `$searchFields` as the primary mechanism** — a field-probe oracle, and the cause of today's drift; inconsistent across surfaces.
+- **A bespoke per-object search service separate from `find()`** — bypasses RLS / visibility and re-implements query + security.
+- **Jump straight to an external engine (ES/Algolia)** — premature at current scale; Tier 1 covers the 90% case; keep the seam for when a tenant needs it.
+
+## Consequences
+- **(+)** One consistent, secure, metadata-driven search across every surface; zero-config sensible default; additive, low-risk rollout; a clear, non-blocking path to FTS / relevance / external engines; closes a declared-but-unenforced gap with a conformance proof.
+- **(−)** Tier 1 unindexed ILIKE is O(n) on large tables (mitigated by Tier-2 trigram); `lookup`/`reference` display-name search is deferred to Tier 2; cross-object relevance is limited until a unified index exists.

docs/audits/2026-06-record-search-liveness.md

@@ -0,0 +1,58 @@
+# Audit: Record search liveness & architecture
+
+**Date**: 2026-06-21
+**Scope**: the end-to-end **record search** capability — the generic data-query `$search` / `FullTextSearchSchema` and every surface that invokes it, across framework (spec, objectql, rest, drivers, services) and objectui (lookup picker, list view, command palette).
+**Method**: cross-reference each search surface's definition against its **actual consumer** (`file:line`) in two layers — framework runtime and objectui renderers. **LIVE** = a consumer reads it and changes behaviour; **DEAD** = parsed/sent but no executor acts on it → a silent no-op for authors/callers. Browser observation is deliberately *not* the signal (tiny datasets make a no-op look like a working search).
+
+> Evidence catalog intended to seed an ADR (record search architecture). It states what is wired; the decision lives in **ADR-0061**.
+
+## Headline
+
+Record search is a **declared-but-unenforced** capability (cf. ADR-0049). `$search` is defined in the spec and **sent by every client surface**, but **no driver or engine executes it** — it is a silent no-op for generic objects. The only server path that actually filters is the public-form picker. Meanwhile `$searchFields` is already sent by the client but is undocumented and unhonored (drift), and three overlapping object/view search-metadata shapes exist with **zero** data-layer consumers.
+
+| Bucket | State |
+|---|---|
+| `$search` execution (engine + drivers) | **DEAD** — no-op |
+| `$searchFields` (client → server) | **DRIFT** — sent, undocumented, unhonored |
+| object / view search metadata | **PARTIAL** — admin UI only; data layer ignores |
+| public-form lookup picker | **LIVE** — the only working path |
+| global `/search` (`searchAll`) | **PLANNED** — 501 |
+| knowledge / RAG / vector | **LIVE but separate** (documents, not CRUD) |
+| external FTS engines | **PLANNED** — schema only |
+
+## 1. Generic `$search` — DEAD
+- **Defined**: `packages/spec/src/data/query.zod.ts:454` `FullTextSearchSchema { query, fields, fuzzy, operator, boost, minScore, language, highlight }`; wired into `BaseQuerySchema` at `:525`.
+- **Not executed**: no path in `packages/objectql/src/engine.ts` or `protocol.ts`; `driver-sql` `fullTextSearch: false` (`packages/plugins/driver-sql/src/sql-driver.ts:166`); `driver-memory` `fullTextSearch: false` (`memory-driver.ts:136`); mongodb no `$text`. → `$search` returns rows **unfiltered**.
+
+## 2. `$searchFields` — DRIFT
+- objectui ListView sends `$searchFields: schema.searchableFields` (`packages/plugin-list/src/ListView.tsx:1001`) — **not** in the `QueryParams` contract (`packages/types/src/data.ts`), and the backend does not honor it.
+
+## 3. Object / view search metadata — PARTIAL
+- `object.searchable` boolean (`object.zod.ts:52`) — consumed by metadata-admin UI + the command-palette object gate; **not** the data layer.
+- `object.search` (`SearchConfigSchema`, `object.zod.ts:113`) — `fields`/`displayFields`/`filters`; **no runtime consumer**.
+- `view.searchableFields` (`view.zod.ts:522`) — UI input control + ListView; not the data layer.
+- field-level `searchable` — **DEAD** (per `2026-06-fieldschema-property-liveness.md`).
+
+## 4. Lookup search (objectui) — sends `$search`, gets a no-op
+- `LookupField.tsx:311` / `RecordPickerDialog.tsx:451` — bare `$search`. The picker "works" visually only because datasets are tiny; there is **no server filtering**.
+- `$filter` `$contains` IS live (user filter bar / `lookupFilters`) — driver-sql implements `$or`/`$contains` (`sql-driver.ts:1919/1962`). This is the lever the ADR's Tier 1 builds on.
+
+## 5. Global search — client fan-out + PLANNED server
+- CommandPalette → `useRecordSearch` (`packages/react/src/hooks/useRecordSearch.ts:235`) fans out bare `$search` per object and ranks **client-side** (`scoreHit`); object gate via `searchable !== false` (`:195`).
+- Server `GET /search` (`searchAll`) skeleton (`packages/rest/src/rest-server.ts:3309`) → 501.
+
+## 6. Public-form picker — LIVE (only working path)
+- `GET /forms/:slug/lookup/:field` (`rest-server.ts:3751`) — `publicPicker` projection + base `filter` + `maxResults` ≤ 50 (anti-enumeration).
+
+## 7. Heavier search — separate / planned
+- `service-knowledge` `IKnowledgeService` (RAG) — LIVE but **document-scoped**, not object CRUD.
+- vector search — PLANNED (`driver.zod.ts:236`).
+- external engines (Elasticsearch / Algolia / MeiliSearch / Typesense) — declared in `system/search-engine.zod`, no runtime.
+
+## Drift / duplication (why "consistency" is itself a requirement)
+- ListView sends `$searchFields`; lookup & command-palette don't → three surfaces, three behaviours.
+- LookupField carries a hardcoded client-side `label + description` local filter (static-options path).
+- Three object/view search-metadata shapes, **zero** data-layer consumers.
+
+## Recommendation
+Treat as an **enforce-or-remove** (ADR-0049) case: either wire `$search` end-to-end through a single **metadata-driven server resolver** (recommended) or mark it experimental. The decision, default behaviour, phasing, and rejected alternatives are in **ADR-0061**.