feat(search): generic pinyin recall via locale-gated __search companion column (#3027)

Implements #2486 / ADR-0097 on top of ADR-0061 Tier-1 $search, purely additive:

- OS_SEARCH_PINYIN_ENABLED platform switch with locale-derived default (any zh-* locale turns it on); CLI serve resolves once and stamps the decision into the env so the SchemaRegistry seam and the plugin gate agree.
- SchemaRegistry provisions a hidden `__search` companion column for the ADR-0079 display/name field at object compile time; FLS-restricted / secret / hidden source fields are excluded fail-closed (ADR-0061 D5).
- New @objectstack/plugin-pinyin-search fills the blob (full pinyin + initials, "zhangwei zw") via lazy-loaded pinyin-pro on before-save, with paged idempotent boot backfill and a rebuildSearchCompanion reconcile entry.
- expandSearchToFilter ORs the companion into each latin term's clause; resolveSearchFields unchanged; zero driver changes.

Verified end-to-end on the showcase app: $search=zhangwei / zw / 张 all recall 张伟; latin and CJK source-column behavior unchanged.

Closes #2486

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMugRriiNv3nsnmLczfhuY

Author: os-zhuang

.changeset/config.json

@@ -41,6 +41,7 @@
       "@objectstack/plugin-email",
       "@objectstack/plugin-hono-server",
       "@objectstack/mcp",
+      "@objectstack/plugin-pinyin-search",
       "@objectstack/plugin-reports",
       "@objectstack/plugin-security",
       "@objectstack/plugin-sharing",

.changeset/pinyin-search-companion.md

@@ -0,0 +1,18 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/plugin-pinyin-search': minor
+'@objectstack/types': minor
+'@objectstack/cli': minor
+---
+
+Generic pinyin search recall (#2486, ADR-0097): a locale-gated
+`OS_SEARCH_PINYIN_ENABLED` switch (auto-on when the stack configures any
+`zh-*` locale) provisions a hidden `__search` companion column for each
+object's display/name field at compile time, the new
+`@objectstack/plugin-pinyin-search` fills it with full pinyin + initials
+("张伟" → "zhangwei zw") on before-save (plus boot backfill and a
+`rebuildSearchCompanion` reconcile entry), and `$search` ORs the column in at
+query time — so lookup pickers, list quick-search and ⌘K transparently match
+`zhangwei` / `zw` against CJK names. Purely additive: `resolveSearchFields`,
+`searchableFields`, drivers and non-Chinese deployments are untouched; FLS
+restricted / secret / PII fields never feed the companion.

docs/adr/0097-pinyin-search-companion-column.md

@@ -0,0 +1,98 @@
+# ADR-0097: Pinyin Search via a Locale-Gated Companion Column
+
+- **Status**: Accepted
+- **Date**: 2026-07-16
+- **Issue**: #2486
+- **Extends**: ADR-0061 (record search architecture — purely additive), ADR-0045 (additive materialization), ADR-0079 (display-name contract)
+
+## Context
+
+Pinyin search is a cross-cutting requirement for Chinese deployments: user
+pickers (`sys_user`), accounts, products, departments — any object with CJK
+names — must be findable by typing full pinyin (`zhangwei`) or initials
+(`zw`). ADR-0061 Tier-1 `$search` expands to `$contains` over source columns,
+which can never match: the stored value is the CJK original. Hand-rolling
+denormalized pinyin columns per object would copy the same hack N times.
+
+Database tokenizers were investigated and rejected: Turso Cloud does not load
+traditional C extensions (no `simple` pinyin tokenizer), neither SQLite FTS5
+nor Turso's Tantivy engine ships native pinyin, and each dialect's FTS is
+mutually incompatible (FTS5 / tsvector·pg_trgm / Tantivy). A denormalized
+companion column + `$contains` is dialect- and engine-agnostic, and a future
+Tier-2 native FTS simply indexes it as one more column.
+
+Of the four input shapes against CJK records, only two need anything new:
+CJK text and other-field originals (email, codes) already hit source columns
+via `$contains`. Full pinyin and initials do not. So the entire net increment
+is: **one normalized companion column for the name field, OR-ed in at query
+time**.
+
+## Decision
+
+1. **Locale-gated platform switch, no field metadata.**
+   `OS_SEARCH_PINYIN_ENABLED` (resolved by `resolveSearchPinyinEnabled()` in
+   `@objectstack/types`) gates the feature end-to-end. When unset, the CLI
+   boot path derives the default from the stack's configured locales (any
+   `zh-*` → on) and stamps the decision back into the env var so every
+   consumer — the per-engine `SchemaRegistry` and the plugin gate — reads the
+   same single decision. No field-level `pinyin` marker exists, so there is
+   no declared-but-unenforced dead metadata (ADR-0049) and no half-state
+   where a field "pretends" to support pinyin.
+
+2. **Materialization set ≠ search set: one column per object.** Only the
+   ADR-0079 display/name field (`resolveDisplayField`) feeds the hidden
+   companion column `__search` — `provisionSearchCompanion` in
+   `packages/objectql/src/search-companion.ts`, invoked from
+   `SchemaRegistry.registerObject` right after `provisionPrimary`. The column
+   is `hidden` / `readonly` / `system` / `searchable: false` / `index: true`;
+   migration is a plain additive column (ADR-0045). `searchableFields` and
+   the auto-default are untouched — text search stays wide via source
+   columns, pinyin materialization stays narrow.
+
+3. **One blob recalls both shapes.** The column stores full pinyin AND
+   initials (`"张伟"` → `"zhangwei zw"`), so a single `$contains` recalls
+   `zhang`/`wei`/`zhangwei` (full-form substrings) and `zw` (initials).
+   Relevance ranking (full-pinyin first) and short-initial noise guards are
+   Tier-2 (native FTS) — deliberately out of scope, hence one column, not two.
+
+4. **Plugin fills, never declares** (the `plugin-sharing` primary-BU
+   projection pattern). `@objectstack/plugin-pinyin-search` binds global
+   `beforeInsert`/`beforeUpdate` hooks that recompute the blob (lazy-loaded
+   `pinyin-pro`, default polyphone heuristics) **only when the source field is
+   in the write** — no write amplification. A non-CJK new value clears the
+   blob (no stale recall).
+
+5. **Query-time is purely additive.** `expandSearchToFilter`
+   (`packages/objectql/src/search-filter.ts`) ORs
+   `{ __search: { $contains: term } }` into each latin term's clause when the
+   object carries the column. `resolveSearchFields` is unchanged; the
+   companion is invisible to `$searchFields` overrides and clients. CJK terms
+   skip the clause (they hit source columns). `$or` + `$contains` are
+   supported by every driver — zero driver changes.
+
+6. **Security guardrail (ADR-0061 D5 extended).** Only fields readable by
+   every accessor may feed the shared companion: fields with
+   `requiredPermissions` (FLS), `hidden` fields, and secret/virtual types are
+   excluded at the single eligibility gate (`isCompanionSourceEligible`) —
+   otherwise "which search hits" becomes an FLS-bypass inference oracle.
+   Fail-closed, not author-remembered.
+
+7. **Write-path fallback.** Hook-bypassing writes (bulk import, direct
+   migration) leave the blob empty → `backfillSearchCompanion` runs once per
+   boot (`kernel:bootstrapped`, paged, idempotent) and
+   `rebuildSearchCompanion` is the explicit reconcile/rebuild entry.
+
+## Generalization seam
+
+The companion is one instance of "search-normalization companion":
+`SEARCH_COMPANION_NORMALIZERS = ['pinyin']` names the applied normalizers.
+Future normalizers (simplified/traditional conversion, width folding, accent
+folding) extend the same list and column; only pinyin is implemented.
+
+## Non-goals
+
+- Relevance ranking, typo tolerance, whole-word guards → Tier-2 (native FTS).
+- Materializing non-name fields or a search blob; changing
+  `searchableFields`/auto-default behavior.
+- Surname polyphone dictionaries (P2; `pinyin-pro` heuristics accepted).
+- Turso/SQLite tokenizers or loadable extensions.

packages/cli/package.json

@@ -62,6 +62,7 @@
     "@objectstack/plugin-hono-server": "workspace:*",
     "@objectstack/plugin-reports": "workspace:*",
     "@objectstack/plugin-security": "workspace:*",
+    "@objectstack/plugin-pinyin-search": "workspace:*",
     "@objectstack/plugin-sharing": "workspace:*",
     "@objectstack/plugin-webhooks": "workspace:*",
     "@objectstack/rest": "workspace:*",

packages/cli/src/commands/serve.ts

@@ -8,7 +8,7 @@ import chalk from 'chalk';
 import { bundleRequire } from 'bundle-require';
 import { loadConfig, BUNDLE_REQUIRE_EXTERNALS } from '../utils/config.js';
 import { isHostConfig, shouldBootWithLibrary } from '../utils/plugin-detection.js';
-import { readEnvWithDeprecation, resolveMultiOrgEnabled, resolveAllowDegradedTenancy, isMcpServerEnabled } from '@objectstack/types';
+import { readEnvWithDeprecation, resolveMultiOrgEnabled, resolveAllowDegradedTenancy, isMcpServerEnabled, resolveSearchPinyinEnabled } from '@objectstack/types';
 import { resolveObjectStackHome } from '@objectstack/runtime';
 import { LOG_LEVELS, resolveLogLevel, readLogLevelEnv } from '../utils/log-level.js';
 import {
@@ -486,6 +486,25 @@ export default class Serve extends Command {
       if (isMcpServerEnabled() && !requires.includes('mcp')) {
         requires.push('mcp');
       }
+      // Pinyin search recall (#2486): locale-gated platform capability. When
+      // `OS_SEARCH_PINYIN_ENABLED` is unset, the default derives from the
+      // stack's configured locales (any `zh-*` → on). This is the ONE place
+      // that sees the stack config, so the resolved decision is stamped back
+      // into the env var — every later consumer (each engine's SchemaRegistry
+      // provisioning the `__search` companion column, the plugin's own gate)
+      // reads the same answer via the no-arg `resolveSearchPinyinEnabled()`.
+      {
+        const i18nCfg = (config as any).i18n ?? {};
+        const configuredLocales = [
+          i18nCfg.defaultLocale,
+          i18nCfg.fallbackLocale,
+          ...(Array.isArray(i18nCfg.supportedLocales) ? i18nCfg.supportedLocales : []),
+        ].filter((l: unknown): l is string => typeof l === 'string');
+        if (resolveSearchPinyinEnabled({ locales: configuredLocales })) {
+          process.env.OS_SEARCH_PINYIN_ENABLED = 'true';
+          if (!requires.includes('pinyin-search')) requires.push('pinyin-search');
+        }
+      }
       // Default capability slate — every preset except `minimal` gets the
       // foundational services (queue + job + cache + settings + email +
       // storage). Opt out with `objectstack serve --preset minimal`.
@@ -1812,6 +1831,13 @@ export default class Serve extends Command {
           export: 'SharingServicePlugin',
           nameMatch: ['plugin-sharing', 'SharingServicePlugin', 'SharingPlugin'],
         },
+        // #2486 — auto-required above when resolveSearchPinyinEnabled()
+        // (explicit env, else any configured zh-* locale) says on.
+        'pinyin-search': {
+          pkg: '@objectstack/plugin-pinyin-search',
+          export: 'PinyinSearchPlugin',
+          nameMatch: ['plugin-pinyin-search', 'PinyinSearchPlugin'],
+        },
         reports: {
           pkg: '@objectstack/plugin-reports',
           export: 'ReportsServicePlugin',

packages/objectql/src/core.ts

@@ -23,6 +23,18 @@ export {
 } from './registry.js';
 export type { ObjectContributor, SchemaRegistryOptions } from './registry.js';
 
+// Search-normalization companion column (#2486 — pinyin recall)
+export {
+  SEARCH_COMPANION_FIELD,
+  SEARCH_COMPANION_NORMALIZERS,
+  provisionSearchCompanion,
+  resolveSearchCompanionSources,
+  isCompanionSourceEligible,
+  isCompanionMatchableTerm,
+  containsCJK,
+} from './search-companion.js';
+export type { CompanionFieldMeta, CompanionObjectMeta } from './search-companion.js';
+
 // Engine
 export { ObjectQL, ObjectRepository, ScopedContext } from './engine.js';
 export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, EngineMiddleware } from './engine.js';

packages/objectql/src/index.ts

@@ -12,6 +12,20 @@ export {
 } from './registry.js';
 export type { ObjectContributor, SchemaRegistryOptions } from './registry.js';
 
+// Search-normalization companion column (#2486 — pinyin recall). Shared by
+// the registry's compile-time provisioning seam, the engine's `$search`
+// expansion, and plugin-pinyin-search's populate hooks.
+export {
+  SEARCH_COMPANION_FIELD,
+  SEARCH_COMPANION_NORMALIZERS,
+  provisionSearchCompanion,
+  resolveSearchCompanionSources,
+  isCompanionSourceEligible,
+  isCompanionMatchableTerm,
+  containsCJK,
+} from './search-companion.js';
+export type { CompanionFieldMeta, CompanionObjectMeta } from './search-companion.js';
+
 // Export Protocol Implementation
 export { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol';
 

packages/objectql/src/registry.ts

@@ -1,7 +1,8 @@
 // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
 
 import { ServiceObject, ObjectSchema, ObjectOwnership, provisionPrimary } from '@objectstack/spec/data';
-import { resolveMultiOrgEnabled } from '@objectstack/types';
+import { resolveMultiOrgEnabled, resolveSearchPinyinEnabled } from '@objectstack/types';
+import { provisionSearchCompanion } from './search-companion.js';
 import { ObjectStackManifest, ManifestSchema, InstalledPackage, InstalledPackageSchema } from '@objectstack/spec/kernel';
 import { AppSchema } from '@objectstack/spec/ui';
 import { applyProtection } from '@objectstack/spec/shared';
@@ -142,6 +143,17 @@ export interface SchemaRegistryOptions {
    */
   multiTenant?: boolean;
 
+  /**
+   * Whether to provision the hidden `__search` search-normalization companion
+   * column on registered objects that have an eligible display/name field
+   * (#2486 — pinyin search recall). Sourced from `OS_SEARCH_PINYIN_ENABLED`
+   * via `resolveSearchPinyinEnabled()` when not explicitly set (the CLI boot
+   * path stamps the locale-derived decision into that env var before the
+   * kernel constructs engines). Default off — non-Chinese deployments carry
+   * no extra column. Pass an explicit boolean to override (useful in tests).
+   */
+  searchCompanion?: boolean;
+
   /**
    * Policy for the install-time namespace gate (ADR-0048 Phase 1) — installing
    * a package whose `manifest.namespace` is already owned by a *different*
@@ -412,6 +424,9 @@ export class SchemaRegistry {
   /** Whether to auto-inject multi-tenant system fields. */
   private readonly multiTenant: boolean;
 
+  /** Whether to provision the `__search` companion column (#2486). */
+  private readonly searchCompanion: boolean;
+
   /** Cross-package base-layer collision policy (ADR-0048). */
   private readonly collisionPolicy: 'error' | 'warn';
 
@@ -423,6 +438,12 @@ export class SchemaRegistry {
       this.multiTenant = resolveMultiOrgEnabled();
     }
 
+    // Pinyin-search companion column (#2486). Env-driven like multiTenant;
+    // the CLI boot path resolves the locale-derived default once and stamps
+    // it into OS_SEARCH_PINYIN_ENABLED so this read agrees with the plugin
+    // gate.
+    this.searchCompanion = options.searchCompanion ?? resolveSearchPinyinEnabled();
+
     // ADR-0048 — default to a loud error on cross-package collision; allow an
     // env opt-out for deliberate migrations.
     this.collisionPolicy =
@@ -581,6 +602,18 @@ export class SchemaRegistry {
     // resolving their title on read via `resolveDisplayField` / `titleFormat`.
     if (ownership === 'own') {
       schema = provisionPrimary(schema, { synthesize: false });
+
+      // [#2486] Search-normalization companion column (`__search`). Runs
+      // AFTER `provisionPrimary` so the just-designated `nameField` is
+      // visible, and ONLY when the deployment enables pinyin search — the
+      // column is a real additive migration (ADR-0045) that the driver's
+      // `syncSchema` materializes, populated by plugin-pinyin-search's
+      // before-save hooks and OR-ed into `$search` by the engine's
+      // `expandSearchToFilter`. Owned objects only: extensions merge into
+      // the owner's already-provisioned shape.
+      if (this.searchCompanion) {
+        schema = provisionSearchCompanion(schema);
+      }
     }
 
     const shortName = schema.name;