feat(pg_textsearch): add language configuration for BM25 text search …

[liling]

…indexes

Add HINDSIGHT_API_TEXT_SEARCH_LANGUAGE env var to control the text search configuration used by pg_textsearch BM25 indexes. On startup, the system detects mismatches between the configured language and the current index definition, and rebuilds the index automatically when needed.

Supported values: "english" (default), "chinese" (uses zhparser via public.zh_cn config), or any PostgreSQL text search configuration name.

[nicoloboschi]

Overview

Adds HINDSIGHT_API_TEXT_SEARCH_LANGUAGE to switch the BM25 text_config used by pg_textsearch indexes (default english, chinese→public.zh_cn). On startup, _ensure_pg_textsearch_language parses each BM25 index definition and rebuilds on mismatch.

Must-fix

1. Silent index rebuild ignores the data-safety check (migrations.py _ensure_pg_textsearch_language)

The sibling logic in ensure_text_search_extension (lines 1007–1027) explicitly refuses to drop/recreate an index when the table has data and tells the operator how to proceed. The new function does the opposite: DROP INDEX … CREATE INDEX … unconditionally, on every startup, against a potentially huge table, without CONCURRENTLY. Consequences:

• BM25 queries silently fail or seq-scan during the rebuild (pg_textsearch's <@> generally requires the index).
• A single misconfigured env flip on a multi-million row deployment becomes a self-inflicted outage.

At minimum, mirror the row-count guard from the existing function and surface a clear error/migration guidance, or use CREATE INDEX CONCURRENTLY + drop-old-after-swap.

2. Index expression for memory_units diverges from the recreation path

• Initial migration (5a366d414dce) and n9i0j1k2l3m4: USING bm25(text) / USING bm25(content) — single column (matches the comment in a2b3c4d5e6f7: "pg_textsearch doesn't support expressions").
• Existing ensure_text_search_extension recreation block (migrations.py:1072–1083): (COALESCE(text,'') || ' ' || COALESCE(context,'')) for memory_units, (COALESCE(name,'') || ' ' || content) for reflections.
• New _ensure_pg_textsearch_language: (text) for memory_units, (COALESCE(name,'') || ' ' || content) for reflections.

A deployment that went through the existing ensure_text_search_extension path on first boot has an enriched expression index; on the first language change after this PR lands, it gets silently downgraded to a single-column (text) index — losing context from the searchable corpus. The three definitions should be reconciled (and the canonical one extracted into a helper) before merging.

3. No tests

The PR adds non-trivial DB migration logic with multiple branches (extension install, config-creation, mismatch detection, idempotency) and zero tests. Per CLAUDE.md, this is required. Recommended cases:

• Idempotency: running twice with english doesn't drop the index the second time.
• Switch english → chinese rebuilds with public.zh_cn.
• Detection robustness: matches both text_config=english and text_config='english' (both forms appear in real indexdef output).
• No-op when text_search_extension != \"pg_textsearch\".

4. Env var name is broader than its actual scope

HINDSIGHT_API_TEXT_SEARCH_LANGUAGE reads as a general control, but _ensure_pg_textsearch_language only runs in the pg_textsearch branch. The native tsvector path still hardcodes to_tsvector('english', …) (migrations.py:1089–1091, 5a366d414dce_initial_schema.py, a2b3c4d5e6f7_add_text_signals_column.py). A Chinese user on the default native backend will set this var and get no effect. Either (a) thread text_search_language through the native path too, or (b) rename to HINDSIGHT_API_PG_TEXTSEARCH_LANGUAGE and document the pg_textsearch-only scope.

5. Documentation not updated

Per CLAUDE.md ("Adding New API Configuration Flags" → step 5), hindsight-docs/docs/developer/configuration.md already documents HINDSIGHT_API_TEXT_SEARCH_EXTENSION (line 129); add HINDSIGHT_API_TEXT_SEARCH_LANGUAGE next to it with supported values and the pg_textsearch-only caveat.

Should-fix

6. SQL injection / quoting on text_config

config.py accepts any string for text_search_language (the validator only rejects empty), then _ensure_pg_textsearch_language interpolates it into raw SQL: WITH (text_config='{target_config}'). Env-sourced and admin-controlled, but a typo with a single quote breaks the statement, and the PR description says "any PostgreSQL text search configuration name" is accepted. Validate with a strict regex (e.g. ^[A-Za-z_][A-Za-z0-9_.]*$) before interpolation.

7. Index detection is too loose

WHERE indexname LIKE '%text_search%' (existing pattern) will match any index containing that substring, e.g. a future idx_memory_units_text_search_signals. Use the exact name idx_{table}_text_search.

8. No advisory lock around rebuild

Multiple API replicas booting simultaneously will race on DROP INDEX … CREATE INDEX …. The existing ensure_text_search_extension has the same gap, but this code runs on every startup when language was previously different, so the race window is wider. Wrap the rebuild in pg_advisory_lock(_get_schema_lock_id(schema)).

9. Misleading validation comment

```python

Validate text_search_language (informational only — any value is accepted)

if not self.text_search_language:
raise ValueError("text_search_language must not be empty")
```
This is dead code (from_env always supplies DEFAULT_TEXT_SEARCH_LANGUAGE), and the comment contradicts the actual rejection. Either drop the check or replace with the regex from #6.

10. Field annotation comment outdated

config.py:836 — text_search_language: str # \"english\" or \"chinese\". Per the PR description and _language_to_text_config passthrough, any PG text-search config name is allowed. Update the comment.

Nits

• _KNOWN_MAP is a function-local; underscore prefix is the module-private convention. Just KNOWN_MAP (or hoist to module scope as _LANGUAGE_TO_TEXT_CONFIG) is clearer.
• conn.commit() is called three times inside one engine.connect() block; if a later step fails, you've half-committed (zhparser extension/config installed, index not yet rebuilt). Consider one transaction per table, or document the partial-commit recovery path.
• The membership check uses both f\"text_config={target_config}\" and f\"text_config='{target_config}'\" — good, but a regex text_config=\\s*'?([^'\\s)]+)'? would be more robust to whitespace/format differences across PG versions.

Risk summary

The feature itself is reasonable, but the rebuild path is currently a foot-gun (#1) and the index-expression divergence (#2) introduces silent corpus loss. Combined with no tests and missing docs, this isn't ready to land. Once #1–#5 are addressed, the rest are small.

[liling]

I've found that vchord offers excellent support for multilingual BM25 search, so there’s no need to tweak pg_textsearch anymore.