Vector chunk index has no lifecycle: orphaned chunks survive restores and reindexes, with no safe recreate path

[tomasmontielp]

Problem

The dedicated vector chunk index (data_asset_embeddings_chunks, introduced by the multi-chunk embedding work) is written per entity and cleaned only by entity-delete events. That leaves it with no maintainable lifecycle:

1. Orphaned chunks are immortal. Entities removed without delete events (DB restore, wipe-and-remigrate, environment clone) leave chunks behind that no reindex ever revisits — a full recreate rebuilds entity indexes from the DB, but entities that no longer exist produce no writes and no deletes. AI retrieval (context finder, RAG, attached knowledge) keeps surfacing this ghost knowledge indefinitely.
2. Fingerprint skipping leaves even live entities' chunks untouched when their content is unchanged, so routine reindexing can never sweep the index either.
3. There is no product-level recovery path. The only remedy for orphans — and for an incompatible chunk-mapping upgrade, which wedges in a retry loop — is manual OpenSearch surgery: stop the server, DELETE the index, restart, reindex.
4. Naive fixes are dangerous. Dropping the index at the start of a recreate turns any mid-run failure (misconfigured embedding client, provider outage, job crash) into a retrieval outage with nothing to roll back to; and a raw chunk write against a missing index auto-creates it with dynamic mappings (embedding as float, not knn_vector), breaking every vector and hybrid query until manual repair.

Fix (#30049)

Staged, generation-based recreate behind the read alias:

• A full-recreate run writes into a bare next-generation index (<base>_g<N>), invisible to reads; the old chunks stay live for the entire run.
• Promotion is a single atomic _aliases call (read + search aliases attached, previous target removed) and fires only after every vector-indexable entity type finalizes successfully.
• An embedding-client pre-flight runs before anything is created; a failed entity type abandons the staged generation immediately; generations orphaned by crashed runs are swept at the next staged recreate.
• Distributed workers resolve the staged write target from cluster state, so no cross-node coordination is needed; live-update writers always target the read alias.
• Chunk writes refuse to proceed when the index cannot be ensured, closing the wrong-mapping auto-creation trap.
• The legacy layout migrates in place: the first promotion atomically replaces the physical index with an alias of the same name (remove_index + add in one call), so nothing needs reconfiguring.

Net effect: there is no failure path that loses the old chunks before their replacements are ready, and the "recreate" recovery an operator would naturally reach for now actually recovers — including from the mapping-wedge case.

Residual (known, bounded)

A JVM crash mid-run leaves an orphan generation until the next staged recreate sweeps it; reindex-sink writes from other nodes can land in that orphan during the window. Live-path writes are unaffected, and chunk-0 / entity-doc retrieval is unaffected throughout.