feat: preserve indexdata during serialization

[BorysTheDev]

Summary: This PR makes HNSW (global) vector indices safe to serialize when vectors are stored as external pointers into HASH StringMap entries (copy_vector=false).

Changes:

• Extend HNSW remove APIs to report whether a remove ran synchronously or was deferred under MRMW read locks.
• Add DrainPendingOps() to block on the HNSW write lock and drain deferred ops after serialization.
• Allow index-removal callbacks to mutate PrimeValue so HASH field sds entries can be swapped/copied for pointer preservation.
• Preserve old StringMap sds entries for deferred HNSW removes, and clear preserved data once deferred ops have drained.
• Update HASH command paths (e.g. HSET/HDEL/HINCRBY) to pass modified field names for selective preservation.
• After serializing all HNSW graphs, drain pending ops and clear preserved field data across shards.
• Extend replication stress test to cover “external vector” mode (DIM=256) and validate hashtable encoding.

[Copilot]

Pull request overview

This PR addresses a correctness issue for HNSW indices in “external vector storage” mode (copy_vector=false) by preserving underlying HASH field storage during deferred removals (e.g., while serialization holds read locks), and adds replication coverage for that mode.

Changes:

• Add preservation of StringMap-backed field storage during deferred HNSW removals, plus cleanup hooks to clear preserved data once deferred ops are drained.
• Extend HNSW index APIs to report whether removes executed synchronously and add an explicit “drain pending ops” operation.
• Expand replication test coverage to exercise external-vector mode and assert the HASH encoding matches the expected storage mode.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/dragonfly/replication_test.py	Parameterizes replication test to cover copied vs external vector modes and asserts encoding in external mode.
src/server/search/serialization_utils.cc	After global HNSW serialization, drains pending ops and clears preserved field data across shards.
src/server/search/doc_index.h	Changes removal APIs, adds preserved-field-data storage and cleanup entrypoints.
src/server/search/doc_index.cc	Implements preservation by swapping StringMap entries and tying preservation to deferred vs synchronous removes.
src/server/search/doc_index_fallback.cc	Updates fallback signatures and adds no-op preserved-data cleanup.
src/server/hset_family.cc	Collects modified HASH fields and passes them to search index removal for targeted preservation.
src/server/generic_family.cc	Uses FindMutable to allow preservation logic to mutate PrimeValue during overwrite handling.
src/server/family_utils.h	Updates RemoveKeyFromIndexesIfNeeded to accept mutable PrimeValue for preservation.
src/server/db_slice.h	Updates deletion callback to pass mutable PrimeValue for preservation.
src/core/search/hnsw_index.h	Changes Remove() to return sync/deferred status and adds DrainPendingOps().
src/core/search/hnsw_index.cc	Implements boolean Remove return and DrainPendingOps logic in the adapter.

[augmentcode]

🤖 Augment PR Summary

Summary: This PR makes HNSW (global) vector indices safe to serialize when vectors are stored as external pointers into HASH StringMap entries (copy_vector=false).

Changes:

• Extend HNSW remove APIs to report whether a remove ran synchronously or was deferred under MRMW read locks.
• Add DrainPendingOps() to block on the HNSW write lock and drain deferred ops after serialization.
• Allow index-removal callbacks to mutate PrimeValue so HASH field sds entries can be swapped/copied for pointer preservation.
• Preserve old StringMap sds entries for deferred HNSW removes, and clear preserved data once deferred ops have drained.
• Update HASH command paths (e.g. HSET/HDEL/HINCRBY) to pass modified field names for selective preservation.
• After serializing all HNSW graphs, drain pending ops and clear preserved field data across shards.
• Extend replication stress test to cover “external vector” mode (DIM=256) and validate hashtable encoding.

Technical Notes: The approach relies on swapping StringMap entries to keep old sds allocations alive until deferred HNSW removes execute, then draining deferred ops immediately after serialization to release preserved memory.

_{🤖 Was this summary useful? React with 👍 or 👎}

[augmentcode]

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

[dranikpg]

I still think that keeping the shard index as a regular DocIndex inside FieldIndices will remove all the code for custom handling of this index type

[dranikpg]

why not store the shard index inside the field indices? this way you don't need any special hnsw code - and if you need it as it's own type you can cast it back with static_cast or reinterpret_cast

[BorysTheDev]

I will make these changes as a separate PR

[dranikpg]

and I assume that you then also don't need those parts