Check if merge is aborted during HNSW graph construction

[jeho-rpls]

Fixes #16367.

HNSW graph construction never checked OneMerge's abort flag. It is a pure-CPU loop that performs no writes, so IndexWriter#rollback and abortMerges() blocked until the entire graph was built. For production-sized vector segments that means tens of minutes (measurements in the issue).

Changes

• HnswBuilder gains setAbortCheck(IORunnable), and HnswGraphBuilder invokes it before every node insertion in addGraphNodeInternal.
  • This is the single point that all insertion paths funnel through: full rebuild via addVectors, graph join via MergingHnswGraphBuilder, and HnswConcurrentMergeBuilder workers (which forward the check).
• Lucene99HnswVectorsWriter passes mergeState::checkAborted into the graph mergers (new constructor overloads, existing constructors unchanged).
• Flush-time graph builds are unaffected (no check set), matching the null contract of checkIntegrity(OneMerge) from Check if merge is aborted during file integrity checksums #16281.
• Overhead: one null check per node insertion, where each insertion costs hundreds of vector comparisons.

Testing

• New TestHnswMergeAbort exercises all three merge paths end-to-end (rollback during a forceMerge of a 48k x 96d index). Without the fix, rollback blocks until the build finishes. With the fix, it returns in milliseconds.
  • full rebuild (every segment above 40% deletions, so no base graph is eligible): blocked 27 s without the fix
  • graph join (deletion-free segments, MergingHnswGraphBuilder): blocked 31 s
  • concurrent merge (numMergeWorkers = 2, HnswConcurrentMergeBuilder): blocked 14 s
• ./gradlew tidy and :lucene:core:check pass.
• The approach was also validated on a production Elasticsearch 9.4.2 cluster via a 10.4 backport.
  • Node shutdown during a live HNSW merge went from 24–31 min (grace period exceeded, force-killed) to 2.2 s.
  • Undisturbed merges complete unchanged (details in HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes #16367).

[jeho-rpls]

The CI failure is unrelated to this change: TestForceNoBulkScoringQuery.testBulkScoringIsDisabled (lucene/monitor) fails on a clean main checkout (a9eb52b, no changes from this PR) with the same seed:

gradlew -p lucene/monitor test --tests TestForceNoBulkScoringQuery.testBulkScoringIsDisabled -Dtests.seed=BDD18A3ADD885193 -Dtests.locale=ii-CN -Dtests.timezone=Etc/Zulu

fails deterministically with Expected exception AssertionError but no exception was thrown (test environment: codec=CheapBastard). Re-pushed to re-trigger CI with fresh seeds.

[jeho-rpls]

Opened #16369 with a fix for the unrelated test failure above.

[msokolov]

Is this really needed? When do we ever call abortMerges()? It looks to me as if it gets called when an exception happens while closing an IndexWriter, or explicitly rolling back. Seem like kind of exceptional cases, although maybe we would want rollbacks to be faster? Do other merge operations check for aborted status?

[jeho-rpls]

Hi @msokolov, thanks for taking a look!

Is this really needed? When do we ever call abortMerges()? It looks to me as if it gets called when an exception happens while closing an IndexWriter, or explicitly rolling back. Seem like kind of exceptional cases

abortMerges() is routine at the deployment level even though it looks exceptional at the API level.
Elasticsearch closes its shard engines through IndexWriter#rollback on node shutdown, so every rolling restart of a data node goes through it.

That is how we hit this.

• When a node is asked to shut down while an HNSW merge is in the middle of graph construction, that rollback blocks until the build completes. In containerized deployments it easily outlives the pod termination grace period, so the process ends up force-killed mid-merge.
• In our case, a routine rolling restart of a vector-heavy cluster blocked for 24–31 minutes per node behind a single in-flight merge, past the 30-minute pod grace in the worst case (measurements in HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes #16367).

Elasticsearch reports hitting a similar issue in production and has an open downstream workaround (elastic/elasticsearch#152342), but it lives in their stateless plugin only, and the workaround's author notes it is partial coverage. A check in the graph builder covers every directory implementation.

although maybe we would want rollbacks to be faster?

Yes, exactly. Faster rollback is the whole motivation. The wait also buys nothing. The writer is rolling back, so the graph being built is discarded the moment it completes.

Do other merge operations check for aborted status?

They do.

• Writes: every merge output is wrapped via ConcurrentMergeScheduler#wrapForMerge into RateLimitedIndexOutput, and MergeRateLimiter#maybePause throws MergeAbortedException once the flag is set. This is the "they periodically check if they are aborted" that abortMerges relies on.
• Checksums: since Check if merge is aborted before executing file integrity checks #16264 and Check if merge is aborted during file integrity checksums #16281, every format consumer calls MergeState#checkAborted before each producer's integrity check, and CodecUtil#checksumEntireFile(input, merge) re-checks the flag every megabyte read.

Graph construction is the odd one out. It performs no writes and no checksum reads for tens of minutes, so it never observes the flag. In our profiling it was 93.6% of the merge CPU, while the checksum phase covered by #16281 was 0.1%.