Skip to content

Commit 4494239

Browse files
committed
Fix HNSW InfoStream duplicate times and add per-chunk completion logging (#15967)
Signed-off-by: prithvi <prithvisivasankar@gmail.com>
1 parent db3222d commit 4494239

2 files changed

Lines changed: 19 additions & 4 deletions

File tree

lucene/CHANGES.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,9 @@ Optimizations
148148

149149
Bug Fixes
150150
---------------------
151+
* GITHUB#15967: Fix InfoStream progress lines from concurrent HNSW merging reporting duplicate
152+
times, and add per-chunk completion logging to aid concurrency debugging. (Prithvi S)
153+
151154
* GITHUB#14049: Randomize KNN codec params in RandomCodec. Fixes scalar quantization div-by-zero
152155
when all values are identical. (Mike Sokolov)
153156

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -205,15 +205,27 @@ protected void addVectors(int minOrd, int maxOrd) throws IOException {
205205
throw new IllegalStateException("This HnswGraphBuilder is frozen and cannot be updated");
206206
}
207207
long start = System.nanoTime(), t = start;
208-
if (infoStream.isEnabled(HNSW_COMPONENT)) {
209-
infoStream.message(HNSW_COMPONENT, "addVectors [" + minOrd + " " + maxOrd + ")");
210-
}
208+
int numVectors = maxOrd - minOrd;
211209
for (int node = minOrd; node < maxOrd; node++) {
212210
addGraphNode(node);
213-
if ((node % 10000 == 0) && infoStream.isEnabled(HNSW_COMPONENT)) {
211+
// Skip in-loop progress for ranges <= 10000 where it would fire at most once
212+
// with identical incremental/total times (#15967).
213+
if (numVectors > 10000 && (node % 10000 == 0) && infoStream.isEnabled(HNSW_COMPONENT)) {
214214
t = printGraphBuildStatus(node, start, t);
215215
}
216216
}
217+
if (infoStream.isEnabled(HNSW_COMPONENT)) {
218+
long elapsed = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);
219+
infoStream.message(
220+
HNSW_COMPONENT,
221+
String.format(
222+
Locale.ROOT,
223+
"addVectors [%d %d): %d vectors in %d ms",
224+
minOrd,
225+
maxOrd,
226+
numVectors,
227+
elapsed));
228+
}
217229
}
218230

219231
private void addVectors(int maxOrd) throws IOException {

0 commit comments

Comments
 (0)