Skip to content

Commit e63d00f

Browse files
committed
fix(batch): require both counts unchanged before the idle flush
The wait loop in the batching threads breaks out early when a second has passed and nothing new arrived, but the guard ORs the two "unchanged" checks: if time.time() - start >= 1 and ( len_o == len(self.__batch_objects) or len_r == len(self.__batch_references) ): In an objects-only batch -- what `add_object` without `add_reference` produces -- `len_r` and `len(self.__batch_references)` are both always 0, so the right operand is always True and the whole guard collapses to `if time.time() - start >= 1`. The loop then stops waiting on the first tick past one second no matter how fast objects are still streaming in, so the batch is flushed well short of its configured size. A references-only batch has the mirror-image problem. The comment above the break says "no new objects were added in the last second", i.e. the batch is idle -- which is true only when *both* counts are unchanged. Switch `or` to `and` at the three places that share this loop shape (sync.py, async_.py, base.py). Fixes #2105
1 parent ae327ca commit e63d00f

3 files changed

Lines changed: 3 additions & 3 deletions

File tree

weaviate/collections/batch/async_.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ async def __loop(self) -> None:
225225
# wait for more objects to be added up to the batch size
226226
await asyncio.sleep(refresh_time)
227227
if time.time() - start >= 1 and (
228-
len_o == len(self.__batch_objects) or len_r == len(self.__batch_references)
228+
len_o == len(self.__batch_objects) and len_r == len(self.__batch_references)
229229
):
230230
# no new objects were added in the last second, exit the loop
231231
break

weaviate/collections/batch/base.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,7 @@ def __batch_send(self) -> None:
434434
# shutdown was requested, exit the loop
435435
break
436436
if time.time() - start >= 1 and (
437-
len_o == len(self.__batch_objects) or len_r == len(self.__batch_references)
437+
len_o == len(self.__batch_objects) and len_r == len(self.__batch_references)
438438
):
439439
# no new objects were added in the last second, exit the loop
440440
break

weaviate/collections/batch/sync.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ def __loop(self) -> None:
177177
# wait for more objects to be added up to the batch size
178178
time.sleep(refresh_time)
179179
if time.time() - start >= 1 and (
180-
len_o == len(self.__batch_objects) or len_r == len(self.__batch_references)
180+
len_o == len(self.__batch_objects) and len_r == len(self.__batch_references)
181181
):
182182
# no new objects were added in the last second, exit the loop
183183
break

0 commit comments

Comments
 (0)