fix: don't log async queue-time ingests as "permanently rejected"

[Marvinthebored]

Summary

The chunk ingest loop (src/ingest-queue.ts) logs Chunk permanently rejected and shrinks the chunk budget whenever the daemon's ingest_markdown_document returns nodesAccepted === 0. But the daemon ingests asynchronously — the RPC returns at queue-time, before the embed/commit runs — so a perfectly good ingest reports nodesAccepted: 0 (and nodesRejected: 0) on return; the content commits moments later as the WAL drains.

Under load (a backed-up WAL) nodesAccepted: 0 is the normal return, so this fired on nearly every chunk, flooding logs with false Chunk permanently rejected warnings even though nothing was rejected and all content committed.

Evidence (running daemon, GGUF backend): the ingest feedback for a wide range of inputs — a full 8.9 KB document, a 2 KB slice, an 80-char slice, even plain English — is uniformly:

nodesAccepted: 0   nodesRejected: 0   tokensIngested: 0   walDepth: climbing

nodesRejected is 0 throughout; the WAL depth climbs (queued) and the documents are retrievable seconds later.

Fix

Gate the rejection path on nodesRejected > 0 — the only signal of an actual rejection — instead of nodesAccepted === 0:

• nodesRejected === 0 (the normal async-queued case) → success: no warning, no shrink, advance.
• nodesRejected > 0 → a real rejection: shrink-and-retry the same offset if the daemon reported a lower tokenBurstLimit, otherwise log an accurate Chunk rejected including the nodesRejected count.

This removes the false-alarm flood without weakening genuine rejection handling. (The ok === false transport-error path is unchanged.)

Testing

• tsc --noEmit clean; unit suite green (5/5), including two new tests:
  • async queue-time nodesAccepted=0 / nodesRejected=0 produces no rejection warning and submits the whole document;
  • a real nodesRejected > 0 with a lower burst limit shrinks and retries the same offset.
• Validated against a live deployment: re-ingesting a 646-file markdown corpus produced 0 rejection lines (the previous build logged 128+ before being interrupted), errors=0, and the previously-"rejected" dense documents are retrievable. Driving the three worst offenders straight through the patched queue (fresh hashes, bypassing dedup) produced 0 warnings.

Note

This supersedes #360, which I closed — that PR chased a chunk-size theory, but the symptom was never a size rejection (the daemon never set nodesRejected > 0); it was this async-queue mislabeling. Apologies for the detour.

Fixed chunk ingest handling so async queue-time ingests with nodesAccepted === 0 and nodesRejected === 0 are treated as successful instead of being logged as permanently rejected.

• Rejection detection now keys off nodesRejected > 0
• Real rejections still shrink/retry when a lower tokenBurstLimit is reported
• Otherwise, rejections log Chunk rejected with the rejection count
• Added tests for:
  • async zero-accept/zero-reject success path
  • real rejection retry/shrink behavior

Complexity:

• Big O: no regression
• Cyclomatic complexity: no regression

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b2c19fff-221e-4524-b899-e423a4ae6452

📥 Commits

Reviewing files that changed from the base of the PR and between be9cbcb and 8e583c8.

📒 Files selected for processing (2)

• src/ingest-queue.ts
• test/unit/ingest-queue.test.ts

---

📝 Walkthrough

Walkthrough

In enqueueIngest, the rejection condition is changed from nodesAccepted === 0 to nodesRejected > 0. The shrink-and-retry behavior using tokenBurstLimit is now tied to the new condition. Two new tests cover the async success case (nodesAccepted=0, nodesRejected=0) and the real rejection shrink-and-retry case.

Async ingest rejection fix

Layer / File(s)	Summary
Rejection condition, retry logic, and tests src/ingest-queue.ts, test/unit/ingest-queue.test.ts	enqueueIngest now treats nodesRejected > 0 as rejection instead of nodesAccepted === 0. Two new tests verify that async-queued feedback with zero rejections is treated as success, and that a real rejection triggers shrink-and-retry at the same offset.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

• Chunker permanently drops dense/CJK documents (4-chars/token estimate + non-converging token-burst retry) #359: Directly related — both modify the enqueueIngest retry/rejection logic around tokenBurstLimit in src/ingest-queue.ts.

Possibly related PRs

• xDarkicex/openclaw-memory-libravdb#188: Modifies the same feedback-driven chunking and retry flow in enqueueIngest, making it directly related to the nodesRejected > 0 condition change here.

Suggested labels

release:patch

🐇 No more phantom rejections in the queue today,
When nodes are merely sleeping, we won't shout "nay!"
nodesRejected > 0 — that's the real alarm,
Zero accepted? No worries, no harm! 🌿
Hop along, little chunks, you're safe on your way~

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: preventing async queue-time ingests from being logged as permanently rejected.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[compoodment]

Vale Review — PR #361

Quality: Q3/5 — promising
Head: 8e583c8

Findings:
None

Proof gaps: Targeted tests were not run by Vale; live-daemon proof is PR-author supplied. Mixed feedback with both nodesAccepted > 0 and nodesRejected > 0 is still not covered by the new unit tests, so the feedback contract should stay on the reviewer radar.

Verdict: comment-only — focused fix with relevant tests, but not independently verified in this run.

– Vale