fix: batch limits and failed task reque limit

[dartpain]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

• Why was this change needed? (You can also link to an open issue here)

• Other information:

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nextra-docsgpt	Building	Preview, Comment	May 18, 2026 9:01pm
oss-docsgpt	Ready	Preview, Comment	May 18, 2026 9:01pm

[Copilot]

Pull request overview

This PR tightens resource usage and failure-handling around ingestion by (1) capping Docling’s threaded pipeline buffering to reduce worker memory spikes and (2) hardening Celery task idempotency/poison-loop behavior for keyless dispatches, including emitting a terminal failure SSE event when the poison guard trips.

Changes:

• Add Docling threaded-pipeline caps (queue depth + per-stage batch sizes) controlled via a new DOCLING_PIPELINE_QUEUE_MAX_SIZE setting.
• Extend with_idempotency to synthesize a deterministic guard key from source_id when idempotency_key is missing, preventing infinite redelivery poison loops.
• Add an on_poison hook and wire ingest-related tasks to publish source.ingest.failed when the poison guard stops execution.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
application/parser/file/docling_parser.py	Caps Docling pipeline buffering via _apply_pipeline_caps() to reduce memory footprint.
application/core/settings.py	Introduces DOCLING_PIPELINE_QUEUE_MAX_SIZE setting with a safer default.
application/api/user/idempotency.py	Adds synthesized guard keys for keyless source_id dispatches and an on_poison hook.
application/api/user/tasks.py	Emits terminal source.ingest.failed when poison guard trips for ingest-family tasks.
tests/parser/file/test_docling_parser.py	Adds unit tests validating pipeline cap behavior and legacy no-op behavior.
tests/api/user/test_idempotency_decorator.py	Adds unit tests for synthesized key behavior and poison hook semantics.
tests/api/user/test_tasks.py	Adds unit tests for the poison-path ingest failure event publisher.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.37%. Comparing base (9b8fe2d) to head (afb69bb).
⚠️ Report is 36 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2484      +/-   ##
==========================================
+ Coverage   91.34%   91.37%   +0.03%     
==========================================
  Files         248      265      +17     
  Lines       20709    22781    +2072     
==========================================
+ Hits        18916    20817    +1901     
- Misses       1793     1964     +171     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.