Describe the bug
When multiple large PDF documents are uploaded simultaneously, the system spawns parallel extraction and OCR (pytesseract) threads/subprocesses without bounding. Under load, this causes extreme CPU/memory spikes, unhandled OS-level exceptions (e.g. out of memory), and database connection lockups on SQLite/ChromaDB.
Describe the fix
- Implement a concurrency semaphore or task queue inside
backend/app/services/document_ingestion.py to bound the number of active document parsing tasks executing concurrently.
- Ensure file handles, PDF readers, and high-memory image objects are explicitly closed and cleaned up using
contextmanager blocks.
- Add defensive exception handling for out-of-memory or timeout errors to fail individual files gracefully instead of crashing the entire backend.
Level: Critical / Bug
Affected Files: backend/app/services/document_ingestion.py, backend/app/tasks.py
Describe the bug
When multiple large PDF documents are uploaded simultaneously, the system spawns parallel extraction and OCR (pytesseract) threads/subprocesses without bounding. Under load, this causes extreme CPU/memory spikes, unhandled OS-level exceptions (e.g. out of memory), and database connection lockups on SQLite/ChromaDB.
Describe the fix
backend/app/services/document_ingestion.pyto bound the number of active document parsing tasks executing concurrently.contextmanagerblocks.Level: Critical / Bug
Affected Files:
backend/app/services/document_ingestion.py,backend/app/tasks.py