Skip to content

[BUG] Resolve Memory Leak and Process Crashing during Concurrent PDF Ingestion #565

Description

@knoxiboy

Describe the bug

When multiple large PDF documents are uploaded simultaneously, the system spawns parallel extraction and OCR (pytesseract) threads/subprocesses without bounding. Under load, this causes extreme CPU/memory spikes, unhandled OS-level exceptions (e.g. out of memory), and database connection lockups on SQLite/ChromaDB.

Describe the fix

  1. Implement a concurrency semaphore or task queue inside backend/app/services/document_ingestion.py to bound the number of active document parsing tasks executing concurrently.
  2. Ensure file handles, PDF readers, and high-memory image objects are explicitly closed and cleaned up using contextmanager blocks.
  3. Add defensive exception handling for out-of-memory or timeout errors to fail individual files gracefully instead of crashing the entire backend.

Level: Critical / Bug
Affected Files: backend/app/services/document_ingestion.py, backend/app/tasks.py

Metadata

Metadata

Assignees

Labels

gssocGirlScript Summer of Code 2026 issue/PR

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions