Feat:Backend Image Processing Blocks the Single-Worker ProcessPoolExecutor

[akhil06232]

Describe the feature

Heavy processing tasks like AI tagging + face detection + face clustering are all submitted to this single worker. If a user adds a large folder, every subsequent request (sync, AI tagging enable) is queued behind it.Additionally, in
images.py #L105-L137, images are processed one-by-one in a loop — no batching or parallelism.

Add ScreenShots

Proposed Fix:
--Increase max_workers or use a proper async task queue (e.g., Celery, RQ, or even asyncio.to_thread)
--Implement batch inference for YOLO/FaceNet instead of per-image model invocations
--Add progress reporting (SSE or WebSocket) to the frontend

Record

• I agree to follow this project's Code of Conduct
• I want to work on this issue

[github-actions]

👋 Thanks for opening this issue! The maintainers will review it shortly. Till then, do not open any PR. This is a strict rule we like to follow around here :)

[codewith-shivansh]

Hi @rahulharpal1603

I went through the issue and reviewed the current implementation using a single-worker ProcessPoolExecutor along with the per-image processing loop in images.py. I agree that this design can easily become a bottleneck when large folders are added, as all AI-related tasks get queued behind one worker.

I would like to work on this issue. My proposed approach would be incremental:

Phase 1:

Refactor the image processing loop to support batch inference (for YOLO/FaceNet where applicable) instead of per-image model calls.

Adjust the ProcessPoolExecutor configuration to use a dynamic max_workers based on available CPU cores while ensuring system stability.

Add proper task queue handling to avoid blocking subsequent lightweight requests.

Phase 2 (if aligned with project direction):

Evaluate whether introducing a background task queue (e.g., Celery/RQ) is appropriate, or if improving the current executor-based design is preferred.

Before proceeding, I would like to confirm the preferred architectural direction so the solution aligns with the project's roadmap.

Looking forward to your guidance.