✨ Feature Description
Implement a Redis-backed request coalescing layer that detects identical incoming RAG queries and prevents redundant LLM executions.
When multiple users submit the same query concurrently, the system should execute only a single retrieval and LLM inference pipeline while other requests wait for the result. Once completed, the response should be shared with all waiting requests.
This feature reduces duplicate computation, improves throughput, lowers infrastructure costs, and enhances system scalability under concurrent workloads.
🤔 Problem It Solves
Currently, if multiple users send the same query simultaneously, the platform may execute:
- Duplicate vector retrieval operations
- Duplicate context assembly workflows
- Duplicate LLM inference requests
This leads to:
- Increased latency
- Higher infrastructure costs
- Reduced throughput
- Wasted compute resources
In high-traffic environments, identical requests can cause significant unnecessary load on retrieval and inference systems.
💡 Proposed Solution
Introduce a Redis-backed request deduplication layer.
Workflow
-
Generate a deterministic hash for incoming queries.
-
Check Redis for an active in-flight request.
-
If no active request exists:
- Acquire a distributed lock.
- Execute retrieval and inference.
- Store result in Redis cache.
-
If an identical request is already running:
- Subscribe or wait for completion.
- Return the shared result once available.
-
Expose observability metrics for:
- Cache hits
- Cache misses
- Deduplicated requests
- Average wait time
- LLM requests avoided
Infrastructure Components
- Redis Distributed Locking
- FastAPI Middleware / Service Layer
- Request Fingerprinting
- Result Caching
- Prometheus Metrics
- OpenTelemetry Tracing
🔄 Alternatives Considered
Traditional Cache Only
Caching reduces repeated requests over time but does not prevent concurrent duplicate requests that arrive before the cache is populated.
Local In-Memory Locks
Works only for a single process and does not scale across multiple application instances.
Queue-Based Deduplication
More complex operationally and introduces additional infrastructure overhead.
Redis-based distributed coordination provides a simpler and scalable solution.
📸 Mockups / Examples
Current Behavior
User A → "What is RAG?"
User B → "What is RAG?"
Result:
2 Retrieval Pipelines
2 LLM Calls
Proposed Behavior
User A → "What is RAG?"
User B → "What is RAG?"
Result:
1 Retrieval Pipeline
1 LLM Call
2 Responses Returned
✅ Acceptance Criteria
📋 Additional Context
This feature aligns with the project's goals around:
- Distributed systems experimentation
- Async backend infrastructure
- Observability-first architectures
- Backend reliability engineering
- Scalable AI infrastructure
It also provides a practical demonstration of request coalescing, distributed locking, and infrastructure optimization patterns commonly used in production-scale backend systems.
🌱 Contributor Checklist
✨ Feature Description
Implement a Redis-backed request coalescing layer that detects identical incoming RAG queries and prevents redundant LLM executions.
When multiple users submit the same query concurrently, the system should execute only a single retrieval and LLM inference pipeline while other requests wait for the result. Once completed, the response should be shared with all waiting requests.
This feature reduces duplicate computation, improves throughput, lowers infrastructure costs, and enhances system scalability under concurrent workloads.
🤔 Problem It Solves
Currently, if multiple users send the same query simultaneously, the platform may execute:
This leads to:
In high-traffic environments, identical requests can cause significant unnecessary load on retrieval and inference systems.
💡 Proposed Solution
Introduce a Redis-backed request deduplication layer.
Workflow
Generate a deterministic hash for incoming queries.
Check Redis for an active in-flight request.
If no active request exists:
If an identical request is already running:
Expose observability metrics for:
Infrastructure Components
🔄 Alternatives Considered
Traditional Cache Only
Caching reduces repeated requests over time but does not prevent concurrent duplicate requests that arrive before the cache is populated.
Local In-Memory Locks
Works only for a single process and does not scale across multiple application instances.
Queue-Based Deduplication
More complex operationally and introduces additional infrastructure overhead.
Redis-based distributed coordination provides a simpler and scalable solution.
📸 Mockups / Examples
Current Behavior
User A → "What is RAG?"
User B → "What is RAG?"
Result:
2 Retrieval Pipelines
2 LLM Calls
Proposed Behavior
User A → "What is RAG?"
User B → "What is RAG?"
Result:
1 Retrieval Pipeline
1 LLM Call
2 Responses Returned
✅ Acceptance Criteria
📋 Additional Context
This feature aligns with the project's goals around:
It also provides a practical demonstration of request coalescing, distributed locking, and infrastructure optimization patterns commonly used in production-scale backend systems.
🌱 Contributor Checklist