[Jobs Match Making] Scale testing and performance analysis

[aldbr]

Description

Push the implementation toward the target scale from the spec: 10M waiting jobs, 200 Hz sustained matching (stretch goal: 1 KHz), ~1000 sites, several hundred tags, non-uniform distributions.

Identify bottlenecks and experiment with Redis configuration tuning. Produce a comparative analysis across all implementations.

Also address robustness:

• Recovery from Redis data loss: rebuild state from a simulated JobDB
• Concurrent matching: verify no duplicate job assignments

Definition of Done

• Synthetic dataset at 10M jobs
• Locust tests with multiple concurrent workers
• Performance report with plots: throughput and latency at increasing scales
• Bottleneck analysis
• Comparison table: all implementations (Python, Redis+Python, Redis+Lua)
• Rebuild mechanism tested at scale
• Concurrent matching tested