Keep status dependency checks from blocking probes

[wayyoungboy]

Summary:

• Run dependency status probes off the event loop with bounded timeouts.
• Cache dependency statuses briefly and coalesce same-dependency checks.
• Add tests for timeout behavior, status caching, concurrent probe coalescing, and configuration validation.

Root cause:
/api/v1/system/status performs dependency checks for dashboard status data. In a single-worker deployment, a slow synchronous dependency probe can occupy the event loop long enough for lightweight health probes to time out.

Validation:

• Using the project virtual environment, ran pytest tests/unit/server/test_health_check.py tests/unit/server/test_settings.py (6 passed).
• Using the project virtual environment, ran pytest tests/unit/server (79 passed, 2 warnings from existing dependency deprecations).
• Using the project virtual environment, ran python -m compileall src/server/utils/health_check.py src/server/config.py.
• Ran git diff --check.
• Ran scripts/branch_name_check.sh --repo ..
• E2E: started a local FastAPI server, simulated a slow dependency probe, and confirmed a concurrent /api/v1/system/health request stayed responsive while /api/v1/system/status returned a degraded dependency result.

[wayyoungboy]

P2: the timeout currently only releases the coroutine waiting for the probe; it does not stop the synchronous probe already running in the default executor. After _cached_probe() stores the timeout result and releases the dependency lock, a later status poll after the cache TTL can start another worker even if the previous probe is still hung. In a real stuck dependency initialization, repeated /system/status polling can eventually exhaust default-executor workers and degrade unrelated asyncio.to_thread users.

Please keep same-dependency coalescing active until the underlying worker exits, or run these probes through a bounded/isolateable executor so a stuck dependency check cannot consume an unbounded number of shared workers.

[wayyoungboy]

Updated this PR to address the dependency probe timeout review finding:

• Reuse the same in-flight dependency probe after timeout instead of starting another worker on the next status poll.
• Run dependency probes through a small dedicated executor so stuck probes cannot consume the shared default executor.
• Added a regression test covering repeated timeout calls while the original probe is still blocked.

Validation:

• pytest tests/unit/server/test_health_check.py tests/unit/server/test_settings.py
• pytest tests/unit/server
• python -m compileall src/server/utils/health_check.py src/server/config.py
• git diff --check upstream/main...HEAD
• E2E: slow dependency probe with concurrent /api/v1/system/health and /api/v1/system/status.

[lightzt99]

LGTM