Skip to content

Check is_ready flag in /healthz endpoint #2255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ecarrara wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Check is_ready flag in /healthz endpoint #2255

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7b2aef5
Check is_ready flag in /healthz endpoint
ecarrara Apr 21, 2026
1b63263
Merge branch 'main' into worktree-erle+healthz-is-ready-flag
ecarrara Apr 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions inference/core/interfaces/http/http_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2521,12 +2521,25 @@ def readiness(
)

@app.get("/healthz", status_code=200)
def healthz():
def healthz(
state: ModelInitState = Depends(lambda: model_init_state),
):
"""Health endpoint for Kubernetes liveness probe.

Verifies CUDA context health when running on GPU. Returns 503 if
CUDA is corrupted (unrecoverable - requires process restart).
CUDA is corrupted (unrecoverable - requires process restart) or
if model initialization has not completed yet.
"""
with state.lock:
if not state.is_ready:
return JSONResponse(
content={
"status": "unhealthy",
"reason": "not_ready",
},
status_code=503,
)

from inference.core.utils.cuda_health import check_cuda_health

is_healthy, error = check_cuda_health()
Expand Down
Loading