feat(extraction): log sticky runtime start failures via domain probe#790
Merged
Merged
Conversation
The Graph Management Assistant "no credentials resolved" / RUNTIME_START_FAILED error was invisible server-side: it only ever reached the browser via the runtime/warm NDJSON stream, with no corresponding log line in the api container. Add a StickySessionRuntimeProbe (Domain-Oriented Observability) that logs the raw failure/timeout detail whenever get_or_start_runtime or the health wait fails, so future occurrences show up in kubectl/ArgoCD logs without needing to reproduce them interactively. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
📝 WalkthroughSummary by CodeRabbit
WalkthroughNew Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant StickySessionRuntimeService
participant ContainerRuntime
participant StickySessionRuntimeProbe
participant structlog
StickySessionRuntimeService->>ContainerRuntime: start runtime
ContainerRuntime--xStickySessionRuntimeService: ContainerRuntimeError/RuntimeError
StickySessionRuntimeService->>StickySessionRuntimeProbe: runtime_start_failed(session_id, knowledge_graph_id, mode, error)
StickySessionRuntimeProbe->>structlog: logger.error("sticky_runtime_start_failed", context)
StickySessionRuntimeService->>ContainerRuntime: health check
ContainerRuntime--xStickySessionRuntimeService: TimeoutError
StickySessionRuntimeService->>StickySessionRuntimeProbe: runtime_unhealthy(session_id, knowledge_graph_id, mode, error)
StickySessionRuntimeProbe->>structlog: logger.error("sticky_runtime_unhealthy", context)
Related PRs: None identified from provided data. Suggested labels: observability, testing Suggested reviewers: No mentions provided; assign per team ownership of Security note (per role): No dependency, CI/CD, IDE config, or Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (9 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/api/extraction/application/observability/sticky_session_runtime_probe.py`:
- Around line 75-113: The sticky session runtime probe methods are logging and
forwarding raw exception text, which can leak sensitive Docker/Podman/OpenShell
details. Update runtime_start_failed and runtime_unhealthy to sanitize the error
argument before calling self._logger.error, and apply the same redaction before
any API response uses the exception text. Keep the existing context fields, but
ensure ContainerRuntimeError/RuntimeError messages are scrubbed for secrets or
connection data before emission.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 3d28b8c9-50a5-4cd6-9415-a20906ae0068
⛔ Files ignored due to path filters (1)
src/api/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
src/api/extraction/application/observability/__init__.pysrc/api/extraction/application/observability/sticky_session_runtime_probe.pysrc/api/extraction/application/sticky_session_runtime_service.pysrc/api/tests/unit/extraction/application/observability/test_sticky_session_runtime_probe.pysrc/api/tests/unit/extraction/application/test_sticky_session_runtime_service.py
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
openshift-hyperfleet/architecture(manual)openshift-hyperfleet/hyperfleet-api(manual)openshift-hyperfleet/hyperfleet-sentinel(manual)openshift-hyperfleet/hyperfleet-adapter(manual)openshift-hyperfleet/hyperfleet-broker(manual)
Comment on lines
+75
to
+113
| def runtime_start_failed( | ||
| self, | ||
| *, | ||
| session_id: str, | ||
| knowledge_graph_id: str, | ||
| mode: str, | ||
| error: str, | ||
| ) -> None: | ||
| context_kwargs = self._get_context_kwargs( | ||
| exclude={"session_id", "knowledge_graph_id", "mode", "error"} | ||
| ) | ||
| self._logger.error( | ||
| "sticky_runtime_start_failed", | ||
| session_id=session_id, | ||
| knowledge_graph_id=knowledge_graph_id, | ||
| mode=mode, | ||
| error=error, | ||
| **context_kwargs, | ||
| ) | ||
|
|
||
| def runtime_unhealthy( | ||
| self, | ||
| *, | ||
| session_id: str, | ||
| knowledge_graph_id: str, | ||
| mode: str, | ||
| error: str, | ||
| ) -> None: | ||
| context_kwargs = self._get_context_kwargs( | ||
| exclude={"session_id", "knowledge_graph_id", "mode", "error"} | ||
| ) | ||
| self._logger.error( | ||
| "sticky_runtime_unhealthy", | ||
| session_id=session_id, | ||
| knowledge_graph_id=knowledge_graph_id, | ||
| mode=mode, | ||
| error=error, | ||
| **context_kwargs, | ||
| ) |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== file outline ==\n'
ast-grep outline src/api/extraction/application/observability/sticky_session_runtime_probe.py --view expanded || true
printf '\n== relevant references ==\n'
rg -n "runtime_start_failed|runtime_unhealthy|ContainerRuntimeError|str\\(exc\\)|no credentials resolved|redact|sanitize|secret|credential" src/api/extraction src/api/shared_kernel src/api -g '!**/__pycache__/**' || true
printf '\n== nearby file slice ==\n'
sed -n '1,220p' src/api/extraction/application/observability/sticky_session_runtime_probe.py
printf '\n== runtime service slice ==\n'
sed -n '1,260p' src/api/extraction/application/sticky_session_runtime_service.pyRepository: openshift-hyperfleet/kartograph
Length of output: 50388
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '260,380p' src/api/extraction/application/sticky_session_runtime_service.py
printf '\n== probe file ==\n'
sed -n '1,180p' src/api/extraction/application/observability/sticky_session_runtime_probe.py
printf '\n== runtime error type ==\n'
sed -n '1,120p' src/api/shared_kernel/container_runtime/ports.pyRepository: openshift-hyperfleet/kartograph
Length of output: 10319
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,240p' src/api/shared_kernel/container_runtime/cli_runtime.py
printf '\n== grep for runtime error propagation ==\n'
rg -n "ContainerRuntimeError\\(|raise RuntimeError\\(|detail|stderr|stdout|redact|sanitize" src/api/shared_kernel/container_runtime src/api/extraction/application src/api/extraction/infrastructure -g '!**/__pycache__/**'Repository: openshift-hyperfleet/kartograph
Length of output: 24565
CWE-532/CWE-209: redact runtime exception text before logging or returning it.
error=str(exc) is emitted unchanged here, and the same text is also forwarded to the API response. ContainerRuntimeError/RuntimeError can carry Docker/Podman/OpenShell stderr/stdout, so any token, connection string, or credential fragment in the failure text will land in logs and user-visible errors. Redact known secret patterns before emitting.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/api/extraction/application/observability/sticky_session_runtime_probe.py`
around lines 75 - 113, The sticky session runtime probe methods are logging and
forwarding raw exception text, which can leak sensitive Docker/Podman/OpenShell
details. Update runtime_start_failed and runtime_unhealthy to sanitize the error
argument before calling self._logger.error, and apply the same redaction before
any API response uses the exception text. Keep the existing context fields, but
ensure ContainerRuntimeError/RuntimeError messages are scrubbed for secrets or
connection data before emission.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RUNTIME_START_FAILEDerror is completely invisible server-side today: it's only ever surfaced to the browser via the/runtime/warmNDJSON stream body, with zero corresponding log line in theapicontainer (confirmed via 9+ hours of production logs with no matches for "vertex", "provider", "credential", etc).StickySessionRuntimeProbe(Domain-Oriented Observability) that logs the raw failure detail (error=str(exc)) wheneverget_or_start_runtimeraises or the health wait times out, in bothstream_runtime_warmupandensure_runtime_for_chatcode paths.kubectl logs/ArgoCD without needing to reproduce it live.Test plan
uv run pytest tests/unit/extraction/application/observability/test_sticky_session_runtime_probe.py -vuv run pytest tests/unit/extraction/application/test_sticky_session_runtime_service.py -vuv run pytest tests/unit -q(full suite, 3512 passed)uv run ruff format/ruff checkon changed filesMade with Cursor