Skip to content

DEBUGGING_PLAYBOOK.md

Latest commit

 

History

History
384 lines (271 loc) · 20 KB

DEBUGGING_PLAYBOOK.md

File metadata and controls

384 lines (271 loc) · 20 KB
Raw

Debugging Playbook

Reference for querying structured logs across all Open-Inspect services.

All services emit flat JSON lines with a common envelope. Use the msg field (stable event identifier) for filtering, and correlation fields (trace_id, session_id, message_id) for joining across service boundaries.

Common Envelope

Every log line includes:

Field Type Description
level string debug | info | warn | error
service string control-plane | modal-infra | slack-bot
component string Sub-area (e.g. router, session-do, bridge)
msg string Stable event identifier for querying
ts number Epoch milliseconds

Correlation Fields

Include whichever are available at the call site:

Field Scope Generated by
trace_id Cross-service Router / slack-bot edge
request_id Single HTTP hop Router
session_id Session lifetime Control-plane on create
message_id Single prompt run Control-plane on enqueue
sandbox_id Sandbox lifetime Modal on create/restore
opencode_session_id OpenCode process Supervisor on ensure

Joining across services

A prompt run flows: slack-bot → control-plane → modal-infra → sandbox.

  1. Slack-bot generates trace_id and sends it as x-trace-id header.
  2. Control-plane router propagates trace_id into Durable Object and Modal calls.
  3. Modal endpoints read x-trace-id and bind it to the sandbox logger.

To trace a full request: filter by trace_id across all three services, or narrow by session_id + message_id for a specific prompt run.

Error Shape

All services use the same error fields:

Field Description
error_type Error class name (e.g. TimeoutError)
error_message Human-readable message
error_stack Stack trace (error level only, truncated)
error_code Application error code (when applicable)

Outcome Convention

Wide events use outcome to indicate result:

  • success — completed normally
  • error — failed with an error
  • rejected — denied before processing (auth, validation)
  • timeout — timed out
  • cancelled — cancelled by caller

Event Catalog

Control Plane (service: "control-plane")

Router (component: "router")

Event Level Key Fields Description
http.request info, error http_method, http_path, http_status, duration_ms, outcome One per incoming HTTP request

Session Durable Object (component: "session-do")

Event Level Key Fields Description
do.request info http_method, http_path, http_status, duration_ms, outcome One per DO internal route call
ws.connect info, warn ws_type (sandbox|client), outcome, reject_reason, sandbox_id, participant_id, duration_ms WebSocket lifecycle
prompt.enqueue info message_id, source, author_id, user_id, model, content_length, has_attachments, queue_position Message queued
prompt.dispatch info message_id, outcome, reason, model, has_sandbox_ws, queue_wait_ms Message sent to sandbox
prompt.complete info, warn message_id, outcome, total_duration_ms, processing_duration_ms, queue_duration_ms Prompt run finished

Lifecycle Manager (component: "lifecycle-manager")

Event Level Key Fields Description
sandbox.spawn info expected_sandbox_id, repo_owner, repo_name Spawn attempt started
sandbox.spawned info sandbox_id, provider_object_id Spawn succeeded
sandbox.spawn_failed error error Spawn failed
sandbox.restore info snapshot_image_id Restore attempt started
sandbox.restored info sandbox_id, provider_object_id Restore succeeded
sandbox.snapshot info reason, modal_object_id Snapshot attempt started
sandbox.snapshot_saved info image_id, reason Snapshot saved
sandbox.heartbeat_stale warn last_heartbeat_ms, threshold_ms Heartbeat missed
sandbox.timeout info last_activity, timeout_ms Inactivity timeout reached

Modal Client (component: "modal-client")

Event Level Key Fields Description
modal.request info endpoint, session_id, sandbox_id, http_status, duration_ms, outcome One per control-plane → Modal call

Modal Infra (service: "modal-infra")

API (component: "api")

Event Level Key Fields Description
modal.http_request info http_method, http_path, http_status, duration_ms, outcome, endpoint_name, trace_id, request_id One per Modal endpoint call

Sandbox Manager (component: "sandbox")

Event Level Key Fields Description
sandbox.create info sandbox_id, modal_object_id, repo_owner, repo_name, duration_ms, outcome Sandbox creation
sandbox.snapshot info sandbox_id, snapshot_id, image_id, duration_ms, outcome Snapshot taken
sandbox.restore info sandbox_id, modal_object_id, snapshot_image_id, duration_ms, outcome Sandbox restored

Supervisor (component: "supervisor")

Event Level Key Fields Description
sandbox.startup info repo_owner, repo_name, restored_from_snapshot, git_sync_success, opencode_ready, duration_ms One per sandbox boot
supervisor.start info repo_owner, repo_name Supervisor process started
supervisor.error error exc Unhandled supervisor error
supervisor.fatal error message Fatal error, sandbox will exit

Bridge (component: "bridge")

Event Level Key Fields Description
bridge.connect info outcome WebSocket connected to control-plane
bridge.disconnect info, warn reason, ws_close_code WebSocket disconnected
bridge.reconnect info attempt, delay_s Reconnection attempt
prompt.start info message_id, model Prompt processing started
prompt.run info message_id, model, outcome, duration_ms Prompt processing completed (wide)
prompt.error error message_id, exc Prompt processing failed

Git Operations (component: "bridge" / "supervisor")

Event Level Key Fields Description
git.clone_start info repo_owner, repo_name, authenticated Clone started
git.clone_complete info repo_path Clone succeeded
git.clone_error error stderr, exit_code Clone failed
git.sync_complete info head_sha Repo sync complete
git.push_start info branch_name, repo_owner, repo_name, token_source Push started
git.push_complete info branch_name Push succeeded
git.push_error warn, error reason or exc, branch_name Push failed

OpenCode (component: "bridge")

Event Level Key Fields Description
opencode.session.ensure info opencode_session_id, action Session created or loaded
opencode.start info — OpenCode process started
opencode.ready info — OpenCode health check OK
opencode.crash error exit_code, restart_count OpenCode process died

Slack Bot (service: "slack-bot")

Handler (component: "handler")

Event Level Key Fields Description
http.request info, warn trace_id, http_method, http_path, http_status, event_type, outcome, duration_ms One per /events or /interactions
slack.event.duplicate debug trace_id, event_id Deduplicated Slack event
thread_session.stale warn trace_id, session_id, channel, thread_ts Stale session in thread
control_plane.create_session info, error trace_id, repo_owner, repo_name, model, session_id, http_status, duration_ms Session creation call
control_plane.send_prompt info, error trace_id, session_id, message_id, http_status, duration_ms, outcome Prompt submission call
slack.app_home error user_id, outcome, slack_error App Home publish failed
kv.get / kv.put / kv.delete error, warn key_prefix, contextual IDs KV storage operation failed

Callback (component: "callback")

Event Level Key Fields Description
http.request info, warn trace_id, http_method, http_path, http_status, session_id, message_id, outcome, duration_ms One per /callbacks/complete
callback.complete info, error trace_id, session_id, message_id, channel, agent_success, tool_call_count, artifact_count, duration_ms Completion callback processed

Extractor (component: "extractor")

Event Level Key Fields Description
control_plane.fetch_events info, error trace_id, session_id, message_id, event_count, tool_call_count, duration_ms Fetch events for a message

Repos (component: "repos")

Event Level Key Fields Description
control_plane.fetch_repos info, warn, error trace_id, repo_count, source, duration_ms Fetch available repositories

Classifier (component: "classifier")

Event Level Key Fields Description
classifier.classify error trace_id, method, channel_id, outcome LLM classification failed

Debugging Scenarios

"What happened to message X?"

Trace a prompt from submission to completion across all services.

# 1. Find the prompt in control-plane
service="control-plane" msg="prompt.enqueue" message_id="<MSG_ID>"

# 2. Was it dispatched to a sandbox?
service="control-plane" msg="prompt.dispatch" message_id="<MSG_ID>"

# 3. Did the sandbox process it?
service="modal-infra" msg="prompt.run" message_id="<MSG_ID>"

# 4. Did it complete?
service="control-plane" msg="prompt.complete" message_id="<MSG_ID>"

Key fields to check:

  • prompt.dispatch → outcome and reason (was a sandbox connected?)
  • prompt.run → outcome and duration_ms (did it succeed? how long?)
  • prompt.complete → total_duration_ms (end-to-end time)

"Why is the sandbox not connecting?"

# 1. Did the sandbox spawn?
service="control-plane" msg="sandbox.spawn" session_id="<SESSION_ID>"
service="control-plane" msg="sandbox.spawned" session_id="<SESSION_ID>"

# 2. Did Modal create it?
service="modal-infra" msg="sandbox.create" sandbox_id="<SANDBOX_ID>"

# 3. Did startup succeed?
service="modal-infra" msg="sandbox.startup" sandbox_id="<SANDBOX_ID>"

# 4. Did the bridge connect?
service="modal-infra" msg="bridge.connect" sandbox_id="<SANDBOX_ID>"

# 5. Did the control-plane accept the WebSocket?
service="control-plane" msg="ws.connect" ws_type="sandbox" session_id="<SESSION_ID>"

Check outcome and reject_reason on ws.connect. Common reasons:

  • sandbox_id_mismatch — sandbox was replaced (stale sandbox reconnecting)
  • token_mismatch — auth token doesn't match
  • sandbox_stopped — session already terminated

"Why did a sandbox spawn fail?"

service="control-plane" msg="sandbox.spawn_failed" session_id="<SESSION_ID>"

Check the error_type and error_message. Then look at the Modal side:

service="modal-infra" msg="sandbox.create" outcome="error"

If the circuit breaker opened:

service="control-plane" component="lifecycle-manager" msg="Circuit breaker open"

"Why did snapshot restore fail?"

# Control-plane side
service="control-plane" msg="sandbox.restore" session_id="<SESSION_ID>"

# Modal side
service="modal-infra" msg="sandbox.restore" outcome="error"

"Slack message isn't getting a response"

Trace from the Slack bot edge inward:

# 1. Did the slack-bot receive the event?
service="slack-bot" msg="http.request" http_path="/events"

# 2. Was it a duplicate?
service="slack-bot" msg="slack.event.duplicate"

# 3. Did classification succeed?
service="slack-bot" msg="classifier.classify"

# 4. Was a session created?
service="slack-bot" msg="control_plane.create_session" trace_id="<TRACE_ID>"

# 5. Was the prompt sent?
service="slack-bot" msg="control_plane.send_prompt" trace_id="<TRACE_ID>"

# 6. Did the callback arrive?
service="slack-bot" msg="callback.complete" trace_id="<TRACE_ID>"

Use the trace_id from step 1 to follow the request into the control-plane and modal-infra logs.

"Errors by type across all services"

level="error" | group by error_type, service, msg | count

"Slow sandbox spawns"

service="control-plane" msg="sandbox.spawned" | where duration_ms > 30000

Or on the Modal side:

service="modal-infra" msg="sandbox.create" | where duration_ms > 30000

"Prompt failures by model"

service="control-plane" msg="prompt.complete" outcome!="success"
  | group by model, outcome | count

"Slack bot → control-plane latency"

service="slack-bot" msg="control_plane.create_session"
  | stats avg(duration_ms), p95(duration_ms), max(duration_ms)

service="slack-bot" msg="control_plane.send_prompt"
  | stats avg(duration_ms), p95(duration_ms), max(duration_ms)

"OpenCode crashes"

service="modal-infra" msg="opencode.crash"
  | group by exit_code, restart_count | count

Redaction Rules

The following must never appear in logs:

  • Authorization headers
  • HMAC tokens (INTERNAL_CALLBACK_SECRET derivatives)
  • GitHub App private keys or installation tokens
  • GitHub OAuth access/refresh tokens
  • Sandbox auth tokens

Body content is omitted unless explicitly whitelisted and truncated.

Log Level Guidelines

Level Use for
debug High-frequency internals (heartbeats, per-token events, cache)
info Wide events for normal operations
warn Degraded but recoverable (stale sessions, fallbacks, retries)
error Failures requiring investigation

Set LOG_LEVEL environment variable per service to control verbosity. Default is info. Use debug in staging, info or warn in production.