Skip to content

Commit 835e664

Browse files
authored
docs: Enhance investigation and integration documentation (Tracer-Cloud#2807)
* docs: Enhance investigation and integration documentation - Added a new "Investigation workflow" section detailing the structured RCA pipeline for alerts. - Updated "Integrations overview" to clarify setup paths and added a catalog of available integrations. - Introduced a new "Benchmark" group in the documentation navigation. - Improved descriptions and examples for using the interactive shell and session management. - Added a link to the hermes_runbook for contributors in the Hermes documentation. - Removed outdated AWS and macOS integration pages to streamline content. * docs: Update link to hermes_runbook in Hermes documentation - Corrected the file extension in the link to the hermes_runbook from `.md` to `.mdx` for accuracy.
1 parent 22e638b commit 835e664

9 files changed

Lines changed: 144 additions & 95 deletions

File tree

docs/docs.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,17 @@
3939
"group": "First steps",
4040
"pages": [
4141
"quickstart",
42+
"investigation-overview",
4243
"deployment",
4344
"faq"
4445
]
4546
},
47+
{
48+
"group": "Benchmark",
49+
"pages": [
50+
"cloudopsbench"
51+
]
52+
},
4653
{
4754
"group": "Community",
4855
"pages": [

docs/features.mdx

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,37 @@ title: "Features"
33
description: "What OpenSRE can do today"
44
---
55

6-
## Coming soon
6+
## Investigation workflow
77

8-
- Investigation workflow (evidence → hypotheses → report)
9-
- Integrations (60+ tools)
10-
- Masking and safe sharing
11-
- Remote runtime investigations
8+
OpenSRE runs a structured RCA pipeline for each alert:
129

10+
1. **Extract context** from the alert payload and connected integrations
11+
2. **Plan evidence collection** across logs, metrics, deploys, and dependencies
12+
3. **Test hypotheses** in a tool-calling loop until confidence is high enough to stop
13+
4. **Publish findings** as `problem.md`, `theory/hypothesis_*.md`, and `report.md` (or JSON with `--output`)
14+
15+
Run investigations from the interactive shell (`opensre`) or one-shot via `opensre investigate -i <alert-file>`. See [Investigations overview](/investigation-overview).
16+
17+
## Integrations (60+ tools)
18+
19+
Connect observability, cloud, databases, incident management, messaging, and workflow systems so investigations can query the same tools your engineers use. Setup via `opensre onboard`, `opensre integrations setup`, or environment variables. See [Integrations overview](/integrations-overview).
20+
21+
## Interactive shell
22+
23+
The TTY REPL (`opensre` with no subcommand) supports:
24+
25+
- Plain-language incident descriptions and follow-up questions
26+
- Slash commands for health checks, investigations, integrations, remote agents, and more (`/help`)
27+
- [Session history](/sessions) with `/sessions`, `/resume`, and `/new`
28+
- [Local agent fleet](/agents) monitoring for Claude Code, Cursor, Codex, and other coding agents
29+
- [Scheduled deliveries](/cron) and [Hermes log watch](/hermes)
30+
31+
## Masking and safe sharing
32+
33+
Reversible masking replaces sensitive infrastructure identifiers (pods, clusters, hostnames, account IDs) with stable placeholders before external LLM calls, then restores originals in user-facing output. See [Masking](/masking).
34+
35+
Command-history redaction and persistence controls live under [Interactive Shell Privacy](/interactive-shell-privacy).
36+
37+
## Remote runtime investigations
38+
39+
Investigate deployed OpenSRE services by name — OpenSRE gathers live deployment status, recent logs, and health probes, then runs the standard RCA pipeline. See [Remote runtime investigation](/remote-runtime-investigation).

docs/hermes.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,5 +150,6 @@ Contributors add scenarios under `tests/synthetic/hermes/` (fixtures, `scenario.
150150
## Further reading
151151

152152
- Implementation lives under `app/integrations/hermes/` in the OpenSRE repository (tailer, parser, classifier, correlator, sinks, CLI wiring).
153+
- Surface-attribution evaluation workflow for contributors: [`docs/hermes_runbook.mdx`](https://github.com/Tracer-Cloud/opensre/blob/main/docs/hermes_runbook.mdx) in the repo (not on the public docs nav).
153154
- If you also run [OpenClaw](/openclaw) alongside Hermes, that page covers the OpenClaw bridge (context lookup and RCA write-back).
154155
- For false-positive risks when rules match both `message` and raw line text, see [issue #1874](https://github.com/Tracer-Cloud/opensre/issues/1874).

docs/integrations-overview.mdx

Lines changed: 44 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,70 @@
11
---
22
title: "Integrations overview"
3+
description: "Connect observability, cloud, data, and messaging tools for automated RCA"
34
---
45

56
OpenSRE investigates alerts by querying the same tools your engineers use. Connect your stack so OpenSRE can pull logs, check recent deploys, map dependencies, and deliver findings where your team already works.
67

7-
## Before you begin
8+
## Two setup paths
89

9-
Make sure you have:
10+
| Path | Best for | How to configure |
11+
| --- | --- | --- |
12+
| **Local CLI** | Self-hosted `opensre` on your laptop or server | `opensre onboard` or `opensre integrations setup`, plus env vars |
13+
| **Hosted (OpenSRE Cloud)** | Org-wide connectors shared across users | [app.tracer.cloud](https://app.tracer.cloud) web app |
1014

11-
- A OpenSRE account — you can create one at [app.tracer.cloud/sign-up](http://app.tracer.cloud/sign-up)
12-
- Access credentials for the tools you want to connect
13-
- Permissions to create API keys
15+
Most users start with the **local CLI** path. You need credentials for the tools you want to connect and permission to create API keys.
1416

15-
## Enterprise integrations
17+
## Local CLI setup
1618

17-
Enterprise integrations are configured through the OpenSRE web app at [app.tracer.cloud](https://app.tracer.cloud). Once connected, they are available to all users in your organization.
18-
19-
**Observability.** OpenSRE reads your logs, metrics, traces, and alerts to build investigation hypotheses. Connect [Grafana](/grafana) and [Datadog](/datadog).
19+
```bash
20+
opensre integrations setup
21+
```
2022

21-
**Infrastructure.** OpenSRE maps your environment and dependencies across clouds and clusters. Connect [AWS](/aws).
23+
Or set environment variables — OpenSRE picks them up automatically on the next run.
2224

23-
**Communication.** OpenSRE delivers investigation reports where your team already works. Connect [Slack, Discord, or Telegram](/messaging/index).
25+
### Verify
2426

25-
<Frame>
26-
![Integrations](/images/integrations.png)
27-
</Frame>
28-
29-
### Configure enterprise integrations
27+
```bash
28+
opensre integrations verify
29+
opensre integrations verify datadog
30+
```
3031

31-
Each integration follows a simple, guided flow. Let's look at Datadog as an example.
32+
Inside the interactive shell you can also run `/integrations list`, `/integrations verify <name>`, and `/health`.
3233

33-
1. Name - Helps distinguish between different instances
34-
2. API Key - in Datadog, create a new API key and copy and paste into OpenSRE.
35-
3. Application Key - In Datadog, create a new Application Key and copy and paste into OpenSRE.
34+
## Integration catalog
3635

37-
<Frame>
38-
![Connect Datadog](/images/connect_datadog.png)
39-
</Frame>
36+
| Category | Integrations |
37+
| --- | --- |
38+
| **Observability and incidents** | [Alertmanager](/alertmanager), [Azure Monitor](/azure-monitor), [Better Stack](/betterstack), [Coralogix](/coralogix), [Datadog](/datadog), [Grafana](/grafana), [Hermes](/hermes), [Honeycomb](/honeycomb), [Incident.io](/incident_io), [OpenObserve](/openobserve), [OpsGenie](/opsgenie), [Sentry](/sentry), [SignOz](/signoz), [Splunk](/splunk), [Victoria Logs](/victoria_logs) |
39+
| **Cloud and infrastructure** | [AWS](/aws), [Argo CD](/argocd), [Helm](/helm), [Jenkins](/jenkins), [Vercel](/vercel) |
40+
| **Databases and data platforms** | [Azure SQL](/azure-sql), [ClickHouse](/clickhouse), [Kafka](/kafka), [MariaDB](/mariadb), [MongoDB](/mongodb), [MongoDB Atlas](/mongodb-atlas), [MySQL](/mysql), [OpenSearch](/opensearch), [PostgreSQL](/postgresql), [RabbitMQ](/rabbitmq), [RDS](/rds), [Snowflake](/snowflake), [Supabase](/supabase) |
41+
| **Source control and collaboration** | [Bitbucket](/bitbucket), [GitHub](/github), [GitHub Actions](/integrations/github-actions), [GitLab](/gitlab), [Google Docs](/google-docs), [Jira](/jira), [Trello](/integrations/trello) |
42+
| **Workflow orchestration** | [Airflow](/airflow), [Dagster](/dagster), [Prefect](/prefect) |
43+
| **Messaging** | [Slack](/messaging/slack), [Discord](/messaging/discord), [Telegram](/messaging/telegram), [WhatsApp](/messaging/whatsapp), [Twilio SMS](/messaging/twilio-sms) |
44+
| **AI coding assistants** | [OpenClaw](/openclaw) |
4045

41-
## Local integrations
46+
[Multi-instance integrations](/multi-instance-integrations) let you register multiple clusters, regions, or accounts per provider (for example prod and staging Grafana).
4247

43-
Local integrations are configured on the machine running OpenSRE — via the CLI setup wizard or environment variables. They give OpenSRE access to your databases, source control, error trackers, and workflow tools.
48+
## Hosted (OpenSRE Cloud) setup
4449

45-
```bash
46-
opensre integrations setup
47-
```
50+
Enterprise connectors are configured through the OpenSRE web app at [app.tracer.cloud](https://app.tracer.cloud). Once connected, they are available to all users in your organization.
4851

49-
Or set environment variables and OpenSRE picks them up automatically.
52+
**Observability.** Connect [Grafana](/grafana) and [Datadog](/datadog) for logs, metrics, traces, and alerts.
5053

51-
| Category | Integrations |
52-
| --- | --- |
53-
| **Observability** | [Better Stack](/betterstack), [Honeycomb](/honeycomb), [Coralogix](/coralogix), [Sentry](/sentry) |
54-
| **Databases** | [MongoDB](/mongodb), [MariaDB](/mariadb), [ClickHouse](/clickhouse), [Kafka](/kafka) |
55-
| **Source control** | [GitHub](/github), [GitLab](/gitlab), [Bitbucket](/bitbucket) |
56-
| **Kubernetes releases** | [Helm](/helm) |
57-
| **Deployments** | [Vercel](/vercel) |
58-
| **Orchestration** | [Dagster](/dagster), [Prefect](/prefect) |
59-
| **Incident management** | [OpsGenie](/opsgenie), [Jira](/jira) |
60-
| **Reporting** | [Google Docs](/google-docs) |
61-
| **AI coding assistants** | [OpenClaw](/openclaw) |
54+
**Infrastructure.** Connect [AWS](/aws) to map your environment across clouds and clusters.
6255

63-
### Verify local integrations
56+
**Communication.** Deliver investigation reports via [Slack, Discord, or Telegram](/messaging/index).
6457

65-
After setup, confirm everything is working:
58+
<Frame>
59+
![Integrations](/images/integrations.png)
60+
</Frame>
6661

67-
```bash
68-
opensre integrations verify
69-
```
62+
### Example: Datadog in the web app
7063

71-
To check a specific integration:
64+
1. **Name** — distinguish between instances (for example `prod` vs `staging`)
65+
2. **API key** — create in Datadog and paste into OpenSRE
66+
3. **Application key** — create in Datadog and paste into OpenSRE
7267

73-
```bash
74-
opensre integrations verify mongodb
75-
```
68+
<Frame>
69+
![Connect Datadog](/images/connect_datadog.png)
70+
</Frame>

docs/integrations/aws.mdx

Lines changed: 0 additions & 20 deletions
This file was deleted.

docs/integrations/macos.mdx

Lines changed: 0 additions & 16 deletions
This file was deleted.

docs/investigation-overview.mdx

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
---
22
title: "Investigations overview"
3+
description: "Start investigations from the interactive shell or CLI, review artifacts, and use plain-language REPL routing"
34
---
45

56
Use this workflow when running OpenSRE via the local `opensre` binary.
@@ -23,6 +24,32 @@ Then describe the incident or paste alert context at the prompt. Use `/help` for
2324

2425
When `LLM_PROVIDER` is `openai` or `codex`, `/effort` sets how much reasoning the model applies for that REPL session (`low`, `medium`, `high`, `xhigh`, or `max`). Run `/effort` with no arguments to print the current level and usage; `/status` includes the same field. Other providers ignore this setting (the shell prints a hint). You can also set `OPENSRE_REASONING_EFFORT` to `low`, `medium`, `high`, or `xhigh` in the environment for non-interactive defaults.
2526

27+
### Plain language and compound requests
28+
29+
You do not need to memorize every slash command. Describe what you want in natural language and the REPL planner maps intent to the right actions — often a sequence of slash commands executed in order.
30+
31+
Examples:
32+
33+
| You type | Typical action |
34+
| --- | --- |
35+
| "check the health of my opensre and then show connected services" | `/health`, then `/integrations list` |
36+
| "verify datadog" | `/integrations verify datadog` |
37+
| "run the sample alert investigation" | `/investigate` with a built-in template |
38+
| "connect to my remote EC2 instance and send it hello world" | `/remote` subcommands, then a remote investigation |
39+
40+
List-style questions route to **per-domain** commands (there is no global `/list`):
41+
42+
| Intent | Command |
43+
| --- | --- |
44+
| Connected integrations | `/integrations list` |
45+
| Investigation tools | `/tools` |
46+
| Background tasks | `/tasks` |
47+
| MCP servers | `/mcp list` |
48+
| Cron deliveries | `/cron list` |
49+
| Past REPL sessions | `/sessions` |
50+
51+
For procedural questions ("how do I configure Datadog?"), the assistant answers from docs without running mutating commands unless you ask it to.
52+
2653
### Direct investigation (`opensre investigate`)
2754

2855
Pass an alert payload to `opensre investigate`:
@@ -33,6 +60,8 @@ opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json
3360

3461
You can also use `--interactive` to pick an input file from your terminal UI.
3562

63+
For deployed services by name, see [Remote runtime investigation](/remote-runtime-investigation).
64+
3665
## 2) Review investigation artifacts
3766

3867
A local run produces structured RCA artifacts such as:
@@ -56,6 +85,10 @@ Each run captures:
5685
- tool outputs collected from connected integrations
5786
- final diagnosis and recommended remediation steps
5887

88+
## Session continuity
89+
90+
REPL sessions are persisted under `~/.opensre/sessions/`. When you exit, the shell prints a `/resume <session-id>` hint so you can pick up later. See [Session history](/sessions).
91+
5992
## Chat
6093

6194
For local binary usage, the primary workflow is file-based (`problem.md`, `report.md`, and optional JSON output).
@@ -69,4 +102,4 @@ local investigation completes.
69102

70103
<Frame>
71104
![Slack Alert](/images/slack_alert.png)
72-
</Frame>
105+
</Frame>

docs/masking.mdx

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@ Masking is **off by default**. Enable it per investigation via environment varia
1111

1212
## How it works
1313

14-
1. When masking is enabled, `node_investigate` replaces sensitive identifiers in the `evidence` dict with stable placeholders like `<POD_0>`, `<NAMESPACE_0>`, `<CLUSTER_1>`. The placeholder→original map is stored in the investigation state under `masking_map`.
15-
2. The diagnosis LLM receives masked evidence, so raw identifiers never hit the model.
16-
3. After the LLM returns its root-cause analysis, `diagnose_root_cause` unmasks the output so downstream state and display code see real identifiers.
17-
4. `publish_findings` runs a final unmask pass on the Slack message and blocks before delivery, as defence in depth.
14+
1. When masking is enabled, the investigation step replaces sensitive identifiers in collected evidence with stable placeholders like `<POD_0>`, `<NAMESPACE_0>`, `<CLUSTER_1>`. The placeholder→original map is stored in investigation state.
15+
2. The diagnosis model receives masked evidence, so raw identifiers never hit the external LLM.
16+
3. After the model returns its root-cause analysis, OpenSRE restores real identifiers in downstream state and display output.
17+
4. Report delivery (for example Slack) runs a final unmask pass before sending, as defence in depth.
1818

1919
The same identifier always maps to the same placeholder within a single investigation, so the LLM's reasoning about `<POD_0>` remains coherent.
2020

docs/sessions.mdx

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,28 @@ If the prefix is ambiguous (matches more than one session), OpenSRE will ask you
7575

7676
---
7777

78+
## When you exit
79+
80+
When you leave the REPL, OpenSRE prints your session ID so you can resume later:
81+
82+
| Exit path | Behavior |
83+
| --- | --- |
84+
| `/exit` or `/quit` | Prints `Resume this session with:` and `/resume <session-id>`, then goodbye |
85+
| **Ctrl+C** twice within 2 seconds | Same resume hint, then exits |
86+
| **Ctrl+D** (EOF) | Same resume hint when no dispatch is running |
87+
88+
Example:
89+
90+
```
91+
Resume this session with:
92+
/resume 3f8a1c2d-…
93+
Goodbye!
94+
```
95+
96+
Copy the `/resume …` line from the terminal, or find the session later with `/sessions`.
97+
98+
---
99+
78100
## Starting fresh with context — `/new`
79101

80102
`/new` closes the current session and opens a new one, but carries the conversation forward so you don't lose context:

0 commit comments

Comments
 (0)