feat: infrahub-collect troubleshooting-bundle tool (INFP-415)

.github/workflows/ci.yml

@@ -105,7 +105,7 @@ jobs:
       - name: Install tini
         run: apt-get update && apt-get install -y tini
 
-      - name: Build infrahub-backup binary
+      - name: Build binaries (infrahub-backup, infrahub-taskmanager, infrahub-collect)
         run: make build
 
       - name: Install Python dependencies
@@ -161,7 +161,7 @@ jobs:
       - name: Install tini
         run: apt-get update && apt-get install -y tini
 
-      - name: Build infrahub-backup binary
+      - name: Build binaries (infrahub-backup, infrahub-taskmanager, infrahub-collect)
         run: make build
 
       - name: Install Python dependencies

AGENTS.md

@@ -4,20 +4,21 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-`infrahub-ops-cli` is a Go-based toolset for managing and maintaining Infrahub instances. The project provides two specialized CLI binaries:
+`infrahub-ops-cli` is a Go-based toolset for managing and maintaining Infrahub instances. The project provides three specialized CLI binaries:
 
 - **infrahub-backup** - Backup/restore operations and environment detection
 - **infrahub-taskmanager** - Task manager (Prefect) maintenance operations
+- **infrahub-collect** - Troubleshooting-bundle collection (read-only diagnostics) for support
 
-Both tools share common internal application logic but expose different commands through their respective main entry points.
+All three tools share common internal application logic but expose different commands through their respective main entry points.
 
 ## Common Development Commands
 
 ### Building and Running
 
-- `make build` - Build both binaries to `bin/infrahub-backup` and `bin/infrahub-taskmanager`
-- `make build-all` - Cross-compile both binaries for Linux, Darwin, and Windows (amd64/arm64)
-- `make install` - Build and install both binaries to `$GOPATH/bin`
+- `make build` - Build all three binaries to `bin/infrahub-backup`, `bin/infrahub-taskmanager`, and `bin/infrahub-collect`
+- `make build-all` - Cross-compile all three binaries for Linux, Darwin, and Windows (amd64/arm64)
+- `make install` - Build and install all three binaries to `$GOPATH/bin`
 - `make clean` - Remove build artifacts
 
 ### Testing and Quality
@@ -40,7 +41,7 @@ Whenever Go modules change (any modification to `go.mod` or `go.sum` — adding,
 
 ## Architecture
 
-The codebase follows a command-pattern architecture using Cobra for CLI structure, with two separate binary entry points sharing common internal logic:
+The codebase follows a command-pattern architecture using Cobra for CLI structure, with three separate binary entry points sharing common internal logic:
 
 ### Core Components
 
@@ -54,50 +55,64 @@ The codebase follows a command-pattern architecture using Cobra for CLI structur
    - Commands: `flush flow-runs`, `flush stale-runs`, `environment detect`, `environment list`, `version`
    - Uses shared application logic from `src/internal/app`
 
-3. **src/internal/app/app.go** - Core application logic
+3. **src/cmd/infrahub-collect/main.go** - Troubleshooting-bundle tool entry point
+   - Defines root command with troubleshooting-bundle collection
+   - Commands: `create`, `environment detect`, `environment list`, `version`
+   - Uses shared application logic from `src/internal/app`
+
+4. **src/internal/app/app.go** - Core application logic
    - `InfrahubOps` struct - Main application controller
    - `CommandExecutor` - Handles Docker Compose and system command execution
    - Environment detection (Docker vs Kubernetes)
    - Docker project discovery and validation
-   - Shared by both CLI tools
+   - Shared by all three CLI tools
 
-4. **src/internal/app/backup.go** - Backup and restore operations
+5. **src/internal/app/backup.go** - Backup and restore operations
    - Creates tar.gz backups with metadata JSON
    - Backs up Neo4j database, PostgreSQL (task-manager), and artifacts
    - Implements safe backup with container stopping/starting
    - Restore validates metadata and handles version compatibility
 
-5. **src/internal/app/taskmanager.go** - Task management operations
+6. **src/internal/app/taskmanager.go** - Task management operations
    - PostgreSQL database connection management
    - Flow run cleanup operations (completed/failed/cancelled)
    - Stale run cancellation (stuck in running state)
    - Uses embedded Python scripts for Prefect API operations
 
-6. **src/internal/app/utils.go** - Utility functions
+7. **src/internal/app/collect*.go** - Troubleshooting-bundle collection
+   - `collect.go` - Orchestrator (`CollectBundle`), collector run plan, staging/archive lifecycle
+   - `collect_manifest.go` - `bundle_information.json` manifest and per-collector results
+   - `collect_logs.go` - Per-replica service log collector (backend-agnostic)
+   - `collect_diagnostics.go` - Database, message-queue, cache, task-worker, task-manager, and server collectors
+   - `collect_metrics.go` - Container resource metrics collector
+   - `collect_extras.go` - Opt-in `--include-backup` and `--benchmark` collectors
+   - `masking.go` - Key-name secret masking for env and config dumps
+
+8. **src/internal/app/utils.go** - Utility functions
    - File operations, checksum validation
    - Environment variable handling
    - Version detection and comparison
 
-7. **src/internal/app/cli.go** - Shared CLI configuration
+9. **src/internal/app/cli.go** - Shared CLI configuration
    - `ConfigureRootCommand()` - Sets up common flags and configuration
    - `AttachEnvironmentCommands()` - Adds environment detection commands
-   - Shared between both binaries
+   - Shared between all three binaries
 
 ### Key Design Patterns
 
-- **Split Binary Architecture**: Two specialized binaries sharing common internal logic for focused functionality
+- **Split Binary Architecture**: Three specialized binaries sharing common internal logic for focused functionality
 - **Embedded Scripts**: Python scripts are embedded using Go's embed package (src/internal/app/scripts directory)
 - **Docker Compose Integration**: All operations work through Docker Compose commands
 - **Project-based Operations**: Can target specific Docker Compose projects with `--project` flag
 - **Streaming Output**: Commands stream output in real-time for user feedback
-- **Shared Configuration**: Both binaries use the same configuration system and environment variables
+- **Shared Configuration**: All binaries use the same configuration system and environment variables
 
 ## Docker Compose Dependencies
 
-Both tools assume Infrahub is deployed using Docker Compose with these service names:
+All tools assume Infrahub is deployed using Docker Compose with these service names:
 
-- `database` (Neo4j) - Used by infrahub-backup
-- `task-manager-db` (PostgreSQL) - Used by both tools
+- `database` (Neo4j) - Used by infrahub-backup and infrahub-collect
+- `task-manager-db` (PostgreSQL) - Used by infrahub-backup and infrahub-taskmanager
 - `infrahub-server`, `task-worker`, `task-manager`, `task-manager-background-svc` - Application containers
 - `cache`, `message-queue` - Infrastructure services
 
@@ -266,7 +281,10 @@ The codebase uses explicit error wrapping with `fmt.Errorf` for context. All com
 
 - Go 1.25.0 + kloset v1.0.13 (Plakar core), integration-fs (storage), cobra, logrus, viper (002-plakar-integration)
 - Plakar repository (local filesystem or S3 via integration backends) (002-plakar-integration)
+- Go 1.25.0 + cobra, viper, logrus; Docker/Kubernetes via `docker`/`kubectl` CLI shell-out through `CommandExecutor` — no client-go or Docker SDK (003-collect-tool)
+- Local filesystem bundle output under `--output-dir` (default `./infrahub_bundles`); no network egress beyond the target deployment (003-collect-tool)
 
 ## Recent Changes
 
+- 003-collect-tool: Added `infrahub-collect` troubleshooting-bundle binary (collector framework, log/metrics primitives on both backends, key-name secret masking, bundle manifest)
 - 002-plakar-integration: Added kloset (Plakar core library), integration-fs (filesystem storage/exporter), cobra, logrus

Makefile

@@ -1,7 +1,7 @@
 .PHONY: build build-all clean install test lint fmt vet help docker-build docker-build-multi docker-push
 
 # Variables
-BINARIES=infrahub-backup infrahub-taskmanager
+BINARIES=infrahub-backup infrahub-taskmanager infrahub-collect
 BUILD_DIR=$(shell pwd)/bin
 SRC_ROOT=./src
 VERSION?=

dev/README.md

@@ -0,0 +1,28 @@
+# Developer Documentation
+
+Durable engineering knowledge for `infrahub-ops-cli`, extracted from completed
+feature specs (`specs/`) as they land. The lifecycle is
+`specs/ → dev/{adr,knowledge,guidelines}/`, after which the spec is archived
+under `specs/archive/`.
+
+- **adr/** — Architecture Decision Records: why a structural choice was made, and the alternatives rejected.
+- **knowledge/** — How the system works today (descriptive).
+- **guidelines/** — Prescriptive conventions to follow in future code.
+
+## Current ADRs
+
+- [0001-shell-out-to-docker-kubectl-clis.md](adr/0001-shell-out-to-docker-kubectl-clis.md) - Use the `docker`/`kubectl` CLIs instead of client libraries
+- [0002-non-fatal-timeout-bounded-collectors.md](adr/0002-non-fatal-timeout-bounded-collectors.md) - Non-fatal, timeout-bounded collector framework
+- [0003-read-only-collection.md](adr/0003-read-only-collection.md) - Collection never mutates the workload lifecycle
+- [0004-key-name-secret-masking.md](adr/0004-key-name-secret-masking.md) - Key-name secret masking through a single choke-point
+- [0005-uniform-bundle-and-manifest.md](adr/0005-uniform-bundle-and-manifest.md) - Single `.tar.gz` bundle + manifest, uniform across environments
+- [0006-include-backup-reuses-createbackup.md](adr/0006-include-backup-reuses-createbackup.md) - `--include-backup` reuses `CreateBackup` non-interactively
+- [0007-offline-by-default-opt-in-benchmark.md](adr/0007-offline-by-default-opt-in-benchmark.md) - Offline by default; benchmark/image pulls opt-in
+
+## Current Knowledge
+
+- [infrahub-collect.md](knowledge/infrahub-collect.md) - The troubleshooting-bundle tool: CLI, bundle layout, manifest, backend seam
+
+## Current Guidelines
+
+- [collectors.md](guidelines/collectors.md) - Conventions for writing and maintaining collect collectors

dev/adr/0001-shell-out-to-docker-kubectl-clis.md

@@ -0,0 +1,25 @@
+# 1. Shell out to `docker`/`kubectl` CLIs instead of client libraries
+
+**Status**: Accepted
+**Date**: 2026-07-09
+**Source**: specs/archive/003-collect-tool/research.md (R1)
+
+## Context
+
+`infrahub-collect` (and the existing `infrahub-backup`/`infrahub-taskmanager`) must operate against both Docker Compose and Kubernetes deployments of Infrahub. The tools are copied onto customer hosts and CI runners where no Go toolchain or package manager is available (constitution Principle III), and they must cross-compile `CGO_ENABLED=0` for linux/darwin/windows on amd64/arm64.
+
+## Decision
+
+All Docker and Kubernetes interaction shells out to the `docker` and `kubectl` binaries through the shared `CommandExecutor`. No Kubernetes `client-go`, no Docker Engine SDK. New collectors reuse the existing backend abstraction (`EnvironmentBackend`) and executor rather than introducing a client dependency.
+
+## Consequences
+
+- No new Go module dependencies for collect; the vendor hash and cross-compilation stay trivial.
+- `kubectl` handles kubeconfig, contexts, and auth plugins (OIDC, cloud IAM) for free — none of that has to be reimplemented.
+- Remote Docker contexts keep working because the `docker` CLI resolves them.
+- The cost is that behaviour depends on the CLIs being present and on parsing their text/JSON output; absence is detected and reported with an actionable error (`ErrCLIUnavailable`).
+
+## Alternatives Considered
+
+- **`k8s.io/client-go`** — rejected: large dependency tree, auth-plugin complexity, and inconsistent with the existing backends.
+- **Docker Engine API over the socket** — rejected: breaks remote-context workflows the `docker` CLI handles transparently.

dev/adr/0002-non-fatal-timeout-bounded-collectors.md

@@ -0,0 +1,31 @@
+# 2. Non-fatal, timeout-bounded collector framework
+
+**Status**: Accepted
+**Date**: 2026-07-09
+**Source**: specs/archive/003-collect-tool/research.md (R2)
+
+## Context
+
+The primary use case for a troubleshooting bundle is a **degraded** Infrahub instance — a service is down, a pod is crash-looping, an exec hangs. A collection tool that aborts on the first failure, or that hangs on a wedged container, is useless exactly when it is needed most (spec FR-009, SC-005).
+
+## Decision
+
+Collection is a fixed, ordered list of named `collector` units run sequentially by an orchestrator (`CollectBundle` / `runCollectPlan`). Each collector:
+
+- Runs every subprocess under a context timeout (`exec.CommandContext`): 60s for exec/status dumps, 5 minutes for log downloads and file copies. A timeout is recorded verbatim as `timed out after <duration>`.
+- Is **non-fatal**: an error is caught, logged as a `WARN`, and recorded in the bundle manifest as `failed` (with a reason) or `skipped` (when the collector does not apply); the run continues and the command still exits 0 with a partial bundle.
+- A panic in a collector is recovered and converted into a recorded `failed` outcome, so no single collector can abort the whole run.
+
+Only a missing environment, an unwritable output directory, or a failure to write the archive is fatal (exit 1).
+
+## Consequences
+
+- A bundle from a broken instance is still produced, and the manifest accounts for 100% of attempted collectors with an explicit outcome.
+- The tool never hangs on a degraded dependency — timeouts convert a hang into a recorded failure.
+- Sequential execution keeps the streamed progress readable and avoids hammering a fragile instance; wall time stays within the 5-minute budget (SC-001).
+
+## Alternatives Considered
+
+- **Parallel collectors** — rejected for v1: interleaved output, higher load on a degraded instance, and no evidence the time budget needs it.
+- **Abort on first failure** — rejected: contradicts the degraded-instance use case.
+- **Unbounded subprocesses** — rejected: a single wedged `exec`/`cp` would stall the entire run (the exact failure mode a troubleshooting tool must survive).

dev/adr/0003-read-only-collection.md

@@ -0,0 +1,28 @@
+# 3. Collection is strictly read-only with respect to the workload lifecycle
+
+**Status**: Accepted
+**Date**: 2026-07-09
+**Source**: specs/archive/003-collect-tool/spec.md (FR-010, SC-003)
+
+## Context
+
+`infrahub-backup` may stop and restart application containers while it snapshots the database (Neo4j Community). `infrahub-collect` is meant to be run against production instances at any time — including while they are degraded — and often by customers themselves. A diagnostic tool that could stop, restart, or scale a workload would be unsafe to hand out.
+
+## Decision
+
+`infrahub-collect` never stops, restarts, or scales any container, pod, or workload belonging to the deployment. It uses only read/exec operations (`pods/log`, `pods/exec`, `docker logs`, `docker exec`, `docker stats`, `kubectl top`). The stop/start/scale code paths in the shared backend are simply not reached by the collect plan.
+
+Two deliberate exceptions, both operating on the **tool's own** transient resources, not the deployment's:
+
+- `--benchmark` creates and then removes a short-lived benchmark container/pod it owns.
+- `--include-backup` delegates to `CreateBackup`, whose documented behaviour may stop/restart app containers; this is opt-in and called out in the docs (see ADR 0006).
+
+## Consequences
+
+- Safe to run against a live or degraded production instance without risk of an outage attributable to the tool.
+- Kubernetes RBAC for the tool needs only read/exec verbs — no write or scale permissions.
+- Interrupt handling is simpler: there is no workload state to restore on SIGINT, only a staging directory to clean up.
+
+## Alternatives Considered
+
+- **Reuse the backup tool's stop/collect/start pattern for consistency** — rejected: it would make the tool unsafe for its core (production, degraded) use case and require elevated permissions.

dev/adr/0004-key-name-secret-masking.md

@@ -0,0 +1,32 @@
+# 4. Key-name secret masking through a single choke-point
+
+**Status**: Accepted
+**Date**: 2026-07-09
+**Source**: specs/archive/003-collect-tool/research.md (R5)
+
+## Context
+
+A troubleshooting bundle captures environment variables and configuration dumps (server env, server API config, Redis `CONFIG GET`, RabbitMQ `environment`/`status`). These contain secrets. Support engineers receive these bundles, so plaintext secrets must not be written (spec FR-008). The prior Python tool masked by key name, and parity was required.
+
+## Decision
+
+A pure function masks values whose **key name** contains any of `pass`, `secret`, `token`, `key` (case-insensitive substring), replacing the value with `********`. Every dump that can carry secrets is routed through this single choke-point (`masking.go`), and the set of masked outputs is enumerated explicitly:
+
+1. `infrahub-server` environment (`maskEnvOutput`)
+2. server API configuration dump (`maskJSON`, recurses through nested JSON)
+3. Redis `CONFIG GET '*'` (`maskConfigPairs`)
+4. RabbitMQ `rabbitmqctl environment` and `status` (`maskErlangConfig`)
+
+The token list is a documented **minimum**; `pass` was added over the spec's `password` to also catch `requirepass`/`default_pass`. Raw service logs are written as-is (parity with the prior tool; the docs warn users to review before sharing).
+
+## Consequences
+
+- A single, table-tested function governs masking; new sensitive dumps must be added to the enumerated list and routed through it, so masking cannot be silently skipped.
+- Over-masking (a non-secret key that happens to contain `key`) is accepted as safe; under-masking is not.
+- Known limitation: credentials embedded in a value under a non-matching key (e.g. a connection string `..._URL=postgres://user:pw@host`) are not masked. This is a documented parity limitation, not a regression.
+
+## Alternatives Considered
+
+- **Value-pattern scrubbing inside logs** — out of scope for v1 (documented); logs are copied verbatim.
+- **Per-collector ad-hoc masking** — rejected: a single choke-point with an enumerated call-site list is enforceable; scattered masking rots.
+- **Allowlist of safe keys** — rejected: inverts the prior tool's behaviour and breaks parity.