feat(insights): restore bounded investigation agent

[izadoesdev]

Summary

• Replace the deterministic terminal-decision engine and ~2,000-line prose validator with a bounded ToolLoopAgent.
• Let the model choose one of up to five eligible regressions, choose/read evidence, stop, cite receipts, and decide action_ready, needs_context, monitor, or not_a_problem.
• Keep only hard backend boundaries deterministic: tenant/date scope, measured facts, lifecycle, budgets, billing, exact repair authorization, and persistence.
• Fail invalid agent runs so jobs retry; never turn model failure into a persisted monitor decision.
• Keep this slice explicitly regression-only: positive signals create no model or backend decision.
• Let the agent cite detector receipts after a real evidence read, so unrelated fresh reads are not forced into customer cards.
• Rename the dashboard section from “Why it matters” to “What changed” to match the grounded output contract.
• Preserve read-only, anonymized production-history evaluation and self-hosted AI operation without requiring Autumn.
• Remove the unscoped live-now anomaly evidence path; the agent can only use time-scoped detector and evidence receipts.

This supersedes the overengineered implementation approach in #565 without modifying or closing that PR.

Simplification

• 32 touched paths, but 1 file added and 3 deleted.
• 2,055 additions / 3,584 deletions: net -1,529 LOC.
• Deletes terminal-decision.ts and the validate.ts implementation/test pair.
• Removes fake semantic regex grading, deterministic copy generation, synthetic copy-quality claims, and failure-to-monitor fallback behavior.

Evidence

• Full pre-commit workspace typecheck/build gate: 36/36 tasks passed.
• Full pre-push monorepo test gate: 27/27 tasks passed.
• Focused affected suites: 3,805 tests passed; 20 service-dependent integration tests skipped as expected.
• Targeted lint and diff checks pass. Repo-wide lint still reports unrelated pre-existing formatting/regex findings in basket and ClickHouse tooling.
• Final read-only production shadow: 3 high-volume sites × 3 historical dates = 9 cases; 6 useful cards, 0 engine errors, 0 incomplete detection, 0 failed/truncated evidence reads; 50–63 visible words per card; p50 11.1s / p95 17.7s.
• The model selected backend rank 2 and rank 3 candidates in real cases, demonstrating that triage is not a deterministic top-one fallback.

Deliberate scope

• This is a bounded regression investigator, not the finished PR-opening product.
• The production shadow replays historical analytics with currently surviving definitions; it is not a full as-of replay of deleted/disabled definition state.
• Exact action_ready repairs are covered by unit/synthetic tests. The real sampled cards required external deploy/code/campaign context and therefore used needs_context.
• Answer ingestion, deploy/code/campaign connectors, and repository PR creation are the next closed-loop slice.

AI disclosure

AI assistance was used. The implementation, tests, production-shadow output, and final diff were reviewed and verified by the maintainer.

---

Summary by cubic

Replaced the deterministic insights engine with a bounded investigation agent that triages up to five regressions, reads evidence, and returns grounded decisions under 100 words. Investigations are now time-scoped to each website’s timezone; usage is tracked and billed when configured; the UI says “What changed”, and the worker requires AI_GATEWAY_API_KEY/AI_API_KEY.

• New Features

  • Added a bounded agent using ai’s ToolLoopAgent (max 5 steps, 2 tool calls, 120s, 100-word output).
  • Keeps investigations time-scoped to the website timezone to avoid cross-window reads.
  • Tracks usage and bills when Autumn billing is configured; no-op when disabled.
  • Supports four outcomes; invalid runs fail and retry; triages eligible signals in rank order and cites receipts only after real evidence reads.

• Migration

  • If INSIGHTS_WORKER_ENABLED=true, set AI_GATEWAY_API_KEY or AI_API_KEY (added to docker-compose.selfhost.yml).
  • UI and view model now use whatChanged instead of whyItMatters.

^{Written for commit f398f6a. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
databuddy-status	Ready	Preview, Comment	Jul 15, 2026 6:41pm

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
dashboard	Skipped		Jul 15, 2026 6:41pm
documentation	Skipped		Jul 15, 2026 6:41pm

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9bb0077f-a260-41e8-94c9-1ff8395d73fb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/insights-agent-simplification

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[unkey-deploy]

The latest updates on your projects. Learn more about Unkey Deploy

Name	Status	Preview	Inspect	Updated (UTC)
api (preview)	Ready	Visit Preview	Inspect	Jul 15, 2026 6:41pm

[greptile-apps]

Greptile Summary

This PR replaces deterministic insight decisions with a bounded investigation agent. The main changes are:

• Adds model-selected evidence reads and decision submission.
• Keeps persistence, repair authorization, and lifecycle handling in the backend.
• Updates insight evaluation and production-shadow flows.
• Renames the dashboard copy section to “What changed.”

Confidence Score: 4/5

The production read-only investigation path can invoke live anomaly detection.

• Candidate selection and evidence citations are scoped to one signal.
• Invalid agent runs throw instead of persisting a fallback decision.
• The production reader must disable live anomaly detection before merging.

apps/insights/src/generation.ts

Security Review

The new evidence reader keeps website, date, and signal scope server-controlled. The production investigation path still permits live anomaly detection while running in dry-run mode.

Important Files Changed

Filename	Overview
apps/insights/src/agent.ts	Adds bounded candidate selection, evidence reads, and validated finding materialization.
apps/insights/src/generation.ts	Integrates the agent into generation, billing, persistence, and observation handling; the evidence-reader setup leaves live anomaly detection enabled.
packages/ai/src/ai/tools/insights-agent-tools.ts	Adds strict request schemas and signal-bound evidence reads.
packages/ai/src/ai/insights/ops-context.ts	Adds a live anomaly-detection switch that defaults to enabled.
packages/shared/src/insights.ts	Simplifies the needs-context decision shape.
packages/evals/src/insight-production-shadow.ts	Updates historical shadow evaluation to use the bounded agent and disables live anomaly detection there.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Detected regressions] --> B[Eligible candidate shortlist]
  B --> C[Bounded investigation agent]
  C --> D[Signal-scoped evidence reader]
  D --> E[Validated decision]
  E --> F[Persist insight and observation]
  F --> G[Resolve prior insights and effects]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[Detected regressions] --> B[Eligible candidate shortlist]
  B --> C[Bounded investigation agent]
  C --> D[Signal-scoped evidence reader]
  D --> E[Validated decision]
  E --> F[Persist insight and observation]
  F --> G[Resolve prior insights and effects]

Loading

_{Reviews (1): Last reviewed commit: "feat(insights): restore bounded investig..." | Re-trigger Greptile}