Skip to content

[WEB-4956] Add no_tools_found Sentry event to diagnose empty discovery scans#215

Merged
anonpran merged 3 commits into
mainfrom
nanda/discovery-no-tools-sentry-event
Jun 26, 2026
Merged

[WEB-4956] Add no_tools_found Sentry event to diagnose empty discovery scans#215
anonpran merged 3 commits into
mainfrom
nanda/discovery-no-tools-sentry-event

Conversation

@anonpran

@anonpran anonpran commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • A discovery scan that finds zero tools is currently silent — from the backend/Sentry you cannot tell apart an errored scan, a missed detection (tool present, not detected), and a genuinely empty device; nor, for the empty case, an enumeration miss (no human user found) from a timing race (binaries not on disk yet).
  • This emits one fire-and-forget Sentry event (phase=no_tools_found) on the no-tools path, carrying the discriminators needed to diagnose it at a glance.

Why homes_enumerated is the key field

The existing metrics report user_count = len(all_users) after the if not all_users: all_users = [get_user_info()] fallback, so a true enumeration miss (0) is masked as 1. We capture the count before the fallback: homes_enumerated == 0 ⇒ enumeration miss (the macOS/AD network-account class); > 0 ⇒ genuinely empty / timing race.

Changes

  • scripts/coding_discovery_tools/ai_tools_discovery.py
    • Capture homes_enumerated before the user-enumeration fallback.
    • On if not tools:, emit report_to_sentry(RuntimeError("Discovery found no tools"), level="warning") with context: phase, homes_enumerated, users_scanned, used_fallback_user, is_root (when os.getuid exists), os, duration_ms (plus device_id/run_id/system_user from the existing sentry_ctx).
  • tests/test_discovery_flow.py
    • New TestNoToolsSentryEvent: fires exactly once on a zero-tool scan with the right fields; does not fire when tools are found.

Diagnosing from Sentry

Filter phase:no_tools_found, then read per event:

  • homes_enumerated:0 → enumeration miss (network/AD account dropped)
  • homes_enumerated:>0 → genuinely empty disk at scan time (timing race / true-empty)
  • is_root:true → ran in MDM/SYSTEM context

Constraints honored

  • Fire-and-forget — wrapped in try/except: pass; telemetry can never raise into or slow the scan.
  • Curl-only — reuses the existing report_to_sentry (no new network code; no urllib).
  • No PII — counts and booleans only; no username lists or home-dir contents.
  • Fires only when tools == []; relies on report_to_sentry's existing per-run cap + dedup (unique phase) + circuit breaker.

Known limitation (deliberate, minimal scope)

Discriminator fields land in Sentry extra, not searchable tags — only phase:no_tools_found is filterable; values are readable per-event but not facetable/alertable without a follow-up _SENTRY_TAG_KEYS addition. No backend/DeviceScan/migration changes (intentionally out of scope).

Test plan

  • python3 -m pytest tests/test_discovery_flow.py -v — new tests pass
  • Full tests/ suite green (69 passed), no regressions
  • Both changed files byte-compile clean

🤖 Generated with Claude Code


Note

Low Risk
Observability-only changes with fire-and-forget Sentry calls; scan completion path and backend payloads are unchanged except for optional warning telemetry.

Overview
Zero-tool discovery scans were silent in observability—hard to distinguish enumeration misses, timing races, and genuinely empty devices.

The scan now records homes_enumerated before the current-user fallback (so 0 is not masked as 1 in metrics). When no tools are found, it sends one fire-and-forget Sentry warning (phase=no_tools_found) with discriminators: homes_enumerated, users_scanned, used_fallback_user, is_root, duration_ms, plus existing run context. Telemetry is wrapped so it cannot fail the scan.

report_to_sentry gains a priority flag so this terminal diagnostic can bypass the per-run event cap and circuit breaker (still deduped). Tests cover zero-tool vs found-tool behavior and priority send semantics.

Reviewed by Cursor Bugbot for commit ab61e8e. Bugbot is set up for automated code reviews on this repo. Configure here.

Greptile Summary

This PR adds Sentry visibility for discovery scans that find no tools. The main changes are:

  • Adds pre-fallback home enumeration counts to the discovery flow.
  • Sends one priority no_tools_found warning with scan context when no tools are detected.
  • Lets priority Sentry events bypass the per-run cap and circuit breaker while keeping deduplication.
  • Adds tests for the no-tools event and priority Sentry behavior.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
scripts/coding_discovery_tools/ai_tools_discovery.py Adds no-tools scan context and emits the terminal Sentry warning when discovery returns no tools.
scripts/coding_discovery_tools/utils.py Adds priority handling to report_to_sentry for terminal diagnostics while preserving duplicate suppression.
tests/test_discovery_flow.py Adds tests for the zero-tool Sentry event and the tools-found path.
tests/test_send_and_persist.py Adds tests for priority Sentry events bypassing send limits and preserving duplicate suppression.

Reviews (3): Last reviewed commit: "Let priority Sentry events bypass the ci..." | Re-trigger Greptile

A scan that finds zero tools was previously silent — indistinguishable from
an errored scan, a missed detection, or a genuinely empty device. Emit one
fire-and-forget Sentry warning (phase=no_tools_found) carrying the
discriminators needed to tell them apart:
- homes_enumerated: pre-fallback enumerated-user count (0 = enumeration miss;
  the existing post-fallback user_count masks 0 as 1)
- users_scanned, used_fallback_user, is_root, os, duration_ms

Counts/booleans only (no PII); reuses the existing curl-based report_to_sentry;
wrapped so telemetry can never raise into or slow the scan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@anonpran anonpran requested a review from a team June 26, 2026 03:25
Comment thread scripts/coding_discovery_tools/ai_tools_discovery.py

@vigneshsubbiah16 vigneshsubbiah16 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Automated Security Review (consensus)

0 findings — 0 high-confidence, 0 to triage. Reviewers: Cursor, Claude, Semgrep, Gitleaks.

✅ Security consensus: no issues found. (reviewers: Cursor, Claude, Semgrep, Gitleaks)


🤖 consensus review · reviewers: Cursor,Claude,Semgrep,Gitleaks · head 29e59bce · 2026-06-26T03:30Z

@anonpran anonpran changed the title Add no_tools_found Sentry event to diagnose empty discovery scans [WEB-4956] Add no_tools_found Sentry event to diagnose empty discovery scans Jun 26, 2026
A noisy empty scan (many earlier detector errors) could exhaust the shared
30-event per-run cap before the terminal no_tools_found summary runs, dropping
exactly the diagnostic you most need on a likely missed-detection. Add a
`priority` flag to report_to_sentry that bypasses the COUNT cap only — it still
honors the circuit breaker (a dead transport can't be helped) and dedup (no
spam) — and pass priority=True for the no_tools_found event.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread scripts/coding_discovery_tools/utils.py Outdated
The priority flag bypassed the count cap but the circuit breaker — tripped by 3
consecutive, possibly transient, send failures — could still skip the terminal
no_tools_found event without even an attempt. Allow a priority event ONE bounded
attempt past the breaker (at most one ~4s curl at end of run) so a transient
mid-scan outage doesn't drop the terminal diagnostic. Dedup is still honored, so
it can never spam.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vigneshsubbiah16

Copy link
Copy Markdown
Contributor

🛡️ Automated Security Review (consensus)

0 findings — 0 high-confidence, 0 to triage. Reviewers: Lead, Claude, Semgrep, Gitleaks.

✅ Security consensus: no issues found. (reviewers: Lead, Claude, Semgrep, Gitleaks)


🤖 consensus review · reviewers: Cursor,Claude,Semgrep,Gitleaks · head ab61e8eb · 2026-06-26T04:00Z

@anonpran anonpran merged commit c6b102a into main Jun 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants