Skip to content

feat(analytics): durable adoption analytics, diagnosing-analytics skill, hook replay test, Claude schema validation - #260

Merged
ScriptedAlchemy merged 1 commit into
masterfrom
feat/analytics-adoption-diagnostics
Jul 3, 2026
Merged

feat(analytics): durable adoption analytics, diagnosing-analytics skill, hook replay test, Claude schema validation#260
ScriptedAlchemy merged 1 commit into
masterfrom
feat/analytics-adoption-diagnostics

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Why

Investigating "is TraceDecay actually being used?" surfaced four structural problems:

  1. analytics_events looked dead but wasn't. The only writer targets the user-level global.db; the per-project sessions.db carries the same schema with no writer, so anyone inspecting the project store concludes telemetry is broken.
  2. Hook telemetry splits across two JSONL files with no attribution. Hooks write to the project store when they can resolve a project root and silently fall back to ~/.tracedecay/hook_analytics.jsonl when they can't (Claude preToolUse and all Kiro hooks hardcoded None). Rows carried no project field, so the split streams could never be re-joined.
  3. Dashboard diagnostics read only the project-store file, so the active user-level stream was invisible.
  4. No CLI answered adoption questionstracedecay analytics … didn't exist, and nothing taught agents which surface to use.

What

  • Writer attribution — hook analytics rows now carry project_root/event_cwd; Claude preToolUse resolves the root from the event cwd then the process cwd, and the three Kiro hooks resolve from the event cwd.
  • Merged reader — dashboard diagnostics reads both hook files (user-level rows filtered by attribution) and reports a hook_sources list with per-file row counts.
  • analytics_bridge — idempotent hook-JSONL → analytics_events import using parse_offsets byte cursors; partial trailing lines stay unconsumed for the next run.
  • New CLItracedecay analytics diagnostics [--all|--no-sync] (imports, then prints the same summary the dashboard serves) and tracedecay analytics sync.
  • New skilldiagnosing-analytics teaches agents the analytics/doctor/lcm_status/sessions surface and its guardrails (never raw sqlite); added to CURSOR_PLUGIN_SKILLS.
  • Hook replay E2E test — drives all four providers' hook subcommands through the real binary with representative events, asserting attributed JSONL rows land in the project store, nothing spills to the fallback file, and analytics sync bridges every row into analytics_events.
  • Claude schema validation — vendored draft-07 schemas for Claude Code hooks.json, plugin.json, and marketplace.json with positive + negative tests, closing the gap where only Cursor/Codex configs were schema-checked. docs/PLUGIN-VALIDATION.md table updated.

Verification

  • New tests: hook replay E2E (12 provider events, wiring end to end), bridge import integration test (incremental/idempotent/partial-line), 3 bridge unit tests, 4 Claude schema tests.
  • Suites: hooks_lsp_suite 117/117, agent_suite skill/schema subset 43/43, dashboard analytics 6/6, session_suite import test — all green on this branch.
  • Live run on real data imported 38,686 backlogged hook rows into analytics_events and the diagnostics command now answers adoption questions in one shot.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e9b75f3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@ScriptedAlchemy
ScriptedAlchemy force-pushed the feat/analytics-adoption-diagnostics branch 2 times, most recently from 9b98fe0 to e86a876 Compare July 3, 2026 23:16
Adoption telemetry was split and partly unreadable: the only writer of
analytics_events targets the user-level global DB (the per-project copy is
schema-only), hook_analytics.jsonl splits between the project store and the
user-level fallback with no attribution, dashboard diagnostics read only the
project-store file, and there was no CLI to answer adoption questions.

- hooks: stamp project_root/event_cwd on hook analytics rows; resolve the
  project root in Claude preToolUse (event cwd, then process cwd) and the
  Kiro hooks instead of hardcoding None
- dashboard: diagnostics merges the project-store and user-level hook files
  (user-level rows filtered by attribution) and reports hook_sources
- analytics_bridge: idempotent hook-JSONL -> analytics_events import using
  parse_offsets byte cursors; partial trailing lines stay unconsumed
- cli: new `tracedecay analytics diagnostics [--all|--no-sync]` and
  `tracedecay analytics sync`
- plugin: new diagnosing-analytics skill teaching the analytics/doctor/
  lcm_status/sessions surface; added to CURSOR_PLUGIN_SKILLS
- tests: end-to-end hook replay driving all four providers' hook subcommands
  through the binary and asserting JSONL attribution plus durable bridging;
  vendored Claude Code schemas (hooks/plugin/marketplace) with positive and
  negative validation, closing the gap where only Cursor/Codex configs were
  schema-checked

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy force-pushed the feat/analytics-adoption-diagnostics branch from e86a876 to e9b75f3 Compare July 3, 2026 23:24
@ScriptedAlchemy
ScriptedAlchemy merged commit ad60993 into master Jul 3, 2026
16 checks passed
ScriptedAlchemy added a commit that referenced this pull request Jul 3, 2026
Working-tree twin of PR #260 (feat/analytics-adoption-diagnostics) so this
branch carries the same analytics work: hook-row project attribution,
merged hook JSONL diagnostics with hook_sources, the analytics_bridge
hook-JSONL -> analytics_events import, the `tracedecay analytics`
CLI (diagnostics/sync) with rich help, the diagnosing-analytics plugin
skill, the provider hook replay E2E test, and vendored Claude Code
schema validation (hooks/plugin/marketplace).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy deleted the feat/analytics-adoption-diagnostics branch July 3, 2026 23:31
ScriptedAlchemy added a commit that referenced this pull request Jul 4, 2026
Return raw JSON envelopes from the fact_store and fact_feedback memory
write handlers instead of markdown-rendered output. After the markdown
default output change, these write-path handlers rendered their status
JSON as markdown, which interfered with the memory digest export
lifecycle. A new `tool_json` helper emits the truncated JSON envelope
(with project handle) directly; `handle_memory_status` continues to use
the rendered path.

Recovered from a stranded worktree whose stale rebase base had reverted
unrelated master work (PR #258/#260 agent-surface changes); only the
genuine memory.rs fix is carried forward here, rebuilt cleanly on
current origin/master.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant