Skip to content

W2.2 — Telemetry for dynamic tool loading #24

Description

@edelauna

Context

We need data to know whether dynamic loading is actually helping: how often the threshold is hit, how big the attached set ends up after the rolling window, how often the model has to call mcp_load, and which ranker tier won. Without telemetry we can't tune the defaults.

Operational; doesn't close a user-facing issue.

Developer Notes

  • Events to emit (using whatever telemetry pipeline Roo already uses — check src/services/telemetry if present):
    • dynamic_tools.task_start: { scopedCount, attachedCount, ranker: "embeddings" | "bm25" }.
    • dynamic_tools.mcp_load_invoked: { taskTurn, queryLength, returnedCount, alreadyAttachedCount }.
    • dynamic_tools.rolling_window_step: { turn, addedCount, totalAttached } (only when something changes).
    • dynamic_tools.ranker_fallback: { from: "embeddings", to: "bm25", reason } (when Tier 1 fails).
  • Respect existing telemetry opt-out.
  • Files: src/services/tools/ToolRouter.ts (most events fire here), src/core/tools/mcpLoadTool.ts (mcp_load_invoked).
  • Validation: telemetry events fire in integration tests; opt-out correctly suppresses them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions