Goal

Add analytics-only classification that describes what Junior conversations are about. The result should support aggregate analysis and evaluation; it must not affect model profiles, routing, permissions, or runtime behavior.

Taxonomy

Define a configurable, versioned list of categories. Each category supplies a stable ID and a description for the judge:

conversationClassification:
  taxonomy: intent-v1
  categories:
    - id: code_change
      description: Implements, fixes, or reviews repository code
    - id: investigation
      description: Diagnoses behavior using source code or telemetry
    - id: information_lookup
      description: Retrieves and summarizes existing information
    - id: external_action
      description: Creates or changes a resource in another system

The taxonomy should be configuration rather than a hardcoded application enum, while classification output is schema-constrained to one of the configured IDs. Persist the taxonomy ID/version with every result so historical analytics remain interpretable after categories change.

Hypothetical approaches

Conversation-level, after completion

Give a fast judge the completed bounded transcript and configured category list, then choose the primary category for the conversation.

• Best semantic accuracy because it sees the outcome, not only the opening request.
• One classification per conversation/run and no user-facing latency.
• A single label may flatten conversations that change intent.

Turn-level

Classify each user turn, then derive conversation analytics from the ordered labels or an aggregate rule.

• Captures intent changes, follow-ups, and mixed-purpose conversations.
• Costs more and requires defining which turns count, how assistant/tool turns are treated, and how labels roll up.
• Useful if the analytical question is about work units rather than whole conversations.

Hybrid

Classify each meaningful user turn, then run or derive a final conversation-level classification after completion.

• Preserves both granular and aggregate data.
• Highest cost and schema complexity.
• Could begin as an experiment on sampled conversations rather than the default.

First-message proxy

Classify only the first actionable user message.

• Cheapest and available before completion.
• Measures initial request intent, not necessarily what the conversation became.
• Better treated as a separate initial_intent signal than the authoritative conversation category.

Proposed starting point

Start with asynchronous conversation-level classification after completion using a fast, tool-free structured-output judge. Store one primary category, optional confidence/reason for diagnostics, taxonomy version, and classification timestamp. Keep unknown/other as an explicit fallback and sample low-confidence results for taxonomy refinement.

Before expanding to turn-level classification, use real transcripts to answer whether mixed-intent conversations are common enough to justify the added cost. If turn-level data is added later, retain separate fields for turn_category, initial_intent, and conversation_category rather than treating them as interchangeable.

Open questions

• Should the judge emit exactly one primary category, multiple categories, or primary plus secondary?
• Is a Junior run the correct analytics boundary, or should multiple runs in one Slack thread roll up into one conversation?
• Which transcript entries should be visible to the judge: user/assistant text only, or normalized tool names/results too?
• Where should configured taxonomies live, and who owns category/version changes?
• Should classification run on every completed conversation or on a configurable sample?

Requested by immutable dcramer via Junior.

--

View Junior Session in Sentry