Add learning tracking documentation (#141)

* docs: Add comprehensive BYOA learning tracking documentation

Resolves #141

Added complete documentation and examples for integrating learning
tracking into Bring-Your-Own-Agent (BYOA) adapters.

Changes:
- NEW: docs/sdk/learning_tracking.md - Comprehensive guide covering:
  - How to use resolve_playbook() (auto-registers entries)
  - When/how to call detect_and_record() for cue detection
  - When/how to call record_action_adoption() for tool tracking
  - Learning key management for evaluation scenarios
  - Complete working examples
  - Troubleshooting section for common issues

- UPDATED: examples/byoa_structured_adapter.py
  - Added learning_enabled_adapter() function
  - Demonstrates all 4 learning tracking steps
  - Includes planning and execution phase examples
  - Graceful error handling and debug output

- UPDATED: atlas/learning/usage.py
  - Enhanced module docstring with usage overview
  - Added comprehensive docstrings for all public methods
  - Included usage examples in docstrings
  - Added troubleshooting guidance for common issues

- UPDATED: docs/sdk/structured_adapter_payloads.md
  - Added Learning Tracking Integration section
  - Cross-reference to learning_tracking.md
  - Quick example showing integration

This addresses all 4 problems reported in issue #141:
1. Playbook entries not automatically registered (documented existing behavior)
2. Action adoption tracking (complete guide + examples)
3. Cue hit detection (when and how to call)
4. Learning key management (best practices included)

No code changes required - the automatic registration already works in
atlas/learning/playbook.py:59-65. Issue was purely documentation gap.

* docs: Add troubleshooting for disabled learning tracking

Added troubleshooting section for when learning tracking is disabled
via learning_usage_config.enabled flag. This addresses a silent failure
mode where all tracking calls succeed but do nothing.

* chore: Add CLAUDE.md to .gitignore

* refactor: Clarify learning tracking steps and remove duplicate imports

- Clarified 'five steps' wording in learning_tracking.md (get_tracker + 4 tracking calls)
- Removed redundant json imports inside function bodies in byoa_structured_adapter.py

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: jbarnes850

.gitignore

@@ -54,3 +54,4 @@ atlas-postgres.yaml
 results/
 .atlas/
 .DS_Store
+CLAUDE.md

atlas/learning/usage.py

@@ -1,4 +1,37 @@
-"""Runtime usage instrumentation for playbook entries."""
+"""Runtime usage instrumentation for playbook entries.
+
+This module provides learning tracking for Atlas agents, enabling measurement
+of which playbook entries are referenced during execution and which actions
+are adopted based on learning recommendations.
+
+Core Concepts:
+    - Cue Detection: Identifying when user input matches learning cue patterns
+    - Action Adoption: Tracking when tools are executed based on playbook entries
+    - Impact Metrics: Computing effectiveness of learning entries across sessions
+
+Typical Usage (BYOA Adapters):
+    ```python
+    from atlas.learning.playbook import resolve_playbook
+    from atlas.learning.usage import get_tracker
+
+    # 1. Retrieve playbook (auto-registers entries)
+    playbook, digest, metadata = resolve_playbook("student", apply=True)
+
+    # 2. Get tracker
+    tracker = get_tracker()
+
+    # 3. Detect cue hits
+    tracker.detect_and_record("student", user_input, step_id=1)
+
+    # 4. Record action adoptions
+    tracker.record_action_adoption("student", "tool_name", success=True, step_id=1)
+
+    # 5. Record final outcome
+    tracker.record_session_outcome(reward_score=0.85)
+    ```
+
+See docs/sdk/learning_tracking.md for complete guide.
+"""
 
 from __future__ import annotations
 
@@ -17,7 +50,35 @@ class _TrackerConfig:
 
 
 class LearningUsageTracker:
-    """Lightweight helper that records cue hits and action adoption statistics."""
+    """Lightweight helper that records cue hits and action adoption statistics.
+
+    This tracker instruments learning usage during agent execution, enabling
+    measurement of:
+    - Which playbook entries have their cues detected in user input
+    - Which tools/actions are adopted based on playbook recommendations
+    - Whether adopted actions succeed or fail
+    - Session-level metrics (reward, tokens, retries, failures)
+
+    The tracker is automatically created and attached to ExecutionContext.
+    Access it via `get_tracker()` or `get_tracker(context)`.
+
+    Usage Pattern:
+        1. Register entries: Called automatically by resolve_playbook()
+        2. Detect cues: Call detect_and_record() on user input
+        3. Track adoptions: Call record_action_adoption() after tool execution
+        4. Record outcome: Call record_session_outcome() at session end
+
+    Attributes:
+        enabled (bool): Whether tracking is active (from config)
+
+    Methods:
+        register_entries(): Register playbook entries for tracking
+        detect_and_record(): Detect and record cue hits from text
+        record_cue_hit(): Record a specific cue hit manually
+        record_action_adoption(): Record when a tool/action is adopted
+        record_session_outcome(): Record final session metrics
+        snapshot(): Export current tracking state
+    """
 
     def __init__(self, context: ExecutionContext) -> None:
         self._context = context
@@ -53,6 +114,27 @@ def enabled(self) -> bool:
         return self._config.enabled
 
     def register_entries(self, role: str, entries: Iterable[Dict[str, Any]]) -> None:
+        """Register playbook entries for tracking.
+
+        This method is called automatically by resolve_playbook() so BYOA adapters
+        typically don't need to call it manually.
+
+        For each entry, this method:
+        - Creates tracking storage (cue_hits, action_adoptions, step_ids)
+        - Registers cue detection patterns
+        - Stores runtime_handle for action adoption matching
+
+        Args:
+            role: "student" or "teacher"
+            entries: List of playbook entry dicts, each containing:
+                - id: Unique entry identifier
+                - cue: Dict with type and pattern for detection
+                - action: Dict with runtime_handle for adoption tracking
+                - scope: Category and other metadata
+
+        Note:
+            Called automatically by resolve_playbook() in atlas/learning/playbook.py
+        """
         if not self.enabled:
             return
         role_store = self._usage_store["roles"].setdefault(role, {})
@@ -96,6 +178,38 @@ def detect_and_record(
         step_id: int | None = None,
         context_hint: str | None = None,
     ) -> List[Dict[str, Any]]:
+        """Detect and record cue hits from text.
+
+        Scans the provided text for patterns matching registered playbook entry cues.
+        For each match, increments cue hit counters and optionally saves examples.
+
+        Cue Types:
+            - keyword: Simple substring match (case-insensitive)
+            - regex: Regular expression pattern match
+            - predicate: Condition-based match (treated as keyword)
+
+        When to Call:
+            - On user input/task description (planning phase)
+            - On step descriptions (execution phase)
+            - On any text that might contain learning cues
+
+        Args:
+            role: "student" or "teacher"
+            text: Text to scan for cue patterns
+            step_id: Optional step identifier for tracking which steps have cue hits
+            context_hint: Optional text snippet to save as example (truncated to 200 chars)
+
+        Returns:
+            List of matched cue dicts, each containing:
+                - entry_id: Which playbook entry was matched
+                - pattern: The pattern that matched
+                - type: Cue type (keyword, regex, predicate)
+
+        Example:
+            >>> tracker = get_tracker()
+            >>> matches = tracker.detect_and_record("student", "create a new contact", step_id=1)
+            >>> print(f"Detected {len(matches)} cue hits")
+        """
         if not self.enabled or not text:
             return []
         detectors = self._usage_store.get("detectors", {}).get(role) or []
@@ -147,6 +261,46 @@ def record_action_adoption(
         step_id: int | None = None,
         metadata: Optional[Dict[str, Any]] = None,
     ) -> None:
+        """Record when a tool/action is adopted based on playbook recommendation.
+
+        This method tracks whether learning entries actually change agent behavior.
+        It matches the runtime_handle parameter against playbook entries and
+        increments adoption counters when matches are found.
+
+        CRITICAL: The runtime_handle must EXACTLY match the action.runtime_handle
+        stored in playbook entries. This is how Atlas links tool execution to
+        learning recommendations.
+
+        When to Call:
+            - After executing any tool that might be in playbook entries
+            - After taking any action recommended by learning
+            - Typically called once per tool execution
+
+        Args:
+            role: "student" or "teacher"
+            runtime_handle: Tool name or action identifier (must match playbook entries)
+            success: Whether the action succeeded (True) or failed (False)
+            step_id: Optional step identifier for tracking adoption per step
+            metadata: Optional dict with additional context (saved as example if
+                     capture_examples is enabled)
+
+        Example:
+            >>> tracker = get_tracker()
+            >>> # After executing a tool
+            >>> tracker.record_action_adoption(
+            ...     "student",
+            ...     runtime_handle="create_contact",  # Must match entry's runtime_handle
+            ...     success=True,
+            ...     step_id=1,
+            ...     metadata={"tool_name": "create_contact", "args": {...}}
+            ... )
+
+        Troubleshooting:
+            If adoptions remain 0 despite tool execution:
+            1. Verify runtime_handle exactly matches playbook entry's action.runtime_handle
+            2. Ensure playbook entries are registered (call resolve_playbook first)
+            3. Check that learning synthesis has generated entries
+        """
         if not self.enabled:
             return
         if not runtime_handle:
@@ -187,6 +341,45 @@ def record_session_outcome(
         failure_flag: bool | None = None,
         failure_events: Sequence[Dict[str, Any]] | None = None,
     ) -> None:
+        """Record final session metrics and outcome.
+
+        Call this method once at the end of session execution to record overall
+        metrics used for computing learning entry impact.
+
+        When to Call:
+            - At the end of session execution, before returning results
+            - After all steps have been executed
+            - Typically called once per session
+
+        Args:
+            reward_score: Session reward score (0.0-1.0)
+            token_usage: Dict with token counts:
+                - total_tokens: Total tokens used
+                - prompt_tokens: Input tokens
+                - completion_tokens: Output tokens
+                - calls: Number of LLM calls
+            incident_id: Optional incident/task identifier for grouping
+            task_identifier: Optional task type/category identifier
+            incident_tags: Optional list of tags for categorization
+            retry_count: Number of retries during session
+            failure_flag: Whether session failed overall
+            failure_events: List of failure event dicts
+
+        Example:
+            >>> tracker = get_tracker()
+            >>> tracker.record_session_outcome(
+            ...     reward_score=0.85,
+            ...     token_usage={
+            ...         "total_tokens": 2000,
+            ...         "prompt_tokens": 1500,
+            ...         "completion_tokens": 500,
+            ...         "calls": 3
+            ...     },
+            ...     task_identifier="security-review",
+            ...     retry_count=1,
+            ...     failure_flag=False
+            ... )
+        """
         if not self.enabled:
             return
         session_block = self._usage_store["session"]
@@ -239,13 +432,75 @@ def record_session_outcome(
                 session_block["failure_events"] = cleaned
 
     def snapshot(self) -> Dict[str, Any]:
-        """Return the current usage store (already JSON-serialisable)."""
+        """Return the current usage store (already JSON-serializable).
+
+        This method exports all tracked learning usage data for the current session,
+        including:
+        - Per-entry statistics (cue hits, adoptions, success/failure counts)
+        - Session-level aggregates (total cues, total adoptions, reward score)
+        - Step-level tracking (which steps had cue hits or adoptions)
+
+        The returned dict is automatically persisted to the database and used for
+        generating learning impact reports.
+
+        Returns:
+            Dict with structure:
+                {
+                    "roles": {
+                        "student": {
+                            "entry_id_1": {
+                                "cue_hits": int,
+                                "action_adoptions": int,
+                                "successful_adoptions": int,
+                                "failed_adoptions": int,
+                                "step_ids": List[int],
+                                ...
+                            },
+                            ...
+                        }
+                    },
+                    "session": {
+                        "cue_hits": int,
+                        "action_adoptions": int,
+                        "reward_score": float,
+                        "token_usage": {...},
+                        ...
+                    }
+                }
+        """
 
         return dict(self._usage_store)
 
 
 def get_tracker(context: ExecutionContext | None = None) -> LearningUsageTracker:
-    """Helper to fetch the tracker for the active execution context."""
+    """Get the learning usage tracker for the current execution context.
+
+    This helper creates or retrieves the tracker instance attached to the
+    execution context. Call this in BYOA adapters to access learning tracking.
+
+    Args:
+        context: Optional ExecutionContext. If None, uses ExecutionContext.get()
+
+    Returns:
+        LearningUsageTracker instance for the current session
+
+    Raises:
+        Exception: If ExecutionContext is not available (e.g., standalone testing)
+
+    Example:
+        >>> from atlas.learning.usage import get_tracker
+        >>> from atlas.runtime.orchestration.execution_context import ExecutionContext
+        >>>
+        >>> context = ExecutionContext.get()
+        >>> tracker = get_tracker(context)
+        >>> # Or simply:
+        >>> tracker = get_tracker()
+
+    Note:
+        Each execution context has its own tracker instance. Tracking data is
+        stored in context.metadata["learning_usage"] and persisted to database
+        at session end.
+    """
 
     context = context or ExecutionContext.get()
     return LearningUsageTracker(context)