Viewer consolidation, workflow segmentation, and VM monitoring (#9)

* feat: add openadapt-viewer dependency and adapter module (Phase 1)

Phase 1 of viewer consolidation plan: Foundation

Changes:
- Add openadapt-viewer as local file dependency in pyproject.toml
- Create openadapt_ml/training/viewer_components.py adapter module
  * screenshot_with_predictions() - Screenshot with human/AI overlays
  * training_metrics() - Training stats metrics grid
  * playback_controls() - Playback UI controls
  * correctness_badge() - Pass/fail badge component
  * generate_comparison_summary() - Model comparison summary
- Add tests/test_viewer_screenshots.py with component validation tests
- Add openadapt_ml/training/viewer_migration_example.py validation example

Design:
- Zero breaking changes to existing viewer.py code
- Adapter pattern wraps openadapt-viewer with ML-specific context
- Functions accept openadapt-ml data structures
- Can be incrementally adopted in future phases

Next steps (Phase 2):
- Gradually migrate viewer.py to use these adapters
- Replace inline HTML generation with component calls

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat: add workflow segmentation system with capture adapter

Restored and enhanced the workflow segmentation system from commit dd9a393
with new integration for openadapt-capture format.

## What's Added

### Core Segmentation Pipeline (4 stages):

1. **Stage 1 - Frame Description (VLM)**:
   - Converts screenshots + actions into semantic descriptions
   - Supports Gemini, Claude, GPT-4o backends
   - Automatic caching for efficiency
   - File: openadapt_ml/segmentation/frame_describer.py

2. **Stage 2 - Episode Extraction (LLM)**:
   - Identifies coherent workflow boundaries
   - Few-shot prompting for better quality
   - Confidence-based filtering
   - File: openadapt_ml/segmentation/segment_extractor.py

3. **Stage 3 - Deduplication (Embeddings)**:
   - Finds similar workflows across recordings
   - Agglomerative clustering with cosine similarity
   - Supports OpenAI or local HuggingFace embeddings
   - File: openadapt_ml/segmentation/deduplicator.py

4. **Stage 4 - Annotation (VLM Quality Control)**:
   - Auto-annotates episodes for training data quality
   - Detects failures, boundary issues, incompleteness
   - Human-in-the-loop review workflow
   - File: openadapt_ml/segmentation/annotator.py

### Integration Features:

- **CaptureAdapter**: Loads recordings from openadapt-capture SQLite format
  - File: openadapt_ml/segmentation/adapters/capture_adapter.py
  - Automatically used when capture.db is detected
  - Converts events to segmentation format

- **Unified Pipeline**: Run all stages with single API
  - File: openadapt_ml/segmentation/pipeline.py
  - Automatic intermediate result caching
  - Resume support for interrupted runs

- **CLI Interface**: Full command-line interface for all stages
  - File: openadapt_ml/segmentation/cli.py
  - Commands: describe, extract, deduplicate, annotate, review, export-gold

- **Comprehensive Documentation**:
  - File: openadapt_ml/segmentation/README.md
  - 20+ code examples
  - Complete API reference
  - Integration guide
  - Cost estimates and performance benchmarks

## Use Cases

1. **Training Data Curation**: Extract and filter high-quality demonstration episodes
2. **Demo Retrieval**: Build searchable libraries for demo-conditioned prompting
3. **Workflow Documentation**: Auto-generate step-by-step guides from recordings

## Data Schemas

All schemas use Pydantic for type safety (openadapt_ml/segmentation/schemas.py):
- ActionTranscript: Frame-by-frame semantic descriptions
- Episode: Coherent workflow segment with boundaries
- CanonicalEpisode: Deduplicated workflow definition
- EpisodeAnnotation: Quality assessment for training data

## Example Usage

```python
from openadapt_ml.segmentation import SegmentationPipeline, PipelineConfig

config = PipelineConfig(
    vlm_model="gemini-2.0-flash",
    llm_model="gpt-4o",
    similarity_threshold=0.85
)

pipeline = SegmentationPipeline(config)
result = pipeline.run(
    recordings=["/path/to/recording1", "/path/to/recording2"],
    output_dir="workflow_library"
)

print(f"Found {result.unique_episodes} unique workflows")
```

## Next Steps

See openadapt_ml/segmentation/README.md for:
- P0: Integration tests with real openadapt-capture recordings
- P0: Visualization generator for segment boundaries
- P1: Improved prompt engineering and cost optimization
- P2: Active learning and multi-modal features

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Enhance vm monitor command with comprehensive VM usage visibility

Features added:
- Azure ML job tracking: Shows recent jobs from last 7 days with status
- Cost tracking: Real-time uptime, hourly rate, and cost estimation
- VM activity detection: Identifies what VM is currently doing
- Evaluation history: Past benchmark runs and success rates (--details flag)
- Enhanced UI: Structured dashboard with clear sections and icons

New utility functions in vm_monitor.py:
- fetch_azure_ml_jobs(): Fetch recent Azure ML jobs with filtering
- calculate_vm_costs(): Calculate VM costs with hourly/daily/weekly rates
- get_vm_uptime_hours(): Get VM uptime from Azure activity logs
- detect_vm_activity(): Detect current VM activity (idle, running, setup)
- get_evaluation_history(): Load past evaluation runs from results dir

CLI enhancements:
- Added --details flag for extended information
- Improved output formatting with sections and separators
- Better error handling and status icons
- Preserved existing SSH tunnel and dashboard functionality

Documentation:
- Updated CLAUDE.md with new features and usage examples
- Added detailed docstrings to all new functions

This consolidates VM monitoring into a single enhanced command rather than
creating duplicate dashboards, following the viewer consolidation strategy.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Refactor segmentation pipeline to use screen.frame events

Update CaptureAdapter to work with actual openadapt-capture database
format. Key changes:

- Use screen.frame events instead of generic event types
- Pair action events (mouse.down + mouse.up → single click)
- Map frame events to screenshots via timestamp matching
- Update event type filtering to match openadapt-capture schema
- Improve frame-to-action association logic

This enables the segmentation pipeline to process real capture recordings
from openadapt-capture instead of requiring simulated data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add VM monitoring dashboard with comprehensive usage visibility

Enhance vm monitor command to provide complete VM usage tracking:
- Real-time VM status (size, IP, power state)
- Activity detection (idle, benchmark running, setup)
- Cost tracking (uptime hours, hourly rate, total cost)
- Azure ML jobs list (last 7 days with status)
- Evaluation history (with --details flag)
- Mock mode for testing without VM (--mock flag)

Add new API endpoints to local.py dashboard server:
- /api/benchmark/status - current job status with ETA
- /api/benchmark/costs - cost breakdown (Azure VM, API, GPU)
- /api/benchmark/metrics - performance metrics by domain
- /api/benchmark/workers - worker status and utilization
- /api/benchmark/runs - list all benchmark runs
- /api/benchmark/tasks/{run}/{task} - task execution details

Update README with VM monitor section including screenshots and
usage examples.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add segmentation testing documentation and test files

Add comprehensive test plan and results for workflow segmentation pipeline:
- Test plan with 8 stages from environment setup to documentation
- Test results documenting real capture processing outcomes
- Test files for CaptureAdapter and segmentation pipeline

Add VM monitor screenshot generation scripts and documentation:
- Scripts for automated dashboard screenshot generation
- Implementation plan for VM monitor screenshot feature
- Analysis of screenshot capture approaches

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Document archived OpenAdapter repository

- Archive OpenAdapter (incomplete pre-refactor cloud deployment POC)
- Document key takeaways and lessons learned
- Reference modern cloud infrastructure in openadapt-ml
- Add guidelines for when to archive repositories

OpenAdapter was an incomplete proof-of-concept from October 2024
with only 165 lines of code and no ecosystem usage. Cloud deployment
is now production-ready in openadapt_ml/cloud/ and benchmarks/azure.py.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add search functionality to training viewer

- Add search bar to viewer controls with Ctrl+F / Cmd+F keyboard shortcut
- Implement advanced token-based search across step indices, action types, and text
- Search filters step list in real-time with result count display
- Clear button and Escape key support for resetting search
- Consistent UI styling with existing viewer components
- Integrates with existing step list filtering

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve ruff linting and formatting issues

* fix: resolve test failures from missing dependencies

- Remove non-existent openadapt_ml.shared_ui import from viewer.py
- Skip anthropic test when anthropic package not installed (optional dependency)
- Skip viewer_components test when openadapt-viewer not installed (optional dependency)

All tests now pass (334 passed, 6 skipped).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: abrichr

CLAUDE.md

@@ -1,5 +1,16 @@
 # Claude Context for openadapt-ml
 
+## Project Status & Priorities
+
+**IMPORTANT**: Before starting work, always check the project-wide status document:
+- **Location**: `/Users/abrichr/oa/src/STATUS.md`
+- **Purpose**: Tracks P0 priorities, active background tasks, blockers, and strategic decisions
+- **Action**: Read this file at the start of every session to understand current priorities
+
+This ensures continuity between Claude Code sessions and context compactions.
+
+---
+
 This file helps maintain context across sessions.
 
 ---
@@ -18,9 +29,32 @@ This file helps maintain context across sessions.
 uv run python -m openadapt_ml.benchmarks.cli vm monitor
 ```
 
+**ENHANCED FEATURES (as of Jan 2026):**
+The `vm monitor` command now provides comprehensive VM usage visibility:
+- **VM Status**: Real-time VM state, size, and IP
+- **Activity Detection**: What the VM is currently doing (idle, benchmark running, setup)
+- **Cost Tracking**: Current uptime, hourly rate, and total cost for session
+- **Azure ML Jobs**: Recent jobs from last 7 days with status
+- **Evaluation History**: Past benchmark runs and success rates (with --details flag)
+- **Dashboard & Tunnels**: Auto-starts web dashboard and SSH/VNC tunnels
+
+**Usage:**
+```bash
+# Basic monitoring
+uv run python -m openadapt_ml.benchmarks.cli vm monitor
+
+# With detailed information (costs per day/week, evaluation history)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor --details
+
+# With auto-shutdown after 2 hours
+uv run python -m openadapt_ml.benchmarks.cli vm monitor --auto-shutdown-hours 2
+```
+
 **WHY THIS MATTERS:**
 - VNC is ONLY accessible via SSH tunnel at `localhost:8006` (NOT the public IP)
 - The dashboard auto-manages SSH tunnels
+- Shows real-time costs to prevent budget overruns
+- Tracks all Azure ML jobs for visibility into what's running
 - Without it, you cannot see what Windows is doing
 - The user WILL be frustrated if you keep forgetting this
 
@@ -120,6 +154,12 @@ openadapt-ml is a model-agnostic, domain-agnostic ML engine for GUI automation a
 - With demo: 100% correct first actions
 - See `docs/experiments/demo_conditioned_prompting_results.md`
 
+**✅ VALIDATED (Jan 17, 2026)**: Demo persistence fix is working
+- The P0 fix in `openadapt-evals` ensures demo is included at EVERY step, not just step 1
+- Mock test confirms: agent behavior changes from 6.8 avg steps (random) to 3.0 avg steps (focused)
+- See `openadapt-evals/CLAUDE.md` for full validation details
+- **Next step**: Run full WAA evaluation (154 tasks) to measure episode success improvement
+
 **Next step**: Build demo retrieval to automatically select relevant demos from a library.
 
 **Key insight**: OpenAdapt's value is **trajectory-conditioned disambiguation of UI affordances**, not "better reasoning".

README.md

@@ -781,6 +781,52 @@ uv run python -m openadapt_ml.cloud.local serve --port 8080 --open
 
 *View benchmark evaluation results with task-level filtering, success/failure status, and run comparison. Shows Claude achieving 30% on mock evaluation tasks (simulated environment for testing the pipeline - real WAA evaluation requires Windows VMs).*
 
+### 13.4 VM Monitoring Dashboard
+
+For managing Azure VMs used in benchmark evaluations, the `vm monitor` command provides a comprehensive dashboard:
+
+```bash
+# Start VM monitoring dashboard (auto-opens browser)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor
+
+# Show detailed information (evaluation history, daily/weekly costs)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor --details
+```
+
+**VM Monitor Dashboard (Full View):**
+
+![VM Monitor Dashboard](docs/screenshots/vm_monitor_dashboard_full.png)
+
+*The VM monitor dashboard shows: (1) VM status (name, IP, size, state), (2) Current activity (idle/benchmark running), (3) Cost tracking (uptime, hourly rate, total cost), (4) Recent Azure ML jobs from last 7 days, and (6) Dashboard & access URLs.*
+
+**VM Monitor Dashboard (With --details Flag):**
+
+![VM Monitor Dashboard Details](docs/screenshots/vm_monitor_details.png)
+
+*The --details flag adds: (5) Evaluation history with success rates and agent types, plus extended cost information (daily/weekly projections).*
+
+**Features:**
+- **Real-time VM status** - Shows VM size, power state, and IP address
+- **Activity detection** - Identifies if VM is idle, running benchmarks, or in setup
+- **Cost tracking** - Displays uptime hours, hourly rate, and total cost for current session
+- **Azure ML jobs** - Lists recent jobs from last 7 days with status indicators
+- **Evaluation history** - Shows past benchmark runs with success rates (with --details flag)
+- **Dashboard & tunnels** - Auto-starts web dashboard and SSH/VNC tunnels for accessing Windows VM
+
+**Mock mode for testing:**
+```bash
+# Generate screenshots or test dashboard without a VM running
+uv run python -m openadapt_ml.benchmarks.cli vm monitor --mock
+```
+
+**Auto-shutdown option:**
+```bash
+# Automatically deallocate VM after 2 hours to prevent runaway costs
+uv run python -m openadapt_ml.benchmarks.cli vm monitor --auto-shutdown-hours 2
+```
+
+For complete VM management commands and Azure setup instructions, see [`CLAUDE.md`](CLAUDE.md) and [`docs/azure_waa_setup.md`](docs/azure_waa_setup.md).
+
 ---
 
 ## 14. Limitations & Notes

docs/REPOSITORY_HISTORY.md

@@ -0,0 +1,70 @@
+# Repository History
+
+Documentation of deprecated and archived OpenAdapt ecosystem projects.
+
+## Deprecated/Archived Projects
+
+### OpenAdapter (Archived January 2026)
+
+**Repository**: https://github.com/OpenAdaptAI/OpenAdapter (ARCHIVED)
+**Status**: Incomplete proof-of-concept from before OpenAdapt refactor
+
+**Why Archived**:
+- Incomplete proof-of-concept code (only 165 lines, missing imports)
+- Created October 2024, minimal activity (14 commits, only 1 contributor)
+- Cloud infrastructure now handled by `openadapt_ml/cloud/` module
+- No active development, zero ecosystem usage
+- Last substantial commit was February 2025 (marked as WIP)
+
+**Original Purpose**:
+Attempted to provide cloud deployment infrastructure for screenshot parsing and action models, specifically targeting AWS ECS/ECR deployment for OmniParser using CDKTF (Terraform via Python).
+
+**Key Takeaways & Lessons Learned**:
+- Cloud training support is critical for productivity
+- Multiple backends (Lambda Labs, Azure) enable flexibility and cost optimization
+- Infrastructure as Code (Terraform/CDK) is appropriate for cloud setup
+- State management (tracking deployment IPs, configs) is important for multi-region deployments
+- Single-provider solutions are fragile - always support multiple cloud backends
+
+**What Replaced It**:
+- `openadapt_ml/cloud/lambda_labs.py` - Lambda Labs GPU rental and management
+- `openadapt_ml/cloud/azure_inference.py` - Azure ML integration for inference
+- `openadapt_ml/benchmarks/azure.py` - Azure ML for automated WAA evaluation
+- `scripts/setup_azure.py` - Full Azure setup automation with resource management
+- Documentation: `docs/cloud_gpu_training.md`, `docs/azure_waa_setup.md`
+
+**Modern Approach**:
+The current openadapt-ml cloud infrastructure is production-ready and supports:
+- Multiple cloud providers (Lambda Labs, Azure ML, local)
+- Multiple model types (not just OmniParser)
+- Automatic cleanup and quota management
+- Tested deployment patterns with comprehensive documentation
+- Cost estimation and monitoring tools
+
+**References**:
+- Original incomplete code: https://github.com/OpenAdaptAI/OpenAdapter/tree/feat/omniparser
+- Cloud architecture docs: `docs/cloud_gpu_training.md`
+- Azure setup guide: `docs/azure_waa_setup.md`
+
+---
+
+## Notes on Repository Management
+
+**When to Archive**:
+- No active development for 3+ months
+- Incomplete/experimental code that won't be finished
+- Functionality superseded by other ecosystem components
+- Zero usage in production or by other repos
+- Single contributor with no current interest
+
+**Before Archiving**:
+1. Review code for valuable patterns or ideas
+2. Document key takeaways in this file
+3. Update references in other repositories
+4. Remove from GitHub organization profile README
+5. Add archive notice to repository description
+
+**Alternative to Archiving**:
+- Move code to `legacy/` branch in main repository
+- Keep as example/reference in documentation
+- Convert to gist or snippet if very small