Commit 006688c
feat: add OpenAI embedding-based alignment strategy for DemoLibrary (#179)
Add OpenAIEmbeddingAlignment class that implements the AlignmentStrategy
protocol using OpenAI's VLM (gpt-4o-mini) for screenshot description
and text-embedding-3-small for semantic embedding. This provides a
cloud-based alternative to local CLIP models for demo step alignment.
Key changes:
- OpenAIEmbeddingAlignment: two-step pipeline (VLM describe + embed)
with cosine similarity matching against pre-computed demo embeddings
- create_alignment_strategy() factory: accepts string names ("phash",
"clip", "hybrid", "openai") for easy construction
- DemoLibrary constructor now accepts string alignment_strategy names
with automatic fallback to pHash on failure
- enrich_demo() pre-computes OpenAI embeddings when strategy is "openai"
- Embeddings persist to demo.json as plain lists for serialization
- Pure-Python cosine similarity fallback when numpy is not installed
Cost: ~$0.001 per screenshot (~$0.06 for 61 demos total).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent c5550a0 commit 006688c
1 file changed
Lines changed: 361 additions & 13 deletions
0 commit comments