diff --git a/CHANGELOG.md b/CHANGELOG.md index 123962b3..bc7808f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ All notable changes to OpenReels will be documented in this file. +## [0.15.0] - 2026-04-09 + +### Added +- **Director-Critic quality gate**: the pipeline now evaluates and revises the DirectorScore before any expensive generation begins. A Critic agent scores the plan (1-10) and if below 7, provides revision instructions. The Creative Director revises the plan and re-submits. Up to 2 revision rounds, tracking the highest-scoring version. Revision costs ~$0.02-0.13 in LLM calls vs $0.50-2.00+ for regenerating the full pipeline. +- **Revision cost tracking**: cost estimates now include a "Revise" line item showing actual revision LLM cost. The `CostBreakdown` interface includes `revisionCost` and `revisionRounds` fields. +- **Worker revision history**: `meta.json` now includes a `revisionHistory` array with per-round critic scores, enabling future UI display of the revision journey. + +### Changed +- The Creative Director's system prompt loading is now a shared `loadDirectorSystemPrompt()` helper, used by both initial generation and revision. +- The step-6 critic no longer displays "Revision support coming in a future release" since revision is now active. +- Cost estimation uses the final revised DirectorScore for accurate scene counts when visual types change during revision. +- Dry-run mode now runs after the revision loop, showing the revised plan instead of the first draft. + ## [0.14.0] - 2026-04-09 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index ea3d3d28..e5450159 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,7 +46,7 @@ fixtures/ # sample DirectorScore JSONs ```bash pnpm install # install dependencies pnpm start "topic" # run full pipeline (CLI) -pnpm test # run vitest suite (338 tests) +pnpm test # run vitest suite (349 tests) ``` ### Web UI (Docker Compose) diff --git a/TODOS.md b/TODOS.md index 3a782423..4e1a0b49 100644 --- a/TODOS.md +++ b/TODOS.md @@ -96,6 +96,13 @@ **Depends on:** TTS alignment layer (Kokoro + Gemini TTS providers) Deferred from plan: Unified TTS Alignment Layer (CEO review: user prefers unified voice interface later) +## Critic / Quality Gate + +- [ ] **Deterministic structural validation for DirectorScore** — Extract pacing checks (total word count, scene count, per-scene word bounds, consecutive visual_type runs) into a TypeScript validation function. Run it after DirectorScore.parse() in the existing retry loop. Structural violations become retry errors (fed back as error context to the director), not LLM critic concerns. Saves 1-2 LLM calls when the issue is structural, and makes structural quality a guaranteed invariant rather than probabilistic. + **Priority:** P3 + **Depends on:** Director-Critic quality gate (v0.15.0) + Deferred from plan: Director-Critic Quality Gate (eng review: outside voice recommended splitting deterministic vs subjective evaluation) + ## Captions - [ ] **Contextual emphasis on power words** — Creative director marks 1-3 power words per scene with asterisks in script_line. Pipeline strips markers before TTS and records emphasisIndices. CaptionWrapper applies extra animation treatment (larger font-size delta, accent color flash) to emphasis words. Requires: (a) strip asterisks before ALL downstream consumers (critic, image prompter, TTS), not just TTS, (b) creative director prompt update with emphasis marking instructions, (c) emphasisIndices field on CompositionProps (optional, backward compat), (d) CaptionWrapper emphasis detection logic. Per-word asterisks format: `*New* *York*` not `*New York*`. diff --git a/VERSION b/VERSION index a803cc22..a5510516 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.14.0 +0.15.0 diff --git a/docs/content/docs/contributing/architecture.mdx b/docs/content/docs/contributing/architecture.mdx index f577a46b..181b4de7 100644 --- a/docs/content/docs/contributing/architecture.mdx +++ b/docs/content/docs/contributing/architecture.mdx @@ -39,7 +39,7 @@ AI agent functions that make LLM calls with structured output. Each agent has a | `creative-director.ts` | Creative Director | research data + archetype | DirectorScore | | `image-prompter.ts` | Image Prompter | raw visual prompt | optimized prompt for image gen | | `music-prompter.ts` | Music Prompter | music mood | prompt string for AI music gen | -| `critic.ts` | Critic | rendered video + score | score 0-10, strengths, weaknesses | +| `critic.ts` | Critic | DirectorScore + topic | score 0-10, strengths, weaknesses | ### `src/pipeline/` diff --git a/docs/content/docs/pipeline/critic.mdx b/docs/content/docs/pipeline/critic.mdx index 3fee367e..54a3527c 100644 --- a/docs/content/docs/pipeline/critic.mdx +++ b/docs/content/docs/pipeline/critic.mdx @@ -3,7 +3,7 @@ title: Critic Agent description: How the critic evaluates DirectorScore quality using a weighted rubric and flags revision when scores are low. --- -The Critic is the final stage of the pipeline. It evaluates the DirectorScore against a structured rubric and produces a quality score. If the score falls below the pass threshold, it identifies the weakest scene and provides specific revision instructions. +The Critic evaluates the DirectorScore against a structured rubric and produces a quality score. It runs twice in the pipeline: first as a **quality gate** inside the Director stage (before any expensive generation), and again as a **post-mortem** in the final stage. If the score falls below the pass threshold, it identifies the weakest scene and provides specific revision instructions. ## What it evaluates @@ -73,9 +73,16 @@ interface CritiqueResult { } ``` -## When re-runs happen +## Quality gate (Director-Critic revision loop) -If `revision_needed` is true and the score is below 7, the Critic logs the weaknesses and notes that revision support is coming in a future release. Currently, the pipeline uses the existing DirectorScore as-is -- the Critic serves as a quality signal rather than triggering automatic re-generation. +After the Creative Director generates the initial DirectorScore, the Critic evaluates it immediately. If `revision_needed` is true and the score is below 7, the pipeline enters a revision loop: + +1. The Critic's `revision_instructions` and `weaknesses` are passed back to the Creative Director +2. The Director generates a revised DirectorScore addressing the feedback +3. The Critic re-evaluates the revised plan +4. The loop runs up to 2 rounds, tracking the highest-scoring revision + +This happens before TTS, visuals, or music generation, so revisions cost only LLM calls (~$0.02-0.13) instead of re-running the full pipeline ($0.50-2.00+). If the Critic fails during evaluation, the loop exits gracefully and proceeds with the current best score. The critique results are emitted through the progress callback so the web UI can display the score, strengths, and weaknesses to the user. diff --git a/docs/content/docs/pipeline/overview.mdx b/docs/content/docs/pipeline/overview.mdx index 06076958..b010936e 100644 --- a/docs/content/docs/pipeline/overview.mdx +++ b/docs/content/docs/pipeline/overview.mdx @@ -14,8 +14,9 @@ Topic Research ---- web search grounds the script in facts | v -Director ---- writes the DirectorScore (scenes, archetype, music mood) - | +Director ---- writes the DirectorScore, then Critic evaluates it + | ^ (revision loop: up to 2 rounds until score >= 7) + | | v TTS --------- generates voiceover audio with word-level timestamps | @@ -89,7 +90,7 @@ After all stages complete, actual LLM token usage is aggregated and the real cos The pipeline supports two early exit paths: -- **Dry run** (`--dry-run`) -- stops after the Director stage and prints the DirectorScore JSON. No assets are generated. +- **Dry run** (`--dry-run`) -- stops after the Director stage (including the quality gate revision loop) and prints the final DirectorScore JSON. No assets are generated. - **Cost rejection** -- if the user declines at the cost estimate prompt, the pipeline exits cleanly. Both paths still write `log.json` to the output directory. diff --git a/src/agents/creative-director.test.ts b/src/agents/creative-director.test.ts index 6753b1bf..33fb2248 100644 --- a/src/agents/creative-director.test.ts +++ b/src/agents/creative-director.test.ts @@ -1,5 +1,9 @@ -import { describe, expect, it } from "vitest"; -import { buildPacingInstruction, PACING_CONFIG } from "./creative-director.js"; +import { describe, expect, it, vi } from "vitest"; +import { buildPacingInstruction, PACING_CONFIG, reviseDirectorScore } from "./creative-director.js"; +import type { DirectorScore } from "../schema/director-score.js"; +import type { LLMProvider } from "../schema/providers.js"; +import type { CritiqueResult } from "./critic.js"; +import type { ResearchResult } from "./research.js"; describe("buildPacingInstruction", () => { // Path 2: explicit archetype — derive tier from config @@ -102,3 +106,81 @@ describe("PACING_CONFIG", () => { expect(PACING_CONFIG.cinematic.max).toBeLessThan(PACING_CONFIG.fast.max); }); }); + +// ── reviseDirectorScore tests ──────────────────────────────────────────────── + +const baseScore: DirectorScore = { + emotional_arc: "curiosity-to-wisdom", + archetype: "infographic", + music_mood: "epic_cinematic", + scenes: [ + { visual_type: "text_card", visual_prompt: "Title", motion: "static", script_line: "Did you know this?", transition: null }, + { visual_type: "ai_image", visual_prompt: "A diagram", motion: "zoom_in", script_line: "Here is the truth.", transition: "crossfade" }, + { visual_type: "stock_video", visual_prompt: "Ocean", motion: "static", script_line: "It changed everything.", transition: "slide_left" }, + { visual_type: "ai_image", visual_prompt: "Chart", motion: "pan_right", script_line: "The numbers prove it.", transition: "crossfade" }, + { visual_type: "stock_image", visual_prompt: "Sunset", motion: "zoom_out", script_line: "Think about that.", transition: "crossfade" }, + { visual_type: "text_card", visual_prompt: "CTA", motion: "static", script_line: "What do you think?", transition: "wipe" }, + { visual_type: "ai_image", visual_prompt: "Finale", motion: "zoom_in", script_line: "Comment below!", transition: null }, + { visual_type: "stock_video", visual_prompt: "Stars", motion: "static", script_line: "Follow for more.", transition: "crossfade" }, + ], +}; + +const baseResearch: ResearchResult = { + summary: "Test topic summary", + key_facts: ["fact1", "fact2"], + mood: "informative", + sources: [], +}; + +const baseCritique: CritiqueResult = { + score: 5, + strengths: ["good hook"], + weaknesses: ["weak pacing", "repetitive visuals"], + revision_needed: true, + revision_instructions: "Improve pacing in scenes 3-5 and add more visual variety.", + weakest_scene_index: 3, +}; + +function mockRevisionLLM(): LLMProvider & { lastUserMessage: string } { + const mock = { + id: "anthropic" as const, + lastUserMessage: "", + generate: vi.fn(async ({ userMessage }: { userMessage: string }) => { + mock.lastUserMessage = userMessage; + return { + data: { ...baseScore }, + usage: { inputTokens: 200, outputTokens: 100 }, + }; + }), + }; + return mock as unknown as LLMProvider & { lastUserMessage: string }; +} + +describe("reviseDirectorScore", () => { + it("generates a revised DirectorScore from critique feedback", async () => { + const llm = mockRevisionLLM(); + const result = await reviseDirectorScore(llm, "test topic", baseResearch, baseScore, baseCritique); + expect(result.data.archetype).toBe("infographic"); + expect(result.data.scenes.length).toBeGreaterThanOrEqual(3); + expect(result.usage.inputTokens).toBe(200); + }); + + it("includes critique details in the user message", async () => { + const llm = mockRevisionLLM(); + await reviseDirectorScore(llm, "test topic", baseResearch, baseScore, baseCritique); + expect(llm.lastUserMessage).toContain("score: 5/10"); + expect(llm.lastUserMessage).toContain("weak pacing"); + expect(llm.lastUserMessage).toContain("Improve pacing in scenes 3-5"); + expect(llm.lastUserMessage).toContain("Weakest scene: Scene 3"); + }); + + it("falls back to weaknesses when revision_instructions is null", async () => { + const llm = mockRevisionLLM(); + const critique: CritiqueResult = { + ...baseCritique, + revision_instructions: null, + }; + await reviseDirectorScore(llm, "test topic", baseResearch, baseScore, critique); + expect(llm.lastUserMessage).toContain("weak pacing; repetitive visuals"); + }); +}); diff --git a/src/agents/creative-director.ts b/src/agents/creative-director.ts index c6328b7f..1caa87f7 100644 --- a/src/agents/creative-director.ts +++ b/src/agents/creative-director.ts @@ -7,6 +7,7 @@ import { loadPlaybook } from "../config/playbook.js"; import { DirectorScore, Motion, MusicMood, TransitionType, VisualType } from "../schema/director-score.js"; import type { LLMProvider, LLMUsage } from "../schema/providers.js"; import type { ResearchResult } from "./research.js"; +import type { CritiqueResult } from "./critic.js"; const SYSTEM_PROMPT_PATH = path.join(process.cwd(), "prompts", "creative-director.md"); @@ -34,12 +35,8 @@ export interface DirectorScoreOutput { usage: LLMUsage; } -export async function generateDirectorScore( - llm: LLMProvider, - topic: string, - researchContext: ResearchResult, - options?: { archetype?: string; pacing?: string; videoEnabled?: boolean }, -): Promise { +/** Load the creative director system prompt with playbook injection */ +function loadDirectorSystemPrompt(): string { let systemPrompt = buildDefaultPrompt(); try { @@ -56,6 +53,17 @@ export async function generateDirectorScore( console.warn(`[creative-director] Playbook not loaded: ${err}`); } + return systemPrompt; +} + +export async function generateDirectorScore( + llm: LLMProvider, + topic: string, + researchContext: ResearchResult, + options?: { archetype?: string; pacing?: string; videoEnabled?: boolean }, +): Promise { + const systemPrompt = loadDirectorSystemPrompt(); + const archetypes = listArchetypes(); const archetypeInstruction = options?.archetype ? `Use the "${options.archetype}" archetype.` @@ -180,3 +188,92 @@ Per-scene word budget: ${cfg.wordsPerScene} words. Total word budget: ${cfg.tota } export { PACING_CONFIG }; + +// ── Revision ───────────────────────────────────────────────────────────────── + +export async function reviseDirectorScore( + llm: LLMProvider, + topic: string, + researchContext: ResearchResult, + originalScore: DirectorScore, + critique: CritiqueResult, + options?: { archetype?: string; pacing?: string; videoEnabled?: boolean }, +): Promise { + const systemPrompt = loadDirectorSystemPrompt(); + + // Build revision instructions from critique, guarding nullable revision_instructions + const revisionGuidance = critique.revision_instructions + ?? `Address these weaknesses: ${critique.weaknesses.join("; ")}`; + + const pacingInstruction = buildPacingInstruction(options?.archetype, options?.pacing); + + const videoEnabled = options?.videoEnabled ?? false; + const visualTypes = videoEnabled + ? "all 5 visual types (ai_image, ai_video, stock_image, stock_video, text_card)" + : "all 4 visual types (ai_image, stock_image, stock_video, text_card)"; + + const userMessage = `Topic: ${topic} + +Research context: +${researchContext.summary} + +Key facts: +${researchContext.key_facts.map((f) => `- ${f}`).join("\n")} + +Mood: ${researchContext.mood} + +${pacingInstruction} +Use ${visualTypes}. + +## Current Plan (score: ${critique.score}/10) + +${JSON.stringify(originalScore, null, 2)} + +## Critic Feedback + +Strengths: ${critique.strengths.join(", ")} +Weaknesses: ${critique.weaknesses.join(", ")} +${critique.weakest_scene_index != null ? `Weakest scene: Scene ${critique.weakest_scene_index}` : ""} + +## Revision Instructions + +${revisionGuidance} + +Revise the DirectorScore to address the weaknesses while preserving the strengths. +Keep the same archetype. Maintain the GOLDEN RULE: never use the same visual_type more than 2 times in a row.`; + + const maxRetries = 2; + let lastError: Error | null = null; + const totalUsage: LLMUsage = { inputTokens: 0, outputTokens: 0 }; + + for (let attempt = 0; attempt < maxRetries; attempt++) { + try { + const result = await llm.generate({ + systemPrompt, + userMessage: + attempt > 0 + ? `${userMessage}\n\nPREVIOUS ATTEMPT FAILED: ${lastError?.message}. Fix the issue.` + : userMessage, + schema: DirectorScoreRaw, + }); + + totalUsage.inputTokens += result.usage.inputTokens; + totalUsage.outputTokens += result.usage.outputTokens; + + const validated = DirectorScore.parse(result.data); + + // Prevent archetype drift: the LLM may change the archetype during revision + // despite prompt instructions. Force it back to the original. + if (validated.archetype !== originalScore.archetype) { + (validated as { archetype: string }).archetype = originalScore.archetype; + } + + return { data: validated, usage: totalUsage }; + } catch (err) { + lastError = err instanceof Error ? err : new Error(String(err)); + console.warn(`[creative-director] Revision attempt ${attempt + 1} failed: ${lastError.message}`); + } + } + + throw new Error(`Revision failed after ${maxRetries} attempts: ${lastError?.message}`); +} diff --git a/src/cli/cost-estimator.test.ts b/src/cli/cost-estimator.test.ts index eca7eac6..ef596ee6 100644 --- a/src/cli/cost-estimator.test.ts +++ b/src/cli/cost-estimator.test.ts @@ -70,7 +70,7 @@ describe("estimateCost", () => { ]); const result = estimateCost(score); expect(result.totalCost).toBeCloseTo( - result.llmCost + result.ttsCost + result.imageCost + result.videoCost + result.musicCost, + result.llmCost + result.revisionCost + result.ttsCost + result.imageCost + result.videoCost + result.musicCost, ); }); @@ -119,6 +119,30 @@ describe("estimateCost", () => { expect(fal.videoCost).toBeGreaterThan(gemini.videoCost); }); + it("includes revision cost with separate evaluation and revision counts", () => { + const score = makeScore([{ visual_type: "ai_image", script_line: "Test" }]); + // No gate activity + const noGate = estimateCost(score, "gemini", "elevenlabs", undefined, "anthropic", "bundled", 0, 0); + expect(noGate.revisionCost).toBe(0); + expect(noGate.details.gateEvaluations).toBe(0); + expect(noGate.details.revisionRounds).toBe(0); + + // Gate only (score >= 7 on first try): 1 eval, 0 revisions + const gateOnly = estimateCost(score, "gemini", "elevenlabs", undefined, "anthropic", "bundled", 1, 0); + expect(gateOnly.revisionCost).toBeGreaterThan(0); + expect(gateOnly.details.gateEvaluations).toBe(1); + expect(gateOnly.details.revisionRounds).toBe(0); + + // Full revision: 2 evals + 1 revision + const fullRevision = estimateCost(score, "gemini", "elevenlabs", undefined, "anthropic", "bundled", 2, 1); + expect(fullRevision.revisionCost).toBeGreaterThan(gateOnly.revisionCost); + expect(fullRevision.details.gateEvaluations).toBe(2); + expect(fullRevision.details.revisionRounds).toBe(1); + + // Gate-only should cost less than a full revision (no director call) + expect(gateOnly.revisionCost).toBeLessThan(fullRevision.revisionCost); + }); + it("uses gemini LLM pricing when llmProvider is gemini", () => { const score = makeScore([{ visual_type: "ai_image", script_line: "Test" }]); const anthropic = estimateCost(score, "gemini", "elevenlabs", undefined, "anthropic"); diff --git a/src/cli/cost-estimator.ts b/src/cli/cost-estimator.ts index 603a93c4..e774ddf0 100644 --- a/src/cli/cost-estimator.ts +++ b/src/cli/cost-estimator.ts @@ -10,6 +10,7 @@ import type { export interface CostBreakdown { llmCost: number; + revisionCost: number; ttsCost: number; imageCost: number; videoCost: number; @@ -17,6 +18,8 @@ export interface CostBreakdown { totalCost: number; details: { llmCalls: number; + gateEvaluations: number; + revisionRounds: number; ttsCharacters: number; aiImages: number; aiVideos: number; @@ -96,6 +99,8 @@ export function estimateCost( videoProvider?: VideoProviderKey, llmProvider: LLMProviderKey = "anthropic", musicProvider: MusicProviderKey = "bundled", + gateEvaluations = 0, + revisionRounds = 0, ): CostBreakdown { const aiImageScenes = score.scenes.filter((s) => s.visual_type === "ai_image").length; const aiVideoScenes = score.scenes.filter((s) => s.visual_type === "ai_video").length; @@ -114,6 +119,11 @@ export function estimateCost( callCost(TOKEN_ESTIMATES.creativeDirector) + callCost(TOKEN_ESTIMATES.critic) + aiImages * callCost(TOKEN_ESTIMATES.imagePrompter); + + // Quality gate cost: critic calls for evaluation + director calls for revision. + // Split because evaluations > revisions (gate always evaluates, only revises if score < 7). + // Slightly undercounts revise calls because the prompt includes the original score JSON (~1-3K extra tokens). + const revisionCost = gateEvaluations * callCost(TOKEN_ESTIMATES.critic) + revisionRounds * callCost(TOKEN_ESTIMATES.creativeDirector); const ttsPerChar = PRICING.ttsPerChar[ttsProvider]; const ttsCost = ttsCharacters * ttsPerChar; const perImage = imageProvider === "openai" ? PRICING.openaiPerImage : PRICING.geminiPerImage; @@ -127,7 +137,7 @@ export function estimateCost( const musicCost = musicProvider === "lyria" ? PRICING.lyriaPerTrack + callCost(TOKEN_ESTIMATES.imagePrompter) : 0; - const totalCost = llmCost + ttsCost + imageCost + videoCost + musicCost; + const totalCost = llmCost + revisionCost + ttsCost + imageCost + videoCost + musicCost; // Per-scene cost breakdown const perScene = score.scenes.map((s) => { @@ -152,8 +162,8 @@ export function estimateCost( }); return { - llmCost, ttsCost, imageCost, videoCost, musicCost, totalCost, - details: { llmCalls, ttsCharacters, aiImages, aiVideos: aiVideoScenes }, + llmCost, revisionCost, ttsCost, imageCost, videoCost, musicCost, totalCost, + details: { llmCalls, gateEvaluations, revisionRounds, ttsCharacters, aiImages, aiVideos: aiVideoScenes }, perScene, }; } @@ -167,9 +177,17 @@ export function formatCostEstimate( const lines = [ `Estimated cost: $${breakdown.totalCost.toFixed(3)}`, ` LLM: $${breakdown.llmCost.toFixed(4)} (${breakdown.details.llmCalls} calls)`, + ]; + if (breakdown.revisionCost > 0) { + const evals = breakdown.details.gateEvaluations; + const revs = breakdown.details.revisionRounds; + const detail = revs > 0 ? `${evals} eval${evals !== 1 ? "s" : ""}, ${revs} revision${revs !== 1 ? "s" : ""}` : `${evals} eval${evals !== 1 ? "s" : ""}`; + lines.push(` Gate: $${breakdown.revisionCost.toFixed(4)} (${detail})`); + } + lines.push( ` TTS: $${breakdown.ttsCost.toFixed(4)} (${breakdown.details.ttsCharacters} chars)`, ` Images: $${breakdown.imageCost.toFixed(4)} (${breakdown.details.aiImages} AI images @ $${perImage.toFixed(3)}/ea)`, - ]; + ); if (breakdown.details.aiVideos > 0) { lines.push(` Video: $${breakdown.videoCost.toFixed(4)} (${breakdown.details.aiVideos} AI videos)`); } diff --git a/src/pipeline/orchestrator-callbacks.test.ts b/src/pipeline/orchestrator-callbacks.test.ts index 0f407c17..af60294b 100644 --- a/src/pipeline/orchestrator-callbacks.test.ts +++ b/src/pipeline/orchestrator-callbacks.test.ts @@ -76,12 +76,13 @@ describe("createCliCallbacks", () => { const result = await callbacks.onCostEstimate!( { llmCost: 0, + revisionCost: 0, ttsCost: 0, imageCost: 0, totalCost: 0.1, videoCost: 0, musicCost: 0, - details: { llmCalls: 3, ttsCharacters: 500, aiImages: 2, aiVideos: 0 }, + details: { llmCalls: 3, gateEvaluations: 0, revisionRounds: 0, ttsCharacters: 500, aiImages: 2, aiVideos: 0 }, }, "gemini", ); diff --git a/src/pipeline/orchestrator-revision.test.ts b/src/pipeline/orchestrator-revision.test.ts new file mode 100644 index 00000000..104c2e53 --- /dev/null +++ b/src/pipeline/orchestrator-revision.test.ts @@ -0,0 +1,266 @@ +import { describe, expect, it, vi } from "vitest"; +import type { PipelineCallbacks, StageName } from "./orchestrator.js"; +import type { DirectorScore } from "../schema/director-score.js"; +import type { CritiqueResult } from "../agents/critic.js"; + +/** + * Tests for the Director-Critic revision loop inside directorStep.execute. + * + * These test the loop logic by simulating what the orchestrator does: + * generate → evaluate → (revise if score < 7) → use best score. + * + * Full integration tests (real LLM calls) are not covered here. + */ + +const makeScore = (archetype = "infographic", sceneCount = 8): DirectorScore => ({ + emotional_arc: "curiosity-to-wisdom", + archetype, + music_mood: "epic_cinematic", + scenes: Array.from({ length: sceneCount }, (_, i) => { + const visualTypes = ["text_card", "ai_image", "stock_video", "ai_image", "stock_image", "text_card", "ai_image", "stock_video"] as const; + return { + visual_type: visualTypes[i % 8]!, + visual_prompt: `Scene ${i + 1}`, + motion: "static" as const, + script_line: `This is scene ${i + 1}.`, + transition: i < sceneCount - 1 ? ("crossfade" as const) : null, + }; + }), +}); + +const makeCritique = (score: number, revisionNeeded: boolean): CritiqueResult => ({ + score, + strengths: ["good hook", "visual variety"], + weaknesses: score < 7 ? ["weak pacing", "repetitive script"] : [], + revision_needed: revisionNeeded, + revision_instructions: revisionNeeded ? "Improve pacing in the middle scenes." : null, + weakest_scene_index: revisionNeeded ? 3 : null, +}); + +/** + * Simulates the revision loop from orchestrator.ts directorStep.execute. + * Extracted here so we can test the logic without running the full Mastra workflow. + */ +async function simulateRevisionLoop(opts: { + initialScore: DirectorScore; + evaluateFn: (score: DirectorScore) => Promise<{ data: CritiqueResult; usage: { inputTokens: number; outputTokens: number } }>; + reviseFn: (score: DirectorScore, critique: CritiqueResult) => Promise<{ data: DirectorScore; usage: { inputTokens: number; outputTokens: number } }>; + callbacks?: PipelineCallbacks; + maxRounds?: number; +}) { + const { initialScore, evaluateFn, reviseFn, callbacks: cb, maxRounds = 2 } = opts; + const llmUsages: { inputTokens: number; outputTokens: number }[] = []; + let score = initialScore; + let bestScore = score; + let bestCritiqueScore = 0; + let revisionRoundsCompleted = 0; + + for (let round = 0; round < maxRounds; round++) { + try { + const critiqueOutput = await evaluateFn(score); + llmUsages.push(critiqueOutput.usage); + const critique = critiqueOutput.data; + + if (critique.score > bestCritiqueScore) { + bestScore = score; + bestCritiqueScore = critique.score; + } + + if (critique.score >= 7 || !critique.revision_needed) break; + + cb?.onProgress?.("director", { type: "revision", round: round + 1, critiqueScore: critique.score }); + cb?.onLog?.(`\n[director] Critic score: ${critique.score}/10 (round ${round + 1}), revising...`); + + const revised = await reviseFn(score, critique); + llmUsages.push(revised.usage); + score = revised.data; + revisionRoundsCompleted++; + } catch (err) { + console.warn(`[director] Revision round ${round + 1} failed: ${err}`); + break; + } + } + + // Final evaluation: if the last revision was never scored, give it a chance + if (revisionRoundsCompleted > 0 && score !== bestScore) { + try { + const finalCritique = await evaluateFn(score); + llmUsages.push(finalCritique.usage); + if (finalCritique.data.score > bestCritiqueScore) { + bestScore = score; + bestCritiqueScore = finalCritique.data.score; + } + } catch { + // bestScore from the loop is still valid + } + } + + score = bestScore; + return { finalScore: score, bestCritiqueScore, revisionRoundsCompleted, llmUsages }; +} + +describe("Director-Critic revision loop", () => { + it("skips revision when initial score >= 7", async () => { + const evaluateFn = vi.fn().mockResolvedValue({ + data: makeCritique(8, false), + usage: { inputTokens: 100, outputTokens: 50 }, + }); + const reviseFn = vi.fn(); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + expect(evaluateFn).toHaveBeenCalledTimes(1); + expect(reviseFn).not.toHaveBeenCalled(); + expect(result.revisionRoundsCompleted).toBe(0); + expect(result.bestCritiqueScore).toBe(8); + }); + + it("triggers revision when score < 7 and improves to >= 7", async () => { + const revisedScore = makeScore("infographic", 9); + const evaluateFn = vi.fn() + .mockResolvedValueOnce({ data: makeCritique(5, true), usage: { inputTokens: 100, outputTokens: 50 } }) + .mockResolvedValueOnce({ data: makeCritique(8, false), usage: { inputTokens: 100, outputTokens: 50 } }); + const reviseFn = vi.fn().mockResolvedValue({ + data: revisedScore, + usage: { inputTokens: 200, outputTokens: 100 }, + }); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + expect(evaluateFn).toHaveBeenCalledTimes(2); + expect(reviseFn).toHaveBeenCalledTimes(1); + expect(result.revisionRoundsCompleted).toBe(1); + expect(result.finalScore).toBe(revisedScore); // best score is the revised one (scored 8) + expect(result.bestCritiqueScore).toBe(8); + }); + + it("exhausts max rounds and evaluates the final revision", async () => { + const round1Score = makeScore("infographic", 9); + const round2Score = makeScore("infographic", 10); + + const evaluateFn = vi.fn() + .mockResolvedValueOnce({ data: makeCritique(4, true), usage: { inputTokens: 100, outputTokens: 50 } }) + .mockResolvedValueOnce({ data: makeCritique(6, true), usage: { inputTokens: 100, outputTokens: 50 } }) + // Final evaluation of round2Score after loop exhausts + .mockResolvedValueOnce({ data: makeCritique(9, false), usage: { inputTokens: 100, outputTokens: 50 } }); + const reviseFn = vi.fn() + .mockResolvedValueOnce({ data: round1Score, usage: { inputTokens: 200, outputTokens: 100 } }) + .mockResolvedValueOnce({ data: round2Score, usage: { inputTokens: 200, outputTokens: 100 } }); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + // 2 in-loop evaluations + 1 final evaluation + expect(evaluateFn).toHaveBeenCalledTimes(3); + expect(reviseFn).toHaveBeenCalledTimes(2); + expect(result.revisionRoundsCompleted).toBe(2); + // round2Score scored 9 in final eval, beating round1Score's 6 + expect(result.finalScore).toBe(round2Score); + expect(result.bestCritiqueScore).toBe(9); + }); + + it("uses highest-scoring revision when final evaluation degrades", async () => { + const round1Score = makeScore("infographic", 9); + const round2Score = makeScore("infographic", 10); + + // Round 0: initial scores 3, revise → round1Score + // Round 1: round1Score scores 6, revise → round2Score + // Final eval: round2Score scores 4 (degradation) + // Result: should use round1Score (scored 6), not round2Score (scored 4) + const evaluateFn = vi.fn() + .mockResolvedValueOnce({ data: makeCritique(3, true), usage: { inputTokens: 100, outputTokens: 50 } }) + .mockResolvedValueOnce({ data: makeCritique(6, true), usage: { inputTokens: 100, outputTokens: 50 } }) + // Final evaluation of round2Score — degraded + .mockResolvedValueOnce({ data: makeCritique(4, false), usage: { inputTokens: 100, outputTokens: 50 } }); + const reviseFn = vi.fn() + .mockResolvedValueOnce({ data: round1Score, usage: { inputTokens: 200, outputTokens: 100 } }) + .mockResolvedValueOnce({ data: round2Score, usage: { inputTokens: 200, outputTokens: 100 } }); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + // 2 in-loop + 1 final = 3 evaluations + expect(evaluateFn).toHaveBeenCalledTimes(3); + // Best is round1Score which was evaluated at 6 (round2Score's final eval of 4 didn't beat it) + expect(result.finalScore).toBe(round1Score); + expect(result.bestCritiqueScore).toBe(6); + }); + + it("emits revision progress events", async () => { + const events: Array<{ stage: StageName; data: Record }> = []; + const cb: PipelineCallbacks = { + onProgress(stage, data) { events.push({ stage, data }); }, + onLog() {}, + }; + + const evaluateFn = vi.fn() + .mockResolvedValueOnce({ data: makeCritique(5, true), usage: { inputTokens: 100, outputTokens: 50 } }) + .mockResolvedValueOnce({ data: makeCritique(8, false), usage: { inputTokens: 100, outputTokens: 50 } }); + const reviseFn = vi.fn().mockResolvedValue({ + data: makeScore(), + usage: { inputTokens: 200, outputTokens: 100 }, + }); + + await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + callbacks: cb, + }); + + expect(events).toHaveLength(1); + expect(events[0]?.stage).toBe("director"); + expect(events[0]?.data).toEqual({ type: "revision", round: 1, critiqueScore: 5 }); + }); + + it("gracefully handles evaluate() failure", async () => { + const evaluateFn = vi.fn().mockRejectedValue(new Error("LLM JSON parse failed")); + const reviseFn = vi.fn(); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + // Should proceed with initial score, no crash + expect(result.revisionRoundsCompleted).toBe(0); + expect(result.finalScore).toEqual(makeScore()); + expect(reviseFn).not.toHaveBeenCalled(); + }); + + it("accumulates LLM usage across all revision rounds", async () => { + const evaluateFn = vi.fn() + .mockResolvedValueOnce({ data: makeCritique(5, true), usage: { inputTokens: 100, outputTokens: 50 } }) + .mockResolvedValueOnce({ data: makeCritique(8, false), usage: { inputTokens: 120, outputTokens: 60 } }); + const reviseFn = vi.fn().mockResolvedValue({ + data: makeScore(), + usage: { inputTokens: 200, outputTokens: 100 }, + }); + + const result = await simulateRevisionLoop({ + initialScore: makeScore(), + evaluateFn, + reviseFn, + }); + + // 2 evaluate calls + 1 revise call = 3 usage entries + expect(result.llmUsages).toHaveLength(3); + const totalInput = result.llmUsages.reduce((sum, u) => sum + u.inputTokens, 0); + expect(totalInput).toBe(100 + 200 + 120); // eval1 + revise + eval2 + }); +}); diff --git a/src/pipeline/orchestrator.ts b/src/pipeline/orchestrator.ts index e92c9f37..231707eb 100644 --- a/src/pipeline/orchestrator.ts +++ b/src/pipeline/orchestrator.ts @@ -5,7 +5,7 @@ import * as path from "node:path"; import { Mastra } from "@mastra/core"; import { createStep, createWorkflow } from "@mastra/core/workflows"; import { z } from "zod"; -import { generateDirectorScore } from "../agents/creative-director.js"; +import { generateDirectorScore, reviseDirectorScore } from "../agents/creative-director.js"; import { evaluate } from "../agents/critic.js"; import { optimizeImagePrompt } from "../agents/image-prompter.js"; import { research } from "../agents/research.js"; @@ -410,45 +410,114 @@ function buildPipelineWorkflow( cb.onStageStart?.("director"); const start = Date.now(); const videoEnabled = !opts.noVideo && (opts.videoProviders?.length ?? 0) > 0; - const cdOutput = await generateDirectorScore(opts.llm, opts.topic, inputData, { - archetype: opts.archetype, - pacing: opts.pacing, - videoEnabled, - }); - // Store on shared context via closure - directorResult.score = cdOutput.data; - directorResult.config = getArchetype(cdOutput.data.archetype); + const directorOpts = { archetype: opts.archetype, pacing: opts.pacing, videoEnabled }; + + // ── Generate initial DirectorScore ── + const cdOutput = await generateDirectorScore(opts.llm, opts.topic, inputData, directorOpts); llmUsages.push(cdOutput.usage); + let score = cdOutput.data; + + // ── Revision loop: evaluate → revise until score >= 7 or max rounds ── + // + // generate ──► evaluate ──► score >= 7? ──YES──► done + // │ + // NO (round < MAX) + // │ + // revise ──► re-evaluate ──► ... + // + // Tracks the highest-scoring revision (LLM refinement can degrade in later rounds). + const MAX_REVISION_ROUNDS = 2; + let bestScore = score; + let bestCritiqueScore = 0; + let revisionRoundsCompleted = 0; + + let evaluationsCompleted = 0; + + for (let round = 0; round < MAX_REVISION_ROUNDS; round++) { + try { + const critiqueOutput = await evaluate(opts.llm, score, opts.topic, opts.pacing); + llmUsages.push(critiqueOutput.usage); + evaluationsCompleted++; + const critique = critiqueOutput.data; + + // Track highest-scoring revision + if (critique.score > bestCritiqueScore) { + bestScore = score; + bestCritiqueScore = critique.score; + } + + if (critique.score >= 7 || !critique.revision_needed) break; + + cb.onProgress?.("director", { type: "revision", round: round + 1, critiqueScore: critique.score }); + cb.onLog?.(`\n[director] Critic score: ${critique.score}/10 (round ${round + 1}), revising...`); + + const revised = await reviseDirectorScore(opts.llm, opts.topic, inputData, score, critique, directorOpts); + llmUsages.push(revised.usage); + score = revised.data; + revisionRoundsCompleted++; + } catch (err) { + // Graceful degradation: if the critic or revision fails, proceed with current best score + console.warn(`[director] Revision round ${round + 1} failed: ${err}`); + break; + } + } + + // Final evaluation: if the last revision was never scored (loop exhausted), + // give it a chance to compete with bestScore + if (revisionRoundsCompleted > 0 && score !== bestScore) { + try { + const finalCritique = await evaluate(opts.llm, score, opts.topic, opts.pacing); + llmUsages.push(finalCritique.usage); + evaluationsCompleted++; + if (finalCritique.data.score > bestCritiqueScore) { + bestScore = score; + bestCritiqueScore = finalCritique.data.score; + } + } catch { + // If final evaluation fails, bestScore from the loop is still valid + } + } + + // Use the highest-scoring revision + score = bestScore; + + // ── Store final score on shared closure state ── + directorResult.score = score; + directorResult.config = getArchetype(score.archetype); + const dur = (Date.now() - start) / 1000; cb.onStageComplete?.( "director", - `${cdOutput.data.scenes.length} scenes, ${cdOutput.data.archetype}`, + `${score.scenes.length} scenes, ${score.archetype}`, dur, ); log.stages.push({ name: "creative-director", duration: dur, status: "done" }); - fs.writeFileSync(scorePath, JSON.stringify(cdOutput.data, null, 2)); - cb.onProgress?.("director", { type: "score", score: cdOutput.data }); + // Write score.json and emit progress AFTER the revision loop + fs.writeFileSync(scorePath, JSON.stringify(score, null, 2)); + cb.onProgress?.("director", { type: "score", score }); - // Dry run handling + // Dry run handling (after revision loop so --dry-run shows revised score) if (opts.dryRun) { cb.onStageSkip?.("tts", "dry run"); cb.onStageSkip?.("visuals", "dry run"); cb.onStageSkip?.("assembly", "dry run"); cb.onStageSkip?.("critic", "dry run"); cb.onLog?.("\n--- DirectorScore ---"); - cb.onLog?.(JSON.stringify(cdOutput.data, null, 2)); + cb.onLog?.(JSON.stringify(score, null, 2)); directorResult.dryRunExit = true; return { done: true }; } - // Cost estimation - const costBreakdown = estimateCost(cdOutput.data, opts.imageProvider, opts.ttsProvider, opts.videoProvider, opts.llm.id, opts.musicProviderKey); + // Cost estimation (uses the final revised score for accurate scene counts) + // Pass evaluations and revisions separately: evaluations count critic calls, + // revisions count director calls (evaluations >= revisions since gate always evaluates) + const costBreakdown = estimateCost(score, opts.imageProvider, opts.ttsProvider, opts.videoProvider, opts.llm.id, opts.musicProviderKey, evaluationsCompleted, revisionRoundsCompleted); directorResult.costBreakdown = costBreakdown; log.totalCost = { estimated: costBreakdown.totalCost }; if (cb.onCostEstimate) { - const stockSceneCount = cdOutput.data.scenes.filter( + const stockSceneCount = score.scenes.filter( (s) => s.visual_type === "stock_image" || s.visual_type === "stock_video", ).length; const proceed = await cb.onCostEstimate(costBreakdown, opts.imageProvider, stockSceneCount); @@ -734,13 +803,7 @@ function buildPipelineWorkflow( llmUsages.push(critiqueOutput.usage); const dur = (Date.now() - start) / 1000; - if (critique.revision_needed && critique.score < 7) { - cb.onStageComplete?.("critic", `score ${critique.score}/10, revision needed`, dur); - cb.onLog?.(`\nCritic score: ${critique.score}/10 — ${critique.weaknesses.join(", ")}`); - cb.onLog?.("Revision support coming in a future release. Using current version."); - } else { - cb.onStageComplete?.("critic", `score ${critique.score}/10`, dur); - } + cb.onStageComplete?.("critic", `score ${critique.score}/10`, dur); cb.onProgress?.("critic", { type: "review", score: critique.score, diff --git a/src/worker.ts b/src/worker.ts index 2af3e1cd..584e1112 100644 --- a/src/worker.ts +++ b/src/worker.ts @@ -72,6 +72,7 @@ interface JobMeta { criticReview?: { score: number; strengths: string[]; weaknesses: string[] }; musicTrack?: { trackId: string; mood: string; requestedMood: string; fallback: boolean }; musicGeneration?: { provider: string; prompt?: string; metadata?: Record; fallback: boolean }; + revisionHistory?: { round: number; score: number }[]; error?: string; } @@ -166,6 +167,13 @@ const worker = new Worker( fallback: data.fallback as boolean, }; writeMeta(jobDir, meta); + } else if (data.type === "revision") { + if (!meta.revisionHistory) meta.revisionHistory = []; + meta.revisionHistory.push({ + round: data.round as number, + score: data.critiqueScore as number, + }); + writeMeta(jobDir, meta); } else if (data.type === "review") { meta.criticReview = { score: data.score as number, strengths: data.strengths as string[], weaknesses: data.weaknesses as string[] }; writeMeta(jobDir, meta);