|
| 1 | +# Non-linear generation: backport across pipeline stages & Story Builder steps |
| 2 | + |
| 3 | +## Context |
| 4 | + |
| 5 | +Today both the **Create Series Pipeline** and the **Story Builder** are strictly |
| 6 | +*forward-only*. Each generation step assumes its source is the conventional |
| 7 | +upstream artifact: |
| 8 | + |
| 9 | +- Pipeline text stages (`idea → prose → comicScript → teleplay`) are wired so |
| 10 | + `buildStageContext()` (`server/services/pipeline/textStages.js:53`) only ever |
| 11 | + includes stages *before* the current one (`if (id === stageId) break;`), and |
| 12 | + each prompt template hardcodes one source slot (`{{seed}}`, `{{stages.idea.content}}`, |
| 13 | + `{{stages.prose.content}}`). |
| 14 | +- Story Builder gates every step behind `isStepReachable()` |
| 15 | + (`server/services/storyBuilder.js`), which hard-throws `409` unless **every** |
| 16 | + earlier step is locked + not stale; its generators only pull from their |
| 17 | + conventional upstream (seed → idea → aesthetic → arc → readerMap). |
| 18 | + |
| 19 | +The user often has work that was authored out of order — e.g. a series that |
| 20 | +*started* as a drafted comic book. They want to **backfill the prior steps** to |
| 21 | +evaluate whether the story is complete: generate prose from a comic script, |
| 22 | +synthesize the idea/arc from issues that already have comic scripts, etc. |
| 23 | + |
| 24 | +**Goal:** |
| 25 | +1. **Pipeline** — each stage's Generate button offers a **multi-select source |
| 26 | + picker** listing only the stages that currently have content (never the target |
| 27 | + itself), with the conventional forward source pre-checked. Generation feeds |
| 28 | + the chosen sources into the LLM. |
| 29 | +2. **Story Builder** — relax the hard lock-gate to advisory, and make the |
| 30 | + upstream generators able to **pull from existing downstream content** (the |
| 31 | + series' issues' comic-script/prose) — reusing the importer's extraction |
| 32 | + prompts — so you can start from a comic script and backfill idea/arc. |
| 33 | + |
| 34 | +Decisions confirmed with the user: scope = **Pipeline + Story Builder gating**; |
| 35 | +pipeline source picker = **multi-select with smart (conventional-forward) |
| 36 | +default**; Story Builder = **generate upstream from downstream** (full backfill, |
| 37 | +not just navigation unlock). |
| 38 | + |
| 39 | +--- |
| 40 | + |
| 41 | +## Part 1 — Pipeline text-stage source picker |
| 42 | + |
| 43 | +### 1a. Server: source-agnostic context (`server/services/pipeline/textStages.js`) |
| 44 | + |
| 45 | +`buildStageContext({ series, canon, world, issue, stageId, seedInput, sourceStageIds })`: |
| 46 | + |
| 47 | +- **Keep** the existing forward-only `stages` object exactly as-is. Customized |
| 48 | + installs whose prompts still reference `{{stages.idea.content}}` / |
| 49 | + `{{stages.prose.content}}` must keep working (these prompts are NOT migrated for |
| 50 | + customized installs — see 1c). |
| 51 | +- **Add** a new `sourceMaterials` array to the returned context: |
| 52 | + ```js |
| 53 | + // selected = explicit sourceStageIds if provided & non-empty, else the |
| 54 | + // conventional forward-prior stages that have content (mirrors today's default) |
| 55 | + sourceMaterials: selected.map((id) => ({ |
| 56 | + stageId: id, |
| 57 | + label: STAGE_LABELS[id], // reuse existing STAGE_LABELS map (line 22) |
| 58 | + content: contentOf(issue.stages?.[id]) // input?.trim() || output?.trim() || '' |
| 59 | + })).filter((s) => s.content) |
| 60 | + ``` |
| 61 | + - When `sourceStageIds` is omitted/empty → default to the forward-prior stages |
| 62 | + with content (so `autoRunner.js`, which calls `generateStage` with no |
| 63 | + `sourceStageIds`, is byte-for-byte unchanged in behavior). |
| 64 | + - When provided → validate each id is a `TEXT_STAGE_IDS` member, is **not** the |
| 65 | + target `stageId`, and has content; silently drop any that fail (or throw 400 — |
| 66 | + pick throw for explicit user requests). |
| 67 | +- `generateStage(issueId, stageId, options)` passes |
| 68 | + `sourceStageIds: options.sourceStageIds` into `buildStageContext`. |
| 69 | +
|
| 70 | +### 1b. Server route (`server/routes/pipeline.js`) |
| 71 | +
|
| 72 | +Extend `generateSchema` (~line 350): |
| 73 | +```js |
| 74 | +sourceStageIds: z.array(z.enum(['idea', 'prose', 'comicScript', 'teleplay'])).optional(), |
| 75 | +``` |
| 76 | +The handler already spreads the validated body into `generateStage` — no further |
| 77 | +route change. `client/src/services/apiPipeline.js` `generatePipelineStage` already |
| 78 | +forwards arbitrary `opts`, so no client-API change. |
| 79 | +
|
| 80 | +### 1c. Prompt templates + migration |
| 81 | +
|
| 82 | +All four templates (`data.reference/prompts/stages/pipeline-{idea-expansion,prose, |
| 83 | +comic-script,teleplay}.md`) currently hardcode one source slot. Replace that slot |
| 84 | +with a generic, source-agnostic block and make the task wording source-neutral: |
| 85 | +
|
| 86 | +```md |
| 87 | +{{#sourceMaterials}} |
| 88 | +## Source material — {{label}} |
| 89 | +{{content}} |
| 90 | + |
| 91 | +{{/sourceMaterials}} |
| 92 | +``` |
| 93 | +- For `pipeline-idea-expansion.md`: keep the `{{seed}}` block (rough idea) AND add |
| 94 | + the `sourceMaterials` block (so the beat sheet can also be synthesized from an |
| 95 | + existing prose/comic/teleplay). |
| 96 | +- Reword task lines from "adapting a beat sheet"/"adapting a prose story" to |
| 97 | + "adapting the provided source material" so the prompt reads correctly whichever |
| 98 | + source(s) are supplied. |
| 99 | +
|
| 100 | +Because `scripts/setup-data.js` only copies *missing* prompts, ship a migration |
| 101 | +following the canonical pattern in `scripts/migrations/003-update-pipeline-stage-prompts.js`: |
| 102 | +new file `scripts/migrations/0NN-pipeline-stage-prompts-source-agnostic.js` |
| 103 | +(next free number after the highest in `scripts/migrations/`). For each of the 4 |
| 104 | +files: if the installed copy's md5 (line-ending-normalized) matches the |
| 105 | +**pre-change shipped hash**, overwrite with the new shipped version; otherwise |
| 106 | +leave the user's customization untouched. Mirror both `OLD_SHIPPED_MD5` and |
| 107 | +`NEW_SHIPPED_MD5` into the drift warning in `scripts/setup-data.js` so it stays |
| 108 | +actionable (per CLAUDE.md "Stage-prompt template changes need a migration"). |
| 109 | +
|
| 110 | +### 1d. Client UI (`client/src/components/pipeline/stages/TextStagePanel.jsx`) |
| 111 | +
|
| 112 | +- Compute available sources: iterate the text stage order |
| 113 | + (`['idea','prose','comicScript','teleplay']`), exclude the current `stageId`, |
| 114 | + keep those with `input?.trim() || output?.trim()`. |
| 115 | +- Render a compact **multi-select** (checkbox row or chips) labeled "Generate |
| 116 | + from:" above the Generate button, listing only available sources, using |
| 117 | + `PIPELINE_STAGE_LABELS` (`client/src/lib/pipelineStages.js`) for labels. |
| 118 | + - Default-checked = the conventional forward source(s) that exist (e.g. for |
| 119 | + `prose` → `idea`; for `comicScript`/`teleplay` → `prose`). If none of the |
| 120 | + conventional sources exist, leave all unchecked and let the user pick. |
| 121 | + - Hide the picker entirely when no other stage has content (fresh issue) — the |
| 122 | + panel then behaves exactly like today. |
| 123 | +- Track selection in local state; pass `sourceStageIds: selected` in the |
| 124 | + `generatePipelineStage(issue.id, stageId, { ... })` call. |
| 125 | +- Update `PLACEHOLDERS` / `HELP_TEXT` copy to stop asserting a fixed source |
| 126 | + ("Generated from the prose draft above" → "Generated from the selected source"). |
| 127 | +
|
| 128 | +### 1e. Tests |
| 129 | +
|
| 130 | +`server/services/pipeline/textStages.test.js`: |
| 131 | +- Default (no `sourceStageIds`) still produces the forward-only `sourceMaterials` |
| 132 | + and matches prior behavior. |
| 133 | +- Backport: target `prose` with `sourceStageIds:['comicScript']` puts the comic |
| 134 | + script content into `sourceMaterials` and omits `idea`. |
| 135 | +- Invalid source (target===source, or empty stage) is rejected/dropped. |
| 136 | +
|
| 137 | +`server/routes/pipeline.test.js`: `sourceStageIds` accepted by `generateSchema`; |
| 138 | +bad enum value 400s. |
| 139 | +
|
| 140 | +--- |
| 141 | +
|
| 142 | +## Part 2 — Story Builder: backfill upstream from downstream |
| 143 | +
|
| 144 | +### 2a. Relax the reachability gate (advisory, not hard block) |
| 145 | +
|
| 146 | +In `server/services/storyBuilder.js`: |
| 147 | +- `generateStep()` currently throws `409` when `!isStepReachable(session, stepId)`. |
| 148 | + Change so the lock check (`step.locked` → 409) **stays**, but the |
| 149 | + upstream-incomplete condition no longer blocks generation. Keep |
| 150 | + `isStepReachable()` and surface its result to the client (already returned in |
| 151 | + session/step payloads) as an advisory `reachable`/`stale` flag the UI renders as |
| 152 | + a warning badge — do **not** throw on it. |
| 153 | +- The stepper UI (`client/src/pages/StoryBuilder.jsx`) should allow clicking into |
| 154 | + any non-locked step and show a "upstream not locked / may be stale" warning |
| 155 | + instead of disabling it. |
| 156 | +
|
| 157 | +### 2b. Backfill-capable generators (pull from existing issue content) |
| 158 | +
|
| 159 | +The Story Builder generators (`STEP_GENERATORS` in `storyBuilder.js`) operate at |
| 160 | +universe/series level. Add an optional `fromDownstream` path so `idea` and |
| 161 | +`plotArc` can synthesize from the series' existing issue content when upstream is |
| 162 | +empty: |
| 163 | +
|
| 164 | +- Add a helper that collects source text from the session's series issues via |
| 165 | + `getIssuesForSeries(session.seriesId)` (already imported, line 10): concatenate |
| 166 | + each issue's most-authored text stage (prefer `comicScript`, else `teleplay`, |
| 167 | + else `prose`, else `idea`) with a per-issue label, capped to a sane size. |
| 168 | +- **`plotArc` generator:** when `options.fromDownstream` (or when no arc exists yet |
| 169 | + but issues do), feed that collected issue content into arc generation. Reuse the |
| 170 | + importer's arc-extraction prompt (`importer-arc-extract.md`) — the importer |
| 171 | + already reverse-engineers `{logline, summary, seasons[...]}` from a finished |
| 172 | + work; route through the same extraction service the importer uses |
| 173 | + (`server/services/importer.js`) rather than duplicating the prompt call. Persist |
| 174 | + to `series.arc` / `series.seasons` exactly as the forward `generateArcOverview` |
| 175 | + path does. |
| 176 | +- **`idea` generator:** when backfilling, include the collected issue content as |
| 177 | + additional source in the `story-builder-idea-expand` prompt input (add an |
| 178 | + optional `sourceMaterial` field to `data.reference/prompts/stages/story-builder-idea-expand.md`, |
| 179 | + rendered only when present — migrate via the same migration mechanism as 1c). |
| 180 | +- **characters / readerMap:** `characters` can already be backfilled by the |
| 181 | + importer's canon extraction (`importer-canon-extract.md`); `readerMap` derives |
| 182 | + from the (now backfillable) arc, so it works once arc exists. No new generator |
| 183 | + needed for these in this pass — note in PLAN.md if deeper backfill is wanted. |
| 184 | +
|
| 185 | +### 2c. Plumb the option through route + client |
| 186 | +
|
| 187 | +- `server/routes/storyBuilder.js`: add `fromDownstream: z.boolean().optional()` |
| 188 | + (and, if exposing source choice, an optional source descriptor) to the |
| 189 | + generate-step request schema; forward into `generateStep` options. |
| 190 | +- `client/src/services/apiStoryBuilder.js` + `StoryBuilder.jsx`: when a step's |
| 191 | + conventional upstream is empty but downstream issue content exists, offer a |
| 192 | + "Backfill from existing issues" affordance on that step's generate control that |
| 193 | + sends `fromDownstream: true`. |
| 194 | +
|
| 195 | +### 2d. Tests |
| 196 | +
|
| 197 | +- `server/services/storyBuilder.test.js`: generating `plotArc` with |
| 198 | + `fromDownstream` on a session whose series has issues-with-comic-scripts but no |
| 199 | + arc produces a persisted arc; the reachability gate no longer throws 409 for an |
| 200 | + unlocked-but-explicitly-requested upstream step (only `locked` throws). |
| 201 | +- `server/routes/storyBuilder.test.js`: new schema field accepted/validated. |
| 202 | +
|
| 203 | +--- |
| 204 | +
|
| 205 | +## Compatibility notes (per CLAUDE.md) |
| 206 | +
|
| 207 | +- **Migrations required** for every shipped prompt change (1c, 2b idea prompt) — |
| 208 | + other installs/machines run this code and upgrade independently. Use the |
| 209 | + hash-gated rewrite pattern and update the `setup-data.js` drift warning. |
| 210 | +- Keep `buildStageContext`'s legacy `stages` object so customized (un-migrated) |
| 211 | + prompts keep resolving. |
| 212 | +- `autoRunner.js` must remain behaviorally unchanged (it calls `generateStage` |
| 213 | + with no `sourceStageIds` → default forward sources). |
| 214 | +- No sync/schema-version payload changes expected (Story Builder sessions are |
| 215 | + local-only; pipeline issue shape is unchanged — only generation *inputs* change). |
| 216 | +
|
| 217 | +## Verification |
| 218 | +
|
| 219 | +1. `cd server && npm test` (textStages, pipeline route, storyBuilder service+route). |
| 220 | +2. `cd client && npm test` (TextStagePanel / StoryBuilder component tests). |
| 221 | +3. Manual via `npm run dev`: |
| 222 | + - Pipeline: open an issue that has only a comic script; on the Prose stage the |
| 223 | + source picker lists "Comic Script" (and Teleplay if present), Idea is absent. |
| 224 | + Generate → prose is produced from the comic script. Verify the Idea stage can |
| 225 | + be generated from prose/comic (backport). |
| 226 | + - Story Builder: import or hand-build a session whose series has issues with |
| 227 | + comic scripts but no arc; confirm you can enter the Plot Arc step despite |
| 228 | + upstream being unlocked, click "Backfill from existing issues", and get a |
| 229 | + populated arc; confirm a locked step still refuses regeneration. |
| 230 | +4. Migration: on a copy of `data/` with the pre-change stage prompts, run the |
| 231 | + migration and confirm only unmodified prompts are upgraded; customized ones are |
| 232 | + left intact and flagged by the drift warning. |
0 commit comments