Non-linear generation: backport across pipeline stages & Story Builder steps

[atomantic]

Summary

Makes both the Create Series Pipeline and the Story Builder non-linear, so a creator who authored work out of order (e.g. started a series as a drafted comic book) can backfill the prior steps to evaluate whether the story is complete.

Pipeline — per-stage source picker. Each text stage (idea → prose → comicScript → teleplay) was forward-only: it could only be generated from its conventional upstream. Now each stage's Generate button shows a "Generate from:" picker listing every other stage that has content, pre-checked with the conventional forward source. The server renders the chosen stages through a generic {{#sourceMaterials}} block instead of a hardcoded single-source slot, so prose can be written from a comic script, the beat sheet reverse-engineered from finished prose, etc. Omitting sourceStageIds preserves exact prior behavior (auto-run and legacy callers fall back to the forward source).

Story Builder — start anywhere + backfill. Navigation is now advisory rather than gated: setCurrentStep accepts any valid step, and the step rail flags an unlocked/stale upstream with a warning icon instead of disabling it. Lock enforcement stays at the generators (a locked arc/readerMap record still refuses regeneration). The idea and plotArc generators take an optional fromDownstream flag that synthesizes the step from the series' existing issue content (richest stage per issue: comicScript → teleplay → prose → idea). plotArc routes through a new arcPlanner.generateArcFromSource that reuses the importer's importer-arc-extract prompt and returns the same {arc, seasons} shape, committed through the identical commitSeasonsWithRemap path; idea passes the corpus as {{sourceMaterial}} in a new optional prompt block. A "Backfill from existing issues" button appears on the idea + plot-arc steps whenever any issue has content.

Compatibility. Two hash-gated prompt migrations ship the updated templates to existing installs (054 for the four pipeline stage prompts, 055 for the story-builder idea prompt); customized prompts are left intact and warned about. No sync/schema-version payload changes (Story Builder sessions are local-only; pipeline issue shape is unchanged — only generation inputs change).

Test plan

• cd server && npx vitest run for the touched suites — 294 tests pass: textStages (incl. new backport cases), pipeline route (new sourceStageIds schema), storyBuilder service (new fromDownstream backfill cases + relaxed navigation), storyBuilder route, arcPlanner, and both migration suites (054/055, each verifying the accepted-old hash matches git-HEAD and that customized prompts are skipped).
• cd client && npx eslint on TextStagePanel.jsx + StoryBuilder.jsx — clean.
• Manual (npm run dev): open an issue that has only a comic script → the Prose stage's "Generate from:" lists Comic Script (Idea absent); generate produces prose from the script. In Story Builder, a session whose series has drafted issues but no arc → enter Plot Arc out of order, click "Backfill from existing issues", get a populated arc; a locked step still refuses regeneration.

[Copilot]

Pull request overview

Makes the Create Series Pipeline and Story Builder non-linear so creators can backfill earlier steps from later ones. Pipeline text stages gain a multi-select source picker (any populated stage can feed any other) rendered via a generic {{#sourceMaterials}} block, and Story Builder relaxes step gating to advisory while letting the idea and plotArc generators synthesize from existing issue content via the importer's arc-extract prompt.

Changes:

• Pipeline: per-stage source picker (client + server sourceStageIds) with hash-gated migration 054 rewriting four prompt templates to a source-agnostic block.
• Story Builder: setCurrentStep becomes advisory; generateStep({fromDownstream}) adds backfill paths for idea (new optional {{sourceMaterial}} block, migration 055) and plotArc (new arcPlanner.generateArcFromSource reusing importer-arc-extract).
• Drift table in setup-data.js plus old migrations 003/004/013/025/027 updated to accept the new hashes and target the post-054-fence shipped versions.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
server/services/pipeline/textStages.js	Adds sourceMaterials context + resolveSourceStageIds, exports stageContentOf.
server/services/pipeline/textStages.test.js	New backport / default-source / drop-invalid cases.
server/services/pipeline/arcPlanner.js	New generateArcFromSource using importer-arc-extract; adds synopsis to season outline shape.
server/routes/pipeline.js	Adds sourceStageIds enum array to generate schema.
server/routes/pipeline.test.js	Verifies accept/reject of sourceStageIds.
server/services/storyBuilder.js	Removes lock-gate from setCurrentStep; adds collectIssueSourceText + fromDownstream paths for idea/plotArc.
server/services/storyBuilder.test.js	New backfill tests + relaxed-nav test.
server/lib/validation.js	Adds fromDownstream to story step generate schema.
client/src/services/apiPipeline.js	Exports PIPELINE_DEFAULT_FORWARD_SOURCE.
client/src/components/pipeline/stages/TextStagePanel.jsx	Renders "Generate from:" chip picker, sends sourceStageIds.
client/src/pages/StoryBuilder.jsx	Free navigation with warning icons; adds Backfill button on idea + plotArc.
data.reference/prompts/stages/pipeline-{idea-expansion,prose,comic-script,teleplay}.md	Replace hardcoded slot with {{#sourceMaterials}} block + injection-guard fences.
data.reference/prompts/stages/story-builder-idea-expand.md	Adds optional {{sourceMaterial}} backfill block.
scripts/migrations/054-source-agnostic-stage-prompts.{js,test.js}	New hash-gated migration for the four pipeline prompts.
scripts/migrations/055-story-builder-idea-backfill-prompt.{js,test.js}	New hash-gated migration for the idea-expand prompt.
scripts/migrations/003,004,013,025,027-*.js	Extend ACCEPTED_OLD_MD5 chains + retarget NEW_SHIPPED_MD5 to post-054-fence hashes.
scripts/setup-data.js	Drift table updated with the new old/new prompt hashes.
docs/plans/2026-05-30-non-linear-stage-generation.md	Planning doc.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.