chore(pipeline): make full-refresh-parallel the canonical default

- Switch `npm run all` and orchestrator default to `full-refresh-parallel`
- Pin legacy shortcut scripts (getModules, expandModuleList, checkModules,
  collectMetadata) explicitly to `full-refresh` to avoid --only resolution
  regression (stage IDs are validated against the selected pipeline)
- Finalize D1-D4 architecture decisions in worker-pool-design.md
- Add P7.5/P7.6 execution checklist (C1 inventory, C2 done)
- Update roadmap, orchestrator CLI reference, and related docs

Author: KristjanESPERANTO

docs/pipeline-refactor-roadmap.md

@@ -122,23 +122,22 @@ Measure and visualize pipeline performance.
 
 ## Next Concrete Steps
 
-See [worker-pool-design.md](pipeline/worker-pool-design.md) for detailed implementation plan.
+See [worker-pool-design.md](pipeline/worker-pool-design.md) for architecture details and
+[p7-cleanup-incremental-checklist.md](pipeline/p7-cleanup-incremental-checklist.md) for execution tracking.
 
-**Current focus: P7.3** — Implement worker pool orchestration
+**Current focus: P7.5** — Cleanup old stage scripts
 
-- Create child process spawning for parallel workers
-- Implement batch distribution and work stealing algorithm
-- Add progress tracking and health monitoring
-- Test with 4 workers on full module set
+- Inventory legacy stage entry points and consumers (`full-refresh`, npm scripts, docs references)
+- Switch canonical full refresh execution to `full-refresh-parallel`
+- Remove/retire obsolete stage wrappers and stale intermediate artifact dependencies
+- Run regression checks (`lint`, fixtures, schema checks, golden checks) after cleanup
 
-**Completed: P7.2** ✅
+**Next: P7.6** — Incremental mode integration in worker architecture
 
-Successfully implemented and tested the single-worker prototype:
-
-- Merged Stage 3+4+5 logic into `processModule()` function
-- Tested with 20 modules (100% success rate)
-- Average processing time: ~400ms per module (when cached)
-- Code location: `pipeline/workers/`
+- Integrate module-cache hit/miss/prune logic into parallel worker flow
+- Use deterministic cache merge/write strategy to avoid worker write contention
+- Add tests for cache behavior and validate skip rates on repeated runs
+- Capture before/after timing and cache-hit metrics for readiness to start P8
 
 ---
 

docs/pipeline/orchestrator-cli-reference.md

@@ -6,7 +6,7 @@ Task **P1.2** delivered a lightweight Node.js command-line interface that reads
 
 ## Key capabilities
 
-- Provide a single entry point (e.g. `node --run pipeline -- pipeline run full-refresh`) that replaces the existing ad-hoc shell scripts.
+- Provide a single entry point (e.g. `node --run pipeline -- pipeline run full-refresh-parallel`) that replaces the existing ad-hoc shell scripts.
 - Interpret `pipeline/stage-graph.json` at runtime to determine stage ordering, inputs/outputs, and side-effects.
 - Emit structured logs and progress indicators so maintainers can trace stage execution locally and in CI.
 - Persist run metadata (planned, skipped, succeeded, failed) to `.pipeline-runs/` so partial runs stay auditable.
@@ -31,7 +31,7 @@ Task **P1.2** delivered a lightweight Node.js command-line interface that reads
 1. **Command Surface** — Implemented with `commander`, exposing the `pipeline` root command and subcommands:
    - `pipeline list` — enumerate available pipelines/stages from the graph.
    - `pipeline describe <stage|pipeline>` — print detailed metadata for inspection.
-   - `pipeline run <pipelineId>` — execute stages sequentially (default: `full-refresh`).
+   - `pipeline run <pipelineId>` — execute stages sequentially (default: `full-refresh-parallel`).
    - `pipeline logs [runId|--latest]` — inspect structured run metadata saved to `.pipeline-runs/`.
      - `pipeline doctor` — check external prerequisites (Node.js version, Git availability, required env vars).
 

docs/pipeline/p7-cleanup-incremental-checklist.md

@@ -0,0 +1,90 @@
+# P7.5/P7.6 Execution Checklist
+
+This checklist turns roadmap items P7.5 and P7.6 into concrete implementation slices.
+
+Related docs:
+
+- [Pipeline roadmap](../pipeline-refactor-roadmap.md)
+- [Worker pool design](worker-pool-design.md)
+
+## P7.5 Cleanup Old Stage Scripts
+
+- [x] C1: Inventory and dependency map
+  - Completed on 2026-03-19.
+  - npm scripts still exposing legacy flow:
+    - `all`, `getModules`, `expandModuleList`, `checkModules`, `ownList` in [package.json](../../package.json).
+    - Legacy stage numbering/descriptions in `ntl.descriptions` in [package.json](../../package.json).
+    - Status: canonical script usage completed in C2; retire old aliases in C3.
+  - Stage graph still contains both legacy and parallel paths:
+    - Legacy pipeline `full-refresh` plus stages `get-modules`, `expand-module-list`, `check-modules` in [pipeline/stage-graph.json](../../pipeline/stage-graph.json).
+    - Parallel pipeline `full-refresh-parallel` and stage `parallel-processing` in [pipeline/stage-graph.json](../../pipeline/stage-graph.json).
+    - Status: `full-refresh-parallel` set as canonical in C2; remove/legacy-gate old stage chain in C3.
+  - Docs with legacy/default references identified:
+    - Contributor stage table and examples in [docs/CONTRIBUTING.md](../CONTRIBUTING.md).
+    - Sequential five-stage architecture narrative in [docs/architecture.md](../architecture.md).
+    - CLI default examples around `full-refresh` in [docs/pipeline/orchestrator-cli-reference.md](orchestrator-cli-reference.md).
+    - Full-refresh command example in [docs/pipeline/worker-pool-design.md](worker-pool-design.md).
+    - Fixture regeneration guidance using `node --run all` in [fixtures/README.md](../../fixtures/README.md).
+    - Stage-5 reference content in [docs/pipeline/check-modules-reference.md](check-modules-reference.md).
+    - Follow-up: normalize docs/commands in C5 to reflect the new canonical default.
+  - Tests and harness references:
+    - Check-group config unit test points to `scripts/check-modules` in [scripts/check-modules/**tests**/check-group-config.test.js](../../scripts/check-modules/__tests__/check-group-config.test.js).
+    - Compare harness workflow uses `checkModules:compare` in [.github/workflows/check-modules-compare.yaml](../../.github/workflows/check-modules-compare.yaml).
+    - Follow-up: keep harness/tests for parity validation during P7.6; reassess after worker analysis parity is complete.
+  - CI hooks snapshot:
+    - No workflow currently invokes legacy stage chain (`full-refresh` or `--only=get-modules|expand-module-list|check-modules`).
+    - Git hook only runs lint-staged in [.husky/\_/pre-commit](../../.husky/_/pre-commit).
+- [x] C2: Canonical pipeline switch
+  - Completed on 2026-03-19.
+  - `npm run all` now targets `full-refresh-parallel` in [package.json](../../package.json).
+  - `pipeline run` without explicit pipeline id now defaults to `full-refresh-parallel` in [scripts/orchestrator/index.js](../../scripts/orchestrator/index.js).
+  - Legacy stage-specific shortcuts (`collectMetadata`, `getModules`, `expandModuleList`, `checkModules`) are pinned to `full-refresh` as a temporary compatibility path in [package.json](../../package.json).
+  - Legacy `full-refresh` pipeline remains available as compatibility path in [pipeline/stage-graph.json](../../pipeline/stage-graph.json).
+- [ ] C3: Legacy script retirement
+  - Remove or clearly deprecate obsolete wrappers and code paths that are no longer part of canonical flow.
+  - Ensure no dead stage artifact dependencies remain in orchestrator execution path.
+- [ ] C4: Artifact contract cleanup
+  - Confirm expected outputs for parallel path and remove stale assumptions about intermediate stage files.
+  - Validate schema references for the artifacts that remain part of supported flows.
+- [ ] C5: Docs and command surface cleanup
+  - Update README/docs/npm script descriptions to match canonical flow.
+  - Ensure contributor instructions do not point to retired stage sequence.
+- [ ] C6: Validation pass
+  - Run: `npm run lint`
+  - Run: `npm run test:fixtures`
+  - Run: `npm run golden:check`
+  - Run: `npm run schemas:check`
+  - Run one full `full-refresh-parallel` execution and archive summary logs.
+
+## P7.6 Incremental Mode Integration
+
+- [ ] I1: Cache key contract
+  - Define worker-compatible cache key (module identity + repo freshness signal + analysis config).
+  - Include a cache schema/version field for safe future migrations.
+- [ ] I2: Read path integration
+  - Load cache at orchestrator start and provide cache context to workers.
+  - Preserve current behavior for cache miss and partial cache entries.
+- [ ] I3: Write path integration
+  - Aggregate worker cache updates in orchestrator and write once, deterministically.
+  - Prevent write races from child processes.
+- [ ] I4: Skip semantics and reporting
+  - Standardize `status=skipped` and `skippedReason=cached` handling.
+  - Ensure progress output and final summary report skipped/cached totals clearly.
+- [ ] I5: Invalidation and pruning
+  - Prune cache entries for removed modules.
+  - Invalidate entries when key inputs change (checks config/schema version).
+- [ ] I6: Test coverage
+  - Add unit tests for cache hit/miss/invalidation paths.
+  - Add integration test showing improved second-run skip rate.
+- [ ] I7: Runtime controls
+  - Support toggles for cache enable/disable in parallel mode (`--no-cache` and/or env var).
+  - Document default behavior for full vs incremental runs.
+- [ ] I8: Evidence and acceptance
+  - Record before/after performance and cache-hit metrics.
+  - Confirm output parity for representative module subsets.
+
+## Definition of Done
+
+- [ ] P7.5 done: Legacy stage flow is removed or explicitly legacy-only, and all docs/scripts reference the canonical flow.
+- [ ] P7.6 done: Incremental cache behavior is integrated into worker architecture with tests and measurable skip-rate benefit.
+- [ ] Ready for P8: No open P7 blockers remain in roadmap or worker design decision list.

docs/pipeline/worker-pool-design.md

@@ -517,16 +517,20 @@ PIPELINE_CACHE_ENABLED=true
 
 ---
 
-## Open Questions
+## Architecture Decisions (Mar 2026)
 
-1. **Cache location**: Single shared cache file or per-worker cache files?
-   - Recommendation: Single shared cache, locked during writes
+1. **Cache location**: Keep a single shared cache file at `website/data/moduleCache.json`.
 
-2. **Git operations**: Shared `modules/` directory or per-worker clone dirs?
-   - Recommendation: Shared `modules/` with file locking per module
+- Decision details: Workers return cache updates to orchestrator, and orchestrator performs deterministic cache writeback.
 
-3. **Image processing**: Run in worker or separate phase?
-   - Recommendation: In worker (already part of Stage 4)
+2. **Git operations**: Keep shared `modules/` and `modules_temp/` directories.
 
-4. **ESLint/ncu**: Include in worker or optional post-processing?
-   - Recommendation: Optional in worker, controlled by config
+- Decision details: Use per-module isolation/locking to avoid conflicts while preserving clone reuse.
+
+3. **Image processing**: Keep image extraction/resizing inside worker flow.
+
+- Decision details: Image handling remains part of the enrich phase to keep per-module processing self-contained.
+
+4. **ESLint/ncu**: Keep optional in worker, controlled by check-group configuration.
+
+- Decision details: Pipeline mode defaults are explicit and may differ between full-refresh and incremental-oriented runs.

package.json

@@ -28,11 +28,11 @@
   },
   "scripts": {
     "start": "ntl --size 50",
-    "all": "node scripts/orchestrator/index.js run full-refresh",
-    "collectMetadata": "node scripts/orchestrator/index.js run --only=collect-metadata",
-    "getModules": "node scripts/orchestrator/index.js run --only=get-modules",
-    "expandModuleList": "node scripts/orchestrator/index.js run --only=expand-module-list",
-    "checkModules": "node scripts/orchestrator/index.js run --only=check-modules",
+    "all": "node scripts/orchestrator/index.js run full-refresh-parallel",
+    "collectMetadata": "node scripts/orchestrator/index.js run full-refresh --only=collect-metadata",
+    "getModules": "node scripts/orchestrator/index.js run full-refresh --only=get-modules",
+    "expandModuleList": "node scripts/orchestrator/index.js run full-refresh --only=expand-module-list",
+    "checkModules": "node scripts/orchestrator/index.js run full-refresh --only=check-modules",
     "pipeline": "node scripts/orchestrator/index.js",
     "golden:check": "node scripts/golden-artifacts/index.js check",
     "golden:update": "node scripts/golden-artifacts/index.js update",
@@ -96,7 +96,7 @@
   },
   "ntl": {
     "descriptions": {
-      "all": "Run all scripts (1 till 6) on all modules. Requires a lot of time and storage space!",
+      "all": "Run the canonical full refresh pipeline (`full-refresh-parallel`) on all modules. Requires a lot of time and storage space!",
       "collectMetadata": "Script 1+2: Convert the official module list from the wiki into a json file and update the JSON file that collects the GitHub information of the modules.",
       "getModules": "Script 3: Get all modules with `git clone`. Requires a lot of time and storage space!",
       "expandModuleList": "Script 4: Extend the module list with information from the package.json. And get an image if one is available and the license is okay.",

scripts/orchestrator/index.js

@@ -431,7 +431,7 @@ export async function main(argv = process.argv) {
     .option("--json-logs", "Output logs in JSON format")
     .action(async (pipelineId, options) => {
       const graphPath = resolve(options.graph);
-      const selectedPipeline = pipelineId ?? "full-refresh";
+      const selectedPipeline = pipelineId ?? "full-refresh-parallel";
       const filters = {
         only: options.only,
         skip: options.skip