Skip to content

Commit df3ea48

Browse files
chore(docs): centralize open follow-ups and remove historical documentation
- add a single open-items tracker for active and optional backlog work - remove historical-only docs and update references to canonical docs - eliminate milestone/rollout wording (P1/P2/P7) across docs - simplify architecture doc to current-state pipeline only - replace deterministic-output migration wording with current compatibility guidance - normalize worker/check-modules doc wording to current behavior
1 parent 8a511ae commit df3ea48

17 files changed

Lines changed: 72 additions & 990 deletions

docs/CONTRIBUTING.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ For focused manual runs, point the pipeline at a small local wiki snapshot inste
7878

7979
For regression testing, prefer the curated fixtures (`node --run fixtures:generate`, `node --run test:fixtures`, `node --run golden:check`) over ad-hoc subsets.
8080

81-
## Release validation (Phase 4 rollout)
81+
## Release validation
8282

8383
As of September 2025, schema validation is part of the release gate. After you regenerate the website data, run:
8484

@@ -222,7 +222,8 @@ Golden artifacts are reference outputs stored in `fixtures/golden/` that serve a
222222
- [`docs/architecture.md`](architecture.md) – current vs. target pipeline topology.
223223
- [`docs/pipeline/orchestrator-cli-reference.md`](pipeline/orchestrator-cli-reference.md) – command reference for partial runs, diagnostics, and logs.
224224
- [`fixtures/README.md`](../fixtures/README.md) – curated dataset and validation troubleshooting.
225-
- [`docs/pipeline/shared-defs-scope.md`](pipeline/shared-defs-scope.md) – plan for consolidating shared JSON Schema `$defs` (task P1.6).
225+
- [`docs/pipeline/orchestrator-cli-reference.md`](pipeline/orchestrator-cli-reference.md) – current CLI and schema validation flow.
226+
- [`docs/open-items.md`](open-items.md) – central list of active cross-doc follow-up work.
226227

227228
### Troubleshooting
228229

docs/architecture.md

Lines changed: 3 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Pipeline Architecture
22

3-
Visibility into the automation that builds and publishes the third-party module catalogue helps contributors reason about changes and spot failure points early. This document summarizes the current canonical pipeline, the legacy state we migrated from, and the parts of the broader architecture that are still future-facing.
3+
Visibility into the automation that builds and publishes the third-party module catalogue helps contributors reason about changes and spot failure points early. This document summarizes the current canonical pipeline and the parts of the broader architecture that are still future-facing.
44

55
## Current State (March 2026)
66

@@ -68,71 +68,9 @@ The pipeline implements intelligent caching and skip logic to avoid redundant wo
6868

6969
---
7070

71-
## Legacy Workflow Snapshot (pre-September 2025)
71+
### Follow-Up Tracking
7272

73-
```mermaid
74-
flowchart TB
75-
subgraph Stage 1: Create Module List
76-
wiki[("MagicMirror wiki table")] --> createLegacy{{Create module list<br>Node.js}}
77-
createLegacy --> stage1Legacy["legacy stage-1 snapshot"]
78-
end
79-
80-
subgraph Stage 2: Update Repository Data
81-
stage1Legacy --> updateLegacy{{Update repository data<br>Node.js}}
82-
updateLegacy --> cacheLegacy[("gitHubData.json cache")]
83-
updateLegacy --> stage2Legacy["legacy stage-2 snapshot"]
84-
end
85-
86-
subgraph Stage 3: Get Modules
87-
stage2Legacy --> getLegacy{{Fetch repos<br>Python}}
88-
getLegacy --> clonesLegacy[("modules/<br>modules_temp/")]
89-
getLegacy --> stage3Legacy["legacy stage-3 snapshot"]
90-
end
91-
92-
subgraph Stage 4: Expand Module List
93-
stage3Legacy --> expandLegacy{{Enrich metadata<br>Node.js}}
94-
expandLegacy --> imagesLegacy[("website/images/")]
95-
expandLegacy --> stage4Legacy["legacy stage-4 snapshot"]
96-
end
97-
98-
subgraph Stage 5: Check Modules JS
99-
stage4Legacy --> checkjsLegacy{{Static checks<br>Node.js}}
100-
checkjsLegacy --> stage5Legacy["legacy Stage-5 snapshot"]
101-
end
102-
103-
subgraph Stage 6: Check Modules
104-
stage5Legacy --> checkLegacy{{Deep analysis<br>Python}}
105-
checkLegacy --> outputsLegacy[("modules.json<br>modules.min.json<br>stats.json<br>result.md")]
106-
end
107-
108-
note1[["Mixed runtime: Python + Node.js"]]
109-
note2[["No orchestrator: manual script execution"]]
110-
note3[["6 sequential stages"]]
111-
```
112-
113-
This legacy diagram captures the pre-orchestrator, mixed-runtime pipeline. Key issues that motivated the modernization:
114-
115-
- Mixed Python + Node.js runtime made maintenance difficult
116-
- No orchestrator: manual script execution
117-
- No incremental updates: full run required every time
118-
- OOM risk with 1300+ modules loaded into memory
119-
- 6 sequential stages with 6 intermediate JSON files
120-
121-
### Comparison: Legacy vs. Current Flow
122-
123-
| Aspect | Legacy (6 stages) | Current flow (Mar 2026) |
124-
| ------------------ | ------------------------- | -------------------------------------------------------------------------------------------------------- |
125-
| Runtime | Python + Node.js | Node.js with TypeScript-based deep checks |
126-
| Execution | Sequential manual scripts | Orchestrated 4-stage pipeline with in-process handoff |
127-
| Incremental | ❌ No | Partial: metadata cache + clone reuse |
128-
| Memory | Unbounded | Batch-/worker-bounded |
129-
| Intermediate files | 6 | none; only published outputs are written (`modules.json`, `modules.min.json`, `stats.json`, `result.md`) |
130-
131-
### Remaining Gaps
132-
133-
1. Reintegrate worker-compatible `moduleCache.json` handling under P7.6.
134-
2. Record before/after repeated-run performance metrics once cache writes are back in place.
135-
3. Keep the published contract (`modules.json`, `modules.min.json`, `stats.json`, `result.md`) stable while worker caching evolves.
73+
Open follow-up work is tracked centrally in [Open Items](open-items.md).
13674

13775
No persisted intermediate stage boundary remains. Stage handoffs are fully in-memory.
13876

docs/deterministic-outputs.md

Lines changed: 7 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -153,41 +153,16 @@ No hashing required - the module identifier itself is already unique and determi
153153
- **Filename Generation**: Instant string concatenation
154154
- **Overall**: No measurable impact on pipeline runtime
155155

156-
## Migration Notes
156+
## Compatibility Note
157157

158-
### Existing Screenshots
158+
Some older snapshots in the repository may still contain pre-standardized screenshot filenames (for example, names derived from source paths). The current canonical output uses the deterministic `<moduleName>---<maintainer>.<extension>` format.
159159

160-
When this feature was introduced, existing screenshots had filenames like:
160+
Downstream consumers should always read screenshot paths from `modules.json` rather than hard-coding file names.
161161

162-
```text
163-
MMM-Weather---example---screenshot.jpg
164-
```
165-
166-
After the change, new screenshots use:
167-
168-
```text
169-
MMM-Weather---example.jpg
170-
```
171-
172-
**Migration Strategy**:
173-
174-
1. Old screenshots remain in place (not deleted)
175-
2. New pipeline runs generate hash-based filenames
176-
3. Gradual cleanup of old files as modules are re-processed
177-
178-
**Breaking Change**: Modules linking directly to screenshot URLs will need updates. This is considered acceptable because:
179-
180-
- Screenshot URLs should come from `modules.json`, not hard-coded
181-
- The migration improves long-term maintainability
182-
- Old files persist during transition
183-
184-
## Related Tasks
185-
186-
- **P3.4**: Ensure deterministic outputs (sorted keys, simple deterministic image names) ✅ Completed Nov 2025
162+
## Status
187163

188-
## Future Improvements
164+
Deterministic output safeguards (sorted keys and deterministic image naming) are part of the current pipeline behavior.
189165

190-
Potential enhancements (not currently planned):
166+
## Follow-Up Tracking
191167

192-
1. **Canonical JSON**: Use RFC 8785 canonical JSON for cryptographic verification
193-
2. **Reproducible Timestamps**: Strip or normalize timestamps in metadata
168+
Potential deterministic-output enhancements are tracked centrally in [Open Items](open-items.md) under "Backlog (Optional)".

0 commit comments

Comments
 (0)