-
Notifications
You must be signed in to change notification settings - Fork 327
Experiment #127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DmitrievDmitriyA
wants to merge
16
commits into
main
Choose a base branch
from
experiment
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Experiment #127
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
c08f509
Add mapgen: DSL-to-draw.io generator prototype (tools/mapgen)
DmitrievDmitriyA 6b90732
Add mapcheck: CI validation for the roadmap maps (box-fit, overlaps, …
DmitrievDmitriyA 4962c44
mapgen: bilateral spine with per-language computed hub-x
DmitrievDmitriyA e16e2bf
mapgen: center node, vertical centering, local packing, side-aware hi…
DmitrievDmitriyA bb63d09
Updated TODO.md
DmitrievDmitriyA 470d989
Generate the roadmap maps from a language-neutral mapgen source (word…
DmitrievDmitriyA 8f725f7
mapgen: place hints by explicit angle/dist; restructure source Hints …
DmitrievDmitriyA ea5b8c0
mapgen: move extract/remap into bootstrap/; document DSL angle/dist e…
DmitrievDmitriyA 5b9fd1c
Updates to the dsl and to TODO.md
DmitrievDmitriyA fcd3a3c
mapgen: migrate build engine PowerShell -> Python; hint arrow start s…
DmitrievDmitriyA 07735e7
Updated hints' position in dsl
DmitrievDmitriyA 45416c8
Updated hints' position in dsl
DmitrievDmitriyA eb9fc32
Updated the graphes after the migration
DmitrievDmitriyA 63f29a9
Updated cross-references
DmitrievDmitriyA 3cc3aa1
Added unit tests for the mapgen engine
DmitrievDmitriyA 8e7dae9
Fixed broken links
DmitrievDmitriyA File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| name: Check maps | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "*/Graph/roadmap.drawio.svg" | ||
| - "tools/mapgen/**" # DSL/tsv source — the map-vs-DSL check compares against it | ||
| - "tools/mapcheck/**" | ||
| - ".github/workflows/map-check.yml" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| mapcheck: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v7 | ||
| with: | ||
| fetch-depth: 0 # date check reads each map's last-commit date | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Unit-test the generator | ||
| run: python tools/mapgen/test_build.py # parser + geometry math; no Pillow/draw.io needed | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --quiet Pillow | ||
| sudo apt-get update -qq | ||
| # Liberation Sans is metric-compatible with Arial (what draw.io uses); | ||
| # Noto CJK covers the Chinese map. | ||
| sudo apt-get install -y -qq fonts-liberation fonts-noto-cjk | ||
|
|
||
| - name: Validate maps | ||
| run: python tools/mapcheck/check.py | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,147 +1,98 @@ | ||
| # Notes for AI agents: working on the roadmap | ||
|
|
||
| This file helps an AI agent (or a human) get up to speed on the repository quickly: | ||
| how the map is structured, how to edit it safely, and how to port edits between languages. | ||
| This file helps an AI agent (or a human) get up to speed on the repository quickly: how the | ||
| map is produced, how to change it safely, and the conventions to keep. | ||
|
|
||
| This is **not a changelog** — for what changed and when, see the git history; the owner's | ||
| current tasks live in [TODO.md](TODO.md). | ||
|
|
||
| > **IMPORTANT: the repository owner also edits the map by hand, in draw.io.** | ||
| > Before any edit, re-extract the XML from `<Language>/Graph/roadmap.drawio.svg` — | ||
| > do not reuse XML extracted in a previous session, or you will overwrite their changes. | ||
| > Before writing the file, check `LastWriteTime`: if it's newer than your copy, re-read it. | ||
| > **IMPORTANT: the maps are GENERATED, not hand-edited.** | ||
| > Each `<Language>/Graph/roadmap.drawio.svg` is **build output** of the mapgen pipeline. | ||
| > Do not edit those files (or their embedded draw.io XML) directly — edit the source in | ||
| > [`tools/mapgen/roadmap/`](tools/mapgen/roadmap) and rebuild. CI (`mapcheck`) fails if a | ||
| > committed map no longer matches the source, so hand-edits will not pass. | ||
|
|
||
| ## Repository conventions | ||
|
|
||
| - **Commit messages: English only.** Article and map content is maintained in | ||
| Russian/English/Chinese, but the git history is English. | ||
| - `main` is the primary branch; work through separate branches and pull requests. | ||
| - Article file names and URLs must not change — they are linked from outside. | ||
| - Language versions: `Russian/` is the reference (edits are made here first), then ported to | ||
| `English/` (the root `README.md` is the English one) and `Chinese/`. There is no Spanish | ||
| version (`Spanish/`) yet; when one appears, everything is ported to it the same way | ||
| (see "Porting edits between languages"). | ||
|
|
||
| ## How the map is structured | ||
|
|
||
| - Each language's map is a single `<Language>/Graph/roadmap.drawio.svg` file: an SVG image with | ||
| the **embedded draw.io source** in the `content` attribute of the root `<svg>` tag | ||
| (uncompressed mxfile XML). | ||
| - **Do not hand-edit the SVG part** — only via draw.io (desktop/web/VS Code extension) or through | ||
| the cycle "extract XML → edit XML → rebuild via the draw.io CLI". | ||
| - Color legend (fillColor): Optional `#CCEEFF`, Junior `#96BB7C`, Middle `#FAD586`, | ||
| Senior `#BBCCEE`, pink hints `#FFD5E4`, gray stage boxes `#F5F5F5`, | ||
| header date box `#FFF5EB`. | ||
| - External links to the map go through `goto/*.html` redirects (drawio/miro/svg) with a | ||
| GoatCounter counter — file names and URLs must not change. | ||
|
|
||
| ## The edit pipeline via the CLI (Windows) | ||
|
|
||
| 1. Extract the XML from `.drawio.svg` — the `content` attribute of the root `<svg>`. | ||
| **Only via UTF-8** (see "encoding gotcha" below). | ||
| 2. Edit the XML (text edits, new `mxCell` nodes + edges). Give new nodes string ids with a | ||
| prefix (e.g. `n900`+) so they don't collide with the numeric ones. | ||
| 3. Export: `draw.io.exe -x -f svg -e -u --svg-theme auto -o out.drawio.svg in.drawio` | ||
| (draw.io CLI, `%LOCALAPPDATA%\Programs\draw.io\`). | ||
| 4. **Required**: the CLI produces a transparent background — restore it in the SVG tag: | ||
| `style="background: #ffffff; background-color: light-dark(#ffffff, #121212); color-scheme: light dark;"`. | ||
| 5. Verification: render the SVG in **headless Edge** | ||
| (`msedge --headless=new --screenshot=... --window-size=...` plus an HTML wrapper with | ||
| panes over the regions of interest) and run a bbox overlap check (0 overlaps is mandatory). | ||
| Do not trust the interactive browser render or the PNG export on this large SVG. | ||
|
|
||
| > **Encoding gotcha (important!):** the XML must be extracted via UTF-8: | ||
| > `[IO.File]::ReadAllText($svg, [Text.Encoding]::UTF8)` + `LoadXml`, NOT `Get-Content -Raw` | ||
| > (PowerShell 5 reads it with the system codepage and mangles Chinese into double mojibake). | ||
| > Save with a `StreamWriter` using `UTF8Encoding($false)` (no BOM). The English map is pure | ||
| > ASCII and is unaffected; the Chinese map is affected. | ||
|
|
||
| ## Editing via the draw.io MCP (alternative to the CLI) | ||
|
|
||
| This environment also exposes a **draw.io MCP server** (tools named `mcp__drawio__*`) that | ||
| drives a **live draw.io instance** directly — read the model by cell id, edit/add/delete | ||
| cells and edges, and export. It was added recently and hadn't been used before; prefer it | ||
| when an instance is connected, and fall back to the CLI pipeline above when it isn't. | ||
|
|
||
| - **Check connection first:** `list-documents`. An **empty result means no draw.io is | ||
| connected** — use the CLI pipeline instead. (You need the draw.io desktop app open with the | ||
| file; it registers itself with the MCP bridge.) | ||
| - **Read the model instead of grepping XML:** `list-pages`, then `list-paged-model` with a | ||
| `filter` (by `cell_type`, `ids`, `parent_ids`/`layer_ids`, or attribute/style expressions) | ||
| to locate cells. `get-selected-cell` reads whatever the owner has selected in the editor — | ||
| handy when they say "this box". | ||
| - **Edit by id:** `edit-cell` (x / y / **width** / height / style / text — e.g. widen a box | ||
| whose label overflows), `edit-edge` (waypoints, source/target, style), plus `add-rectangle`, | ||
| `add-cell-of-shape`, `add-edge`, `delete-cell-by-id`, `set-cell-parent`. | ||
| - **Render for verification:** `export-diagram` (`format=svg|png|xml`; set `background=#ffffff` | ||
| and `crop=true`, or `selection_only=true` for a quick region check). This replaces the | ||
| headless-Edge crop step. Note the export's `background` is a solid color, **not** the | ||
| `light-dark(...)` CSS the saved file needs — see the layout rules below. | ||
| - **Persistence:** MCP edits mutate the *live* document; the file on disk changes only when | ||
| draw.io saves. Don't treat a change as landed in `roadmap.drawio.svg` until it's saved — | ||
| verify `LastWriteTime` / re-extract the `content` attribute before relying on it. | ||
| - **Same rules apply:** the MCP changes *how* you edit, not *what* a valid edit is — the | ||
| header date, background style, stage-frame growth, and attachment-point normalization in | ||
| "Map layout rules" still hold. | ||
|
|
||
| ## Map layout rules | ||
|
|
||
| - **Update the date in the header on every map edit.** The header has a box | ||
| "Last updated: DD.MM.YYYY" (id `n1100`). The box deliberately has a light fill | ||
| `#FFF5EB` (not bare text): black text on a bare background is invisible in dark theme, because | ||
| the SVG uses a `light-dark()` background. Same rule for any new labels — a bare | ||
| `fontColor=#000000` without a light box is unreadable on a dark background. | ||
| - **Stage-frame title band ~37px.** For the "Stage N" frames (id 2–11) the top margin down to the | ||
| first node is ~37px (exception — frame 2, where it's 61px); the label sits in this band | ||
| (`verticalAlign=top`). Hence the limit on label font size: **~28px max** fits without shifting | ||
| nodes. For emphasis use `fontStyle=1` (bold) rather than a larger size — that doesn't change | ||
| the height. | ||
| - **Stage frames are ordinary vertices, not containers** (`container=0`, children are not nested). | ||
| If you shift content vertically, the frame has to be grown separately, otherwise nodes spill | ||
| out. After any shift, check for each frame: frame bottom ≥ bottom of the lowest element inside it. | ||
| - **Edge attachment points.** A legacy of the Miro import — some edges have arbitrary | ||
| `exitX/exitY`, `entryX/entryY`, so they leave one side of a node in a fan from different points. | ||
| Normalization: attach to the middle of the side facing the line (determine the side from the | ||
| first waypoint for the exit / the last for the entry); if the reference point is inside the | ||
| node's bbox, leave it alone (this preserves intentional downward exits); skip edges with the | ||
| `curved=1` style (hint callouts) entirely. | ||
| - **Known layout gotchas:** | ||
| - edges have explicit waypoint arrays (`<Array as="points">`) and vertical "buses" | ||
| (trunk lines at a fixed x); build new edges to the same pattern, routing horizontal segments | ||
| in the empty bands between rows of boxes (row pitch 60px, box height 30px); | ||
| - if you move a node explicitly and then run a general shift rule, the node moves **twice**; | ||
| exclude such nodes from the general pass or shift them after it; | ||
| - line breaks in node text are `
` in the `value` attribute. | ||
|
|
||
| ## Porting edits between languages | ||
|
|
||
| Edits are made in the Russian map first, then ported to the English and Chinese ones. | ||
|
|
||
| - **The vertical structure of the maps is identical across languages** — the stage frames and all | ||
| rows share y-coordinates. Only x/width differ (translation changes node widths). So vertical | ||
| shifts port one-to-one along the same y-thresholds. | ||
| - **gate and hub-x are language-dependent** (the map center is at a different x): only the right | ||
| (hard skills) half shifts, `x ≥ gate`; section headers ("C++ syntax", "OS", …) sit left of the | ||
| gate and don't shift. The "elbows" of the central trunk (at the level of the "Hard skills" node) | ||
| must not move during vertical shifts, or the map's central line breaks. | ||
| Current-state reference points: RU gate=2900 hub-x=2945.25; EN gate=3000 hub-x=3044.52; | ||
| ZH gate=2900 hub-x=2680.04. | ||
| - **Node ids in the upper part of the map (syntax, templates, libraries) are shared across | ||
| languages**, but they diverge lower down. Known RU→EN/ZH divergences: `242→241` (Dev practices), | ||
| `275→274` (Profilers), `277→276` (Industrial standards), `341→340` (Multithreading), | ||
| `379→378` (Security). Find a new node's parent by this map, not blindly by id. | ||
| - New nodes are copied whole from the Russian map and translated rigidly by | ||
| `delta = parent_position(target) − parent_position(RU)`; the text comes from the translation, | ||
| the width is fit to the translation; edges reuse the same waypoints + delta; remap the | ||
| `source/target` of divergent ids. Then normalize the attachment points. | ||
| - **Chinese nodes are wider** (the "中文(English)" format), so individual columns may need to | ||
| be shifted right so they don't overlap their neighbors; verify with the overlap check. | ||
| - **Articles** (the per-language `.md` content) are still maintained by hand, one copy per | ||
| language; any content change must be mirrored across `English/` `Russian/` `Chinese/` | ||
| (there is no `Spanish/` yet). **The map is different** — it has a single source (below), | ||
| so a structural change is made once and regenerated into every language. | ||
|
|
||
| ## How the map is produced | ||
|
|
||
| ``` | ||
| tools/mapgen/roadmap/structure.dsl + <lang>.tsv ──► build.py ──► <lang>.drawio.svg | ||
| (one language-neutral source) (per-lang text) (layout engine) (copied to <Language>/Graph/) | ||
| ``` | ||
|
|
||
| - **`structure.dsl`** fixes the *logical* map: the node tree (indentation = hierarchy), each | ||
| node's grade (colour) and maturity `stage`, the pink hint boxes (`angle`/`dist`/`arrow`), | ||
| the spine, and the stage frames. Language-neutral **word-ids** (`[standard-library-stl]`) | ||
| are the shared key. | ||
| - **`<lang>.tsv`** is `id <TAB> text` — only the words, per language. Chrome (title, legend, | ||
| About/date) text lives here too, keyed by role; its geometry is in `chrome.tsv`. | ||
| - **`build.py`** computes the *physical* layout (widths from text metrics, packing, routing, | ||
| frame growth) and renders via the draw.io CLI. Output is byte-for-byte reproducible from | ||
| the source. Full grammar and build/flags: [`tools/mapgen/README.md`](tools/mapgen/README.md). | ||
| - Each language's map is still a single `<Language>/Graph/roadmap.drawio.svg` — an SVG image | ||
| with the draw.io source embedded in the `content` attribute of the root `<svg>`. That's the | ||
| **output**; treat it as generated. | ||
|
|
||
| ## Editing the map | ||
|
|
||
| 1. Edit the source in `tools/mapgen/roadmap/`: | ||
| - **structure / grade / stage / hint placement** → `structure.dsl` | ||
| - **text** (a label, a hint's wording, the date, a legend caption) → the relevant | ||
| `<lang>.tsv` row(s). A structural add needs a new row in **all three** tsvs. | ||
| 2. Rebuild and copy over the live maps: | ||
| ```bash | ||
| pip install -r tools/mapgen/requirements.txt # first time (Pillow) | ||
| python tools/mapgen/build.py --dir tools/mapgen/roadmap --langs en,ru,zh | ||
| cp tools/mapgen/roadmap/en.drawio.svg English/Graph/roadmap.drawio.svg | ||
| cp tools/mapgen/roadmap/ru.drawio.svg Russian/Graph/roadmap.drawio.svg | ||
| cp tools/mapgen/roadmap/zh.drawio.svg Chinese/Graph/roadmap.drawio.svg | ||
| ``` | ||
| (needs the draw.io desktop CLI at `%LOCALAPPDATA%\Programs\draw.io\`; `--drawio-cli` to override.) | ||
| 3. Validate: `python tools/mapcheck/check.py --no-date` must report **0 hard errors**. | ||
| 4. Commit the changed source **and** the regenerated maps together. | ||
|
|
||
| Reverse-engineering an existing map back into the DSL, and onboarding a new language, are | ||
| one-off jobs handled by [`tools/mapgen/bootstrap/`](tools/mapgen/bootstrap) — not part of the | ||
| normal edit loop. | ||
|
|
||
| ## Conventions the generator follows (and you should preserve in the DSL) | ||
|
|
||
| - **Colour legend** (grades → fillColor): junior `#96BB7C`, middle `#FAD586`, senior | ||
| `#BBCCEE`, optional `#CCEEFF`; pink hints `#FFD5E4`, grey stage frames `#F5F5F5`, centre | ||
| node `#FFE5B9`, chrome body `#FFF5EB`. Set a node's grade in the DSL, not a colour. | ||
| - **The "Last updated" date** is a `<lang>.tsv` entry (role `last-updated`, | ||
| "Last updated: YYYY-MM-DD"). Bump it when you change a map — `mapcheck`'s date check warns | ||
| if it predates the map's last commit. Its light `#FFF5EB` fill is deliberate: black text on | ||
| a bare background is invisible in the map's dark theme. | ||
| - **Stage frames** grow to fit their `(section, stage)` group automatically; the "N step" | ||
| titles come from the `stage1`..`stage5` tsv keys. Just annotate a subtree root with `stage=N`. | ||
| - **Hints** are placed by explicit polar `angle`/`dist` from their target and a required | ||
| `arrow=` side; one value serves all three languages, so re-check `mapcheck` after moving one. | ||
| - Rows share `y` across languages (`PITCH=60`, box `H=30`); only x/width differ per language. | ||
| The generator handles this — you don't port vertical positions by hand any more. | ||
|
|
||
| ## Validation — `tools/mapcheck` | ||
|
|
||
| `check.py` runs in CI on every PR touching a map ([`.github/workflows/map-check.yml`](.github/workflows/map-check.yml)) | ||
| and scans all `<Language>/Graph/roadmap.drawio.svg`. Hard errors (fail CI): box-fits-text | ||
| (Latin), overlaps, and **map-vs-DSL** (a committed map that no longer matches | ||
| `tools/mapgen/roadmap` — ids, grades, `<lang>.tsv` text, or a missing translation). Warnings: | ||
| cross-language drift and a stale date. See [`tools/mapcheck/README.md`](tools/mapcheck/README.md). | ||
|
|
||
| ## Open questions | ||
|
|
||
| - **The fate of Miro.** The Miro boards (external) don't sync with draw.io and have fallen behind; | ||
| they are marked obsolete in the README of every language. Optional plan: check GoatCounter | ||
| traffic for `/goto/miro-*` vs `/goto/svg-*`; if traffic is low, put a "the map has moved" banner | ||
| on the boards and redirect `goto/miro` to the SVG viewer (without breaking links). | ||
| - **The owner's current tasks are in [TODO.md](TODO.md)**; no need to duplicate them here, | ||
| check it directly. | ||
| - **The fate of Miro.** The Miro boards (external) don't sync with draw.io and have fallen | ||
| behind; they are marked obsolete in every language's README. Optional plan: check | ||
| GoatCounter traffic for `/goto/miro-*` vs `/goto/svg-*`; if low, put a "the map has moved" | ||
| banner on the boards and redirect `goto/miro` to the SVG viewer (without breaking links). | ||
| - **The owner's current tasks are in [TODO.md](TODO.md)**; check it directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you "Description" with a short description what the workflow does