Discuss a sustainable translation/i18n infrastructure for long-term maintainability

[sanchuanhehe]

The ongoing Japanese translation PR (#256), together with earlier work on the Chinese translation, highlights a maintenance question that may become more important as more locales arrive.

From my experience helping with the Chinese translation, I think the project may soon run into a few predictable challenges:

• reviewer scarcity: each locale needs people who understand both the language and the source material
• very large mirrored PRs: full-directory translation updates are hard to review incrementally
• source drift: when English chapters change, it is difficult to see which localized files are now stale
• limited CI: today we do not seem to have build/link/freshness checks for localized content, so regressions and divergence are easy to miss
• weak translation tooling: there is no shared glossary, translation memory, segment-level workflow, or status dashboard for translators/reviewers

Would it make sense to discuss moving toward a docs toolchain that is more translation-aware? One possible direction could be:

1. Sphinx + MyST as the authoring/build layer
2. gettext-based extraction for translatable strings
3. sphinx-intl or a similar gettext workflow for locale sync
4. optional collaboration tooling such as Weblate / Pontoon / Transifex / similar, plus Vale, link checking, spell checking, and preview builds in CI

I am not suggesting that we must do a full migration immediately. An incremental approach might be enough:

• start by adding CI for docs build, link checking, and localized previews
• experiment with gettext extraction on part of the book
• evaluate whether MyST + Sphinx would reduce maintenance cost compared with the current mirrored Markdown tree
• document glossary, style-guide, and reviewer workflow for translators

This issue is intentionally separate from #257, which is about the content and editorial direction of the guidebook itself. This one is about the translation/localization workflow and the long-term maintainability of the project.

If maintainers think this is worthwhile, I would be happy to help bootstrap and maintain the infrastructure work.

[semioticrobotic]

Thanks for this, @sanchuanhehe. I made this a community discussion instead of an issue, because I believe we'll all need some time to talk and think about this one.

I certainly agree that automation and additional CI tooling could help with the work of maintaining translations. I'm more concerned about adding toolchain complexity and what that means for our overall ability to maintain the Open Source Way project. I also must confess that I know very little about this area, so would want to ensure that other maintainers and community members could have an opportunity to review what you're proposing and weigh in.

[sanchuanhehe]

Thanks for moving this to a discussion, @semioticrobotic! I completely understand your concern about toolchain complexity—keeping the project easy to maintain is definitely the top priority.

To help everyone visualize how this could work, I'd be happy to build a small demo (Proof of Concept) in my own fork. I will make sure to minimize the setup complexity as much as possible. This way, the community can evaluate whether the benefits are worth it without any upfront commitment. I'll wait to hear what others think while I put the demo together!

[sanchuanhehe]

I have a working proof-of-concept fork for the sustainable i18n/docs workflow discussed here:

• Fork main branch: https://github.com/sanchuanhehe/guidebook/tree/main
• Feature branch with the full migration history: https://github.com/sanchuanhehe/guidebook/tree/feat-sustainable-translation/i18n-infrastructure
• Deployed multilingual preview: https://sanchuanhehe.github.io/guidebook/
• Updated contributor workflow docs: https://github.com/sanchuanhehe/guidebook/blob/main/CONTRIBUTING.md#building-docs-and-maintaining-translations
• Docs workflow definition: https://github.com/sanchuanhehe/guidebook/actions/workflows/docs.yml
• Latest successful docs workflow run: https://github.com/sanchuanhehe/guidebook/actions/runs/25102894306
• PDF artifact (guidebook-pdf) from the latest run: https://github.com/sanchuanhehe/guidebook/actions/runs/25102894306/artifacts/6704582493

What is included:

• uv-managed Sphinx/MyST documentation toolchain.
• gettext-based zh_CN translation catalogs with existing Chinese content migrated and semantic shift issues corrected.
• Section-level index.md files so navigation matches the GitBook-style document hierarchy more closely.
• sphinx-book-theme for a GitBook-like layout, with a language dropdown that can switch between English and Simplified Chinese builds.
• Markdownlint coverage for structural Markdown issues, including list consistency and invalid fragment links.
• Sphinx LaTeX/PDF output using XeLaTeX and CJK-capable fonts; the workflow uploads English and Simplified Chinese PDFs as artifacts.
• GitHub Actions build for English/Chinese HTML, catalog freshness checks, Markdownlint, PDF artifacts, advisory linkcheck, preview artifact upload, and GitHub Pages deployment.

How to try it locally:

uv sync
uv run --locked sphinx-build -b html . _build/html/en
uv run --locked sphinx-build -b html -D language=zh_CN . _build/html/zh_CN

To build PDF previews locally, install XeLaTeX/latexmk first, then run:

uv run --locked sphinx-build -M latexpdf . _build/pdf/en
uv run --locked sphinx-build -M latexpdf . _build/pdf/zh_CN -D language=zh_CN

To refresh translation catalogs after English source changes:

SOURCE_DATE_EPOCH=0 uv run --locked sphinx-build -b gettext . _build/gettext
uv run --locked sphinx-intl update -p _build/gettext -l zh_CN --locale-dir locales
uv run --locked python tools/i18n/normalize_po_files.py
uv run --locked python tools/i18n/catalog_status.py

To run Markdownlint locally:

npx --yes markdownlint-cli2@0.18.1

What the fork pipeline currently shows:

1. Syncs the locked uv environment.
2. Runs Markdownlint.
3. Builds English HTML.
4. Extracts gettext catalogs and checks committed zh_CN catalogs are current.
5. Builds zh_CN HTML and reports catalog coverage.
6. Builds English and Simplified Chinese PDFs in a dedicated job and uploads them as the guidebook-pdf artifact.
7. Prepares a Pages artifact with /en/, /zh_CN/, .nojekyll, and a root redirect to /en/.
8. Uploads preview artifacts and deploys to GitHub Pages.
9. Runs external link checking as an advisory job with an explicit summary/warning step, so noisy third-party links do not show up as a failed job or block deployment.

The deployed preview is meant as a reviewable fork-based prototype before proposing any upstream PR changes.

[sanchuanhehe]

@semioticrobotic

[semioticrobotic]

This is really fascinating work, @sanchuanhehe. Thank again for putting it together. And thanks, too, for your patience while I took a few days to look through and comprehend what you're proposing here. As I said: very little here is within my area of expertise so I just needed a bit of extra time to sort it all out.

As I see it, the scope of what you're proposing here is even greater than translations alone. You're potentially proposing a new production toolchain for the entire Open Source Way guidebook—albeit one that is "translation-aware" or "translation-first" in ways that our current tooling currently isn't. But what I see here is at its heart a sphynx-based system for producing a multi-lingual version guidebook from Markdown source (a real affirmation of our decision to switch to Markdown-by-default last year!). One valuable knock-on effect of what you're proposing here is the ability to produce PDF versions of the guidebook, something we once did with our now-deprecated toolchain, and something that I had not quite figured out how to replace with the current system. This addresses that in a (seemingly, to my naive eyes) straightforward and simple way.

So I am inclined to endorse the approach and would propose doing so with the following adjustments to our current project and community organization:

• We create a new project, called production, in the organization. It is the home of the production toolchain you have proposed, and we manage that toolchain there.
• We create a new team, Producers, which has Maintain access to that repository and is tasked with maintaining the guidebook's production.
• We maintain for now the current current GitBook-based guide and reposition it as the "bleeding edge" version of the guide—sort of like our community's version of "Debian testing." It reflects the ongoing editorial work of writing the guide, and gets managed and built from the guidebook repository, which contains all the editorial source files.
• When it's time to cut a new version of the guidebook, we create the latest "production version" of the guidebook—that is, activate the tooling in production to build the latest "versioned edition" of the guidebook (pulling source from guidebook) in multiple formats (HTML, PDF, ePub, etc.), and then link to those versions from the project home page (maintained in website) as the "latest stable version" of the Open Source Way guidebook.

What do you think of this, @sanchuanhehe?

I am keen to hear what other editors and maintainers are also thinking of these changes/additions.

[sanchuanhehe]

Thanks again for the thoughtful and detailed response — and for taking the time to really dig into the PoC. I'm glad the PDF angle resonates with you. Restoring the ability to produce high-quality PDFs like the project once had was something I carefully considered when evaluating toolchain options alongside the translation goals, so I'm especially pleased that this turned out to be a good fit.
I think the organizational structure you've outlined is excellent:

• A separate production repo keeps the toolchain self-contained, so contributors working on translations or build improvements don't need to touch the editorial source, and vice versa.
• The Producers team gives clear ownership and reduces bus-factor risk.
• Keeping GitBook as the "bleeding edge" is a smart way to avoid disrupting ongoing editorial work while we validate the new pipeline in parallel.

That said, there is one consideration worth flagging. The MyST setup in the PoC introduces index.md files into the guidebook source tree for structural management, along with CI/CD checks (Markdownlint, link checking, etc.) on the documentation. Keeping these in a single repository makes CI/CD integration more straightforward — PR authors can see all relevant checks in one place. If the toolchain and editorial source live in separate repos, PR checks become less intuitive (the action status lives in a different repository), though there are certainly ways to work around that. Ultimately, this is your call.
If you decide to move ahead with the separate production repo, the PoC I shared is ready to serve as a starting point. I'd be happy to help set up the initial repository structure and CI, and to join the Producers team to maintain the toolchain going forward.
Let me know what the next step is — happy to adapt based on whatever the other editors and maintainers think.

[sanchuanhehe]

Thanks again for the thoughtful and detailed response — and for taking the time to really dig into the PoC. I'm glad the PDF angle resonates with you. Restoring the ability to produce high-quality PDFs like the project once had was something I carefully considered when evaluating toolchain options alongside the translation goals, so I'm especially pleased that this turned out to be a good fit. I think the organizational structure you've outlined is excellent:

• A separate production repo keeps the toolchain self-contained, so contributors working on translations or build improvements don't need to touch the editorial source, and vice versa.
• The Producers team gives clear ownership and reduces bus-factor risk.
• Keeping GitBook as the "bleeding edge" is a smart way to avoid disrupting ongoing editorial work while we validate the new pipeline in parallel.

That said, there is one consideration worth flagging. The MyST setup in the PoC introduces index.md files into the guidebook source tree for structural management, along with CI/CD checks (Markdownlint, link checking, etc.) on the documentation. Keeping these in a single repository makes CI/CD integration more straightforward — PR authors can see all relevant checks in one place. If the toolchain and editorial source live in separate repos, PR checks become less intuitive (the action status lives in a different repository), though there are certainly ways to work around that. Ultimately, this is your call. If you decide to move ahead with the separate production repo, the PoC I shared is ready to serve as a starting point. I'd be happy to help set up the initial repository structure and CI, and to join the Producers team to maintain the toolchain going forward. Let me know what the next step is — happy to adapt based on whatever the other editors and maintainers think.

@semioticrobotic

[semioticrobotic]

Thanks, @sanchuanhehe! I will review when I have a moment. (You don't need to worry about explicitly tagging me. As a maintainer, I've opted into notifications for all activity in/on this project.)

[kkomazaw]

Thank you very much for raising this discussion. Whenever the infrastructure direction is settled, I'd be glad to contribute on the Japanese locale side—whether that means migrating existing translated content into .po catalogs, validating the pipeline against a ja locale, or helping document the translator workflow. Happy to follow the community's lead on timing and scope.

[semioticrobotic]

Hi, all. Sorry to take so long on this. Work has been busy.

All in all, I believe I am in favor of this new arrangement (a new production repository and corresponding team), and I can work to implement it. Of course, @sanchuanhehe and @kkomazaw, I will be very glad to have you on board.

I do want to address this, just to ensure I am getting it right:

The MyST setup in the PoC introduces index.md files into the guidebook source tree for structural management, along with CI/CD checks (Markdownlint, link checking, etc.) on the documentation. Keeping these in a single repository makes CI/CD integration more straightforward — PR authors can see all relevant checks in one place. If the toolchain and editorial source live in separate repos, PR checks become less intuitive (the action status lives in a different repository), though there are certainly ways to work around that. Ultimately, this is your call.

I do see your point here about integration, @sanchuanhehe, and certainly appreciate it. At the moment, I believe that in order to keep the editorial and production workflows separate, I would prefer to keep their respective tooling within the confines of their respective repositories—at least until we have piloted this and believe it is workable. At that point, if we decide it increases efficiency, we can "shift left" with our production tooling and integrate some linting and other production-related automations into the editorial repository.

How does this strike you? Does it create undue or unnecessary burdens for you and the soon-to-be production team? I do not mean to hamstring you right from the start, so please let me know what you think.

[sanchuanhehe]

Thanks for the thoughtful response — and no need to apologize, I completely understand.

I'm on board with keeping editorial and production tooling in separate repositories. That's a reasonable approach, and I agree it makes sense to pilot the separation first. If we find later that shifting some tooling left into the editorial repo would improve the workflow, we can revisit it then.

No undue burden on my end. The only thing I'd flag is that as I set up the production repo and CI, I may need to submit a few small patches to the editorial repo as well — the current Markdown source has some formatting inconsistencies that MyST is more strict about. These would be minor fixes, and I'll keep them focused.

When you're ready, please go ahead and create the production repo — I'm eager to get started. Looking forward to working on this together!

[semioticrobotic]

Thanks, @sanchuanhehe. A blank repository is now ready. Additionally, I have created the Producers team and invited both you and @kkomazaw to join as inaugural members. 😄 Please let me know if you encounter any issues getting set up. When you are ready, let's please ensure that we have all the principal documentation in place (e.g., our license, your contribution instructions in CONTRIBUTING.md, etc.). Thank you again for your dedication to the project.

[sanchuanhehe]

@semioticrobotic Following up on our earlier conversation — the production toolchain is now set up and running. Here is the current status:

Production repository: https://github.com/theopensourceway/production

• The guidebook is tracked as a git submodule, keeping editorial and production concerns separate as discussed
• Sphinx + MyST + sphinx-book-theme build configuration with uv for dependency management
• CI workflow: multilingual HTML builds, PDF generation (XeLaTeX + CJK fonts), advisory linkcheck, GitHub Pages deployment
• Language switcher for en/zh_CN in the sidebar

Deployed preview: https://theopensourceway.github.io/production/

• /en/ — English HTML
• /zh_CN/ — Simplified Chinese HTML
• PDF artifacts available from CI runs

Pull request to the editorial repo: #261

• Adds Sphinx navigation index.md files (toctree directives for chapter structure)
• Adds locales/zh_CN/LC_MESSAGES/*.po gettext catalogs with full Chinese translation
• Includes content formatting fixes for MyST compatibility
• These files are needed by the production repo to build from the guidebook submodule

Current architecture:

• theopensourceway/guidebook — editorial source, translations (via PR Add Sphinx navigation, zh_CN translations, and content formatting fixes #261), and Sphinx navigation
• theopensourceway/production — build toolchain, CI, theme customization, PDF/HTML generation
• The production repo runs prepare-source.sh to merge guidebook submodule content with navigation files; once Add Sphinx navigation, zh_CN translations, and content formatting fixes #261 is merged, the staging step can be simplified

What I would like to ask:

• Could you review the production repo and PR Add Sphinx navigation, zh_CN translations, and content formatting fixes #261 when you have a chance? The PR is intentionally scoped to only add files the production toolchain needs — no changes to existing editorial content or workflows
• Any feedback on the separation between editorial and production repos is welcome
• I am happy to walk through the build process or adjust anything based on your review

[sanchuanhehe]

Thanks for reviewing! Great catch on the GitHub link — I agree it should point to the guidebook issue tracker rather than production. I'm currently going through and cataloging a few more of these inconsistencies (formatting, cross-references, etc.) and will file issues for them in the production repo so we can track them properly.
As for l10n/cn — it will be deprecated, but I'd suggest keeping it in place for now as a transition period. The gettext catalogs in locales/zh_CN/ are now the source of truth for translations, but removing l10n/cn right away might confuse existing translators who are familiar with that layout. We can drop it once everyone has had time to adjust to the new workflow.

[sanchuanhehe]

I've filed theopensourceway/production#2 for the GitHub link issue.

[semioticrobotic]

As for l10n/cn — it will be deprecated, but I'd suggest keeping it in place for now as a transition period. The gettext catalogs in locales/zh_CN/ are now the source of truth for translations, but removing l10n/cn right away might confuse existing translators who are familiar with that layout. We can drop it once everyone has had time to adjust to the new workflow.

Understood! Thanks.

[semioticrobotic]

@sanchuanhehe, once we have resolved the recent build issues and overcome our immediate growing pains, let's please discuss the best ways to surface and share the new PDF-formatted production versions of the guidebooks. I am working on an update to the project home page and would like to feature them there.

[sanchuanhehe]

Sounds good! I'll spend the next few days working through all the open issues under https://github.com/theopensourceway/production/issues — these are all the issues I've been able to find through manual review (most of them are CJK and Chinese-related). If I don't finish by the end of this month, I'll likely be busy starting in early June until around the 9th or 10th. Hopefully I can get through most of them in the next couple of days.