Generator extensions: user-defined output generators

[gennaroprota]

Splitting this out per the #1128 / #1196 discussion. Transformation extensions (scripts that mutate the corpus) shipped in #1196. Generator extensions are the separate capability: letting users add or customize an output generator without touching MrDocs core.

The concrete first approach is tracked in #1082 (format-agnostic Handlebars generator: a base generator parameterized by format, templates under share/mrdocs/addons/generator/<format>/, an escape map, supplemental addon directories; e.g. a Markdown generator); it shipped as data-driven generators in #1197. This issue is the umbrella for the capability, which now has two flavors: data-driven (template) generators (#1197) and script-driven generators (#1218).

Open questions (resolved by #1197 and #1218):

• Template-only generators (the Format-agnostic Handlebars generator extension #1082 flavor) vs script-driven generators (interpreter hooks).
• How a custom format registers and how output file extensions map to template directories.
• Escaping rules per format.
• Interaction with addon layering / supplemental directories.

Distinct from #1210 (richer transformation entry points), #1213, and #1215, which are all about transformation extensions, not generators.

[gennaroprota]

Generator extensions: script-driven generators (design proposal)

1. Goal

Let a user add an output generator written in Lua or JavaScript, with no C++ and no compilation. This complements the data-driven (template) generators that shipped in #1197.

2. Limitations of what already shipped

Templates, helpers, and transforms are all bound to MrDocs's orchestration: multipage writes one file per symbol, single-page writes one file. But nothing can:

• partition output into files on a different axis (one file per namespace, or per module), or
• emit a single artifact aggregated across all symbols (a search index, an SQLite database, a combined data file), or
• own the traversal order and decide what gets written where.

The only way to change the output's shape in the cases above is a generator that owns the emit loop. That is what this proposal specifies.

3. The proposal: a loop-owning script generator

The script exports a single entry point that owns the whole emit:

generate(corpus, output)

• corpus is the flat DOM view scripts already read (feat: support Lua and JavaScript extensions #1196).
• output is a small file-writing API the host provides, for example output.write(relativePath, contents).

The script traverses the corpus itself and writes whatever files it wants under the output directory. Escaping is the script's job (there is no escape map).

This is intentionally the opposite of a per-symbol render: the script gains full control of the output and gives up what MrDocs would otherwise do for it (parallel page rendering, path computation, tagfile generation, incremental output). The script reimplements only what it needs.

4. Design decisions

Location and discovery. A script generator lives at /generator//, but is marked distinctly from a data-driven one by a script: key in mrdocs-generator.yml naming the entry file.

A bounded file-writing API. The host exposes writing, not raw filesystem access: write(relativePath, contents) resolved under the configured output directory, rejecting paths that escape it. This keeps a generator from writing anywhere on disk ("sandbox").

Errors. An uncaught script error, or a missing generate, aborts the build with a diagnostic naming the generator and the script path.

5. Example

A search-index generator, which templates cannot produce because it is a single artifact aggregated across all symbols rather than one page each:

-- <addon>/generator/search-index/generator.lua
return function(corpus, output)
  local entries = {}
  for _, sym in ipairs(corpus.symbols) do
    local name = sym.name or ""
    if name ~= "" then
      entries[#entries + 1] =
        '{"name":"' .. name .. '","url":"' .. sym._id .. '.html"}'
    end
  end
  output.write("search-index.json", "[" .. table.concat(entries, ",") .. "]")
end

mrdocs --generator=search-index ...

6. Testing

• Unit: a temp addon with a script generator.

7. Alternatives considered

(a) A per-symbol render(symbol) -> string generator. Redundant with #1197 plus a one-line layout and a helper. Its only advantage would be removing two stub template files. Not worth a new mechanism.

(b) Native plugin loader. Compiled C++ generators loaded from a shared library. This has a different audience (compiled, performance-sensitive, or library-dependent generators) and looks like a heavier mechanism.

[gennaroprota]

Implemented in PR #1218.

Implementing it also fixed a latent bug in the #1196 Lua DOM bridge: domValue_push aborted on Undefined and SafeString values (for example, reading a symbol with no name), which would equally have crashed a Lua corpus transform.