Skip to content

Commit 8be0cfd

Browse files
committed
docs: parallelize man page generation with make -j
The man_pages Bazel action invoked 'make bazel-manpages' serially, running ~10,000 pandoc/nroff invocations one at a time (~177s on a 64-core host). Split the single make call into two phases: 'preprocess' (serial) generates the md/man*/*.md sources, then 'cat web' fan out pandoc/nroff in parallel with -j. They cannot share one -j invocation because cat/web read the md files that preprocess produces and there is no dependency edge forcing preprocess first. Running 'cat web' as a second invocation also re-parses the Makefile so its $(wildcard md/man*/*.md) picks up the freshly generated sources. Regenerating the full doc set drops from ~177s to ~6s (~30x). Signed-off-by: Matt Liberty <mliberty@precisioninno.com>
1 parent b506257 commit 8be0cfd

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

bazel/man_pages.bzl

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,15 @@ def _man_pages_impl(ctx):
1818
set -euo pipefail
1919
CAT_OUT="$PWD/{cat_out}"
2020
HTML_OUT="$PWD/{html_out}"
21-
make --no-print-directory -C docs -f Makefile bazel-manpages \\
21+
# Two phases: 'preprocess' (serial) generates the md/man*/*.md sources, then
22+
# 'cat web' fan out pandoc/nroff in parallel. They cannot share one -j make
23+
# invocation: cat/web read the md files preprocess produces, and a parallel
24+
# build has no dependency edge forcing preprocess to finish first. Running
25+
# 'cat web' as a second invocation also re-parses the Makefile so its
26+
# $(wildcard md/man*/*.md) picks up the freshly generated sources.
27+
make --no-print-directory -C docs -f Makefile preprocess \\
28+
CAT_ROOT_DIR="$CAT_OUT" HTML_ROOT_DIR="$HTML_OUT"
29+
make --no-print-directory -j"$(nproc)" -C docs -f Makefile cat web \\
2230
CAT_ROOT_DIR="$CAT_OUT" HTML_ROOT_DIR="$HTML_OUT"
2331
""".format(
2432
cat_out = cat_dir.path,

0 commit comments

Comments
 (0)