Skip to content

Commit 30ae277

Browse files
oharboeoharboe
committed
bazel: //:make-yosys-netlist + README determinism docs
yosys-check feeds bazel's netlist into the make flow and compares .odb at every stage; make-yosys-netlist is the inverse — it feeds make's netlist into the bazel flow via the documented bazel-orfs positional KEY=VAL hook (bazel-orfs README:354-368, wired through private/rules.bzl:185 so make sees SYNTH_NETLIST_FILES as a command-line override). Workflow on a failing `_test`: 1. Build the bazel _final target (bazel-natural baseline). 2. Snapshot the bazel-natural .odb chain to a temp dir before step 4 clobbers it. 3. cd flow && make clean_all + make finish DESIGN_CONFIG=… (make's full flow yields its own .odb chain). 4. bazelisk run :<n>_final -- SYNTH_NETLIST_FILES=<make-netlist> to re-run the bazel flow with make's netlist injected. 5. Print a 7-row × 3-column SHA table covering 1_synth.odb through 6_final.odb. All MATCH in the "bazel+make-netlist" column proves bazel-test OpenROAD ≡ tools/install OpenROAD on this design. flow/README.md adds three sections derived from the recent asap7 + sky130hd sweep workflow: a sweep-driver recipe, a triage decision tree for failing tests, and an "OpenROAD determinism across bazel and make" section that documents both wrappers, the underlying SYNTH_NETLIST_FILES hook (both make- and bazel-side), and the BLOCKS= caveat (sub-block abstracts aren't staged into the parent synth action; a separate bazel-orfs patch in this branch fixes that for the shared-block.mk case). Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
1 parent 88848d4 commit 30ae277

3 files changed

Lines changed: 336 additions & 0 deletions

File tree

‎BUILD.bazel‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,12 @@ sh_binary(
1313
name = "yosys-check",
1414
srcs = ["bazel/yosys-check.sh"],
1515
)
16+
17+
# Symmetric companion to //:yosys-check. Feeds make's pre-built yosys
18+
# netlist into a fresh bazel flow and SHA-compares every stage's .odb
19+
# against make's own. Use to determine if a failing bazel _test is a
20+
# yosys-environment false positive (all stages MATCH).
21+
sh_binary(
22+
name = "make-yosys-netlist",
23+
srcs = ["bazel/make-yosys-netlist.sh"],
24+
)

‎bazel/make-yosys-netlist.sh‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Symmetric companion to //:yosys-check. Feeds make's pre-built yosys
4+
# netlist into a fresh bazel run and SHA-compares the resulting .odb at
5+
# every stage against make's own .odb. When MATCH, the bazel-test
6+
# build of OpenROAD is bit-identical to tools/install OpenROAD given
7+
# the same starting netlist — a one-command proof that any bazel
8+
# `_test` QoR failure on this design is yosys-environment drift, not a
9+
# bazel-orfs / OpenROAD bug.
10+
#
11+
# Usage:
12+
# bazelisk run //:make-yosys-netlist //flow/designs/<plat>/<design>:<n>_test
13+
#
14+
# Output: a 7-row × 3-column SHA table covering 1_synth.odb through
15+
# 6_final.odb. Two comparison columns:
16+
# 1. bazel-natural vs make -> usually DIFFER (yosys-env drift)
17+
# 2. bazel-with-make-netlist vs make -> expected MATCH (the proof)
18+
19+
set -e -u -o pipefail
20+
21+
if [[ $# -ne 1 ]]; then
22+
echo "usage: bazelisk run //:make-yosys-netlist <test-label>" >&2
23+
echo " e.g. bazelisk run //:make-yosys-netlist //flow/designs/asap7/jpeg_lvt:jpeg_encoder_test" >&2
24+
exit 2
25+
fi
26+
27+
cd "${BUILD_WORKSPACE_DIRECTORY:?must be invoked via bazelisk run}"
28+
29+
LABEL="$1"
30+
31+
case "$LABEL" in
32+
//flow/designs/*/*:*_test) ;;
33+
*)
34+
echo "make-yosys-netlist: expected //flow/designs/<plat>/<design>:<name>_test, got $LABEL" >&2
35+
exit 2
36+
;;
37+
esac
38+
39+
PKG="${LABEL#//}"; PKG="${PKG%:*}"
40+
NAME="${LABEL##*:}"
41+
DESIGN_DIR="${PKG#flow/designs/}"
42+
PLAT="${DESIGN_DIR%%/*}"
43+
DESIGN="${DESIGN_DIR#*/}"
44+
BASE="${NAME%_test}"
45+
SYNTH_TARGET="//$PKG:${BASE}_synth"
46+
FINAL_TARGET="//$PKG:${BASE}_final"
47+
48+
# BLOCKS= designs don't pass clean bazel-vs-make comparisons because
49+
# bazel-orfs's hierarchical-block plumbing differs from make's
50+
# (sub-block .lef/.lib aren't staged into the parent synth action).
51+
# See aes-block. Refuse rather than emit a misleading DIFFER table.
52+
if grep -qE '^[[:space:]]*export[[:space:]]+BLOCKS[[:space:]]*[?:]?=' \
53+
"flow/designs/$PLAT/$DESIGN/config.mk"; then
54+
echo "make-yosys-netlist: $DESIGN sets BLOCKS=… (hierarchical design)." >&2
55+
echo " bazel-orfs's BLOCKS handling diverges from make's at the parent" >&2
56+
echo " synth step (sub-block abstracts not staged into the parent action)" >&2
57+
echo " so a per-stage .odb comparison won't be apples-to-apples." >&2
58+
echo " Skipping; this is a known bazel-orfs gap, not an OpenROAD-determinism" >&2
59+
echo " question." >&2
60+
exit 2
61+
fi
62+
63+
echo "==> Building $FINAL_TARGET (bazel-natural baseline; produces the full .odb chain)"
64+
bazelisk build "$FINAL_TARGET"
65+
66+
# Bazel-side results subdir glob (bazel uses DESIGN_NAME for the path).
67+
BAZEL_PLAT_DIR="bazel-bin/$PKG/results/$PLAT"
68+
BAZEL_NICK=$(basename "$(ls -d "$BAZEL_PLAT_DIR"/*/ 2>/dev/null | head -1)" 2>/dev/null)
69+
BAZEL_NICK="${BAZEL_NICK:-$DESIGN}"
70+
71+
# Make-side results subdir (make uses DESIGN_NICKNAME).
72+
MAKE_NICK=$(awk '
73+
/^[[:space:]]*export[[:space:]]+DESIGN_NICKNAME[[:space:]]*=/ {
74+
n = index($0, "="); v = substr($0, n + 1)
75+
gsub(/^[[:space:]]+|[[:space:]]+$/, "", v); print v; exit
76+
}
77+
' "flow/designs/$PLAT/$DESIGN/config.mk")
78+
MAKE_NICK="${MAKE_NICK:-$DESIGN}"
79+
echo "==> bazel subdir=$BAZEL_NICK make subdir=$MAKE_NICK"
80+
81+
BAZEL_NATURAL="bazel-bin/$PKG/results/$PLAT/$BAZEL_NICK/base"
82+
MAKE_RES="flow/results/$PLAT/$MAKE_NICK/base"
83+
84+
echo "==> Installing OpenROAD into tools/install/ (idempotent)"
85+
( cd tools/OpenROAD && bazelisk run //:install )
86+
87+
echo "==> Running full make flow (yields make's 1_2_yosys.v and full .odb chain)"
88+
( cd flow && make clean_all DESIGN_CONFIG="designs/$PLAT/$DESIGN/config.mk" ) || true
89+
( cd flow && make finish DESIGN_CONFIG="designs/$PLAT/$DESIGN/config.mk" ) || true
90+
91+
MAKE_NETLIST_RO="$MAKE_RES/1_2_yosys.v"
92+
if [[ ! -f "$MAKE_NETLIST_RO" ]]; then
93+
echo "make-yosys-netlist: make's 1_2_yosys.v not at $MAKE_NETLIST_RO" >&2
94+
echo " (the 'make finish' run above failed before yosys; nothing to feed bazel)" >&2
95+
exit 1
96+
fi
97+
98+
# synth_preamble.tcl copies SYNTH_NETLIST_FILES with `cp -p`; the
99+
# canonicalize step then the synth step each copy. The second cp
100+
# fails on a read-only dest, so stage to a writable temp.
101+
# Stage the bazel-natural .odb files into a temp dir BEFORE the second
102+
# bazel run overwrites them in bazel-bin/.
103+
MAKE_NETLIST="$(mktemp --suffix=-make-1_2_yosys.v)"
104+
NATURAL_STAGED="$(mktemp -d --suffix=-bazel-natural)"
105+
trap 'rm -f "$MAKE_NETLIST"; rm -rf "$NATURAL_STAGED"' EXIT
106+
cp --no-preserve=mode "$MAKE_NETLIST_RO" "$MAKE_NETLIST"
107+
echo "==> Snapshotting bazel-natural .odb to $NATURAL_STAGED"
108+
for f in 1_synth.odb 2_floorplan.odb 3_place.odb 4_cts.odb \
109+
5_1_grt.odb 5_route.odb 6_final.odb; do
110+
[[ -f "$BAZEL_NATURAL/$f" ]] && cp "$BAZEL_NATURAL/$f" "$NATURAL_STAGED/$f"
111+
done
112+
113+
echo "==> Re-running bazel flow with make's netlist injected via SYNTH_NETLIST_FILES"
114+
# bazel-orfs supports positional KEY=VAL args after `bazel run` (see
115+
# bazel-orfs/README.md:354-368). The override propagates to make on
116+
# the command line where it wins over the design config.
117+
bazelisk run "$FINAL_TARGET" -- SYNTH_NETLIST_FILES="$MAKE_NETLIST"
118+
119+
# The second run wrote fresh .odb to the same bazel-bin path.
120+
BAZEL_OVERLAY="$BAZEL_NATURAL"
121+
122+
echo ""
123+
echo "==> .odb SHA matrix"
124+
printf '%-22s %-18s %-18s %-18s %s\n' \
125+
stage bazel-natural make 'bazel+make-netlist' outcome
126+
all_overlay_match=1
127+
for f in 1_synth.odb 2_floorplan.odb 3_place.odb 4_cts.odb \
128+
5_1_grt.odb 5_route.odb 6_final.odb; do
129+
bs=$(sha256sum "$NATURAL_STAGED/$f" 2>/dev/null | cut -c1-16 || true)
130+
ms=$(sha256sum "$MAKE_RES/$f" 2>/dev/null | cut -c1-16 || true)
131+
os=$(sha256sum "$BAZEL_OVERLAY/$f" 2>/dev/null | cut -c1-16 || true)
132+
if [[ -z "$os" && -z "$ms" ]]; then
133+
ovl_tag=skip
134+
elif [[ -n "$os" && "$os" == "$ms" ]]; then
135+
ovl_tag=MATCH
136+
else
137+
ovl_tag=DIFFER
138+
all_overlay_match=0
139+
fi
140+
printf '%-22s %-18s %-18s %-18s %s\n' \
141+
"$f" "${bs:--}" "${ms:--}" "${os:--}" "$ovl_tag"
142+
done
143+
144+
echo ""
145+
if [[ $all_overlay_match -eq 1 ]]; then
146+
echo "make-yosys-netlist: every stage MATCHes in the 'bazel-with-make-netlist'"
147+
echo " column -> bazel-test OpenROAD is bit-identical to tools/install"
148+
echo " OpenROAD given the same yosys netlist. Any bazel _test QoR failure"
149+
echo " on this design is yosys-environment drift, not a bazel-orfs or"
150+
echo " OpenROAD bug."
151+
exit 0
152+
else
153+
echo "make-yosys-netlist: some stages DIFFER even with the same netlist ->"
154+
echo " real bazel-vs-make OpenROAD divergence on this design. Worth"
155+
echo " filing the per-stage SHA matrix as an issue."
156+
exit 1
157+
fi

‎flow/README.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,173 @@ the generated targets get `tags = ["manual"]`.
5656
Each OpenROAD invocation takes `-threads <nproc>`. A wildcard
5757
`bazelisk test` runs designs in parallel and overcommits the host. Cap
5858
with `--jobs=N`.
59+
60+
## Running a full sweep across designs
61+
62+
To verify every design under one or more platforms (e.g. before
63+
bumping `bazel-orfs` or `tools/yosys`), drive `bazelisk test` per
64+
platform with `--keep_going` so one failure doesn't mask the rest.
65+
66+
```bash
67+
# 1. Install the make-flow toolchain too, so failures can be
68+
# cross-checked against `make DESIGN_CONFIG=...`.
69+
(cd tools/OpenROAD && bazelisk run //:install)
70+
71+
# 2. List actual test targets per platform.
72+
bazelisk query 'tests(//flow/designs/asap7/...)'
73+
bazelisk query 'tests(//flow/designs/sky130hd/...)'
74+
75+
# 3. Run the small/fast designs in one invocation (parallel),
76+
# and each big design in its own invocation (sequential) so
77+
# routing/synth don't overcommit RAM. 16-core/30 GiB box:
78+
bazelisk test --keep_going \
79+
//flow/designs/asap7/aes/... //flow/designs/asap7/gcd/... \
80+
//flow/designs/asap7/uart/... //flow/designs/sky130hd/gcd/... # etc.
81+
82+
for big in cva6 ibex jpeg riscv32i swerv_wrapper; do
83+
bazelisk test --keep_going //flow/designs/asap7/${big}/...
84+
done
85+
```
86+
87+
Heuristic: hierarchical / RISC-V cores / image-processing designs run
88+
solo; everything else batches. `bazelisk query 'tests(...)'` returns
89+
the exact target list — capture it before running so the sweep is
90+
reproducible.
91+
92+
## Triaging a failure
93+
94+
For each failing `<design>_test`, decide which bucket it falls into:
95+
96+
1. **Trivial config fix.** Missing variable in
97+
`flow/scripts/variables.yaml`, a typo in `config.mk`, a missing
98+
`exports_files()` for a cross-design SDC/Verilog reference, etc.
99+
Fix in place and commit `git commit -s` — `variables.yaml` is the
100+
single source of truth that bazel-orfs validates against.
101+
102+
2. **Bazel-only divergence.** Run the failing stage through the
103+
classic make flow and compare:
104+
105+
```bash
106+
cd flow
107+
make <stage> DESIGN_CONFIG=designs/<plat>/<design>/config.mk
108+
# then sha256sum the corresponding flow/results/.../<stage>.odb
109+
# against bazel-bin/flow/designs/.../results/.../<stage>.odb
110+
```
111+
112+
If the make flow succeeds where bazel fails, the cause is in the
113+
bazel toolchain pin — most commonly the `bazel-orfs` yosys version
114+
diverges from the `tools/yosys` submodule version that `make`
115+
uses. Note the divergence stage; **don't** file an upstream issue
116+
from these (the artifact in make is too different to reproduce the
117+
bazel-only failure).
118+
119+
3. **Idempotent failure (real bug).** Both flows fail at the same
120+
stage with the same error. `.odb` SHAs may still differ (yosys
121+
version) but the failure mode matches. Generate an issue tarball
122+
from the make flow and file the bug:
123+
124+
```bash
125+
cd flow
126+
make <stage>_issue DESIGN_CONFIG=designs/<plat>/<design>/config.mk \
127+
ISSUE_TAG=<plat>-<design>-<short>
128+
# tarball lands at flow/<stage>_<tag>.tar.gz
129+
130+
gh issue create --repo The-OpenROAD-Project-private/ascenium \
131+
--title "<plat>/<design>: <stage> fails (bazel == make)" \
132+
--body "<reproducer + SHA matrix + tarball path>"
133+
```
134+
135+
`flow/util/utils.mk:142` auto-generates `<script>_issue` targets
136+
for every script in `flow/scripts/`, so `floorplan_issue`,
137+
`global_route_issue`, `detail_route_issue`, `final_issue`, etc.
138+
all work.
139+
140+
### OpenROAD determinism across bazel and make
141+
142+
**TL;DR.** OpenROAD is bit-deterministic across the bazel-test build
143+
and the `tools/install` build when given the same yosys netlist.
144+
Yosys, however, is sensitive to its build environment (abc version,
145+
cxxopts version, compile flags), so bazel-built yosys and make-built
146+
yosys routinely produce different `1_2_yosys.v` for the same RTL.
147+
That netlist drift propagates into different placement/routing
148+
metrics, which can push a design past `rules-base.json` thresholds
149+
and fail the bazel `_test`. These failures are not bazel-orfs or
150+
OpenROAD bugs.
151+
152+
Two wrappers prove this bidirectionally:
153+
154+
#### Check a failing design for yosys-environment false positives
155+
156+
If a bazel `_test` is failing QoR on a specific design, run:
157+
158+
```bash
159+
bazelisk run //:make-yosys-netlist //flow/designs/asap7/uart:uart_test
160+
```
161+
162+
This:
163+
1. Builds the design's full bazel flow (yields a "bazel-natural"
164+
`.odb` chain).
165+
2. Runs the full classic make flow (yields make's `.odb` chain from
166+
make's `1_2_yosys.v`).
167+
3. Re-runs the full bazel flow with **make's** `1_2_yosys.v` injected
168+
via `SYNTH_NETLIST_FILES` ("bazel+make-netlist").
169+
4. SHA-compares every stage three ways.
170+
171+
Interpretation:
172+
173+
- **Every stage MATCH in the `bazel+make-netlist` column** →
174+
confirmed yosys-environment false positive. bazel-test OpenROAD
175+
≡ `tools/install` OpenROAD given the same starting netlist. The
176+
only variable making the bazel test fail QoR is yosys's build
177+
environment. Move on (or relax the threshold).
178+
- **Any DIFFER in the `bazel+make-netlist` column** → real
179+
bazel-vs-make OpenROAD divergence on this design. Worth filing
180+
the per-stage SHA matrix as an issue.
181+
- **Wrapper refuses with "design uses BLOCKS=…"** → bazel-orfs's
182+
`BLOCKS=` handling for hierarchical designs diverges from make's
183+
(sub-block `.lef`/`.lib` abstracts aren't staged into the parent
184+
synth action). Known gap; not a yosys-false-positive question.
185+
See aes-block for the canonical example.
186+
187+
#### Reverse-direction sanity check
188+
189+
```bash
190+
bazelisk run //:yosys-check //flow/designs/asap7/uart:uart_test
191+
```
192+
193+
Same fundamental check, opposite direction: feeds **bazel's**
194+
netlist into a fresh make-flow run. Use when you want to start
195+
from the bazel side or as a second opinion.
196+
197+
#### Manual override
198+
199+
Both wrappers ultimately invoke bazel-orfs's documented positional
200+
`KEY=VAL` mechanism (`bazel-orfs/README.md` "Bazel run with ORFS
201+
variable overrides"):
202+
203+
```bash
204+
# bazel side: re-run a stage with an injected netlist
205+
bazelisk run //flow/designs/asap7/uart:uart_final -- \
206+
SYNTH_NETLIST_FILES=/tmp/my-1_2_yosys.v
207+
208+
# make side: same via make's hook
209+
cd flow
210+
make clean_all DESIGN_CONFIG=designs/asap7/uart/config.mk
211+
make finish DESIGN_CONFIG=designs/asap7/uart/config.mk \
212+
SYNTH_NETLIST_FILES=/tmp/my-1_2_yosys.v
213+
```
214+
215+
Stage the netlist to a writable temp first — `flow/scripts/synth_preamble.tcl`
216+
uses `cp -p`, so a read-only bazel-out source makes the second
217+
invocation (do-yosys after do-yosys-canonicalize) fail with
218+
"Permission denied".
219+
220+
#### BLOCKS= caveat
221+
222+
For designs that set `BLOCKS=` (hierarchical synthesis with sub-block
223+
abstracts — e.g. `asap7/aes-block`), bazel-orfs and make currently
224+
disagree on how the parent's synth_odb action sees the sub-block
225+
`.lef`/`.lib` abstracts. Both wrappers refuse to run on these
226+
designs because the per-stage `.odb` comparison wouldn't be
227+
apples-to-apples. This is a bazel-orfs gap independent of
228+
OpenROAD determinism.

0 commit comments

Comments
 (0)