Skip to content

Commit 2e02b11

Browse files
committed
test/orfs: retire eqy equivalence checking
bazel-orfs upstream dropped eqy_test/sby_test formal verification support (eqy isn't actively maintained). Remove all consumers: eqy-flow.bzl, the ram_8x7 test (whose only purpose was catching eqy bugs), eqy_flow_test/eqy_test calls in mock-array and gcd, and related README documentation. Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
1 parent 0504146 commit 2e02b11

16 files changed

Lines changed: 0 additions & 3309 deletions

File tree

test/orfs/README.md

Lines changed: 0 additions & 121 deletions
Original file line numberDiff line numberDiff line change
@@ -109,127 +109,6 @@ To hunt down missing `tags = ["manual"]` run a query like:
109109

110110
Note that OpenROAD *does* want `bazelisk test ...` to run all tests, so test targets should be marked `tags = ["orfs"]` instead, so that `.bazelrc` can skip builds of those targets with the `build --build_tag_filters=-orfs` line.
111111

112-
## eqy tests
113-
114-
`eqy_test` is used to run equivalence checks before and after an ORFS stage, such as before and after floorplan for mock-array. To run the test and keep all the files from the test and see interactive output, run:
115-
116-
bazelisk test //test/orfs/mock-array:MockArray_4x4_eqy_test --test_output=streamed --sandbox_debug
117-
118-
If this fails, then it will output the line below. `eqy` uses a very, very large number of files and copying out these files to the bazel-testlogs folder for inspection takes some time:
119-
120-
Copying 114462 files to bazel-testlogs/test/orfs/mock-array/MockArray_4x4_eqy_test/test.outputs for inspection.
121-
122-
If you just want the files needed to run a locally installed `eqy`, build all the files used in the run above by:
123-
124-
bazelisk build //test/orfs/mock-array:MockArray_4x4_eqy_test
125-
126-
The files used to run the test are then in `bazel-bin/test/orfs/mock-array/`.
127-
128-
### TL;DR creating archive for standalone eqy test
129-
130-
Create `/tmp/issue.tar.gz`
131-
132-
bazelisk build //test/orfs/gcd:gcd_eqy_synth_test
133-
tar --exclude='*oss_cad_suite*' -czvf /tmp/issue.tar.gz -C "$(bazelisk info bazel-bin)/test/orfs/gcd/gcd_eqy_synth_test.run.sh.runfiles" .
134-
135-
Reproduction instructions:
136-
137-
1. untar archive
138-
2. cd _main
139-
3. ~/oss-cad-suite/bin/eqy test/orfs/gcd/gcd_eqy_synth_test.eqy
140-
141-
Comments:
142-
143-
- `~/oss-cad-suite/` should be familiar with anyone versed in eqy as it refers to the [official binary release binaries](https://github.com/YosysHQ/oss-cad-suite-build)
144-
- Reproduction cases should be minimal in terms of size and context. The above recipe contains some unnecessary files, unnecessary directory structure, etc. Prune the test case further manually to get to the core of the matter.
145-
146-
## Running eqy tests standalone outside bazel
147-
148-
Build the test files:
149-
150-
bazelisk build //test/orfs/gcd:gcd_eqy_synth_test
151-
152-
This outputs:
153-
154-
Target //test/orfs/gcd:gcd_eqy_synth_test up-to-date:
155-
bazel-bin/test/orfs/gcd/gcd_eqy_synth_test.run.sh
156-
157-
Copy all files needed to run the tests into a working folder `foo`:
158-
159-
rsync -aL "$(bazelisk info bazel-bin)/test/orfs/gcd/gcd_eqy_synth_test.run.sh.runfiles/" foo/
160-
161-
Change cwd for running the tests:
162-
163-
cd foo/_main
164-
165-
Bazel sets up both binaries and data files to run the tests, to create a reportable standalone test. A report should include only the data files and not the actual binaries used locally to run the test:
166-
167-
$ find .
168-
.
169-
./test
170-
./test/orfs
171-
./test/orfs/asap7
172-
./test/orfs/asap7/asap7sc7p5t_SIMPLE_RVT_TT_201020.v
173-
./test/orfs/asap7/asap7sc7p5t_INVBUF_RVT_TT_201020.v
174-
./test/orfs/asap7/asap7sc7p5t_AO_RVT_TT_201020.v
175-
./test/orfs/asap7/asap7sc7p5t_OA_RVT_TT_201020.v
176-
./test/orfs/gcd
177-
./test/orfs/gcd/gcd.v
178-
./test/orfs/gcd/gcd_eqy_synth_test.run.sh
179-
./test/orfs/gcd/gcd_eqy_synth.v
180-
./test/orfs/gcd/gcd_eqy_synth_test.eqy
181-
182-
Bazel generated `bazel-bin/test/orfs/gcd/gcd_eqy_synth_test.run.sh` to runs the tests. Examine this script to look at that for clues as to how to run the tests standalone:
183-
184-
$ head -n 5 test/orfs/gcd/gcd_eqy_synth_test.run.sh
185-
# !/bin/sh
186-
set -euo pipefail
187-
test_status=0
188-
(exec ../bazel-orfs++_repo_rules+oss_cad_suite/bin/eqy "$@" test/orfs/gcd/gcd_eqy_synth_test.eqy) || test_status=$?
189-
190-
What we want is:
191-
192-
../bazel-orfs++_repo_rules+oss_cad_suite/bin/eqy test/orfs/gcd/gcd_eqy_synth_test.eqy
193-
194-
Running this, we get:
195-
196-
<pre>$ ../bazel-orfs++_repo_rules+oss_cad_suite/bin/eqy test/orfs/gcd/gcd_eqy_synth_test.eqy
197-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:24</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">read_gold</font>: starting process &quot;yosys -ql gcd_eqy_synth_test/gold.log gcd_eqy_synth_test/gold.ys&quot;
198-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:24</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">read_gold</font>: finished (returncode=0)
199-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:24</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">read_gate</font>: starting process &quot;yosys -ql gcd_eqy_synth_test/gate.log gcd_eqy_synth_test/gate.ys&quot;
200-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:25</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">read_gate</font>: finished (returncode=0)
201-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:25</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">combine</font>: starting process &quot;yosys -ql gcd_eqy_synth_test/combine.log gcd_eqy_synth_test/combine.ys&quot;
202-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:25</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A347BA">combine</font>: finished (returncode=0)
203-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:25</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#A2734C"><b>Warning: Cannot find entity _*_.*.</b></font>
204-
[deleted]
205-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:32</font> [<font color="#12488B">gcd_eqy_synth_test</font>] <font color="#C01C28"><b>Failed to prove equivalence of partition gcd.resp_msg.1</b></font>
206-
[deleted]
207-
<font color="#12488B">EQY</font> <font color="#26A269"> 8:05:32</font> [<font color="#12488B">gcd_eqy_synth_test</font>] DONE (FAIL, rc=2)
208-
</pre>
209-
210-
211-
## using sv-bugpoint to whittle down test-cases
212-
213-
[sv-bugpoint](https://github.com/antmicro/sv-bugpoint) can be used to whittle down test-cases to a minimum example.
214-
215-
Let's say that `bazelisk test test/orfs/gcd:eqy_synth_test` fails, first create a script that looks for the error string in the output, in this case a false positive `Failed to prove equivalence of partition gcd.req_rdy`.
216-
217-
After compiling sv-bugpoint, we create a check.sh script:
218-
219-
```bash
220-
#!/usr/bin/env bash
221-
set -euo pipefail
222-
cp $1 test/orfs/gcd/
223-
(bazelisk test test/orfs/gcd:eqy_synth_test --test_timeout=30 --test_output=streamed || error=$?) | tee /dev/tty | grep "Failed to prove equivalence of partition gcd.req_rdy"
224-
```
225-
226-
- `--test_timeout=30` is a suitable timeout for this test and machine, adjust. Note that the timeout must be handled by Bazel and not a generic `timeout` utility as it would not work correctly with the bazel server.
227-
228-
229-
Next, run sv-bugpoint to whittle `test/orfs/gcd/gcd.v` down to a minimal test case:
230-
231-
~/sv-bugpoint/build/sv-bugpoint fail check.sh test/orfs/gcd/gcd.v
232-
233112
## Using whittle.py to minimize .odb files
234113

235114
While sv-bugpoint minimizes Verilog source files, `whittle.py` minimizes

test/orfs/eqy-flow.bzl

Lines changed: 0 additions & 88 deletions
This file was deleted.

test/orfs/gcd/BUILD

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
load("@bazel-orfs//:sweep.bzl", "orfs_sweep")
22
load("@rules_shell//shell:sh_test.bzl", "sh_test")
33
load("//test/orfs:check_same.bzl", "check_same")
4-
load("//test/orfs:eqy-flow.bzl", "eqy_flow_test")
54

65
package(features = ["layering_check"])
76

@@ -36,14 +35,6 @@ orfs_sweep(
3635
verilog_files = ["gcd.v"],
3736
)
3837

39-
eqy_flow_test(
40-
name = "gcd_eqy",
41-
flow = "gcd",
42-
module_top = "gcd",
43-
tags = ["manual"],
44-
verilog_files = ["gcd.v"],
45-
)
46-
4738
filegroup(
4839
name = "gcd_floorplan_odb",
4940
srcs = [":gcd_floorplan"],

test/orfs/gcd/README.md

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -11,31 +11,6 @@ This design has about 250 cells.
1111

1212
Re-used code derived from http://opencelerity.org/ project.
1313

14-
## LEC tests
15-
16-
LEC tests exist for each of the ORFS stages as well as a self-test for the original source.
17-
18-
In terms of expectations for LEC tests, this is experimental and contributions are highly appreciated. As of writing, even for such a small test case as gcd, the running time is considerable and eqy has false positives reported to the eqy project.
19-
20-
```bash
21-
$ cd test/orfs/gcd
22-
$ bazelisk query :* | grep eqy | grep _test
23-
Loading: 0 packages loaded
24-
//test/orfs/gcd:gcd_eqy_cts_test
25-
//test/orfs/gcd:gcd_eqy_final_test
26-
//test/orfs/gcd:gcd_eqy_floorplan_test
27-
//test/orfs/gcd:gcd_eqy_grt_test
28-
//test/orfs/gcd:gcd_eqy_place_test
29-
//test/orfs/gcd:gcd_eqy_route_test
30-
//test/orfs/gcd:gcd_eqy_source_test
31-
//test/orfs/gcd:gcd_eqy_synth_test
32-
//test/orfs/gcd:gcd_eqy_tests
33-
```
34-
35-
To run all the tests:
36-
37-
bazelisk test //test/orfs/gcd:gcd_eqy_tests --keep_going
38-
3914
## Whittle test
4015

4116
The `gcd_whittle_test` is an integration test for `etc/whittle.py`, the

test/orfs/mock-array/BUILD

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -83,23 +83,6 @@ CONFIGS = {name: config(name, rows, cols) for name, rows, cols in [
8383
config = config,
8484
) for name, config in CONFIGS.items()]
8585

86-
test_suite(
87-
name = "eqy_tests",
88-
tags = ["manual"],
89-
tests = ["Element_eqy_{variant}".format(variant = name) for name in CONFIGS] +
90-
[
91-
"MockArray_eqy_{name}_{variant}".format(
92-
name = name,
93-
variant = variant,
94-
)
95-
for name in CONFIGS
96-
for variant in [
97-
"flat",
98-
"base",
99-
]
100-
],
101-
)
102-
10386
orfs_run(
10487
name = "write_macro_placement",
10588
src = ":MockArray_4x4_base_floorplan",

test/orfs/mock-array/README.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -63,9 +63,6 @@ power reporting. It has exposed:
6363
names (`475eb39584`).
6464
- Verilator reported 17k UNDRIVEN warnings on hierarchical output
6565
(issue #8108).
66-
- Equivalence checking (eqy) failures from `write_verilog` output and
67-
FILLER/TAPCELL cell interactions (issues #8869, #8905, #9035; fix
68-
PR #8966).
6966

7067
## What mock-array does NOT test
7168

@@ -151,7 +148,4 @@ bazelisk build //test/orfs/mock-array:Element_4x4_slang_synth
151148
# Full: place-and-route (takes minutes)
152149
bazelisk test //test/orfs/mock-array:MockArray_4x4_base_test
153150
bazelisk test //test/orfs/mock-array:MockArray_4x4_flat_test
154-
155-
# Power and equivalence (slowest)
156-
bazelisk test //test/orfs/mock-array:eqy_tests
157151
```

test/orfs/mock-array/mock-array.bzl

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ load("@rules_cc//cc:defs.bzl", "cc_binary")
99
load("@rules_shell//shell:sh_test.bzl", "sh_test")
1010
load("@rules_verilator//verilator:defs.bzl", "verilator_cc_library")
1111
load("@rules_verilator//verilog:defs.bzl", "verilog_library")
12-
load("//test/orfs:eqy-flow.bzl", "eqy_flow_test")
1312

1413
FIRTOOL_OPTIONS = [
1514
"-disable-all-randomization",
@@ -107,13 +106,6 @@ def element(name, config):
107106
verilog_files = [":{name}_verilog".format(name = name)],
108107
variant = "{name}_base".format(name = name),
109108
)
110-
eqy_flow_test(
111-
name = "Element_eqy_{variant}".format(variant = name),
112-
flow = "Element_{variant}_base".format(variant = name),
113-
verilog_files = [":{name}_verilog".format(name = name)],
114-
tags = ["manual"],
115-
module_top = "Element",
116-
)
117109

118110
POWER_STAGES = {
119111
"cts": {
@@ -302,14 +294,6 @@ def mock_array(name, config):
302294
variant = variant,
303295
verilog_files = [":{name}_verilog".format(name = name)],
304296
)
305-
eqy_flow_test(
306-
name = "MockArray_eqy_{variant}".format(variant = variant),
307-
flow = "MockArray_{variant}".format(variant = variant),
308-
verilog_files = [":{name}_verilog".format(name = name)],
309-
other_verilog_files = [":Element_eqy_{name}_final_verilog".format(name = name)],
310-
tags = ["manual"],
311-
module_top = "MockArray",
312-
)
313297

314298
for stage in POWER_STAGES:
315299
for macro in MACROS:

0 commit comments

Comments
 (0)