|
| 1 | +--- |
| 2 | +name: add-test |
| 3 | +description: > |
| 4 | + Add integration or unit tests to an OpenROAD module. Handles the |
| 5 | + complete workflow: writing the Tcl/C++ test, generating golden files, |
| 6 | + and registering the test in BOTH CMake and Bazel build systems (the |
| 7 | + most common mistake is forgetting Bazel registration). Use when asked |
| 8 | + to add tests, improve test coverage, create regression tests, or write |
| 9 | + unit tests for any OpenROAD module (ant, cts, dpl, drt, gpl, grt, mpl, |
| 10 | + odb, pad, pdn, psm, rcx, rsz, sta, stt, tap, upf, utl, etc.). |
| 11 | + Also triggers on: "add test", "write test", "test coverage", |
| 12 | + "missing tests", "create regression test", "add unit test". |
| 13 | +argument-hint: "<module> [test-description]" |
| 14 | +--- |
| 15 | + |
| 16 | +# Add Test to OpenROAD Module |
| 17 | + |
| 18 | +You are adding tests for: **$ARGUMENTS** |
| 19 | + |
| 20 | +## 1. Identify the module and scope |
| 21 | + |
| 22 | +Parse `$ARGUMENTS` to determine: |
| 23 | +- **Module name** (e.g., `rsz`, `drt`, `gpl`) |
| 24 | +- **What to test** (specific feature, edge case, or general coverage) |
| 25 | + |
| 26 | +Explore existing tests to understand conventions: |
| 27 | +```bash |
| 28 | +ls src/MODULE/test/*.tcl | head -20 |
| 29 | +``` |
| 30 | + |
| 31 | +Read a representative test for the module's style: |
| 32 | +```bash |
| 33 | +# Good reference for integration test patterns: |
| 34 | +cat src/rsz/test/repair_tie12_hier.tcl |
| 35 | +``` |
| 36 | + |
| 37 | +## 2. Write the integration test (Tcl) |
| 38 | + |
| 39 | +Create `src/MODULE/test/TEST_NAME.tcl`: |
| 40 | + |
| 41 | +```tcl |
| 42 | +# Brief description of what this test verifies |
| 43 | +set test_name TEST_NAME |
| 44 | +source "helpers.tcl" |
| 45 | +
|
| 46 | +# Read design files |
| 47 | +read_lef "TECH.lef" |
| 48 | +read_lef "CELLS.lef" |
| 49 | +read_def "DESIGN.def" |
| 50 | +
|
| 51 | +# ... test operations ... |
| 52 | +
|
| 53 | +# Use make_result_file for temporary output |
| 54 | +set def_filename "${test_name}.def" |
| 55 | +set out_def [make_result_file $def_filename] |
| 56 | +write_def $out_def |
| 57 | +
|
| 58 | +# Compare against golden file |
| 59 | +diff_file ${test_name}.defok $out_def |
| 60 | +``` |
| 61 | + |
| 62 | +### Test design files |
| 63 | + |
| 64 | +Prefer reusing existing test data in `src/MODULE/test/`. Only create |
| 65 | +new LEF/DEF files when testing a specific geometry or edge case that |
| 66 | +existing files don't cover. Keep test data minimal. |
| 67 | + |
| 68 | +### Naming conventions |
| 69 | + |
| 70 | +- Test files: `src/MODULE/test/TEST_NAME.tcl` |
| 71 | +- Golden log: `src/MODULE/test/TEST_NAME.ok` |
| 72 | +- Golden output: `src/MODULE/test/TEST_NAME.{defok,vok,spefok,...}` |
| 73 | +- Test data: `src/MODULE/test/*.{lef,def,lib,sdc,v}` |
| 74 | + |
| 75 | +## 3. Generate golden files |
| 76 | + |
| 77 | +Run the test and capture output: |
| 78 | +```bash |
| 79 | +cd src/MODULE/test |
| 80 | + |
| 81 | +# Run with openroad to generate output |
| 82 | +openroad -no_splash -no_init -exit TEST_NAME.tcl > TEST_NAME.ok 2>&1 |
| 83 | + |
| 84 | +# Remove the openroad banner from the .ok file if present |
| 85 | +# The .ok file should contain only the test's stdout |
| 86 | +``` |
| 87 | + |
| 88 | +If the test uses `diff_file`, also generate the golden output file |
| 89 | +(e.g., `TEST_NAME.defok`) by running once and copying the result file. |
| 90 | + |
| 91 | +## 4. Register in BOTH build systems |
| 92 | + |
| 93 | +This is the most common mistake. **Every test must be in both CMake AND |
| 94 | +Bazel.** CMake-only tests silently pass locally but are missing from |
| 95 | +Bazel CI. |
| 96 | + |
| 97 | +### CMake -- `src/MODULE/test/CMakeLists.txt` |
| 98 | + |
| 99 | +Find the `or_integration_tests()` call. The first positional argument |
| 100 | +is the **module name**, followed by the `TESTS` keyword and the list: |
| 101 | + |
| 102 | +```cmake |
| 103 | +or_integration_tests( |
| 104 | + "MODULE" # <-- module name (e.g. "rsz") |
| 105 | + TESTS |
| 106 | + existing_test1 |
| 107 | + existing_test2 |
| 108 | + TEST_NAME # <-- add here |
| 109 | +) |
| 110 | +``` |
| 111 | + |
| 112 | +Some modules also have a `PASSFAIL_TESTS` section after `TESTS` -- add |
| 113 | +to the appropriate list. Look at the module's existing `CMakeLists.txt` |
| 114 | +before editing. |
| 115 | + |
| 116 | +### Bazel -- `src/MODULE/test/BUILD` |
| 117 | + |
| 118 | +The typical pattern is a `TESTS` list consumed by a list comprehension |
| 119 | +that calls `regression_test()` for each name: |
| 120 | + |
| 121 | +```python |
| 122 | +TESTS = [ |
| 123 | + "buffer_ports1", |
| 124 | + "buffer_ports10", |
| 125 | + "buffer_ports11", |
| 126 | + "TEST_NAME", # <-- add here, in the existing sort order |
| 127 | + # (other tests omitted) |
| 128 | +] |
| 129 | + |
| 130 | +[ |
| 131 | + regression_test( |
| 132 | + name = test_name, |
| 133 | + ) |
| 134 | + for test_name in TESTS |
| 135 | +] |
| 136 | +``` |
| 137 | + |
| 138 | +Some modules split tests across multiple lists (`TESTS`, |
| 139 | +`PASSFAIL_TESTS`, `BIG_TESTS`, etc.) and define a combined |
| 140 | +`ALL_TESTS = TESTS + PASSFAIL_TESTS + ...` that the comprehension |
| 141 | +iterates over. Read the existing `BUILD` to find which list to |
| 142 | +extend, and insert the new test name in whatever sort order that |
| 143 | +file already uses (most modules are alphabetical). |
| 144 | + |
| 145 | +## 5. Write unit tests (C++, if applicable) |
| 146 | + |
| 147 | +Prefer C++ unit tests over Tcl for testing internal logic, algorithms, |
| 148 | +and data structure operations. Use Tcl integration tests for |
| 149 | +command-level behavior and end-to-end flows. |
| 150 | + |
| 151 | +### Test file |
| 152 | + |
| 153 | +Create `src/MODULE/test/cpp/TestName.cpp`: |
| 154 | + |
| 155 | +```cpp |
| 156 | +#include "gtest/gtest.h" |
| 157 | +#include "odb/db.h" |
| 158 | + |
| 159 | +// Simple test (no fixture) |
| 160 | +namespace module { |
| 161 | +namespace { |
| 162 | + |
| 163 | +TEST(ModuleTest, TestDescription) { |
| 164 | + // ARRANGE - ACT - ASSERT |
| 165 | + EXPECT_EQ(expected, actual); |
| 166 | +} |
| 167 | + |
| 168 | +} // namespace |
| 169 | +} // namespace module |
| 170 | +``` |
| 171 | +
|
| 172 | +### Using test fixtures |
| 173 | +
|
| 174 | +OpenROAD provides a fixture hierarchy for tests that need database |
| 175 | +or tool setup. Choose the simplest one that covers your needs: |
| 176 | +
|
| 177 | +| Fixture | Use when | |
| 178 | +|---------|----------| |
| 179 | +| `tst::Fixture` | Need bare `dbDatabase` + STA, load LEFs manually | |
| 180 | +| `tst::Nangate45Fixture` | Need pre-loaded Nangate45 tech/lib | |
| 181 | +| `tst::Sky130Fixture` | Need pre-loaded Sky130 tech/lib | |
| 182 | +| `tst::IntegratedFixture` | Need full tool integration (STA, DPL, GRT, RSZ) | |
| 183 | +
|
| 184 | +```cpp |
| 185 | +#include "gtest/gtest.h" |
| 186 | +#include "tst/nangate45_fixture.h" |
| 187 | +
|
| 188 | +class TestFeature : public tst::Nangate45Fixture |
| 189 | +{ |
| 190 | + protected: |
| 191 | + void SetUp() override { /* additional setup */ } |
| 192 | +}; |
| 193 | +
|
| 194 | +TEST_F(TestFeature, HandlesEdgeCase) { |
| 195 | + // block_, lib_ are available from the fixture |
| 196 | + EXPECT_TRUE(condition); |
| 197 | +} |
| 198 | +``` |
| 199 | + |
| 200 | +### CMake registration -- `src/MODULE/test/cpp/CMakeLists.txt` |
| 201 | + |
| 202 | +```cmake |
| 203 | +add_executable(TestName TestName.cpp) |
| 204 | +target_link_libraries(TestName |
| 205 | + MODULE_lib |
| 206 | + GTest::gtest |
| 207 | + GTest::gtest_main |
| 208 | + tst |
| 209 | + odb |
| 210 | +) |
| 211 | +gtest_discover_tests(TestName |
| 212 | + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/.. |
| 213 | +) |
| 214 | +add_dependencies(build_and_test TestName) |
| 215 | +``` |
| 216 | + |
| 217 | +### Bazel registration -- `src/MODULE/test/BUILD` |
| 218 | + |
| 219 | +Add the `cc_test` target alongside the existing `regression_test` |
| 220 | +entries. Most modules keep C++ tests in the same BUILD file: |
| 221 | + |
| 222 | +```python |
| 223 | +load("@rules_cc//cc:cc_test.bzl", "cc_test") |
| 224 | + |
| 225 | +cc_test( |
| 226 | + name = "TestName", |
| 227 | + srcs = ["cpp/TestName.cpp"], |
| 228 | + deps = [ |
| 229 | + "//src/MODULE", |
| 230 | + "//src/tst", |
| 231 | + "//src/tst:nangate45_fixture", # if using Nangate45Fixture |
| 232 | + "@googletest//:gtest", |
| 233 | + "@googletest//:gtest_main", |
| 234 | + ], |
| 235 | +) |
| 236 | +``` |
| 237 | + |
| 238 | +## 6. Run and verify |
| 239 | + |
| 240 | +`./regression` is a thin ctest wrapper that filters by module label, so |
| 241 | +the standard ctest flags apply. Use `-R` to match a single test by name: |
| 242 | + |
| 243 | +```bash |
| 244 | +cd src/MODULE/test |
| 245 | +./regression -R TEST_NAME |
| 246 | + |
| 247 | +# Verify test is discoverable in both systems |
| 248 | +cd ../../../build |
| 249 | +ctest -N | grep MODULE.*TEST_NAME |
| 250 | +``` |
| 251 | + |
| 252 | +## 7. Format and commit |
| 253 | + |
| 254 | +```bash |
| 255 | +# Format any C++ files (NEVER format src/sta/* or *.i files) |
| 256 | +clang-format -i <changed-cpp-files> |
| 257 | + |
| 258 | +git add src/MODULE/test/TEST_NAME.tcl \ |
| 259 | + src/MODULE/test/TEST_NAME.ok \ |
| 260 | + src/MODULE/test/CMakeLists.txt \ |
| 261 | + src/MODULE/test/BUILD |
| 262 | +git commit -s -m "MODULE: add TEST_NAME test |
| 263 | +
|
| 264 | +Test for DESCRIPTION." |
| 265 | +``` |
| 266 | + |
| 267 | +## Checklist |
| 268 | + |
| 269 | +- [ ] Test runs successfully: `./regression -R TEST_NAME` |
| 270 | +- [ ] Golden file generated and committed |
| 271 | +- [ ] Registered in CMake `CMakeLists.txt` |
| 272 | +- [ ] Registered in Bazel `BUILD` |
| 273 | +- [ ] Test data is minimal (reuse existing files when possible) |
| 274 | +- [ ] C++ files formatted with clang-format |
| 275 | +- [ ] Commit is signed off (`-s`) |
0 commit comments