Skip to content

Commit 697959a

Browse files
crazydemocrazydemo
committed
[None][test] CBTS coverage: enable for all stages, kill-switch via two knobs
Swap the dogfood allowlist for an enable-by-default gate so the author can pick stages via /bot run --stage-list rather than maintaining a hard-coded list of stage names that drifts as stages get renamed. L0_Test.groovy: * Replace CBTS_DOGFOOD_STAGES = {"DGX_H100-PyTorch-1"} with two knobs: ENABLE_CBTS_COVERAGE default true global kill-switch CBTS_EXCLUDE_STAGES default {} per-stage opt-out isCbtsStage() now returns true for every stage unless one of these knobs takes effect. coverage_utils/README.md: * Rename the "L0 dogfood" section to "L0 rollout" and document both kill-switches and the recommended /bot run --stage-list workflow. * Update the validation checklist to match: A. now points at the new kill-switches instead of the removed allowlist; D. explains the non-CBTS-stage check in terms of an explicit opt-out rather than a default-disabled stage. No behavior change for stages that were already covered; legacy- pytest-cov stages now exist only when explicitly opted out. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
1 parent 6967fc4 commit 697959a

2 files changed

Lines changed: 118 additions & 6 deletions

File tree

‎jenkins/L0_Test.groovy‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -143,14 +143,22 @@ ENABLE_NGC_RELEASE_IMAGE_TEST = params.enableNgcReleaseImageTest ?: false
143143

144144
COMMON_SSH_OPTIONS = "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o TCPKeepAlive=no -o ServerAliveInterval=30 -o ServerAliveCountMax=20"
145145

146-
// CBTS dogfood gate: only the named stage(s) use the new per-test coverage
147-
// pipeline (cbts_plugin + sitecustomize). Everything else stays on the
148-
// legacy pytest-cov + .coveragerc(branch=True) path. Expand the list once
149-
// the dogfood validates.
150-
CBTS_DOGFOOD_STAGES = ["DGX_H100-PyTorch-1"] as Set
146+
// CBTS coverage gate. The per-test coverage pipeline (cbts_plugin +
147+
// sitecustomize) replaces the legacy pytest-cov + .coveragerc(branch=True)
148+
// path on every stage we let through this gate. The author triggers
149+
// individual stages by passing them via `/bot run --stage-list ...` so
150+
// the allowlist itself does not need to narrow the scope -- it's left
151+
// here only as a quick kill-switch:
152+
// * to disable CBTS on one bad stage: add it to CBTS_EXCLUDE_STAGES.
153+
// * to disable CBTS everywhere: flip ENABLE_CBTS_COVERAGE to false.
154+
ENABLE_CBTS_COVERAGE = true
155+
CBTS_EXCLUDE_STAGES = [] as Set
151156

152157
def isCbtsStage(String stageName) {
153-
return CBTS_DOGFOOD_STAGES.contains(stageName)
158+
if (!ENABLE_CBTS_COVERAGE) {
159+
return false
160+
}
161+
return !CBTS_EXCLUDE_STAGES.contains(stageName)
154162
}
155163

156164
def scpFromRemoteCmd(Map remote, String remotePath, String localPath) {

‎jenkins/scripts/cbts/coverage_utils/README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,110 @@ coverage combine --rcfile="${JOB_WORKSPACE}/.coveragerc"
5656
# → ${JOB_WORKSPACE}/.coverage.${STAGE_NAME}
5757
```
5858

59+
## L0 rollout: triggering and validation checklist
60+
61+
The CBTS path is enabled for **every** Jenkins stage by default; the
62+
author scopes which stage(s) actually run via `/bot run --stage-list`.
63+
Two kill-switches in `jenkins/L0_Test.groovy`:
64+
65+
| Knob | What it does |
66+
|---|---|
67+
| `ENABLE_CBTS_COVERAGE = false` | flip off CBTS everywhere; every stage reverts to the legacy pytest-cov path |
68+
| `CBTS_EXCLUDE_STAGES = ["X", ...]` | keep CBTS on but skip it for the listed stages (e.g. a stage that turns out to be incompatible) |
69+
70+
### Trigger from a PR
71+
72+
```
73+
/bot run # full default pre-merge stage set
74+
/bot run --stage-list "DGX_H100-PyTorch-1" # one stage only
75+
/bot run --stage-list "DGX_H100-PyTorch-1, A30-PyTorch-1" # a few stages
76+
/bot run --disable-fail-fast # keep going even if an earlier stage fails
77+
/bot help # full bot syntax
78+
```
79+
80+
### A. Read the Jenkins console log
81+
82+
Search the CBTS stage's log for these markers; missing any of them
83+
usually means `ENABLE_CBTS_COVERAGE` was flipped off or the stage is
84+
in `CBTS_EXCLUDE_STAGES`.
85+
86+
| Search for | Expect |
87+
|---|---|
88+
| `cat ./.coveragerc` | the CBTS rcfile: `core = sysmon`, `branch = False`, `parallel = True`, `sigterm = True` |
89+
| pytest command line | **no** `--cov=*` or `--cov-config=*`; instead `-p cbts_plugin` |
90+
| `plugins: ... cbts_plugin` | plugin loaded by pytest |
91+
| env-var dump | `CBTS_COVERAGE_CONFIG`, `CBTS_MARKER_FILE`, `CBTS_SENTINEL_DIR`, `PYTHONPATH=...coverage_utils...` |
92+
| `Submit Test Result` stage | `CBTS coverage uploaded to <UPLOAD_PATH>/cbts-coverage/cbts-<stage>.tar.gz` |
93+
94+
### B. Verify the uploaded artifact
95+
96+
```bash
97+
tar tzf cbts-DGX_H100-PyTorch-1.tar.gz | head -20
98+
```
99+
100+
Expect a `cbts/` directory with:
101+
102+
- one or more `.coverage.DGX_H100-PyTorch-1.<host>.pid<N>.X<rand>` per-process files
103+
- a `cbts_seen_tests/` directory containing one sentinel file per test (filename = `sha1(nodeid)`)
104+
105+
### C. Validate the data is queryable
106+
107+
```bash
108+
cd cbts/
109+
# Combine all per-process files into one DB.
110+
coverage combine --keep --rcfile=/dev/stdin <<EOF
111+
[run]
112+
data_file = .coverage.DGX_H100-PyTorch-1
113+
EOF
114+
115+
# Count distinct test contexts (should be close to the pytest PASSED count).
116+
python3 -c "
117+
import sqlite3
118+
c = sqlite3.connect('.coverage.DGX_H100-PyTorch-1')
119+
n = c.execute('select count(*) from context where context != \"\"').fetchone()[0]
120+
print(f'distinct test contexts: {n}')
121+
"
122+
123+
# Run the audit; expect OK + MAIN_ONLY + EMPTY totals to match the test count.
124+
python3 ../coverage_audit.py \
125+
--db .coverage.DGX_H100-PyTorch-1 \
126+
--sentinel-dir cbts_seen_tests \
127+
--out-csv audit.csv --out-json audit.json
128+
```
129+
130+
### D. Regression check on a non-CBTS stage
131+
132+
If you flipped `ENABLE_CBTS_COVERAGE = false` or added a stage to
133+
`CBTS_EXCLUDE_STAGES`, confirm that excluded stage:
134+
135+
| Check | Expect |
136+
|---|---|
137+
| Its `.coveragerc` content | still `branch = True` (the legacy block) |
138+
| Its pytest command line | still has `--cov=*` and `--cov-config=*` |
139+
| Wall-clock vs historical runs | no significant change |
140+
| No `CBTS_*` env vars in env-dump | side-effect-free |
141+
142+
### E. Wall-clock overhead
143+
144+
Compare the stage's wall time to the last few historical runs on the
145+
same stage. Expected overhead is **+5-10%** (sysmon core is ~3-4%
146+
per coverage.py docs; the extra is sitecustomize startup + periodic-
147+
save in each subprocess).
148+
149+
> If overhead is >30%, `core = sysmon` likely did not take effect.
150+
> Check that `branch = False` in the materialised `.coveragerc` and
151+
> that no upstream `[run]` block overrides `core`.
152+
153+
### Common failure modes
154+
155+
| Symptom | Likely cause |
156+
|---|---|
157+
| None of the A. markers show up | `ENABLE_CBTS_COVERAGE` is false, or the stage is in `CBTS_EXCLUDE_STAGES` |
158+
| `ModuleNotFoundError: No module named 'cbts_plugin'` | `PYTHONPATH` not propagated to the pytest invocation -- check the env-dump line in the log |
159+
| Zero `.coverage.*` files scp'd back | `covScp`/`sentScp` status non-zero in the log: remote glob did not match, usually means pytest crashed before producing any per-process file |
160+
| `audit.json` reports `FATAL: 0 named contexts` | `sitecustomize.py` did not fire in any subprocess -- usually means `PYTHONPATH` was stripped between pytest main and the subprocess (check the env whitelist in `mpi_session._start_mpi_pool`) |
161+
| Wall-clock doubled | `branch = True` somehow leaked in, falling back to settrace |
162+
59163
## Key design decisions (defended in `LAYER_C_PLAN.md`)
60164

61165
- **No product code change.** Env propagation to MPI workers is handled

0 commit comments

Comments
 (0)