Skip to content

Commit a0d4b14

Browse files
mballanceCopilot
andcommitted
docs: add P2 reports and TUI views 6-8 to RST documentation
- reports.rst: add usage examples and autofunction/autoclass entries for report_test_budget (H), report_safety_matrix (L), report_seed_reliability (M) - exploring-tui.rst: document views 6 (Code Coverage), 7 (Test History with v2 flake stats), and 8 (Testplan closure); update global key table 1-8 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent d396b87 commit a0d4b14

2 files changed

Lines changed: 100 additions & 1 deletion

File tree

doc/source/working-with-coverage/exploring-tui.rst

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,24 @@ Press the number key at any time to switch views.
5353
Statistical analysis: hit-count distribution histogram, mean/median/stddev,
5454
coverage tier breakdown (complete / high / medium / low), bin utilization rate.
5555

56+
**6 — Code Coverage**
57+
File-level code coverage table showing statement, branch, and toggle hit
58+
rates per source file. Requires a database with code-coverage data (e.g.
59+
imported from Verilator).
60+
61+
**7 — Test History**
62+
Per-test contribution analysis. Each row shows total and unique coverage
63+
items hit by that test. When the database includes v2 history (NCDB v2),
64+
the detail panel also shows flake score, pass/fail counts, and mean CPU
65+
time. Sort with ``N`` (name), ``D`` (date), ``C`` (coverage), ``U``
66+
(unique).
67+
68+
**8 — Testplan**
69+
Testplan closure status. Requires a testplan embedded in the database
70+
(see :doc:`testplan`). Shows every testpoint with its stage, closure
71+
status, and pass/fail counts. The header displays a stage roll-up with
72+
colour-coded progress. Press ``r`` to refresh.
73+
5674
Global Keys
5775
===========
5876

@@ -62,7 +80,7 @@ Global Keys
6280

6381
* - Key
6482
- Action
65-
* - ``1`` – ``5``
83+
* - ``1`` – ``8``
6684
- Switch view
6785
* - ``?``
6886
- Help overlay

doc/source/working-with-coverage/reports.rst

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,75 @@ In a GitHub Actions workflow::
194194

195195
-----------
196196

197+
**********************
198+
Test budget by stage
199+
**********************
200+
201+
Estimate CPU-hour cost per stage from v2 test history mean CPU times::
202+
203+
from ucis.ncdb.ncdb_ucis import NcdbUCIS
204+
from ucis.ncdb.testplan import get_testplan
205+
from ucis.ncdb.reports import report_test_budget, format_test_budget
206+
207+
db = NcdbUCIS("coverage.cdb")
208+
plan = db.getTestplan()
209+
rpt = report_test_budget(plan, db)
210+
print(format_test_budget(rpt))
211+
212+
# JSON export
213+
import json
214+
print(json.loads(rpt.to_json())["stage_totals"])
215+
216+
Testpoints whose tests have no CPU time recorded appear in
217+
``rpt.missing_stats``.
218+
219+
-----------
220+
221+
**********************
222+
Safety traceability matrix
223+
**********************
224+
225+
Build a requirement-to-testpoint matrix (suitable for safety audits)::
226+
227+
from ucis.ncdb.reports import report_safety_matrix, format_safety_matrix
228+
229+
rpt = report_safety_matrix(results) # results from compute_closure
230+
print(format_safety_matrix(rpt))
231+
232+
# CSV for a spreadsheet or audit tool
233+
with open("traceability.csv", "w") as f:
234+
f.write(rpt.to_csv())
235+
236+
# Add a WaiverSet to flag waived testpoints
237+
from ucis.ncdb.waivers import WaiverSet
238+
waivers = WaiverSet.from_file("waivers.hjson")
239+
rpt = report_safety_matrix(results, waivers=waivers)
240+
241+
-----------
242+
243+
**********************
244+
Seed reliability heat-map
245+
**********************
246+
247+
Identify seeds that are disproportionately flaky::
248+
249+
from ucis.ncdb.ncdb_ucis import NcdbUCIS
250+
from ucis.ncdb.reports import report_seed_reliability, format_seed_reliability
251+
252+
db = NcdbUCIS("coverage.cdb")
253+
rpt = report_seed_reliability(db, "uart_smoke")
254+
print(format_seed_reliability(rpt))
255+
# Seeds with flake_score ≥ 0.2 are flagged with ⚠
256+
257+
# JSON heat-map for a custom dashboard
258+
import json
259+
data = json.loads(rpt.to_json())
260+
for row in data["rows"]:
261+
if row["flake"] >= 0.2:
262+
print(f"Seed {row['seed']}: {row['fail']} failures")
263+
264+
-----------
265+
197266
**********************
198267
GitHub Step Summary
199268
**********************
@@ -241,6 +310,18 @@ API reference
241310
.. autofunction:: ucis.ncdb.reports.format_coverage_contribution
242311
.. autoclass:: ucis.ncdb.reports.CoverageContribution
243312

313+
.. autofunction:: ucis.ncdb.reports.report_test_budget
314+
.. autofunction:: ucis.ncdb.reports.format_test_budget
315+
.. autoclass:: ucis.ncdb.reports.TestBudget
316+
317+
.. autofunction:: ucis.ncdb.reports.report_safety_matrix
318+
.. autofunction:: ucis.ncdb.reports.format_safety_matrix
319+
.. autoclass:: ucis.ncdb.reports.SafetyMatrix
320+
321+
.. autofunction:: ucis.ncdb.reports.report_seed_reliability
322+
.. autofunction:: ucis.ncdb.reports.format_seed_reliability
323+
.. autoclass:: ucis.ncdb.reports.SeedReliability
324+
244325
.. autofunction:: ucis.ncdb.testplan_export.export_junit_xml
245326
.. autofunction:: ucis.ncdb.testplan_export.export_github_annotations
246327
.. autofunction:: ucis.ncdb.testplan_export.export_summary_markdown

0 commit comments

Comments
 (0)