Skip to content

Commit 9622c18

Browse files
dblnzCopilot
dblnz
and
Copilot
committed
feat: add coverage report generation with cargo-llvm-cov
Add coverage infrastructure for the Hyperlight project: - Justfile: add coverage-run, coverage, coverage-html, coverage-lcov, and coverage-ci recipes using cargo-llvm-cov for LLVM source-based code coverage. Tests run once via coverage-run; report recipes just generate the desired output format from collected profdata. - CI: add Coverage.yml weekly workflow (Monday 06:00 UTC, manual trigger) running on kvm/amd with self-built guest binaries - coverage-ci mirrors test-like-ci by running multiple test phases with different feature combinations (default, single-driver, crashdump, tracing) and merging profdata into a single unified report - Coverage summary is displayed in the GitHub Actions Job Summary for quick viewing; full HTML report is downloadable as an artifact - docs: add how-to-run-coverage.md with local and CI usage instructions Guest/no_std crates are excluded from coverage because they define coverage instrumentation. Coverage targets host-side crates only. Closes #1190 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Doru Blânzeanu <dblnz@pm.me>
1 parent 3517c0c commit 9622c18

File tree

3 files changed

+293
-0
lines changed

3 files changed

+293
-0
lines changed

‎.github/workflows/Coverage.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json
2+
3+
name: Weekly Coverage
4+
5+
on:
6+
pull_request:
7+
paths:
8+
- .github/workflows/Coverage.yml
9+
schedule:
10+
# Runs every Monday at 06:00 UTC
11+
- cron: '0 6 * * 1'
12+
workflow_dispatch: # Allow manual trigger
13+
14+
env:
15+
CARGO_TERM_COLOR: always
16+
RUST_BACKTRACE: full
17+
18+
permissions:
19+
contents: read
20+
21+
defaults:
22+
run:
23+
shell: bash
24+
25+
jobs:
26+
coverage:
27+
timeout-minutes: 90
28+
strategy:
29+
fail-fast: false
30+
matrix:
31+
hypervisor: [kvm]
32+
cpu: [amd]
33+
runs-on: ${{ fromJson(
34+
format('["self-hosted", "Linux", "X64", "1ES.Pool=hld-{0}-{1}"]',
35+
matrix.hypervisor,
36+
matrix.cpu)) }}
37+
steps:
38+
- uses: actions/checkout@v6
39+
40+
- uses: hyperlight-dev/ci-setup-workflow@v1.8.0
41+
with:
42+
rust-toolchain: "1.89"
43+
env:
44+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
45+
46+
- name: Fix cargo home permissions
47+
run: |
48+
sudo chown -R $(id -u):$(id -g) /opt/cargo || true
49+
50+
- name: Rust cache
51+
uses: Swatinem/rust-cache@v2
52+
with:
53+
shared-key: "${{ runner.os }}-debug"
54+
cache-on-failure: "true"
55+
56+
- name: Build guest binaries
57+
run: just guests
58+
59+
- name: Install nightly toolchain
60+
run: |
61+
rustup toolchain install nightly
62+
rustup component add llvm-tools --toolchain nightly
63+
64+
- name: Generate coverage report
65+
run: just coverage-ci ${{ matrix.hypervisor }}
66+
67+
- name: Coverage summary
68+
if: always()
69+
run: |
70+
echo '## Code Coverage Report' >> $GITHUB_STEP_SUMMARY
71+
echo '' >> $GITHUB_STEP_SUMMARY
72+
if [ -f target/coverage/summary.txt ]; then
73+
echo '```' >> $GITHUB_STEP_SUMMARY
74+
cat target/coverage/summary.txt >> $GITHUB_STEP_SUMMARY
75+
echo '```' >> $GITHUB_STEP_SUMMARY
76+
else
77+
echo 'Coverage report was not generated.' >> $GITHUB_STEP_SUMMARY
78+
fi
79+
echo '' >> $GITHUB_STEP_SUMMARY
80+
echo '> For a detailed per-file breakdown, download the **HTML coverage report** from the Artifacts section below.' >> $GITHUB_STEP_SUMMARY
81+
82+
- name: Upload HTML coverage report
83+
if: always()
84+
uses: actions/upload-artifact@v7
85+
with:
86+
name: coverage-html-${{ matrix.hypervisor }}-${{ matrix.cpu }}
87+
path: target/coverage/html/
88+
if-no-files-found: error
89+
90+
- name: Upload LCOV coverage report
91+
if: always()
92+
uses: actions/upload-artifact@v7
93+
with:
94+
name: coverage-lcov-${{ matrix.hypervisor }}-${{ matrix.cpu }}
95+
path: target/coverage/lcov.info
96+
if-no-files-found: error
97+
98+
notify-failure:
99+
runs-on: ubuntu-latest
100+
needs: [coverage]
101+
if: always() && needs.coverage.result == 'failure'
102+
permissions:
103+
issues: write
104+
steps:
105+
- name: Checkout code
106+
uses: actions/checkout@v6
107+
108+
- name: Notify Coverage Failure
109+
run: ./dev/notify-ci-failure.sh --title="Weekly Coverage Failure - ${{ github.run_number }}" --labels="area/ci-periodics,area/testing,lifecycle/needs-review"
110+
env:
111+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎Justfile‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,84 @@ fuzz-trace-timed max_time fuzz-target="fuzz_guest_trace":
459459
build-trace-fuzzers:
460460
cargo +nightly fuzz build fuzz_guest_trace --features trace
461461

462+
####################
463+
### COVERAGE #######
464+
####################
465+
466+
# install cargo-llvm-cov if not already installed and ensure nightly toolchain + llvm-tools are available
467+
ensure-cargo-llvm-cov:
468+
command -v cargo-llvm-cov >/dev/null 2>&1 || cargo install cargo-llvm-cov
469+
rustup toolchain install nightly 2>/dev/null || true
470+
rustup component add llvm-tools --toolchain nightly 2>/dev/null || true
471+
472+
# host-side packages to collect coverage for (guest/no_std crates are excluded because they
473+
# define #[panic_handler] and cannot be compiled for the host target under coverage instrumentation)
474+
coverage-packages := "-p hyperlight-common -p hyperlight-host -p hyperlight-testing -p hyperlight-component-util -p hyperlight-component-macro"
475+
476+
# run all tests and examples with coverage instrumentation, collecting profdata without
477+
# generating a report. Mirrors test-like-ci + run-examples-like-ci to exercise all code paths
478+
# across all feature combinations. Uses nightly for doc tests and branch coverage.
479+
# (run `just guests` first to build guest binaries)
480+
coverage-run hypervisor="kvm": ensure-cargo-llvm-cov
481+
cargo +nightly llvm-cov clean --workspace
482+
483+
@# tests with default features (all drivers; skip stress tests — too slow under instrumentation)
484+
cargo +nightly llvm-cov {{ coverage-packages }} --branch --no-report -- --skip stress_test
485+
486+
@# tests with single driver + build-metadata
487+
cargo +nightly llvm-cov {{ coverage-packages }} --branch --no-default-features --features build-metadata,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report -- --skip stress_test
488+
489+
@# isolated tests (require running separately due to global state)
490+
cargo +nightly llvm-cov -p hyperlight-host --branch --no-report --lib -- sandbox::uninitialized::tests::test_log_trace --exact --ignored
491+
cargo +nightly llvm-cov -p hyperlight-host --branch --no-report --lib -- sandbox::outb::tests::test_log_outb_log --exact --ignored
492+
cargo +nightly llvm-cov -p hyperlight-host --branch --no-report --test integration_test -- log_message --exact --ignored
493+
cargo +nightly llvm-cov -p hyperlight-host --branch --no-default-features -F function_call_metrics,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report --lib -- metrics::tests::test_metrics_are_emitted --exact
494+
495+
@# integration test with executable_heap feature
496+
cargo +nightly llvm-cov {{ coverage-packages }} --branch --no-default-features -F executable_heap,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report --test integration_test -- execute_on_heap
497+
498+
@# doc tests (requires nightly; excludes component crates which have broken doc tests)
499+
cargo +nightly llvm-cov -p hyperlight-common -p hyperlight-host -p hyperlight-testing --branch --no-report --doctests -- --skip stress_test
500+
501+
@# crashdump tests + example
502+
cargo +nightly llvm-cov {{ coverage-packages }} --branch --no-default-features --features crashdump,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report -- test_crashdump
503+
cargo +nightly llvm-cov --branch --no-default-features --features crashdump,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report --example crashdump
504+
505+
@# tracing feature tests (host-side only; hyperlight-guest-tracing is no_std)
506+
cargo +nightly llvm-cov -p hyperlight-common --branch --no-default-features --features trace_guest --no-report -- --skip stress_test
507+
cargo +nightly llvm-cov -p hyperlight-host --branch --no-default-features --features trace_guest,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report -- --skip stress_test
508+
509+
@# examples: metrics, logging, tracing
510+
cargo +nightly llvm-cov --branch --no-report --example metrics
511+
cargo +nightly llvm-cov --branch --no-default-features -F function_call_metrics,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report --example metrics
512+
cargo +nightly llvm-cov --branch --no-report --example logging
513+
cargo +nightly llvm-cov --branch --no-report --example tracing
514+
cargo +nightly llvm-cov --branch --no-default-features -F function_call_metrics,{{ if hypervisor == "mshv3" { "mshv3" } else { "kvm" } }} --no-report --example tracing
515+
516+
# generate a text coverage summary to stdout
517+
# for this to work you need to run `coverage-run hypervisor` before hand
518+
coverage hypervisor="kvm":
519+
cargo +nightly llvm-cov report
520+
521+
# generate an HTML coverage report to target/coverage/html/
522+
# for this to work you need to run `coverage-run hypervisor` before hand
523+
coverage-html hypervisor="kvm":
524+
cargo +nightly llvm-cov report --html --output-dir target/coverage/html
525+
526+
# generate LCOV coverage output to target/coverage/lcov.info
527+
# for this to work you need to run `coverage-run hypervisor` before hand
528+
coverage-lcov hypervisor="kvm":
529+
mkdir -p target/coverage
530+
cargo +nightly llvm-cov report --lcov --output-path target/coverage/lcov.info
531+
532+
# generate all coverage reports for CI: HTML + LCOV + text summary.
533+
# (run `just guests` first to build guest binaries)
534+
coverage-ci hypervisor="kvm": (coverage-run hypervisor)
535+
mkdir -p target/coverage
536+
cargo +nightly llvm-cov report --html --output-dir target/coverage/html
537+
cargo +nightly llvm-cov report --lcov --output-path target/coverage/lcov.info
538+
cargo +nightly llvm-cov report | tee target/coverage/summary.txt
539+
462540
###################
463541
### FLATBUFFERS ###
464542
###################

‎docs/how-to-run-coverage.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# How to Run Coverage
2+
3+
This guide explains how to generate code coverage reports for Hyperlight.
4+
5+
## Prerequisites
6+
7+
- A working Rust toolchain
8+
- Rust nightly toolchain (required for doc test coverage and branch coverage; installed automatically by the `just` recipes)
9+
- [cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov) (installed automatically by the `just` recipes)
10+
- Guest binaries must be built first: `just guests`
11+
12+
## Local Usage
13+
14+
Coverage is a two-step process: first **collect** profiling data by running the tests, then **generate** a report in the desired format.
15+
16+
### Step 1: Run tests with coverage instrumentation
17+
18+
Build guest binaries (required before running coverage):
19+
20+
```sh
21+
just guests
22+
```
23+
24+
Run all tests and examples under coverage instrumentation:
25+
26+
```sh
27+
just coverage-run
28+
```
29+
30+
This collects profiling data without generating a report. You only need to run this once — you can then generate multiple report formats from the same data.
31+
32+
### Step 2: Generate a report
33+
34+
#### Text Summary
35+
36+
Print a coverage summary to the terminal:
37+
38+
```sh
39+
just coverage
40+
```
41+
42+
#### HTML Report
43+
44+
Generate a browsable HTML report in `target/coverage/html/`:
45+
46+
```sh
47+
just coverage-html
48+
```
49+
50+
Open `target/coverage/html/index.html` in a browser to explore per-file and per-line coverage.
51+
52+
#### LCOV Output
53+
54+
Generate an LCOV file at `target/coverage/lcov.info` for use with external tools or CI integrations:
55+
56+
```sh
57+
just coverage-lcov
58+
```
59+
60+
## Available Recipes
61+
62+
| Recipe | Output | Description |
63+
|---|---|---|
64+
| `just coverage-run` | profiling data | Runs tests with coverage instrumentation (must be run first) |
65+
| `just coverage` | stdout | Text summary of line coverage |
66+
| `just coverage-html` | `target/coverage/html/` | HTML report for browsing |
67+
| `just coverage-lcov` | `target/coverage/lcov.info` | LCOV format for tooling |
68+
| `just coverage-ci <hypervisor>` | All of the above | CI recipe: runs tests + generates HTML + LCOV + text summary |
69+
70+
> **Note:** `coverage`, `coverage-html`, and `coverage-lcov` require `coverage-run` to have been executed first. Only `coverage-ci` runs tests and generates all reports in a single command.
71+
72+
## CI Integration
73+
74+
Coverage runs automatically on a **weekly schedule** (every Monday at 06:00 UTC) via the `Coverage.yml` workflow. It can also be triggered manually from the Actions tab using `workflow_dispatch`. The workflow runs on a single configuration (kvm/amd) to keep resource usage reasonable. It:
75+
76+
1. Builds guest binaries (`just guests`)
77+
2. Runs `just coverage-ci kvm` — this mirrors `test-like-ci` by running multiple test phases with different feature combinations and merging the results into a single coverage report
78+
3. Displays a coverage summary directly in the **GitHub Actions Job Summary** (visible on the workflow run page)
79+
4. Uploads the full HTML report and LCOV file as downloadable build artifacts
80+
81+
### Viewing Coverage Results
82+
83+
- **Quick view**: Open the workflow run in the Actions tab — the coverage table is displayed in the **Job Summary** section at the bottom of the run page.
84+
- **Detailed view**: Download the `coverage-html-*` artifact from the Artifacts section, extract the ZIP, and open `index.html` in a browser for per-file, per-line drill-down.
85+
- **Tooling integration**: Download the `coverage-lcov-*` artifact for use with IDE plugins, Codecov, Coveralls, or other coverage services.
86+
87+
## How It Works
88+
89+
`cargo-llvm-cov` instruments Rust code using LLVM's source-based code coverage. It replaces `cargo test` — when you run `cargo llvm-cov`, it compiles the project with coverage instrumentation, runs the test suite, and then merges the raw profiling data into a human-readable report. The nightly toolchain is used to enable **branch coverage** (`--branch`) and **doc test coverage** (`--doctests`).
90+
91+
The `coverage-run` recipe mirrors the `test-like-ci` + `run-examples-like-ci` workflows by running all test phases and examples with different feature combinations:
92+
93+
1. **Default features** — all drivers enabled (kvm + mshv3 + build-metadata)
94+
2. **Single driver** — only one hypervisor driver + build-metadata
95+
3. **Isolated tests** — tests that require running separately due to global state
96+
4. **Integration tests** — including `executable_heap` feature
97+
5. **Doc tests** — requires nightly
98+
6. **Crashdump** — tests + example with the `crashdump` feature enabled
99+
7. **Tracing** — tests with `trace_guest` feature (host-side crates only)
100+
8. **Examples** — metrics, logging, tracing (with and without `function_call_metrics`)
101+
102+
Each phase uses `--no-report` to accumulate raw profiling data. The report recipes (`coverage`, `coverage-html`, `coverage-lcov`) then generate the desired output format from the collected data. `coverage-ci` combines both steps into a single command.
103+
104+
Coverage is collected for the host-side workspace crates (`hyperlight_common`, `hyperlight_host`, `hyperlight_testing`, `hyperlight_component_util`, `hyperlight_component_macro`). Guest crates (`hyperlight-guest`, `hyperlight-guest-bin`, `hyperlight-guest-capi`, `hyperlight-guest-tracing`) and the `fuzz` crate are excluded because guest crates are `no_std` and cannot be compiled for the host target under coverage instrumentation.

0 commit comments

Comments
 (0)