Skip to content

coverage.md

Latest commit

 

History

History
358 lines (270 loc) · 9.58 KB

coverage.md

File metadata and controls

358 lines (270 loc) · 9.58 KB
Raw

Code Coverage

Code coverage measures how much of your source code is executed when running tests. It helps identify untested code paths and ensures your test suite exercises the important parts of your application.

Quick Start

Enable coverage tracking with the --coverage flag:

::: code-group

bashunit tests/ --coverage
bashunit tests/ --coverage-paths src/
bashunit - 0.30.0 | Tests: 5
.....

Tests:      5 passed, 5 total
Assertions: 12 passed, 12 total

All tests passed
Time taken: 1 s

Coverage Report
---------------
src/math.sh                                2/  3 lines ( 66%)
src/utils.sh                               8/ 10 lines ( 80%)
---------------
Total: 10/13 (76%)

Coverage report written to: coverage/lcov.info

:::

How It Works

bashunit uses Bash's built-in DEBUG trap mechanism to track line execution:

  1. Trap Setup: When coverage is enabled, a DEBUG trap is set that fires before every command execution
  2. Line Recording: Each executed line's file path and line number are recorded
  3. Filtering: Only files matching your coverage paths (and not excluded) are tracked
  4. Aggregation: After tests complete, hit data is aggregated and reported

::: tip Performance The DEBUG trap adds overhead to test execution. For large test suites, consider running coverage periodically rather than on every test run. :::

Configuration

Command Line Options

Option Description
--coverage Enable code coverage tracking
--coverage-paths <paths> Comma-separated paths to track (default: auto-discover from test files)
--coverage-exclude <patterns> Comma-separated exclusion patterns
--coverage-report <file> LCOV report output path (default: coverage/lcov.info)
--coverage-report-html <dir> Generate HTML coverage report with line-by-line details
--coverage-min <percent> Minimum coverage threshold (fails if below)
--no-coverage-report Disable LCOV file generation (console only)

::: tip Auto-enable Coverage is automatically enabled when using --coverage-report, --coverage-report-html, or --coverage-min. You don't need to specify --coverage explicitly with these options. :::

Auto-Discovery

When BASHUNIT_COVERAGE_PATHS is not set, bashunit automatically discovers source files based on your test file names:

Test File Discovers
tests/unit/assert_test.sh src/assert.sh, src/assert_*.sh
tests/unit/helperTest.sh src/helper.sh, src/helper*.sh

This convention follows the common pattern of naming test files after their source files with a _test.sh or Test.sh suffix.

::: tip Zero Configuration For most projects following standard naming conventions, you can simply run bashunit tests/ --coverage without any path configuration. :::

Environment Variables

You can also configure coverage via environment variables in your .env file:

# Enable coverage
BASHUNIT_COVERAGE=true

# Paths to track (comma-separated)
BASHUNIT_COVERAGE_PATHS=src/,lib/

# Patterns to exclude (comma-separated)
BASHUNIT_COVERAGE_EXCLUDE=tests/*,vendor/*,*_test.sh

# LCOV report output path
BASHUNIT_COVERAGE_REPORT=coverage/lcov.info

# HTML report output directory (generates line-by-line coverage view)
BASHUNIT_COVERAGE_REPORT_HTML=coverage/html

# Minimum coverage percentage (optional)
BASHUNIT_COVERAGE_MIN=80

# Color thresholds for console output
BASHUNIT_COVERAGE_THRESHOLD_LOW=50   # Red below this
BASHUNIT_COVERAGE_THRESHOLD_HIGH=80  # Green above this, yellow between

Examples

Basic Coverage

Track coverage for the default src/ directory:

::: code-group

bashunit tests/ --coverage

:::

Custom Source Paths

Track multiple directories:

::: code-group

bashunit tests/ --coverage --coverage-paths "src/,lib/,bin/"
BASHUNIT_COVERAGE_PATHS=src/,lib/,bin/

:::

Exclusion Patterns

Exclude specific files or directories:

::: code-group

bashunit tests/ --coverage --coverage-exclude "vendor/*,*_mock.sh,deprecated/"
BASHUNIT_COVERAGE_EXCLUDE=vendor/*,*_mock.sh,deprecated/

:::

Setting Minimum Threshold

Fail the test run if coverage drops below a threshold:

::: code-group

bashunit tests/ --coverage-min 80
Coverage Report
---------------
src/math.sh                               10/ 12 lines ( 83%)
---------------
Total: 10/12 (83%)

Coverage Report
---------------
src/math.sh                                5/ 12 lines ( 41%)
---------------
Total: 5/12 (41%)

Coverage 41% is below minimum 80%

:::

Console-Only Output

Skip generating the LCOV file:

::: code-group

bashunit tests/ --coverage --no-coverage-report

:::

HTML Coverage Report

Generate a detailed HTML report showing line-by-line coverage:

::: code-group

bashunit tests/ --coverage-report-html coverage/html
BASHUNIT_COVERAGE_REPORT_HTML=coverage/html

:::

This creates a directory with:

  • index.html - Summary page with per-file coverage percentages
  • files/*.html - Individual source file views with line highlighting

Line highlighting:

  • Green background: Lines executed during tests (covered)
  • Red background: Executable lines not executed (uncovered)
  • No background: Non-executable lines (comments, function declarations, etc.)

Each line also shows the number of times it was executed, helping identify hot paths and dead code.

CI/CD Integration

Generate coverage for CI tools like Codecov or Coveralls:

::: code-group

- name: Run tests with coverage
  run: bashunit tests/ --coverage-min 80

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    files: ./coverage/lcov.info
    fail_ci_if_error: true
test:
  script:
    - bashunit tests/ --coverage
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/lcov.info

:::

Understanding the Console Report

The console report shows coverage per file with color coding:

Coverage Report
---------------
src/math.sh                       10/ 12 lines ( 83%)  # Green (>= 80%)
src/parser.sh                      7/ 10 lines ( 70%)  # Yellow (50-79%)
src/legacy.sh                      2/ 15 lines ( 13%)  # Red (< 50%)
---------------
Total: 19/37 (51%)

Color thresholds (configurable via environment variables):

  • Green: Coverage >= 80% (BASHUNIT_COVERAGE_THRESHOLD_HIGH)
  • Yellow: Coverage 50-79%
  • Red: Coverage < 50% (BASHUNIT_COVERAGE_THRESHOLD_LOW)

Understanding LCOV Format

The coverage/lcov.info file uses the industry-standard LCOV format, compatible with most CI coverage tools.

File Structure

TN:
SF:/path/to/source/file.sh
DA:2,5
DA:3,0
LF:2
LH:1
end_of_record

Field Reference

Field Description Example
TN: Test Name (usually empty) TN:
SF: Source File path SF:/home/user/project/src/math.sh
DA: Line Data: line_number,hit_count DA:15,3 (line 15 hit 3 times)
LF: Lines Found (total executable lines) LF:25
LH: Lines Hit (lines with hits > 0) LH:20
end_of_record Marks end of file entry end_of_record

Example Breakdown

Given this source file src/math.sh:

#!/usr/bin/env bash           # Line 1 - not executable (comment/shebang)
function add() {              # Line 2 - not executable (function declaration)
  echo $(($1 + $2))           # Line 3 - executable
}                             # Line 4 - not executable (closing brace)
function multiply() {         # Line 5 - not executable (function declaration)
  echo $(($1 * $2))           # Line 6 - executable
}                             # Line 7 - not executable (closing brace)

If tests call add twice but never call multiply, the LCOV output would be:

TN:
SF:/path/to/src/math.sh
DA:3,2
DA:6,0
LF:2
LH:1
end_of_record

Interpretation:

  • Line 3 (add body): 2 hits
  • Line 6 (multiply body): 0 hits
  • 2 executable lines found, 1 line was hit (50% coverage)

Parallel Execution

Coverage works seamlessly with parallel test execution (-p flag):

::: code-group

bashunit tests/ --coverage -p

:::

How it works:

  • Each parallel worker writes to its own coverage file
  • After all tests complete, coverage data is aggregated
  • The final report combines hits from all workers

::: tip Coverage percentages should be identical whether running in parallel or sequential mode. :::

What Gets Tracked

Executable Lines

bashunit counts these as executable lines:

  • Commands and statements
  • Single-line function bodies (function foo() { echo "hi"; })

Non-Executable Lines (Skipped)

These lines are not counted toward coverage:

  • Empty lines
  • Comment lines (including shebang #!/usr/bin/env bash)
  • Function declaration lines (function foo() {)
  • Lines with only braces ({ or })
  • Control flow keywords (then, else, fi, do, done, esac, in)
  • Case statement patterns (--option), *)) and terminators (;;, ;&, ;;&)

Limitations

External Commands

Coverage only tracks Bash code. External commands (like grep, sed, etc.) are not tracked, though the lines that call them are.

Subshell Behavior

Due to Bash's process model, some subshell contexts may not have full coverage tracking. The DEBUG trap is inherited into subshells, but complex nested scenarios may have edge cases.