-
Notifications
You must be signed in to change notification settings - Fork 0
feat(coverage): Add coverage.py code coverage collection #87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
67c60bb
feat: add coverage snapshot server for code coverage POC
sohil-kshirsagar 3261a40
feat: add ?baseline=true parameter using coverage.py analysis2
sohil-kshirsagar 2a54664
chore: add thread safety lock and clean shutdown for coverage server
sohil-kshirsagar da89503
feat: add branch coverage tracking
sohil-kshirsagar 020e2af
wip: migrate coverage to protobuf channel (Python handler timing out …
sohil-kshirsagar b145c08
fix: Python protobuf coverage handler - use 'is None' not truthiness
sohil-kshirsagar e0e6945
refactor: remove HTTP server, clean up coverage module
sohil-kshirsagar 5e4aa97
fix: prod readiness - thread-safe coverage shutdown
sohil-kshirsagar 18e8349
feat: use TUSK_COVERAGE instead of NODE_V8_COVERAGE for Python
sohil-kshirsagar 5d438df
docs: add code coverage documentation
sohil-kshirsagar 2a383b0
fix: coverage code quality improvements
sohil-kshirsagar 5b3354b
docs: clean up AI writing patterns in coverage doc
sohil-kshirsagar 4d156c4
fix: address bugbot review feedback
sohil-kshirsagar a9eeb5e
chore: update tusk-drift-schemas to >=0.1.34
sohil-kshirsagar f0723e8
fix: address lint, type check, and coverage restart safety
sohil-kshirsagar 5cc5a07
fix: remove unused imports and simplify _is_user_file return
sohil-kshirsagar 97d824e
fix: restore re-exported imports removed by mistake (BranchInfo, Cove…
sohil-kshirsagar 01a0b3a
ref: remove proto re-exports from types.py, import directly from tusk…
sohil-kshirsagar c71dd37
fix: guard coverage with REPLAY mode check, add coverage_server unit …
sohil-kshirsagar 06b6c27
fix: add coverage optional extra, fix docs install instructions, reor…
sohil-kshirsagar e1fa8c0
fix: cache branch structure from baseline for deterministic per-test …
sohil-kshirsagar 9b197f7
test: add Tusk-generated tests for coverage server and communicator h…
sohil-kshirsagar 1eb2e39
fix: move start_coverage_collection after _initialized guard, reset _…
sohil-kshirsagar c0fac85
ref: extract _group_arcs_by_line helper to deduplicate arc grouping
sohil-kshirsagar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Code Coverage (Python) | ||
|
|
||
| The Python SDK collects per-test code coverage during Tusk Drift replay using `coverage.py`. Unlike Node.js (which uses V8's built-in coverage), Python requires the `coverage` package to be installed. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ```bash | ||
| pip install coverage | ||
| # or | ||
| pip install tusk-drift[coverage] | ||
| ``` | ||
|
|
||
| If `coverage` is not installed when coverage is enabled, the SDK logs a warning and coverage is skipped. Tests still run normally. | ||
|
|
||
| ## How It Works | ||
|
|
||
| ### coverage.py Integration | ||
|
|
||
| When coverage is enabled (via `--show-coverage`, `--coverage-output`, or `coverage.enabled: true` in config), the CLI sets `TUSK_COVERAGE=true`. The SDK detects this during initialization and starts coverage.py: | ||
|
|
||
| ```python | ||
| # What the SDK does internally: | ||
| import coverage | ||
| cov = coverage.Coverage( | ||
| source=[os.path.realpath(os.getcwd())], | ||
| branch=True, | ||
| omit=["*/site-packages/*", "*/venv/*", "*/.venv/*", "*/tests/*", "*/test_*.py", "*/__pycache__/*"], | ||
| ) | ||
| cov.start() | ||
| ``` | ||
|
|
||
| Key points: | ||
| - `branch=True` enables branch coverage (arc-based tracking) | ||
| - `source` is set to the real path of the working directory (symlinks resolved) | ||
| - Third-party code (site-packages, venv) is excluded by default | ||
|
|
||
| ### Snapshot Flow | ||
|
|
||
| 1. **Baseline**: CLI sends `CoverageSnapshotRequest(baseline=true)`. The SDK: | ||
| - Calls `cov.stop()` | ||
| - Uses `cov.analysis2(filename)` for each measured file to get ALL coverable lines (statements + missing) | ||
| - Returns lines with count=0 for uncovered, count=1 for covered | ||
| - Calls `cov.erase()` then `cov.start()` to reset counters | ||
|
|
||
| 2. **Per-test**: CLI sends `CoverageSnapshotRequest(baseline=false)`. The SDK: | ||
| - Calls `cov.stop()` | ||
| - Uses `cov.get_data().lines(filename)` to get only executed lines since last reset | ||
| - Returns only covered lines (count=1) | ||
| - Calls `cov.erase()` then `cov.start()` to reset | ||
|
|
||
| 3. **Communication**: Results are sent back to the CLI via the existing protobuf channel — same socket used for replay. No HTTP server or extra ports. | ||
|
|
||
| ### Branch Coverage | ||
|
|
||
| Branch coverage uses coverage.py's arc tracking. The SDK extracts per-line branch data using: | ||
|
|
||
| ```python | ||
| analysis = cov._analyze(filename) # Private API | ||
| missing_arcs = analysis.missing_branch_arcs() | ||
| executed_arcs = set(data.arcs(filename) or []) | ||
| ``` | ||
|
|
||
| For each branch point (line with multiple execution paths), the SDK reports: | ||
| - `total`: number of branch paths from that line | ||
| - `covered`: number of paths that were actually taken | ||
|
|
||
| **Note:** `_analyze()` is a private coverage.py API. It's the only way to get per-line branch arc data. The public API (`analysis2()`) only provides aggregate branch counts. This means branch coverage may break on major coverage.py version upgrades. | ||
|
|
||
| ### Path Handling | ||
|
|
||
| The SDK uses `os.path.realpath()` for the source root to handle symlinked project directories. File paths reported by coverage.py are also resolved via `realpath` before comparison. This prevents the silent failure where all files get filtered out because symlink paths don't match. | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| Set automatically by the CLI. You should not set these manually. | ||
|
|
||
| | Variable | Description | | ||
| |----------|-------------| | ||
| | `TUSK_COVERAGE` | Set to `true` by the CLI when coverage is enabled. The SDK checks this to decide whether to start coverage.py. | | ||
|
|
||
| Note: `NODE_V8_COVERAGE` is also set by the CLI (for Node.js), but the Python SDK ignores it — it only checks `TUSK_COVERAGE`. | ||
|
|
||
| ## Thread Safety | ||
|
|
||
| Coverage collection uses a module-level lock (`threading.Lock`) to ensure thread safety: | ||
|
|
||
| - `start_coverage_collection()`: Acquires lock while initializing. Guards against double initialization — if called twice, stops the existing instance first. | ||
| - `take_coverage_snapshot()`: Acquires lock for the entire stop/read/erase/start cycle. | ||
| - `stop_coverage_collection()`: Acquires lock while stopping and cleaning up. | ||
|
|
||
| This is important because the protobuf communicator runs coverage handlers in a background thread. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **`coverage` package required**: Unlike Node.js (V8 coverage is built-in), Python needs `pip install coverage`. If not installed, coverage silently doesn't work (warning logged). | ||
| - **Performance overhead**: coverage.py uses `sys.settrace()` which adds 10-30% execution overhead. This only applies during coverage replay runs. | ||
| - **Multi-process servers**: gunicorn with `--workers > 1` forks worker processes. The SDK starts coverage.py in the main process; forked workers don't inherit it. Use `--workers 1` during coverage runs. | ||
| - **Private API for branches**: `_analyze()` is not part of coverage.py's public API. Branch coverage detail may break on future coverage.py versions. | ||
| - **Python 3.12+ recommended for async**: coverage.py's `sys.settrace` can miss some async lines on Python < 3.12. Python 3.12+ uses `sys.monitoring` for better async tracking. | ||
| - **Startup ordering**: coverage.py starts during SDK initialization. Code that executes before `TuskDrift.initialize()` (e.g., module-level code in `tusk_drift_init.py`) isn't tracked. This is why `tusk_drift_init.py` typically shows 0% coverage. | ||
| - **C extensions invisible**: coverage.py can't track C extensions (numpy, Cython modules). Not relevant for typical web API servers. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.