Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
Changelog
---------

Unreleased
~~~~~~~~~~

* Per-function source hashing for incremental cache invalidation — only re-test mutants in functions that changed

* Cross-call dependency tracking — invalidate mutants in callers when a called function changes

* Use git to detect non-Python dependency file changes; falls back to a curated file list when git is unavailable

* Add `cache_invalidation_exclude` config to suppress noisy files from change detection

* Add `use_git_change_detection` config (default true) to opt out of git-based detection

* Invalidate cached results automatically when result-affecting config fields change


3.6.0
~~~~~

Expand Down
71 changes: 71 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,77 @@ You can add and override pytest arguments:
also_copy = ["mutmut_pytest.ini"]


Detecting dependency and config changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Between runs, mutmut only re-tests mutants in functions whose source changed.
Changes outside your Python source — a dependency upgrade, a data file, a
config file — cannot be tied to a function, so they would otherwise be missed
and you would get cached results that no longer reflect reality.

To catch this, mutmut detects non-Python files that changed since the last full
run and warns you about them. If your project is a git repository and git is
installed, mutmut uses git (a soft dependency no extra package is required) to
find every changed non-Python file, respecting your `.gitignore`. Python files
are excluded because their changes are already tracked per function.

On a full run with git available, mutmut also records the content hashes of the
tracked non-Python files. This means a later run in an environment without git
(for example a different CI stage) can still detect changes to that known set of
files, even though it cannot discover brand-new ones.

When git is unavailable, mutmut falls back to hashing a curated set of build and
dependency files:

- `pyproject.toml`
- `setup.cfg`
- `setup.py`
- `requirements*.txt`
- `poetry.lock`
- `uv.lock`
- `Pipfile`
- `Pipfile.lock`

You can watch additional files (for example data files your tests depend on)
with the `cache_invalidation_files` config, which accepts glob patterns
resolved against the project root. These are checked even when git ignores them,
and are never dropped by the exclusions below:

.. code-block:: toml

cache_invalidation_files = [ "queries/*.sql", "config/*.yaml" ]

Git detection reports every changed non-Python file, so mutmut drops files that
practically never affect tests (markdown, `LICENSE`, `CHANGELOG`, `docs/`, git
and editor metadata, ...). Exclude additional noisy files with
`cache_invalidation_exclude` (glob patterns, `*` spans directories):

.. code-block:: toml

cache_invalidation_exclude = [ "*.json", "fixtures/snapshots/*" ]

When a watched file changes, `on_dependency_change` controls what happens:

- `warn` (default): list the changed files and keep the cache.
- `rerun`: re-test all mutants.
- `ignore`: do nothing.

.. code-block:: toml

on_dependency_change = "warn"

Git detection is on by default; disable it (forcing the curated-list fallback)
with:

.. code-block:: toml

use_git_change_detection = false

Changes to mutmut's own result-affecting config (such as `pytest_add_cli_args`,
`type_check_command`, or the timeout settings) are always detected and
invalidate the affected cached results automatically.


Unstable configs
~~~~~~~~~~~~~~~~

Expand Down
60 changes: 60 additions & 0 deletions e2e_projects/benchmark_1k/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Benchmark 1K

A synthetic benchmark project with 1000 mutants for validating mutmut's process isolation and hot-fork warmup strategy performance.

**TL;DR:**
- `fork` is fastest and nearly immune to import delays (requires fork-safe libraries)
- `collect` (hot-fork default) is 2-9x faster than `import`/`none` depending on import cost
- Higher import delays dramatically penalize `import` and `none` strategies


## Mutant Distribution

| Type | Total | Killed | Survived | Kill Rate |
|------------|-------|--------|----------|-----------|
| return | 221 | 161 | 60 | 73% |
| number | 159 | 99 | 60 | 62% |
| argument | 141 | 132 | 9 | 94% |
| string | 125 | 78 | 47 | 62% |
| boolean | 120 | 47 | 73 | 39% |
| comparison | 119 | 19 | 100 | 16% |
| operator | 115 | 90 | 25 | 78% |
| **Total** | **1000** | **626** | **374** | **63%** |

## Usage

### Run mutation testing

```bash
cd e2e_projects/benchmark_1k
mutmut run
```

### Run benchmark comparison

```bash
python run_benchmark.py
```

This runs `mutmut run` under each strategy (`fork`, `collect`, `import`, `none`) and outputs:
- Throughput (mutations/second) for each strategy
- Results saved to `benchmark_results.json`

### View results

```bash
cat mutants/summary.json | python -m json.tool
```

## Test Design

Tests are fast unit tests with instant assertions. Configurable delays simulate real-world costs:

- **Import delay**: Simulates library import time (Flask, SQLAlchemy, etc.)
- **Conftest delay**: Simulates fixture/plugin setup time
- **Test delay**: Per-test runtime with +/-10% gaussian jitter for realistic variance

Usage:
```bash
python run_benchmark.py --test-delay 0.01 # Add 10ms per-test with jitter
```
Loading
Loading