feat: add mktestdocs to catch docs/code drift (closes #1083)

Adds automated testing of Python code examples in the documentation
to prevent examples from drifting out of sync with the library.

Changes:

- Add `mktestdocs` and `pytest` to the `doc` dependency group in
  `pyproject.toml` so they are available alongside the other doc-build
  tools without pulling in the full `dev` group.

- Add `scripts/check-docs-drift.py`: a pytest-based script that uses
  `mktestdocs.grab_code_blocks()` to collect every ```python fenced
  block under `docs/`, skips any block whose first line is `# skip`,
  and executes the rest via `exec_python()`. A new taskipy task
  `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.

- Fix all ```python code blocks across `docs/` so they are correctly
  picked up by mktestdocs:
  - Remove the stray space in ``` python fences (changed to ```python)
    so that mktestdocs can identify them (it matches on the exact string
    "python" immediately after the backticks).
  - Add `save_to_file=False, log_level="error"` to `EmissionsTracker`
    and `OfflineEmissionsTracker` instantiations to avoid creating CSV
    files or noisy output during CI runs.
  - Add `# skip` as the first line of blocks that cannot run in CI
    because they depend on external services or optional heavy
    dependencies (TensorFlow, Prometheus, Logfire, Google Cloud,
    Comet ML, live CodeCarbon API).
  - Correct a `pip install` command that was incorrectly fenced as
    ```python in `comet.md`; changed to ```console.

- Update `.github/workflows/build-docs.yml` to run `docs-check-drift`
  as a step before the docs build, triggered on changes to `docs/**`,
  `mkdocs.yml`, or `scripts/check-docs-drift.py`.

- Document the drift check and the `# skip` convention in
  `CONTRIBUTING.md` under the "Build Documentation" section.

Author: cianc

.github/workflows/build-docs.yml

@@ -9,11 +9,13 @@ on:
     paths:
       - "docs/**"
       - "mkdocs.yml"
+      - "scripts/check-docs-drift.py"
   pull_request:
     branches: [master]
     paths:
       - "docs/**"
       - "mkdocs.yml"
+      - "scripts/check-docs-drift.py"
 
 jobs:
   build:
@@ -32,5 +34,8 @@ jobs:
       - name: Install dependencies
         run: uv sync --group doc
 
+      - name: Check docs code examples
+        run: uv run task docs-check-drift
+
       - name: Build docs
         run: uv run zensical build -f mkdocs.yml --clean

CONTRIBUTING.md

@@ -284,6 +284,28 @@ uv run --only-group doc task docs
 
 to regenerate the html files. For local preview with live reload, run `uv run --only-group doc task docs-serve`.
 
+#### Testing documentation code examples
+
+Python code blocks in the docs can be checked to catch examples that have drifted out of sync with the library. Run the check with:
+
+```sh
+uv run task docs-check-drift
+```
+
+This executes every ` ```python ` fenced block found under `docs/` using [mktestdocs](https://github.com/koaning/mktestdocs). The check is driven by `scripts/check-docs-drift.py`.
+
+**Adding a new code block to the docs?** Use a ` ```python ` fence so it is picked up. If the block cannot run in CI (e.g. it requires TensorFlow, a live API, or other external dependencies), add `# skip` as the first line inside the block:
+
+````markdown
+```python
+# skip
+import tensorflow as tf
+...
+```
+````
+
+Blocks marked `# skip` are excluded from the check but still rendered normally in the documentation.
+
 ### Rebase your branch on master 
 
 Before creating a PR, please make sure to rebase your branch on master to avoid merge conflicts and make the review easier. You can do it with the following command:

docs/getting-started/api.md

@@ -32,14 +32,16 @@ codecarbon monitor
 
 Or use the API in your code:
 
-``` python
+```python
+# skip
 from codecarbon import track_emissions
 
 @track_emissions(save_to_api=True)
 def train_model():
-    # GPU intensive training code  goes here
+    # GPU intensive training code goes here
+    pass
 
-if __name__ =="__main__":
+if __name__ == "__main__":
     train_model()
 ```
 
@@ -59,14 +61,16 @@ You then have to set your experiment id in CodeCarbon, with two options:
 
 In the code:
 
-``` python
+```python
 from codecarbon import track_emissions
 
 @track_emissions(
     measure_power_secs=30,
     api_call_interval=4,
     experiment_id="your experiment id",
     save_to_api=True,
+    save_to_file=False,
+    log_level="error",
 )
 def train_model():
     ...

docs/getting-started/comet.md

@@ -13,7 +13,7 @@ and more.
 To get started with the Comet-CodeCarbon integration, make sure you have
 comet-ml installed:
 
-``` python
+``` console
 pip install comet_ml>=3.2.2
 ```
 
@@ -24,7 +24,8 @@ In the
 [mnist-comet.py](https://github.com/mlco2/codecarbon/blob/master/examples/mnist-comet.py)
 example file, replace the placeholder code with your API key:
 
-``` python
+```python
+# skip
 experiment = Experiment(api_key="YOUR API KEY")
 ```
 

docs/getting-started/examples.md

@@ -13,7 +13,8 @@ automatically and printed at the end of the training.
 But you can't get them in your code, see the Context Manager section
 below for that.
 
-``` python
+```python
+# skip
 import tensorflow as tf
 from codecarbon import track_emissions
 
@@ -49,7 +50,8 @@ if __name__ == "__main__":
 We think this is the best way to use CodeCarbon. Still only two lines of
 code, and you can get the emissions in your code.
 
-``` python
+```python
+# skip
 import tensorflow as tf
 
 from codecarbon import EmissionsTracker
@@ -95,7 +97,8 @@ CodeCarbon scheduler is stopped. If you don't use
 background after your computation code has crashed, so your program will
 never finish.
 
-``` python
+```python
+# skip
 import tensorflow as tf
 
 from codecarbon import EmissionsTracker

docs/getting-started/usage.md

@@ -145,9 +145,9 @@ code base, users can instantiate a `EmissionsTracker` object and pass it
 as a parameter to function calls to start and stop the emissions
 tracking of the compute section.
 
-``` python
+```python
 from codecarbon import EmissionsTracker
-tracker = EmissionsTracker()
+tracker = EmissionsTracker(save_to_file=False, log_level="error")
 tracker.start()
 try:
      # Compute intensive code goes here
@@ -167,7 +167,8 @@ depending on the configuration, but keep running the experiment.
 If you want to monitor small piece of code, like a model inference, you
 could use the task manager:
 
-``` python
+```python
+# skip
 try:
     tracker = EmissionsTracker(project_name="bert_inference", measure_power_secs=10)
     tracker.start_task("load dataset")
@@ -193,11 +194,11 @@ to interfere with the task measurement.
 
 The `Emissions tracker` also works as a context manager.
 
-``` python
+```python
 from codecarbon import EmissionsTracker
 
-with EmissionsTracker() as tracker:
-    # Compute intensive training code goes here
+with EmissionsTracker(save_to_file=False, log_level="error") as tracker:
+    _ = 1 + 1  # Compute intensive training code goes here
 ```
 
 This mode is recommended when you want to monitor a specific block of
@@ -209,12 +210,13 @@ In case the training code base is wrapped in a function, users can use
 the decorator `@track_emissions` within the function to enable tracking
 emissions of the training code.
 
-``` python
+```python
 from codecarbon import track_emissions
 
-@track_emissions
+@track_emissions(save_to_file=False, log_level="error")
 def training_loop():
     # Compute intensive training code goes here
+    pass
 ```
 
 This mode is recommended if you have a training function.
@@ -239,9 +241,9 @@ can be found on
 Developers can use the `OfflineEmissionsTracker` object to track
 emissions as follows:
 
-``` python
+```python
 from codecarbon import OfflineEmissionsTracker
-tracker = OfflineEmissionsTracker(country_iso_code="CAN")
+tracker = OfflineEmissionsTracker(country_iso_code="CAN", save_to_file=False, log_level="error")
 tracker.start()
 # GPU intensive training code
 tracker.stop()
@@ -251,11 +253,12 @@ tracker.stop()
 
 The `OfflineEmissionsTracker` also works as a context manager
 
-``` python
+```python
 from codecarbon import OfflineEmissionsTracker
 
-with OfflineEmissionsTracker() as tracker:
-# GPU intensive training code  goes here
+with OfflineEmissionsTracker(country_iso_code="CAN", save_to_file=False, log_level="error") as tracker:
+    # GPU intensive training code goes here
+    pass
 ```
 
 ### Decorator
@@ -270,7 +273,7 @@ parameters:
 
 ```python
 from codecarbon import track_emissions
-@track_emissions(offline=True, country_iso_code="CAN")
+@track_emissions(offline=True, country_iso_code="CAN", save_to_file=False, log_level="error")
 def training_loop():
     # training code goes here
     pass
@@ -334,7 +337,8 @@ CodeCarbon is structured so that you can configure it in a hierarchical manner:
     -   script parameters will override environment variables if the same
         parameter is set in both:
 
-        ``` python
+        ```python
+        # skip
         EmissionsTracker(
            api_call_interval=4,
            save_to_api=True,
@@ -343,7 +347,7 @@ CodeCarbon is structured so that you can configure it in a hierarchical manner:
 
 Yields attributes:
 
-``` python
+```python
 {
     "measure_power_secs": 10,  # from ~/.codecarbon.config
     "save_to_file": True,   # from ./.codecarbon.config (override ~/.codecarbon.config)
@@ -382,7 +386,7 @@ export HTTPS_PROXY="http://0.0.0.0:0000"
 
 Or in your Python code:
 
-``` python
+```python
 import os
 
 os.environ["HTTPS_PROXY"] = "http://0.0.0.0:0000"

docs/index.md

@@ -38,7 +38,7 @@ Only a few lines of code:
 ```python
 from codecarbon import EmissionsTracker
 
-tracker = EmissionsTracker()
+tracker = EmissionsTracker(save_to_file=False, log_level="error")
 tracker.start()
 
 # Your code here

docs/logging/output.md

@@ -63,6 +63,7 @@ docker-compose up
 Run your EmissionsTracker as usual, with `save_to_prometheus=True`:
 
 ```python
+# skip
 tracker = OfflineEmissionsTracker(
     project_name="my_project",
     country_iso_code="USA",
@@ -86,6 +87,7 @@ CodeCarbon exposes all its metrics with the suffix `codecarbon_`.
 Run your EmissionsTracker as usual, with `save_to_logfire=True`:
 
 ```python
+# skip
 tracker = OfflineEmissionsTracker(
     project_name="my_project",
     country_iso_code="USA",

docs/logging/to_logger.md

@@ -20,7 +20,8 @@ or cloud-based collector.
 
 ### Python logger
 
-``` python
+```python
+# skip
 import logging
 
 # Create a dedicated logger (log name can be the CodeCarbon project name for example)
@@ -40,7 +41,8 @@ my_logger = LoggerOutput(_logger, logging.INFO)
 
 ### Google Cloud Logging
 
-``` python
+```python
+# skip
 import google.cloud.logging
 
 
@@ -61,7 +63,8 @@ documentation](https://cloud.google.com/logging/docs/reference/libraries#setting
 Create an EmissionTracker saving output to the logger. Other save
 options are still usable and valid.
 
-``` python
+```python
+# skip
 tracker = EmissionsTracker(save_to_logger=True, logging_logger=my_logger)
 tracker.start()
 # Your code here

pyproject.toml

@@ -123,6 +123,8 @@ doc = [
   "zensical",
   "mike",
   "mkdocstrings[python]>=0.26",
+  "mktestdocs",
+  "pytest",
 ]
 
 [project.optional-dependencies]
@@ -191,7 +193,7 @@ test-coverage = "CODECARBON_ALLOW_MULTIPLE_RUNS=True pytest --cov --cov-report=x
 test-package-integ = "CODECARBON_ALLOW_MULTIPLE_RUNS=True python -m pytest -vv tests/"
 docs = "zensical build -f mkdocs.yml"
 docs-serve = "zensical serve -f mkdocs.yml"
-docs-check-drift = "python scripts/check-docs-drift.py"
+docs-check-drift = "pytest scripts/check-docs-drift.py -v"
 carbonboard = "python codecarbon/viz/carbonboard.py"
 
 [tool.bumpver]