Python(chore): pytest docs reorganization (#589)

Author: alexluck-sift

python/docs/examples/index.md

@@ -6,9 +6,12 @@ This section contains interactive Jupyter notebook examples demonstrating how to
 
 - **[Basic Usage](basic.ipynb)** - Introduction to the Sift Python client, covering basic operations and API usage
 - **[Data Ingestion](ingestion.ipynb)** - Learn how to ingest telemetry data into Sift using various methods
-- **[Pytest Plugin](pytest_plugin.md)** - Turn a pytest run into a Sift TestReport with measurements, nested steps, and pass/fail outcomes
 - **[Pytest Plugin Quickstart](pytest_plugin_quickstart.md)** - Guided tour of the runnable demo project under `python/examples/pytest_plugin/`
 
+For the conceptual reference on the pytest plugin (fixtures, configuration,
+report structure, and pass/fail behavior), see the
+[Pytest Plugin guide](../guides/pytest_plugin/index.md).
+
 ## Running Examples Locally
 
 To run these examples on your local machine:

python/docs/examples/pytest_plugin.md

@@ -8,8 +8,8 @@ axes, manual substeps, and gate markers. It also includes a tests directory
 that uses no Sift APIs at all, to show how the autouse fixtures capture plain
 pytest tests for free.
 
-For a conceptual reference (fixtures, ini flags, status semantics), see
-[Pytest Plugin](pytest_plugin.md).
+For a conceptual reference (fixtures, ini flags, status semantics), see the
+[Pytest Plugin guide](../guides/pytest_plugin/index.md).
 
 ## Project layout
 
@@ -172,7 +172,7 @@ Flip any of the `sift_*_step` / `sift_parametrize_nesting` flags in
 
 ## Next steps
 
-- [Pytest Plugin](pytest_plugin.md): conceptual reference covering fixtures,
-  ini flags, status semantics, and layout-mapping examples.
+- [Pytest Plugin guide](../guides/pytest_plugin/index.md): conceptual reference
+  covering fixtures, configuration, report structure, and pass/fail behavior.
 - The demo's [README](https://github.com/sift-stack/sift/blob/main/python/examples/pytest_plugin/README.md)
   on GitHub mirrors this page and is the canonical source.

python/docs/examples/pytest_plugin_quickstart.md

@@ -0,0 +1,11 @@
+# Guides
+
+Conceptual references for the Sift Python client. Guides explain how a feature
+works and how to configure it. For runnable, end-to-end walkthroughs see the
+[Examples](../examples/index.md) section.
+
+## Available guides
+
+- [Pytest Plugin](pytest_plugin/index.md): turn a pytest run into a `TestReport`
+  in Sift. Each test becomes a `TestStep`, measurements are recorded as rows, and
+  failures propagate up through nested substeps to the report.

python/docs/guides/index.md

@@ -0,0 +1,220 @@
+# Configuration & Defaults
+
+This page is the full reference for everything the plugin exposes: fixtures, CLI
+flags, ini options, credential handling, and the markers that control which
+tests report.
+
+!!! info "Where the plugin lives"
+    The plugin lives at `sift_client.pytest_plugin`. It is **not** registered as
+    a `pytest11` entry point. Projects opt in with a `pytest_plugins` declaration
+    in their top-level `conftest.py`. Pytest then loads the module as a real
+    plugin: the fixtures, CLI options, and `pytest_runtest_makereport` hook all
+    register through standard pytest machinery, so `pytest --trace-config` lists
+    it and `pytest -p no:sift_client.pytest_plugin` disables it.
+
+## Credentials
+
+Set the connection details in a `.env` next to your tests:
+
+```bash
+SIFT_API_KEY="your-api-key"
+SIFT_GRPC_URI="..."
+SIFT_REST_URI="..."
+```
+
+The `SIFT_GRPC_URI` and `SIFT_REST_URI` are the gRPC and REST endpoints for your
+Sift organization. You can find these on the Sift Manage page as well as
+generate an API key.
+
+The default `sift_client` fixture reads its two URIs from environment first and
+falls back to ini keys when the env vars are unset. `SIFT_API_KEY` is
+intentionally env-only, so keep it out of source control and supply it through
+`pytest-dotenv` (see [API key handling](#api-key-handling) below). The env var
+wins when both are set, so secrets injected into a CI environment continue to
+override values committed to `pyproject.toml`. There are no CLI flags for
+credentials.
+
+| Ini key | Environment variable | Notes |
+|---|---|---|
+| _(none)_ | `SIFT_API_KEY` | Env-only. Use `.env` + `pytest-dotenv` locally; inject from your secret store in CI. |
+| `sift_grpc_uri` | `SIFT_GRPC_URI` | Stable per-org gRPC endpoint; safe to commit. |
+| `sift_rest_uri` | `SIFT_REST_URI` | Stable per-org REST endpoint; safe to commit. |
+
+### API key handling
+
+`SIFT_API_KEY` is deliberately read from the process environment only. The
+recommended workflow uses the
+[`pytest-dotenv`](https://pypi.org/project/pytest-dotenv/) plugin (already a
+dependency of `sift-stack-py`), which loads variables from a `.env` file into
+`os.environ` before tests run.
+
+1. Add `.env` to `.gitignore`.
+2. Drop your key into `.env` at the project root:
+
+    ```bash title=".env"
+    SIFT_API_KEY=sk-...your-key...
+    ```
+
+3. In CI, set `SIFT_API_KEY` directly via your provider's secret manager
+   instead of committing a `.env` file.
+
+`pytest-dotenv` picks the file up automatically; no `pytest_configure` glue is
+needed.
+
+!!! warning "FedRAMP / shared environments"
+    Pass `--sift-log-file=false` (or set the ini key to `"false"`) to skip the
+    temp file + worker pipeline. Create/update calls then run inline against the
+    API instead of being deferred through a subprocess.
+
+## Wire the plugin into `conftest.py`
+
+A single `pytest_plugins` declaration in your top-level `conftest.py` is all
+that's required. The plugin ships a default `sift_client` fixture that reads
+`SIFT_API_KEY`, `SIFT_GRPC_URI`, and `SIFT_REST_URI` from the environment.
+
+```python title="conftest.py"
+from dotenv import load_dotenv
+
+load_dotenv()
+
+pytest_plugins = ["sift_client.pytest_plugin"]
+```
+
+That's the whole setup. Every test in the session will now create a step on a
+single shared `TestReport`.
+
+### Customizing the `SiftClient`
+
+To construct the client differently (custom TLS, timeouts, alternate
+credentials, etc.), override the `sift_client` fixture in your conftest. The
+plugin's default falls away in favor of your definition.
+
+```python title="conftest.py"
+import os
+
+import pytest
+from dotenv import load_dotenv
+
+from sift_client import SiftClient, SiftConnectionConfig
+
+load_dotenv()
+
+pytest_plugins = ["sift_client.pytest_plugin"]
+
+
+@pytest.fixture(scope="session")
+def sift_client() -> SiftClient:
+    return SiftClient(
+        connection_config=SiftConnectionConfig(
+            api_key=os.getenv("SIFT_API_KEY"),
+            grpc_url=os.getenv("SIFT_GRPC_URI"),
+            rest_url=os.getenv("SIFT_REST_URI"),
+            use_ssl=False,
+        )
+    )
+```
+
+## Plugin provided fixtures
+
+| Name | Kind | Scope | Purpose |
+|---|---|---|---|
+| `report_context` | fixture (autouse) | session | The `ReportContext` backing the run's `TestReport`. Use it to attach metadata or open ad-hoc steps. |
+| `step` | fixture (autouse) | function | A `NewStep` created for the current test function. Exposes `measure*`, `substep`, `report_outcome`, `fail_if_measurements_failed`, and `current_step`. |
+| `_hierarchy_parents` | internal fixture (autouse) | function | Opens a parent step for each `pytest.Package`, `pytest.Module`, and `pytest.Class` ancestor of the current test. Each layer is gated independently; see [ini options](#ini-options). |
+| `_parametrize_parents` | internal fixture (autouse) | function | Opens a parent step for each `@pytest.mark.parametrize` axis (and fixture parametrization), nested inside the hierarchy parents. |
+| `client_has_connection` | fixture | session | Calls `sift_client.ping.ping()`; consulted by `report_context` at session start in online mode (the default). Override to skip the ping or use a different reachability signal. |
+
+## CLI options
+
+| Flag | Default | Effect |
+|---|---|---|
+| `--sift-offline` | off (online) | Skip the session-start ping and don't contact Sift. All create/update calls go to the JSONL log file for later replay via `import-test-result-log`. Missing `SIFT_*` env vars are tolerated; placeholders are filled. |
+| `--sift-disabled` | off | Skip Sift entirely. Nothing contacts the API and no log file is written; `step.measure(...)` still evaluates bounds and returns a real pass/fail boolean. Also honored via `SIFT_DISABLED=1`. Supersedes every other flag (disabled wins over offline). |
+| `--sift-log-file=<path\|true\|false>` | temp file | Where the JSONL log of create/update calls goes. With a log file set, the plugin spawns an `import-test-result-log --incremental` worker that polls the file and replays entries against Sift while the run is in flight. Pass `false` to disable the file entirely; create/update calls then go straight to the API synchronously during tests. Incompatible with `--sift-offline` since offline mode needs the log file as its sole sink. |
+| `--no-sift-git-metadata` | git metadata on | Skip capturing git repo/branch/commit on the report's metadata. |
+
+These can be passed permanently via `addopts`:
+
+```ini title="pytest.ini"
+[pytest]
+addopts = --sift-offline
+```
+
+## Ini options
+
+Set the matching ini key directly (recommended for stable per-project
+configuration). Each CLI flag has a corresponding key under
+`[tool.pytest.ini_options]` in `pyproject.toml` or `[pytest]` in `pytest.ini`.
+CLI flags, when passed, override the ini values.
+
+| Ini key | Type | Equivalent CLI flag |
+|---|---|---|
+| `sift_log_file` | string (`true` / `false` / `none` / path) | `--sift-log-file=<value>` |
+| `sift_git_metadata` | bool (default `true`) | `--no-sift-git-metadata` (sets to `false`) |
+| `sift_offline` | bool (default `false`) | `--sift-offline` |
+| `sift_disabled` | bool (default `false`) | `--sift-disabled` (also honors `SIFT_DISABLED` env var) |
+| `sift_autouse` | bool (default `true`) | _(no CLI flag; controls the marker gate below)_ |
+| `sift_package_step` | bool (default `true`) | _(ini-only)_. Opens a parent step for each Python package (directory with `__init__.py`) in the test path. |
+| `sift_module_step` | bool (default `true`) | _(ini-only)_. Opens a parent step for each test module (file). |
+| `sift_class_step` | bool (default `true`) | _(ini-only)_. Opens a parent step for each test class, including nested classes. |
+| `sift_parametrize_nesting` | bool (default `true`) | _(ini-only)_. Clusters parametrized tests under shared parents (`test_x`, `axis=value`) instead of flat leaves (`test_x[value]`). |
+
+```toml title="pyproject.toml"
+[tool.pytest.ini_options]
+sift_offline = true
+sift_git_metadata = false
+sift_grpc_uri = "your-org.sift.example:443"
+sift_rest_uri = "https://your-org.sift.example"
+```
+
+```ini title="pytest.ini"
+[pytest]
+sift_offline = true
+sift_git_metadata = false
+sift_grpc_uri = your-org.sift.example:443
+sift_rest_uri = https://your-org.sift.example
+```
+
+## Controlling which tests produce reports
+
+By default every test in the session produces a Sift step. Two markers and one
+ini key let you narrow that to a specific set of tests, which is useful when a
+repo holds tests that you don't want included in the Sift test report.
+
+| Setting                                                 | Effect                                                                                       |
+|---------------------------------------------------------|----------------------------------------------------------------------------------------------|
+| `sift_autouse = false` in `pyproject.toml` | Flip the project-wide default off. Tests no longer produce steps unless explicitly opted in. |
+| `@pytest.mark.sift_include` on a test, class, or module | Force reporting on for that scope, regardless of the project default.                        |
+| `@pytest.mark.sift_exclude` on a test, class, or module | Force reporting off for that scope, regardless of the project default.                       |
+
+Closest marker determines setting. `sift_exclude` beats `sift_include` when both apply.
+`pytestmark` at the class or module level inherits to every test in scope.
+
+### Bulk-applying a marker to a directory
+
+To opt an entire directory in (or out) without editing each file, hook
+`pytest_collection_modifyitems` in the directory's `conftest.py`:
+
+```python title="tests/example/conftest.py"
+from pathlib import Path
+
+import pytest
+
+_HERE = Path(__file__).parent
+
+
+def pytest_collection_modifyitems(config, items):
+    for item in items:
+        try:
+            item.path.relative_to(_HERE)
+        except ValueError:
+            continue
+        item.add_marker(pytest.mark.sift_include)
+```
+
+This applies `sift_include` to every test collected under `tests/example/`.
+Combine with `sift_autouse = false` in `pyproject.toml` for opting in to
+specific directories.
+
+`pytest_collection_modifyitems` receives every item in the session, not just
+this directory's, so the `relative_to` filter is what scopes the marker.

python/docs/guides/pytest_plugin/configuration.md

@@ -0,0 +1,122 @@
+# Pytest Plugin
+
+The Sift Python client ships a pytest plugin that turns a pytest run into a
+`TestReport` in Sift. Each test function becomes a `TestStep`, measurements are presented
+as rows under that step, and failures propagate up through nested substeps to
+the report itself.
+
+## Quick start
+
+Install the client and pytest:
+
+```bash
+pip install sift-stack-py pytest python-dotenv
+```
+
+Set your connection details in a `.env` next to your tests:
+
+```bash title=".env"
+SIFT_API_KEY="..."
+SIFT_GRPC_URI="..."
+SIFT_REST_URI="..."
+```
+
+Find these on the Sift Manage page, where you can also generate an API key.
+
+Register the plugin with a single `pytest_plugins` declaration in your top-level
+`conftest.py`:
+
+```python title="conftest.py"
+from dotenv import load_dotenv
+
+load_dotenv()
+
+pytest_plugins = ["sift_client.pytest_plugin"]
+```
+
+Write a test. The `step` fixture is `autouse`, so any test becomes a step on the
+report. Take it as an argument when you want to record a measurement:
+
+```python title="test_battery.py"
+def test_battery_voltage(step):
+    step.measure(
+        name="battery_voltage",
+        value=4.97,
+        bounds={"min": 4.8, "max": 5.2},
+        unit="V",
+    )
+    step.fail_if_measurements_failed()
+```
+
+Run it:
+
+```bash
+pytest
+```
+
+A `TestReport` shows up in Sift once the session finishes.
+
+!!! tip "Fail at the end, not per measurement"
+    `step.measure(...)` returns a pass/fail boolean and never raises, so a
+    failing measurement marks the step failed without aborting the test. Take
+    every measurement first, then call `step.fail_if_measurements_failed()` once
+    at the end, so every measurement still lands in the report even when one
+    fails. It fails the test via `pytest.fail` (no assertion noise in
+    `error_info`), and unlike asserting on an individual `step.measure(...)` call
+    it does not short-circuit on the first failure and skip every measurement
+    after it.
+
+## Sensible defaults
+
+With nothing but the `conftest.py` above, you get:
+
+- **Full step tree.** Every Python package, test module, test class, and
+  parametrize axis above a test becomes a parent step, so the report mirrors
+  your test layout.
+- **Online mode.** The plugin pings Sift at session start and streams
+  create/update calls to your tenant during the run.
+- **Git metadata.** Repo, branch, and commit are captured on the report
+  automatically.
+
+Everything is on by default and individually overridable. See
+[Configuration & Defaults](configuration.md) for the full audit of every knob,
+marker, flag, and fixture.
+
+## Running modes
+
+The plugin runs in one of three modes, picked at invocation.
+
+| Mode | How to select | Contacts Sift | When to use                                                   |
+|---|---|---|---------------------------------------------------------------|
+| **Online** | default (no flag) | Yes, during the run | Default choice                                                |
+| **Offline** | `--sift-offline` | No; records to a log file for later replay | Environments without Sift access.                             |
+| **Disabled** | `--sift-disabled` | No | Local dev. Bounds still evaluate and return a real pass/fail. |
+
+Online mode pings Sift once at session start and aborts if Sift is unreachable or the credentials are invalid, 
+so a misconfigured job fails immediately instead of silently producing no report. 
+During the run, every create and update is appended to a JSONL log file. 
+A background worker uploads new entries to Sift incrementally. 
+If the connection drops mid-test, the test keeps running and the log keeps writing locally. 
+The remaining entries can be uploaded afterward by running import-test-result-log, which the plugin prints on exit.
+
+See [Running Modes](running_modes.md) for the log-file and replay pipeline,
+overriding the connection check, and replaying a saved log.
+
+## Report structure
+
+The report tree mirrors your test layout: packages, modules, classes, and
+parametrize axes nest automatically, and you can open arbitrary substeps inside
+a test. See [Report Structure](report_structure.md) for the layout-to-tree
+mapping, measurement variants, and report metadata.
+
+## Pass/fail outcomes
+
+Every pytest outcome (pass, assertion failure, exception, skip, xfail, hard
+exit) maps to a `TestStatus`, and failures roll up to the parent steps and the
+report. See [Pass/Fail Behavior](pass_fail_behavior.md).
+
+## Try the runnable demo
+
+The [Pytest Plugin Quickstart](../../examples/pytest_plugin_quickstart.md) walks
+through a self-contained demo project that exercises every layer of the step
+tree, with instructions to run it with or without a Sift tenant.