Skip to content

Commit 9d02cde

Browse files
authored
fix: remove references to old pytest markers (#776)
1 parent c887d60 commit 9d02cde

1 file changed

Lines changed: 43 additions & 21 deletions

File tree

docs/docs/community/contributing-guide.md

Lines changed: 43 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -198,24 +198,46 @@ git commit -n -m "wip: intermediate work"
198198

199199
### Test markers
200200

201-
Tests are categorized using pytest markers:
202-
203-
| Marker | Requirement |
204-
| ------ | ----------- |
205-
| `@pytest.mark.ollama` | Ollama running locally (lightweight) |
206-
| `@pytest.mark.huggingface` | HuggingFace backend (local, heavy) |
207-
| `@pytest.mark.vllm` | vLLM backend (GPU required) |
208-
| `@pytest.mark.openai` | OpenAI API key |
209-
| `@pytest.mark.watsonx` | Watsonx API key |
210-
| `@pytest.mark.litellm` | LiteLLM backend |
211-
| `@pytest.mark.requires_gpu` | GPU available |
212-
| `@pytest.mark.requires_heavy_ram` | 48 GB+ RAM |
213-
| `@pytest.mark.requires_api_key` | External API key |
214-
| `@pytest.mark.qualitative` | LLM output quality (skipped in CI via `CICD=1`) |
215-
| `@pytest.mark.llm` | Makes LLM calls (needs at least Ollama) |
216-
| `@pytest.mark.slow` | Tests taking more than 5 minutes |
217-
218-
> **Warning:** Do not add `qualitative` to trivial tests — keep the fast loop fast. Mark tests taking more than 5 minutes with `slow`.
201+
Tests use a four-tier granularity system. Every test belongs to exactly one tier:
202+
203+
| Tier | When to use | How to apply |
204+
| ---- | ----------- | ------------ |
205+
| `unit` | Self-contained, no services, no I/O | Auto-applied — never write `@pytest.mark.unit` |
206+
| `integration` | Real SDK/library boundary or multi-component wiring | `@pytest.mark.integration` |
207+
| `e2e` | Real backends (Ollama, APIs, GPU models), deterministic assertions | `@pytest.mark.e2e` + backend marker(s) |
208+
| `qualitative` | Subset of e2e with non-deterministic output assertions | `@pytest.mark.qualitative` per-function, `e2e` + backend at module level |
209+
210+
**Backend markers** (only for e2e/qualitative tests):
211+
212+
| Marker | Backend | Resources |
213+
| ------ | ------- | --------- |
214+
| `ollama` | Ollama (port 11434) | Local, light (~2–4 GB RAM) |
215+
| `openai` | OpenAI API or compatible | API calls (may use Ollama `/v1`) |
216+
| `watsonx` | Watsonx API | API calls, requires credentials |
217+
| `huggingface` | HuggingFace transformers | Local, GPU required |
218+
| `vllm` | vLLM | Local, GPU required |
219+
| `litellm` | LiteLLM (wraps other backends) | Depends on underlying backend |
220+
| `bedrock` | AWS Bedrock | API calls, requires credentials |
221+
222+
**Resource predicates** (from `test/predicates.py`, for e2e/qualitative tests):
223+
224+
| Predicate | Use when test needs |
225+
| --------- | ------------------- |
226+
| `require_gpu()` | Any GPU (CUDA or MPS) |
227+
| `require_gpu(min_vram_gb=N)` | GPU with at least N GB VRAM |
228+
| `require_ram(min_gb=N)` | N GB+ system RAM |
229+
| `require_api_key("ENV_VAR")` | Specific API credentials |
230+
| `require_package("pkg")` | Optional dependency |
231+
| `require_python((3, 11))` | Minimum Python version |
232+
233+
**Other markers:**
234+
235+
| Marker | Purpose |
236+
| ------ | ------- |
237+
| `slow` | Tests taking >1 minute (excluded by default) |
238+
| `qualitative` | Non-deterministic output (skipped when `CICD=1`) |
239+
240+
For more information, see our [Markers Guide](https://github.com/generative-computing/mellea/blob/main/test/MARKERS_GUIDE.md).
219241

220242
### Running tests
221243

@@ -239,8 +261,8 @@ uv run pytest -m slow
239261
uv run pytest -m "ollama"
240262
uv run pytest -m "openai"
241263

242-
# Run tests without LLM calls (unit tests only)
243-
uv run pytest -m "not llm"
264+
# Run unit tests only (no backends needed)
265+
uv run pytest -m unit
244266

245267
# CI/CD mode (skips qualitative tests)
246268
CICD=1 uv run pytest
@@ -252,7 +274,7 @@ CICD=1 uv run pytest
252274
| --- | -------- |
253275
| Fast tests (`-m "not qualitative"`) | ~2 minutes |
254276
| Default (qualitative, no slow) | Several minutes |
255-
| Slow tests (`-m slow`) | More than 5 minutes |
277+
| Slow tests (`-m slow`) | More than 1 minute |
256278
| Pre-commit hooks | 1–5 minutes |
257279

258280
### Replicate CI locally

0 commit comments

Comments
 (0)