@@ -198,24 +198,46 @@ git commit -n -m "wip: intermediate work"
198198
199199### Test markers
200200
201- Tests are categorized using pytest markers:
202-
203- | Marker | Requirement |
204- | ------ | ----------- |
205- | ` @pytest.mark.ollama ` | Ollama running locally (lightweight) |
206- | ` @pytest.mark.huggingface ` | HuggingFace backend (local, heavy) |
207- | ` @pytest.mark.vllm ` | vLLM backend (GPU required) |
208- | ` @pytest.mark.openai ` | OpenAI API key |
209- | ` @pytest.mark.watsonx ` | Watsonx API key |
210- | ` @pytest.mark.litellm ` | LiteLLM backend |
211- | ` @pytest.mark.requires_gpu ` | GPU available |
212- | ` @pytest.mark.requires_heavy_ram ` | 48 GB+ RAM |
213- | ` @pytest.mark.requires_api_key ` | External API key |
214- | ` @pytest.mark.qualitative ` | LLM output quality (skipped in CI via ` CICD=1 ` ) |
215- | ` @pytest.mark.llm ` | Makes LLM calls (needs at least Ollama) |
216- | ` @pytest.mark.slow ` | Tests taking more than 5 minutes |
217-
218- > ** Warning:** Do not add ` qualitative ` to trivial tests — keep the fast loop fast. Mark tests taking more than 5 minutes with ` slow ` .
201+ Tests use a four-tier granularity system. Every test belongs to exactly one tier:
202+
203+ | Tier | When to use | How to apply |
204+ | ---- | ----------- | ------------ |
205+ | ` unit ` | Self-contained, no services, no I/O | Auto-applied — never write ` @pytest.mark.unit ` |
206+ | ` integration ` | Real SDK/library boundary or multi-component wiring | ` @pytest.mark.integration ` |
207+ | ` e2e ` | Real backends (Ollama, APIs, GPU models), deterministic assertions | ` @pytest.mark.e2e ` + backend marker(s) |
208+ | ` qualitative ` | Subset of e2e with non-deterministic output assertions | ` @pytest.mark.qualitative ` per-function, ` e2e ` + backend at module level |
209+
210+ ** Backend markers** (only for e2e/qualitative tests):
211+
212+ | Marker | Backend | Resources |
213+ | ------ | ------- | --------- |
214+ | ` ollama ` | Ollama (port 11434) | Local, light (~ 2–4 GB RAM) |
215+ | ` openai ` | OpenAI API or compatible | API calls (may use Ollama ` /v1 ` ) |
216+ | ` watsonx ` | Watsonx API | API calls, requires credentials |
217+ | ` huggingface ` | HuggingFace transformers | Local, GPU required |
218+ | ` vllm ` | vLLM | Local, GPU required |
219+ | ` litellm ` | LiteLLM (wraps other backends) | Depends on underlying backend |
220+ | ` bedrock ` | AWS Bedrock | API calls, requires credentials |
221+
222+ ** Resource predicates** (from ` test/predicates.py ` , for e2e/qualitative tests):
223+
224+ | Predicate | Use when test needs |
225+ | --------- | ------------------- |
226+ | ` require_gpu() ` | Any GPU (CUDA or MPS) |
227+ | ` require_gpu(min_vram_gb=N) ` | GPU with at least N GB VRAM |
228+ | ` require_ram(min_gb=N) ` | N GB+ system RAM |
229+ | ` require_api_key("ENV_VAR") ` | Specific API credentials |
230+ | ` require_package("pkg") ` | Optional dependency |
231+ | ` require_python((3, 11)) ` | Minimum Python version |
232+
233+ ** Other markers:**
234+
235+ | Marker | Purpose |
236+ | ------ | ------- |
237+ | ` slow ` | Tests taking >1 minute (excluded by default) |
238+ | ` qualitative ` | Non-deterministic output (skipped when ` CICD=1 ` ) |
239+
240+ For more information, see our [ Markers Guide] ( https://github.com/generative-computing/mellea/blob/main/test/MARKERS_GUIDE.md ) .
219241
220242### Running tests
221243
@@ -239,8 +261,8 @@ uv run pytest -m slow
239261uv run pytest -m " ollama"
240262uv run pytest -m " openai"
241263
242- # Run tests without LLM calls (unit tests only )
243- uv run pytest -m " not llm "
264+ # Run unit tests only (no backends needed )
265+ uv run pytest -m unit
244266
245267# CI/CD mode (skips qualitative tests)
246268CICD=1 uv run pytest
@@ -252,7 +274,7 @@ CICD=1 uv run pytest
252274| --- | -------- |
253275| Fast tests (` -m "not qualitative" ` ) | ~ 2 minutes |
254276| Default (qualitative, no slow) | Several minutes |
255- | Slow tests (` -m slow ` ) | More than 5 minutes |
277+ | Slow tests (` -m slow ` ) | More than 1 minute |
256278| Pre-commit hooks | 1–5 minutes |
257279
258280### Replicate CI locally
0 commit comments