|
| 1 | +# Testing Guidelines |
| 2 | + |
| 3 | +The existing suite already exercises uploads, conversions, compression, |
| 4 | +redaction, metadata queries, and file utilities through both synchronous and |
| 5 | +asynchronous clients. The expectations below condense every technique we rely on |
| 6 | +so new endpoints launch with complete coverage on the first pass—no reviewer |
| 7 | +iteration required. |
| 8 | + |
| 9 | +## Core Principles |
| 10 | + |
| 11 | +- **Cover both transports everywhere.** Write distinct sync (`PdfRestClient`) |
| 12 | + and async (`AsyncPdfRestClient`) tests for every scenario—such as happy paths, |
| 13 | + request customization, validation failures, file helpers, and live calls. Do |
| 14 | + not hide the transport behind a parameter; the test name itself should reveal |
| 15 | + which client is under test. |
| 16 | +- **Exercise both sides of the contract.** Hermetic tests (via |
| 17 | + `httpx.MockTransport`) validate serialization and local validation. Live |
| 18 | + suites prove the server behaves the same way, including invalid literal |
| 19 | + handling. |
| 20 | +- **Reset global state per test.** Use |
| 21 | + `monkeypatch.delenv("PDFREST_API_KEY", raising=False)` (or `setenv`) so |
| 22 | + clients never inherit accidental API keys. Patch `importlib.metadata.version` |
| 23 | + when asserting SDK headers. |
| 24 | +- **Lean on shared helpers.** Reuse `tests/graphics_test_helpers.py` |
| 25 | + (`make_pdf_file`, `build_file_info_payload`, `PdfRestFileID.generate()`), |
| 26 | + `tests/resources/`, and fixtures from `tests/conftest.py` to keep payloads |
| 27 | + deterministic. |
| 28 | +- **Assert behaviour, not just invocation.** Validate outbound payloads, |
| 29 | + headers, query params, response contents, warnings, and timeouts. Track |
| 30 | + request counts (`seen = {"post": 0, "get": 0}`) so redundant HTTP calls fail |
| 31 | + loudly. |
| 32 | + |
| 33 | +## Environment & Configuration Coverage |
| 34 | + |
| 35 | +- Verify API keys sourced from kwargs vs environment variables, and ensure |
| 36 | + invalid/missing keys raise `PdfRestConfigurationError`. |
| 37 | +- Confirm SDK identity headers (`wsn`, `User-Agent`, `Accept`) by patching |
| 38 | + `importlib.metadata.version`. |
| 39 | +- Assert `PdfRestClient` omits `Api-Key` when pointed at custom hosts and honors |
| 40 | + caller-provided headers/query params even for control-plane calls like |
| 41 | + `.up()`. |
| 42 | + |
| 43 | +## Mocked (Unit) Tests |
| 44 | + |
| 45 | +### Transports & Request Inspection |
| 46 | + |
| 47 | +- Use `httpx.MockTransport` handlers that assert: |
| 48 | + - HTTP method + path (`/png`, `/compressed-pdf`, `/resource/{id}`, etc.). |
| 49 | + - Query parameters and headers (trace/debug flags, mode switches, custom auth |
| 50 | + headers). |
| 51 | + - JSON payloads obtained via `json.loads(request.content)` and compared to the |
| 52 | + relevant Pydantic payload’s |
| 53 | + `.model_dump(mode="json", by_alias=True, exclude_none=True, exclude_unset=True)`. |
| 54 | +- For “should not happen” cases (invalid IDs, missing profiles), set the |
| 55 | + transport to immediately raise |
| 56 | + (`lambda request: (_ for _ in ()).throw(RuntimeError)` or `pytest.fail`) so |
| 57 | + local validation is guaranteed. |
| 58 | + |
| 59 | +### Error Translation & Retries |
| 60 | + |
| 61 | +- Simulate 4xx/5xx responses and assert the correct exception surfaces: |
| 62 | + `PdfRestAuthenticationError` (401/403), `PdfRestApiError` (other status codes, |
| 63 | + include error text), `PdfRestTimeoutError` (raise `httpx.TimeoutException`), |
| 64 | + `PdfRestTransportError` (raise `httpx.TransportError`). |
| 65 | +- Cover retry logic by returning retriable responses multiple times and |
| 66 | + confirming the client retries before raising (see `tests/test_client.py` |
| 67 | + patterns for `Retry-After`). |
| 68 | +- Include `pytest.raises(..., match="...")` to ensure exception messages capture |
| 69 | + server warnings, retry hints, or timeout wording. |
| 70 | + |
| 71 | +### Sync vs Async Coverage |
| 72 | + |
| 73 | +- Sync tests wrap clients in `with PdfRestClient(...):`. |
| 74 | +- Async tests use `@pytest.mark.asyncio` and |
| 75 | + `async with AsyncPdfRestClient(...):`. |
| 76 | +- When asserting async failures, place `pytest.raises` inside the `async with` |
| 77 | + block. Python forbids mixing `with` and `async with` in a single statement. |
| 78 | + |
| 79 | +### Request Customization |
| 80 | + |
| 81 | +- For every endpoint that accepts `extra_query`, `extra_headers`, `extra_body`, |
| 82 | + or `timeout`, add explicit tests (sync + async) proving those options |
| 83 | + propagate. Capture `request.extensions["timeout"]` and assert every component |
| 84 | + equals `pytest.approx(expected)`. |
| 85 | + |
| 86 | +### Validation & Payload Modeling |
| 87 | + |
| 88 | +- Use the payload models directly (`PngPdfRestPayload`, `PdfCompressPayload`, |
| 89 | + `PdfMergePayload`, `PdfSplitPayload`, `PdfRedactionApplyPayload`, etc.) to |
| 90 | + assert serialization, output-prefix validation, and range normalization in |
| 91 | + isolation from the client. |
| 92 | +- Through the client surface, pair calls with |
| 93 | + `pytest.raises(ValidationError, match="...")` for MIME enforcement (“Must be a |
| 94 | + PDF file”), dependency rules (“compression_level 'custom' requires a |
| 95 | + profile”), profile MIME validation (“Profile must be a JSON file”), and list |
| 96 | + length bounds. |
| 97 | +- Cover all accepted literal shapes: single literal vs list vs tuple for |
| 98 | + `PdfInfoQuery`; dict vs tuple vs sequence for `PdfMergeInput`; tuple/list/None |
| 99 | + for `PdfRGBColor`; JSON-friendly dicts for redaction instructions. |
| 100 | + |
| 101 | +### Enumerations, Numeric Ranges, and Options |
| 102 | + |
| 103 | +- Use `pytest.mark.parametrize` with `pytest.param(..., id="friendly")` to |
| 104 | + enumerate literals such as `color_model`, `smoothing`, `compression_level`, |
| 105 | + `page_range`, merge selectors, JPEG quality boundaries, or any future literal |
| 106 | + surfaced by new APIs. |
| 107 | +- Include invalid literals (such as `"extreme"` compression levels, unsupported |
| 108 | + `color_model` values, or smoothing arrays containing |
| 109 | + duplicates/more-than-allowed entries) to ensure validation errors remain |
| 110 | + descriptive—use `re.escape(...)` when asserting. |
| 111 | +- For numeric fields (resolution, DPI, percentages, counts, radii, opacity, |
| 112 | + etc.), exercise the extremes: the documented minimum/maximum, the first legal |
| 113 | + value just inside each bound, and at least one value just outside the range. |
| 114 | + Treat every `ge`, `le`, `gt`, `lt`, or `Annotated` constraint as requiring |
| 115 | + explicit boundary tests. |
| 116 | +- For textual ranges, cover ascending permutations, `"last"`, `"1-last"`, |
| 117 | + descending segments (where allowed), and disallowed selectors (such as |
| 118 | + `"even"`/`"odd"` when the server forbids them). |
| 119 | +- When endpoints expose optional payload arguments (output prefixes, diagnostics |
| 120 | + toggles, merge metadata, future knobs), include both defaulted and explicitly |
| 121 | + provided cases so serialization doesn’t regress. |
| 122 | + |
| 123 | +### Response Verification |
| 124 | + |
| 125 | +- Assert the concrete response types (`PdfRestFileBasedResponse`, `PdfRestFile`, |
| 126 | + `PdfRestInfoResponse`, etc.). |
| 127 | +- Inspect every relevant attribute: |
| 128 | + - File metadata (`name`, `type`, `size`, `url`, `warning`). |
| 129 | + - `input_id` echoes the uploaded file ID (string comparison). |
| 130 | + - `output_files` count matches the number of IDs returned by the mock service. |
| 131 | +- For file-service helpers, compare results against `_build_file_info_payload` |
| 132 | + via `_assert_file_matches_payload`. |
| 133 | + |
| 134 | +### Files API Scenarios |
| 135 | + |
| 136 | +- Uploads: assert multipart bodies include the correct number of `name="file"` |
| 137 | + parts and filenames, and that |
| 138 | + `client.files.create`/`create_from_paths`/`create_from_urls` fetch info |
| 139 | + documents afterward. |
| 140 | +- Downloads: cover `download_file`, `files.read_bytes/text/json`, and |
| 141 | + `files.write_bytes` with `tmp_path`. Confirm file contents match expected |
| 142 | + bytes. |
| 143 | +- Streaming: tests should enter `files.stream()` via `with`/`async with`, |
| 144 | + iterate over `iter_raw`, `iter_bytes`, `iter_text`, and `iter_lines`, and join |
| 145 | + chunks back to the original payload. Manage nested async context managers |
| 146 | + using `ExitStack` / `AsyncExitStack`. |
| 147 | +- ID validation: ensure malformed IDs raise before sending HTTP requests |
| 148 | + (transport should error if called). |
| 149 | + |
| 150 | +### Document, Compression, and Other Endpoint Examples |
| 151 | + |
| 152 | +- Conversions (such as `convert_to_png`, `convert_to_jpeg`, `convert_to_word`, |
| 153 | + or any future format helper) must verify payload serialization, request |
| 154 | + customization, MIME enforcement, multi-file guards, and smoothing/quality |
| 155 | + enumerations. |
| 156 | +- Compression helpers (such as `compress_pdf`) enforce the profile dependency |
| 157 | + (custom requires JSON profile; presets reject profiles) and validate MIME |
| 158 | + types for both PDFs and profiles in sync + async contexts. |
| 159 | +- Split/Merge style endpoints (such as `split_pdf` or `merge_pdfs`) should |
| 160 | + exercise tuples/dicts/lists, ensure payload serializers emit the correct |
| 161 | + parallel arrays, and include validation errors for insufficient sources or |
| 162 | + invalid page groups. |
| 163 | +- Redaction and metadata helpers (such as `preview_redactions`, |
| 164 | + `apply_redactions`, `query_pdf_info`) must cover literal shapes, optional |
| 165 | + parameters, and invalid presets. |
| 166 | +- Treat these as templates for any future API from |
| 167 | + `pdfrest_api_reference_guide.html`: identify its payload model, enumerate |
| 168 | + literals/numeric bounds, and apply the same sync+async/unit+live layering. |
| 169 | + |
| 170 | +### File Fixtures & Helpers |
| 171 | + |
| 172 | +- Generate fake uploads with `make_pdf_file`, `build_file_info_payload`, and |
| 173 | + `PdfRestFileID.generate()` to keep IDs valid. |
| 174 | +- When triggering MIME validation, fabricate `PdfRestFile` objects with |
| 175 | + deliberately incorrect `type` values (PNG for PDF-only endpoints, PDF for |
| 176 | + JSON-only profiles). |
| 177 | +- Use `_StaticStream` / `_StaticAsyncStream` from `tests/test_files.py` to |
| 178 | + simulate streaming responses without touching disk. |
| 179 | + |
| 180 | +## Live Tests |
| 181 | + |
| 182 | +- **Location & structure:** Place suites under `tests/live/` with one module per |
| 183 | + endpoint (`test_live_compress_pdf.py`, `test_live_convert_to_png.py`, |
| 184 | + `test_live_files.py`, etc.). |
| 185 | +- **Fixtures:** Reuse shared fixtures (`pdfrest_api_key`, |
| 186 | + `pdfrest_live_base_url`). Upload deterministic assets from `tests/resources/` |
| 187 | + via `create_from_paths` (or `client.files.create`) so responses are |
| 188 | + predictable. Use `pytest.fixture(scope="module"/"class")` and |
| 189 | + `pytest_asyncio.fixture` to cache uploaded PDFs/profiles for both transports. |
| 190 | +- **Sync + async parity:** Every live module should contain matching sync and |
| 191 | + async tests for success, customization, streaming, and invalid paths |
| 192 | + (compression levels, conversion options, file streaming helpers). |
| 193 | +- **Enumerate literals:** Parameterize over every accepted literal (compression |
| 194 | + levels, `color_model`, `smoothing`, merge selectors, redaction presets). Each |
| 195 | + literal should hit the server once per transport. |
| 196 | +- **Optional arguments:** Exercise options such as custom output prefixes, |
| 197 | + diagnostics toggles, merge metadata, and URL uploads. Validate the server |
| 198 | + honors them (filenames start with the user-provided prefix, warnings appear |
| 199 | + when expected). |
| 200 | +- **Negative live cases:** Override JSON via `extra_body`/`extra_query` to |
| 201 | + bypass local validation and assert `PdfRestApiError` (or the exact server |
| 202 | + exception) surfaces—for example, sending an invalid compression literal or |
| 203 | + smoothing option. |
| 204 | +- **Streaming + downloads:** In live `files` suites, cover `write_bytes`, |
| 205 | + `files.stream().iter_*`, and URL uploads. Manage nested `async with` blocks |
| 206 | + using `AsyncExitStack` to ensure resources are released. |
| 207 | +- **Assertions:** Verify file names, MIME types, sizes, warnings, and that |
| 208 | + `input_id` matches the uploaded ID. When fixtures are deterministic |
| 209 | + (`report.pdf`, `compression_profile.json`), assert exact values rather than |
| 210 | + generic truthiness. |
| 211 | +- **Resource reuse:** For `create_from_urls`, first upload files to retrieve |
| 212 | + stable URLs, then call the URL endpoint—never rely on arbitrary third-party |
| 213 | + hosts. |
| 214 | + |
| 215 | +## Error Handling Patterns |
| 216 | + |
| 217 | +- Always combine clients with `pytest.raises` (including descriptive `match=`) |
| 218 | + when testing validation or HTTP errors. For sync contexts you can use a |
| 219 | + compound `with (PdfRestClient(...) as client, pytest.raises(...)):`; for |
| 220 | + async, place `pytest.raises` inside the `async with` block. |
| 221 | +- Distinguish between: |
| 222 | + - Local validation failures (`ValidationError`, `ValueError`) that should |
| 223 | + prevent HTTP calls. |
| 224 | + - Server/transport failures (`PdfRestApiError`, `PdfRestAuthenticationError`, |
| 225 | + `PdfRestTimeoutError`, `PdfRestTransportError`). |
| 226 | +- When behaviour should short-circuit locally (bad UUIDs, empty query lists, |
| 227 | + missing profiles), configure the transport to raise if invoked so the test |
| 228 | + proves no HTTP request occurs. |
| 229 | + |
| 230 | +## Additional Expectations |
| 231 | + |
| 232 | +- **Context managers everywhere:** Treat clients and file streams as context |
| 233 | + managers so transports close cleanly. |
| 234 | +- **pytest fixtures:** Use readable fixtures (such as `client`, |
| 235 | + `uploaded_pdf_for_compression`, `live_async_file`) with appropriate scopes. |
| 236 | + Prefer `pytest.param(..., id="...")` so parametrized IDs stay intelligible. |
| 237 | +- **No real network in unit tests:** Hermetic tests must rely solely on |
| 238 | + `httpx.MockTransport`. |
| 239 | +- **ID serialization:** Confirm payloads serialize uploaded `PdfRestFile` |
| 240 | + objects as IDs via `_serialize_as_first_file_id` rather than embedding nested |
| 241 | + structures. |
| 242 | +- **Timeout propagation:** Every endpoint that accepts `timeout` needs both sync |
| 243 | + and async coverage that inspects `request.extensions["timeout"]`. |
| 244 | +- **Multi-file safeguards:** Assert endpoints that accept exactly one |
| 245 | + file/profile reject extra inputs (such as conversions or compression |
| 246 | + profiles). Conversely, endpoints that require multiple sources (such as merge |
| 247 | + operations) should test both valid (≥2) and invalid (\<2) cases. |
| 248 | +- **Shared validation suites:** When new payload shapes or validators emerge, |
| 249 | + add/update suites such as `tests/test_graphic_payload_validation.py` so every |
| 250 | + endpoint inheriting the behaviour gains coverage automatically. |
| 251 | + |
| 252 | +## Planning for Future APIs |
| 253 | + |
| 254 | +pdfRest will keep expanding. When implementing a new helper from |
| 255 | +`pdfrest_api_reference_guide.html`—whether it resembles existing conversions, |
| 256 | +merges, inspections, or something entirely new—follow this playbook: |
| 257 | + |
| 258 | +1. **Capture inputs and constraints.** Translate every documented literal, |
| 259 | + numeric range, dependency, and optional field into payload annotations/tests. |
| 260 | + Cover boundary values (minimum, maximum, first legal values inside the range, |
| 261 | + and at least one outside value). |
| 262 | +2. **Map outputs.** Determine whether the endpoint returns files, JSON, or both, |
| 263 | + and assert every returned attribute or warning. |
| 264 | +3. **Layer coverage.** For each behaviour, add sync + async unit tests (mocked) |
| 265 | + plus sync + async live tests hitting the real service with both valid and |
| 266 | + intentionally invalid requests. |
| 267 | +4. **Reuse patterns.** If the endpoint resembles an existing suite (such as |
| 268 | + conversions, redaction, compression, file uploads, metadata queries), mirror |
| 269 | + the structure and assertions to stay consistent. |
| 270 | +5. **Evolve shared tests.** Whenever a new validation rule becomes reusable—such |
| 271 | + as a fresh output-prefix constraint or numeric range validator—extend the |
| 272 | + shared helper modules and suites so future endpoints benefit automatically. |
| 273 | + |
| 274 | +Following these rules ensures new endpoints debut with deterministic unit tests |
| 275 | +and fully instrumented live coverage. Treat the existing conversion, |
| 276 | +compression, redaction, split/merge, and file suites as templates—if a behaviour |
| 277 | +exists today (or will exist tomorrow), there should either be a matching test |
| 278 | +pattern already or one added alongside the new API. |
0 commit comments