Skip to content

Commit 38ec62d

Browse files
Document Python external payload storage APIs
1 parent 8c8300e commit 38ec62d

2 files changed

Lines changed: 62 additions & 1 deletion

File tree

docs/polyglot/python.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,55 @@ async with Client(
134134
set, control-plane methods use `control_token` and worker-plane methods use
135135
`worker_token`.
136136

137+
### External Payload Storage
138+
139+
The SDK exports the same external-payload reference contract used by the server
140+
and CLI storage APIs. Use it when Python activity handlers or invocable
141+
carriers need to decode large payload references from workflow history, or when
142+
a Python process needs to create a language-neutral payload envelope without
143+
embedding large bytes inline.
144+
145+
| API | Role | Failure surface |
146+
| --- | --- | --- |
147+
| `ExternalStorageDriver` | Protocol for `put(data, sha256=..., codec=...)`, `get(uri)`, and `delete(uri)`. | Driver-raised storage errors. |
148+
| `LocalFilesystemExternalStorage` | Dependency-free `file://` driver for local development and tests. | `ValueError` when a referenced URI escapes the configured root. |
149+
| `S3ExternalStorage` | Adapter for a boto3-compatible client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
150+
| `GCSExternalStorage` | Adapter for a google-cloud-storage-style client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
151+
| `AzureBlobExternalStorage` | Adapter for an Azure container client. | `ValueError` for foreign container/prefix references or non-byte responses. |
152+
| `ExternalPayloadReference` | Immutable wire reference with `uri`, `sha256`, `size_bytes`, `codec`, and schema. | `ValueError` when `from_dict()` receives an unsupported schema or malformed fields. |
153+
| `ExternalPayloadCache` | Bounded replay cache for already verified external payload bytes. | Constructor rejects non-positive entry or byte limits. |
154+
| `store_external_payload()` | Stores encoded bytes through a driver and returns `ExternalPayloadReference`. | Driver errors. |
155+
| `fetch_external_payload()` | Fetches referenced bytes, then verifies size and SHA-256 before decode. | `ExternalPayloadIntegrityError` on size/hash mismatch. |
156+
| `external_storage_envelope()` | Encodes a value inline until the threshold is crossed, then writes bytes through a driver. | `ValueError` when the threshold is invalid or no driver can resolve a reference. |
157+
158+
```python
159+
from durable_workflow import (
160+
ExternalPayloadCache,
161+
LocalFilesystemExternalStorage,
162+
external_storage_envelope,
163+
to_avro_payload_value,
164+
)
165+
from durable_workflow.external_storage import fetch_external_payload, store_external_payload
166+
167+
storage = LocalFilesystemExternalStorage("/var/lib/durable-workflow/payloads")
168+
cache = ExternalPayloadCache(max_entries=256, max_bytes=32 * 1024 * 1024)
169+
170+
payload = to_avro_payload_value({"invoice_pdf": "x" * 1_000_000})
171+
envelope = external_storage_envelope(
172+
payload,
173+
external_storage=storage,
174+
threshold_bytes=64 * 1024,
175+
)
176+
177+
reference = store_external_payload(storage, b'{"archived":true}', codec="json")
178+
payload_bytes = fetch_external_payload(storage, reference, cache=cache)
179+
```
180+
181+
The reference schema is `EXTERNAL_PAYLOAD_REFERENCE_SCHEMA`
182+
(`durable-workflow.v2.external-payload-reference.v1`). Object-store adapters do
183+
not add cloud SDK dependencies to `durable-workflow`; applications pass their
184+
already configured S3, GCS, or Azure clients.
185+
137186
### Cluster and Task Queues
138187

139188
| Method | Returns | Notes |

scripts/reference-docs-contract.json

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,7 @@
224224
"Quickstart",
225225
"Client API Reference",
226226
"Constructor",
227+
"External Payload Storage",
227228
"Cluster and Task Queues",
228229
"Workflow Operations",
229230
"Schedules",
@@ -234,9 +235,20 @@
234235
"Failure surface",
235236
"exception classes",
236237
"thin, async-first client",
237-
"language-neutral payload codec"
238+
"language-neutral payload codec",
239+
"durable-workflow.v2.external-payload-reference.v1"
238240
],
239241
"requiredMethods": [
242+
"ExternalStorageDriver",
243+
"LocalFilesystemExternalStorage",
244+
"S3ExternalStorage",
245+
"GCSExternalStorage",
246+
"AzureBlobExternalStorage",
247+
"ExternalPayloadReference",
248+
"ExternalPayloadCache",
249+
"store_external_payload",
250+
"fetch_external_payload",
251+
"external_storage_envelope",
240252
"client.health",
241253
"client.get_cluster_info",
242254
"client.list_task_queues",

0 commit comments

Comments
 (0)