Skip to content

Commit 66a8367

Browse files
committed
docs: add filesystem serdes section
Document the FileSystem serdes as a new section in the serialization reference page. The section covers storage prerequisites (Amazon EFS and Amazon S3 Files), the createFileSystemSerdes signature and FileSystemSerdesConfig, storage modes (ALWAYS and OVERFLOW), path encoding (URI and HASH), the buildPreview helper for inline previews and PII masking, and how to set the serdes as the handler default with configureSerdes. The walkthrough leads with the conservative per-operation StepConfig pattern. Setting it as the handler default lives in its own subsection. Coverage is TypeScript only at this stage. Python and Java tabs link to the parity issues aws-durable-execution-sdk-python#479 and aws-durable-execution-sdk-java#463. Verified against packages/aws-durable-execution-sdk-js/src/utils/serdes/ filesystem-serdes.ts and preview.ts at origin/main. Closes #207
1 parent de63adf commit 66a8367

8 files changed

Lines changed: 330 additions & 0 deletions

File tree

docs/sdk-reference/state/serialization.md

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,230 @@ Map and parallel perations support two SerDes fields that apply at different lev
281281
--8<-- "examples/java/sdk-reference/serialization/PassThroughSerdesExample.java"
282282
```
283283

284+
## FileSystem serdes
285+
286+
The FileSystem serdes stores data on the filesystem and keeps a file pointer in
287+
the checkpoint. This can be useful when you do not want to store your data in the
288+
checkpoint itself, for example when your payload exceeds the durable execution
289+
checkpoint size limit. The persisted value lives on the filesystem and the SDK
290+
reads it on replay.
291+
292+
The FileSystem serdes requires a filesystem that persists across invocations and
293+
that every Lambda execution environment can read. In AWS Lambda, this means:
294+
295+
- [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/lambda/latest/dg/configuration-filesystem-efs.html).
296+
Serverless file system that scales automatically with your workloads.
297+
- [Amazon S3 Files](https://docs.aws.amazon.com/lambda/latest/dg/configuration-filesystem-s3files.html).
298+
Serverless file system for mounting an Amazon S3 bucket. Amazon S3 Files
299+
provides access to Amazon S3 objects as files using standard file system
300+
operations such as read and write on the local mount path.
301+
302+
!!! warning "Do not use Lambda's `/tmp` directory"
303+
304+
Lambda's `/tmp` filesystem is local to a single execution environment. A
305+
different execution environment may run the next invocation, so the SDK cannot
306+
find files written to `/tmp` on replay and deserialization fails. Always mount
307+
a durable, shared filesystem.
308+
309+
=== "TypeScript"
310+
311+
Pass the FileSystem serdes to a single operation through `StepConfig`,
312+
`CallbackConfig`, `MapConfig`, or `ParallelConfig`. Other operations in the
313+
same handler continue to use the default serdes.
314+
315+
```typescript
316+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-walkthrough.ts"
317+
```
318+
319+
=== "Python"
320+
321+
Coming soon. See
322+
[aws-durable-execution-sdk-python#479](https://github.com/aws/aws-durable-execution-sdk-python/issues/479).
323+
324+
=== "Java"
325+
326+
Coming soon. See
327+
[aws-durable-execution-sdk-java#463](https://github.com/aws/aws-durable-execution-sdk-java/issues/463).
328+
329+
### createFileSystemSerdes signature
330+
331+
=== "TypeScript"
332+
333+
```typescript
334+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-signature.ts"
335+
```
336+
337+
**Parameters:**
338+
339+
- `basePath` Directory path where data files are written. Set this to the
340+
mount point of your S3 Files or EFS volume.
341+
- `config` (optional) `FileSystemSerdesConfig` controlling storage mode, path
342+
encoding, and preview generation.
343+
344+
**Returns:** A `Serdes` that reads and writes JSON files under `basePath`.
345+
346+
=== "Python"
347+
348+
Coming soon.
349+
350+
=== "Java"
351+
352+
Coming soon.
353+
354+
### FileSystemSerdesConfig
355+
356+
=== "TypeScript"
357+
358+
```typescript
359+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-config.ts"
360+
```
361+
362+
**Fields:**
363+
364+
- `storageMode` (optional) A `FileSystemSerdesMode` value. Default:
365+
`FileSystemSerdesMode.ALWAYS`.
366+
- `pathEncoding` (optional) A `FileSystemPathEncoding` value. Default:
367+
`FileSystemPathEncoding.URI`.
368+
- `generatePreview` (optional) Function that returns a preview object. The SDK
369+
stores the preview inline in the checkpoint envelope alongside the file
370+
pointer. Use the [`buildPreview`](#preview-and-pii-masking) helper or
371+
write your own.
372+
373+
=== "Python"
374+
375+
Coming soon.
376+
377+
=== "Java"
378+
379+
Coming soon.
380+
381+
### Storage modes
382+
383+
The `storageMode` enumerated field controls when the SDK writes to the filesystem.
384+
385+
`FileSystemSerdesMode.ALWAYS` writes every value to a file. The checkpoint stores
386+
only the file pointer.
387+
388+
`FileSystemSerdesMode.OVERFLOW` uses the standard durable execution checkpoint
389+
store and only writes to the filesystem when the value would exceed the durable
390+
execution checkpoint size limit.
391+
392+
=== "TypeScript"
393+
394+
```typescript
395+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-overflow.ts"
396+
```
397+
398+
=== "Python"
399+
400+
Coming soon.
401+
402+
=== "Java"
403+
404+
Coming soon.
405+
406+
### Path encoding
407+
408+
The `pathEncoding` field controls how the durable execution ARN and the entity ID
409+
become the on-disk directory and file names.
410+
411+
`FileSystemPathEncoding.URI` builds a per-execution directory from the function
412+
name, execution name, and invocation ID parsed from the ARN, and encodes the entity
413+
ID with `encodeURIComponent` for the file name. Names stay readable when you read
414+
files directly from the mount. A very long entity ID may exceed the filesystem's
415+
per-name length limit, commonly 255 bytes.
416+
417+
`FileSystemPathEncoding.HASH` replaces the ARN and the entity ID with their SHA-256
418+
hex digests. Names are a fixed 64 characters and are always filesystem-safe
419+
regardless of the input characters or length. Choose `HASH` when entity IDs may
420+
contain characters that are unsafe in a file name, such as `/`, or may be long
421+
enough to exceed the name-length limit.
422+
423+
=== "TypeScript"
424+
425+
```typescript
426+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-path-encoding.ts"
427+
```
428+
429+
=== "Python"
430+
431+
Coming soon.
432+
433+
=== "Java"
434+
435+
Coming soon.
436+
437+
### Preview and PII masking
438+
439+
When the FileSystem serdes writes to a file, the checkpoint envelope only contains
440+
the file pointer, so the GetDurableExecution API and the AWS Console cannot show
441+
the actual data for that operation. Configure `generatePreview` to embed a small
442+
object inline so you can see the data without reading the file.
443+
444+
Use `buildPreview` to compute the preview from a `PreviewConfig`. The config
445+
selects which fields to include, exclude, or mask. Masking replaces a field's value
446+
with `***` or a custom string. The preview object is capped at 4096 bytes by
447+
default.
448+
449+
=== "TypeScript"
450+
451+
```typescript
452+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-preview.ts"
453+
```
454+
455+
**`PreviewConfig` fields:**
456+
457+
- `mode` Either `PreviewMode.INCLUDE_ALL` or `PreviewMode.EXCLUDE_ALL`. Sets
458+
the starting point before applying `include`, `exclude`, and `mask`.
459+
- `include` (optional) Fields to add when starting from `EXCLUDE_ALL`, or to
460+
force-include when starting from `INCLUDE_ALL`.
461+
- `exclude` (optional) Fields to remove. Always wins over `mask`.
462+
- `mask` (optional) Fields whose values become `maskString`. A masked field is
463+
visible in the preview unless it is also excluded.
464+
- `maskString` (optional) Replacement value for masked fields. Default: `***`.
465+
- `maxPreviewBytes` (optional) Maximum size in bytes for the preview object
466+
when JSON-serialized. Default: `4096`.
467+
468+
Each `PreviewField` selector has a `name` and an optional `match`. Use
469+
`FieldMatchMode.ANYWHERE` (default) to match the field name at any depth in
470+
the object tree, or `FieldMatchMode.PATH` to match an exact dot-notation path
471+
from the root.
472+
473+
Field names that contain a dot are not supported in selectors because a dot is
474+
indistinguishable from a path separator. Array structure is not preserved in
475+
the output: fields from array elements merge into a plain object at the
476+
array's path.
477+
478+
=== "Python"
479+
480+
Coming soon.
481+
482+
=== "Java"
483+
484+
Coming soon.
485+
486+
### Set as the default for the handler
487+
488+
When you want every step result, child-context result, invoke result, and
489+
waitForCondition result in the handler to use the FileSystem serdes, configure it
490+
once with `configureSerdes`.
491+
492+
=== "TypeScript"
493+
494+
```typescript
495+
--8<-- "examples/typescript/sdk-reference/serialization/filesystem-serdes-default.ts"
496+
```
497+
498+
=== "Python"
499+
500+
Coming soon. See
501+
[aws-durable-execution-sdk-python#479](https://github.com/aws/aws-durable-execution-sdk-python/issues/479).
502+
503+
=== "Java"
504+
505+
Coming soon. See
506+
[aws-durable-execution-sdk-java#463](https://github.com/aws/aws-durable-execution-sdk-java/issues/463).
507+
284508
## Serialization errors
285509

286510
When serialization or deserialization fails, each SDK raises or throws a specific
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import {
2+
FileSystemSerdesMode,
3+
FileSystemPathEncoding,
4+
} from "@aws/durable-execution-sdk-js";
5+
6+
export interface FileSystemSerdesConfig {
7+
storageMode?: FileSystemSerdesMode;
8+
pathEncoding?: FileSystemPathEncoding;
9+
generatePreview?: (value: unknown) => Record<string, unknown> | undefined;
10+
}
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import {
2+
DurableContext,
3+
withDurableExecution,
4+
createFileSystemSerdes,
5+
} from "@aws/durable-execution-sdk-js";
6+
7+
export const handler = withDurableExecution(
8+
async (event: { customerId: string }, context: DurableContext) => {
9+
// Configure once. Every step result, child-context result, invoke result,
10+
// and waitForCondition result in this handler is now stored on /mnt/s3.
11+
context.configureSerdes({
12+
defaultSerdes: createFileSystemSerdes("/mnt/s3"),
13+
});
14+
15+
const profile = await context.step("load-profile", async () => {
16+
return loadLargeProfile(event.customerId);
17+
});
18+
19+
return { name: profile.name };
20+
},
21+
);
22+
23+
declare function loadLargeProfile(
24+
customerId: string,
25+
): Promise<{ name: string }>;
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import {
2+
createFileSystemSerdes,
3+
FileSystemSerdesMode,
4+
} from "@aws/durable-execution-sdk-js";
5+
6+
// Small payloads stay inline in the checkpoint. The SDK only writes to the
7+
// filesystem when the checkpoint envelope would exceed the size limit.
8+
export const overflowFileSystemSerdes = createFileSystemSerdes("/mnt/s3", {
9+
storageMode: FileSystemSerdesMode.OVERFLOW,
10+
});
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import {
2+
createFileSystemSerdes,
3+
FileSystemPathEncoding,
4+
} from "@aws/durable-execution-sdk-js";
5+
6+
// Hash the ARN and the entity ID into fixed-length, filesystem-safe segments.
7+
// Use HASH when entity IDs may contain unsafe characters or be very long.
8+
export const hashedFileSystemSerdes = createFileSystemSerdes("/mnt/s3", {
9+
pathEncoding: FileSystemPathEncoding.HASH,
10+
});
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import {
2+
createFileSystemSerdes,
3+
buildPreview,
4+
PreviewMode,
5+
} from "@aws/durable-execution-sdk-js";
6+
7+
// The checkpoint envelope stores the file pointer plus a small preview object,
8+
// so the console can show id and status without reading the full file. The
9+
// email field is visible but masked, so PII does not land in the checkpoint.
10+
export const previewFileSystemSerdes = createFileSystemSerdes("/mnt/s3", {
11+
generatePreview: (value) =>
12+
buildPreview(value, {
13+
mode: PreviewMode.EXCLUDE_ALL,
14+
include: [{ name: "id" }, { name: "status" }],
15+
mask: [{ name: "email" }],
16+
}),
17+
});
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
import {
2+
Serdes,
3+
FileSystemSerdesConfig,
4+
} from "@aws/durable-execution-sdk-js";
5+
6+
export type CreateFileSystemSerdes = (
7+
basePath: string,
8+
config?: FileSystemSerdesConfig,
9+
) => Serdes<unknown>;
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import {
2+
DurableContext,
3+
withDurableExecution,
4+
createFileSystemSerdes,
5+
} from "@aws/durable-execution-sdk-js";
6+
7+
export const handler = withDurableExecution(
8+
async (event: { customerId: string }, context: DurableContext) => {
9+
const fsSerdes = createFileSystemSerdes("/mnt/s3");
10+
11+
// Pass the FileSystem serdes to one step. Other operations in this handler
12+
// keep using the default JSON serdes.
13+
const profile = await context.step(
14+
"load-profile",
15+
async () => loadLargeProfile(event.customerId),
16+
{ serdes: fsSerdes },
17+
);
18+
19+
return { name: profile.name };
20+
},
21+
);
22+
23+
declare function loadLargeProfile(
24+
customerId: string,
25+
): Promise<{ name: string }>;

0 commit comments

Comments
 (0)