Amazon S3 storage backend for @backloghq/opslog. Store your event-sourced data in S3 and enable multi-writer concurrency across machines.
npm install @backloghq/opslog @backloghq/opslog-s3import { Store } from "@backloghq/opslog";
import { S3Backend } from "@backloghq/opslog-s3";
const store = new Store<{ title: string; status: string }>();
await store.open("my-store", {
backend: new S3Backend({
bucket: "my-bucket",
prefix: "opslog/my-store",
region: "us-east-1",
}),
agentId: "agent-A",
});
await store.set("task-1", { title: "Build API", status: "active" });
const task = store.get("task-1");
await store.close();Another agent on a different machine can write to the same store:
const store = new Store<{ title: string; status: string }>();
await store.open("my-store", {
backend: new S3Backend({
bucket: "my-bucket",
prefix: "opslog/my-store",
region: "us-east-1",
}),
agentId: "agent-B",
});
// Sees agent-A's writes
store.get("task-1"); // { title: "Build API", status: "active" }
// Writes to its own WAL — no contention with agent-A
await store.set("task-2", { title: "Write tests", status: "active" });
await store.close();new S3Backend({
bucket: "my-bucket", // S3 bucket name (required)
prefix: "path/to/store", // Key prefix (optional, no trailing slash)
region: "us-east-1", // AWS region (required if no client provided)
client: myS3Client, // Pre-configured S3Client (optional)
lockTtlMs: 60000, // Lock TTL in ms (default: 60000)
});Pass your own S3Client if you need custom configuration (credentials, endpoint, middleware):
import { S3Client } from "@aws-sdk/client-s3";
const client = new S3Client({
region: "us-east-1",
endpoint: "http://localhost:9000", // MinIO, LocalStack, etc.
forcePathStyle: true,
});
const backend = new S3Backend({ bucket: "my-bucket", client });s3://bucket/prefix/
manifest.json # Store metadata
snapshots/snap-<ts>.jsonl # Immutable state captures (JSONL)
ops/agent-<id>-<ts>.jsonl # Per-agent WAL files
archive/archive-<period>.json # Archived records
S3 has no native append. Instead of downloading, concatenating, and re-uploading a single growing file, the S3Backend stores each write as a separate S3 object:
ops/agent-A-1744200000/
batch-1744200001-0000.jsonl (ops from first write)
batch-1744200002-0000.jsonl (ops from second write)
batch-1744200003-0000.jsonl (ops from third write)
- Append: single PutObject per write (no download needed)
- Read: ListObjectsV2 + parallel GetObject per batch, with incremental caching (only new batches are downloaded on subsequent reads)
- Truncate: delete last batch object (or modify if it contains multiple ops)
Stores created with v0.1.0 (single .jsonl files) are transparently supported — the backend detects the format automatically.
Uses S3 conditional writes (IfNoneMatch: *) for lock acquisition — fails atomically if the lock already exists. Lock objects carry a TTL; stale locks from crashed agents are automatically recovered.
await backend.writeBlob("data/records.jsonl", buffer);
const full = await backend.readBlob("data/records.jsonl");
const range = await backend.readBlobRange("data/records.jsonl", 1024, 256); // HTTP Range headerreadBlobRange uses S3 GetObject with Range: bytes=N-M header — reads only the requested bytes. Enables O(1) point lookups in JSONL record stores without downloading the full object.
When multiple agents write to the same key, opslog's Lamport clock ordering resolves conflicts via last-writer-wins. See opslog multi-writer docs for details.
The S3Backend needs these S3 permissions on the bucket/prefix:
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/opslog/*"
]
}npm run build # Compile TypeScript
npm run lint # ESLint
npm test # Run tests (uses in-memory mock S3, no AWS needed)
npm run test:coverage # Tests with coverageMIT