Skip to content

Concepts

Jump to bottom
Eugene Lazutkin edited this page Apr 23, 2026 · 3 revisions

Concepts

A one-page reference for the vocabulary the rest of the wiki uses. Two buckets:

  1. Toolkit concepts — things this library introduces.
  2. DynamoDB concepts — AWS terms the toolkit wraps but does not redefine.

Every other page links in here on first use rather than redefining terms locally.

How to read the wiki

The toolkit is a thin helper on top of the AWS JS SDK v3. Most pages open with a See also (AWS JS SDK v3) header pointing at the relevant SDK / DynamoDB docs. Rules of thumb:

  • If the toolkit wraps an SDK concept, read the SDK page first, then read the wiki page for what the wrapper adds.
  • If the toolkit does not wrap something, the wiki tells you to use the SDK directly and links the reference.
  • All code examples are extracted from the project's test suite — they run against DynamoDB Local. They are not runnable standalone from the wiki; they assume the fixtures in the test harness.

Toolkit concepts

Adapter as the composition root

Adapter is the entry point. One Adapter instance owns a client, a table, a keyFields list, an optional declarative schema (structuralKey, indices, typeLabels, typeDiscriminator, typeField, technicalPrefix, searchable, filterable, versionField, createdAtField, relationships, descriptorKey), and a hooks bag. It delegates real work to orthogonal sub-exports:

  • dynamodb-toolkit/expressions — build UpdateExpression, ConditionExpression, FilterExpression, ProjectionExpression, KeyConditionExpression.
  • dynamodb-toolkit/batch — applyBatch, applyTransaction, explainTransactionCancellation, getBatch, getTransaction.
  • dynamodb-toolkit/mass — paginateList, iterateItems, readByKeys, writeItems, deleteByKeys, deleteList, copyList, moveList, plus the cursor helpers.
  • dynamodb-toolkit/paths — dotted-path get / set / patch utilities.
  • dynamodb-toolkit/marshalling — Marshaller<TRuntime, TStored> pairs for Date / URL / Map.
  • dynamodb-toolkit/hooks — canned prepare-hook builders (stampCreatedAtISO, stampCreatedAtEpoch).
  • dynamodb-toolkit/rest-core — framework-agnostic parsers + builders for the REST layer.
  • dynamodb-toolkit/handler — node:http handler on top of rest-core.
  • dynamodb-toolkit/provisioning — planTable, ensureTable, verifyTable, plus the descriptor record.

Use the Adapter for "one entity, one table" flows. Drop to a sub-export when you need a slice of the toolkit without the full composition root.

Additive params (the builder contract)

Every expression builder in this toolkit mutates a params object in place and returns the same object. The design is deliberately additive: a caller accumulates a full DynamoDB Command input by calling one builder after another against the same params.

import {buildUpdate, buildCondition, cleanParams} from 'dynamodb-toolkit/expressions';

let params = {TableName: 'planets', Key: {name: 'Hoth'}};  // mandatory SDK fields
params = buildUpdate({diameter: 13000}, {}, params);        // adds UpdateExpression + names/values
params = buildCondition([{path: 'climate', op: '=', value: 'frozen'}], params);  // adds ConditionExpression
params = cleanParams(params);                               // drops unused names/values

await docClient.send(new UpdateCommand(params));

What each builder adds:

Builder Populates Works with
buildUpdate UpdateExpression, ExpressionAttributeNames, ExpressionAttributeValues UpdateCommand, TransactWriteItems.Update
buildCondition ConditionExpression (AND-merges with existing), names / values any Command that accepts ConditionExpression
buildSearch / buildFilterByExample FilterExpression, names / values QueryCommand, ScanCommand
buildKeyCondition KeyConditionExpression (AND-merges), names / values QueryCommand
addProjection ProjectionExpression, names any Command that accepts ProjectionExpression
adapter.applyFilter KeyConditionExpression (auto-promoted clauses) + FilterExpression (rest), names / values QueryCommand, ScanCommand
adapter.buildKey KeyConditionExpression, names / values QueryCommand — ergonomic wrapper around buildKeyCondition that uses the Adapter's declared keyFields / structuralKey

Simple case: pass {} as the starting params, the builder fills in what it needs, use the returned object directly.

Typical case: start with the mandatory SDK fields (TableName, Key, Item, IndexName, ConsistentRead…), pass through every builder you need, call cleanParams at the end, hand the result to the SDK.

params is not a fluent chain (.a().b()). It is a passed-through mutable bag: each builder reads the running name / value counters, appends its own entries with a per-builder prefix (#upk/:upv for update, #kc/:kcv for key-condition, #cd/:cdv for condition, #sr/:flt for search, #pj for projection, #ff/:ffv for the Adapter's applyFilter), writes the expression field, and returns. Builders can be run in any order; merging is safe because every builder allocates fresh aliases from the existing counters.

cleanParams

The builders allocate every name / value placeholder they might need, but conditional paths (e.g. buildUpdate with an empty patch, or buildCondition with no clauses) can leave entries in ExpressionAttributeNames / ExpressionAttributeValues that don't appear in any expression field. DynamoDB rejects unused entries at request time.

cleanParams(params) removes every unused name / value, leaving only the entries referenced by one of KeyConditionExpression, ConditionExpression, UpdateExpression, ProjectionExpression, FilterExpression. Called automatically by every Adapter method; call it yourself at the end of any builder pipeline you construct by hand.

See src/expressions/clean-params.js for the implementation.

Declarative schema

Pre-3.2 Adapters were minimal: {client, table, keyFields} plus any hooks you wired by hand. Everything beyond single-field partition keys (composite keys, searchable mirrors, type dispatch, OC, filter allowlisting, cascade, scope-freeze) was your prepare / revive hooks' job.

3.2.0–3.7.0 folded those patterns into declarative constructor options. The Adapter now owns the mechanical work via built-in steps that run before your hooks. Your prepare is reserved for domain transforms the declaration can't express.

The rest of this section defines each declarative concept. Pull the corresponding entry on Adapter: Constructor options for the wire shape.

technicalPrefix and managed fields

When the Adapter is constructed with technicalPrefix: '-', the toolkit treats every field whose name starts with that prefix as an adapter-managed namespace:

  • Write-side validation. The built-in prepare step rejects incoming user items that carry a prefixed field (unless the field is one of versionField / createdAtField, which round-trip through reads). Throws Error at the hook boundary; the item never hits DynamoDB.
  • Read-side stripping. The built-in revive step drops every prefixed field from the returned item (again excepting versionField / createdAtField).
  • Mechanical write fields. Every adapter-managed field the toolkit itself adds (structuralKey.name, searchablePrefix + field mirrors, typeField, versionField, createdAtField, descriptorKey) must start with technicalPrefix. The constructor validates this at build time — a misconfigured Adapter throws.

Without technicalPrefix: the toolkit does not reserve any field-name convention, and the pre-3.2 "author-convention" pattern (hooks add -search-… mirrors, revive strips --prefixed keys in user code) still works exactly the same. The v3 examples on older pages that show '-t': 1 as a "v3 row marker" and revive(...) { if (!k.startsWith('-')) ...} are the pre-technicalPrefix pattern. Declaring technicalPrefix moves the validation + stripping from your hooks into the toolkit.

Pick a prefix that doesn't clash with DynamoDB reserved words or legitimate domain fields. - is the wiki convention; _, $, and __ are common alternatives.

Structural key

DynamoDB primary keys are at most two scalars: one partition key and one optional sort key. Real domains are often deeper — a rental-agency row is scoped by (state, rentalId, carVin). The structural key is how the toolkit maps an N-dimensional keyFields tuple onto DynamoDB's 2-scalar limit:

new Adapter({
  client, table: 'rentals',
  keyFields: [
    'state',
    {name: 'rentalId', type: 'number', width: 5},
    'carVin'
  ],
  structuralKey: {name: '-sk', separator: '|'},
  technicalPrefix: '-'
});
  • keyFields[0].name (state) becomes the DynamoDB partition key.
  • The structuralKey.name attribute (-sk) becomes the DynamoDB sort key, carrying the joined value of keyFields[1..] (and optionally keyFields[0]).
  • The built-in prepare step writes -sk for you on every full write: it walks keyFields contiguously-from-start, stops at the first missing value, zero-pads {type: 'number'} components per their width, joins with separator. The value for a row with {state: 'TX', rentalId: 42, carVin: '1HG…'} becomes -sk: 'TX|00042|1HG…'.
  • The built-in prepareKey does the same composition for reads, so caller-supplied {state, rentalId, carVin} reaches DynamoDB as {state, '-sk': 'TX|00042|1HG…'}.
  • width on {type: 'number'} components is required in a composite — without zero-padding, string-sorted comparisons between '5|…' and '42|…' break. The constructor validates.

Why not multi-column primary keys? DynamoDB's data model has them baked in at two scalars. The structural-key pattern is the standard single-table-design workaround; the toolkit formalises it as declaration so your buildKey / getListUnder / cascade code can compose subtree queries naturally. See adapter.buildKey and the List-records-of-a-tier recipe.

Type tags: typeLabels, typeDiscriminator, typeField, adapter.typeOf

When one table holds multiple types of record (e.g., parent state rows alongside child rental rows and grandchild car rows under the same composite keyFields), you need a way to tell them apart. The declarative schema gives four coordinated pieces:

  • typeLabels: ['state', 'rental', 'car'] — names paired 1:1 with keyFields. A record with {state: 'TX'} and no further keyFields is depth-1 = 'state'; {state, rentalId} is depth-2 = 'rental'; {state, rentalId, carVin} is depth-3 = 'car'. Labels must be pairwise non-prefix-colliding (a 'car' / 'card' split would break adapter.buildKey({self: true}) queries).
  • typeDiscriminator: 'kind' — explicit type field. When an item carries this field, its value wins over depth-based detection. Use when depth alone doesn't disambiguate (cars-vs-boats at the same leaf tier: both are depth-3 but carry different kind).
  • typeField: 'kind' — auto-populate. When set, the built-in prepare step stamps adapter.typeOf(item) at this field on every full write, unless the user already set it. Enables the Pattern-1 sparse-GSI recipe (pk = typeField) without any hook code. Typically same name as typeDiscriminator so the value round-trips.
  • adapter.typeOf(item) — the resolver. Priority: typeDiscriminator value on the item → typeLabels[depth-1] → raw depth number → undefined.

Minimum declaration for depth-based dispatch: typeLabels. Add typeDiscriminator when you need an explicit override. Add typeField when you want the toolkit to stamp it automatically. All three are additive and opt-in.

See Recipe: List records of a tier for the end-to-end query pattern.

searchablePrefix and searchable mirror columns

The Adapter maintains lowercase "mirror" columns for substring filtering. Declare searchable: {name: 1, climate: 1} on the Adapter; the built-in prepare step writes -search-name / -search-climate (lowercase) on every write whose body carries the source field — full writes and patches alike. The list-read ?search= option and the buildSearch helper build a contains(...) OR contains(...) FilterExpression over the mirrors.

  • Prefix: default '-search-'. Override via searchablePrefix on the Adapter constructor, or on a per-call basis via ListOptions.prefix and the buildSearch options bag.
  • When technicalPrefix is declared, the searchablePrefix must start with it (constructor validates).
  • Only this one prefix is defaulted + auto-written by the toolkit. Everything else field-prefix-related is the technicalPrefix umbrella.
  • Pre-3.2 adapters wired this pattern by hand in prepare; the declaration replaces the boilerplate.

See Expressions: Filter builder for the read path.

Filter URL grammar — <op>-<field>=<value>

The REST handler parses every query parameter shaped as <op>-<field>=<value> into a structured filter clause. The Adapter compiles the clauses against its filterable allowlist:

new Adapter({
  client, table, keyFields: ['name'],
  filterable: {
    status: ['eq', 'in'],
    year: {ops: ['eq', 'ge', 'le', 'btw'], type: 'number'}
  }
});

Ops: eq ne lt le gt ge in btw beg ct ex nx (the last two take no value). Multi-value ops (in, btw) use first-character delimiter on the value: ?in-status=,active,pending splits on the leading ,; ?btw-year=-2020-2024 splits on -. Clauses that match a declared pk / sk auto-promote to KeyConditionExpression; the rest land in FilterExpression. Unlisted fields throw BadFilterField; unallowed ops throw BadFilterOp.

Types: inferred from keyFields / indices when the field is a key, overridden by the {ops, type} shape for plain data fields. Value coercion happens before the expression is built.

See Expressions: Filter builder and adapter.applyFilter in Adapter: Constructor options.

Raw<T> — the bypass marker

raw(item) wraps a value in a Raw<T> sentinel that tells the Adapter to skip hooks on that one call:

import {raw} from 'dynamodb-toolkit';

// Write exactly this, no hooks applied:
await adapter.put(raw({name: 'X', '-internal': 'kept'}), {force: true});

// Read as-is, skip revive:
const item = await adapter.getByKey(key, undefined, {reviveItems: false});

Raw bypasses both the built-in prepare / revive and your user-supplied hooks. The caller is responsible for the stored shape — including structuralKey.name, searchablePrefix + field mirrors, typeField, and the technicalPrefix namespace. Use when you already have the canonical DB shape (a replication sink, a re-import, a low-level fixup) and don't want hooks to transform it.

See Adapter: Raw marker.

Projection and patches — performance levers for read and write

DynamoDB items are often wide: dozens or hundreds of attributes, some of them large (embedded blobs, serialized graphs, long text). Reading and writing the full item every time is wasteful both on the DynamoDB side (read / write capacity, scanned bytes) and on the network (egress bytes back to your client). The toolkit ships two symmetric tools to trim that cost:

  • Projection (read side) — request only the attributes your caller actually needs. DynamoDB reads and returns just the projected slice, not the full item. The ProjectionExpression names exactly which fields come back; everything else stays on the server. Saves read capacity, scan bytes, and network egress. The toolkit's addProjection builds the expression; the Adapter's getByKey / getByKeys / getList / getListByParams take a fields argument that feeds it straight in. Projection is the primary read-performance lever — reach for it any time the caller doesn't need every attribute.
  • Patch (write side) — modify only the fields that changed. The UpdateExpression contains exactly the attributes being set / removed / incremented; DynamoDB leaves everything else alone. Saves write traffic (you ship only the delta, not the whole item) and avoids the read-modify-write round-trip you would need with PutItem. The toolkit's buildUpdate builds the expression; the Adapter's patch(key, patch, options) takes a partial object and wires it through. Patches are the write-side counterpart of projection — same philosophy, opposite direction.

These two tools pair naturally: read a projection of an item, mutate the fields you care about on the client, patch them back. The other fields never leave the server.

Optimistic concurrency — versionField

Declaring versionField: '-v' turns on opt-in optimistic concurrency: the toolkit maintains a numeric counter on every item and conditions every write on the counter's observed value. No lost updates, no read-modify-write-race, no explicit locking.

  • post initialises the counter to 1 with attribute_not_exists(<pk>).
  • put(item) reads the counter from the caller's item, conditions on attribute_not_exists(<pk>) OR <versionField> = :observed, writes observed + 1. {force: true} bypasses the condition but still bumps the version.
  • patch(key, patch, {expectedVersion?}) conditions on <versionField> = :expectedVersion when supplied; always emits ADD <versionField> :1. Leaving expectedVersion off means "any version, just increment" — a last-writer-wins flavour that still maintains the counter for readers that care.
  • delete(key, {expectedVersion?}) conditions when supplied; no increment (the item is gone).
  • edit(key, mapFn) and editListByParams(params, mapFn) handle OC automatically via read-then-write. Version conflicts in mass ops bucket into MassOpResult.conflicts: [{key, reason: 'VersionConflict'}].
  • revive preserves versionField (unlike other technicalPrefix fields) so callers round-trip version counters through read-modify-write cycles without explicit handling.

versionField must start with technicalPrefix when declared. See Adapter: Constructor options — Concurrency and scope-freeze.

Scope-freeze — createdAtField + asOf

Declaring createdAtField: '-createdAt' enables the scope-freeze pattern on mass ops: pass options.asOf to AND-merge <createdAtField> <= :asOf into the Query / Scan FilterExpression. Restricts the op to items that existed at or before a point in time.

// Replay every item created up to the cutoff; skip anything newer.
await adapter.editListByParams(params, applyFn, {asOf: '2026-04-01T00:00:00Z'});

Typical uses: historical replays, audit exports, snapshot backfills, "rerun the migration against the data that existed when it started."

The toolkit does not auto-write createdAtField. Wire one of the canned stamp builders in prepare (stampCreatedAtISO / stampCreatedAtEpoch) or write your own. The stored format dictates what asOf values to pass; Date is auto-ISO-converted as a convenience, other types pass through. asOf without createdAtField throws CreatedAtFieldNotDeclared.

createdAtField must start with technicalPrefix when declared. revive preserves the field (like versionField) so callers can compare timestamps or round-trip the value through mutations. See the stamp builders on Adapter: Hooks.

Cascade — relationships

Declaring relationships: {structural: true} treats the composite structuralKey as a parent-child hierarchy and unlocks cascade primitives:

  • adapter.deleteAllUnder(srcKey, options) — leaf-first: descendants first, then the self node.
  • adapter.cloneAllUnder(srcKey, dstKey, options) / cloneAllUnderBy(srcKey, mapFn, options) — root-first prefix-swap or mapFn-driven fan-out. Source stays intact.
  • adapter.moveAllUnder(srcKey, dstKey, options) / moveAllUnderBy(srcKey, mapFn, options) — leaf-first, two-phase idempotent.

Each primitive honours options.maxItems + options.resumeToken on the descendants pass; the self-node phase runs exactly once per full sequence (root-first for clone, last for delete / move). Undeclared adapters throw CascadeNotDeclared on call.

Why an explicit declaration. Composite keyFields alone don't imply a hierarchy — many schemas stack keyFields for access-pattern reasons that don't mean "the shorter key is a parent." The declaration is the toolkit's signal that the hierarchy is intentional. See Adapter: Constructor options — Cascade gate.

Mass-op envelope — resumable cursors, partial failure, version conflict

Every list-op variant (deleteListByParams, cloneListByParams, moveListByParams, editListByParams, the cascade primitives, the rename / cloneWithOverwrite macros) returns a MassOpResult with uniform shape:

interface MassOpResult {
  processed: number;
  skipped: number;
  failed: {key, reason, details?, sdkError?}[];
  conflicts: {key, reason: 'VersionConflict', sdkError?}[];   // populated when versionField declared
  cursor?: string;                                             // present iff stopped early
}
  • options.maxItems — soft cap, page-boundary enforced. When reached, cursor is populated.
  • options.resumeToken — accept a prior cursor to continue from the page boundary.
  • cursor is opaque base64-url JSON. decodeCursor(cursor) is a debug / test helper; callers round-trip it unopened.
  • failed[] carries a closed reason enum ('ConditionalCheckFailed', 'ValidationException', 'ProvisionedThroughputExceeded', 'Unknown'); sdkError preserves the underlying AWS error for inspection.
  • conflicts[] is separated from the general failed bucket specifically so optimistic-concurrency retries are distinguishable.

Re-running a mass op with the same params + resumeToken is safe: deletes are idempotent (DynamoDB Delete succeeds on missing items); clones / moves with {ifNotExists} bucket put-collisions into skipped; vanished items during edit bucket into skipped via the CCF on the existence guard.

See Mass operations for the full primitive catalogue.

Descriptor record — descriptorKey

Opt-in reserved-row key (e.g., descriptorKey: '__adapter__') that the provisioning helpers use to store a JSON snapshot of the Adapter's declaration. The snapshot captures what DescribeTable can't see — searchable, searchablePrefix, filterable, versionField, createdAtField, typeField, typeDiscriminator, typeLabels. verifyTable diffs the snapshot against the current declaration to detect drift beyond the raw table schema.

When declared, the Adapter also auto-filters this row out of every getList / getListByParams result (injects <pk> <> :descriptorKey into the FilterExpression). Pass {includeDescriptor: true} on an individual call to include it — typically only useful for tooling that inspects the descriptor itself.

Absent by default — IaC-managed tables (Terraform / CDK / CloudFormation) that own their schema ignore the descriptor entirely. See Adapter: Provisioning.

Marshalling pairs — Marshaller<TRuntime, TStored>

The toolkit ships symmetric marshaller pairs in dynamodb-toolkit/marshalling. Each pair packages write + read directions in a single object:

interface Marshaller<TRuntime, TStored> {
  marshall: (value: TRuntime) => TStored;
  unmarshall: (stored: TStored) => TRuntime;
}

Predefined pairs: dateISO (Date ⇔ ISO string), dateEpoch (Date ⇔ epoch ms), url (URL ⇔ string). Loose helpers for Map ⇔ plain object via marshallMap / unmarshallMap.

Typical use: call the marshall direction in prepare, unmarshall in revive. Packaging both directions in one object nudges callers to keep them in sync — changing the wire format is one object literal, not two scattered call sites. Every pair passes undefined / null through untouched. See Adapter: Marshalling.

checkConsistency — invariants enforced at the database, no round-trip

The toolkit ships a checkConsistency hook plus a transaction auto-upgrade path (Adapter: Transaction auto-upgrade) that together let you enforce invariants on every write without reading the current item back to the client first.

When a partial update touches a subset of fields, the natural worry is "did I just break an invariant this item was supposed to hold?" The naive approach is: read the full item, combine it with the patch in memory, check the invariant in JavaScript, write back. That's two round-trips and a race window. The toolkit approach is: the checkConsistency(batch) hook inspects the outgoing write descriptor and returns extra makeCheck descriptors — each a ConditionExpression that DynamoDB evaluates server-side. If all checks pass, the write commits atomically as part of the same TransactWriteItems; if any fail, the whole transaction is rejected with TransactionCanceledException and the database is unchanged.

Net effect: the invariant check happens at the database level, without transferring anything to the client and back. The caller sees either the write commit cleanly or a single rejection — never a torn update or a stale snapshot. Use it for referential integrity ("parent must exist"), optimistic locking (revision / version checks), quota enforcement, tenant scoping, or any rule where the answer is "it depends on other rows".

Hooks

Per-Adapter customization lives in the hooks bag. The toolkit composes built-in steps around your hooks — see Adapter: Hooks — Built-in steps for the composition rules. At a glance:

Hook When What your override adds
prepare(item, isPatch) Every non-Raw write Domain-level derived fields, cross-field coercions, transient-field stripping. (Built-in covers searchable mirrors, structuralKey composition, typeField stamping, technicalPrefix validation, versionField init.)
prepareKey(key, index) Every keyed op Index-specific key reshaping, external-ID translation. (Built-in covers structuralKey composition.)
prepareListInput(example, index) getList Build the Query / Scan input from an example + index name.
updateInput(input, op) Every single write Last-chance mutation of the built params before SDK dispatch.
revive(item, fields) Every read Domain-level decoding (marshaller unwraps, field-shape fixes). (Built-in covers technicalPrefix stripping.)
validateItem(item, isPatch) Every validated write Throw on invalid items before hitting DynamoDB.
checkConsistency(batch) Every single-op write Return extra makeCheck descriptors DynamoDB evaluates atomically with the write — invariant enforcement at the DB level, no client round-trip. See Adapter: Transaction auto-upgrade.

Indirect indices

A sparse GSI (one that projects KEYS_ONLY or a small subset) plus an indirect: true entry in indices (or the legacy indirectIndices: {name: 1} map) tells the Adapter to do a second-hop read. The Adapter queries the GSI to get the primary keys, then does a BatchGetItem against the base table to hydrate full items. Callers get a single-shot list read with the GSI's ordering and the base table's full projection.

See Adapter: Indirect indices for the full pattern, including ignoreIndirection to opt out on a per-call basis.

DynamoDB concepts

Terms that belong to DynamoDB itself. These short definitions are here so every wiki page can link in rather than repeat them.

Primary index — the base table itself

The table's primary key (partition key + optional sort key) is the primary index. Every DynamoDB table is a key-value store sorted by the primary key; there is no separate "primary index" object to create. GetItem / Query / Scan on the table name target this index.

Everything in the rest of this section — GSI, LSI, projection, KeyConditionExpression — is about adding secondary access paths on top of the primary key. If your read pattern fits the primary key directly (e.g. "get item by id"), you don't need any of it.

Why reach for a secondary index

The primary key gives you exactly one way to look items up — the partition key plus (optionally) the sort key. Real apps need more:

  • A different ordering. Primary key orders by partition then sort. Want chronological? Need a GSI with created_at as sort key. Want alphabetical by name? Need a GSI with the name as sort key. The base table can have only one ordering; each GSI / LSI adds another.
  • A different access predicate. Look items up by email, by tenant, by status, by tag — anything that isn't the partition key. Each of these is a GSI whose partition key is the field you want to query by.
  • Compound / structured sort keys that support BEGINS_WITH(...) and range queries. Concatenate multiple attribute values (often with a separator like # or |) into a single sort-key string; query for begins_with(sortKey, 'active#2026-04-') to get "all active items from April 2026". This is the workhorse single-table-design pattern — one GSI can serve many access patterns if the sort key is designed around the prefixes you want to query. See Structural key for the toolkit's declarative take.
  • Sparse indexes — include / exclude items by criteria. A secondary index only contains items that have a value for the index's key fields. Items where those fields are missing (undefined / not-set) are silently omitted from the index. This gives you "only items in state X" as a cheap read: populate the index key field conditionally in your prepare hook (or declaratively via indices[*].sparse: {onlyWhen}); items that don't qualify never enter the index, so a full scan of the GSI returns exactly the qualifying subset. Pairs well with the toolkit's indirect-indices pattern when the index projection is KEYS_ONLY.

Worked examples of each pattern:

  • Chronological GSI — created_at as GSI partition key means you can Query the whole table by time buckets (usually with a coarse partition like date-or-month) and get sorted results. Without it, listing "latest N items" requires a Scan with in-memory sort.
  • Email lookup GSI — base table keyed by user_id; GSI with email as partition key. Login flow: getByKey({email: 'x@y'}, …, {params: {IndexName: 'email-index'}}) → one point read on the GSI returns the matching user.
  • Compound sort key GSI — store 'tenant#status#name' as the GSI sort key (populated by prepare or structuralKey). One GSI serves: "all items for tenant T" (begins_with('T#')), "active items for tenant T" (begins_with('T#active#')), "active items alphabetically" (range query on the active# prefix).
  • Sparse tombstone GSI — indices: {'by-active': {type: 'gsi', pk: 'status', sparse: true}} holds only rows whose status is set. A full scan of the index is the "list all live" query, no filter needed.

Start picking the projection — KEYS_ONLY / INCLUDE / ALL — based on what the read side needs. See Projection below.

GSI — Global Secondary Index

A secondary index with its own partition / sort key, replicated asynchronously, with independent read / write capacity. Supports any primary key schema (independent of the base table). Can be created and deleted at any time. See Global Secondary Indexes (AWS docs).

LSI — Local Secondary Index

A secondary index that shares the base table's partition key but chooses a different sort key. Created only at table-creation time — can't be added later. Strongly-consistent reads are available (unlike GSI, which is eventually consistent). Use when you need a second sort order within each partition. See Local Secondary Indexes (AWS docs).

Projection (index attribute copy)

What attributes the secondary index copies from the base table. Three options, picked at index-create time:

  • KEYS_ONLY — only the base-table key and the index key. Cheapest; best for indirect indices.
  • INCLUDE — the keys plus a declared list of extra attributes.
  • ALL — the full item on every index write. Most expensive, fastest for direct reads.

See Attribute projections in secondary indexes (AWS docs).

KeyConditionExpression

The partition-key-plus-optional-sort-key predicate on a Query. Must reference the partition key with =; the sort key can use =, range ops (<, <=, >, >=, BETWEEN), or BEGINS_WITH(...). This is not the same as ConditionExpression — you cannot filter arbitrary attributes here.

The toolkit ships two builders:

  • Primitive: buildKeyCondition(input, params?) from dynamodb-toolkit/expressions — takes a fully-prepared value string, appropriate for callers composing the key value themselves.
  • Ergonomic: adapter.buildKey(values, options?, params?) — takes an object keyed by keyFields names, uses the Adapter's declared keyFields / structuralKey to compose the prefix automatically. Supports {self: true} (self + descendants) and {partial: 'abc'} (narrow next tier).

Most callers reach for adapter.buildKey. The primitive is the escape hatch for custom schemes. See AWS Query API reference for the underlying grammar.

ConditionExpression

A predicate that gates a Put / Update / Delete (fail-on-mismatch semantics). The toolkit ships buildCondition for composing these. See Condition expressions (AWS docs).

FilterExpression

A predicate applied after the server-side Query / Scan returns matching items. Does not reduce read capacity — the scanned bytes still count.

The toolkit ships three ways to build one:

See Filter expressions (AWS docs).

ProjectionExpression

The list of attributes a read returns. The toolkit ships addProjection. See Projection expressions (AWS docs).

UpdateExpression

The SET / REMOVE / ADD / DELETE mutation on an UpdateCommand. The toolkit ships buildUpdate. See Update expressions (AWS docs).

BatchWriteItem / BatchGetItem

Non-atomic bulk endpoints. Per-call limits are asymmetric: BatchWriteItem caps at 25 items, BatchGetItem at 100. BatchWriteItem is the outlier — the other three DynamoDB bulk APIs (BatchGetItem, TransactWriteItems, TransactGetItems) are all 100. TransactWriteItems was raised from 25 to 100 in September 2022; BatchWriteItem never got the same treatment. See Batch and transactions → Per-call item limits for the full table. The toolkit's applyBatch chunks transparently and retries the SDK's UnprocessedItems with exponential backoff. See BatchWriteItem (AWS docs) and BatchGetItem (AWS docs).

TransactWriteItems / TransactGetItems

Atomic bulk endpoints. Per-call limit: 100 actions. The toolkit does not chunk transactions — atomicity rules that out. See TransactWriteItems (AWS docs) and Batch and transactions.

DocumentClient and marshalling

@aws-sdk/lib-dynamodb's DynamoDBDocumentClient.from(...) wraps the low-level client to marshal plain JS values (numbers, strings, booleans, arrays, Sets, nested objects) to / from DynamoDB's wire format automatically. Construct with {marshallOptions: {removeUndefinedValues: true}} for v2-parity behavior. See the lib-dynamodb package.

The toolkit assumes you are using the DocumentClient throughout — every example and type does. If you wire a raw DynamoDBClient in by mistake, marshalling errors surface immediately.

Glossary: cross-references

Clone this wiki locally