-
Notifications
You must be signed in to change notification settings - Fork 0
Mass operations
dynamodb-toolkit/mass exports primitives the Adapter composes for its mass methods (see Adapter: Mass methods). Use them directly when you need finer control or when you don't have an Adapter.
See also (AWS JS SDK v3):
QueryCommand/ScanCommand·BatchWriteCommand·BatchGetCommand· Paginating Query results. Vocabulary: Concepts.
import {
paginateList, iterateList, iterateItems,
readList, readListGetItems, readListByKeys, readOrderedListByKeys,
writeList, deleteList, deleteListByKeys,
copyList, moveList, getTotal
} from 'dynamodb-toolkit/mass';Offset/limit pagination over a Query or Scan. Returns {data, offset, limit, total?}.
When params.FilterExpression is set, paginateList accumulates matches across pages — DynamoDB's Limit is pre-filter, so naive use returns short pages and false-empty results. paginateList paginates until enough matches are accumulated.
const r = await paginateList(client, {TableName: 'planets'}, {offset: 0, limit: 10});
// → {data: [...10 items...], offset: 0, limit: 10, total: 61}needTotal: false skips the Select: 'COUNT' round trips for the rest of the dataset — much cheaper for endless-scroll feeds.
Async generators. iterateList yields raw page objects ({Items, Count, LastEvaluatedKey, …}); iterateItems yields individual items.
for await (const item of iterateItems(client, {TableName: 'planets'})) {
console.log(item.name);
}Reads one page from Query / Scan, hands the raw SDK response to fn, and returns either the next params (with ExclusiveStartKey already set) or null when there are no more pages.
readList(
client: DynamoDBDocumentClient,
params: QueryCommandInput | ScanCommandInput,
fn: (data: {Items?: unknown[]; Count?: number; LastEvaluatedKey?: unknown; /* full SDK response */}) => void | Promise<void>
): Promise<typeof params | null>;The callback receives the full SDK response for the page — .Items, .Count, .LastEvaluatedKey, .ScannedCount, etc. It returns nothing; use it to push items into an accumulator, send them downstream, etc. readList does not loop — it reads one page, calls the callback once, returns the next params so the caller can drive the loop.
The nextParams return value carries ExclusiveStartKey = LastEvaluatedKey plus everything else the caller passed in. Re-enter readList with it to read the next page.
Drain loop example:
let p = {TableName: 'planets', FilterExpression: '#c = :c',
ExpressionAttributeNames: {'#c': 'climate'}, ExpressionAttributeValues: {':c': 'frozen'}};
const all = [];
while (p) {
p = await readList(client, p, data => {
if (data.Items) all.push(...data.Items);
// could also: break early, flush a buffer, send to a stream, etc.
});
}
// all now holds every matching itemSame shape, but returns {nextParams, items} directly — no callback. Equivalent to readList with a push-to-array callback, inlined.
const {nextParams, items} = await readListGetItems(client, params);
// items is always an array (possibly empty); nextParams is null on the last pageWrapper over getBatch that returns just the items as a plain array.
const items = await readListByKeys(client, 'planets', [{name: 'Hoth'}, {name: 'Bespin'}]);Same but preserves caller key order, with undefined for missing keys. The SDK's BatchGetItem returns items in arbitrary order — this helper rebuilds the original order.
const items = await readOrderedListByKeys(client, 'planets', [{name: 'A'}, {name: 'B'}, {name: 'C'}]);
// items[0] is A, items[1] is B, items[2] is C — or undefined for missesBatchWriteItem puts. mapFn is optional and defaults to identity (x => x) — use it to transform each item in flight, typically to add technical fields or to normalize the wire shape. Returns the count of items processed.
writeList(
client: DynamoDBDocumentClient,
tableName: string,
items: unknown[],
mapFn?: (item: unknown) => unknown
): Promise<number>;Example — same job the Adapter's prepare hook does, but for a standalone write:
// Item shape on the wire: {name, climate}
// DB shape: adds a sentinel '-t' field the caller uses to distinguish "v3 items"
await writeList(client, 'planets', planets, item => ({...item, '-t': 1}));The -t here is author convention for a technical field (see Concepts → technicalPrefix and managed fields for the declarative version that validates + strips this prefix for you). The toolkit does not require or reserve the - prefix unless technicalPrefix is declared.
Read items matching a Query / Scan, extract a key object from each, and batch-delete. keyFn is optional and defaults to identity — which is correct when ProjectionExpression on params already limits the read to just key attributes; otherwise you pass a keyFn that extracts the key.
deleteList(
client: DynamoDBDocumentClient,
params: QueryCommandInput | ScanCommandInput,
keyFn?: (item: unknown) => object
): Promise<number>;Example — delete all planets with climate = 'frozen', extracting just the name key:
const n = await deleteList(
client,
{
TableName: 'planets',
FilterExpression: '#c = :c',
ExpressionAttributeNames: {'#c': 'climate'},
ExpressionAttributeValues: {':c': 'frozen'}
},
item => ({name: item.name})
);
// → number of deletes DynamoDB acceptedBatch-delete a list of keys directly — no read phase. Use when you already have the key list in memory.
Read items via scan/query, optionally transform via mapFn, batch-write back to the same table (pulled from params.TableName). Default mapFn is identity, which effectively re-writes every matching item unchanged — rarely useful; almost every call supplies a mapFn that rewrites the key.
copyList(
client: DynamoDBDocumentClient,
params: QueryCommandInput | ScanCommandInput,
mapFn?: (item: unknown) => unknown
): Promise<number>;Read via scan/query, paired puts + deletes per chunk (12 puts + 12 deletes per BatchWrite to stay under the 25-action BatchWriteItem limit — see src/mass/move-list.js).
moveList(
client: DynamoDBDocumentClient,
params: QueryCommandInput | ScanCommandInput,
mapFn?: (item: unknown) => unknown,
keyFn?: (item: unknown) => object
): Promise<number>;Both callbacks default to identity. Same caveat as copyList: without a mapFn that changes the item, the "move" puts the same item back with the same key (no-op) — typical use supplies a mapFn that rewrites the key. The keyFn extracts what to delete from the source row; defaults to identity, which is correct when params projects only key attributes.
mapFn returning a falsy value skips both legs for that item — the put is not queued and the paired delete is not queued either. This prevents the "source deleted but copy never written" silent data-loss that an asymmetric filter would produce. If you want to drop an item selectively, make your mapFn return null / undefined / false and the source remains untouched.
Returns the total batch-actions count (puts + deletes ≈ 2× the moved-item count on success).
Select: 'COUNT' pagination — counts items matching the query/scan without returning them. Useful for the total in pagination envelopes when needTotal: true.