ElectroDB Bug Report: hidden: true attributes break synthetic cursors with count pagination

There's seems to be an issue with either how I'm using hidden attributes or with the ElectroDB library. I had Cursor try to findf the code in question and put together this bug issue. Hopefully this is helpful and not just AI barf.

Describe the bug

When an attribute used as a GSI partition key composite has hidden: true, ElectroDB's count-based pagination generates a synthetic cursor with a truncated partition key value. The cursor omits the attribute's actual value from the composite key string, causing DynamoDB to reject the ExclusiveStartKey on subsequent pages with:

"The query can return at most one row and cannot be restarted"

The first page returns correctly. The error only occurs when the cursor from page 1 is passed to retrieve page 2, because the cursor's partition key no longer matches the index's actual partition key value stored in DynamoDB.

Root cause (two layers of hidden stripping):

Hidden attributes are stripped at two levels before the synthetic cursor is constructed:

1. Attribute getter (schema.js, _makeGet): the generated get() method returns undefined immediately when this.hidden is true, before any value processing occurs.
2. formatItemForRetrieval (schema.js): deletes any attribute in hiddenAttributes from the result object.

When executeQuery builds the synthetic cursor (in the count path at entity.js ~line 774), it calls _fromCompositeToKeysByIndex with lastItem — which has already passed through both stripping layers. _findFacets then reads undefined for the hidden attribute, and _makeIndexKeys builds a composite key string with the value missing (e.g. $service#attr___ instead of $service#attr___value).

Fix:

Branch: fix/hidden-attribute-cursor-pagination (forked)

The fix is a single-file change to src/entity.js. Before formatResponse strips hidden attributes, the raw DynamoDB response.Items are captured. When building the synthetic cursor, translateFromFields is called directly on the raw item — bypassing both the attribute getter and formatItemForRetrieval. This ensures all index key facets are present regardless of the hidden flag.

--- a/src/entity.js
+++ b/src/entity.js
@@ -709,6 +709,14 @@
       ExclusiveStartKey = response.LastEvaluatedKey;

+      // When using count-based pagination, we need access to the raw
+      // DynamoDB items to build accurate synthetic cursors. Hidden
+      // attributes (hidden: true) are stripped by formatItemForRetrieval,
+      // but they may be part of index key composites needed for the cursor.
+      let rawItems = config.count && response.Items
+        ? response.Items.slice()
+        : undefined;
+
       response = this.formatResponse(response, parameters.IndexName, {
@@ -755,6 +763,9 @@
         if (moreItemsThanRequired) {
           items = items.slice(0, config.count - prevCount);
+          if (rawItems) {
+            rawItems = rawItems.slice(0, config.count - prevCount);
+          }
         }
@@ -772,9 +783,18 @@
         if (moreItemsThanRequired || count === config.count) {
-          const lastItem = results[results.length - 1];
+          // Build the cursor from the last item. Use the raw DynamoDB
+          // item (before hidden-attribute stripping) and translateFromFields
+          // directly (bypassing formatItemForRetrieval) to ensure all index
+          // key facets are present. Both the attribute getter and the
+          // retrieval formatter strip hidden attributes, which breaks
+          // synthetic cursor construction when a hidden attribute is part
+          // of an index key composite.
+          let cursorSource = results[results.length - 1];
+          if (rawItems && rawItems.length > 0) {
+            cursorSource = this.model.schema.translateFromFields(
+              rawItems[rawItems.length - 1],
+              config,
+            );
+          }
           ExclusiveStartKey = this._fromCompositeToKeysByIndex({
             indexName,
-            provided: lastItem,
+            provided: cursorSource,
           });

Key insight: formatItemForRetrieval cannot simply be called with a "skip hidden" flag because the attribute's own get() method (built by _makeGet in schema.js line 274-277) also returns undefined for hidden attributes — this happens inside _fulfillAttributeMutationMethod before the deletion loop even runs. Using translateFromFields directly is the correct approach because it performs a simple field-name-to-attribute-name remapping without invoking getters or stripping hidden attributes.

ElectroDB Version

Reproduced on both 3.5.3 and 3.7.2.

ElectroDB Playground Link

N/A — the bug requires a live DynamoDB table with enough items to trigger multi-page results, which the playground cannot provide.

Entity/Service Definitions

Minimal reproduction model:

import { Entity } from 'electrodb';

const MyEntity = new Entity({
  model: { entity: 'MyEntity', version: '1', service: 'myapp' },
  attributes: {
    itemId: { type: 'string', required: true },
    name: { type: 'string', required: true },
    status: { type: ['active', 'disabled', 'deleted'] as const, default: 'active', required: true },
    // This attribute is used as a GSI PK composite.
    // BUG: hidden: true causes cursor generation to omit the value.
    __type__: { type: 'string', default: 'MYENTITY', hidden: true, readOnly: true },
    createdAt: { type: 'number', default: () => Date.now(), readOnly: true, required: true },
    updatedAt: {
      type: 'number',
      watch: '*',
      default: () => Date.now(),
      set: () => Date.now(),
      readOnly: true,
      required: true,
    },
  },
  indexes: {
    primary: {
      pk: { field: 'pk', composite: ['itemId'] },
      sk: { field: 'sk', composite: [] },
    },
    // Fan-out index: __type__ as PK allows listing all entities of this type
    byCollection: {
      index: 'gsi1pk-gsi1sk-index',
      pk: { field: 'gsi1pk', composite: ['__type__'] },
      sk: { field: 'gsi1sk', composite: ['createdAt'] },
    },
  },
}, { client: dynamoClient, table: 'MyTable' });

Steps to reproduce:

1. Create the entity above and seed the table with more items than your page size (e.g., 10+ items).
2. Query using count pagination:

// Page 1 — works fine, returns 5 items
const page1 = await MyEntity.query
  .byCollection({ __type__: 'MYENTITY' })
  .go({ count: 5, order: 'desc' });

console.log(page1.data.length); // 5
console.log(page1.cursor);      // non-null cursor string

// Page 2 — FAILS with DynamoDB error
const page2 = await MyEntity.query
  .byCollection({ __type__: 'MYENTITY' })
  .go({ count: 5, cursor: page1.cursor, order: 'desc' });
// Error: "The query can return at most one row and cannot be restarted"

3. Decode the base64 cursor from page 1 to see the problem:

{
  "gsi1pk": "$myapp#__type___",
  "gsi1sk": "$myentity_1#createdat_1775503542082",
  "pk": "$myapp#itemid_abc123",
  "sk": "$myentity_1"
}

Note gsi1pk is "$myapp#__type___" — the attribute value myentity is missing. The actual value stored in DynamoDB is "$myapp#__type___myentity".

Expected behavior

The synthetic cursor should include the full partition key value regardless of the hidden flag. The cursor should contain:

{
  "gsi1pk": "$myapp#__type___myentity",
  "gsi1sk": "$myentity_1#createdat_1775503542082",
  "pk": "$myapp#itemid_abc123",
  "sk": "$myentity_1"
}

This would allow DynamoDB to correctly position the ExclusiveStartKey and return the next page of results.

Errors

Error: Error thrown by DynamoDB client: "The query can return at most one row and cannot be restarted"
  - For more detail on this error reference: https://electrodb.dev/en/reference/errors/#aws-error

Additional context

• The bug affects any attribute with hidden: true that is used as a composite in a GSI partition key or sort key. This includes both custom sentinel attributes (like __type__) and built-in patterns like entityType.
• The hidden flag works correctly for data storage (items are written with the full key value) and for first-page queries (the query parameters include the full key). It only fails during cursor generation because ElectroDB reconstructs the cursor from the hydrated result object, which has hidden fields stripped at both the getter and formatter levels.
• Workaround: Remove hidden: true from any attribute used as a GSI key composite. The attribute will then appear in query results, but can be filtered out at the application layer during serialization. This does not require any data migration since the stored DynamoDB items are unaffected.
• Using limit + pages: 1 instead of count avoids the bug because DynamoDB's native LastEvaluatedKey is used instead of ElectroDB's synthetic cursor. However, this changes pagination semantics (DynamoDB's Limit applies before FilterExpression, so page sizes become inconsistent when filters exclude items).