Bug in addAttributeToProject in enhanced client for list attributes

[DanielBauman88]

Describe the bug

addAttributeToProject generates invalid ExpressionAttributeName key when projecting list elements

Summary

ScanEnhancedRequest.builder().addAttributeToProject("tags[0]") generates an ExpressionAttributeNames entry with brackets in the key (#AMZN_MAPPED_tags[0]), which DynamoDB rejects with a validation error. The [0] list dereference belongs in the projection expression itself, not inside the #placeholder key.

Affected Component

• Package: AwsJavaSdk-DynamoDb-Enhanced (2.0)
• Class: software.amazon.awssdk.enhanced.dynamodb.internal.ProjectionExpression
• Source: ProjectionExpression.java

Reproduction

DynamoDbTable<Item> table = enhancedClient.table("MyTable", TableSchema.fromBean(Item.class));

// Documented as valid per NestedAttributeName javadoc:
//   "List item 0 of ListAttribute can be created as NestedAttributeName.create("ListAttribute[0]")"
ScanEnhancedRequest request = ScanEnhancedRequest.builder()
        .addAttributeToProject("tags[0]")
        .build();

table.scan(request).iterator().next(); // THROWS

Note the docs here imply that this should work: https://docs.aws.amazon.com/java/api/latest/software/amazon/awssdk/enhanced/dynamodb/NestedAttributeName.html (the addAttributeToProject delegates down - just sugar and turns the string into a NestedAttributeName)

Quote from docs:
List item 0 of ListAttribute can be created as NestedAttributeName.create("ListAttribute[0]")

Error:

software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
  ExpressionAttributeNames contains invalid key: Syntax error; key: "#AMZN_MAPPED_tags[0]"
  (Service: DynamoDb, Status Code: 400)

The request payload that us generated looks like this and is invalid

{"TableName":"BugRepro-Projection","ProjectionExpression":"#AMZN_MAPPED_tags[0]","ExpressionAttributeNames":{"#AMZN_MAPPED_tags[0]":"tags[0]"}}

Expected Behavior

The SDK should generate:

ExpressionAttributeNames: { "#AMZN_MAPPED_tags": "tags" }
ProjectionExpression: "#AMZN_MAPPED_tags[0]"

i.e., the placeholder #AMZN_MAPPED_tags maps to the attribute name tags, and [0] is appended to the placeholder in the expression string to dereference the list element.

Root Cause

In ProjectionExpression.java:

1. NestedAttributeName.create("tags[0]") produces a single element "tags[0]"
2. createAttributePlaceholders calls PROJECTION_EXPRESSION_KEY_MAPPER which does AMZN_MAPPED + cleanAttributeName(k) — but cleanAttributeName doesn't strip the [0] suffix
3. This produces the key #AMZN_MAPPED_tags[0]
4. convertToNameExpression then uses this key directly in the expression string

The fix should:

• Strip [N] suffixes from the attribute name when generating the placeholder key
• Keep [N] in the expression output appended after the placeholder

Workaround

Use the low-level DynamoDbClient.scan() with a raw projectionExpression string:

client.scan(ScanRequest.builder()
        .tableName("MyTable")
        .projectionExpression("tags[0]")
        .build());

This works because DynamoDB natively supports tags[0] in projection expressions — the issue is solely in how the Enhanced Client maps it to ExpressionAttributeNames.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

See above.

Current Behavior

See above.

Reproduction Steps

Note, this repro will leave behind a table in your account called BugRepro-Projection which you should delete when you're done.

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.NestedAttributeName;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.model.ScanEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.Page;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.PutItemRequest;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.model.ResourceInUseException;

import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

/**
 * Minimal reproduction of addAttributeToProject bug with list index projection.
 *
 * The Enhanced Client's addAttributeToProject("myList[0]") generates an invalid
 * ExpressionAttributeName key "#AMZN_MAPPED_myList[0]" which DynamoDB rejects.
 *
 * Run: java -cp <classpath> org.junit.runner.JUnitCore amazon.rds.mrr.AddAttributeToProjectBugRepro
 * Requires: AWS credentials with DynamoDB access in us-east-1
 */
public class PotatoTest {

    private static final String TABLE_NAME = "BugRepro-Projection";

    private DynamoDbClient client;
    private DynamoDbEnhancedClient enhancedClient;
    private DynamoDbTable<Item> table;

    @DynamoDbBean
    public static class Item {
        private String id;
        private List<String> tags;

        @DynamoDbPartitionKey
        public String getId() { return id; }
        public void setId(String id) { this.id = id; }
        public List<String> getTags() { return tags; }
        public void setTags(List<String> tags) { this.tags = tags; }
    }

    @Before
    public void setUp() {
        client = DynamoDbClient.builder().region(Region.US_EAST_1).build();
        enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(client).build();
        table = enhancedClient.table(TABLE_NAME, TableSchema.fromBean(Item.class));
        try {
            table.createTable(b -> b.provisionedThroughput(
                    ProvisionedThroughput.builder().readCapacityUnits(5L).writeCapacityUnits(5L).build()));
            client.waiter().waitUntilTableExists(b -> b.tableName(TABLE_NAME));
        } catch (ResourceInUseException e) {
            System.out.println("Caught ResourceInUseException");
        }

        // Put an item with a list attribute using low-level client
        Map<String, AttributeValue> item = new HashMap<>();
        item.put("id", AttributeValue.builder().s("item-1").build());
        item.put("tags", AttributeValue.builder().l(
                AttributeValue.builder().s("alpha").build(),
                AttributeValue.builder().s("beta").build(),
                AttributeValue.builder().s("gamma").build()
        ).build());
        client.putItem(PutItemRequest.builder().tableName(TABLE_NAME).item(item).build());
    }

    @After
    public void tearDown() {
    }

    @Test
    public void addAttributeToProject_withListIndex_failsAgainstRealDynamoDB() {
        // This is documented as valid usage per NestedAttributeName javadoc:
        //   "List item 0 of ListAttribute can be created as NestedAttributeName.create("ListAttribute[0]")"
        // But it generates an invalid ExpressionAttributeName key with brackets.
        ScanEnhancedRequest request = ScanEnhancedRequest.builder()
                //.addAttributeToProject("tags[0]")
                .addNestedAttributesToProject(NestedAttributeName.create("tags[0]"))
                .build();

        // Throws: DynamoDbException: ExpressionAttributeNames contains invalid key:
        //         Syntax error; key: "#AMZN_MAPPED_tags[0]"
        Iterator<Page<Item>> pages = table.scan(request).iterator();
        while (pages.hasNext()) {
            pages.next();
        }
    }

    @Test
    public void rawProjectionExpression_withListIndex_worksAgainstRealDynamoDB() {
        // Workaround: use the low-level client with a raw projection expression
        var response = client.scan(software.amazon.awssdk.services.dynamodb.model.ScanRequest.builder()
                .tableName(TABLE_NAME)
                .projectionExpression("tags[0]")
                .build());

        // This succeeds - DynamoDB supports tags[0] in projection expressions natively
        assert response.items().size() == 1;
        assert response.items().get(0).get("tags").l().get(0).s().equals("alpha");
    }

    @Test
    public void projectionExpressionWithExpressionAttributeNames_withListIndex_worksAgainstRealDynamoDB() {
        // Workaround: use the low-level client with expressionAttributeNames
        // The key insight: the placeholder key must NOT contain brackets,
        // but the projection expression string can append [0] to the placeholder.
        var response = client.scan(software.amazon.awssdk.services.dynamodb.model.ScanRequest.builder()
                .tableName(TABLE_NAME)
                .projectionExpression("#t[0]")
                .expressionAttributeNames(Map.of("#t", "tags"))
                .build());

        // This succeeds - DynamoDB allows [0] appended to a placeholder in the expression
        assert response.items().size() == 1;
        assert response.items().get(0).get("tags").l().get(0).s().equals("alpha");
    }

}

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

latest

JDK version used

17

Operating System and version

osx and alinux