fix: compact nested projection list elements by index#171
Merged
pdf-amzn merged 1 commit intoJun 10, 2026
Conversation
Multi-index list projections (e.g. `a[0], a[2]`) collapsed to one element: each path built a one-element list and the merge overwrote it. Replace the per-path wrap/merge with a projection trie that walks the item once, emitting selected indices in ascending order. Preserves projected NULLs and merges same-index sub-paths.
yesyayen
requested review from
LeeroyHannigan,
amrith,
c33howard,
jcshepherd and
pdf-amzn
as code owners
pdf-amzn
approved these changes
Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This makes nested
ProjectionExpressionresults match Amazon DynamoDB when a projection selects multiple elements of the same list. When a projection selects more than one element of a list (for examplea[0], a[2]), Amazon DynamoDB returns those elements compacted into a new list, ordered by their original index. ExtendDB collapsed them to a single element.I replaced the per-path resolve / wrap / merge assembly in
crates/core/src/expression/projection.rswith a projection trie.apply_projectionnow builds one trie from all paths (resolving#namerefs, rejecting index-start paths) and walks the item once: a map keeps only the selected keys, and a list emits the selected indices in ascending order.Behavior: today vs Amazon DynamoDB
The item under test has
mylist = [zero, one, two, three],listOfMaps = [{val:a0,x:x0},{val:a1,x:x1},{val:a2,x:x2}], andlistWithNull = [keep0, NULL, keep2].mylist[0], mylist[2]{"mylist":["two"]}{"mylist":["zero","two"]}mylist[2], mylist[0]{"mylist":["zero"]}{"mylist":["zero","two"]}(ordered by index, not by expression)mylist[1], mylist[3]{"mylist":["three"]}{"mylist":["one","three"]}listOfMaps[0].val, listOfMaps[2].val{"listOfMaps":[{"val":"a2"}]}{"listOfMaps":[{"val":"a0"},{"val":"a2"}]}listWithNull[0], listWithNull[2]{"listWithNull":["keep2"]}{"listWithNull":["keep0","keep2"]}listWithNull(whole list){"listWithNull":["keep0",NULL,"keep2"]}listWithNull[1](the NULL){"listWithNull":[NULL]}Why
This is a conformance gap. It is a data-correctness bug: customers got the wrong elements back from a list projection. #162 corrected the read-path validation and parse-error labels and called out nested projection correctness as a separate follow-up; this is that follow-up.
Closes: n/a (no separate issue; tracked as the deferred nested-projection item from #162).
Testing done
Added
tests/test_nested_projection.py(9 dual-target cases). Every expected result was captured from Amazon DynamoDB.projection.rs: 17 pass, including 8 new cases covering compaction, index ordering, NULL handling, list-of-maps sub-field projection, and same-index merge.Out of scope
The following are tracked separately:
apply_projectionrebuilds the projection trie per returned item on Query and Scan. The paths are identical across items, so building the trie once per request and reusing it is a follow-up optimization. This matches the old apply path, which also did per-item work, so it is not a regression.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).ADR / RFC: n/a.
Breaking changes
n/a
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.