Commit c259da2
[SPARK-48091][SQL] Preserve aliases inside lambda when ExtractGenerator restructures plan
### What changes were proposed in this pull request?
Fix `ExtractGenerator` to preserve aliases inside lambda functions when restructuring the plan.
Previously, `ExtractGenerator` called `trimNonTopLevelAliases` on all expressions in the project list before extracting the generator. This stripped aliases inside lambda functions (e.g., struct(x.as("data"))) before `CreateStruct` could resolve them into struct field names.
The fix uses `trimNonTopLevelAliases` only for pattern matching (to detect generators via `AliasedGenerator`), but preserves the original untrimmed expression for non-generator project items.
### Why are the changes needed?
When using explode together with transform in the same `select statement`, aliases used inside the transformed column's `struct()` are ignored. Field names become auto-generated (x_1, x_2) instead of the user-specified alias. This only happens with the DataFrame/Dataset API, not with SQL.
### Does this PR introduce _any_ user-facing change?
Yes. Struct field aliases inside transform lambdas are now correctly preserved when explode (or any generator) is in the same `select`.
### How was this patch tested?
Added a test in `GeneratorFunctionSuite` verifying that struct field aliases are preserved when explode and transform are used together, including single and multiple aliases.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #55892 from shrirangmhalgi/SPARK-48091-explode-transform-alias.
Authored-by: Shrirang Mhalgi <shrirangmhalgi@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit ccdb31a)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent 5dacfe3 commit c259da2
2 files changed
Lines changed: 37 additions & 2 deletions
File tree
- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions
- core/src/test/scala/org/apache/spark/sql
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
115 | 119 | | |
116 | 120 | | |
117 | 121 | | |
| |||
Lines changed: 32 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| |||
765 | 765 | | |
766 | 766 | | |
767 | 767 | | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
768 | 799 | | |
769 | 800 | | |
770 | 801 | | |
| |||
0 commit comments