fix(clickhouse): support arrayMap and arrayFilter roundtrips [CODEX]#7794
Conversation
…order functions [CLAUDE] ClickHouse's higher-order array functions arrayMap(lambda, arr) and arrayFilter(lambda, arr) were being parsed as Anonymous expressions because the ClickHouse parser lacked entries for their uppercased forms (ARRAYMAP, ARRAYFILTER). As a result, transpiling to DuckDB produced invalid ARRAYMAP() and ARRAYFILTER() calls instead of LIST_TRANSFORM() and LIST_FILTER(). Additionally, the ClickHouse generator had no Transform entry, so DuckDB's LIST_TRANSFORM and Spark's TRANSFORM arrived in ClickHouse as the wrong name. Note: argument order in ClickHouse is reversed relative to the canonical exp.Transform/exp.ArrayFilter convention (lambda first, array second).
There was a problem hiding this comment.
Nice work, I left some comments.
My suggestion is the following:
Remove the current validate_all tests of this PR, and only use vaildate_identity tests for testing the roundtrip (input clickhouse -> output clickhouse) of the added functions for example self.validate_identity("arrayFilter(x -> x > 0, arr)").assert_is(exp.ArrayFilter). This would make the PR simple and mergable.
Then, if you want you can investigate various inputs - outputs for the transpilation, as you saw the NULL creates semantic issues for the transpilation. After, detecting all this edge cases let's make a following PR that contains the robust transpilation.
ruff-format requires the multi-arg lambda to fit on one line.
gaoflow
changed the title
|
Addressed in
I also updated the PR title and description to reflect the narrowed roundtrip scope. |
geooo109
left a comment
geooo109
left a comment
There was a problem hiding this comment.
thank you, great work!
3 participants
Problem
ClickHouse higher-order array functions
arrayMap(lambda, arr)andarrayFilter(lambda, arr)were parsed as generic/unsupported function expressions instead of SQLGlot's semantic array function nodes.This made ClickHouse -> ClickHouse roundtrips lossy for these functions, and it also meant
arrayMapdid not use the existingexp.Transformrepresentation.Fix
ARRAYMAPandARRAYFILTER.arrayMap(lambda, array)toexp.Transform(this=array, expression=lambda)andarrayFilter(lambda, array)toexp.ArrayFilter(this=array, expression=lambda).exp.TransformsoarrayMaproundtrips back to ClickHouse syntax.Tests
Added focused ClickHouse identity tests:
arrayMap(x -> x + 1, arr)roundtrips and parses asexp.Transform.arrayFilter(x -> x > 0, arr)roundtrips and parses asexp.ArrayFilter.Local verification:
This pull request was prepared with the assistance of AI, under my direction and review.