Skip to content

Commit dc2a994

Browse files
authored
[yaml]: fix filter sql issue (#35490)
* fix keep sql expression * add filter sql test pipeline
1 parent f4f0e9a commit dc2a994

2 files changed

Lines changed: 24 additions & 1 deletion

File tree

sdks/python/apache_beam/yaml/tests/filter.yaml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,29 @@ pipelines:
116116
- {transaction_id: "T0302", product_name: "Monitor", category: "Electronics", price: 249.99}
117117

118118

119+
# Simple Filter using SQL
120+
- pipeline:
121+
type: chain
122+
transforms:
123+
- type: Create
124+
config:
125+
elements:
126+
- {transaction_id: "T0012", product_name: "Headphones", category: "Electronics", price: 59.99}
127+
- {transaction_id: "T5034", product_name: "Leather Jacket", category: "Apparel", price: 109.99}
128+
- {transaction_id: "T0024", product_name: "Aluminum Mug", category: "Kitchen", price: 29.9}
129+
- {transaction_id: "T0104", product_name: "Headphones", category: "Electronics", price: 59.99}
130+
- {transaction_id: "T0302", product_name: "Monitor", category: "Electronics", price: 249.99}
131+
- type: Filter
132+
config:
133+
language: sql
134+
keep: category = 'Electronics' and price < 100
135+
- type: AssertEqual
136+
config:
137+
elements:
138+
- {transaction_id: "T0012", product_name: "Headphones", category: "Electronics", price: 59.99}
139+
- {transaction_id: "T0104", product_name: "Headphones", category: "Electronics", price: 59.99}
140+
141+
119142
# Simple Filter with error handling
120143
- pipeline:
121144
type: composite

sdks/python/apache_beam/yaml/yaml_mapping.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -724,7 +724,7 @@ def _PyJsMapToFields(
724724
@beam.ptransform.ptransform_fn
725725
def _SqlFilterTransform(pcoll, sql_transform_constructor, keep, language):
726726
return pcoll | sql_transform_constructor(
727-
f'SELECT * FROM PCOLLECTION WHERE {keep}')
727+
f"SELECT * FROM PCOLLECTION WHERE {keep.get('expression')}")
728728

729729

730730
@beam.ptransform.ptransform_fn

0 commit comments

Comments
 (0)