Skip to content

Commit bdac5d6

Browse files
committed
Fix: preserve reverse_row_groups/reverse_rows across proto roundtrip
When PushdownSort removes SortExec and sets reverse_row_groups=true / reverse_rows=true on ParquetSource, these runtime flags must survive proto serialization. Without this, remote executors (RemoteExec) would deserialize the plan without reverse scanning, producing wrong order. Added reverse_row_groups and reverse_rows fields to ParquetScanExecNode proto message, serialized on encode and restored on decode.
1 parent 381faa1 commit bdac5d6

4 files changed

Lines changed: 56 additions & 0 deletions

File tree

datafusion/proto/proto/datafusion.proto

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1058,6 +1058,12 @@ message ParquetScanExecNode {
10581058
PhysicalExprNode predicate = 3;
10591059

10601060
datafusion_common.TableParquetOptions parquet_options = 4;
1061+
1062+
// Runtime reverse-scan flags set by PushdownSort optimizer.
1063+
// Must be preserved across proto roundtrips so that remote executors
1064+
// produce correctly reversed output after SortExec removal.
1065+
bool reverse_row_groups = 5;
1066+
bool reverse_rows = 6;
10611067
}
10621068

10631069
message CsvScanExecNode {

datafusion/proto/src/generated/pbjson.rs

Lines changed: 36 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datafusion/proto/src/generated/prost.rs

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datafusion/proto/src/physical_plan/mod.rs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -724,6 +724,14 @@ impl protobuf::PhysicalPlanNode {
724724
if let Some(predicate) = predicate {
725725
source = source.with_predicate(predicate);
726726
}
727+
728+
// Restore runtime reverse-scan flags from proto
729+
if scan.reverse_row_groups {
730+
source = source.with_reverse_row_groups(true);
731+
}
732+
if scan.reverse_rows {
733+
source = source.with_reverse_rows(true);
734+
}
727735
let base_config = parse_protobuf_file_scan_config(
728736
base_conf,
729737
ctx,
@@ -2672,6 +2680,8 @@ impl protobuf::PhysicalPlanNode {
26722680
)?),
26732681
predicate,
26742682
parquet_options: Some(conf.table_parquet_options().try_into()?),
2683+
reverse_row_groups: conf.reverse_row_groups(),
2684+
reverse_rows: conf.reverse_rows(),
26752685
},
26762686
)),
26772687
}));

0 commit comments

Comments
 (0)