Commit 3796eb7
committed
refactor: simplify CDC writer wiring and nest CdcOptions proto message
Follow-up tidy-ups to the parquet CDC options:
* Drop the chunk-size validation in the parquet writer path; the bounds are
enforced downstream by parquet-rs, so the extra DataFusion-side checks (and
their two unit tests) are redundant. Reference the `content_defined_chunking`
fields directly instead of aliasing them.
* Nest the proto `CdcOptions` message inside `ParquetOptions`
(`parquet_options::CdcOptions`), since it is a parquet write option. Field
tags are unchanged, so the wire format is unaffected.
* Add `From` helpers in `datafusion-proto-common` for the config <-> proto
`CdcOptions` mapping in both directions, replacing the inline field copies in
the `ParquetOptions` conversions. The `datafusion-proto` `file_formats.rs`
mapping stays inline (the orphan rule blocks a shared impl there) and only
switches to the nested type path.
Regenerated prost/pbjson for proto-common and proto-models.1 parent 81e756c commit 3796eb7
8 files changed
Lines changed: 222 additions & 241 deletions
File tree
- datafusion
- common/src/file_options
- proto-common
- proto
- src
- from_proto
- generated
- to_proto
- proto-models/src/generated
- proto/src/logical_plan
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
250 | 250 | | |
251 | 251 | | |
252 | 252 | | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | 253 | | |
266 | 254 | | |
267 | | - | |
268 | | - | |
269 | | - | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
270 | 258 | | |
271 | 259 | | |
272 | 260 | | |
| |||
887 | 875 | | |
888 | 876 | | |
889 | 877 | | |
890 | | - | |
891 | | - | |
892 | | - | |
893 | | - | |
894 | | - | |
895 | | - | |
896 | | - | |
897 | | - | |
898 | | - | |
899 | | - | |
900 | | - | |
901 | | - | |
902 | | - | |
903 | | - | |
904 | | - | |
905 | | - | |
906 | | - | |
907 | | - | |
908 | | - | |
909 | | - | |
910 | | - | |
911 | | - | |
912 | | - | |
913 | | - | |
914 | | - | |
915 | 878 | | |
916 | 879 | | |
917 | 880 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
636 | 636 | | |
637 | 637 | | |
638 | 638 | | |
639 | | - | |
640 | 639 | | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
645 | | - | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
646 | 648 | | |
647 | 649 | | |
648 | 650 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1130 | 1130 | | |
1131 | 1131 | | |
1132 | 1132 | | |
1133 | | - | |
1134 | | - | |
1135 | | - | |
1136 | | - | |
1137 | | - | |
1138 | | - | |
| 1133 | + | |
1139 | 1134 | | |
1140 | 1135 | | |
1141 | 1136 | | |
1142 | 1137 | | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
1143 | 1149 | | |
1144 | 1150 | | |
1145 | 1151 | | |
| |||
0 commit comments