Commit f6312fb
[SPARK-56650][SDP][CONNECT] Add AutoCDC Spark Connect APIs
### What changes were proposed in this pull request?
Add the spark-connect protos to support AutoCDC flows. This is mainly two new protos:
`AutoCdcFlowDetails`: similar to the existing `WriteRelationFlowDetails`, describes an AutoCDC flow.
`SCDType` an enum to specify the type of SCD (slow changing dimension) storage type.
For now, throw "not implemented" for the new flow type.
### Why are the changes needed?
This new API is needed to specify an AutoCDC flow in a declarative pipeline.
### Does this PR introduce _any_ user-facing change?
It introduces a new Spark Connect API, however, no implementation yet, and also no language wrappers (e.g. Python), yet.
### How was this patch tested?
No tests, as this API is unimplemented for now.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
Closes #55589 from anew/add-sc-apis.
Authored-by: Andreas Neumann <anew@apache.org>
Signed-off-by: DB Tsai <dbtsai@dbtsai.com>
(cherry picked from commit a88c275)
Signed-off-by: DB Tsai <dbtsai@dbtsai.com>1 parent 5c81510 commit f6312fb
4 files changed
Lines changed: 331 additions & 76 deletions
File tree
- python/pyspark/sql/connect/proto
- sql/connect
- common/src/main/protobuf/spark/connect
- server/src/main/scala/org/apache/spark/sql/connect/pipelines
0 commit comments