Skip to content

Commit f6312fb

Browse files
anewdbtsai
authored andcommitted
[SPARK-56650][SDP][CONNECT] Add AutoCDC Spark Connect APIs
### What changes were proposed in this pull request? Add the spark-connect protos to support AutoCDC flows. This is mainly two new protos: `AutoCdcFlowDetails`: similar to the existing `WriteRelationFlowDetails`, describes an AutoCDC flow. `SCDType` an enum to specify the type of SCD (slow changing dimension) storage type. For now, throw "not implemented" for the new flow type. ### Why are the changes needed? This new API is needed to specify an AutoCDC flow in a declarative pipeline. ### Does this PR introduce _any_ user-facing change? It introduces a new Spark Connect API, however, no implementation yet, and also no language wrappers (e.g. Python), yet. ### How was this patch tested? No tests, as this API is unimplemented for now. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 4.6 Closes #55589 from anew/add-sc-apis. Authored-by: Andreas Neumann <anew@apache.org> Signed-off-by: DB Tsai <dbtsai@dbtsai.com> (cherry picked from commit a88c275) Signed-off-by: DB Tsai <dbtsai@dbtsai.com>
1 parent 5c81510 commit f6312fb

4 files changed

Lines changed: 331 additions & 76 deletions

File tree

0 commit comments

Comments
 (0)