Add documentation for Data Prepper Iceberg source plugin#12164
Add documentation for Data Prepper Iceberg source plugin#12164lawofcycles wants to merge 4 commits into
Conversation
lawofcycles
requested review from
cwperks,
dlvenable,
epugh,
kolchfa-aws,
peterzhuamazon and
sumobrian
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). |
|
@dlvenable Could you please review this PR? |
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
…e options Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Reflect changes from data-prepper#6682 (shuffle) and data-prepper#6727 (shared catalog) that have been merged since the initial documentation was written. Changes: - Add top-level catalog configuration option with table-level override - Add shared catalog usage example - Add storage_path and ssl_client_auth to shuffle configuration table - Update io-impl description with default FileIO per catalog type - Fix iceberg_operation metadata description to reflect actual behavior - Clarify document_id description (concatenates instead of joins) Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
lawofcycles
force-pushed
the
doc/iceberg-source
branch
from
980ec79 to
6c273b6
Compare
|
@dlvenable I updated the documentation to reflect the latest merged code. The dependent PRs (opensearch-project/data-prepper#6682 and opensearch-project/data-prepper#6727) have both been merged now. Could you take a look when you get a chance? |
3 participants
Description
Add documentation for the Data Prepper Iceberg source plugin, an experimental CDC source introduced in Data Prepper 2.15. The plugin detects changes in Apache Iceberg tables by computing differences between snapshots.
This PR depends on opensearch-project/data-prepper#6682, which must be merged before this documentation can be published.
Issues Resolved
Closes #12163
Version
All (Data Prepper 2.15)
Frontend features
N/A
Checklist
Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check
here.