Query Iceberg topics using Databricks and Unity Catalog#1050
Closed
Query Iceberg topics using Databricks and Unity Catalog#1050
Conversation
This reverts commit 8b09d9a.
This reverts commit 2a6f5a1.
This reverts commit fbf0815.
Co-authored-by: Joyce Fee <joyce@redpanda.com> Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
Co-authored-by: Michele Cyran <michele@redpanda.com>
Co-authored-by: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com>
Co-authored-by: Martin Schneppenheim <23424570+weeco@users.noreply.github.com> Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
Co-authored-by: Jake Cahill <45230295+JakeSCahill@users.noreply.github.com>
This reverts commit c129523.
Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> Co-authored-by: Angela Simms <102690377+asimms41@users.noreply.github.com>
kbatuigas
commented
Apr 3, 2025
| ---- | ||
| iceberg_enabled: true | ||
| iceberg_catalog_type: rest | ||
| iceberg_rest_catalog_endpoint: https://<workspace-instance>/api/2.1/unity-catalog/iceberg |
Contributor
Author
There was a problem hiding this comment.
The Databricks docs specify /api/2.1/unity-catalog/iceberg but our internal examples seem to use /api/2.1/unity-catalog/iceberg-rest. Can someone confirm what the correct value is?
kbatuigas
commented
Apr 3, 2025
| ---- | ||
| # Example for redpanda.iceberg.mode=key_value with 1 record produced to topic | ||
| +--------------------------------------------------------------------------+------------+-------------------------+ | ||
| | redpanda | value | redpanda.timestamp_hour | |
Contributor
Author
There was a problem hiding this comment.
Is timestamp_hour a new addition to this schema?
kbatuigas
commented
Apr 3, 2025
| | "headers":null,"key":"68656c6c6f"} | | | | ||
| +--------------------------------------------------------------------------+------------+-------------------------+ | ||
| ---- | ||
|
|
Contributor
Author
There was a problem hiding this comment.
Is there anything we should mention in this doc regarding metadata refresh? Does Unity automatically refresh table metadata?
Co-authored-by: Jake Cahill <45230295+JakeSCahill@users.noreply.github.com> Co-authored-by: Michele Cyran <michele@redpanda.com>
Co-authored-by: Michele Cyran <michele@redpanda.com>
Co-authored-by: Michele Cyran <michele@redpanda.com>
…led for new installs (#965) Co-authored-by: Andrew Stucki <andrew.stucki@gmail.com> Co-authored-by: David Yu <yongshinyu@gmail.com> Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Co-authored-by: Joyce Fee <joyce@redpanda.com> Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> Co-authored-by: Chris Seto <chriskseto@gmail.com> Co-authored-by: Angela Simms <102690377+asimms41@users.noreply.github.com>
![hyperlint-ai-deprecated[bot]](https://avatars.githubusercontent.com/in/718456?s=60&v=4){kind=small}
|
|
||
| A storage credential is a Databricks object that controls access to external object storage, in this case S3. You associate a storage credential with an AWS IAM role that defines what actions Unity Catalog can perform in the S3 bucket. | ||
|
|
||
| Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials[official Databricks documentation^] to create an AWS IAM role that has the required permissions for the bucket. When you have completed these steps, you should have the following configured in AWS and Databricks: |
Contributor
There was a problem hiding this comment.
Identified issues
- Vale Style Guide - (Spelling-error) Did you really mean 'Databricks'?
- Vale Style Guide - (CustomStyle.SwapWords-warning) Consider using 'will' or 'must' instead of 'should'
Proposed fix
Suggested change
| Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials[official Databricks documentation^] to create an AWS IAM role that has the required permissions for the bucket. When you have completed these steps, you should have the following configured in AWS and Databricks: | |
| Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials[official Databricks documentation^] to create an AWS IAM role that has the required permissions for the bucket. When you have completed these steps, you will have the following configured in AWS and Databricks: |
The spelling of 'Databricks' is correct, and I believe the use of 'should' can be replaced with 'will' to make the instruction more definitive.
Co-authored-by: Michele Cyran <michele@redpanda.com> Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Co-authored-by: Suslik Da-Rete <suslik_da-rete@protonmail.ch> Co-authored-by: Rogger Vasquez <59714880+r-vasquez@users.noreply.github.com> Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> Co-authored-by: Yaniv Ben Hemo <70286779+yanivbh1@users.noreply.github.com> Co-authored-by: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com> Co-authored-by: Paulo Borges <paulohtb@hotmail.com> Co-authored-by: Bill Chambers <wchambers@ischool.berkeley.edu> Co-authored-by: David Yu <yongshinyu@gmail.com> Co-authored-by: Angela Simms <102690377+asimms41@users.noreply.github.com> Co-authored-by: Joyce Fee <joyce@redpanda.com> Co-authored-by: Martin Schneppenheim <23424570+weeco@users.noreply.github.com> Co-authored-by: Stephan Dollberg <stephan@redpanda.com> Co-authored-by: Andrew Stucki <andrew.stucki@gmail.com> Co-authored-by: Chris Seto <chriskseto@gmail.com>
Contributor
|
@kbatuigas Please update/provide a status so folks know where this PR stands, what the delay is, etc. Thanks. |
Contributor
Author
|
@mattschumpert given our internal conversations with Databricks regarding the availability of managed tables as well as creating Iceberg tables using the metadata path, can we go ahead and close this PR for now, and reopen in June? |
|
Whatever is most efficient. I'm sure we will publish this content. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request adds a new guide for querying Iceberg topics using Databricks and Unity Catalog to the documentation. The key changes include updating the navigation to include the new guide and adding detailed instructions for the integration process.
Documentation updates:
modules/ROOT/nav.adoc: Added a new link to the guide for querying Iceberg topics with Databricks.New guide:
modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc: Added a comprehensive guide on how to query Redpanda topics as Iceberg tables in Databricks using Unity Catalog. This includes prerequisites, creating storage credentials and external locations, updating cluster configurations, and querying the Iceberg table using Databricks SQL.Resolves https://redpandadata.atlassian.net/browse/
Review deadline: 4 April
Page previews
https://deploy-preview-1050--redpanda-docs-preview.netlify.app/25.1/manage/iceberg/iceberg-topics-databricks-unity/
Checks