Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,40 @@
/**
* A connector that reads from <a href="https://delta.io/">Delta Lake</a> tables.
*
* <p>This is work in progress. For more details and to track progress, please see <a
* href="https://github.com/apache/beam/issues/21100">Issue 21100</a>.
* <p>{@link DeltaIO} is offered as a Managed transform. This class is subject to change and should
* not be used directly. Instead, use it like so:
*
* <pre>{@code
* Map<String, Object> config = Map.of(
* "table", "gs://my-bucket/delta-table",
* "hadoop_config", Map.of(
* "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem",
* "fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS",
* "fs.gs.project.id", "my-project-id"));
*
* pipeline
* .apply(Managed.read(Managed.DELTA_LAKE).withConfig(config))
* .getSinglePCollection()
* .apply(ParDo.of(...));
* }</pre>
*
* <h2>Configuration Options</h2>
*
* Please check the <a href="https://beam.apache.org/documentation/io/managed-io/">Managed IO
* configuration page</a> for more details.
*
* <p>This is work in progress and is subject to change. For more details and to track progress, see
* <a href="https://github.com/apache/beam/issues/21100">Issue 21100</a>.
*/
@Internal
public class DeltaIO {

/**
* Reads rows from a Delta Lake table.
*
* <p>Normally, it is recommended to use {@link org.apache.beam.sdk.managed.Managed#read(String)}

Check warning on line 84 in sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaIO.java

View workflow job for this annotation

GitHub Actions / beam_PreCommit_Java_Delta_IO_Direct (Run Java_Delta_IO_Direct PreCommit)

Tag @link: reference not found: org.apache.beam.sdk.managed.Managed#read(String)
* with {@code Managed.DELTA_LAKE} instead of directly using this transform.
*/
public static ReadRows readRows() {
return new AutoValue_DeltaIO_ReadRows.Builder().build();
}
Expand Down Expand Up @@ -108,6 +136,15 @@
if (path == null) {
throw new IllegalArgumentException("Table path must be set.");
}
if (getTimestamp() != null) {
throw new UnsupportedOperationException(
"Reading from a specific timestamp is not supported yet");
}

if (getVersion() != null) {
throw new UnsupportedOperationException(
"Reading from a specific version is not supported yet");
}

Configuration conf = new Configuration();
Map<String, String> hadoopConfig = getHadoopConfig();
Expand Down
68 changes: 68 additions & 0 deletions website/www/site/content/en/documentation/io/managed-io.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,18 @@ and Beam SQL is invoked via the Managed API under the hood.
<th>Read Configuration</th>
<th>Write Configuration</th>
</tr>
<tr>
<td><strong>DELTA</strong></td>
<td>
<strong>table</strong> (<code style="color: green">str</code>)<br>
hadoop_config (<code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>)<br>
timestamp (<code style="color: green">str</code>)<br>
version (<code style="color: #f54251">int64</code>)<br>
</td>
<td>
Unavailable
</td>
</tr>
<tr>
<td><strong>ICEBERG</strong></td>
<td>
Expand Down Expand Up @@ -238,6 +250,62 @@ and Beam SQL is invoked via the Managed API under the hood.

## Configuration Details

### `DELTA` Read

<div class="table-container-wrapper">
<table class="table table-bordered">
<tr>
<th>Configuration</th>
<th>Type</th>
<th>Description</th>
</tr>
<tr>
<td>
<strong>table</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
Identifier of the Delta Lake table.
</td>
</tr>
<tr>
<td>
hadoop_config
</td>
<td>
<code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>
</td>
<td>
Properties passed to the Hadoop Configuration.
</td>
</tr>
<tr>
<td>
timestamp
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
Timestamp of the Delta Lake table to read.
</td>
</tr>
<tr>
<td>
version
</td>
<td>
<code style="color: #f54251">int64</code>
</td>
<td>
Version of the Delta Lake table to read.
</td>
</tr>
</table>
</div>

### `ICEBERG` Read

<div class="table-container-wrapper">
Expand Down
Loading