Skip to content

Commit ac52623

Browse files
chamikaramjchamikaramj
authored
Adds documentation for the Delta Lake Read Managed I/O (#39495)
1 parent de234c7 commit ac52623

3 files changed

Lines changed: 124 additions & 3 deletions

File tree

‎sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaIO.java‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,12 +50,40 @@
5050
/**
5151
* A connector that reads from <a href="https://delta.io/">Delta Lake</a> tables.
5252
*
53-
* <p>This is work in progress. For more details and to track progress, please see <a
54-
* href="https://github.com/apache/beam/issues/21100">Issue 21100</a>.
53+
* <p>{@link DeltaIO} is offered as a Managed transform. This class is subject to change and should
54+
* not be used directly. Instead, use it like so:
55+
*
56+
* <pre>{@code
57+
* Map<String, Object> config = Map.of(
58+
* "table", "gs://my-bucket/delta-table",
59+
* "hadoop_config", Map.of(
60+
* "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem",
61+
* "fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS",
62+
* "fs.gs.project.id", "my-project-id"));
63+
*
64+
* pipeline
65+
* .apply(Managed.read(Managed.DELTA_LAKE).withConfig(config))
66+
* .getSinglePCollection()
67+
* .apply(ParDo.of(...));
68+
* }</pre>
69+
*
70+
* <h2>Configuration Options</h2>
71+
*
72+
* Please check the <a href="https://beam.apache.org/documentation/io/managed-io/">Managed IO
73+
* configuration page</a> for more details.
74+
*
75+
* <p>This is work in progress and is subject to change. For more details and to track progress, see
76+
* <a href="https://github.com/apache/beam/issues/21100">Issue 21100</a>.
5577
*/
5678
@Internal
5779
public class DeltaIO {
5880

81+
/**
82+
* Reads rows from a Delta Lake table.
83+
*
84+
* <p>Normally, it is recommended to use {@link org.apache.beam.sdk.managed.Managed#read(String)}
85+
* with {@code Managed.DELTA_LAKE} instead of directly using this transform.
86+
*/
5987
public static ReadRows readRows() {
6088
return new AutoValue_DeltaIO_ReadRows.Builder().build();
6189
}
@@ -108,6 +136,15 @@ public PCollection<Row> expand(PBegin input) {
108136
if (path == null) {
109137
throw new IllegalArgumentException("Table path must be set.");
110138
}
139+
if (getTimestamp() != null) {
140+
throw new UnsupportedOperationException(
141+
"Reading from a specific timestamp is not supported yet");
142+
}
143+
144+
if (getVersion() != null) {
145+
throw new UnsupportedOperationException(
146+
"Reading from a specific version is not supported yet");
147+
}
111148

112149
Configuration conf = new Configuration();
113150
Map<String, String> hadoopConfig = getHadoopConfig();

‎website/www/site/content/en/documentation/io/connectors.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Apache Beam I/O connectors provide read and write transforms for the most popula
2222

2323
I/O connectors denoted _via X-language_ have been made available using the Apache Beam [multi-language pipelines framework](/documentation/programming-guide/#multi-language-pipelines).
2424

25-
I/O connectors denoted as _Supported via Managed API_ can be accessed using the simplified _managed I/O_ APIs for [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/managed/Managed.html) and [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.managed.html#module-apache_beam.transforms.managed).
25+
I/O connectors denoted as _Supported via Managed API_ can be accessed using the simplified [managed I/O_ APIs](https://beam.apache.org/documentation/io/managed-io/).
2626

2727
## Built-in I/O Connectors
2828

@@ -1297,6 +1297,22 @@ This table provides a consolidated, at-a-glance overview of the available built-
12971297
<td class="absent">✘</td>
12981298
<td class="absent">✘</td>
12991299
</tr>
1300+
<tr>
1301+
<td>Delta Lake</td>
1302+
<td class="present">✔</td>
1303+
<td class="present">✘</td>
1304+
<td class="present">
1305+
✔
1306+
<a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/managed/Managed.html">native</a>
1307+
</td>
1308+
<td>Not available</td>
1309+
<td>Not available</td>
1310+
<td>Not available</td>
1311+
<td>Not available</td>
1312+
<td class="present">✔</td>
1313+
<td class="absent">✘</td>
1314+
<td class="absent">✔</td>
1315+
</tr>
13001316
</table>
13011317
</div>
13021318

‎website/www/site/content/en/documentation/io/managed-io.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,18 @@ and Beam SQL is invoked via the Managed API under the hood.
5858
<th>Read Configuration</th>
5959
<th>Write Configuration</th>
6060
</tr>
61+
<tr>
62+
<td><strong>DELTA</strong></td>
63+
<td>
64+
<strong>table</strong> (<code style="color: green">str</code>)<br>
65+
hadoop_config (<code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>)<br>
66+
timestamp (<code style="color: green">str</code>)<br>
67+
version (<code style="color: #f54251">int64</code>)<br>
68+
</td>
69+
<td>
70+
Unavailable
71+
</td>
72+
</tr>
6173
<tr>
6274
<td><strong>ICEBERG</strong></td>
6375
<td>
@@ -238,6 +250,62 @@ and Beam SQL is invoked via the Managed API under the hood.
238250

239251
## Configuration Details
240252

253+
### `DELTA` Read
254+
255+
<div class="table-container-wrapper">
256+
<table class="table table-bordered">
257+
<tr>
258+
<th>Configuration</th>
259+
<th>Type</th>
260+
<th>Description</th>
261+
</tr>
262+
<tr>
263+
<td>
264+
<strong>table</strong>
265+
</td>
266+
<td>
267+
<code style="color: green">str</code>
268+
</td>
269+
<td>
270+
Identifier of the Delta Lake table.
271+
</td>
272+
</tr>
273+
<tr>
274+
<td>
275+
hadoop_config
276+
</td>
277+
<td>
278+
<code>map[<span style="color: green;">str</span>, <span style="color: green;">str</span>]</code>
279+
</td>
280+
<td>
281+
Properties passed to the Hadoop Configuration.
282+
</td>
283+
</tr>
284+
<tr>
285+
<td>
286+
timestamp
287+
</td>
288+
<td>
289+
<code style="color: green">str</code>
290+
</td>
291+
<td>
292+
Timestamp of the Delta Lake table to read.
293+
</td>
294+
</tr>
295+
<tr>
296+
<td>
297+
version
298+
</td>
299+
<td>
300+
<code style="color: #f54251">int64</code>
301+
</td>
302+
<td>
303+
Version of the Delta Lake table to read.
304+
</td>
305+
</tr>
306+
</table>
307+
</div>
308+
241309
### `ICEBERG` Read
242310

243311
<div class="table-container-wrapper">

0 commit comments

Comments
 (0)