Skip to content

Commit a95e164

Browse files
authored
docs: clarify remove_orphan_files with vended credentials (#4170)
1 parent 95e0d62 commit a95e164

1 file changed

Lines changed: 2 additions & 3 deletions

File tree

site/content/in-dev/unreleased/_index.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,9 @@ To secure interactions with service connections, Polaris vends temporary storage
151151
execution. These credentials allow the query engine to run the query without requiring access to your cloud storage for
152152
Iceberg tables. This process is called credential vending.
153153

154-
As of now, the following limitation is known regarding Apache Iceberg support:
154+
The following applies to Apache Iceberg with Apache Spark:
155155

156-
- **remove_orphan_files:** Apache Spark can't use credential vending
157-
for this due to a known issue. See [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914) for details.
156+
- **remove_orphan_files**: By default this procedure lists storage through Spark's Hadoop layer, not Iceberg, so Polaris vended credentials may not apply to that step. From **Iceberg 1.10**, use **`prefix_listing => true`** in `CALL ... remove_orphan_files(...)` (or **`.usePrefixListing(true)`** on `SparkActions.deleteOrphanFiles`) so listing uses Iceberg's file client and vended credentials apply. See [apache/iceberg#12254](https://github.com/apache/iceberg/pull/12254). Note: Listing runs on the driver for this mode and can be slow for very large file sets.
158157

159158
### Identity and access management (IAM)
160159

0 commit comments

Comments
 (0)