Skip to content

Commit 8c8c391

Browse files
authored
Docs: clarify SparkSessionCatalog function limitations (#15736)
1 parent 8a51a68 commit 8c8c391

2 files changed

Lines changed: 18 additions & 0 deletions

File tree

docs/docs/spark-configuration.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,16 @@ Spark's built-in catalog supports existing v1 and v2 tables tracked in a Hive Me
112112

113113
This configuration can use same Hive Metastore for both Iceberg and non-Iceberg tables.
114114

115+
`SparkSessionCatalog` is useful when you want `spark_catalog` to work with both Iceberg and non-Iceberg
116+
tables in the same metastore.
117+
118+
!!! note
119+
Spark before 4.2.0 does not support `V2Function` in the session catalog. See
120+
[SPARK-54760](https://issues.apache.org/jira/browse/SPARK-54760) ([apache/spark#53531](https://github.com/apache/spark/pull/53531)) for details. As a result,
121+
catalog-scoped SQL functions such as `system.bucket`, `system.days`, and `system.iceberg_version`
122+
are not available through `spark_catalog`. To work around this limitation, configure a separate
123+
Iceberg catalog with `org.apache.iceberg.spark.SparkCatalog` and call them through that catalog.
124+
115125
### Using catalog specific Hadoop configuration values
116126

117127
Similar to configuring Hadoop properties by using `spark.hadoop.*`, it's possible to set per-catalog Hadoop configuration values when using Spark by adding the property for the catalog with the prefix `spark.sql.catalog.(catalog-name).hadoop.*`. These properties will take precedence over values configured globally using `spark.hadoop.*` and will only affect Iceberg tables.

docs/docs/spark-queries.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,14 @@ writing filters that match Iceberg partition transforms. These functions are ava
5151
[Iceberg catalog](spark-configuration.md#catalog-configuration); they are not registered in Spark's
5252
built-in catalog.
5353

54+
!!! note
55+
Spark before 4.2.0 does not support `V2Function` in the session catalog.
56+
Queries such as `SELECT spark_catalog.system.bucket(16, id)` fail even when
57+
`spark_catalog` is configured with `org.apache.iceberg.spark.SparkSessionCatalog`.
58+
See [SPARK-54760](https://issues.apache.org/jira/browse/SPARK-54760) ([apache/spark#53531](https://github.com/apache/spark/pull/53531)) for details.
59+
To use Iceberg SQL functions, call them through a catalog configured with
60+
`org.apache.iceberg.spark.SparkCatalog`.
61+
5462
Use the `system` namespace when calling these functions:
5563

5664
```sql

0 commit comments

Comments
 (0)