Skip to content

Commit 3b80a63

Browse files
morningmanmorningman
authored
change parquet_meta release version to 4.1 (#3278)
1 parent 72df070 commit 3b80a63

12 files changed

Lines changed: 120 additions & 35 deletions

File tree

‎docs/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,12 +87,18 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8787

8888
### Supported Iceberg Versions
8989

90-
The current Iceberg dependency is version 1.6.1, which is compatible with higher versions of Iceberg.
90+
| Doris Version | Iceberg SDK Version |
91+
| --- | --- |
92+
| 2.1 | 1.6.1 |
93+
| 3.0 | 1.6.1 |
94+
| 3.1 | 1.9.1 |
95+
| 4.0 | 1.9.1 |
9196

9297
### Supported Iceberg Formats
9398

9499
* Supports Iceberg V1/V2 formats.
95-
* Supports Position Delete and Equality Delete.
100+
* Supports reading Position Delete and Equality Delete.
101+
* Supports reading Deletion Vector (Since 4.1.0).
96102

97103
### Supported Metadata Services
98104

‎docs/sql-manual/sql-functions/table-valued-functions/parquet-meta.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
The `parquet_meta` table-valued-function (tvf) can be used to read Footer metadata of Parquet files without scanning data pages. It allows for quickly viewing Row Group statistics, Schema, file-level metadata, KV metadata, and Bloom Filter probe results.
1010

11-
> This is an experimental feature, supported since version 4.0.3.
11+
> This is an experimental feature, supported since version 4.1.0.
1212
1313
## Syntax
1414

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,14 +89,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8989

9090
### 支持的 Iceberg 版本
9191

92-
当前使用的 Iceberg 依赖为 1.6.1 版本，可以兼容更高版本的 Iceberg。
92+
| Doris 版本 | Iceberg SDK 版本 |
93+
| --- | --- |
94+
| 2.1 | 1.6.1 |
95+
| 3.0 | 1.6.1 |
96+
| 3.1 | 1.9.1 |
97+
| 4.0 | 1.9.1 |
9398

9499
### 支持的 Iceberg 格式
95100

96101
* 支持 Iceberg V1/V2 格式。
97102

98103
* 支持 Position Delete 和 Equality Delete。
99104

105+
* 支持读取 Deletion Vector（自 4.1.0 版本支持）。
106+
100107
### 支持的元数据服务
101108

102109
* [Hive Metastore](../metastores/hive-metastore.md)

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-functions/table-valued-functions/parquet-meta.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
`parquet_meta` 表函数（table-valued-function，tvf）可以用于读取 Parquet 文件的 Footer 元数据，不会扫描数据页。它可以快速查看 Row Group 统计、Schema、文件级元数据、KV 元数据以及 Bloom Filter 探测结果。
1010

11-
> 该功能为实验功能，自 4.0.3 版本支持。
11+
> 该功能为实验功能，自 4.1.0 版本支持。
1212
1313
## 语法
1414

‎i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -89,14 +89,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8989

9090
### 支持的 Iceberg 版本
9191

92-
当前使用的 Iceberg 依赖为 1.6.1 版本，可以兼容更高版本的 Iceberg。
92+
| Doris 版本 | Iceberg SDK 版本 |
93+
| --- | --- |
94+
| 2.1 | 1.6.1 |
95+
| 3.0 | 1.6.1 |
96+
| 3.1 | 1.9.1 |
97+
| 4.0 | 1.9.1 |
9398

9499
### 支持的 Iceberg 格式
95100

96101
* 支持 Iceberg V1/V2 格式。
97102

98103
* 支持 Position Delete 和 Equality Delete。
99104

105+
* 支持读取 Deletion Vector（自 4.1.0 版本支持）。
106+
100107
### 支持的元数据服务
101108

102109
* [Hive Metastore](../metastores/hive-metastore.md)
@@ -153,7 +160,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
153160
| fixed(N) | char(N) | |
154161
| string | string | |
155162
| binary | string/varbinary |由 properties 中 `enable.mapping.varbinary` (4.0.2 后开始支持) 属性控制。默认为 `false`, 则映射到 `string`; 为 `true` 时，则映射到 `varbinary` 类型。 |
156-
| uuid | string/varbinary |4.0.2 之前映射成 `string`，之后则映射成 `varbinary` 类型。 |
163+
| uuid | string/varbinary |version < 4.0.2 映射成 `string` 类型; version == 4.0.2 则映射成 `varbinary` 类型。version > 4.0.2，由 properties 中 `enable.mapping.varbinary`属性控制。默认为 `false`, 则映射到 `string`; |
157164
| struct | struct（2.1.3 版本开始支持） | |
158165
| map | map（2.1.3 版本开始支持） | |
159166
| list | array | |
@@ -1151,17 +1158,19 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
11511158
<details>
11521159
<summary>3.1+ 版本 </summary>
11531160
<Tabs>
1154-
<TabItem value='FileSystem Catalog' label='FileSystem Catalog' default>
1161+
<TabItem value='S3 Tables Rest Catalog' label='S3 Tables Rest Catalog' default>
11551162
可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。
11561163
```sql
1157-
CREATE CATALOG test_s3tables_write_insert PROPERTIES (
1164+
CREATE CATALOG aws_s3_tables PROPERTIES (
11581165
'type' = 'iceberg',
1159-
'iceberg.catalog.type' = 's3tables',
1160-
'warehouse' = 'arn:aws:s3tables:ap-east-1:12345:bucket/doris-s3-table-bucket',
1161-
's3.region' = 'ap-east-1',
1162-
's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com',
1163-
's3.access_key' = '<ak>',
1164-
's3.secret_key' = '<sk>'
1166+
'iceberg.catalog.type' = 'rest',
1167+
'warehouse' = 'arn:aws:s3tables:us-east-1:<account_id>:bucket/<s3_table_bucket_name>',
1168+
'iceberg.rest.uri' = 'https://s3tables.us-east-1.amazonaws.com/iceberg',
1169+
'iceberg.rest.sigv4-enabled' = 'true',
1170+
'iceberg.rest.signing-name' = 's3tables',
1171+
'iceberg.rest.signing-region' = 'us-east-1',
1172+
'iceberg.rest.access-key-id' = '<ak>',
1173+
'iceberg.rest.secret-access-key' = '<sk>'
11651174
);
11661175
```
11671176
</TabItem>
@@ -1181,6 +1190,21 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
11811190
);
11821191
```
11831192
</TabItem>
1193+
<TabItem value='FileSystem Catalog' label='FileSystem Catalog' default>
1194+
可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。
1195+
```sql
1196+
CREATE CATALOG test_s3tables_write_insert PROPERTIES (
1197+
'type' = 'iceberg',
1198+
'iceberg.catalog.type' = 's3tables',
1199+
'warehouse' = 'arn:aws:s3tables:ap-east-1:12345:bucket/doris-s3-table-bucket',
1200+
's3.region' = 'ap-east-1',
1201+
's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com',
1202+
's3.access_key' = '<ak>',
1203+
's3.secret_key' = '<sk>'
1204+
);
1205+
```
1206+
</TabItem>
1207+
11841208
</Tabs>
11851209

11861210

‎i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,14 +89,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8989

9090
### 支持的 Iceberg 版本
9191

92-
当前使用的 Iceberg 依赖为 1.6.1 版本，可以兼容更高版本的 Iceberg。
92+
| Doris 版本 | Iceberg SDK 版本 |
93+
| --- | --- |
94+
| 2.1 | 1.6.1 |
95+
| 3.0 | 1.6.1 |
96+
| 3.1 | 1.9.1 |
97+
| 4.0 | 1.9.1 |
9398

9499
### 支持的 Iceberg 格式
95100

96101
* 支持 Iceberg V1/V2 格式。
97102

98103
* 支持 Position Delete 和 Equality Delete。
99104

105+
* 支持读取 Deletion Vector（自 4.1.0 版本支持）。
106+
100107
### 支持的元数据服务
101108

102109
* [Hive Metastore](../metastores/hive-metastore.md)

‎i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,14 +89,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8989

9090
### 支持的 Iceberg 版本
9191

92-
当前使用的 Iceberg 依赖为 1.6.1 版本，可以兼容更高版本的 Iceberg。
92+
| Doris 版本 | Iceberg SDK 版本 |
93+
| --- | --- |
94+
| 2.1 | 1.6.1 |
95+
| 3.0 | 1.6.1 |
96+
| 3.1 | 1.9.1 |
97+
| 4.0 | 1.9.1 |
9398

9499
### 支持的 Iceberg 格式
95100

96101
* 支持 Iceberg V1/V2 格式。
97102

98103
* 支持 Position Delete 和 Equality Delete。
99104

105+
* 支持读取 Deletion Vector（自 4.1.0 版本支持）。
106+
100107
### 支持的元数据服务
101108

102109
* [Hive Metastore](../metastores/hive-metastore.md)

‎i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-functions/table-valued-functions/parquet-meta.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
`parquet_meta` 表函数（table-valued-function，tvf）可以用于读取 Parquet 文件的 Footer 元数据，不会扫描数据页。它可以快速查看 Row Group 统计、Schema、文件级元数据、KV 元数据以及 Bloom Filter 探测结果。
1010

11-
> 该功能为实验功能，自 4.0.3 版本支持。
11+
> 该功能为实验功能，自 4.1.0 版本支持。
1212
1313
## 语法
1414

‎versioned_docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 34 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
{
33
"title": "Iceberg Catalog",
44
"language": "en",
5-
"description": "Iceberg Catalog in Apache Doris 2.1 connects to multiple Iceberg metadata services to query and write Iceberg tables across HDFS and cloud object storage, with detailed configuration, properties, query and write operations."
5+
"description": "Iceberg Catalog in Apache Doris connects to multiple Iceberg metadata services to query and write Iceberg tables across HDFS and cloud object storage, with detailed configuration, properties, query and write operations."
66
}
77
---
88

@@ -87,12 +87,18 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8787

8888
### Supported Iceberg Versions
8989

90-
The current Iceberg dependency is version 1.6.1, which is compatible with higher versions of Iceberg.
90+
| Doris Version | Iceberg SDK Version |
91+
| --- | --- |
92+
| 2.1 | 1.6.1 |
93+
| 3.0 | 1.6.1 |
94+
| 3.1 | 1.9.1 |
95+
| 4.0 | 1.9.1 |
9196

9297
### Supported Iceberg Formats
9398

9499
* Supports Iceberg V1/V2 formats.
95-
* Supports Position Delete and Equality Delete.
100+
* Supports reading Position Delete and Equality Delete.
101+
* Supports reading Deletion Vector (Since 4.1.0).
96102

97103
### Supported Metadata Services
98104

@@ -138,7 +144,7 @@ The current Iceberg dependency is version 1.6.1, which is compatible with higher
138144
| fixed(N) | char(N) | |
139145
| string | string | |
140146
| binary | string/varbinary | Controlled by the `enable.mapping.varbinary` property of Catalog (supported since 4.0.2). The default is `false`, which maps to `string`; when `true`, it maps to `varbinary` type.|
141-
| uuid | string/varbinary | Before 4.0.2, it was mapped to `string`; after that, it is mapped to `varbinary` type.|
147+
| uuid | string/varbinary | version < 4.0.2 maps to `string` type; version == 4.0.2 maps to `varbinary` type. For version > 4.0.2, it is controlled by the `enable.mapping.varbinary` property in the properties. The default is `false`, which maps to `string`.|
142148
| struct | struct (supported from version 2.1.3) | |
143149
| map | map (supported from version 2.1.3) | |
144150
| list | array | |
@@ -1137,17 +1143,19 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please
11371143
<details>
11381144
<summary>3.1+ Version</summary>
11391145
<Tabs>
1140-
<TabItem value='FileSystem Catalog' label='FileSystem Catalog' default>
1146+
<TabItem value='S3 Tables Rest Catalog' label='S3 Tables Rest Catalog' default>
11411147
Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation.
11421148
```sql
1143-
CREATE CATALOG test_s3tables_write_insert PROPERTIES (
1149+
CREATE CATALOG aws_s3_tables PROPERTIES (
11441150
'type' = 'iceberg',
1145-
'iceberg.catalog.type' = 's3tables',
1146-
'warehouse' = 'arn:aws:s3tables:ap-east-1:12345:bucket/doris-s3-table-bucket',
1147-
's3.region' = 'ap-east-1',
1148-
's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com',
1149-
's3.access_key' = '<ak>',
1150-
's3.secret_key' = '<sk>'
1151+
'iceberg.catalog.type' = 'rest',
1152+
'warehouse' = 'arn:aws:s3tables:us-east-1:<account_id>:bucket/<s3_table_bucket_name>',
1153+
'iceberg.rest.uri' = 'https://s3tables.us-east-1.amazonaws.com/iceberg',
1154+
'iceberg.rest.sigv4-enabled' = 'true',
1155+
'iceberg.rest.signing-name' = 's3tables',
1156+
'iceberg.rest.signing-region' = 'us-east-1',
1157+
'iceberg.rest.access-key-id' = '<ak>',
1158+
'iceberg.rest.secret-access-key' = '<sk>'
11511159
);
11521160
```
11531161
</TabItem>
@@ -1167,6 +1175,20 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please
11671175
);
11681176
```
11691177
</TabItem>
1178+
<TabItem value='FileSystem Catalog' label='FileSystem Catalog' default>
1179+
Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation.
1180+
```sql
1181+
CREATE CATALOG test_s3tables_write_insert PROPERTIES (
1182+
'type' = 'iceberg',
1183+
'iceberg.catalog.type' = 's3tables',
1184+
'warehouse' = 'arn:aws:s3tables:ap-east-1:12345:bucket/doris-s3-table-bucket',
1185+
's3.region' = 'ap-east-1',
1186+
's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com',
1187+
's3.access_key' = '<ak>',
1188+
's3.secret_key' = '<sk>'
1189+
);
1190+
```
1191+
</TabItem>
11701192
</Tabs>
11711193

11721194

‎versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,12 +87,18 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
8787

8888
### Supported Iceberg Versions
8989

90-
The current Iceberg dependency is version 1.6.1, which is compatible with higher versions of Iceberg.
90+
| Doris Version | Iceberg SDK Version |
91+
| --- | --- |
92+
| 2.1 | 1.6.1 |
93+
| 3.0 | 1.6.1 |
94+
| 3.1 | 1.9.1 |
95+
| 4.0 | 1.9.1 |
9196

9297
### Supported Iceberg Formats
9398

9499
* Supports Iceberg V1/V2 formats.
95-
* Supports Position Delete and Equality Delete.
100+
* Supports reading Position Delete and Equality Delete.
101+
* Supports reading Deletion Vector (Since 4.1.0).
96102

97103
### Supported Metadata Services
98104

0 commit comments

Comments
 (0)