Skip to content

Commit 508f99b

Browse files
CalvinKirsmorningman
CalvinKirs
and
morningman
authored
Update OSS_HDFS docs (#3108)
## Versions - [x] dev - [x] 4.x - [x] 3.x - [x] 2.1 ## Languages - [x] Chinese - [ ] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built --------- Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>
1 parent b65bd01 commit 508f99b

16 files changed

Lines changed: 436 additions & 131 deletions

File tree

‎docs/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,8 @@ The currently dependent Paimon version is 1.0.0.
490490
```
491491
</TabItem>
492492
<TabItem value='OSS' label='OSS'>
493+
Use OSS
494+
493495
```sql
494496
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
495497
'type' = 'paimon',
@@ -501,6 +503,21 @@ The currently dependent Paimon version is 1.0.0.
501503
'oss.secret_key'='<sk>'
502504
);
503505
```
506+
507+
Use OSS-HDFS
508+
509+
```sql
510+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
511+
'type' = 'paimon',
512+
'paimon.catalog.type' = 'filesystem',
513+
'warehouse' = 'oss://bucket/regression/paimon1',
514+
'fs.oss-hdfs.support' = 'true',
515+
'oss.hdfs.access_key' = '<ak>',
516+
'oss.hdfs.secret_key' = '<sk>',
517+
'oss.hdfs.endpoint' = 'cn-beijing.oss-dls.aliyuncs.com',
518+
'oss.hdfs.region' = 'cn-beijing'
519+
);
520+
```
504521
</TabItem>
505522
<TabItem value='COS' label='COS'>
506523
```sql
@@ -584,6 +601,8 @@ The currently dependent Paimon version is 1.0.0.
584601
```
585602
</TabItem>
586603
<TabItem value='OSS' label='OSS'>
604+
Use OSS
605+
587606
```sql
588607
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
589608
'type' = 'paimon',
@@ -595,6 +614,21 @@ The currently dependent Paimon version is 1.0.0.
595614
'oss.secret_key'='<sk>'
596615
);
597616
```
617+
618+
Use OSS-HDFS
619+
620+
```sql
621+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
622+
'type' = 'paimon',
623+
'paimon.catalog.type' = 'filesystem',
624+
'warehouse' = 'oss://bucket/regression/paimon1',
625+
'oss.hdfs.enabled' = 'true',
626+
'oss.access_key' = 'your-access-key',
627+
'oss.secret_key' = 'your-secret-key',
628+
'oss.endpoint' = 'cn-hangzhou.oss-dls.aliyuncs.com',
629+
'oss.region' = 'cn-hangzhou'
630+
);
631+
```
598632
</TabItem>
599633
<TabItem value='COS' label='COS'>
600634
```sql

‎docs/lakehouse/storages/aliyun-oss.md‎

Lines changed: 20 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,11 @@ This document describes the parameters required to access Alibaba Cloud OSS, whi
1313
- Export properties
1414
- Outfile properties
1515

16-
**Doris uses S3 Client to access Alibaba Cloud OSS through S3-compatible protocol.**
16+
## OSS
1717

18-
## Parameter Overview
18+
Doris uses S3 Client to access Alibaba Cloud OSS through S3-compatible protocol.
19+
20+
### Parameter Overview
1921

2022
| Property Name | Legacy Name | Description | Default Value |
2123
| ------------------------------ | ---------------------------- | ------------------------------------------------------------ | ------------- |
@@ -48,7 +50,7 @@ For versions before 3.1:
4850
"s3.region" = "cn-beijing"
4951
```
5052

51-
## Usage Recommendations
53+
### Usage Recommendations
5254

5355
* It is recommended to use the `oss.` prefix for configuration parameters to ensure consistency and clarity with Alibaba Cloud OSS.
5456
* For versions before 3.1, please use the legacy name `s3.` as the prefix.
@@ -63,14 +65,15 @@ Accessing data stored on OSS-HDFS is slightly different from directly accessing
6365

6466
### Parameter Overview
6567

66-
| Property Name | Legacy Name | Description | Default Value | Required |
67-
| ------------------------------ | ---------------------------- | ------------------------------------------------------------ | ------------- | -------- |
68-
| oss.hdfs.endpoint | s3.endpoint | Alibaba Cloud OSS-HDFS service endpoint, e.g., `cn-hangzhou.oss-dls.aliyuncs.com`. | None | Yes |
69-
| oss.hdfs.access_key | s3.access_key | OSS Access Key for authentication | None | Yes |
70-
| oss.hdfs.secret_key | s3.secret_key | OSS Secret Key, used together with Access Key | None | Yes |
71-
| oss.hdfs.region | s3.region | Region ID where the OSS bucket is located, e.g., `cn-beijing`. | None | Yes |
72-
| oss.hdfs.fs.defaultFS | | Supported in version 3.1. Specifies the file system access path for OSS, e.g., `oss://my-bucket/`. | None | No |
73-
| oss.hdfs.hadoop.config.resources | | Supported in version 3.1. Specifies the path containing OSS file system configuration. Requires relative path. Default directory is `/plugins/hadoop_conf/` under the (FE/BE) deployment directory (can be changed by modifying hadoop_config_dir in fe.conf/be.conf). All FE and BE nodes need to configure the same relative path. Example: `hadoop/conf/core-site.xml,hadoop/conf/hdfs-site.xml`. | None | No |
68+
| Property Name | Legacy Name | Description | Default Value | Required |
69+
| ------------------------------ |----------------| ----------------------------------------------------------- | ------------- | -------- |
70+
| oss.hdfs.endpoint | oss.endpoint | Alibaba Cloud OSS-HDFS service endpoint, e.g., `cn-hangzhou.oss-dls.aliyuncs.com`. | None | Yes |
71+
| oss.hdfs.access_key | oss.access_key | OSS Access Key for authentication | None | Yes |
72+
| oss.hdfs.secret_key | oss.secret_key | OSS Secret Key, used together with Access Key | None | Yes |
73+
| oss.hdfs.region | oss.region | Region ID where the OSS bucket is located, e.g., `cn-beijing`. | None | Yes |
74+
| oss.hdfs.fs.defaultFS | | Supported in version 3.1. Specifies the file system access path for OSS, e.g., `oss://my-bucket/`. | None | No |
75+
| oss.hdfs.hadoop.config.resources | | Supported in version 3.1. Specifies the path containing OSS file system configuration. Requires relative path. Default directory is `/plugins/hadoop_conf/` under the (FE/BE) deployment directory (can be changed by modifying hadoop_config_dir in fe.conf/be.conf). All FE and BE nodes need to configure the same relative path. Example: `hadoop/conf/core-site.xml,hadoop/conf/hdfs-site.xml`. | None | No |
76+
| fs.oss-hdfs.support |oss.hdfs.enabled | Supported in version 3.1. Explicitly declares the enabling of OSS-HDFS functionality. Needs to be set to true | None | No |
7477

7578
> For versions before 3.1, please use legacy names.
7679
@@ -99,6 +102,7 @@ If the configuration files contain the parameters mentioned above in this docume
99102
### Example Configuration
100103

101104
```properties
105+
"fs.oss-hdfs.support" = "true",
102106
"oss.hdfs.access_key" = "your-access-key",
103107
"oss.hdfs.secret_key" = "your-secret-key",
104108
"oss.hdfs.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
@@ -108,8 +112,9 @@ If the configuration files contain the parameters mentioned above in this docume
108112
For versions before 3.1:
109113

110114
```
111-
"s3.access_key" = "your-access-key",
112-
"s3.secret_key" = "your-secret-key",
113-
"s3.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
114-
"s3.region" = "cn-hangzhou"
115+
"oss.hdfs.enabled" = "true",
116+
"oss.access_key" = "your-access-key",
117+
"oss.secret_key" = "your-secret-key",
118+
"oss.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
119+
"oss.region" = "cn-hangzhou"
115120
```

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -489,6 +489,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
489489
```
490490
</TabItem>
491491
<TabItem value='OSS' label='OSS'>
492+
使用 OSS
492493
```sql
493494
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
494495
'type' = 'paimon',
@@ -500,6 +501,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
500501
'oss.secret_key'='<sk>'
501502
);
502503
```
504+
505+
使用 OSS-HDFS
506+
507+
```sql
508+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
509+
'type' = 'paimon',
510+
'paimon.catalog.type' = 'filesystem',
511+
'warehouse' = 'oss://bucket/regression/paimon1',
512+
'fs.oss-hdfs.support' = 'true',
513+
'oss.hdfs.access_key' = '<ak>',
514+
'oss.hdfs.secret_key' = '<sk>',
515+
'oss.hdfs.endpoint' = 'cn-beijing.oss-dls.aliyuncs.com',
516+
'oss.hdfs.region' = 'cn-beijing'
517+
);
518+
```
503519
</TabItem>
504520
<TabItem value='COS' label='COS'>
505521
```sql
@@ -583,6 +599,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
583599
```
584600
</TabItem>
585601
<TabItem value='OSS' label='OSS'>
602+
使用 OSS
603+
586604
```sql
587605
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
588606
'type' = 'paimon',
@@ -594,6 +612,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
594612
'oss.secret_key'='<sk>'
595613
);
596614
```
615+
616+
使用 OSS-HDFS
617+
618+
```sql
619+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
620+
'type' = 'paimon',
621+
'paimon.catalog.type' = 'filesystem',
622+
'warehouse' = 'oss://bucket/regression/paimon1',
623+
'oss.hdfs.enabled' = 'true',
624+
'oss.access_key' = 'your-access-key',
625+
'oss.secret_key' = 'your-secret-key',
626+
'oss.endpoint' = 'cn-hangzhou.oss-dls.aliyuncs.com',
627+
'oss.region' = 'cn-hangzhou'
628+
);
629+
```
597630
</TabItem>
598631
<TabItem value='COS' label='COS'>
599632
```sql

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/storages/aliyun-oss.md‎

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,11 @@
1313
- Export 属性
1414
- Outfile 属性
1515

16-
**Doris 使用 S3 Client，通过 S3 兼容协议访问阿里云 OSS。**
16+
## OSS
1717

18-
## 参数总览
18+
Doris 使用 S3 Client，通过 S3 兼容协议访问阿里云 OSS。
19+
20+
### 参数总览
1921

2022
| 属性名称 | 曾用名 | 描述 | 默认值 |
2123
| ------------------------------ | ---------------------------- | ------------------------------------------------------------ | ------ |
@@ -30,7 +32,7 @@
3032

3133
> 3.1 版本之前，请使用曾用名。
3234
33-
## 示例配置
35+
### 示例配置
3436

3537
```properties
3638
"oss.access_key" = "your-access-key",
@@ -48,7 +50,7 @@
4850
"s3.region" = "cn-beijing"
4951
```
5052

51-
## 使用建议
53+
### 使用建议
5254

5355
* 推荐使用 `oss.` 前缀配置参数，保证与阿里云 OSS 的一致性和清晰度。
5456
* 3.1 之前的版本，请使用曾用名 `s3.` 作为前缀。
@@ -63,14 +65,15 @@ OSS-HDFS 服务（JindoFS 服务）是一个阿里云云原生数据湖存储功
6365

6466
### 参数总览
6567

66-
| 属性名称 | 曾用名 | 描述 | 默认值 |是否必须 |
67-
| ------------------------------ | ---------------------------- | ------------------------------------------------------------ | ------ | --- |
68-
| oss.hdfs.endpoint | s3.endpoint | 阿里云 OSS-HDFS 服务的 Endpoint，例如 `cn-hangzhou.oss-dls.aliyuncs.com`。 | 无 | 是 |
69-
| oss.hdfs.access_key | s3.access_key | OSS Access Key，用于身份验证 | 无 | 是 |
70-
| oss.hdfs.secret_key | s3.secret_key | OSS Secret Key，与 Access Key 配合使用 | 无 | 是 |
71-
| oss.hdfs.region | s3.region | OSS bucket 所在的地域 ID，例如 `cn-beijing`。 | 无 | 是 |
72-
| oss.hdfs.fs.defaultFS | | 3.1 版本支持。指定 OSS 的文件系统访问路径，例如 `oss://my-bucket/`。 | 无 | 否 |
73-
| oss.hdfs.hadoop.config.resources | | 3.1 版本支持。指定包含 OSS 文件系统配置的路径，需使用相对路径，默认目录为（FE/BE）部署目录下的 /plugins/hadoop_conf/（可修改 fe.conf/be.conf 中的 hadoop_config_dir 来更改默认路径）。所有 FE 和 BE 节点需配置相同相对路径。示例：`hadoop/conf/core-site.xml,hadoop/conf/hdfs-site.xml`。 | 无 | 否 |
68+
| 属性名称 | 曾用名 | 描述 | 默认值 |是否必须 |
69+
|----------------------------------|---------------| ------------------------------------------------------------ | ------ | --- |
70+
| oss.hdfs.endpoint | oss.endpoint | 阿里云 OSS-HDFS 服务的 Endpoint，例如 `cn-hangzhou.oss-dls.aliyuncs.com`。 | 无 | 是 |
71+
| oss.hdfs.access_key | oss.access_key | OSS Access Key，用于身份验证 | 无 | 是 |
72+
| oss.hdfs.secret_key | oss.secret_key | OSS Secret Key，与 Access Key 配合使用 | 无 | 是 |
73+
| oss.hdfs.region | oss.region | OSS bucket 所在的地域 ID，例如 `cn-beijing`。 | 无 | 是 |
74+
| oss.hdfs.fs.defaultFS | | 3.1 版本支持。指定 OSS 的文件系统访问路径，例如 `oss://my-bucket/`。 | 无 | 否 |
75+
| oss.hdfs.hadoop.config.resources | | 3.1 版本支持。指定包含 OSS 文件系统配置的路径，需使用相对路径，默认目录为（FE/BE）部署目录下的 /plugins/hadoop_conf/（可修改 fe.conf/be.conf 中的 hadoop_config_dir 来更改默认路径）。所有 FE 和 BE 节点需配置相同相对路径。示例：`hadoop/conf/core-site.xml,hadoop/conf/hdfs-site.xml`。 | 无 | 否 |
76+
| fs.oss-hdfs.support |oss.hdfs.enabled | 3.1 版本支持。显示声明启用 OSS-HDFS 功能。需要设置为 true | 无 | 否 |
7477

7578
> 3.1 版本之前，请使用曾用名。
7679
@@ -100,17 +103,19 @@ OSS-HDFS 支持通过 `oss.hdfs.hadoop.config.resources` 参数来指定 HDFS
100103
### 示例配置
101104

102105
```properties
106+
"fs.oss-hdfs.support" = "true",
103107
"oss.hdfs.access_key" = "your-access-key",
104108
"oss.hdfs.secret_key" = "your-secret-key",
105109
"oss.hdfs.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
106110
"oss.hdfs.region" = "cn-hangzhou"
107111
```
108112

109-
3.1 之前的版：
113+
3.1 之前的版本：
110114

111115
```
112-
"s3.access_key" = "your-access-key",
113-
"s3.secret_key" = "your-secret-key",
114-
"s3.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
115-
"s3.region" = "cn-hangzhou"
116+
"oss.hdfs.enabled" = "true",
117+
"oss.access_key" = "your-access-key",
118+
"oss.secret_key" = "your-secret-key",
119+
"oss.endpoint" = "cn-hangzhou.oss-dls.aliyuncs.com",
120+
"oss.region" = "cn-hangzhou"
116121
```

‎i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,6 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
188188
's3.access_key' = '<ak>',
189189
's3.secret_key' = '<sk>'
190190
);
191-
```
192191
使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.2+)
193192
```sql
194193
CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
@@ -490,6 +489,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
490489
```
491490
</TabItem>
492491
<TabItem value='OSS' label='OSS'>
492+
使用 OSS
493493
```sql
494494
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
495495
'type' = 'paimon',
@@ -501,6 +501,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
501501
'oss.secret_key'='<sk>'
502502
);
503503
```
504+
505+
使用 OSS-HDFS
506+
507+
```sql
508+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
509+
'type' = 'paimon',
510+
'paimon.catalog.type' = 'filesystem',
511+
'warehouse' = 'oss://bucket/regression/paimon1',
512+
'fs.oss-hdfs.support' = 'true',
513+
'oss.hdfs.access_key' = '<ak>',
514+
'oss.hdfs.secret_key' = '<sk>',
515+
'oss.hdfs.endpoint' = 'cn-beijing.oss-dls.aliyuncs.com',
516+
'oss.hdfs.region' = 'cn-beijing'
517+
);
518+
```
504519
</TabItem>
505520
<TabItem value='COS' label='COS'>
506521
```sql
@@ -584,6 +599,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
584599
```
585600
</TabItem>
586601
<TabItem value='OSS' label='OSS'>
602+
使用 OSS
603+
587604
```sql
588605
CREATE CATALOG paimon_base_filesystem_paimon_oss PROPERTIES (
589606
'type' = 'paimon',
@@ -595,6 +612,21 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
595612
'oss.secret_key'='<sk>'
596613
);
597614
```
615+
616+
使用 OSS-HDFS
617+
618+
```sql
619+
CREATE CATALOG paimon_base_filesystem_paimon_oss_hdfs PROPERTIES (
620+
'type' = 'paimon',
621+
'paimon.catalog.type' = 'filesystem',
622+
'warehouse' = 'oss://bucket/regression/paimon1',
623+
'oss.hdfs.enabled' = 'true',
624+
'oss.access_key' = 'your-access-key',
625+
'oss.secret_key' = 'your-secret-key',
626+
'oss.endpoint' = 'cn-hangzhou.oss-dls.aliyuncs.com',
627+
'oss.region' = 'cn-hangzhou'
628+
);
629+
```
598630
</TabItem>
599631
<TabItem value='COS' label='COS'>
600632
```sql

0 commit comments

Comments
 (0)