Skip to content

Commit 50abecf

Browse files
shuaiqidexiaosushuaiqidexiaosu
authored
Merge branch 'master' into patch-1
2 parents 852f7b2 + 96f1410 commit 50abecf

33 files changed

Lines changed: 478 additions & 39 deletions

File tree

‎config/custom-docusaurus-plugin.js‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
const path = require('path');
2+
3+
module.exports = function (context, options) {
4+
return {
5+
name: 'custom-docusaurus-plugin',
6+
configureWebpack(config, isServer, utils) {
7+
const localePublicPath =
8+
context.i18n.currentLocale === 'en'
9+
? '/'
10+
: context.i18n.currentLocale === 'zh-CN'
11+
? '/zh-CN/'
12+
: '/ja/';
13+
14+
return {
15+
output: {
16+
...config.output,
17+
publicPath:
18+
context.i18n.currentLocale === 'en'
19+
? 'https://cdnd.selectdb.com/' :
20+
context.i18n.currentLocale === 'zh-CN' ?
21+
'https://cdnd.selectdb.com/zh-CN/' : 'https://cdnd.selectdb.com/ja/',
22+
// Keep general assets on CDN, but force worker chunks to same-origin
23+
// to satisfy browser worker same-origin policy for Search worker bootstrap.
24+
workerPublicPath: localePublicPath,
25+
},
26+
};
27+
},
28+
};
29+
};

‎config/ssrTemplate.js‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
module.exports = {
2+
ssrTemplate: `<!DOCTYPE html>
3+
<html <%~ it.htmlAttributes %>>
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no">
7+
<meta name="generator" content="Docusaurus v<%= it.version %>">
8+
<% if (it.noIndex) { %>
9+
<meta name="robots" content="noindex, nofollow" />
10+
<% } %>
11+
<%~ it.headTags %>
12+
<% it.metaAttributes.forEach((metaAttribute) => { %>
13+
<%~ metaAttribute %>
14+
<% }); %>
15+
<% it.stylesheets.forEach((stylesheet) => { %>
16+
<link rel="stylesheet" href="<%= 'https://cdnd.selectdb.com' %><%= it.baseUrl %><%= stylesheet %>" />
17+
<% }); %>
18+
<% it.scripts.forEach((script) => { %>
19+
<link rel="preload" href="<%= 'https://cdnd.selectdb.com' %><%= it.baseUrl %><%= script %>" as="script">
20+
<% }); %>
21+
</head>
22+
<body <%~ it.bodyAttributes %>>
23+
<%~ it.preBodyTags %>
24+
<div id="__docusaurus">
25+
<%~ it.appHtml %>
26+
</div>
27+
<% it.scripts.forEach((script) => { %>
28+
<script src="<%= 'https://cdnd.selectdb.com' %><%= it.baseUrl %><%= script %>"></script>
29+
<% }); %>
30+
<%~ it.postBodyTags %>
31+
</body>
32+
</html>`,
33+
};

‎docs/data-operate/import/handling-messy-data.md‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ The system employs different strategies based on the strict mode setting:
3535
| ----------------- | -------------------- | ----------------------------- | ----------- | ------ |
3636
| NULL | \N | NULL | ON/OFF | NULL |
3737
| Non-NULL | "abc" or 2000 | NULL | ON | Invalid (Filtered) |
38-
| Non-NULL | "abc" | NULL | OFF | NULL |
38+
| Non-NULL | "abc" or 2000 | NULL | OFF | NULL |
3939
| Non-NULL | 1 | 1 | ON/OFF | Loaded Successfully |
4040

4141
:::tip
@@ -51,14 +51,28 @@ The system employs different strategies based on the strict mode setting:
5151
| NULL | \N | NULL | ON/OFF | NULL |
5252
| Non-NULL | aaa | NULL | ON | Invalid (Filtered) |
5353
| Non-NULL | aaa | NULL | OFF | NULL |
54-
| Non-NULL | 1 or 10 | 1 or 10 | ON/OFF | Loaded Successfully |
54+
| Non-NULL | 10 | NULL (overflow) | ON | Filtered |
55+
| Non-NULL | 10 | NULL (overflow) | OFF | NULL |
5556

5657
:::tip
5758
1. The column in the table allows NULL values
5859

5960
2. `abc` becomes NULL after conversion to Decimal due to type issues. When strict mode is ON, such data will be filtered out. When OFF, NULL will be loaded.
6061

61-
3. Although `10` exceeds the range, since its type meets decimal requirements, strict mode does not affect it.
62+
3. `10` is a value outside the range of Decimal(1, 0) and will be converted to NULL. It will be filtered out if strict mode is enabled. If strict mode is disabled, `null` will be imported.
63+
:::
64+
65+
**3. Example of a column type char(10)**
66+
67+
| Original data type | Original Data Example | Value after conversion to char(10) | Strict mode | Result |
68+
| ------------------ | --------------------- | ---------------------------------- | ---------- | ------ |
69+
| NULL | \N | NULL | ON/OFF | NULL |
70+
| Non-NULL | a1234567890 | a1234567890 | ON | Too long, filtered |
71+
| Non-NULL | a1234567890 | a1234567890 | OFF | a123456789 (truncated) |
72+
73+
:::tip
74+
75+
1. The column in the table allows NULL values
6276
:::
6377

6478
### Enable Strict Mode
@@ -183,5 +197,5 @@ SET insert_max_filter_ratio = 0.1;
183197
INSERT INTO test_table FROM S3/HDFS/LOCAL();```
184198
185199
:::tip
186-
For Insert Into statements, `insert_max_filter_ratio` only takes effect when `enable_insert_strict = false`, and only applies to `INSERT INTO FROM S3/HDFS/LOCAL()` syntax. The default value is 1.0, which means that all abnormal data are allowed to be filtered.
200+
For Insert Into statements, `insert_max_filter_ratio` only takes effect when `enable_insert_strict = false`. The default value is 1.0, which means that all abnormal data are allowed to be filtered.
187201
:::

‎docs/ecosystem/doris-kafka-connector/doris-kafka-connector.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ The Doris community provides the [doris-kafka-connector](https://github.com/apac
1818
| 1.1.0 | 2.4+ | 2.0+ | 8 |
1919
| 24.0.0 | 2.4+ | 2.0+ | 8 |
2020
| 25.0.0 | 2.4+ | 2.0+ | 8 |
21+
| 26.0.0 | 2.4+ | 2.0+ | 8 |
2122

2223
## Usage
2324

‎docs/ecosystem/doris-kafka-connector/release-notes.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,24 @@
77

88
# Doris Kafka Connector Release Notes
99

10+
## 26.0.0
11+
12+
### Features & Improvements
13+
14+
- Enable gzip compression by default for StreamLoad [#95](https://github.com/apache/doris-kafka-connector/pull/95)
15+
- Fix JDBC connection surge caused by improper connection reuse in schema evolution [#94](https://github.com/apache/doris-kafka-connector/pull/94)
16+
- Add Confluent build scripts [#86](https://github.com/apache/doris-kafka-connector/pull/86)
17+
- Update config for ConfigDef [#85](https://github.com/apache/doris-kafka-connector/pull/85)
18+
19+
### Bug Fixes
20+
21+
- Fix the check of 'topic2table' configuration [#83](https://github.com/apache/doris-kafka-connector/pull/83)
22+
- Fix the link to the website doc in the readme [#84](https://github.com/apache/doris-kafka-connector/pull/84)
23+
24+
### Credits
25+
26+
@JNSimba @rnb-tron
27+
1028
## 25.0.0
1129

1230
### Features & Improvements

‎docs/ecosystem/flink-doris-connector/flink-doris-connector.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Using the Flink Connector, you can perform the following operations:
3434
| 25.0.0 | 1.15 - 1.20 | 1.0+ | 8 | - |
3535
| 25.1.0 | 1.15 - 1.20 | 1.0+ | 8 | - |
3636
| 26.0.0 | 1.15 - 1.20,2.0 - 2.2 | 1.0+ | 8(1.x),17(2.x) | - |
37+
| 26.1.0 | 1.15 - 1.20,2.0 - 2.2 | 1.0+ | 8(1.x),17(2.x) | - |
3738

3839
## Usage
3940

@@ -832,9 +833,9 @@ After starting the Flink cluster, you can directly run the following command:
832833
| Key | Default Value | Required | Comment |
833834
| --------------------------- | ------------- | -------- | ------------------------------------------------------------ |
834835
| sink.label-prefix | -- | Y | The label prefix used for Stream load import. In the 2pc scenario, it is required to be globally unique to ensure the EOS semantics of Flink. |
835-
| sink.properties.* | -- | N | Import parameters for Stream Load. For example, 'sink.properties.column_separator' = ', ' defines the column separator, and 'sink.properties.escape_delimiters' = 'true' means that special characters as delimiters, like \x01, will be converted to binary 0x01. For JSON format import, 'sink.properties.format' = 'json', 'sink.properties.read_json_by_line' = 'true'. For detailed parameters, refer to [here](../data-operate/import/import-way/stream-load-manual.md#load-configuration-parameters). For Group Commit mode, for example, 'sink.properties.group_commit' = 'sync_mode' sets the group commit to synchronous mode. The Flink connector has supported import configuration group commit since version 1.6.2. For detailed usage and limitations, refer to [group commit](../data-operate/import/group-commit-manual.md). |
836+
| sink.properties.* | -- | N | Import parameters for Stream Load. For example, 'sink.properties.column_separator' = ', ' defines the column separator, and 'sink.properties.escape_delimiters' = 'true' means that special characters as delimiters, like \x01, will be converted to binary 0x01. For JSON format import, 'sink.properties.format' = 'json', 'sink.properties.read_json_by_line' = 'true'. For detailed parameters, refer to [here](../../data-operate/import/import-way/stream-load-manual.md#load-configuration-parameters). For Group Commit mode, for example, 'sink.properties.group_commit' = 'sync_mode' sets the group commit to synchronous mode. The Flink connector has supported import configuration group commit since version 1.6.2. For detailed usage and limitations, refer to [group commit](../../data-operate/import/group-commit-manual.md). |
836837
| sink.enable-delete | TRUE | N | Whether to enable deletion. This option requires the Doris table to have the batch deletion feature enabled (enabled by default in Doris 0.15+ versions), and only supports the Unique model. |
837-
| sink.enable-2pc | TRUE | N | Whether to enable two-phase commit (2pc). The default is true, ensuring Exactly-Once semantics. For details about two-phase commit, refer to [here](../data-operate/transaction.md#streamload-2pc). |
838+
| sink.enable-2pc | TRUE | N | Whether to enable two-phase commit (2pc). The default is true, ensuring Exactly-Once semantics. For details about two-phase commit, refer to [here](../../data-operate/transaction.md#streamload-2pc). |
838839
| sink.buffer-size | 1MB | N | The size of the write data cache buffer, in bytes. It is not recommended to modify it, and the default configuration can be used. |
839840
| sink.buffer-count | 3 | N | The number of write data cache buffers. It is not recommended to modify it, and the default configuration can be used. |
840841
| sink.max-retries | 3 | N | The maximum number of retries after a Commit failure. The default is 3 times. |
@@ -1129,7 +1130,7 @@ In the whole database synchronization tool provided by the Connector, no additio
11291130
11301131
3. **errCode = 2, detailMessage = current running txns on db 10006 is 100, larger than limit 100**
11311132
1132-
This is because the concurrent imports into the same database exceed 100. It can be solved by adjusting the parameter `max_running_txn_num_per_db` in `fe.conf`. For specific details, please refer to [max_running_txn_num_per_db](../admin-manual/config/fe-config#max_running_txn_num_per_db).
1133+
This is because the concurrent imports into the same database exceed 100. It can be solved by adjusting the parameter `max_running_txn_num_per_db` in `fe.conf`. For specific details, please refer to [max_running_txn_num_per_db](../../admin-manual/config/fe-config#max_running_txn_num_per_db).
11331134
11341135
Meanwhile, frequently modifying the label and restarting a task may also lead to this error. In the 2pc scenario (for Duplicate/Aggregate models), the label of each task needs to be unique. And when restarting from a checkpoint, the Flink task will actively abort the transactions that have been pre-committed successfully but not yet committed. Frequent label modifications and restarts will result in a large number of pre-committed successful transactions that cannot be aborted and thus occupy transactions. In the Unique model, 2pc can also be disabled to achieve idempotent writes.
11351136

‎docs/ecosystem/flink-doris-connector/release-notes.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,16 @@
77

88
# Flink Doris Connector Release Notes
99

10+
## 26.1.0
11+
12+
### Features & Improvements
13+
14+
- Enable gz compression by default for StreamLoad [#648](https://github.com/apache/doris-flink-connector/pull/648)
15+
16+
### Credits
17+
18+
@JNSimba
19+
1020
## 26.0.0
1121

1222
### Features & Improvements

‎docs/ecosystem/spark-doris-connector/release-notes.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,22 @@
77

88
# Spark Doris Connector Release Notes
99

10+
## 26.0.0
11+
12+
### Improvements
13+
14+
- Enable gzip compression by default for StreamLoad writes [#356](https://github.com/apache/doris-spark-connector/pull/356)
15+
- Add log for Arrow Flight read [#354](https://github.com/apache/doris-spark-connector/pull/354)
16+
- Support http request use UTF-8 charset [#347](https://github.com/apache/doris-spark-connector/pull/347)
17+
18+
### Bug Fixes
19+
20+
- Fix column projection issue in Spark 3.3, 3.4, and 3.5 [#353](https://github.com/apache/doris-spark-connector/pull/353)
21+
22+
### Credits
23+
24+
@JNSimba @gnehil
25+
1026
## 25.2.0
1127

1228
### Improvements

‎docs/ecosystem/spark-doris-connector/spark-doris-connector.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Code repository: https://github.com/apache/doris-spark-connector
2020

2121
| Connector | Spark | Doris | Java | Scala |
2222
|-----------|---------------------|-------------|------|------------|
23+
| 26.0.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2324
| 25.2.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2425
| 25.1.0 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
2526
| 25.0.1 | 3.5 - 3.1, 2.4 | 1.0 + | 8 | 2.12, 2.11 |
@@ -416,8 +417,8 @@ Java version examples are provided under `samples/doris-demo/spark-demo/` for re
416417
| doris.sink.label.prefix | spark-doris | Import label prefix when writing in Stream Load mode. |
417418
| doris.thrift.max.message.size | 2147483647 | Maximum message size when reading data through Thrift. |
418419
| doris.fe.auto.fetch | false | Whether to automatically fetch FE information. When set to true, all FE node information will be requested based on the nodes configured in `doris.fenodes`, without needing to configure multiple nodes additionally or separately configure `doris.read.arrow-flight-sql.port` and `doris.query.port`. |
419-
| doris.read.bitmap-to-string | false | Whether to convert Bitmap type to a string composed of array indexes when reading. For specific result format, refer to function definition [BITMAP_TO_STRING](../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-string.md). |
420-
| doris.read.bitmap-to-base64 | false | Whether to convert Bitmap type to Base64 encoded string when reading. For specific result format, refer to function definition [BITMAP_TO_BASE64](../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-base64.md). |
420+
| doris.read.bitmap-to-string | false | Whether to convert Bitmap type to a string composed of array indexes when reading. For specific result format, refer to function definition [BITMAP_TO_STRING](../../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-string.md). |
421+
| doris.read.bitmap-to-base64 | false | Whether to convert Bitmap type to Base64 encoded string when reading. For specific result format, refer to function definition [BITMAP_TO_BASE64](../../sql-manual/sql-functions/scalar-functions/bitmap-functions/bitmap-to-base64.md). |
421422
| doris.query.port | - | Doris FE query port, used for overwrite write and Catalog metadata retrieval. |
422423

423424
### SQL and Dataframe Specific Configuration

‎docusaurus.config.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@ const versionsPlugin = require('./config/versions-plugin');
33
const VERSIONS = require('./versions.json');
44
const { markdownBoldPlugin } = require('./config/markdown-bold-plugin');
55
const { DEFAULT_VERSION } = require('./src/constant/version');
6+
const { ssrTemplate } = require('./config/ssrTemplate');
7+
const customDocusaurusPlugin = require('./config/custom-docusaurus-plugin');
8+
69

710
// Allow filtering doc versions via environment variable.
811
// Usage: DOCS_VERSIONS="current,4.x" yarn docusaurus build
@@ -139,6 +142,7 @@ const config = {
139142
sidebarPath: require.resolve('./sidebarsReleases.json'),
140143
}),
141144
],
145+
process.env.NODE_ENV === 'development' ? null : customDocusaurusPlugin,
142146
async function tailwindcssPlugin(context, options) {
143147
return {
144148
name: 'docusaurus-tailwindcss',
@@ -671,6 +675,7 @@ const config = {
671675
// },
672676
// ],s
673677
}),
678+
ssrTemplate
674679
};
675680

676681
module.exports = config;

0 commit comments

Comments
 (0)