Skip to content

Commit bc2589c

Browse files
authored
docs: add sql block partition filter docs (#3617)
## Versions - [ ] dev - [ ] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) ## Languages - [ ] Chinese - [ ] English - [ ] Japanese candidate translation needed ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built - [ ] Updated required version and language counterparts, or explained why not - [ ] If only one language changed, confirmed whether source/translation counterparts need sync
1 parent ff58390 commit bc2589c

6 files changed

Lines changed: 108 additions & 24 deletions

File tree

i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/workload-management/sql-blocking.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ SQL Block Rule 用于在查询规划阶段阻止符合特定模式的语句执
3131
| `cardinality` | 允许扫描的最大行数 | 正整数 |
3232
| `partition_num` | 允许扫描的最大分区数 | 正整数 |
3333
| `tablet_num` | 允许扫描的最大分桶数 | 正整数 |
34+
| `require_partition_filter` | 查询分区内表和 Hive 表时是否必须包含有效分区过滤条件。该属性在 Doris 4.0 系列中从 4.0.6 起支持,在 Doris 4.1 系列中从 4.1.2 起支持。 | `"true"``"false"` |
3435
| `global` | 是否为全局规则 | `"true"`(全局生效)或 `"false"`(仅对绑定用户生效) |
3536
| `enable` | 是否启用规则 | `"true"``"false"` |
3637

@@ -143,6 +144,22 @@ PROPERTIES(
143144
- 如果想对一个用户添加多个规则,在规则列表中列举所有的规则名字,以英文逗号隔开。
144145
- 如果想移除一个用户的所有规则,将规则列表置为空字符串即可:`SET PROPERTY FOR 'root' 'SQL_block_rules' = '';`
145146

147+
#### 场景六:要求分区过滤条件
148+
149+
对于分区内表和 Hive 表,缺少分区列过滤条件可能导致全分区扫描。可以要求这些分区表查询必须包含有效的分区过滤条件:
150+
151+
```sql
152+
CREATE SQL_BLOCK_RULE rule_require_partition_filter
153+
PROPERTIES
154+
(
155+
"require_partition_filter" = "true",
156+
"global" = "true",
157+
"enable" = "true"
158+
);
159+
```
160+
161+
开启规则后,`SELECT * FROM partitioned_table` 这类未包含分区过滤条件的查询会被阻止。包含分区过滤条件的查询,例如 `SELECT * FROM partitioned_table WHERE dt = '2024-01-01'`,可以继续执行。
162+
146163
如果需要修改或者删除阻止规则,可以参考阻止规则的 SQL 手册。
147164

148165
### 注意事项

i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-statements/data-governance/CREATE-SQL_BLOCK_RULE.md

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ PROPERTIES (
5151
> - partition_num: 一个表将扫描的最大 partition 数量
5252
> - tablet_num: 一个表将扫描的最大 tablet 数量
5353
> - cardinality: 一个表将扫描的数据行数
54+
> - require_partition_filter:查询分区内表或 Hive 表时是否必须包含有效的分区过滤条件。设置为 `true` 后,如果查询支持的分区表但没有在分区列上添加过滤条件,则会被阻止执行。默认值为 `false`。该属性在 Doris 4.0 系列中从 4.0.6 起支持,在 Doris 4.1 系列中从 4.1.2 起支持。目前仅对内表和 Hive 表生效。
5455
>
5556
> **开关类**
5657
>
@@ -125,6 +126,30 @@ PROPERTIES (
125126
SET PROPERTY FOR 'jack' 'sql_block_rules' = 'test_rule4';
126127
```
127128

129+
5. 创建要求分区内表和 Hive 表查询必须包含分区过滤条件的规则
130+
131+
```sql
132+
CREATE SQL_BLOCK_RULE test_rule5
133+
PROPERTIES(
134+
"require_partition_filter"="true",
135+
"global"="true",
136+
"enable"="true"
137+
);
138+
```
139+
140+
开启规则后,未包含分区列过滤条件的受支持分区表查询会被阻止:
141+
142+
```sql
143+
SELECT * FROM partitioned_table;
144+
ERROR 1105 (HY000): errCode = 2, detailMessage = sql hits sql block rule: test_rule5, missing partition filter
145+
```
146+
147+
包含有效分区过滤条件的查询可以继续执行:
148+
149+
```sql
150+
SELECT * FROM partitioned_table WHERE dt = '2024-01-01';
151+
```
152+
128153
## 其它
129154

130155
常用正则表达式如下:
@@ -157,4 +182,4 @@ $ :匹配字符串的结尾。例如,xyz$ 匹配以 'xyz' 结尾的字符串
157182
\d:匹配任何数字字符,相当于 [0-9]。
158183
159184
\w:匹配任何单词字符,包括字母、数字和下划线,相当于 [a-zA-Z0-9_]。
160-
```
185+
```

i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-statements/data-governance/SHOW-SQL_BLOCK_RULE.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,12 @@ SHOW SQL_BLOCK_RULE;
3737
```
3838

3939
```text
40-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
41-
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
42-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
43-
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true |
44-
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
45-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
40+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
41+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable | RequirePartitionFilter |
42+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
43+
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true | false |
44+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true | false |
45+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
4646
```
4747

4848
2. 查看指定的 SQL 阻止规则
@@ -51,9 +51,9 @@ SHOW SQL_BLOCK_RULE FOR test_rule2;
5151
```
5252

5353
```text
54-
+------------+------+---------+-------------+------------+-------------+--------+--------+
55-
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
56-
+------------+------+---------+-------------+------------+-------------+--------+--------+
57-
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
58-
+------------+------+---------+-------------+------------+-------------+--------+--------+
54+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
55+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable | RequirePartitionFilter |
56+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
57+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true | false |
58+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
5959
```

versioned_docs/version-4.x/admin-manual/workload-management/sql-blocking.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ Each rule is defined by the following properties that control its behavior and s
3131
| `cardinality` | Maximum number of rows allowed to scan | Positive integer |
3232
| `partition_num` | Maximum number of partitions allowed to scan | Positive integer |
3333
| `tablet_num` | Maximum number of buckets allowed to scan | Positive integer |
34+
| `require_partition_filter` | Whether partitioned internal table and Hive table queries must include an effective partition filter. Supported in Doris 4.0.6 and later in the 4.0 series, and in Doris 4.1.2 and later in the 4.1 series. | `"true"` or `"false"` |
3435
| `global` | Whether the rule is global | `"true"` (applies to all users) or `"false"` (applies only to bound users) |
3536
| `enable` | Whether the rule is enabled | `"true"` or `"false"` |
3637

@@ -143,6 +144,22 @@ By default, block rules apply globally (`"global" = "true"`). To apply a rule on
143144
- To add multiple rules for a user, list all rule names in the rule list, separated by commas.
144145
- To remove all rules for a user, set the rule list to an empty string: `SET PROPERTY FOR 'root' 'SQL_block_rules' = '';`
145146

147+
#### Case 6: Requiring Partition Filters
148+
149+
For partitioned internal tables and Hive tables, missing partition-column filters can cause full partition scans. You can require queries on these partitioned tables to include an effective partition filter:
150+
151+
```sql
152+
CREATE SQL_BLOCK_RULE rule_require_partition_filter
153+
PROPERTIES
154+
(
155+
"require_partition_filter" = "true",
156+
"global" = "true",
157+
"enable" = "true"
158+
);
159+
```
160+
161+
After the rule is enabled, a query such as `SELECT * FROM partitioned_table` will be blocked. Queries with partition predicates, such as `SELECT * FROM partitioned_table WHERE dt = '2024-01-01'`, can continue to run.
162+
146163
To modify or delete block rules, refer to the SQL manual for block rules.
147164

148165
### Important Notes

versioned_docs/version-4.x/sql-manual/sql-statements/data-governance/CREATE-SQL_BLOCK_RULE.md

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ PROPERTIES (
4848
> - partition_num: The maximum number of partitions that a table will scan.
4949
> - tablet_num: The maximum number of tablets that a table will scan.
5050
> - cardinality: The number of rows of data that a table will scan.
51+
> - require_partition_filter: Whether a query on a partitioned internal table or Hive table must contain an effective partition filter. When set to `true`, queries that scan a supported partitioned table without filtering on a partition column will be blocked. The default is `false`. This property is supported in Doris 4.0.6 and later in the 4.0 series, and in Doris 4.1.2 and later in the 4.1 series. Currently, it only takes effect on internal tables and Hive tables.
5152
>
5253
> **Switch Category**
5354
>
@@ -129,6 +130,30 @@ The user executing this SQL command must have at least the following permissions
129130
SET PROPERTY FOR 'jack' 'sql_block_rules' = 'test_rule4';
130131
```
131132

133+
5. Create a rule that requires partitioned internal table and Hive table queries to include a partition filter
134+
135+
```sql
136+
CREATE SQL_BLOCK_RULE test_rule5
137+
PROPERTIES(
138+
"require_partition_filter"="true",
139+
"global"="true",
140+
"enable"="true"
141+
);
142+
```
143+
144+
After the rule is enabled, a query on a supported partitioned table without a partition-column filter will be blocked:
145+
146+
```sql
147+
SELECT * FROM partitioned_table;
148+
ERROR 1105 (HY000): errCode = 2, detailMessage = sql hits sql block rule: test_rule5, missing partition filter
149+
```
150+
151+
Queries that include an effective partition filter can continue to run:
152+
153+
```sql
154+
SELECT * FROM partitioned_table WHERE dt = '2024-01-01';
155+
```
156+
132157
## Others
133158

134159
Common regular expressions are as follows:
@@ -163,4 +188,4 @@ $ : Matches the end of a string. For example, xyz$ matches strings ending with '
163188
\d : Matches any digit character, equivalent to [0-9].
164189
165190
\w : Matches any word character, including letters, digits, and underscores, equivalent to [a-zA-Z0-9_].
166-
```
191+
```

versioned_docs/version-4.x/sql-manual/sql-statements/data-governance/SHOW-SQL_BLOCK_RULE.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -36,12 +36,12 @@ SHOW SQL_BLOCK_RULE;
3636
```
3737

3838
```text
39-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
40-
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
41-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
42-
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true |
43-
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
44-
+------------+----------------------------+---------+-------------+------------+-------------+--------+--------+
39+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
40+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable | RequirePartitionFilter |
41+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
42+
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true | false |
43+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true | false |
44+
+------------+------------------------------+---------+-------------+-----------+-------------+--------+--------+------------------------+
4545
```
4646

4747
2. Display a specific SQL blocking rule
@@ -50,9 +50,9 @@ SHOW SQL_BLOCK_RULE FOR test_rule2;
5050
```
5151

5252
```text
53-
+------------+------+---------+-------------+------------+-------------+--------+--------+
54-
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
55-
+------------+------+---------+-------------+------------+-------------+--------+--------+
56-
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
57-
+------------+------+---------+-------------+------------+-------------+--------+--------+
53+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
54+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable | RequirePartitionFilter |
55+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
56+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true | false |
57+
+------------+------+---------+-------------+-----------+-------------+--------+--------+------------------------+
5858
```

0 commit comments

Comments
 (0)