Skip to content

Commit 069d354

Browse files
sollhuisollhui
authored
[doc](load) add field examples and troubleshooting query for routine_load_job (#3635)
1 parent e59ff46 commit 069d354

11 files changed

Lines changed: 591 additions & 272 deletions

File tree

‎docs-next/admin-manual/system-tables/information_schema/routine_load_job.md‎

Lines changed: 53 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,31 +10,60 @@
1010

1111
Used to view information about routine load jobs.
1212

13-
## Database
13+
## Database and Table
1414

15-
`information_schema`
15+
`information_schema.routine_load_jobs`
1616

1717
## Table Information
1818

19-
| Column Name | Type | Description |
20-
| :-------------------- | :-------- | :----------------------------------------- |
21-
| JOB_ID | text | Job ID |
22-
| JOB_NAME | text | Job name |
23-
| CREATE_TIME | text | Job creation time |
24-
| PAUSE_TIME | text | Job pause time |
25-
| END_TIME | text | Job end time |
26-
| DB_NAME | text | Database name |
27-
| TABLE_NAME | text | Table name |
28-
| STATE | text | Job status |
29-
| CURRENT_TASK_NUM | text | Current number of subtasks |
30-
| JOB_PROPERTIES | text | Job property configurations |
31-
| DATA_SOURCE_PROPERTIES| text | Data source property configurations |
32-
| CUSTOM_PROPERTIES | text | Custom property configurations |
33-
| STATISTIC | text | Job statistics information |
34-
| PROGRESS | text | Job progress information |
35-
| LAG | text | Job delay information |
36-
| REASON_OF_STATE_CHANGED| text | Reason for job status change |
37-
| ERROR_LOG_URLS | text | Error log URLs |
38-
| USER_NAME | text | Username |
39-
| CURRENT_ABORT_TASK_NUM| int | Current number of failed tasks |
40-
| IS_ABNORMAL_PAUSE | boolean | Whether paused by system (non-user pause) |
19+
| Column Name | Type | Description | Example |
20+
| :----------------------- | :------ | :------------------------------------------------------------------------------------------------------------------------------------------------------ | :------ |
21+
| JOB_ID | text | Job ID generated by Doris. | `12025` |
22+
| JOB_NAME | text | Routine Load job name. | `example_routine_load` |
23+
| CREATE_TIME | text | Job creation time. | `2024-01-15 08:12:42` |
24+
| PAUSE_TIME | text | Most recent job pause time. It is `NULL` if the job has not been paused. | `NULL` |
25+
| END_TIME | text | Job end time. It is `NULL` if the job has not ended. | `NULL` |
26+
| DB_NAME | text | Database name of the job. | `default_cluster:testdb` |
27+
| TABLE_NAME | text | Target table name of the job. For multi-table import jobs, this value is `multi-table`. | `test_routineload_tbl` |
28+
| STATE | text | Job running status, including `NEED_SCHEDULE`, `RUNNING`, `PAUSED`, `STOPPED`, and `CANCELLED`. | `RUNNING` |
29+
| CURRENT_TASK_NUM | text | Number of subtasks currently being scheduled or executed. | `1` |
30+
| JOB_PROPERTIES | text | Job property configurations, including batch size, concurrency, import format, column mapping, and error tolerance. | `{"max_batch_rows":"200000","format":"csv","columnToColumnExpr":"user_id,name,age","max_filter_ratio":"1.0"}` |
31+
| DATA_SOURCE_PROPERTIES | text | Data source property configurations. For Kafka, this includes topic, broker list, and current Kafka partitions. | `{"topic":"test-topic","currentKafkaPartitions":"0","brokerList":"192.168.88.62:9092"}` |
32+
| CUSTOM_PROPERTIES | text | Custom properties configured when creating the job. For Kafka, this usually includes offsets, group id, and Kafka client parameters passed with `property.` prefixes. | `{"kafka_default_offsets":"OFFSET_BEGINNING","group.id":"example_routine_load_73daf600-884e-46c0-a02b-4e49fdf3b4dc"}` |
33+
| STATISTIC | text | Job runtime statistics. Common fields include `receivedBytes`, `loadedRows`, `errorRows`, `committedTaskNum`, `abortedTaskNum`, `loadRowsRate`, and `taskExecuteTimeMs`. | `{"receivedBytes":28,"runningTxns":[],"errorRows":0,"committedTaskNum":3,"loadedRows":3,"loadRowsRate":0,"abortedTaskNum":0,"errorRowsAfterResumed":0,"totalRows":3,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":30069}` |
34+
| PROGRESS | text | Job running progress. For Kafka, it shows the consumed offset of each partition. | `{"0":"2"}` |
35+
| LAG | text | Job lag information. For Kafka, it shows the consumption lag of each partition. | `{"0":0}` |
36+
| REASON_OF_STATE_CHANGED | text | Reason for the job state change. It is usually empty for normally running jobs, and records the specific reason when the job is paused or cancelled abnormally. | `The number of failed task exceeded max_error_number` |
37+
| ERROR_LOG_URLS | text | Error log URLs for viewing filtered data that failed quality checks. It is empty if there is no error log. | `http://fe_host:8030/api/_load_error_log?file=error.log` |
38+
| USER_NAME | text | User who created or operated the job. | `root` |
39+
| CURRENT_ABORT_TASK_NUM | int | Current number of failed subtasks. | `0` |
40+
| IS_ABNORMAL_PAUSE | boolean | Whether the job was paused abnormally by the system instead of manually by a user. `true` indicates an abnormal system pause, and `false` indicates no abnormal pause. | `false` |
41+
42+
## Query Abnormal Jobs
43+
44+
If a job is abnormally paused, has failed tasks, or is in the `RUNNING` state but has no running subtasks while Kafka still has consumption lag, it should be investigated. You can use the following SQL to query these jobs:
45+
46+
```sql
47+
SELECT DB_NAME, JOB_NAME
48+
FROM information_schema.routine_load_jobs
49+
WHERE IS_ABNORMAL_PAUSE = TRUE
50+
OR (
51+
STATE = 'RUNNING'
52+
AND (
53+
CURRENT_ABORT_TASK_NUM > 0
54+
OR (
55+
CAST(CURRENT_TASK_NUM AS INT) = 0
56+
AND `LAG` REGEXP ':[[:space:]]*[1-9][0-9]*'
57+
)
58+
)
59+
);
60+
```
61+
62+
After finding an abnormal job, switch to the corresponding database and use `SHOW ROUTINE LOAD` to view the job details:
63+
64+
```sql
65+
USE `db_name`;
66+
SHOW ROUTINE LOAD FOR `job_name`;
67+
```
68+
69+
`LAG` is the Kafka consumption lag information for each partition. `LAG REGEXP ':[[:space:]]*[1-9][0-9]*'` matches jobs where at least one partition has a lag greater than 0.

‎docs/admin-manual/system-tables/information_schema/routine_load_job.md‎

Lines changed: 53 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,31 +10,60 @@
1010

1111
Used to view information about routine load jobs.
1212

13-
## Database
13+
## Database and Table
1414

15-
`information_schema`
15+
`information_schema.routine_load_jobs`
1616

1717
## Table Information
1818

19-
| Column Name | Type | Description |
20-
| :-------------------- | :-------- | :----------------------------------------- |
21-
| JOB_ID | text | Job ID |
22-
| JOB_NAME | text | Job name |
23-
| CREATE_TIME | text | Job creation time |
24-
| PAUSE_TIME | text | Job pause time |
25-
| END_TIME | text | Job end time |
26-
| DB_NAME | text | Database name |
27-
| TABLE_NAME | text | Table name |
28-
| STATE | text | Job status |
29-
| CURRENT_TASK_NUM | text | Current number of subtasks |
30-
| JOB_PROPERTIES | text | Job property configurations |
31-
| DATA_SOURCE_PROPERTIES| text | Data source property configurations |
32-
| CUSTOM_PROPERTIES | text | Custom property configurations |
33-
| STATISTIC | text | Job statistics information |
34-
| PROGRESS | text | Job progress information |
35-
| LAG | text | Job delay information |
36-
| REASON_OF_STATE_CHANGED| text | Reason for job status change |
37-
| ERROR_LOG_URLS | text | Error log URLs |
38-
| USER_NAME | text | Username |
39-
| CURRENT_ABORT_TASK_NUM| int | Current number of failed tasks |
40-
| IS_ABNORMAL_PAUSE | boolean | Whether paused by system (non-user pause) |
19+
| Column Name | Type | Description | Example |
20+
| :----------------------- | :------ | :------------------------------------------------------------------------------------------------------------------------------------------------------ | :------ |
21+
| JOB_ID | text | Job ID generated by Doris. | `12025` |
22+
| JOB_NAME | text | Routine Load job name. | `example_routine_load` |
23+
| CREATE_TIME | text | Job creation time. | `2024-01-15 08:12:42` |
24+
| PAUSE_TIME | text | Most recent job pause time. It is `NULL` if the job has not been paused. | `NULL` |
25+
| END_TIME | text | Job end time. It is `NULL` if the job has not ended. | `NULL` |
26+
| DB_NAME | text | Database name of the job. | `default_cluster:testdb` |
27+
| TABLE_NAME | text | Target table name of the job. For multi-table import jobs, this value is `multi-table`. | `test_routineload_tbl` |
28+
| STATE | text | Job running status, including `NEED_SCHEDULE`, `RUNNING`, `PAUSED`, `STOPPED`, and `CANCELLED`. | `RUNNING` |
29+
| CURRENT_TASK_NUM | text | Number of subtasks currently being scheduled or executed. | `1` |
30+
| JOB_PROPERTIES | text | Job property configurations, including batch size, concurrency, import format, column mapping, and error tolerance. | `{"max_batch_rows":"200000","format":"csv","columnToColumnExpr":"user_id,name,age","max_filter_ratio":"1.0"}` |
31+
| DATA_SOURCE_PROPERTIES | text | Data source property configurations. For Kafka, this includes topic, broker list, and current Kafka partitions. | `{"topic":"test-topic","currentKafkaPartitions":"0","brokerList":"192.168.88.62:9092"}` |
32+
| CUSTOM_PROPERTIES | text | Custom properties configured when creating the job. For Kafka, this usually includes offsets, group id, and Kafka client parameters passed with `property.` prefixes. | `{"kafka_default_offsets":"OFFSET_BEGINNING","group.id":"example_routine_load_73daf600-884e-46c0-a02b-4e49fdf3b4dc"}` |
33+
| STATISTIC | text | Job runtime statistics. Common fields include `receivedBytes`, `loadedRows`, `errorRows`, `committedTaskNum`, `abortedTaskNum`, `loadRowsRate`, and `taskExecuteTimeMs`. | `{"receivedBytes":28,"runningTxns":[],"errorRows":0,"committedTaskNum":3,"loadedRows":3,"loadRowsRate":0,"abortedTaskNum":0,"errorRowsAfterResumed":0,"totalRows":3,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":30069}` |
34+
| PROGRESS | text | Job running progress. For Kafka, it shows the consumed offset of each partition. | `{"0":"2"}` |
35+
| LAG | text | Job lag information. For Kafka, it shows the consumption lag of each partition. | `{"0":0}` |
36+
| REASON_OF_STATE_CHANGED | text | Reason for the job state change. It is usually empty for normally running jobs, and records the specific reason when the job is paused or cancelled abnormally. | `The number of failed task exceeded max_error_number` |
37+
| ERROR_LOG_URLS | text | Error log URLs for viewing filtered data that failed quality checks. It is empty if there is no error log. | `http://fe_host:8030/api/_load_error_log?file=error.log` |
38+
| USER_NAME | text | User who created or operated the job. | `root` |
39+
| CURRENT_ABORT_TASK_NUM | int | Current number of failed subtasks. | `0` |
40+
| IS_ABNORMAL_PAUSE | boolean | Whether the job was paused abnormally by the system instead of manually by a user. `true` indicates an abnormal system pause, and `false` indicates no abnormal pause. | `false` |
41+
42+
## Query Abnormal Jobs
43+
44+
If a job is abnormally paused, has failed tasks, or is in the `RUNNING` state but has no running subtasks while Kafka still has consumption lag, it should be investigated. You can use the following SQL to query these jobs:
45+
46+
```sql
47+
SELECT DB_NAME, JOB_NAME
48+
FROM information_schema.routine_load_jobs
49+
WHERE IS_ABNORMAL_PAUSE = TRUE
50+
OR (
51+
STATE = 'RUNNING'
52+
AND (
53+
CURRENT_ABORT_TASK_NUM > 0
54+
OR (
55+
CAST(CURRENT_TASK_NUM AS INT) = 0
56+
AND `LAG` REGEXP ':[[:space:]]*[1-9][0-9]*'
57+
)
58+
)
59+
);
60+
```
61+
62+
After finding an abnormal job, switch to the corresponding database and use `SHOW ROUTINE LOAD` to view the job details:
63+
64+
```sql
65+
USE `db_name`;
66+
SHOW ROUTINE LOAD FOR `job_name`;
67+
```
68+
69+
`LAG` is the Kafka consumption lag information for each partition. `LAG REGEXP ':[[:space:]]*[1-9][0-9]*'` matches jobs where at least one partition has a lag greater than 0.

0 commit comments

Comments
 (0)