Skip to content

Commit f3b41d5

Browse files
authored
Add object import and export 2092 (#1112)
* fix slow query of audit log version * support obejct import(import-data) and export(tsfile-backup and pipe sql) from 2092
1 parent 7ce3c3f commit f3b41d5

12 files changed

Lines changed: 1013 additions & 115 deletions

File tree

src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md

Lines changed: 81 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
# Data Export
22

33
## 1. Function Overview
4-
The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows:
4+
5+
IoTDB supports two methods for data export:
6+
7+
* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files.
8+
* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework.
59

610
<table style="text-align: left;">
711
<tbody>
@@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an
1923
<td>File containing custom SQL statements.</td>
2024
</tr>
2125
<tr>
22-
<td >TsFile</td>
26+
<td rowspan="2">TsFile</td>
2327
<td>Open-source time-series file format.</td>
2428
</tr>
29+
<tr>
30+
<td>tsfile-backup.sh/bat</td>
31+
<td>An open-source time-series data file format,and this script supports the Object data type.</td>
32+
</tr>
2533
</tbody>
2634
</table>
2735

28-
- Does not support the Object data type.
2936

30-
## 2. Detailed Features
37+
## 2. Data Export Tool
3138
### 2.1 Common Parameters
3239
| Short | Full Parameter | Description | Required | Default |
3340
|----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------|
@@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an
9198
> export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1"
9299
Parse error: Missing required option: db
93100
```
94-
## 2.3 SQL Format
101+
### 2.3 SQL Format
95102
#### 2.3.1 Command
96103
```Shell
97104
# Unix/OS X
@@ -168,3 +175,72 @@ Parse error: Missing required option: db
168175
> /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0
169176
Parse error: Missing required option: db
170177
```
178+
179+
180+
## 3. TsFileBackup Based on PIPE Framework
181+
Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format.
182+
183+
**Notes:**
184+
1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb-<version>-extension`.**
185+
2. **This script supports exporting Object-type data to TsFile files.**
186+
187+
188+
### 3.1 Execution Commands
189+
```Shell
190+
# Unix/OS X
191+
> tools/tsfile-backup.sh [-sql_dialect <sql_dialect>] [-h <host>] [-p <port>]
192+
[-u <username>] [-pw <password>] [-path <path>] [-db <db>] [-table
193+
<table>] [-s <start_time>] [-e <end_time>] [-t <target_directory>]
194+
[-th <target_host>] [-tu <target_host_user>] [-tp <target_host_port>]
195+
[--rate_limit] [--plugin_jar] [-help]
196+
197+
# Windows
198+
> tools\windows\tsfile-backup.bat [-sql_dialect <sql_dialect>] [-h <host>] [-p <port>]
199+
[-u <username>] [-pw <password>] [-path <path>] [-db <db>] [-table
200+
<table>] [-s <start_time>] [-e <end_time>] [-t <target_directory>]
201+
[-th <target_host>] [-tu <target_host_user>] [-tp <target_host_port>]
202+
[--rate_limit] [--plugin_jar] [-help]
203+
```
204+
205+
206+
### 3.2 Script Parameters
207+
| Abbreviation | Full Name | Description | Required | Default |
208+
| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- |
209+
| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - |
210+
| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` |
211+
| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` |
212+
| `-u` | `--user` | Username for IoTDB authentication. | No | `root` |
213+
| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` |
214+
| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - |
215+
| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` |
216+
| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` |
217+
| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - |
218+
| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - |
219+
| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - |
220+
| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - |
221+
| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - |
222+
| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` |
223+
| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - |
224+
| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - |
225+
| `-help` | `--help` | Show help information. | No | - |
226+
227+
228+
### 3.3 Execution Examples
229+
230+
Example 1: SCP Remote Export (Send Data to Another Server)
231+
232+
```Bash
233+
./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123!
234+
```
235+
236+
Example 2: Remote Object Data Export with Rate Limiting
237+
238+
```Bash
239+
./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880
240+
```
241+
242+
Example 3: Specify Pipe Plugin JAR Directory
243+
244+
```Bash
245+
./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar
246+
```

0 commit comments

Comments
 (0)