Skip to content

Commit dc0d578

Browse files
leto-bbqleto-bbq
committed
add new params and note in scp datasync and tsfilebackup
1 parent 1f5fa77 commit dc0d578

8 files changed

Lines changed: 162 additions & 114 deletions

File tree

‎src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md‎

Lines changed: 25 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -204,25 +204,27 @@ Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This scrip
204204

205205

206206
### 3.2 Script Parameters
207-
| Abbreviation | Full Name | Description | Required | Default |
208-
| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- |
209-
| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - |
210-
| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` |
211-
| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` |
212-
| `-u` | `--user` | Username for IoTDB authentication. | No | `root` |
213-
| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` |
214-
| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - |
215-
| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` |
216-
| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` |
217-
| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - |
218-
| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - |
219-
| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - |
220-
| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - |
221-
| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - |
222-
| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` |
223-
| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - |
224-
| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - |
225-
| `-help` | `--help` | Show help information. | No | - |
207+
| Abbreviation | Full Name | Description | Required | Default |
208+
|-------------------------|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -------- | --------------- |
209+
| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - |
210+
| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` |
211+
| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` |
212+
| `-u` | `--user` | Username for IoTDB authentication. | No | `root` |
213+
| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` |
214+
| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - |
215+
| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` |
216+
| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` |
217+
| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - |
218+
| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - |
219+
| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - |
220+
| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - |
221+
| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - |
222+
| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` |
223+
| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - |
224+
| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - |
225+
| `--object-parallelism` | `--object-parallelism` | Specifies the maximum parallelism for object file transmission. | No | - |
226+
| `--object-batch-size` | `--object-batch-size` | Limits the total byte size of each object file upload batch, used to control memory usage and single SCP transfer size. | No | - |
227+
| `-help` | `--help` | Show help information. | No | - |
226228

227229

228230
### 3.3 Execution Examples
@@ -244,3 +246,7 @@ Example 3: Specify Pipe Plugin JAR Directory
244246
```Bash
245247
./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar
246248
```
249+
250+
**Note**: When exporting Object-type data in SCP mode, to avoid handshake exceptions, connection failures, or frequent Pipe restarts, it is recommended to take any of the following measures:
251+
* Appropriately lower the configuration parameter `object-parallelism`
252+
* Increase the `MaxStartups` value on the target machine as needed. After modification, execute `sshd reload` or `sshd restart` for the configuration to take effect.

‎src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -608,6 +608,10 @@ WITH SINK (
608608
);
609609
```
610610

611+
**Note**: When exporting Object-type data in SCP mode, to avoid handshake exceptions, connection failures, or frequent Pipe restarts, it is recommended to take any of the following measures:
612+
* Appropriately lower the configuration parameter `sink.scp.object-parallelism`
613+
* Increase the `MaxStartups` value on the target machine as needed. After modification, execute `sshd reload` or `sshd restart` for the configuration to take effect.
614+
611615
**Sink Exported TSFile and Object Format:**
612616

613617
```Bash
@@ -809,13 +813,15 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
809813
| sink.rate-limit-bytes-per-second | Rate limit threshold (unit: bytes/second). Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 |
810814

811815
#### tsfile-remote-sink
812-
| Parameter | Description | Value Range | Required | Default |
813-
|------------------------------------|-----------------------------------------------------------------------------|-------------------------|----------|---------|
814-
| sink | Component name | String: tsfile-remote-sink | Yes | - |
815-
| sink.scp.host | Remote host IP | String | Yes | - |
816-
| sink.scp.port | Remote SSH port | Long | No | 22 |
817-
| sink.scp.user | Remote SSH user | String | Yes | - |
818-
| sink.scp.password | Remote SSH password | String | Yes | - |
819-
| sink.scp.remote-path | Remote target directory | String | Yes | - |
820-
| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 |
816+
| Parameter | Description | Value Range | Required | Default |
817+
|------------------------------------|----------------------------------------------------------------------------|-------------------------|----------|---------|
818+
| sink | Component name | String: tsfile-remote-sink | Yes | - |
819+
| sink.scp.host | Remote host IP | String | Yes | - |
820+
| sink.scp.port | Remote SSH port | Long | No | 22 |
821+
| sink.scp.user | Remote SSH user | String | Yes | - |
822+
| sink.scp.password | Remote SSH password | String | Yes | - |
823+
| sink.scp.remote-path | Remote target directory | String | Yes | - |
824+
| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 |
825+
| sink.scp.object-parallelism | Maximum parallelism for object file transmission | Long | No |` min（cpu/4，16）` |
826+
| sink.scp.object-batch-size-bytes | Maximum size of Object files sent per asynchronous thread, unit: MB | Long | No | 200 |
821827

0 commit comments

Comments
 (0)