You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Verify installation by checking for the dscli command-line tool
59
-
dscli -h
56
+
dscli -h # Check datasystem CLI tool
60
57
```
61
58
62
-
63
-
#### 3. (Required for NPU Transfer) Install CANN and torch-npu
59
+
#### 2. (Optional for NPU Transfer) Install CANN and torch-npu
64
60
65
61
If you have NPU devices and want to accelerate the transmission of NPU tensor, you need to install **Ascend-cann-toolkit** and **torch-npu**.
66
62
67
63
Then check whether CANN is already installed:
68
64
69
65
```bash
70
66
# For root users
71
-
ll /usr/local/Ascend/ascend-toolkit/latest
67
+
ls /usr/local/Ascend/ascend-toolkit/latest
72
68
73
69
# For non-root users
74
-
ll${HOME}/Ascend/ascend-toolkit/latest
70
+
ls${HOME}/Ascend/ascend-toolkit/latest
75
71
```
76
72
77
73
If not installed, and you do need to install it, please skip to [Appendix A](#a-install-cann-for-npu-acceleration).
@@ -86,7 +82,7 @@ pip install torch-npu==2.8.0
86
82
87
83
After installation, you can run TransferQueue with Yuanrong backend.
88
84
89
-
First, start a local Ray cluster. Yuanrong backend relies on Ray for distributed management:
85
+
First, start a local Ray cluster. TransferQueue relies on Ray for distributed management:
90
86
```bash
91
87
ray start --head
92
88
```
@@ -120,12 +116,13 @@ tq.close()
120
116
121
117
## Deployment
122
118
119
+
Yuanrong datasystem is deployed **per-host** (one worker per node), managing all TransferQueue clients on the same node. It is not a per-client deployment.
120
+
123
121
When `auto_init: True` is set in the configuration, TransferQueue automatically initializes the Yuanrong backend during `tq.init()`. The deployment process:
124
122
125
123
1.**Detects Ray cluster nodes** - identifies all alive nodes in the Ray cluster
126
-
2.**Creates placement group** - uses `STRICT_SPREAD` strategy to ensure workers are distributed across nodes
127
-
3.**Launches YuanrongWorkerActor** - creates one actor per node to manage the datasystem worker
128
-
4.**Sets up metastore service** - the head node (driver node) starts the metastore service, other nodes connect as workers
124
+
2.**Launches YuanrongWorkerActor** - creates one actor per node to manage the datasystem worker
125
+
3.**Sets up metastore service** - the head node (driver node) starts the metastore service, other nodes connect as workers
129
126
130
127
### Configuration
131
128
@@ -146,14 +143,14 @@ backend:
146
143
- `metastore_port`: Port for metastore service on the head node.
147
144
- `worker_args`: Additional arguments passed to `dscli start` command:
148
145
- `--shared_memory_size_mb`: Shared memory size in MB for datasystem worker.
149
-
- `--enable_huge_tlb`: Configure huge page memory to reduce TLB misses and improve memory access efficiency. Note: may cause system memory shortage, kernel OOM, or system instability. Required for >21GB shared memory on Ascend 910B.
146
+
- `--enable_huge_tlb`: Configure huge page memory to reduce TLB misses and improve memory access efficiency. Note: may cause system memory shortage, kernel OOM, or system instability. **Please allocate huge pages before starting datasystem** - refer to [Huge Page Guide](https://pages.openeuler.openatom.cn/openyuanrong-datasystem/docs/zh-cn/latest/appendix/hugepage_guide.html).
150
147
151
148
**NPU Transfer Options:**
152
149
- `enable_yr_npu_transport`: Enable NPU transport for high-performance device-to-device data transfer. Set to `true` when using NPU tensors.
153
-
- `worker_args` (recommended when `enable_yr_npu_transport: true`):
154
-
- `--remote_h2d_device_ids`: Enable RH2D (Remote Host-to-Device) for efficient cross-node NPU data transfer. Specify NPU device IDs as comma-separated values (e.g., `0,1,2,3`).
150
+
- `worker_args` (**mandatory** when `enable_yr_npu_transport: true`):
151
+
- `--remote_h2d_device_ids`: Enable RH2D (Remote Host-to-Device) for efficient cross-node NPU data transfer. Specify NPU device IDs as comma-separated values (e.g., `0,1,2,3`). Yuanrong manages all specified devices - to put/get tensors on NPU `X`, device ID `X` must be included in this argument.
155
152
156
-
> More configuration parameters for deploying the data system can refer to [dscli config](https://gitcode.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/dscli.md).
153
+
> More configuration parameters for deploying the datasystem can refer to [dscli config](https://gitcode.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/dscli.md).
157
154
158
155
### Multi-Node Deployment
159
156
@@ -162,13 +159,15 @@ TransferQueue automatically deploys Yuanrong datasystem workers across all Ray c
162
159
#### Deploy Ray Cluster
163
160
164
161
```bash
165
-
# On head node
162
+
# On head node (assume IP of head_node is 192.168.0.1)
166
163
ray start --head --resources='{"node:192.168.0.1": 1}'
167
164
168
-
# On worker node (assume ray port of head_node is 6379)
165
+
# On worker node (assume IP of worker_node is 192.168.0.2)
169
166
ray start --address="192.168.0.1:6379" --resources='{"node:192.168.0.2": 1}'
170
167
```
171
168
169
+
The `--resources` parameter defines node-specific resources. It can be used to control Ray actor placement across nodes. For NPU environments, you may also add `--resources='{"NPU": 4}'` or configure `ASCEND_RT_VISIBLE_DEVICES`.
170
+
172
171
#### Multi-Node Configuration
173
172
174
173
```yaml
@@ -186,6 +185,8 @@ TransferQueue will detect all Ray nodes and deploy datasystem workers automatica
186
185
187
186
#### Multi-Node Demo
188
187
188
+
> **Note**: Before running the demo below, modify `HEAD_NODE_IP` and `WORKER_NODE_IP` to match your actual node IPs.
189
+
189
190
```python
190
191
import torch
191
192
import ray
@@ -311,60 +312,91 @@ Note: In manual startup mode, you need to manage the lifecycle of Yuanrong worke
311
312
312
313
## FAQ
313
314
314
-
### Port Conflict
315
+
### Failed to Start Datasystem Worker
315
316
316
-
If `worker_port` or `metastore_port` is already in use, initialization will fail:
317
+
If initialization fails with `RuntimeError: Failed to start datasystem worker...`, check the following possible causes:
317
318
318
-
```
319
-
RuntimeError: Failed to start datasystem worker...
320
-
```
319
+
**1. Port Conflict**
321
320
322
-
Check port usage:
321
+
Check if `worker_port` or `metastore_port` is already in use:
323
322
```bash
324
323
netstat -tlnp | grep 31501
325
324
netstat -tlnp | grep 2379
326
325
```
327
-
328
326
Solution: Change the port or clean up the occupying process.
329
327
330
-
> If a TransferQueue task terminates abnormally without calling `tq.close()`, the datasystem will become a defunct process and occupy the port.
328
+
> If a TransferQueue task terminates abnormally without calling `tq.close()`, the datasystem may become a defunct process and occupy the port.
329
+
330
+
**2. Shared Memory Allocation Failure**
331
+
332
+
If you encounter an error like:
333
+
```
334
+
Runtime error: failed to mmap shared memory: Cannot allocate memory
335
+
```
336
+
Check the following:
337
+
- Docker container shared memory limit (default is 64MB, may need increase)
338
+
- System available memory for shared memory allocation
339
+
- Huge page configuration if `--enable_huge_tlb true` is enabled
HTTP/HTTPS proxy settings may interfere with Yuanrong's internal communication, causing metastore connection timeout errors.
346
+
347
+
Yuanrong datasystem uses IP addresses directly for internal node communication. If proxy environment variables (`http_proxy`, `https_proxy`, `HTTP_PROXY`, `HTTPS_PROXY`) are set, they may route internal traffic through the proxy instead of direct connections.
If the previous run did not close properly, datasystem worker processes may remain:
358
+
If the previous run did not close properly (e.g., task crashed without `tq.close()`), datasystem worker processes may remain:
335
359
336
360
```bash
337
361
# Check residual processes
338
-
ps aux | grep dscli
362
+
ps aux | grep datasystem_worker
339
363
340
-
# Clean up
341
-
dscli stop --worker_address <IP>:31501
342
-
# Or force cleanup
343
-
pkill -f dscli
364
+
# Clean up gracefully
365
+
dscli stop --worker_address <IP>:<PORT>
366
+
367
+
# Force cleanup (use with caution)
368
+
pkill -f datasystem_worker
344
369
```
345
370
346
371
### Multi-Process Initialization
347
372
348
-
Each process must call `tq.init()` to obtain a TransferQueue client before using `tq.get_client()`:
349
-
- The first process initializes the TransferQueueController and Yuanrong backend
350
-
- Other processes automatically connect to the existing TransferQueueController
373
+
In multi-process scenarios, each process must call `tq.init()` before using TransferQueue APIs:
374
+
- The first process initializes the `TransferQueueController` and Yuanrong backend
375
+
- Subsequent processes automatically connect to the existing controller
351
376
352
-
Recommendation: Let the first process (which initialized the backend) call `tq.close()` to cleanup Yuanrong workers. Other processes only need to close their clients.
377
+
Best practice: Let the process that initialized the backend (typically the main/driver process) call `tq.close()` for cleanup. Other processes can simply close their clients without affecting the shared backend.
353
378
354
379
355
380
### NPU Transfer Issues
356
381
357
-
When enabling `enable_yr_npu_transport: true`, ensure:
When using `enable_yr_npu_transport: true`, ensure:
383
+
- CANN toolkit is properly installed
384
+
- `torch-npu`version matches `torch` version
385
+
- `--remote_h2d_device_ids`includes all device IDs you intend to use
386
+
387
+
Common errors and solutions:
388
+
- `Device not found`: Check if device ID is included in `--remote_h2d_device_ids`
389
+
- `CANN error`: Verify CANN installation path and environment variables
361
390
362
391
### Out of Memory Error
363
-
If you encounter an OutOfMemoryError (OOM) thrown by DataSystems during operation, please increase the value of the configuration option `--shared_memory_size_mb`.
392
+
393
+
If Yuanrong throws an OOM error during operation:
364
394
```
365
395
RuntimeError: code: [Out of memory], msg: [Shared memory no space in arena: ...]
366
396
```
367
397
398
+
Solution: Increase `--shared_memory_size_mb` in `worker_args`, or reduce the data volume being cached.
399
+
368
400
369
401
## Datasystem Logs
370
402
@@ -384,13 +416,13 @@ We recommend developing inside a CANN container.
384
416
385
417
#### Option 1: Docker Image (Recommended)
386
418
387
-
First, select the appropriate [CANN image](https://hub.docker.com/r/ascendai/cann) aligned with your **CANN version**, **Ascend hardware**, **OS**, and **Python version**. For examples:
419
+
First, select the appropriate [CANN image](https://hub.docker.com/r/ascendai/cann) aligned with your **CANN version**, **Ascend hardware**, **OS**, and **Python version**. For example:
388
420
389
421
| CANN Version | Ascend Hardware | OS | Python Version | Image Name |
0 commit comments