Skip to content

Commit 8222cc2

Browse files
committed
Fixed comments
Signed-off-by: dpj135 <958208521@qq.com>
1 parent 0c31647 commit 8222cc2

1 file changed

Lines changed: 87 additions & 55 deletions

File tree

docs/storage_backends/openyuanrong_datasystem.md

Lines changed: 87 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -26,52 +26,48 @@ When Yuanrong backend is selected, `YuanrongStorageManager` and `YuanrongStorage
2626
## Quick Start
2727

2828
### Prerequisites
29-
- **Python Version**: $ \geq 3.10~and \leq 3.11 $
29+
- **Python Version**: >= 3.10, <= 3.11
3030
- **Architecture**: aarch64 or x86_64
3131

3232
### Installation Steps
3333

3434
Follow these steps to build and install:
3535

36-
#### 1. Install Core Dependencies
36+
#### 1. Install TransferQueue with Yuanrong
37+
38+
Use the `[yuanrong]` extras to install PyTorch, TransferQueue, and openYuanrong-datasystem in one command:
3739

38-
Install PyTorch and TransferQueue
3940
```bash
40-
# Install Torch (matching the version specified for your hardware)
41+
# Install torch, recommended version: 2.8.0 or higher.
42+
# Version 2.8.0 is used as an example.
4143
pip install torch==2.8.0
4244

43-
# Install TransferQueue from pypi
44-
pip install TransferQueue
45-
# or install from source code
45+
# Install from PyPI
46+
pip install TransferQueue[yuanrong]
47+
48+
# Or install from source
4649
git clone https://github.com/Ascend/TransferQueue/
4750
cd TransferQueue
48-
pip install -r requirements.txt
49-
python -m build --wheel
50-
pip install dist/*.whl
51+
pip install -e ".[yuanrong]"
5152
```
5253

53-
#### 2. Install Datasystem
54+
Verify installation:
5455
```bash
55-
# Install the OpenYuanrong Datasystem package
56-
pip install openyuanrong-datasystem
57-
58-
# Verify installation by checking for the dscli command-line tool
59-
dscli -h
56+
dscli -h # Check datasystem CLI tool
6057
```
6158

62-
63-
#### 3. (Required for NPU Transfer) Install CANN and torch-npu
59+
#### 2. (Optional for NPU Transfer) Install CANN and torch-npu
6460

6561
If you have NPU devices and want to accelerate the transmission of NPU tensor, you need to install **Ascend-cann-toolkit** and **torch-npu**.
6662

6763
Then check whether CANN is already installed:
6864

6965
```bash
7066
# For root users
71-
ll /usr/local/Ascend/ascend-toolkit/latest
67+
ls /usr/local/Ascend/ascend-toolkit/latest
7268

7369
# For non-root users
74-
ll ${HOME}/Ascend/ascend-toolkit/latest
70+
ls ${HOME}/Ascend/ascend-toolkit/latest
7571
```
7672

7773
If not installed, and you do need to install it, please skip to [Appendix A](#a-install-cann-for-npu-acceleration).
@@ -86,7 +82,7 @@ pip install torch-npu==2.8.0
8682

8783
After installation, you can run TransferQueue with Yuanrong backend.
8884

89-
First, start a local Ray cluster. Yuanrong backend relies on Ray for distributed management:
85+
First, start a local Ray cluster. TransferQueue relies on Ray for distributed management:
9086
```bash
9187
ray start --head
9288
```
@@ -120,12 +116,13 @@ tq.close()
120116

121117
## Deployment
122118

119+
Yuanrong datasystem is deployed **per-host** (one worker per node), managing all TransferQueue clients on the same node. It is not a per-client deployment.
120+
123121
When `auto_init: True` is set in the configuration, TransferQueue automatically initializes the Yuanrong backend during `tq.init()`. The deployment process:
124122

125123
1. **Detects Ray cluster nodes** - identifies all alive nodes in the Ray cluster
126-
2. **Creates placement group** - uses `STRICT_SPREAD` strategy to ensure workers are distributed across nodes
127-
3. **Launches YuanrongWorkerActor** - creates one actor per node to manage the datasystem worker
128-
4. **Sets up metastore service** - the head node (driver node) starts the metastore service, other nodes connect as workers
124+
2. **Launches YuanrongWorkerActor** - creates one actor per node to manage the datasystem worker
125+
3. **Sets up metastore service** - the head node (driver node) starts the metastore service, other nodes connect as workers
129126

130127
### Configuration
131128

@@ -146,14 +143,14 @@ backend:
146143
- `metastore_port`: Port for metastore service on the head node.
147144
- `worker_args`: Additional arguments passed to `dscli start` command:
148145
- `--shared_memory_size_mb`: Shared memory size in MB for datasystem worker.
149-
- `--enable_huge_tlb`: Configure huge page memory to reduce TLB misses and improve memory access efficiency. Note: may cause system memory shortage, kernel OOM, or system instability. Required for >21GB shared memory on Ascend 910B.
146+
- `--enable_huge_tlb`: Configure huge page memory to reduce TLB misses and improve memory access efficiency. Note: may cause system memory shortage, kernel OOM, or system instability. **Please allocate huge pages before starting datasystem** - refer to [Huge Page Guide](https://pages.openeuler.openatom.cn/openyuanrong-datasystem/docs/zh-cn/latest/appendix/hugepage_guide.html).
150147

151148
**NPU Transfer Options:**
152149
- `enable_yr_npu_transport`: Enable NPU transport for high-performance device-to-device data transfer. Set to `true` when using NPU tensors.
153-
- `worker_args` (recommended when `enable_yr_npu_transport: true`):
154-
- `--remote_h2d_device_ids`: Enable RH2D (Remote Host-to-Device) for efficient cross-node NPU data transfer. Specify NPU device IDs as comma-separated values (e.g., `0,1,2,3`).
150+
- `worker_args` (**mandatory** when `enable_yr_npu_transport: true`):
151+
- `--remote_h2d_device_ids`: Enable RH2D (Remote Host-to-Device) for efficient cross-node NPU data transfer. Specify NPU device IDs as comma-separated values (e.g., `0,1,2,3`). Yuanrong manages all specified devices - to put/get tensors on NPU `X`, device ID `X` must be included in this argument.
155152

156-
> More configuration parameters for deploying the data system can refer to [dscli config](https://gitcode.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/dscli.md).
153+
> More configuration parameters for deploying the datasystem can refer to [dscli config](https://gitcode.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/dscli.md).
157154

158155
### Multi-Node Deployment
159156

@@ -162,13 +159,15 @@ TransferQueue automatically deploys Yuanrong datasystem workers across all Ray c
162159
#### Deploy Ray Cluster
163160

164161
```bash
165-
# On head node
162+
# On head node (assume IP of head_node is 192.168.0.1)
166163
ray start --head --resources='{"node:192.168.0.1": 1}'
167164
168-
# On worker node (assume ray port of head_node is 6379)
165+
# On worker node (assume IP of worker_node is 192.168.0.2)
169166
ray start --address="192.168.0.1:6379" --resources='{"node:192.168.0.2": 1}'
170167
```
171168

169+
The `--resources` parameter defines node-specific resources. It can be used to control Ray actor placement across nodes. For NPU environments, you may also add `--resources='{"NPU": 4}'` or configure `ASCEND_RT_VISIBLE_DEVICES`.
170+
172171
#### Multi-Node Configuration
173172

174173
```yaml
@@ -186,6 +185,8 @@ TransferQueue will detect all Ray nodes and deploy datasystem workers automatica
186185

187186
#### Multi-Node Demo
188187

188+
> **Note**: Before running the demo below, modify `HEAD_NODE_IP` and `WORKER_NODE_IP` to match your actual node IPs.
189+
189190
```python
190191
import torch
191192
import ray
@@ -311,60 +312,91 @@ Note: In manual startup mode, you need to manage the lifecycle of Yuanrong worke
311312

312313
## FAQ
313314

314-
### Port Conflict
315+
### Failed to Start Datasystem Worker
315316

316-
If `worker_port` or `metastore_port` is already in use, initialization will fail:
317+
If initialization fails with `RuntimeError: Failed to start datasystem worker...`, check the following possible causes:
317318

318-
```
319-
RuntimeError: Failed to start datasystem worker...
320-
```
319+
**1. Port Conflict**
321320

322-
Check port usage:
321+
Check if `worker_port` or `metastore_port` is already in use:
323322
```bash
324323
netstat -tlnp | grep 31501
325324
netstat -tlnp | grep 2379
326325
```
327-
328326
Solution: Change the port or clean up the occupying process.
329327

330-
> If a TransferQueue task terminates abnormally without calling `tq.close()`, the datasystem will become a defunct process and occupy the port.
328+
> If a TransferQueue task terminates abnormally without calling `tq.close()`, the datasystem may become a defunct process and occupy the port.
329+
330+
**2. Shared Memory Allocation Failure**
331+
332+
If you encounter an error like:
333+
```
334+
Runtime error: failed to mmap shared memory: Cannot allocate memory
335+
```
336+
Check the following:
337+
- Docker container shared memory limit (default is 64MB, may need increase)
338+
- System available memory for shared memory allocation
339+
- Huge page configuration if `--enable_huge_tlb true` is enabled
340+
341+
Solution: Increase container shared memory (`--shm-size` flag), or reduce `--shared_memory_size_mb` value.
342+
343+
**3. Proxy Configuration**
344+
345+
HTTP/HTTPS proxy settings may interfere with Yuanrong's internal communication, causing metastore connection timeout errors.
346+
347+
Yuanrong datasystem uses IP addresses directly for internal node communication. If proxy environment variables (`http_proxy`, `https_proxy`, `HTTP_PROXY`, `HTTPS_PROXY`) are set, they may route internal traffic through the proxy instead of direct connections.
348+
349+
Solution: unset proxy variables before running:
350+
```bash
351+
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
352+
```
353+
354+
331355

332356
### Residual Worker Process
333357

334-
If the previous run did not close properly, datasystem worker processes may remain:
358+
If the previous run did not close properly (e.g., task crashed without `tq.close()`), datasystem worker processes may remain:
335359

336360
```bash
337361
# Check residual processes
338-
ps aux | grep dscli
362+
ps aux | grep datasystem_worker
339363
340-
# Clean up
341-
dscli stop --worker_address <IP>:31501
342-
# Or force cleanup
343-
pkill -f dscli
364+
# Clean up gracefully
365+
dscli stop --worker_address <IP>:<PORT>
366+
367+
# Force cleanup (use with caution)
368+
pkill -f datasystem_worker
344369
```
345370

346371
### Multi-Process Initialization
347372

348-
Each process must call `tq.init()` to obtain a TransferQueue client before using `tq.get_client()`:
349-
- The first process initializes the TransferQueueController and Yuanrong backend
350-
- Other processes automatically connect to the existing TransferQueueController
373+
In multi-process scenarios, each process must call `tq.init()` before using TransferQueue APIs:
374+
- The first process initializes the `TransferQueueController` and Yuanrong backend
375+
- Subsequent processes automatically connect to the existing controller
351376

352-
Recommendation: Let the first process (which initialized the backend) call `tq.close()` to cleanup Yuanrong workers. Other processes only need to close their clients.
377+
Best practice: Let the process that initialized the backend (typically the main/driver process) call `tq.close()` for cleanup. Other processes can simply close their clients without affecting the shared backend.
353378

354379

355380
### NPU Transfer Issues
356381

357-
When enabling `enable_yr_npu_transport: true`, ensure:
358-
- CANN is properly installed
359-
- torch-npu version matches torch version
360-
- `--remote_h2d_device_ids` parameter correctly specifies NPU device IDs
382+
When using `enable_yr_npu_transport: true`, ensure:
383+
- CANN toolkit is properly installed
384+
- `torch-npu` version matches `torch` version
385+
- `--remote_h2d_device_ids` includes all device IDs you intend to use
386+
387+
Common errors and solutions:
388+
- `Device not found`: Check if device ID is included in `--remote_h2d_device_ids`
389+
- `CANN error`: Verify CANN installation path and environment variables
361390

362391
### Out of Memory Error
363-
If you encounter an OutOfMemoryError (OOM) thrown by DataSystems during operation, please increase the value of the configuration option `--shared_memory_size_mb`.
392+
393+
If Yuanrong throws an OOM error during operation:
364394
```
365395
RuntimeError: code: [Out of memory], msg: [Shared memory no space in arena: ...]
366396
```
367397

398+
Solution: Increase `--shared_memory_size_mb` in `worker_args`, or reduce the data volume being cached.
399+
368400

369401
## Datasystem Logs
370402

@@ -384,13 +416,13 @@ We recommend developing inside a CANN container.
384416

385417
#### Option 1: Docker Image (Recommended)
386418

387-
First, select the appropriate [CANN image](https://hub.docker.com/r/ascendai/cann) aligned with your **CANN version**, **Ascend hardware**, **OS**, and **Python version**. For examples:
419+
First, select the appropriate [CANN image](https://hub.docker.com/r/ascendai/cann) aligned with your **CANN version**, **Ascend hardware**, **OS**, and **Python version**. For example:
388420

389421
| CANN Version | Ascend Hardware | OS | Python Version | Image Name |
390422
| ------------ | --------------- | ------------ | -------------- | ------------------------------------ |
391423
| 8.2.rc1 | A3 | Ubuntu 22.04 | 3.11 | cann:8.2.rc1-a3-ubuntu22.04-py3.11 |
392424
| 8.2.rc1 | 910B | Ubuntu 22.04 | 3.11 | cann:8.2.rc1-910b-ubuntu22.04-py3.11 |
393-
425+
---
394426
Pull the image:
395427

396428
```bash

0 commit comments

Comments
 (0)