Skip to content

Commit b6b41ae

Browse files
authored
[chore] Update README (Ascend#17)
1. Update pictures. 2. Update `StreamingDataLoader` related descriptions. 3. Simplifies the expression about user interface. 4. Update install commands. --------- Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
1 parent 5f793df commit b6b41ae

1 file changed

Lines changed: 26 additions & 15 deletions

File tree

README.md

Lines changed: 26 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,16 @@ TransferQueue is a high-performance data storage and transfer module with panora
2222
<img src="https://github.com/TransferQueue/community_doc/blob/main/docs/tq_arch.png?raw=true" width="70%">
2323
</p>
2424

25-
TransferQueue offers **fine-grained, sample-level** data management and **load-balancing** (on the way) capabilities, serving as a data gateway that decouples explicit data dependencies across computational tasks. This enables a divide-and-conquer approach, significantly simplifies the algorithm controller design.
25+
TransferQueue offers **fine-grained, sub-sample-level** data management and **load-balancing** (on the way) capabilities, serving as a data gateway that decouples explicit data dependencies across computational tasks. This enables a divide-and-conquer approach, significantly simplifies the algorithm controller design.
2626

2727
<p align="center">
2828
<img src="https://github.com/TransferQueue/community_doc/blob/main/docs/main_func.png?raw=true" width="70%">
2929
</p>
3030

3131
<h2 id="updates">🔄 Updates</h2>
3232

33-
- **Dec 30, 2025**: **TransferQueue x verl** integration is tested with the DAPO algorithm at scale **(64 nodes, 1024 cards)**. It significantly optimizes host memory utilization and accelerates data transfers. Stay tuned for more details!
33+
- **Jan 28, 2026**: We experimentally introduce `StreamingDataloader` interface for fully-streamed production-consumption pipeline. Refer to our [tutorials/05_streaming_dataloader.py](https://github.com/Ascend/TransferQueue/blob/main/tutorial/05_streaming_dataloader.py) for details.
34+
- **Dec 30, 2025**: **TransferQueue x verl** integration is tested with the DAPO algorithm at scale **(64 nodes, 1024 cards)**. It significantly optimizes host memory utilization and accelerates data transfers. Stay tuned for more details!
3435
- **Dec 20, 2025**: 🔥 The official [tutorial](https://github.com/Ascend/TransferQueue/tree/main/tutorial) is released! Feel free to check it out.
3536
- **Nov 10, 2025**: We disentangle the data retrieval logic from TransferQueueController [PR#101](https://github.com/TransferQueue/TransferQueue/pull/101). Now you can implement your own `Sampler` to control how to consume the data.
3637
- **Nov 5, 2025**: We provide a `KVStorageManager` that simplifies the integration with KV-based storage backends [PR#96](https://github.com/TransferQueue/TransferQueue/pull/96). The first available KV-based backend is [Yuanrong](https://gitcode.com/openeuler/yuanrong-datasystem).
@@ -90,16 +91,10 @@ This data structure design is motivated by the computational characteristics of
9091
</p>
9192

9293
### User Interface: Asynchronous & Synchronous Client
94+
To simplify the usage of TransferQueue, we have encapsulated this process into `TransferQueueClient`. The client provides both asynchronous and synchronous interfaces for data transfer, allowing users to easily integrate TransferQueue into their framework.
9395

94-
The interaction workflow of TransferQueue system is as follows:
95-
96-
1. A process sends a read request to the `TransferQueueController`.
97-
2. `TransferQueueController` scans the production and consumption metadata for each sample (row), and dynamically assembles a micro-batch metadata according to the load-balancing policy. This mechanism enables sample-level data scheduling.
98-
3. The process retrieves the actual data from distributed storage units using the metadata provided by the controller.
99-
100-
To simplify the usage of TransferQueue, we have encapsulated this process into `AsyncTransferQueueClient` and `TransferQueueClient`. These clients provide both asynchronous and synchronous interfaces for data transfer, allowing users to easily integrate TransferQueue into their framework.
101-
102-
> In the future, we will provide a `StreamingDataLoader` interface for disaggregated frameworks as discussed in [issue#85](https://github.com/TransferQueue/TransferQueue/issues/85) and [verl/RFC#2662](https://github.com/volcengine/verl/discussions/2662). Leveraging this abstraction, each rank can automatically get its own data like `DataLoader` in PyTorch. The TransferQueue system will handle the underlying data scheduling and transfer logic caused by different parallelism strategies, significantly simplifying the design of disaggregated frameworks.
96+
We also experimentally provide a `StreamingDataLoader` interface as a standard PyTorch DataLoader. Leveraging this abstraction, each rank can automatically get its own data like `DataLoader` in PyTorch. The TransferQueue system will handle the underlying data scheduling and transfer logic caused by different parallelism strategies, significantly simplifying the design of disaggregated frameworks.
97+
This interface simplifies TransferQueue's integration, ensuring seamless compatibility with existing training workflows. Please refer to our [Roadmap](https://github.com/Ascend/TransferQueue/issues/1) and [tutorials/05_streaming_dataloader.py](https://github.com/Ascend/TransferQueue/blob/main/tutorial/05_streaming_dataloader.py) for more details.
10398

10499
<h2 id="show-cases">🔥 Showcases</h2>
105100

@@ -135,7 +130,11 @@ You may refer to the [recipe](https://github.com/Ascend/TransferQueue/tree/dev/r
135130

136131
### Disaggregated Example
137132

138-
Work in progress :)
133+
We have implemented a series of PRs ([#4](https://github.com/Ascend/TransferQueue/pull/4), [#7](https://github.com/Ascend/TransferQueue/pull/7), [#9](https://github.com/Ascend/TransferQueue/pull/9), [#16](https://github.com/Ascend/TransferQueue/pull/16)) to establish a **standardized, fully-streamed distributed** workflow via TransferQueue.
134+
135+
By leveraging the `RankAwareSampler` and `StreamingDataLoader` interfaces, we achieve a **streamlined micro-batch-level producer-consumer pipeline**. This design eliminates the need to manually determine data dispatching logic across varying parallelism strategies—a typical complexity in the single-controller paradigm—thereby greatly simplifying framework design.
136+
137+
Please refer to our [Roadmap](https://github.com/Ascend/TransferQueue/issues/1) and [tutorials/05_streaming_dataloader.py](https://github.com/Ascend/TransferQueue/blob/main/tutorial/05_streaming_dataloader.py) for more details.
139138

140139
<p align="center">
141140
<img src="https://github.com/TransferQueue/community_doc/blob/main/docs/tq_streaming_dataloader.png?raw=true" width="70%">
@@ -148,18 +147,30 @@ Work in progress :)
148147
pip install TransferQueue
149148
```
150149

151-
### Build wheel package from source code
150+
### Install from source code
152151

153-
Follow these steps to build and install:
154152
1. Clone the source code from the GitHub repository
155153
```bash
156154
git clone https://github.com/Ascend/TransferQueue/
157155
cd TransferQueue
158156
```
159157

158+
2. Install from source code
159+
```bash
160+
pip install .
161+
```
162+
163+
### Build wheel package from source code
164+
165+
1. Clone the source code from the GitHub repository
166+
```bash
167+
git clone https://github.com/Ascend/TransferQueue/
168+
cd TransferQueue
169+
```
170+
160171
2. Install dependencies
161172
```bash
162-
pip install -r requirements.txt
173+
pip install build
163174
```
164175

165176
3. Build and install

0 commit comments

Comments
 (0)