[rayorch] integrate RayOrch for parallel data processing with new operators and storage classes#501
Merged
[rayorch] integrate RayOrch for parallel data processing with new operators and storage classes#501
Conversation
…rators and storage classes - Added `RayAcceleratedOperator` for transparent data-parallel execution of DataFlow operators. - Introduced `InMemoryStorage` for efficient in-memory data handling within Ray actors. - Created dummy operators for testing and validation of the Ray integration. - Updated `requirements.txt` to include `rayorch` dependency. - Added comprehensive tests for both serial and parallel execution scenarios. This implementation enhances the DataFlow framework by enabling efficient multi-GPU processing without modifying existing operators.
Collaborator
Author
|
@haolpku 先不要合并，这个还是等试用期各自体验下，再进主分支。 |
- Added logic to automatically release actor resources after each stage in PipelineABC, BatchedPipelineABC, and StreamBatchedPipelineABC. - Updated documentation in README files to clarify the necessity of calling shutdown() for multiple RayAcceleratedOperators. - Introduced tests to verify the auto-shutdown functionality across different pipeline types. This enhancement ensures efficient resource management and prevents GPU resource blocking during pipeline execution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RayAcceleratedOperatorfor transparent data-parallel execution of DataFlow operators.InMemoryStoragefor efficient in-memory data handling within Ray actors.requirements.txtto includerayorchdependency.This implementation enhances the DataFlow framework by enabling efficient multi-GPU processing without modifying existing operators.