Improve dataset splitting#1589
Open
arteymix wants to merge 4 commits intohotfix-1.32.7from
Open
Conversation
Collaborator
Author
|
The only missing part is to persist the single-cell stuff in the Persister. |
arteymix
force-pushed
the
feature-improve-dataset-splitting
branch
from
9824fd1 to
1f0011a
Compare
Collaborator
Author
|
Involving @cyan864 since we'll need some examples to test this on. |
arteymix
force-pushed
the
hotfix-1.32.6
branch
from
b48a6e4 to
2dd881d
Compare
arteymix
force-pushed
the
feature-improve-dataset-splitting
branch
from
1f0011a to
2052ad8
Compare
Make splitting the experiment transactional by isolating the relevant operations in a helper service. Use generic BulkDataSlicerUtils and SingleCellSlicerUtils for slicing samples in splits which handle all the possible fixed-size encoding cases and single-cell data slicing. Add options to the splitExperiment CLI to skip post-processing and delete the original experiment automatically.
arteymix
force-pushed
the
feature-improve-dataset-splitting
branch
from
2052ad8 to
f26e61e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make splitting the experiment transactional by isolating the relevant operations in a helper service.
Use generic BulkDataSlicerUtils and SingleCellSlicerUtils for slicing samples in splits which handle all the possible fixed-size encoding cases and single-cell data slicing.
Add options to the splitExperiment CLI to skip post-processing and delete the original experiment automatically.