Commit 5ab750d
feat: let Dask use shard-aligned writes when sharding enabled (#54)
When --enable-sharding is True, set align_chunks=False to let Dask
rechunk to shard boundaries instead of chunk boundaries.
Aligns with Dask PRs #12104/#12105 for atomic shard writes.
Prevents partial-shard read-modify-rewrite cycles.
Non-sharded workflows unchanged (align_chunks=True preserved).
Co-authored-by: Emmanuel Mathot <emmanuel.mathot@gmail.com>1 parent 47f5282 commit 5ab750d
1 file changed
Lines changed: 3 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
716 | 716 | | |
717 | 717 | | |
718 | 718 | | |
| 719 | + | |
| 720 | + | |
719 | 721 | | |
720 | 722 | | |
721 | 723 | | |
722 | 724 | | |
723 | 725 | | |
724 | 726 | | |
725 | 727 | | |
726 | | - | |
| 728 | + | |
727 | 729 | | |
728 | 730 | | |
729 | 731 | | |
| |||
0 commit comments