|
1 | | -# xfer — Scalable S3↔S3 Transfers with rclone, Slurm, and pyxis |
| 1 | +# xfer |
| 2 | +Scalable S3↔S3 Transfers with rclone, Slurm, and pyxis |
2 | 3 |
|
3 | 4 |  |
4 | 5 |
|
@@ -113,8 +114,6 @@ uv run xfer run \ |
113 | 114 | --array-concurrency 96 \ |
114 | 115 | --rclone-image rclone/rclone:latest \ |
115 | 116 | --rclone-config ~/.config/rclone/rclone.conf \ |
116 | | - --rclone-source-opts "--s3-endpoint https://objects.source.example.com" \ |
117 | | - --rclone-dest-opts "--s3-endpoint https://objects.dest.example.com" \ |
118 | 117 | --rclone-flags "--transfers 48 --checkers 96 --fast-list --stats 30s" \ |
119 | 118 | --partition transfer \ |
120 | 119 | --cpus-per-task 4 \ |
@@ -195,9 +194,7 @@ uv run xfer slurm render \ |
195 | 194 | --cpus-per-task 4 \ |
196 | 195 | --mem 8G \ |
197 | 196 | --rclone-image rclone/rclone:latest \ |
198 | | - --rclone-config ~/.config/rclone/rclone.conf \ |
199 | | - --rclone-source-opts "--s3-endpoint https://objects.source.example.com" \ |
200 | | - --rclone-dest-opts "--s3-endpoint https://objects.dest.example.com" |
| 197 | + --rclone-config ~/.config/rclone/rclone.conf |
201 | 198 | ``` |
202 | 199 |
|
203 | 200 | --- |
@@ -319,3 +316,12 @@ run/ |
319 | 316 | * **rclone handles object-level idempotency** |
320 | 317 | * **Slurm handles node-level failures** |
321 | 318 | * **xfer handles orchestration only** |
| 319 | + |
| 320 | + |
| 321 | +## Contributing |
| 322 | +* To enable pre-commit `black` formatting, run `uv run pre-commit install` |
| 323 | + * If necessary, you can format locally with `uv run black .` |
| 324 | +* Name branches as either: |
| 325 | + * `<your name>/<branch name>` (e.g., `alice/update-readme`) |
| 326 | + * `<type of contribution>/<branch name>` (e.g., `feature/claude-integration`) (these are usually `feature`, `patch`, or `docs`) |
| 327 | +* Do NOT squash PRs into a single commit |
0 commit comments