Commit 2c5d857
feat(preprod): Add snapshots subcommand (#3110)
Updated version of #3049 to
discuss and iterate on things.
Notable changes:
- Removed `shard_index` parameter from the command. I'm not sure what
the purpose of that was originally.
- This uses the new `many` (batch) API from `objectstore_client`. All
uploads are executed as batch requests, reducing network overhead.
Unfortunately, with they way things are implemented now, we will still
have to buffer all files in memory before sending the request, as we
need to hash their contents to determine the filename. If we could just
use the filename as the key in objectstore, it would be much better
because that way we could stream the files over.
Note that the objectstore endpoint is currently gated by a feature flag,
and only enabled for internal orgs/teams.
Ref FS-233
---------
Co-authored-by: Noah Martin <noahm444@gmail.com>
Co-authored-by: Ryan Brooks <ryanjeffrey.brooks112@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent 466a2f0 commit 2c5d857
11 files changed
Lines changed: 1420 additions & 34 deletions
File tree
- src
- api
- data_types
- commands/build
- tests/integration
- _cases/build
- build
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
3 | 11 | | |
4 | 12 | | |
5 | 13 | | |
| |||
0 commit comments