Commit c9749ef
authored
feat: add HuggingFace Buckets API support (#22)
* docs: add buckets support design spec
Design for adding HuggingFace Buckets API to the Rust SDK,
covering HFBucket handle, API methods, xet integration,
error handling improvements, blocking API, and CLI commands
matching the Python hf buckets interface.
* docs: add buckets implementation plan
13-task plan covering error handling, types, HFBucket handle,
xet refactoring, API methods, blocking wrapper, and CLI commands.
* feat: add BucketNotFound, Forbidden, Conflict, RateLimited error variants
Extend HFError with dedicated variants for HTTP 403, 409, and 429 status
codes, plus BucketNotFound for bucket-specific 404s. Update check_response
to map these status codes to the new variants instead of the generic Http
error. Remove now-redundant inner status code matching in format_hf_error.
* feat: add bucket data types and param structs
* feat: add HFBucket handle with factory method and URL helpers
* refactor: extract xet token URL builders for repo and bucket
* feat: add bucket lifecycle API methods on HFClient
* feat: add bucket scoped API, xet upload/download methods
Add impl HFBucket with info, list_tree, get_paths_info,
get_file_metadata, batch, delete_files, upload_files,
and download_files methods. Xet integration uses
bucket-specific token URLs.
* feat: add HFBucketSync blocking wrapper
* feat: add hfrs buckets CLI scaffolding with create, info, delete, move
* feat: add hfrs buckets list, remove, and cp commands
* fix: add missing filename field to XetBatchFile in bucket downloads
* fix: address code review findings for buckets implementation
- Restore RepoNotFound error context in fetch_xet_connection_info by
passing NotFoundContext and identifier through to check_response
- Add trailing newline to NDJSON batch body per spec
- Use keyed HashMap lookup in download_files instead of positional zip
to prevent silent data corruption on out-of-order server responses
- Simplify get_file_metadata by calling check_response directly,
removing unreachable!() and redundant status checks
- Update CLAUDE.md project layout with new bucket modules
- Add buckets example demonstrating create/list/info workflows
* docs: add bucket sync design spec
* docs: move progress into BucketSyncParams
* docs: add bucket sync implementation plan
* feat: add BucketSyncParams and SyncDirection types
* feat: add SyncPlan, SyncOperation, SyncAction types
* test: add unit tests for SyncPlan summary methods
* feat: implement HFBucket::sync() with plan computation and execution
* feat: add sync to blocking API wrapper
* feat: add hfrs buckets sync CLI subcommand
* test: add bucket sync integration tests
* feat: add buckets cargo feature gate
All bucket-related code (types, API, blocking, xet impl, CLI) is now
behind the "buckets" feature which requires "xet". The "cli" feature
includes "buckets" automatically.
* test: switch test repos to hf-internal-testing owned repos
- Models: openai-community/gpt2 → hf-internal-testing/tiny-gemma3
- Datasets: rajpurkar/squad, xet-team/xet-spec-reference-files → hf-internal-testing/cats_vs_dogs_sample
- Xet repo: mcpotato/42-xet-test-repo → hf-internal-testing/tiny-gemma3 (model.safetensors)
- Keep openai-community as test_model_author for list_models search tests
(hf-internal-testing models aren't returned by the listing API)
* test: add 1MB random binary file to bucket sync seed data
* fix: address code review findings for sync implementation
- Pass path as parameter to compare_files instead of returning empty
placeholder that callers must set after the fact
- Remove unused bucket_resolve_url from client.rs
- Wire SyncPlan.download_entries into execute_download_plan to avoid
redundant get_paths_info API call and ensure plan integrity
- Add structured tracing to HFBucket::xet_download_batch
* rm docs superpowers
* re-org
* try to fix rate limiting1 parent 54ea2a4 commit c9749ef
44 files changed
Lines changed: 3923 additions & 2162 deletions
File tree
- docs/superpowers
- plans
- specs
- huggingface_hub
- examples
- src
- api
- buckets
- bin/hfrs
- commands
- buckets
- types
- buckets
- tests
- cli
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| |||
96 | 97 | | |
97 | 98 | | |
98 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
99 | 104 | | |
100 | 105 | | |
101 | 106 | | |
| |||
104 | 109 | | |
105 | 110 | | |
106 | 111 | | |
107 | | - | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
108 | 126 | | |
109 | 127 | | |
110 | 128 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments