Commit d07ad02
authored
feat: add S3ArtifactService with native async and atomic versioning (#115)
* feat: add S3ArtifactService with native async and atomic versioning
Adds S3ArtifactService to the artifacts module, providing:
- Native async I/O via aioboto3 (no asyncio.to_thread wrappers)
- Atomic versioning using S3 IfNoneMatch conditional writes
- Session-scoped and user-scoped artifact namespacing
- Custom metadata (JSON-serialised in S3 user-metadata)
- Batch delete (1000 keys per request) for efficient cleanup
- Paginated listing for large artifact collections
- Parallel head_object calls in list_artifact_versions
- Optional [s3] dependency group (aioboto3>=13.0.0)
- Comprehensive test suite with full async mock infrastructure
Closes #37
Closes #71
* style: match codebase conventions for docstrings and comments
- Remove section comment banners (# --- heading --- patterns)
- Simplify module and class docstrings to one-liners
- Move Args section to __init__ docstring (matches RedisSessionService)
- Shorten method docstrings to single line
- Remove explanatory inline comments
- Clean up blank lines left from removals
* chore: remove accidental tests/__init__.py
* feat: add S3ArtifactService using aioboto3
Adds S3-backed artifact storage with:
- Atomic versioning via IfNoneMatch conditional writes
- Async-safe session initialization with asyncio.Lock
- Bounded concurrent head_object calls (semaphore=10)
- S3 metadata size validation (2KB limit)
- User-scoped artifact namespace support
- Full test coverage with mocked S3 client
* refactor(s3): address PR review comments
- Metadata size check now accounts for S3 x-amz-meta- prefix overhead
(~11 bytes per key) in the 2KB limit calculation.
- Replace unconventional iter(int, 1) infinite loop with while True and
add exponential backoff (100ms, 200ms, ... capped at 5s) between
version conflict retries.
- list_artifact_keys now uses S3 Delimiter='/' with CommonPrefixes for
O(unique-keys) efficiency instead of listing every version object.
- load_artifact returns Part.from_text() for text/* content types so
consumers can check part.text consistently.
* fix(test): add importorskip guard for S3 optional dependencies
The S3 artifact tests require aioboto3 and botocore which are only
installed with the [s3] extra. CI runs with --extra test only, so
these tests fail with ModuleNotFoundError on collection.
Adding pytest.importorskip() at module level gracefully skips the
entire test file when the S3 dependencies aren't available.
* fix: address remaining review comments
- Consolidate aioboto3 import: cache module ref on self._aioboto3 in
__init__, remove duplicate import in _get_session
- Default save_max_retries to 10 (safe cap) instead of -1 (infinite)
- Normalize metadata keys to lowercase in _flatten_metadata with
case-collision warning (S3 metadata keys are case-insensitive)
- Handle ClientError in _head helper: return None for objects deleted
between list_objects_v2 and head_object calls, skip in result loop1 parent a53f6a4 commit d07ad02
7 files changed
Lines changed: 1232 additions & 2 deletions
File tree
- src/google/adk_community
- artifacts
- tests/unittests/artifacts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | 27 | | |
| 28 | + | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | | - | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
0 commit comments