Skip to content

Commit d07ad02

Browse files
authored
feat: add S3ArtifactService with native async and atomic versioning (#115)
* feat: add S3ArtifactService with native async and atomic versioning Adds S3ArtifactService to the artifacts module, providing: - Native async I/O via aioboto3 (no asyncio.to_thread wrappers) - Atomic versioning using S3 IfNoneMatch conditional writes - Session-scoped and user-scoped artifact namespacing - Custom metadata (JSON-serialised in S3 user-metadata) - Batch delete (1000 keys per request) for efficient cleanup - Paginated listing for large artifact collections - Parallel head_object calls in list_artifact_versions - Optional [s3] dependency group (aioboto3>=13.0.0) - Comprehensive test suite with full async mock infrastructure Closes #37 Closes #71 * style: match codebase conventions for docstrings and comments - Remove section comment banners (# --- heading --- patterns) - Simplify module and class docstrings to one-liners - Move Args section to __init__ docstring (matches RedisSessionService) - Shorten method docstrings to single line - Remove explanatory inline comments - Clean up blank lines left from removals * chore: remove accidental tests/__init__.py * feat: add S3ArtifactService using aioboto3 Adds S3-backed artifact storage with: - Atomic versioning via IfNoneMatch conditional writes - Async-safe session initialization with asyncio.Lock - Bounded concurrent head_object calls (semaphore=10) - S3 metadata size validation (2KB limit) - User-scoped artifact namespace support - Full test coverage with mocked S3 client * refactor(s3): address PR review comments - Metadata size check now accounts for S3 x-amz-meta- prefix overhead (~11 bytes per key) in the 2KB limit calculation. - Replace unconventional iter(int, 1) infinite loop with while True and add exponential backoff (100ms, 200ms, ... capped at 5s) between version conflict retries. - list_artifact_keys now uses S3 Delimiter='/' with CommonPrefixes for O(unique-keys) efficiency instead of listing every version object. - load_artifact returns Part.from_text() for text/* content types so consumers can check part.text consistently. * fix(test): add importorskip guard for S3 optional dependencies The S3 artifact tests require aioboto3 and botocore which are only installed with the [s3] extra. CI runs with --extra test only, so these tests fail with ModuleNotFoundError on collection. Adding pytest.importorskip() at module level gracefully skips the entire test file when the S3 dependencies aren't available. * fix: address remaining review comments - Consolidate aioboto3 import: cache module ref on self._aioboto3 in __init__, remove duplicate import in _get_session - Default save_max_retries to 10 (safe cap) instead of -1 (infinite) - Normalize metadata keys to lowercase in _flatten_metadata with case-collision warning (S3 metadata keys are case-insensitive) - Handle ClientError in _head helper: return None for objects deleted between list_objects_v2 and head_object calls, skip in result loop
1 parent a53f6a4 commit d07ad02

7 files changed

Lines changed: 1232 additions & 2 deletions

File tree

pyproject.toml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,12 @@ classifiers = [ # List of https://pypi.org/classifiers/
2424
]
2525
dependencies = [
2626
# go/keep-sorted start
27-
"google-genai>=1.21.1, <2.0.0", # Google GenAI SDK
2827
"google-adk", # Google ADK
28+
"google-genai>=1.21.1, <2.0.0", # Google GenAI SDK
2929
"httpx>=0.27.0, <1.0.0", # For OpenMemory service
30+
"orjson>=3.11.3",
3031
"redis>=5.0.0, <6.0.0", # Redis for session storage
3132
# go/keep-sorted end
32-
"orjson>=3.11.3",
3333
]
3434
dynamic = ["version"]
3535

@@ -40,6 +40,9 @@ changelog = "https://github.com/google/adk-python-community/blob/main/CHANGELOG.
4040
documentation = "https://google.github.io/adk-docs/"
4141

4242
[project.optional-dependencies]
43+
s3 = [
44+
"aioboto3>=13.0.0", # For S3ArtifactService
45+
]
4346
test = [
4447
"pytest>=8.4.2",
4548
"pytest-asyncio>=1.2.0",

src/google/adk_community/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15+
from . import artifacts
1516
from . import memory
1617
from . import sessions
1718
from . import version
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Community Artifact Services
2+
3+
This module contains community-contributed artifact service implementations for ADK.
4+
5+
## Available Services
6+
7+
### S3ArtifactService
8+
9+
Production-ready artifact storage using Amazon S3 (or any S3-compatible service such as MinIO, DigitalOcean Spaces, etc.).
10+
11+
**Installation:**
12+
```bash
13+
pip install google-adk-community[s3]
14+
```
15+
16+
**Usage:**
17+
```python
18+
from google.adk_community.artifacts import S3ArtifactService
19+
20+
artifact_service = S3ArtifactService(
21+
bucket_name="my-adk-artifacts",
22+
aws_configs={"region_name": "us-east-1"},
23+
)
24+
```
25+
26+
**Features:**
27+
- Native async I/O via `aioboto3` (no `asyncio.to_thread` wrappers)
28+
- Atomic versioning using S3 conditional writes (`IfNoneMatch`)
29+
- Session-scoped and user-scoped artifacts
30+
- Automatic version management
31+
- Custom metadata support (JSON-serialised)
32+
- Batch delete for efficient cleanup
33+
- Paginated listing for large artifact collections
34+
- Works with S3-compatible services (MinIO, DigitalOcean Spaces, etc.)
35+
36+
**See Also:**
37+
- [S3ArtifactService Implementation](./s3_artifact_service.py)
38+
- [Tests](../../../tests/unittests/artifacts/test_s3_artifact_service.py)
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2025 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
from .s3_artifact_service import S3ArtifactService
16+
17+
__all__ = [
18+
'S3ArtifactService',
19+
]

0 commit comments

Comments
 (0)