Add S3 backend core

[shaypal5]

Summary

Closes #41.

Adds a new s3 backend to cachier, allowing function results to be persistently cached in AWS S3 (or any S3-compatible service: MinIO, localstack, etc.).

• New backend: @cachier(backend="s3", s3_bucket="my-bucket") - same decorator API as all other cores
• Flexible client config: pass a pre-built boto3 client (s3_client), a callable factory (s3_client_factory), or let the core auto-create one from s3_region / s3_endpoint_url / s3_config
• Data model: one pickled CacheEntry per cache key, stored at <s3_prefix>/<func_str>/<cache_key>.pkl
• Async: async cached functions are supported via asyncio.to_thread delegation (boto3 is sync-only)
• No import-time warnings: boto3 absence is only signalled when the S3 core is actually instantiated

Files changed

File	Change
src/cachier/cores/s3.py	New _S3Core class implementing the full _BaseCore interface
src/cachier/_types.py	Add S3Client type alias; extend Backend literal with "s3"
src/cachier/core.py	Wire S3 decorator parameters; import and instantiate _S3Core
pyproject.toml	Add s3 pytest marker; add [project.optional-dependencies] extras
tests/s3_tests/	18 tests using moto[s3] (no real AWS account required)
tests/requirements_s3.txt	boto3 + moto[s3] test deps
examples/s3_example.py	Runnable demo

Test plan

• pytest -m s3 - all 18 S3 tests pass (uses moto for offline mocking)
• pytest -m "not (mongo or redis or sql or s3)" - 219 existing tests pass, 0 regressions
• ruff check - no linting errors on changed files
• mypy src/cachier/ - only pre-existing SQL overload error; boto3 stubs handled with # type: ignore[import-untyped]

Open questions (from issue)

The following points from the original issue are left for follow-up if desired:

• Async s3_client_factory (currently sync-only, matching boto3's nature)
• Server-side encryption options in the core API
• Multipart uploads for very large cached values

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (d3b000a) to head (6b3e297).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##            master      #338    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           11        12     +1     
  Lines         1477      1693   +216     
  Branches       185       211    +26     
==========================================
+ Hits          1477      1693   +216     

Flag	Coverage Δ
local	58.35% <25.11%> (-5.02%)	⬇️
mongodb	41.05% <21.91%> (-2.96%)	⬇️
postgres	44.00% <21.91%> (-3.60%)	⬇️
redis	47.31% <21.46%> (-3.95%)	⬇️
s3	43.00% <98.63%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/cachier/_types.py	100.00% <100.00%> (ø)
src/cachier/core.py	100.00% <100.00%> (ø)
src/cachier/cores/s3.py	100.00% <100.00%> (ø)

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3b000a...6b3e297. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gencurrent]

Just 2 notes:

1. For async operations it's possible to use aioboto3.
2. README.rst contains a table of sync / async cores, which would be great to have completed with the new core.

[shaypal5]

@codex Work on the existing branch feature/s3-backend in /Users/shaypalachy/clones/cachier.

Goal: make PR #338 merge-ready by fixing the two real gaps:

1. S3 async behavior should be safe for event loops (no direct blocking sync I/O in async code paths).
2. README must fully reflect S3 support, including the sync/async compatibility table.

Constraints:

• Keep this PR scoped. Do NOT add native aioboto3 support in this change.
• Keep S3 async mode as “delegated” (thread-offloaded sync boto3), not “native async”.
• Preserve backward compatibility.
• Follow repo conventions in AGENTS.md and existing style.

Required code changes:

1. In src/cachier/cores/s3.py, override async methods used by async wrappers so they offload sync boto3 calls via asyncio.to_thread:
   • aget_entry (or aget_entry_by_key)
   • aset_entry
   • amark_entry_being_calculated
   • amark_entry_not_calculated
     Ensure async-decorated functions using S3 do not run blocking boto3 calls directly on the event-loop thread.
2. Add/adjust comments/docstrings to clearly state S3 async support is delegated, not native.
3. Add async S3 tests in tests/s3_tests/test_s3_core.py:
   • at least one @pytest.mark.asyncio test that decorates an async def with @cachier(backend="s3", ...) and verifies cache hit behavior.
   • ensure tests fail on regression (not just smoke).
4. Update README.rst:
   • Move S3 from “Future features” to current supported cores.
   • Add S3 to any “additional core dependencies” text.
   • Update “Core Sync/Async Compatibility” table with S3 row:
     • Sync: Yes
     • Async: Yes (delegated)
     • Constraint: delegated/thread-offloaded sync boto3 implementation; no async client required.
   • Keep wording consistent with existing table semantics.
5. If needed, add a short S3 “Sync/Async Support” subsection near other backend sections.

Validation:

• Run targeted tests: pytest tests/s3_tests/test_s3_core.py
• Run lint on changed files (ruff).
• If full test suite is too heavy, report exactly what was and wasn’t run.

Deliverable:

• Implement changes directly.
• Return a concise summary with:
  • files changed
  • key behavior change for async S3
  • test results
  • any follow-up recommendation (e.g., separate future issue for native aioboto3 support).

[chatgpt-codex-connector]

Summary

• Updated the S3 core async behavior to be explicitly event-loop safe by overriding async methods and offloading sync boto3 operations with asyncio.to_thread (aget_entry, aget_entry_by_key, aset_entry, amark_entry_being_calculated, amark_entry_not_calculated). This preserves delegated async semantics (not native async). src/cachier/cores/s3.pyL29-L35 src/cachier/cores/s3.pyL374-L422
• Added an async regression test for S3 cache-hit behavior using @pytest.mark.asyncio and @cachier(backend="s3", ...), asserting second async call is served from cache (call_count == 1). tests/s3_tests/test_s3_core.pyL191-L209
• Updated README to reflect S3 as a current supported backend, removed it from future features, added S3 dependency install guidance, documented delegated sync/async behavior, and added S3 to the core compatibility table as Yes (delegated) with thread-offloaded constraint wording. README.rstL55-L76 README.rstL583-L587 README.rstL598-L644 README.rstL669-L676

Key behavior change for async S3

• Async-decorated functions using the S3 backend now avoid direct blocking I/O on the event-loop thread by routing key async cache operations through asyncio.to_thread, while still using sync boto3 underneath (delegated mode, backward compatible). src/cachier/cores/s3.pyL374-L422

Testing

• ⚠️ pytest tests/s3_tests/test_s3_core.py (all tests skipped in this environment because boto3/moto are not installed).
• ✅ ruff check src/cachier/cores/s3.py tests/s3_tests/test_s3_core.py

Follow-up recommendation

• If desired later, open a separate issue/PR for optional native async S3 support (aioboto3) as an additive enhancement; keep current delegated mode as default for compatibility.

View task →

[shaypal5]

@codex Line 328 in core.py ( core = _S3Core() and 155 different lines in src/cachier/cores/s3.py are not currently covered by our tests. Please add tests to achieve full test coverage of the src/cachier/cores/s3.py file, and of line 328 in src/cachier/core.py.

[chatgpt-codex-connector]

Summary

• Added a decorator wiring regression test (test_core_decorator_constructs_s3_core) that monkeypatches _S3Core and verifies the backend="s3" instantiation path is exercised, covering the missing core.py S3 branch target (line 328). tests/s3_tests/test_s3_core.pyL385-L437
• Added comprehensive branch-focused S3 core tests to cover internal helpers and error handling paths, including:
  • _get_s3_client factory and auto-created client kwargs behavior,
  • _load_entry deserialization/construction failures,
  • get_entry_by_key success + client/general exception paths,
  • set_entry/mark/unmark/wait/recalculation paths,
  • clear_cache batching/error behavior,
  • clear_being_calculated and delete_stale_entries branch/error behavior. tests/s3_tests/test_s3_core.pyL441-L717
• Added an import-fallback test that executes src/cachier/cores/s3.py with forced boto3 import failure to cover the except ImportError initialization branch (BOTO3_AVAILABLE = False). tests/s3_tests/test_s3_core.pyL721-L740

Testing

• ✅ ruff check tests/s3_tests/test_s3_core.py
• ✅ pytest tests/s3_tests/test_s3_core.py -q
• ✅ coverage run -m pytest tests/s3_tests/test_s3_core.py -q
• ✅ coverage report -m src/cachier/cores/s3.py src/cachier/core.py (shows src/cachier/cores/s3.py at 100%)
• ✅ coverage json -o /tmp/cov.json && python - <<'PY'\nimport json\nj=json.load(open('/tmp/cov.json'))\nlines=set(j['files']['src/cachier/core.py']['executed_lines'])\nprint(328 in lines, 327 in lines, 329 in lines)\nPY (confirmed line 328 executed)

View task →

[shaypal5]

@Borda Please take a loot at this PR (#338)