feat: multi-level caching with automatic backfill

Cherry-picked from upstream PR mozilla#2581.

Adds hierarchical cache storage (e.g., disk -> redis -> s3) with
configurable write tolerance policies and automatic backfill from
slower tiers to faster ones.

Upstream: mozilla#2581

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: mmastrac

README.md

@@ -10,7 +10,7 @@
 sccache - Shared Compilation Cache
 ==================================
 
-sccache is a [ccache](https://ccache.dev/)-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on [local disk](docs/Local.md) or in one of [several cloud storage backends](#storage-options).
+sccache is a [ccache](https://ccache.dev/)-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on [local disk](docs/Local.md) or in one of [several cloud storage backends](#storage-options). Multi-level caching with automatic backfill is supported for hierarchical cache architectures (see [Multi-Level Cache](docs/MultiLevel.md)).
 
 sccache includes support for caching the compilation of Assembler, C/C++ code, [Rust](docs/Rust.md), as well as NVIDIA's CUDA using [nvcc](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html), and [clang](https://llvm.org/docs/CompileCudaWithLLVM.html), [AMD's ROCm HIP](https://rocm.docs.amd.com/projects/HIP/en/latest/index.html).
 
@@ -33,6 +33,7 @@ Table of Contents (ToC)
 * [Interaction with GNU `make` jobserver](#interaction-with-gnu-make-jobserver)
 * [Known Caveats](#known-caveats)
 * [Storage Options](#storage-options)
+  * [Multi-Level (Hierarchical Caching)](docs/MultiLevel.md)
   * [Local](docs/Local.md)
   * [S3](docs/S3.md)
   * [R2](docs/S3.md#R2)

docs/Configuration.md

@@ -39,6 +39,13 @@ cache_dir = "/home/user/.cache/sccache-dist-client"
 type = "token"
 token = "secrettoken"
 
+# Multi-level cache configuration
+# Define cache levels in order (fast to slow).
+# Each level must be separately configured below.
+# See docs/MultiLevel.md for details.
+[cache.multilevel]
+chain = ["disk", "redis", "s3"]
+write_policy = "l0"  # Optional: ignore, l0 (default), or all
 
 #[cache.azure]
 # Azure Storage connection string (see <https://docs.azure.cn/en-us/storage/common/storage-configure-connection-string>)
@@ -176,6 +183,41 @@ Note that some env variables may need sccache server restart to take effect.
 
 ### cache configs
 
+#### multi-level cache
+
+Multi-level caching enables hierarchical cache storage with automatic backfill. See the [Multi-Level Cache documentation](MultiLevel.md) for detailed information.
+
+* `SCCACHE_MULTILEVEL_CHAIN` comma-separated list of cache backend names to use in hierarchy (e.g., `disk,redis,s3`)
+  - Order matters: left-to-right is fast-to-slow (L0, L1, L2, ...)
+  - Valid names: `disk`, `redis`, `memcached`, `s3`, `gcs`, `azure`, `gha`, `webdav`, `oss`, `cos`
+  - Each level must be separately configured with its own environment variables
+  - If not set, sccache uses single-level mode (legacy behavior)
+* `SCCACHE_MULTILEVEL_WRITE_POLICY` controls error handling on cache writes (default: `l0`)
+  - `ignore` - never fail on write errors, log warnings only (most permissive)
+  - `l0` - fail only if L0 (first level) write fails (default, balances reliability and performance)
+  - `all` - fail if any read-write level fails (most strict)
+  - Read-only levels are always skipped and never cause failures
+
+**Basic example**:
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+export SCCACHE_DIR="/tmp/cache"              # for disk level
+export SCCACHE_REDIS_ENDPOINT="redis://..."  # for redis level
+export SCCACHE_BUCKET="my-bucket"            # for s3 level
+```
+
+**Write policy examples**:
+```bash
+# Default: Fail only if disk write fails
+export SCCACHE_MULTILEVEL_WRITE_POLICY="l0"
+
+# Best effort: Never fail on cache writes
+export SCCACHE_MULTILEVEL_WRITE_POLICY="ignore"
+
+# Strict: Fail if any level write fails
+export SCCACHE_MULTILEVEL_WRITE_POLICY="all"
+```
+
 #### disk (local)
 
 * `SCCACHE_DIR` local on disk artifact cache directory

docs/MultiLevel.md

@@ -0,0 +1,249 @@
+# Multi-Level Cache
+
+Multi-level caching enables hierarchical cache storage, similar to how CPUs use L1/L2/L3 caches or CDNs use edge/regional/origin tiers. This feature allows sccache to check multiple storage backends in sequence, dramatically improving cache hit rates and reducing latency.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Architecture](#architecture)
+- [Use Cases](#use-cases)
+- [Configuration](#configuration)
+- [Best Practices](#best-practices)
+
+## Overview
+
+Multi-level caching allows you to configure multiple cache storage backends that work together:
+
+- **Fast, small caches** (e.g., local disk) are checked first
+- **Slower, larger caches** (e.g., S3) are checked if earlier levels miss
+- **Cache hits at any level** return immediately to the compiler
+- **Automatic backfill** copies data from slower to faster levels for future requests
+- **Write-through** ensures all levels stay synchronized on writes
+
+This creates a cache hierarchy where frequently accessed artifacts stay in fast storage while less common ones are still available from slower storage.
+
+## Architecture
+
+### Cache Hierarchy
+
+```
+┌─────────────────────────────────────────────────┐
+│               Compiler Request                  │
+└─────────────────────┬───────────────────────────┘
+                      │
+         ┌────────────▼────────────┐
+         │   Multi-Level Storage   │
+         └────────────┬────────────┘
+                      │
+      ┌───────────────┼───────────────┐
+      │               │               │
+┌─────▼─────┐    ┌────▼────┐     ┌────▼────┐
+│ Level 0   │    │ Level 1 │     │ Level 2 │
+│  (Disk)   │    │ (Redis) │     │  (S3)   │
+│           │    │         │     │         │
+│ Fast      │    │ Medium  │     │ Slow    │
+│ Small     │    │ Medium  │     │ Large   │
+│ ~5ms      │    │ ~10ms   │     │ ~200ms  │
+└───────────┘    └─────────┘     └─────────┘
+```
+
+### Read Path (Cache Hit at Level 2)
+
+```
+1. Check L0 (disk)    → Miss (5ms)
+2. Check L1 (redis)   → Miss (10ms)
+3. Check L2 (s3)      → Hit! (200ms)
+4. Return to compiler wrapper / sccache (Total: 215ms)
+5. Background: Backfill L2→L1 (async, non-blocking)
+6. Background: Backfill L2→L0 (async, non-blocking)
+7. Next request: Check L0 → Hit! (10ms)
+```
+
+### Write Path
+
+All write operations go to **all configured levels** in parallel:
+
+```
+Compiler writes artifact
+    ├─> L0 (disk)  ✓
+    ├─> L1 (redis) ✓
+    └─> L2 (s3)    ✓
+```
+
+If any level fails, the error is logged but the write succeeds if at least one level accepts it.
+
+## Use Cases
+
+### 1. CI/CD with Shared Team Cache
+
+**Problem**: Each CI runner has isolated disk cache, no sharing across machines.
+
+**Solution**: Add Redis or Memcached as L1
+```bash
+SCCACHE_MULTILEVEL_CHAIN="disk,redis"
+SCCACHE_DIR="/tmp/sccache"
+SCCACHE_REDIS_ENDPOINT="redis://cache.internal:6379"
+```
+
+**Result**: Fast local hits when available, team-shared cache otherwise.
+
+### 2. Enterprise with CDN-like Architecture
+
+**Problem**: Global team with high S3 latency, want local speed.
+
+**Solution**: Multi-tier hierarchy
+```bash
+SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+```
+
+- L0: Local disk (instant)
+- L1: Regional Redis (5-10ms)
+- L2: Global S3 bucket (50-200ms)
+
+**Result**: 90%+ hits at L0/L1, L2 as long-term backup.
+
+### 3. Developer Workstation with Cloud Backup
+
+**Problem**: Local disk fills up, don't want to lose cache history.
+
+**Solution**: Disk + cloud storage
+```bash
+SCCACHE_MULTILEVEL_CHAIN="disk,s3"
+SCCACHE_DIR="$HOME/.cache/sccache"
+SCCACHE_BUCKET="my-personal-sccache"
+SCCACHE_CACHE_SIZE="5G"  # Keep disk small
+```
+
+**Result**: Unlimited cloud storage, fast local hits.
+
+## Configuration
+
+### Via Environment Variables
+
+The primary configuration is `SCCACHE_MULTILEVEL_CHAIN`:
+
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+```
+
+**Format**: Comma-separated list of cache backend names
+**Order**: Left-to-right is fast-to-slow (L0, L1, L2, ...)
+**Valid names**: `disk`, `redis`, `memcached`, `s3`, `gcs`, `azure`, `gha`, `webdav`, `oss`, `cos`
+
+### Write Policy Configuration
+
+Control how sccache handles write failures across cache levels using `SCCACHE_MULTILEVEL_WRITE_POLICY`:
+
+**Available policies**:
+- **`ignore`** - Never fail on write errors, log warnings only (most permissive)
+- **`l0`** - Fail only if L0 (first level) write fails (default - balances reliability and performance)
+- **`all`** - Fail if any read-write level write fails (most strict)
+
+**Note**: Read-only levels are always skipped during writes and never cause failures.
+
+#### Write Policy Examples
+
+**Example 1: Default Behavior (l0 policy)**
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+export SCCACHE_MULTILEVEL_WRITE_POLICY="l0"  # or omit, it's the default
+```
+Compilation succeeds if disk write succeeds. Redis/S3 failures are logged but don't block compilation. Ensures local cache is always populated. **Best for most use cases.**
+
+**Example 2: Best Effort (ignore policy)**
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+export SCCACHE_MULTILEVEL_WRITE_POLICY="ignore"
+```
+Compilation always succeeds, even if all writes fail. Write failures are logged as warnings. **Best for unstable cache backends** where you don't want cache issues blocking builds.
+
+**Example 3: Strict Consistency (all policy)**
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+export SCCACHE_MULTILEVEL_WRITE_POLICY="all"
+```
+Compilation succeeds only if all read-write levels succeed. Any write failure fails the compilation. **Best for critical environments** where cache consistency is mandatory.
+
+#### Read-Only Levels
+
+Any level configured as read-only (e.g., `SCCACHE_LOCAL_RW_MODE=READ_ONLY`) is automatically skipped during writes, regardless of write policy:
+
+```bash
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis"
+export SCCACHE_MULTILEVEL_WRITE_POLICY="all"
+export SCCACHE_LOCAL_RW_MODE="READ_ONLY"  # Disk is read-only
+# Compilation succeeds if Redis write succeeds (disk is skipped)
+```
+
+### Complete Example
+
+```bash
+# Multi-level configuration
+export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
+export SCCACHE_MULTILEVEL_WRITE_POLICY="l0"  # Default: fail only if disk fails
+
+# Level 0: Disk cache
+export SCCACHE_DIR="/var/cache/sccache"
+export SCCACHE_CACHE_SIZE="10G"
+
+# Level 1: Redis cache
+export SCCACHE_REDIS_ENDPOINT="redis://localhost:6379"
+export SCCACHE_REDIS_EXPIRATION="86400"  # 24 hours
+
+# Level 2: S3 cache
+export SCCACHE_BUCKET="my-sccache-bucket"
+export SCCACHE_REGION="us-east-1"
+export SCCACHE_S3_USE_SSL="true"
+```
+
+### Via Configuration File
+
+```toml
+# ~/.config/sccache/config
+[cache.multilevel]
+chain = ["disk", "redis", "s3"]
+write_policy = "l0"  # Optional: ignore, l0 (default), or all
+
+[cache.disk]
+dir = "/var/cache/sccache"
+size = 10737418240  # 10GB
+
+[cache.redis]
+endpoint = "redis://localhost:6379"
+expiration = 86400
+
+[cache.s3]
+bucket = "my-sccache-bucket"
+endpoint = "s3-us-east-1.amazonaws.com"
+use_ssl = true
+```
+
+### Single Level (No Multi-Level)
+
+If `SCCACHE_MULTILEVEL_CHAIN` is not set, sccache uses the first configured cache backend (legacy behavior):
+
+```bash
+# Just uses disk (backwards compatible)
+export SCCACHE_DIR="/tmp/cache"
+```
+
+## Best Practices
+
+### 1. Order Levels by Latency (Fastest First)
+
+**Good**: `disk,redis,s3` (10ms → 50ms → 200ms)
+**Bad**: `s3,disk,redis` (slow L0 blocks every request)
+
+### 2. Match Cache Sizes to Access Patterns
+
+- **L0 (disk)**: Small, hot data (5-10GB)
+- **L1 (redis)**: Team shared, medium (50-100GB)
+- **L2 (s3)**: Unlimited, cold storage
+
+## See Also
+
+- [Configuration Options](Configuration.md) - Full config reference
+- [Local Cache](Local.md) - Disk cache details
+- [Redis Cache](Redis.md) - Redis configuration
+- [S3 Cache](S3.md) - S3 configuration
+- [Caching](Caching.md) - How cache keys are computed