fix: fail closed on cache lookup failures

Author: seungpyoson

Cargo.toml

@@ -142,6 +142,7 @@ cc = "1.0"
 chrono = "0.4.42"
 divan = { package = "codspeed-divan-compat", version = "4" }
 filetime = "0.2"
+opendal = { version = "0.55.0", default-features = false, features = ["services-memory"] }
 predicates = "=3.1.0"
 serial_test = "3.1"
 temp-env = "0.3.6"

docs/Configuration.md

@@ -218,6 +218,16 @@ export SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY="ignore"
 export SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY="all"
 ```
 
+For every backend with an `*_RW_MODE` setting, sccache verifies the requested
+capability at server startup. `READ_WRITE` requires both readable storage and a
+successful write probe. `READ_ONLY` does not attempt a write, but still requires
+readable existing storage. A capability-check error prevents server startup.
+For object backends whose API reports missing objects and missing namespaces
+with the same status, sccache verifies namespace readability once at startup
+when list capability is available and reuses that result for subsequent misses.
+Credentialless public S3 and OSS caches remain exempt because public object-read
+access does not imply list access.
+
 #### disk (local)
 
 * `SCCACHE_DIR` local on disk artifact cache directory

docs/FailClosedCacheErrors.md

@@ -3,50 +3,79 @@
 ## Goal
 
 sccache must distinguish a missing cache entry from a cache lookup that
-failed. A missing entry remains a normal cache miss. Any other read used to
-determine a lookup result, capability-check, transport, or cache-entry decoding
-failure remains an error until it reaches the existing caller, compiler, or
-statistics handling.
+failed. Only confirmed absence is a normal cache miss. A capability, transport,
+filesystem, timeout, or decoding failure is fatal to that compilation; sccache
+does not hide the failure by invoking the compiler and publishing a replacement
+entry.
 
 ## Behavior
 
-- `NotFound` from a cache backend is a cache miss.
-- Every other backend read failure is an error.
+- A backend `NotFound` is a miss only after any separately addressable
+  bucket, container, or root namespace has been shown to be readable during
+  startup validation. That proof is cached rather than repeated for every
+  cold-cache lookup.
+- Every other backend read failure, and a cache lookup timeout, is an error.
 - A multi-level cache does not convert a level read failure into a miss.
 - Client-side path, raw, and preprocessor storage RPCs preserve daemon storage
   errors instead of encoding them as absent values.
-- Invalid cached bytes are errors rather than misses.
+- Invalid archive bytes and corrupt present members, including stdout, stderr,
+  required outputs, and optional outputs, are errors rather than misses.
 - Disk path and preprocessor entry read failures are errors rather than misses.
+- Preprocessor dependency validation treats confirmed deletion or content
+  change as invalidation, but propagates other filesystem failures.
 - A failed storage handshake is an error rather than an assumed read-write
   capability.
+- Client-side storage RPCs have an absolute deadline shorter than the compiler
+  lookup deadline. When it expires, sccache shuts down the IPC connection to
+  interrupt synchronous I/O, including a response that is arriving in small
+  increments.
 - Explicitly configured read-only storage does not require write capability.
 - Requested read-write storage fails its capability check when it cannot write.
 - A server disconnect after `CompileStarted` is an error unless
   `SCCACHE_IGNORE_SERVER_IO_ERROR=1` explicitly enables local fallback.
 
-These rules apply consistently to direct daemon compilation, client-side
-compilation, single-level storage, and multi-level storage.
+These rules apply to direct daemon compilation, client-side compilation,
+single-level storage, and multi-level storage. Only a confirmed miss or an
+explicit no-cache/recache request can reach compiler execution.
 
-Optional post-hit backfill remains best-effort. A failure in that optimization
-does not replace an already validated cache hit with a miss.
+Two optimizations remain best-effort:
+
+- A preprocessor entry metadata refresh does not discard an already validated
+  preprocessor hit when the storage is read-only or the refresh fails.
+- Multi-level backfill starts only after the consumer has validated the entire
+  hit. It copies the exact bytes that were validated, and a backfill failure
+  does not invalidate the completed hit.
 
 ## Implementation Boundaries
 
 `RemoteStorage` owns the OpenDAL error classification. `MultiLevelStorage`
 preserves errors returned by its levels. `DiskCache` distinguishes absent
-entries from path and preprocessor read failures. The storage IPC protocol
-carries fallible handshake, path, raw-read, and preprocessor-read results.
-`IpcStorage` converts those protocol errors and cache decoding errors into its
-existing `Storage` error result. Existing compiler handling remains responsible
-for recording cache errors and deciding the compilation outcome.
-
-No alternate fallback path or second error policy is introduced.
+entries from path and preprocessor read failures, initializes according to its
+configured mode, and avoids filesystem mutation in read-only mode. The storage
+IPC protocol carries an explicit version and fallible path, raw-read, and
+preprocessor-read results. The daemon validates capability and size once at
+startup; client-side handshakes return that validated metadata rather than
+re-probing the backend for every compiler invocation.
+
+Public S3 and OSS caches configured with the existing no-credentials option
+are the deliberate namespace-validation exception: public object reads do not
+imply bucket-list permission, so sccache preserves credentialless read-only
+operation and uses the provider's object-read `NotFound` classification without
+an independent list probe.
+
+Multi-level storage also caches every level's startup-validated mode. Writes
+skip read-only levels without re-running remote capability probes, and a mixed
+read-write/read-only chain remains writable through its read-write levels.
+
+The compiler boundary accepts only a hit, confirmed miss, or explicit cache
+control result. Storage errors, lookup timeouts, and cache-entry extraction
+errors return before compiler dispatch.
 
 ## Compatibility
 
-The storage IPC response shape changes, so clients and daemons must use the
-same sccache build. A mismatched peer fails instead of silently treating an
-error as a cache miss.
+The storage IPC protocol is versioned, so clients and daemons must use a
+compatible sccache build. A mismatched peer fails during the handshake instead
+of interpreting a changed payload shape.
 
 The capability-check behavior intentionally reverses the rate-limit tolerance
 introduced by sccache PR #1557. A rate-limited read or unavailable requested
@@ -57,15 +86,23 @@ operation must configure read-only mode explicitly.
 
 Behavior tests must prove:
 
-1. A direct remote `NotFound` is a miss and another remote read error is an
-   error.
+1. A remote missing object in a readable namespace is a miss, while an
+   unreadable namespace and other remote read failures are errors.
 2. A multi-level read error is not converted into a miss.
-3. Client-side path and raw-read errors cross the IPC boundary as errors.
-4. Disk path and preprocessor read failures are errors.
-5. Invalid client-side cached bytes are errors.
-6. Handshake capability and metadata errors cross the IPC boundary as errors.
-7. Explicit read-only mode does not require a successful write check.
-8. A post-`CompileStarted` disconnect fails by default and falls back locally
+3. A later-level hit returns without waiting for backfill, and corrupt or
+   inconsistent bytes are never backfilled.
+4. Client-side path, raw-read, and preprocessor-read errors cross the IPC
+   boundary as errors.
+5. Invalid cached bytes and corrupt present members are errors; absent optional
+   members and empty stdout/stderr remain valid.
+6. Disk read-write checks prove write capability, while read-only checks do not
+   create or mutate cache files.
+7. Preprocessor corruption and unexpected dependency I/O failures are errors,
+   while confirmed dependency change remains invalidation.
+8. Lookup errors and timeouts return before compiler dispatch.
+9. Handshake capability and metadata are checked once at startup, and a
+   protocol mismatch fails explicitly.
+10. A post-`CompileStarted` disconnect fails by default and falls back locally
    only with the documented opt-in.
 
 The affected feature configurations, formatter, linter, and library test suite

docs/GHA.md

@@ -19,4 +19,7 @@ The `SCCACHE_GHA_RW_MODE` environment variable can be set to `READ_ONLY` to make
 
 ## Behavior
 
-In case sccache reaches the rate limit of the service, the build will continue, but the storage might not be performed.
+At startup, sccache verifies the configured cache capability. A rate-limited
+read or a failed requested write check prevents the server from starting.
+Configure `SCCACHE_GHA_RW_MODE=READ_ONLY` when write capability is intentionally
+unavailable; read and transport failures remain errors.

docs/OSS.md

@@ -13,3 +13,8 @@ The `SCCACHE_OSS_RW_MODE` environment variable can be set to `READ_ONLY` to make
 Sccache is able to load credentials from environment variables: `ALIBABA_CLOUD_ACCESS_KEY_ID` and `ALIBABA_CLOUD_ACCESS_KEY_SECRET`.
 
 Alternatively, the `SCCACHE_OSS_NO_CREDENTIALS` environment variable can be set to use public readonly access to the OSS bucket, without the need for credentials. Valid values for this environment variable are `true`, `1`, `false`, and `0`. This can be useful for implementing a readonly cache for pull requests, which typically cannot be given access to credentials for security reasons.
+
+Set `SCCACHE_OSS_RW_MODE=READ_ONLY` with credentialless mode. It requires public
+object-read access but does not require public bucket-list access. Because the
+bucket cannot be probed independently in that mode, sccache uses the OSS
+object-read response to classify a missing cache key.

docs/S3.md

@@ -35,3 +35,8 @@ Sccache is able to load credentials from various sources. Including:
 - AssumeRoleWithWebIdentity: assume role with web webIdentity specified by `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`.
 
 Alternatively, the `SCCACHE_S3_NO_CREDENTIALS` environment variable can be set to use public readonly access to the S3 bucket, without the need for credentials. Valid values for this environment variable are `true`, `1`, `false`, and `0`. This can be useful for implementing a readonly cache for pull requests, which typically cannot be given access to credentials for security reasons.
+
+Set `SCCACHE_S3_RW_MODE=READ_ONLY` with credentialless mode. It requires public
+object-read access but does not require public bucket-list access. Because the
+bucket cannot be probed independently in that mode, sccache uses the S3
+object-read response to classify a missing cache key.