envbuilder_cached_image outputs wrong digest — doesn't match what envbuilder actually pushes

[alirezamirsepassi]

I'm running into a digest mismatch between what envbuilder_cached_image computes and what envbuilder actually pushes to the cache registry.

Setup: envbuilder v1.3.0, provider v1.0.0, registry is CNCF Distribution v3 (registry:3), running on Kubernetes with Coder v2.30.3.

I have a template that uses the exists attribute to fall back to the builder image on cold cache:

image = envbuilder_cached_image.cached[0].exists ? envbuilder_cached_image.cached[0].image : local.devcontainer_builder_image

First run works fine — exists=false, falls back to builder image, envbuilder builds the devcontainer and pushes to cache with ENVBUILDER_PUSH_IMAGE=true. All good.

On the second run the probe finds all cached layers, returns exists=true, and outputs something like:

registry.example.com/coder/cache@sha256:4778c7ff...

But the manifest that envbuilder actually pushed to the registry has a different digest:

sha256:a4ccf717...

Kubernetes tries to pull the provider's digest → ImagePullBackOff because that manifest simply doesn't exist in the registry.

I dug into this and the issue is that runCacheProbe doesn't read the manifest from the registry. It assembles a new v1.Image from the cached layers and computes .Digest() on that. The resulting manifest bytes are different from what envbuilder pushed — so different SHA256.

I checked the actual manifests and the layers are identical. The difference is in the manifest serialization itself. For example, the pushed manifest has "mediaType":"application/vnd.oci.image.manifest.v1+json" but the Manifest struct in go-containerregistry tags that field with omitempty — so if the probe doesn't set it, the field gets dropped from the JSON and the hash changes completely. The config digest also likely differs (different timestamps/history between the probe's fake build and the real build).

I think the fix would be for the provider to either:

• Read the actual manifest back from the registry (e.g. fetch the latest tag) after confirming layers exist, instead of computing the digest from a locally-assembled image
• Or have envbuilder store the digest somewhere the provider can read it back

Happy to provide more details if needed. For now I've had to disable the provider and rely on ENVBUILDER_CACHE_REPO env var for runtime caching, which works but doesn't give the fast-startup benefit of pulling a pre-built image.