Skip to content

Support OCI registries for chunk storage#284

Draft
folbricht wants to merge 6 commits into
masterfrom
oci-store
Draft

Support OCI registries for chunk storage#284
folbricht wants to merge 6 commits into
masterfrom
oci-store

Conversation

@folbricht

@folbricht folbricht commented Apr 5, 2025

Copy link
Copy Markdown
Owner

Adds support for using OCI container registries (ghcr.io, Docker Hub, Harbor, zot, distribution, ...) as chunk stores, addressing #253. Registries are widely available, often free or low cost, and typically CDN-backed, making them an attractive way to distribute chunks publicly without running any server infrastructure.

A registry store is used like any other store with the new oci+https (or oci+http) scheme and supports reading, writing, pruning, and use as a cache:

desync make -s oci+https://ghcr.io/myuser/mystore file.iso.caibx file.iso
desync extract -s oci+https://ghcr.io/myuser/mystore -c /var/tmp/cache file.iso.caibx file.iso

Design

Every chunk is stored as its own OCI artifact: a blob holding the chunk data (zstd-compressed unless the store is configured with uncompressed), referenced by a minimal image manifest with artifact type application/vnd.desync.chunk.v1 that is tagged with the chunk ID in hex.

The chunk ID appearing only in the tag — never as a blob digest — is the key design point, and differs from the earlier draft of this PR which stored bare blobs whose digest was the chunk ID. The rework has these consequences:

  • Default digest algorithm works. OCI blob digests must be SHA256, which previously forced --digest=sha256 and made OCI stores incompatible with every store and index using desync's default SHA512/256. With the chunk ID in the tag, both algorithms work and OCI stores compose freely with other stores and caches.
  • Compression works. Registries verify that uploaded content matches the declared digest, so bare blobs could only ever hold uncompressed data. With the digest decoupled from the chunk ID, chunks are stored compressed by default like in every other store type.
  • Chunks are safe from garbage collection. Registries GC blobs that aren't referenced by a manifest, and some (ghcr.io) won't even serve unreferenced blobs — the earlier bare-blob approach didn't reliably work there at all. Tagged manifests keep every chunk referenced.

The cost is one extra round trip per read (manifest by tag, then blob; writes take three requests) and one tag per chunk in registry UIs, so a dedicated repository is recommended. Unrelated artifacts sharing the repository are safe, including from prune, which only touches tags that parse as chunk IDs.

prune deletes the manifests of unreferenced chunks; reclaiming blob space is left to the registry's own garbage collection. Manifest deletion must be supported and enabled on the registry (distribution: REGISTRY_STORAGE_DELETE_ENABLED=true; ghcr.io only deletes via the GitHub Packages API).

Authentication

Credentials are resolved in order: DESYNC_OCI_USERNAME/DESYNC_OCI_PASSWORD environment variables (convenient in CI), the new oci-credentials config section (keyed by host/repository, glob patterns supported), and finally the Docker credential store — registries logged into with docker login or oras login work with zero desync configuration. Anonymous access works for public repositories.

The standard store options (uncompressed, timeout, error-retry, error-retry-base-interval, trust-insecure, ca-cert, client-cert/client-key) are all wired up.

Trying it out locally

No credentials or external services needed — a throwaway registry container is enough:

# Start a local registry (with deletes enabled so prune works too)
podman run -d --rm --name registry -p 5000:5000 \
  -e REGISTRY_STORAGE_DELETE_ENABLED=true docker.io/library/registry:2

# Chunk a file and store the chunks in the registry
desync make -s oci+http://127.0.0.1:5000/test/store file.iso.caibx file.iso

# The chunks are visible as tagged artifacts in the registry
curl -s http://127.0.0.1:5000/v2/test/store/tags/list | jq .

# Reassemble the file from the registry
desync extract -s oci+http://127.0.0.1:5000/test/store file.iso.caibx restored.iso
cmp file.iso restored.iso

# Drop all chunks not referenced by the index
desync prune -s oci+http://127.0.0.1:5000/test/store file.iso.caibx

(docker works the same as podman here.)

Testing

  • Unit tests run against a minimal in-process OCI registry covering roundtrips with the default SHA512/256 digest, compressed and uncompressed stores, missing chunks, corrupted chunk detection, remove, prune (including foreign artifacts surviving), plain-HTTP scheme handling, and credential resolution.
  • Verified end-to-end against a real distribution registry (registry:2): make/extract/info/prune round trips with the default digest algorithm, byte-identical extraction, cache composition, and prune removing exactly the unreferenced chunks while other indexes keep extracting.

Not included: verify support (the command is currently local-store only, and verifying a registry store means downloading it in full) — possible follow-up if there's demand.

Closes #253

Squashed and rebased from PR #284 (oci-store) onto master. Adds an
OCIStore backend using oras-go that stores chunks as blobs in an OCI
registry, addressable via oci+https:// and oci+http:// store URLs,
with registry credentials configurable via oci-credentials in the
config file. Requires --digest=sha256 since OCI identifies blobs by
SHA256.
folbricht added 5 commits July 8, 2026 15:23
- Fix plain-HTTP detection: the scheme check looked for a '-http'
  suffix but the scheme is 'oci+http', so PlainHTTP was never enabled
- Enable the auth token cache. Without it, every request replayed the
  full 401 challenge and token fetch
- Fetch blobs with a single GET (FetchReference) instead of a HEAD
  followed by a GET
- Make the store explicitly uncompressed-only: registries verify that
  uploaded content matches the declared digest (the chunk ID, a SHA256
  of the plain data), so compression converters can never apply
- Add tests covering the digest requirement, scheme detection, and a
  store/get/has roundtrip against a minimal in-process registry
Store every chunk as its own OCI artifact: a blob holding the chunk in
storage format (compressed by default now, converters are honored),
referenced by a minimal image manifest with artifact type
application/vnd.desync.chunk.v1 that is tagged with the chunk ID hex.

The chunk ID only appears in the tag, decoupling it from the OCI blob
digest. That removes the --digest=sha256 requirement, so OCI stores now
work with the default SHA512/256 algorithm and compose with other
stores and existing indexes. The tagged manifest also keeps the blob
referenced, protecting chunks from registry garbage collection of
unreferenced blobs, which some registries (e.g. GHCR) additionally
won't even serve.

Reads fetch the manifest by tag and then the blob (2 requests),
existence checks are a single HEAD on the tag, writes push the blob
followed by the manifest (3 requests). RemoveChunk deletes only the
manifest, leaving blob cleanup to the registry's garbage collection.

Also wire up the remaining store options (timeout, error-retry with
linear backoff, ca-cert, client-cert/key) matching the semantics of
the HTTP store.

Tested against a real distribution registry: make/extract/info round
trips with the default digest algorithm, byte-identical extraction,
and cache composition.
Resolve OCI registry credentials in order: DESYNC_OCI_USERNAME and
DESYNC_OCI_PASSWORD environment variables, oci-credentials config
entries (now supporting glob patterns in keys, erroring when multiple
entries match), and finally the Docker credential store so registries
logged into with 'docker login' or 'oras login' work without any
desync configuration.

Update the README for the new store layout: no more digest algorithm
restriction, compression support, credential lookup order, an OCI
column in the store capabilities table, and the new environment
variables.
Iterate the repository's tags and delete the manifests of chunks not
in the keep list. Only tags that parse as chunk IDs are considered, so
other artifacts can share the repository. Reclaiming the space of the
unreferenced blobs is left to the registry's garbage collection.

Manifest deletion isn't supported by every registry: the distribution
registry needs REGISTRY_STORAGE_DELETE_ENABLED=true and ghcr.io only
deletes through the GitHub Packages API. Documented in the README.

Verified against a distribution registry with deletes enabled: pruning
to a subset index removed exactly the unreferenced chunk tags, the kept
index still extracts, and extraction of the pruned index reports the
chunks as missing.
Describe the on-registry artifact layout with a real manifest example,
the properties that follow from it (default digest algorithm support,
GC safety, blob dedup), the credential lookup order with a note on
ghcr.io token scopes, applicable store options, prune behavior and
registry support caveats, and usage examples covering publishing,
extraction with a cache, CI credentials, update estimation, pruning,
and a local test registry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

container registry as chunk store

1 participant