Conversation
Squashed and rebased from PR #284 (oci-store) onto master. Adds an OCIStore backend using oras-go that stores chunks as blobs in an OCI registry, addressable via oci+https:// and oci+http:// store URLs, with registry credentials configurable via oci-credentials in the config file. Requires --digest=sha256 since OCI identifies blobs by SHA256.
- Fix plain-HTTP detection: the scheme check looked for a '-http' suffix but the scheme is 'oci+http', so PlainHTTP was never enabled - Enable the auth token cache. Without it, every request replayed the full 401 challenge and token fetch - Fetch blobs with a single GET (FetchReference) instead of a HEAD followed by a GET - Make the store explicitly uncompressed-only: registries verify that uploaded content matches the declared digest (the chunk ID, a SHA256 of the plain data), so compression converters can never apply - Add tests covering the digest requirement, scheme detection, and a store/get/has roundtrip against a minimal in-process registry
Store every chunk as its own OCI artifact: a blob holding the chunk in storage format (compressed by default now, converters are honored), referenced by a minimal image manifest with artifact type application/vnd.desync.chunk.v1 that is tagged with the chunk ID hex. The chunk ID only appears in the tag, decoupling it from the OCI blob digest. That removes the --digest=sha256 requirement, so OCI stores now work with the default SHA512/256 algorithm and compose with other stores and existing indexes. The tagged manifest also keeps the blob referenced, protecting chunks from registry garbage collection of unreferenced blobs, which some registries (e.g. GHCR) additionally won't even serve. Reads fetch the manifest by tag and then the blob (2 requests), existence checks are a single HEAD on the tag, writes push the blob followed by the manifest (3 requests). RemoveChunk deletes only the manifest, leaving blob cleanup to the registry's garbage collection. Also wire up the remaining store options (timeout, error-retry with linear backoff, ca-cert, client-cert/key) matching the semantics of the HTTP store. Tested against a real distribution registry: make/extract/info round trips with the default digest algorithm, byte-identical extraction, and cache composition.
Resolve OCI registry credentials in order: DESYNC_OCI_USERNAME and DESYNC_OCI_PASSWORD environment variables, oci-credentials config entries (now supporting glob patterns in keys, erroring when multiple entries match), and finally the Docker credential store so registries logged into with 'docker login' or 'oras login' work without any desync configuration. Update the README for the new store layout: no more digest algorithm restriction, compression support, credential lookup order, an OCI column in the store capabilities table, and the new environment variables.
Iterate the repository's tags and delete the manifests of chunks not in the keep list. Only tags that parse as chunk IDs are considered, so other artifacts can share the repository. Reclaiming the space of the unreferenced blobs is left to the registry's garbage collection. Manifest deletion isn't supported by every registry: the distribution registry needs REGISTRY_STORAGE_DELETE_ENABLED=true and ghcr.io only deletes through the GitHub Packages API. Documented in the README. Verified against a distribution registry with deletes enabled: pruning to a subset index removed exactly the unreferenced chunk tags, the kept index still extracts, and extraction of the pruned index reports the chunks as missing.
Describe the on-registry artifact layout with a real manifest example, the properties that follow from it (default digest algorithm support, GC safety, blob dedup), the credential lookup order with a note on ghcr.io token scopes, applicable store options, prune behavior and registry support caveats, and usage examples covering publishing, extraction with a cache, CI credentials, update estimation, pruning, and a local test registry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for using OCI container registries (ghcr.io, Docker Hub, Harbor, zot, distribution, ...) as chunk stores, addressing #253. Registries are widely available, often free or low cost, and typically CDN-backed, making them an attractive way to distribute chunks publicly without running any server infrastructure.
A registry store is used like any other store with the new
oci+https(oroci+http) scheme and supports reading, writing, pruning, and use as a cache:Design
Every chunk is stored as its own OCI artifact: a blob holding the chunk data (zstd-compressed unless the store is configured with
uncompressed), referenced by a minimal image manifest with artifact typeapplication/vnd.desync.chunk.v1that is tagged with the chunk ID in hex.The chunk ID appearing only in the tag — never as a blob digest — is the key design point, and differs from the earlier draft of this PR which stored bare blobs whose digest was the chunk ID. The rework has these consequences:
--digest=sha256and made OCI stores incompatible with every store and index using desync's default SHA512/256. With the chunk ID in the tag, both algorithms work and OCI stores compose freely with other stores and caches.The cost is one extra round trip per read (manifest by tag, then blob; writes take three requests) and one tag per chunk in registry UIs, so a dedicated repository is recommended. Unrelated artifacts sharing the repository are safe, including from
prune, which only touches tags that parse as chunk IDs.prunedeletes the manifests of unreferenced chunks; reclaiming blob space is left to the registry's own garbage collection. Manifest deletion must be supported and enabled on the registry (distribution:REGISTRY_STORAGE_DELETE_ENABLED=true; ghcr.io only deletes via the GitHub Packages API).Authentication
Credentials are resolved in order:
DESYNC_OCI_USERNAME/DESYNC_OCI_PASSWORDenvironment variables (convenient in CI), the newoci-credentialsconfig section (keyed by host/repository, glob patterns supported), and finally the Docker credential store — registries logged into withdocker loginororas loginwork with zero desync configuration. Anonymous access works for public repositories.The standard store options (
uncompressed,timeout,error-retry,error-retry-base-interval,trust-insecure,ca-cert,client-cert/client-key) are all wired up.Trying it out locally
No credentials or external services needed — a throwaway registry container is enough:
(
dockerworks the same aspodmanhere.)Testing
registry:2):make/extract/info/pruneround trips with the default digest algorithm, byte-identical extraction, cache composition, and prune removing exactly the unreferenced chunks while other indexes keep extracting.Not included:
verifysupport (the command is currently local-store only, and verifying a registry store means downloading it in full) — possible follow-up if there's demand.Closes #253