fix(providers): manifest-based version source system#160
Merged
Conversation
Add internal/manifest package for managing runtime version manifests: - Manifest types with version/platform download info and SHA256 checksums - Source interface with FileSource and EmbeddedSource implementations - CachedSource wrapper with 24-hour TTL for future remote fetching - Platform key helpers matching Go's runtime.GOOS-GOARCH format - Empty placeholder manifests for python, node, ruby (to be populated) Add SHA256 verification to download package: - FileVerified() downloads with checksum verification - VerifyFile() and ComputeSHA256() utility functions Add JSON Schema for manifest files (schemas/manifest.schema.json) Part of #140
Add scripts/generate-manifests/ to fetch version data from upstream: - Node.js: fetches from nodejs.org with SHA256 checksums - Python/Ruby: placeholder implementations Add .github/workflows/update-manifests.yml: - Runs daily at 6 AM UTC or on manual trigger - Generates manifests and creates PR if changes detected Add npm tasks: - manifest:node, manifest:python, manifest:ruby, manifest:all Populate node.json with 701 versions across all supported platforms Part of #140
Add Go scripts to generate manifests for all runtime providers: - Node.js: Fetches from nodejs.org/dist with SHASUMS256.txt checksums - 703 versions across all platforms - Python: Fetches from python-build-standalone GitHub releases - Uses smaller page size (10) due to large asset counts per release - 83 unique Python versions from 95 dated releases - Ruby: Fetches from ruby-builder and RubyInstaller GitHub releases - ruby-builder provides macOS/Linux builds - RubyInstaller provides Windows builds - 98 versions with Windows support for recent versions Features: - Full GitHub API pagination for all releases - SHA256 checksums are optional (included when available) - Shared pagination utilities in main.go - npm tasks: manifest:node, manifest:python, manifest:ruby, manifest:all - GitHub Action workflow for automated updates
Update Node.js, Python, and Ruby providers to use embedded manifests for version listing and download URLs instead of fetching from upstream APIs directly. This ensures ListAvailable() and Install() use the same data source, eliminating version availability mismatches. Changes: - Node.js: Use manifest for getDownloadURL() and ListAvailable() - Python: Use manifest, remove getStandaloneBuildURL() helper - Ruby: Use manifest, remove platform-specific URL helpers - Add SortVersionsDesc() for semantic version sorting Resolves consistency issues where users could see versions they couldn't actually install (issue #140 Phase 2).
Add infrastructure for fetching manifests from remote server with graceful fallback to embedded manifests when offline or on error. New components: - HTTPSource: fetches manifests from manifests.dtvem.io - FallbackSource: tries primary source, falls back on failure - DefaultSource(): factory that wires cached remote + embedded fallback Flow: cache (24hr TTL) -> remote fetch -> embedded fallback Update all providers to use DefaultSource() instead of direct embedded access, enabling automatic updates when remote hosting is configured.
- Add `dtvem update` command to force refresh manifest cache - Add `dtvem request` command to request builds for unavailable versions - Update default.go with ForceRefreshRuntime and ListAvailableRuntimes - Add build_request.yml issue template for GitHub
Deploys manifest JSON files to manifests.dtvem.io when changes are merged to main. Uses S3-compatible API with Cloudflare R2. Required secrets: - CLOUDFLARE_ACCOUNT_ID - CLOUDFLARE_R2_ACCESS_KEY_ID - CLOUDFLARE_R2_SECRET_ACCESS_KEY - CLOUDFLARE_R2_MANIFESTS_BUCKET
Contributor
Author
|
/release-note DTVEM will now utilize runtime manifests for installation requests hosted from our own domain (manifests.dtvem.io). The installs themselves still come from the appropriate runtime provider sources, and we'll automatically verify a checksum/digest before installing if the upstream provider provides it to us. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a manifest-based architecture for runtime version management, solving the version source consistency issues described in #140.
Problem
ListAvailable()fetched versions from one sourceInstall()downloaded from a different sourceSolution
All runtimes now use curated manifest files that contain direct URLs to pre-built binaries with SHA256 checksums.
Changes
Manifest Infrastructure (
src/internal/manifest/)Manifest,Download,SourceinterfaceEmbeddedSource,HTTPSource,CachedSource,FallbackSourceGOOS-GOARCHto manifest platform keysProvider Updates
manifest.DefaultSource().GetManifest()ListAvailable()andInstall()now use the same data sourceNew Commands
dtvem update: Force refresh manifest cachedtvem request <runtime> <version>: Opens browser to request a build for unavailable versionsCI/CD
update-manifests.yml: Daily workflow to regenerate manifests from upstream and create PRdeploy-manifests.yml: Deploys manifests to Cloudflare R2 on merge to mainManifest Generation Scripts (
scripts/generate-manifests/)Testing
Required Setup
After merge, the following GitHub secrets need to be configured for manifest deployment:
CLOUDFLARE_ACCOUNT_IDCLOUDFLARE_R2_ACCESS_KEY_IDCLOUDFLARE_R2_SECRET_ACCESS_KEYCLOUDFLARE_R2_MANIFESTS_BUCKETCloses #140