Make CI artifacts reproducible#317601
Draft
hediet wants to merge 6 commits into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to make CI-produced distribution artifacts byte-reproducible across runs of the same commit by normalizing archive creation (ordering, ownership, timestamps) and by using the HEAD commit time as the canonical build timestamp throughout packaging steps.
Changes:
- Introduces
SOURCE_DATE_EPOCHhelpers and uses commit time instead of wall-clock time for several build metadata fields. - Routes
.tar.gz/.ziparchive creation in CI through new “reproducible” wrapper scripts. - Makes built-in extension scanning deterministic by sorting
readdirresults.
Show a summary per file
| File | Description |
|---|---|
| build/lib/sourceDateEpoch.ts | New helper to compute SOURCE_DATE_EPOCH (env override → git HEAD committer time → wall clock fallback). |
| build/lib/policies/render.ts | Pins macOS policy manifest pfm_last_modified to the source date instead of new Date(). |
| build/lib/extensions.ts | Sorts built-in extension directory traversal for deterministic scanning output. |
| build/lib/electron.ts | Pins darwinCredits date to the source date instead of wall-clock time. |
| build/gulpfile.vscode.linux.ts | Uses commit-time-based revision and passes SOURCE_DATE_EPOCH into dpkg-deb and rpmbuild. |
| build/azure-pipelines/win32/codesign.ts | Switches Windows zipping to the new reproducible PowerShell helper. |
| build/azure-pipelines/web/product-build-web.yml | Switches web tarball creation to reproducible-tar.sh. |
| build/azure-pipelines/linux/steps/product-build-linux-compile.yml | Switches Linux client/server tarballs to reproducible-tar.sh. |
| build/azure-pipelines/linux/build-snap.sh | Pins Snap BUILD_VERSION to the HEAD commit time. |
| build/azure-pipelines/darwin/steps/product-build-darwin-compile.yml | Switches macOS zipping to reproducible-zip.sh. |
| build/azure-pipelines/darwin/product-build-darwin-universal.yml | Switches universal macOS zipping to reproducible-zip.sh. |
| build/azure-pipelines/common/reproducible-zip.sh | New helper to pre-touch entries and zip from a sorted file list. |
| build/azure-pipelines/common/reproducible-zip.ps1 | New helper to pre-stamp timestamps before zipping with 7-Zip on Windows. |
| build/azure-pipelines/common/reproducible-tar.sh | New helper to create normalized tar.gz archives (sorted entries, fixed mtime/owner/group, reproducible gzip header). |
| build/azure-pipelines/alpine/product-build-alpine.yml | Switches Alpine tarball creation to reproducible-tar.sh. |
Copilot's findings
- Files reviewed: 15/15 changed files
- Comments generated: 4
Comment on lines
23
to
27
| const root = path.dirname(import.meta.dirname); | ||
| const commit = getVersion(root); | ||
|
|
||
| const linuxPackageRevision = Math.floor(new Date().getTime() / 1000); | ||
| const linuxPackageRevision = getSourceDateEpoch(); | ||
|
|
Comment on lines
+22
to
+25
| try { | ||
| const cwd = path.dirname(path.dirname(import.meta.dirname)); | ||
| const out = cp.execSync('git log -1 --pretty=%ct', { cwd, stdio: ['ignore', 'pipe', 'ignore'] }).toString().trim(); | ||
| const parsed = parseInt(out, 10); |
Comment on lines
+33
to
+37
| TOUCH_DATE=$(date -r "$SOURCE_DATE_EPOCH" "+%Y%m%d%H%M.%S") | ||
|
|
||
| ( | ||
| cd "$SOURCE_DIR" | ||
| find . -exec touch -h -t "$TOUCH_DATE" {} + |
Comment on lines
+35
to
+39
| ( | ||
| cd "$SOURCE_DIR" | ||
| find . -exec touch -h -t "$TOUCH_DATE" {} + | ||
| find . -print | LC_ALL=C sort | zip -X -y "$ARCHIVE_PATH" -@ | ||
| ) |
- Add build/lib/sourceDateEpoch.ts helper that derives the timestamp from the HEAD commit time (overridable via SOURCE_DATE_EPOCH) - Add reproducible-tar.sh / reproducible-zip.sh / reproducible-zip.ps1 shared helpers and route the 6 tar + 5 darwin zip + 3 win32 7z invocations through them - Sort readdir output in scanBuiltinExtensions so the built-in extensions list is deterministic - Use the commit time for linuxPackageRevision, the macOS Credits.rtf date, the macOS policy plist date, and the snap BUILD_VERSION - Pass SOURCE_DATE_EPOCH to dpkg-deb and rpmbuild
hediet
force-pushed
the
hediet/b/low-lobster
branch
from
2f243e3 to
531bfb3
Compare
reproducible-zip.sh now takes <cwd> <pattern>; client zips pass '*' (flat), server zips pass the dir name (wrapped) - matching the pre-refactor layout per call site.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP. Routes archive creation through new shared helpers (reproducible-tar.sh, reproducible-zip.sh, reproducible-zip.ps1) that pin entry order, ownership, and timestamps to the HEAD commit time; sorts readdir output in the built-in extensions scanner; pins linuxPackageRevision, Credits.rtf, the macOS policy plist date, and the snap BUILD_VERSION to the commit time; passes SOURCE_DATE_EPOCH to dpkg-deb and rpmbuild.
Driven by an audit of two CI runs of the same SHA producing byte-different artifacts.