refactor: replacing modify_sources script with toml driven archive ov…#17816
Draft
Tonisal-byte wants to merge 1 commit into
Draft
refactor: replacing modify_sources script with toml driven archive ov…#17816Tonisal-byte wants to merge 1 commit into
Tonisal-byte wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors how scanner-flagged test fixtures are stripped from upstream source archives across nine components. It replaces the previous mechanism — a source-files entry with replace-upstream = true pointing at a pre-repacked tarball in the AZL lookaside (produced by a per-component modify_source.sh) — with file-remove / file-search-replace overlays that target paths inside the source archive (e.g. firefox-148.0.source.tar.xz/dom/base/crashtests/607222.html). The corresponding rendered specs and sources manifests are regenerated.
Changes:
- Convert 9 components (yara, rubygem-pdf-reader, libkml, libabigail, kf6-karchive, gdal, firefox, exfatprogs, espeak-ng, apache-commons-compress) from
replace-upstreamrepacked tarballs to in-tree archive overlays, plus accompanying source/spec edits (e.g. yaraMakefile.am, libkmlzip_file_test.cc, rubygemreader_spec_helper.rb). - Regenerate rendered
sources(now pointing at the unmodified upstream tarball hash) and bumpReleasefor each component. - Update the explanatory comments in each
comp.tomlto describe the overlay approach.
A few cross-cutting concerns: the rendered sources now reference the genuine upstream tarball (which still contains the AV-flagged content), so it must be verified that the produced SRPM no longer carries those files; three rendered specs contain azldev <azldev@local> "Local changes (uncommitted)" placeholder changelogs (a non-finalized render that will fail Check Rendered Specs); the orphaned modify_source.sh scripts and some stale comments remain. Note also that no locks/*.lock updates appear in this change set — overlay edits change the component fingerprint, so locks likely need refreshing for the Update Locks check.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| base/comps/firefox/firefox.comp.toml | Replace repacked-tarball source with two file-remove overlays; comment still references orphaned modify_source.sh/REMOVE_PATHS |
| base/comps/gdal/gdal.comp.toml | file-remove overlay; description regresses from AV/Trojan rationale to "reduce source size" |
| base/comps/libkml/libkml.comp.toml | file-remove + file-search-replace; brace-matching regex and its justifying comment are unsafe |
| base/comps/libabigail/libabigail.comp.toml | file-remove directory glob + existing patch overlay; residual "repacked Source0" wording |
| base/comps/kf6-karchive/kf6-karchive.comp.toml | Two file-remove overlays replacing repacked tarball |
| base/comps/exfatprogs/exfatprogs.comp.toml | tests/** file-remove overlay |
| base/comps/espeak-ng/espeak-ng.comp.toml | file-remove of PHP webshell demo fixture |
| base/comps/apache-commons-compress/apache-commons-compress.comp.toml | Four file-remove overlays |
| base/comps/yara/yara.comp.toml | Eight file-remove + four Makefile.am file-search-replace overlays |
| base/comps/rubygem-pdf-reader/rubygem-pdf-reader.comp.toml | 35 file-remove overlays + two Ruby helper file-search-replace overlays |
| specs/f/firefox/sources | Hash reverts to genuine upstream tarball (still contains flagged files) |
| specs/{e,l,k,g,a,r,y}/*/sources | Same upstream-hash revert across converted components |
| specs/y/yara/yara.spec, specs/e/exfatprogs/…, specs/a/apache-commons-compress/… | Release bump + non-finalized azldev@local changelog entries |
| specs/{l,k,g,r}//.spec | Release-tag bumps |
| SHA512 (cbindgen-vendor.tar.xz) = bea420e66bdd1c7c944655dd3e01abd6e7d6ac4b245c7ee190f31d800f7786f21e5cae11715b479bf795f4369d18c40dc12df19e0b643664f2f78e5c8a681415 | ||
| SHA512 (firefox-langpacks-148.0-20260223.tar.xz) = 7e5d283e1a83787984e63901c915f6672eae48c38c5fd64b9f8055f154d83be77f76cb77de2048f2ea263353313a90eefa54b7c173533bec2db4dddf32436302 | ||
| SHA512 (firefox-148.0.source.tar.xz) = c9c9f29fbd8f889bf3cf3d88776e1a62da7b2a65d386538d2bd048dd70caaaf8324adb5303a5fa9aa73c2cf6eb9f89cb4b34f9e67c4208d88636dd5376af90a9 | ||
| SHA512 (firefox-148.0.source.tar.xz) = b0e862091f3a07a02890f6414e77b433893364a8beaf522d440e97ed0060c9b14bdb2fffdecdf12dca849efce8c57d95a534b23e04259d83a96ee8f29e078349 |
| [[components.gdal.overlays]] | ||
| type = "file-remove" | ||
| file = "gdalautotest-3.11.5.tar.gz/gcore/data/zero_5GB_sozip_of_sozip.zip" | ||
| description = "Remove large test fixture to reduce source size" |
Comment on lines
+10
to
+16
| # The sed range `TEST_F(ZipFileTest, TestBadTooLarge) {` through the next | ||
| # column-0 `}` is safe because nested braces inside the test body are | ||
| # always indented. | ||
| [[components.libkml.overlays]] | ||
| type = "file-search-replace" | ||
| file = "libkml-1.3.0.tar.gz/tests/kml/base/zip_file_test.cc" | ||
| regex = 'TEST_F\(ZipFileTest, TestBadTooLarge\) \{[^}]*\}\n' |
Comment on lines
+146
to
+151
| # Archive overlays strip the scanner-flagged fixtures from upstream Source0 | ||
| # during source preparation, avoiding a separate modified-source tarball. | ||
| # | ||
| # When bumping the firefox version (or changing REMOVE_PATHS): | ||
| # 1. Edit VERSION and UPSTREAM_SHA512 in modify_source.sh. | ||
| # 2. Re-run the script and copy the new SHA-512 into the source-files entry. | ||
| # 3. Upload the new tarball to the modified-source lookaside (see script output). | ||
| # 1. Update the overlay paths below if upstream fixture paths change. | ||
| # 2. Re-render and build to verify the archive overlays still apply. |
Tonisal-byte
force-pushed
the
asalinas/replace-modify-source-script
branch
from
6919a49 to
1caf30d
Compare
📄❌ Rendered specs are out of dateFIX: — run this and commit the result: azldev component render -a --clean-staleOr download the fix patch and apply it: gh run download 28267373213 -R microsoft/azurelinux -n rendered-specs-patch
git apply rendered-specs.patch
Files to addThese files are produced by
Files to removeThese files are in your branch but are not produced by render. Remove them.
… and 57 more file(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.