Add no_compression flag support for release and package archives

[mdzhigarov]

This feature adds the ability to control compression for both individual package archives and the outer release archive.

Features

Package-level compression

Packages can specify no_compression: true in their spec file to disable compression for individual package tarballs.

Release-level compression

The no_compression flag can be set in final.yml to disable compression for the outer release tarball.

Implementation Details

• Added NoCompression field to package manifest schema
• Added NoCompression field to release manifest schema
• Added NoCompression() method to FSConfig to read from final.yml
• Added NoCompression() method to ReleaseDir interface
• Updated BuildRelease to respect no_compression from final.yml
• Updated archive factory to accept NoCompression parameter
• Updated archive writer to use release NoCompression() when compressing

Testing

The feature includes comprehensive unit tests covering:

• Reading no_compression from final.yml (FSConfig)
• Setting no_compression in release manifest (BuildRelease)
• Passing no_compression from package specs to archive factory
• Writing no_compression to release.MF file

Files Changed

• 27 files changed
• 505 insertions(+), 82 deletions(-)

[rkoster]

Should be updated to point to upstream bosh-util once other PR get's merged.

[Copilot]

Pull Request Overview

This PR prepares the BOSH CLI infrastructure for adding a --no-compression flag to the bosh create-release command. The changes introduce a compression option parameter throughout the codebase while maintaining backward compatibility by defaulting to compression enabled.

Key changes:

• Added noCompression parameter to ArchiveWriter and updated its constructor signature
• Updated all compressor method calls to accept CompressorOptions struct
• Added placeholder comments indicating where the flag will be exposed in the CLI

Reviewed Changes

Copilot reviewed 11 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cmd/opts/opts.go	Added commented-out NoCompression field to CreateReleaseOpts struct
cmd/cmd.go	Updated all NewArchiveWriter() calls to pass false for noCompression parameter
release/provider.go	Modified NewArchiveWriter() to accept noCompression boolean parameter
release/archive_writer.go	Added noCompression field and updated compression call to use CompressorOptions
release/archive_writer_test.go	Updated test to pass false for noCompression parameter
release/resource/archive.go	Updated CompressFilesInDir call to include empty CompressorOptions
installation/pkg/compiler.go	Updated CompressFilesInDir call to include empty CompressorOptions
installation/job_renderer.go	Updated CompressFilesInDir call to include empty CompressorOptions
templatescompiler/rendered_job_list_compressor.go	Updated CompressFilesInDir call to include empty CompressorOptions
stemcell/stemcell.go	Updated CompressSpecificFilesInDir call to include empty CompressorOptions
go.mod	Updated bosh-utils dependency from v0.0.555 to v0.0.557

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 11 out of 16 changed files in this pull request and generated 2 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[rkoster]

In general I don't like the ux of passing a flag, It would be better to be able to disable compression through the package spec file. This gives more fine-grained control and a deterministic result (always same result in depended of flags being passed).

[mdzhigarov]

In general I don't like the ux of passing a flag, It would be better to be able to disable compression through the package spec file. This gives more fine-grained control and a deterministic result (always same result in depended of flags being passed).

That does mean we'd need to change the packaging spec format (add new field) but also the release. MF manifest schema so that we have info about whether the whole tar should be compressed or not.

Do you agree? I would like that approach even better.

Is there a forum where I can discuss this stuff live?

[mdzhigarov]

I updated the PR. Now the implementation is much cleaner - the bosh create-release takes a --no-compression flag (currently commented out) which is used to control the compression of the final tar.
Each Package spec contains a no_compression: true/false property that can be used to control the compression of the package tars individually.

I tested this extensively with a mix of compressed and non-compressed options and it seems to produce the proper POSIX tars.

The PR is ready for review and potentially merging.

@rkoster would you please check this today? I need to get this merged so that I can update the Ops Manager dependency to bosh cli. Currently OpsMan uses bosh deploy and bosh upload-release to upload the releases onto the Director so it will benefit from the no_compression flag in the manifest because apparently to bosh cli re-archives the tar during upload-release/deploy.

[rkoster]

Yes there is a weekly Foundational Infrastructure Working groups meeting (details can be found here: https://github.com/cloudfoundry/community/blob/main/toc/working-groups/WORKING-GROUPS.md).

I don’t see why the release.MF would have to change, but yes the package spec would have to change, but this feels like a fairly small change an I’m suspecting that the spec parsing code lives closer to the package compression.

Actually thinking about it a bit more did you mean the release.MF so that no compression would be used when compiling these non compressed packages at a later point in time?

[mdzhigarov]

Here is the reasoning...

A typical boshrelease tar contains somewhat similar directory structure:

.
├── jobs
│   ├── cpu-monitor.tgz
│   ├── hubsm-install.tgz
│   ├── post-verify.tgz
...
├── packages
│   ├── helpers.tgz
│   ├── hubsm-cli.tgz
│   ├── imgpkg.tgz
...
└── release.MF

Even if we change the Packaging spec manifest to take no_compression param for each individual package - at the end of the day we have to tar the whole boshrelease.
Introducing the bosh create-release --no-compression flag would control this behavior - whether the parent tar is compressed or not. Disabling compression on the parent tar has significant performance benefits for larger tars (tp registry data, I'm looking at your :) )

The reason why we need to change the release.MF is because:
1] The code in the bosh cli which deals with compressing the parent tar is cleaner - it can simply read the release.MF manifest. This way I won't have to pass down a noCompressionOpt arg down the calling stack to the archive writer.
2] Inside the bosh agent it would be easier to decide whether to use compression once the boshrelease is compiled and re-archived back.

[rkoster]

@mdzhigarov during the FI working group meeting this PR was discussed and decided to put all related PRs back into draft state until the new spec based direction has been realised. The bosh utils change was also reverted because it was blocking dependency bumps. If the other PR's are ready for merge we can merge it all at the same time.

[mdzhigarov]

@rkoster can you please clarify what do you mean by "spec-based direction has been realised"?
What part of this is not currently realised?

Afaik, what we agreed on is:

• Having "no_compression" flag in the Package spec - This is already done.
• Havinh "--no-compression" flag in bosh create-release command and also in the release.MF manifest - This is also done.

There are currently no outstanding comments on my PR so I'm not sure what do you want me to fix.

[rkoster]

My bad, I was under the impression that you wanted to join the working group meeting because you wanted to discuss the implementation. I did not realize the work was already done. Will re-review this PR.

@aramprice mentioned during the meeting that it probably makes sense to have spec related logic also in the bosh-agent and maybe bosh director for compilation. And release tarball creation of compiled releases.

[Copilot]

Pull Request Overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mdzhigarov]

Addressed the code review comments:

• Added a new integration test with no_compression: true
• Changed the BuildRelease() function signature as suggested
• Adopted the Copilot improvement with var assignment

[rkoster]

It's getting closer! Thanks for adding the integration test. I'm just concerned about false positives on the current implementation of the package spec integration test.

Also it think it would be good to have the cli display a warning if a release author disables compression for a package but does not disable it in the final.yml because I can not think of a scenario where this would be the intended outcome.

[Copilot]

Pull Request Overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[beyhan]

LGTM. @mdzhigarov Thank you for the prompt reaction on the discussions and reviews.

[aramprice]

@mdzhigarov - thank you!