Skip to content

Commit f979da7

Browse files
fix: remove ~48 transitive dependencies from asset packaging (#1654)
Whenever the CDK Toolkit packages file assets — Lambda function code, S3 file assets and the like — it builds a ZIP archive. That work was previously done by `archiver`, a dependency that brings roughly fifty transitive packages along with it into every install of the CDK CLI and `@aws-cdk/toolkit-lib`. This pull request replaces it with `yazl`, a focused, write-only ZIP library, which brings the footprint for this functionality down to just two packages (`yazl` and `buffer-crc32`). The reason for the change is customer experience. A smaller dependency graph means a faster `npm install` and a lighter footprint on disk and in CI for everyone who installs the CLI or builds on the toolkit library. It also shrinks the set of third-party code that has to be audited and patched when security advisories are published, which is a recurring cost for our users. Because the CDK only ever *creates* archives and never parses untrusted ones, a write-only library is all we need, so this also drops a large amount of reader and format-handling code that was never exercised in our use case. Asset packaging gets a little faster as a side effect. Both libraries hand compression to Node's native zlib, and in local benchmarks across representative asset trees `yazl` produced archives 8–11% faster than `archiver` at comparable memory use. The existing streaming behaviour is kept, so large assets are still written to disk without buffering the whole archive in memory. The change is otherwise invisible to users. Archives stay byte-for-byte deterministic — entry timestamps remain pinned to the 1980 epoch, so identical content continues to produce an identical hash — Unix file modes such as the executable bit are still preserved, and symbolic links are still followed. The existing unit tests for the zip tool pass unchanged. A full benchmark and security report is attached as a separate comment on this PR. ### Checklist - [ ] This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed - Release notes for the new version: --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 078ce70 commit f979da7

20 files changed

Lines changed: 2176 additions & 8126 deletions

File tree

.projenrc.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -716,8 +716,8 @@ const tools = defineTools({
716716
parent: repo,
717717
tools: {
718718
zip: {
719-
deps: ['archiver@^7.0.1', 'fast-glob@^3.3.3'],
720-
devDeps: ['@types/archiver', 'jszip'],
719+
deps: ['yazl@^3.3.1', 'fast-glob@^3.3.3'],
720+
devDeps: ['@types/yazl', 'jszip'],
721721
},
722722
},
723723
});
@@ -1223,7 +1223,7 @@ const cli = configureProject(
12231223
devDeps: [
12241224
yargsGen,
12251225
cliPluginContract,
1226-
'@types/archiver',
1226+
'@types/yazl',
12271227
'@types/fs-extra@^11',
12281228
'@types/mockery',
12291229
'@types/picomatch',
@@ -1252,7 +1252,7 @@ const cli = configureProject(
12521252
// Already bundled by the CLI: depend on the private tools package
12531253
// directly (the bundler inlines it and dedupes its transitive deps).
12541254
tools,
1255-
'archiver',
1255+
'yazl',
12561256
sdkDep('@aws-sdk/client-appsync'),
12571257
sdkDep('@aws-sdk/client-bedrock-agentcore-control'),
12581258
sdkDep('@aws-sdk/client-cloudformation'),

packages/@aws-cdk/cdk-assets-lib/.projen/deps.json

Lines changed: 5 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/cdk-assets-lib/package.json

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/integ-runner/THIRD_PARTY_LICENSES

Lines changed: 739 additions & 2567 deletions
Large diffs are not rendered by default.

packages/@aws-cdk/private-tools/.projen/deps.json

Lines changed: 8 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/private-tools/.projen/tasks.json

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/private-tools/lib/zip/index.ts

Lines changed: 22 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
import { createWriteStream, promises as fs } from 'fs';
22
import * as path from 'path';
3-
import { Writable } from 'stream';
43
import type { Options } from 'fast-glob';
54
import { globSync } from 'fast-glob';
5+
import { ZipFile } from 'yazl';
66

7-
// namespace object imports won't work in the bundle for function exports
8-
// eslint-disable-next-line @typescript-eslint/no-require-imports
9-
const archiver = require('archiver');
7+
/**
8+
* Fixed epoch used for every entry so that equal content yields an equal
9+
* zip (deterministic output, stable content hash).
10+
*/
11+
const EPOCH = new Date('1980-01-01T00:00:00.000Z');
1012

1113
/**
1214
* Receives informational messages (e.g. retries on EPERM on Windows).
@@ -49,21 +51,15 @@ export function zipString(fileName: string, rawString: string): Promise<Buffer>
4951
return new Promise((resolve, reject) => {
5052
const buffers: Buffer[] = [];
5153

52-
const converter = new Writable();
53-
converter._write = (chunk: Buffer, _: string, callback: () => void) => {
54-
buffers.push(chunk);
55-
process.nextTick(callback);
56-
};
57-
converter.on('finish', () => resolve(Buffer.concat(buffers)));
54+
const zip = new ZipFile();
55+
zip.outputStream.on('data', (chunk: Buffer) => buffers.push(chunk));
56+
zip.outputStream.on('error', reject);
57+
zip.outputStream.on('end', () => resolve(Buffer.concat(buffers)));
5858

59-
const archive = archiver('zip');
60-
archive.on('error', reject);
61-
archive.pipe(converter);
62-
archive.append(rawString, {
63-
name: fileName,
64-
date: new Date('1980-01-01T00:00:00.000Z'),
59+
zip.addBuffer(Buffer.from(rawString), fileName, {
60+
mtime: EPOCH, // reset date to get the same hash for the same content
6561
});
66-
void archive.finalize();
62+
zip.end();
6763
});
6864
}
6965

@@ -80,33 +76,29 @@ function writeZipFile(directory: string, outputFile: string): Promise<void> {
8076
};
8177
const files = globSync('**', globOptions); // The output here is already sorted
8278

83-
const output = createWriteStream(outputFile);
84-
85-
const archive = archiver('zip');
86-
archive.on('warning', fail);
87-
archive.on('error', fail);
79+
const zip = new ZipFile();
80+
zip.outputStream.on('error', fail);
8881

89-
// archive has been finalized and the output file descriptor has closed, resolve promise
90-
// this has to be done before calling `finalize` since the events may fire immediately after.
91-
// see https://www.npmjs.com/package/archiver
82+
const output = createWriteStream(outputFile);
83+
output.on('error', fail);
84+
// resolve once the output file descriptor has closed
9285
output.once('close', ok);
9386

94-
archive.pipe(output);
87+
zip.outputStream.pipe(output);
9588

9689
// Append files serially to ensure file order
9790
for (const file of files) {
9891
const fullPath = path.resolve(directory, file);
9992
// There are exactly 2 promises
10093
// eslint-disable-next-line @cdklabs/promiseall-no-unbounded-parallelism
10194
const [data, stat] = await Promise.all([fs.readFile(fullPath), fs.stat(fullPath)]);
102-
archive.append(data, {
103-
name: file,
104-
date: new Date('1980-01-01T00:00:00.000Z'), // reset dates to get the same hash for the same content
95+
zip.addBuffer(data, file, {
96+
mtime: EPOCH, // reset dates to get the same hash for the same content
10597
mode: stat.mode,
10698
});
10799
}
108100

109-
await archive.finalize();
101+
zip.end();
110102
});
111103
}
112104

packages/@aws-cdk/private-tools/package.json

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/private-tools/tsconfig.dev.json

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk/private-tools/tsconfig.json

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)