Skip to content

Commit fbd9941

Browse files
bmiddhabmiddha
committed
zipsync
1 parent afd2b72 commit fbd9941

44 files changed

Lines changed: 3109 additions & 107 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apps/zipsync/.npmignore‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# THIS IS A STANDARD TEMPLATE FOR .npmignore FILES IN THIS REPO.
2+
3+
# Ignore all files by default, to avoid accidentally publishing unintended files.
4+
*
5+
6+
# Use negative patterns to bring back the specific things we want to publish.
7+
!/bin/**
8+
!/lib/**
9+
!/lib-*/**
10+
!/dist/**
11+
12+
!CHANGELOG.md
13+
!CHANGELOG.json
14+
!heft-plugin.json
15+
!rush-plugin-manifest.json
16+
!ThirdPartyNotice.txt
17+
18+
# Ignore certain patterns that should not get published.
19+
/dist/*.stats.*
20+
/lib/**/test/
21+
/lib-*/**/test/
22+
*.test.js
23+
24+
# NOTE: These don't need to be specified, because NPM includes them automatically.
25+
#
26+
# package.json
27+
# README.md
28+
# LICENSE
29+
30+
# ---------------------------------------------------------------------------
31+
# DO NOT MODIFY ABOVE THIS LINE! Add any project-specific overrides below.
32+
# ---------------------------------------------------------------------------

‎apps/zipsync/LICENSE‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
@rushstack/zipsync
2+
3+
Copyright (c) Microsoft Corporation. All rights reserved.
4+
5+
MIT License
6+
7+
Permission is hereby granted, free of charge, to any person obtaining
8+
a copy of this software and associated documentation files (the
9+
"Software"), to deal in the Software without restriction, including
10+
without limitation the rights to use, copy, modify, merge, publish,
11+
distribute, sublicense, and/or sell copies of the Software, and to
12+
permit persons to whom the Software is furnished to do so, subject to
13+
the following conditions:
14+
15+
The above copyright notice and this permission notice shall be
16+
included in all copies or substantial portions of the Software.
17+
18+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
19+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
20+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
21+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
22+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
23+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
24+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎apps/zipsync/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# @rushstack/zipsync
2+
3+
zipsync is a tool to pack and unpack zip archives. It is designed as a single-purpose tool to pack and unpack build cache entries.
4+
5+
## Implementation
6+
7+
### Unpack
8+
9+
- Read the zip central directory record at the end of the zip file and enumerate zip entries
10+
- Parse the zipsync metadata file in the archive. This contains the SHA-1 hashes of the files
11+
- Enumerate the target directories, cleanup any files or folders that aren't in the archive
12+
- If a file exists with matching size + SHA‑1, skip writing; else unpack it
13+
14+
### Pack
15+
16+
- Enumerate the target directories.
17+
- For each file compute a SHA-1 hash for the zipsync metadata file, and the CRC32 (required by zip format), then compress it if needed. Write the headers and file contents to the zip archive.
18+
- Write the metadata file to the zip archive and the zip central directory record.
19+
20+
## Constraints
21+
22+
Though archives created by zipsync can be used by other zip compatible programs, the opposite is not the case. zipsync only implements a subset of zip features to achieve greater performance.

‎apps/zipsync/config/jest.config.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"extends": "local-node-rig/profiles/default/config/jest.config.json",
3+
"setupFilesAfterEnv": ["<rootDir>/config/jestSymbolDispose.js"]
4+
}

‎apps/zipsync/config/jestSymbolDispose.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
2+
// See LICENSE in the project root for license information.
3+
4+
const disposeSymbol = Symbol('Symbol.dispose');
5+
const asyncDisposeSymbol = Symbol('Symbol.asyncDispose');
6+
7+
Symbol.asyncDispose ??= asyncDisposeSymbol;
8+
Symbol.dispose ??= disposeSymbol;

‎apps/zipsync/config/rig.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
// The "rig.json" file directs tools to look for their config files in an external package.
3+
// Documentation for this system: https://www.npmjs.com/package/@rushstack/rig-package
4+
"$schema": "https://developer.microsoft.com/json-schemas/rig-package/rig.schema.json",
5+
6+
"rigPackageName": "local-node-rig"
7+
}

‎apps/zipsync/eslint.config.js‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
2+
// See LICENSE in the project root for license information.
3+
4+
const nodeTrustedToolProfile = require('local-node-rig/profiles/default/includes/eslint/flat/profile/node-trusted-tool');
5+
const friendlyLocalsMixin = require('local-node-rig/profiles/default/includes/eslint/flat/mixins/friendly-locals');
6+
7+
module.exports = [
8+
...nodeTrustedToolProfile,
9+
...friendlyLocalsMixin,
10+
{
11+
files: ['**/*.ts', '**/*.tsx'],
12+
languageOptions: {
13+
parserOptions: {
14+
tsconfigRootDir: __dirname
15+
}
16+
},
17+
rules: {
18+
'no-console': 'off'
19+
}
20+
}
21+
];

‎apps/zipsync/package.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"name": "@rushstack/zipsync",
3+
"version": "0.0.0",
4+
"description": "CLI tool for creating and extracting ZIP archives with intelligent filesystem synchronization",
5+
"repository": {
6+
"type": "git",
7+
"url": "https://github.com/microsoft/rushstack.git",
8+
"directory": "apps/zipsync"
9+
},
10+
"bin": {
11+
"zipsync": "./bin/zipsync"
12+
},
13+
"license": "MIT",
14+
"scripts": {
15+
"start": "node lib/start",
16+
"build": "heft build --clean",
17+
"_phase:build": "heft run --only build -- --clean",
18+
"_phase:test": "heft run --only test -- --clean"
19+
},
20+
"dependencies": {
21+
"@rushstack/node-core-library": "workspace:*",
22+
"@rushstack/terminal": "workspace:*",
23+
"@rushstack/ts-command-line": "workspace:*",
24+
"semver": "~7.5.4",
25+
"typescript": "~5.8.2",
26+
"@rushstack/lookup-by-path": "workspace:*"
27+
},
28+
"devDependencies": {
29+
"@rushstack/heft": "workspace:*",
30+
"@types/semver": "7.5.0",
31+
"eslint": "~9.25.1",
32+
"local-node-rig": "workspace:*"
33+
}
34+
}

‎apps/zipsync/src/ZipSyncCommandLineParser.ts‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
2+
// See LICENSE in the project root for license information.
3+
4+
import { CommandLineParser } from '@rushstack/ts-command-line/lib/providers/CommandLineParser';
5+
import type {
6+
CommandLineFlagParameter,
7+
IRequiredCommandLineStringParameter,
8+
IRequiredCommandLineChoiceParameter,
9+
IRequiredCommandLineStringListParameter,
10+
CommandLineChoiceParameter
11+
} from '@rushstack/ts-command-line/lib/index';
12+
import { InternalError } from '@rushstack/node-core-library/lib/InternalError';
13+
import { Colorize } from '@rushstack/terminal/lib/Colorize';
14+
import type { ConsoleTerminalProvider } from '@rushstack/terminal/lib/ConsoleTerminalProvider';
15+
import type { ITerminal } from '@rushstack/terminal/lib/ITerminal';
16+
17+
import { type IZipMode, zipSync } from './zipSync';
18+
19+
export class ZipSyncCommandLineParser extends CommandLineParser {
20+
private readonly _debugParameter: CommandLineFlagParameter;
21+
private readonly _verboseParameter: CommandLineFlagParameter;
22+
private readonly _modeParameter: IRequiredCommandLineChoiceParameter<IZipMode>;
23+
private readonly _archivePathParameter: IRequiredCommandLineStringParameter;
24+
private readonly _baseDirParameter: IRequiredCommandLineStringParameter;
25+
private readonly _targetDirectoriesParameter: IRequiredCommandLineStringListParameter;
26+
private readonly _compressionParameter: CommandLineChoiceParameter<'store' | 'deflate' | 'auto'>;
27+
private readonly _terminal: ITerminal;
28+
private readonly _terminalProvider: ConsoleTerminalProvider;
29+
30+
public constructor(terminalProvider: ConsoleTerminalProvider, terminal: ITerminal) {
31+
super({
32+
toolFilename: 'zipsync',
33+
toolDescription: ''
34+
});
35+
36+
this._terminal = terminal;
37+
this._terminalProvider = terminalProvider;
38+
39+
this._debugParameter = this.defineFlagParameter({
40+
parameterLongName: '--debug',
41+
parameterShortName: '-d',
42+
description: 'Show the full call stack if an error occurs while executing the tool'
43+
});
44+
45+
this._verboseParameter = this.defineFlagParameter({
46+
parameterLongName: '--verbose',
47+
parameterShortName: '-v',
48+
description: 'Show verbose output'
49+
});
50+
51+
this._modeParameter = this.defineChoiceParameter<IZipMode>({
52+
parameterLongName: '--mode',
53+
parameterShortName: '-m',
54+
description:
55+
'The mode of operation: "pack" to create a zip archive, or "unpack" to extract files from a zip archive',
56+
alternatives: ['pack', 'unpack'],
57+
required: true
58+
});
59+
60+
this._archivePathParameter = this.defineStringParameter({
61+
parameterLongName: '--archive-path',
62+
parameterShortName: '-a',
63+
description: 'Zip file path',
64+
argumentName: 'ARCHIVE_PATH',
65+
required: true
66+
});
67+
68+
this._targetDirectoriesParameter = this.defineStringListParameter({
69+
parameterLongName: '--target-directory',
70+
parameterShortName: '-t',
71+
description: 'Target directories to pack or unpack',
72+
argumentName: 'TARGET_DIRECTORIES',
73+
required: true
74+
});
75+
76+
this._baseDirParameter = this.defineStringParameter({
77+
parameterLongName: '--base-dir',
78+
parameterShortName: '-b',
79+
description: 'Base directory for relative paths within the archive',
80+
argumentName: 'BASE_DIR',
81+
required: true
82+
});
83+
84+
this._compressionParameter = this.defineChoiceParameter<'store' | 'deflate' | 'auto'>({
85+
parameterLongName: '--compression',
86+
parameterShortName: '-z',
87+
description:
88+
'Compression strategy when packing. "deflate" attempts DEFLATE for every file (keeps only if smaller); "auto" first skips likely-compressed types before attempting; "store" disables compression.',
89+
alternatives: ['store', 'deflate', 'auto'],
90+
required: true
91+
});
92+
}
93+
94+
protected override async onExecuteAsync(): Promise<void> {
95+
if (this._debugParameter.value) {
96+
InternalError.breakInDebugger = true;
97+
this._terminalProvider.debugEnabled = true;
98+
this._terminalProvider.verboseEnabled = true;
99+
}
100+
if (this._verboseParameter.value) {
101+
this._terminalProvider.verboseEnabled = true;
102+
}
103+
try {
104+
zipSync({
105+
terminal: this._terminal,
106+
mode: this._modeParameter.value,
107+
archivePath: this._archivePathParameter.value,
108+
targetDirectories: this._targetDirectoriesParameter.values,
109+
baseDir: this._baseDirParameter.value,
110+
compression: (this._compressionParameter.value as 'store' | 'deflate' | 'auto' | undefined) ?? 'auto'
111+
});
112+
} catch (error) {
113+
if (this._debugParameter.value) {
114+
console.error('\n' + error.stack);
115+
} else {
116+
console.error('\n' + Colorize.red('ERROR: ' + error.message.trim()));
117+
}
118+
}
119+
}
120+
}

‎apps/zipsync/src/__snapshots__/start.test.ts.snap‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
// Jest Snapshot v1, https://goo.gl/fbAQLP
2+
3+
exports[`CLI Tool Tests should display help for "zipsync --help" 1`] = `
4+
"
5+
zipsync 0.0.0 - https://rushstack.io
6+
7+
usage: zipsync [-h] [-d] [-v] -m {pack,unpack} -a ARCHIVE_PATH -t
8+
TARGET_DIRECTORIES -b BASE_DIR -z {store,deflate,auto}
9+
10+
11+
Optional arguments:
12+
-h, --help Show this help message and exit.
13+
-d, --debug Show the full call stack if an error occurs while
14+
executing the tool
15+
-v, --verbose Show verbose output
16+
-m {pack,unpack}, --mode {pack,unpack}
17+
The mode of operation: \\"pack\\" to create a zip archive,
18+
or \\"unpack\\" to extract files from a zip archive
19+
-a ARCHIVE_PATH, --archive-path ARCHIVE_PATH
20+
Zip file path
21+
-t TARGET_DIRECTORIES, --target-directory TARGET_DIRECTORIES
22+
Target directories to pack or unpack
23+
-b BASE_DIR, --base-dir BASE_DIR
24+
Base directory for relative paths within the archive
25+
-z {store,deflate,auto}, --compression {store,deflate,auto}
26+
Compression strategy when packing. \\"deflate\\" attempts
27+
DEFLATE for every file (keeps only if smaller);
28+
\\"auto\\" first skips likely-compressed types before
29+
attempting; \\"store\\" disables compression.
30+
31+
For detailed help about a specific command, use: zipsync <command> -h
32+
"
33+
`;

0 commit comments

Comments
 (0)