Add ability to force tar compression (#277)

Author: mitchpaulus

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- The tar write functions (`tarDirInc`, `tarDirExc`, `tarPack`) now accept a
+  dictionary destination `{path: str|path, compress?: bool}`, where `compress`
+  overrides the extension-based gzip inference in either direction — useful
+  for destinations without a meaningful extension (e.g. `redo`'s `$3` temp files).
+
 - Stream merge redirects `2>&1` (stderr to stdout's destination) and `1>&2`
   (stdout to stderr's destination). Each is a single token with no internal
   spaces, and works on command lists, pipeline stages, and quotations.

doc/functions.inc.html

@@ -126,10 +126,10 @@ <h1 id="functions-file-directory">File and Directory <a class="section-link" hre
         <tr> <td><code>zipExtract</code></td> <td>Extract an entire archive. Options dict is required; defaults: <code>overwrite=false</code>, <code>skipExisting=false</code> (mutually exclusive), <code>stripComponents=0</code>, <code>pattern=""</code> (glob matched before stripping), <code>preservePermissions=true</code>, <code>maxBytes=0</code> (0 = unlimited; a cap on the total uncompressed bytes written, guarding against decompression bombs). Destination is created if missing.</td> <td><code>(<span class="sig-type sig-type-path">path</span>:zipPath <span class="sig-type sig-type-path">path</span>:destDir <span class="sig-type sig-type-dict">dict</span>:options -- )</code></td> </tr>
         <tr> <td><code>zipExtractEntry</code></td> <td>Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: <code>overwrite=false</code>, <code>skipExisting=false</code> (mutually exclusive), <code>preservePermissions=true</code>, <code>mkdirs=true</code> (create parent directories when needed), <code>maxBytes=0</code> (0 = unlimited uncompressed-byte cap).</td> <td><code>(<span class="sig-type sig-type-path">path</span>:zipPath <span class="sig-type sig-type-str">str</span>:entry <span class="sig-type sig-type-path">path</span>:dest <span class="sig-type sig-type-dict">dict</span>:options -- )</code></td> </tr>
         <tr> <td><code>zipRead</code></td> <td>Read an entry’s bytes directly into the stack without writing to disk. Returns <code>none</code> when the entry does not exist.</td> <td><code>(<span class="sig-type sig-type-path">path</span>:zipPath <span class="sig-type sig-type-str">str</span>:entry -- <span class="sig-type sig-type-maybe">Maybe</span>[<span class="sig-type sig-type-binary">binary</span>])</code></td> </tr>
-        <tr> <td colspan="3"><em>Tar functions mirror the <code>zip*</code> surface (same argument order and option dicts). Compression is chosen from the destination extension when writing (<code>.tar.gz</code> / <code>.tgz</code> → gzip, <code>.tar</code> → uncompressed) and auto-detected from the gzip magic bytes when reading. Symlinks are preserved: packed as symlink entries and recreated on extraction (with an escape guard); hard links and device nodes are rejected.</em></td> </tr>
-        <tr> <td><code>tarDirInc</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> from a directory; the archive root contains the directory’s contents (no parent folder).</td> <td><code>(<span class="sig-type sig-type-path">path</span>:sourceDir <span class="sig-type sig-type-path">path</span>:tarPath -- )</code></td> </tr>
-        <tr> <td><code>tarDirExc</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> that includes the source directory itself at the archive root (entries are prefixed with the directory name).</td> <td><code>(<span class="sig-type sig-type-path">path</span>:sourceDir <span class="sig-type sig-type-path">path</span>:tarPath -- )</code></td> </tr>
-        <tr> <td><code>tarPack</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> by packing a list of entries. Each entry is either a bare string/path (the file or directory to add, keeping its base name and mode) or a dictionary requiring <code>path</code>; in the dictionary form <code>archivePath</code> (override the in-archive name) and <code>mode</code> are optional. <code>mode</code> is a Go <a href="https://pkg.go.dev/io/fs#FileMode" target="_blank" rel="noopener noreferrer"><code>os.FileMode</code></a>; write it with an octal literal, e.g. <code>0o644</code> (<code>rw-r--r--</code>), <code>0o755</code> (<code>rwxr-xr-x</code>), <code>0o600</code>. If <code>mode</code> is omitted, the entry keeps the source file’s own mode.</td> <td><code>([<span class="sig-type sig-type-str">str</span>|<span class="sig-type sig-type-path">path</span>|<span class="sig-type sig-type-dict">dict</span>] <span class="sig-type sig-type-path">path</span>:tarPath -- )</code></td> </tr>
+        <tr> <td colspan="3"><em>Tar functions mirror the <code>zip*</code> surface (same argument order and option dicts). Compression is chosen from the destination extension when writing (<code>.tar.gz</code> / <code>.tgz</code> → gzip, <code>.tar</code> → uncompressed) and auto-detected from the gzip magic bytes when reading. The write destination may also be a dictionary <code>{path: str|path, compress?: bool}</code>, where <code>compress</code> overrides the extension inference in either direction — useful for destinations without a meaningful extension (e.g. <code>redo</code>’s <code>$3</code> temp files). Symlinks are preserved: packed as symlink entries and recreated on extraction (with an escape guard); hard links and device nodes are rejected.</em></td> </tr>
+        <tr> <td><code>tarDirInc</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> from a directory; the archive root contains the directory’s contents (no parent folder). The destination may be a dictionary <code>{path, compress?}</code> to force compression on or off.</td> <td><code>(<span class="sig-type sig-type-path">path</span>:sourceDir <span class="sig-type sig-type-path">path</span>|<span class="sig-type sig-type-dict">dict</span>:dest -- )</code></td> </tr>
+        <tr> <td><code>tarDirExc</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> that includes the source directory itself at the archive root (entries are prefixed with the directory name). The destination may be a dictionary <code>{path, compress?}</code> to force compression on or off.</td> <td><code>(<span class="sig-type sig-type-path">path</span>:sourceDir <span class="sig-type sig-type-path">path</span>|<span class="sig-type sig-type-dict">dict</span>:dest -- )</code></td> </tr>
+        <tr> <td><code>tarPack</code></td> <td>Create/overwrite a <code>.tar</code> / <code>.tar.gz</code> by packing a list of entries. Each entry is either a bare string/path (the file or directory to add, keeping its base name and mode) or a dictionary requiring <code>path</code>; in the dictionary form <code>archivePath</code> (override the in-archive name) and <code>mode</code> are optional. <code>mode</code> is a Go <a href="https://pkg.go.dev/io/fs#FileMode" target="_blank" rel="noopener noreferrer"><code>os.FileMode</code></a>; write it with an octal literal, e.g. <code>0o644</code> (<code>rw-r--r--</code>), <code>0o755</code> (<code>rwxr-xr-x</code>), <code>0o600</code>. If <code>mode</code> is omitted, the entry keeps the source file’s own mode. The destination may be a dictionary <code>{path, compress?}</code> to force compression on or off.</td> <td><code>([<span class="sig-type sig-type-str">str</span>|<span class="sig-type sig-type-path">path</span>|<span class="sig-type sig-type-dict">dict</span>] <span class="sig-type sig-type-path">path</span>|<span class="sig-type sig-type-dict">dict</span>:dest -- )</code></td> </tr>
         <tr> <td><code>tarList</code></td> <td>List archive entries as dictionaries with keys: <code>name</code> (string, forward-slash paths, directories end with <code>/</code>), <code>compressedSize</code> and <code>uncompressedSize</code> (int bytes; equal, since tar has no per-entry compressed size), <code>isDir</code> (bool), <code>perm</code> (int POSIX permission bits), <code>executable</code> (bool), <code>modified</code> (datetime), <code>type</code> (<code>"file"</code>/<code>"dir"</code>/<code>"symlink"</code>), and <code>linkTarget</code> (symlink target, empty otherwise).</td> <td><code>(<span class="sig-type sig-type-path">path</span> -- [<span class="sig-type sig-type-dict">dict</span>])</code></td> </tr>
         <tr> <td><code>tarExtract</code></td> <td>Extract an entire archive. Options dict is required; defaults: <code>overwrite=false</code>, <code>skipExisting=false</code> (mutually exclusive), <code>stripComponents=0</code>, <code>pattern=""</code> (glob matched before stripping), <code>preservePermissions=true</code>, <code>maxBytes=0</code> (0 = unlimited; a cap on the total uncompressed bytes written, guarding against decompression bombs). Destination is created if missing.</td> <td><code>(<span class="sig-type sig-type-path">path</span>:tarPath <span class="sig-type sig-type-path">path</span>:destDir <span class="sig-type sig-type-dict">dict</span>:options -- )</code></td> </tr>
         <tr> <td><code>tarExtractEntry</code></td> <td>Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: <code>overwrite=false</code>, <code>skipExisting=false</code> (mutually exclusive), <code>preservePermissions=true</code>, <code>mkdirs=true</code> (create parent directories when needed), <code>maxBytes=0</code> (0 = unlimited uncompressed-byte cap).</td> <td><code>(<span class="sig-type sig-type-path">path</span>:tarPath <span class="sig-type sig-type-str">str</span>:entry <span class="sig-type sig-type-path">path</span>:dest <span class="sig-type sig-type-dict">dict</span>:options -- )</code></td> </tr>

doc/mshell.md

@@ -1484,6 +1484,11 @@ Two things differ, both driven by the tar format:
   `.tar.gz` or `.tgz` produce a gzip-compressed tarball, `.tar` is uncompressed.
   When reading, the gzip magic bytes are auto-detected, so a gzipped tarball is
   read transparently regardless of its filename.
+  The write destination (`tarDirInc`/`tarDirExc`/`tarPack`) may also be a dictionary
+  `{path: str|path, compress?: bool}`, where `compress` overrides the extension
+  inference in either direction.
+  This is useful for destinations without a meaningful extension,
+  e.g. `redo`'s `$3` temp files: `` `src` { "path": @dest, "compress": true } tarDirExc ``.
 - Symlinks are preserved: `tarPack`/`tarDir*` store symlinks as symlink entries,
   and `tarExtract`/`tarExtractEntry` recreate them (rejecting any whose target
   would escape the destination directory). Extraction also refuses to write
@@ -1495,8 +1500,8 @@ bytes; `0` = unlimited) to guard against decompression bombs, and packing never
 follows a source symlink, so a symlink loop or a link to `/dev/zero` cannot hang
 or inflate the archive.
 
-- `tarDirInc`: Create/overwrite a `.tar`/`.tar.gz` from a directory; the archive root contains the directory's contents (no parent folder). `(path:sourceDir path:tarPath -- )`
-- `tarDirExc`: Create/overwrite a `.tar`/`.tar.gz` that includes the source directory itself at the archive root (entries are prefixed with the directory name). `(path:sourceDir path:tarPath -- )`
+- `tarDirInc`: Create/overwrite a `.tar`/`.tar.gz` from a directory; the archive root contains the directory's contents (no parent folder). `(str | path str | path | {path: str | path, compress?: bool} -- )`
+- `tarDirExc`: Create/overwrite a `.tar`/`.tar.gz` that includes the source directory itself at the archive root (entries are prefixed with the directory name). `(str | path str | path | {path: str | path, compress?: bool} -- )`
 - `tarPack`: Create/overwrite a `.tar`/`.tar.gz` by packing a list of entries.
   Each entry is either a bare string/path (the file or directory to add,
   keeping its base name and mode) or a dictionary.
@@ -1505,7 +1510,7 @@ or inflate the archive.
   `mode` is a Go `os.FileMode`; write it with an octal literal,
   e.g. `0o644` (`rw-r--r--`), `0o755` (`rwxr-xr-x`), `0o600`.
   If `mode` is omitted, the entry keeps the source file's own mode.
-  Type: `([str | path | {path: str | path, archivePath?: str | path, mode?: int}] str | path -- )`
+  Type: `([str | path | {path: str | path, archivePath?: str | path, mode?: int}] str | path | {path: str | path, compress?: bool} -- )`
 - `tarList`: List archive entries as dictionaries with keys: `name` (string, forward-slash paths, directories end with `/`), `compressedSize` and `uncompressedSize` (int bytes; equal, since tar has no per-entry compressed size), `isDir` (bool), `perm` (int POSIX permission bits), `executable` (bool), `modified` (datetime from the archive entry), `type` (`"file"`/`"dir"`/`"symlink"`), and `linkTarget` (symlink target, empty otherwise). `(path -- [dict])`
 - `tarExtract`: Extract an entire archive. Options dict is required; defaults: `overwrite=false`, `skipExisting=false` (mutually exclusive), `stripComponents=0`, `pattern=""` (glob matched before stripping), `preservePermissions=true`, `maxBytes=0` (0 = unlimited; caps the total uncompressed bytes written to guard against decompression bombs). Destination is created if missing. `(path:tarPath path:destDir dict:options -- )`
 - `tarExtractEntry`: Extract a single entry (file or directory subtree) to a destination path. Options dict is required; defaults: `overwrite=false`, `skipExisting=false` (mutually exclusive), `preservePermissions=true`, `mkdirs=true`, `maxBytes=0` (0 = unlimited uncompressed-byte cap). `(path:tarPath str:entry path:dest dict:options -- )`

mshell/Evaluator.go

@@ -5279,6 +5279,36 @@ func parseZipExtractEntryOptions(dict *MShellDict) (zipExtractEntryOptions, erro
 	return options, nil
 }
 
+// parseTarDestination interprets the destination argument of the tar write
+// builtins. A plain string/path infers gzip compression from the extension
+// (isGzipTarget); a dict form {path, compress?} makes the choice explicit,
+// overriding the extension in either direction.
+func parseTarDestination(obj MShellObject) (string, bool, error) {
+	if dict, ok := obj.(*MShellDict); ok {
+		pathObj, ok := dict.Items["path"]
+		if !ok {
+			return "", false, fmt.Errorf("destination dict is missing required 'path'")
+		}
+		tarPath, err := pathObj.CastString()
+		if err != nil {
+			return "", false, fmt.Errorf("destination 'path' must be a string or path, found %s", pathObj.TypeName())
+		}
+		compress := isGzipTarget(tarPath)
+		if val, ok, err := boolOption(dict, "compress"); err != nil {
+			return "", false, err
+		} else if ok {
+			compress = val
+		}
+		return tarPath, compress, nil
+	}
+
+	tarPath, err := obj.CastString()
+	if err != nil {
+		return "", false, fmt.Errorf("Cannot tar into a %s", obj.TypeName())
+	}
+	return tarPath, isGzipTarget(tarPath), nil
+}
+
 func boolOption(dict *MShellDict, key string) (bool, bool, error) {
 	item, ok := dict.Items[key]
 	if !ok {
@@ -7853,9 +7883,9 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu
 						return state.FailWithMessage(err.Error())
 					}
 
-					tarPath, err := obj1.CastString()
+					tarPath, compress, err := parseTarDestination(obj1)
 					if err != nil {
-						return state.FailWithMessage(fmt.Sprintf("%d:%d: Cannot tar into a %s.\n", t.Line, t.Column, obj1.TypeName()))
+						return state.FailWithMessage(fmt.Sprintf("%d:%d: %s: %s\n", t.Line, t.Column, t.Lexeme, err.Error()))
 					}
 
 					sourceDir, err := obj2.CastString()
@@ -7864,7 +7894,7 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu
 					}
 
 					preserveRoot := t.Lexeme == "tarDirInc"
-					if err := tarDirectory(sourceDir, tarPath, preserveRoot); err != nil {
+					if err := tarDirectory(sourceDir, tarPath, preserveRoot, compress); err != nil {
 						return state.FailWithMessage(fmt.Sprintf("%d:%d: %s\n", t.Line, t.Column, err.Error()))
 					}
 				} else if t.Lexeme == "tarPack" {
@@ -7873,9 +7903,9 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu
 						return state.FailWithMessage(err.Error())
 					}
 
-					tarPath, err := obj1.CastString()
+					tarPath, compress, err := parseTarDestination(obj1)
 					if err != nil {
-						return state.FailWithMessage(fmt.Sprintf("%d:%d: Cannot tar into a %s.\n", t.Line, t.Column, obj1.TypeName()))
+						return state.FailWithMessage(fmt.Sprintf("%d:%d: tarPack: %s\n", t.Line, t.Column, err.Error()))
 					}
 
 					list, ok := obj2.(*MShellList)
@@ -7928,7 +7958,7 @@ func (state *EvalState) evaluateToken(t Token, stack *MShellStack, context Execu
 						entries = append(entries, packItem)
 					}
 
-					if err := buildTarFromEntries(entries, tarPath); err != nil {
+					if err := buildTarFromEntries(entries, tarPath, compress); err != nil {
 						return state.FailWithMessage(fmt.Sprintf("%d:%d: %s\n", t.Line, t.Column, err.Error()))
 					}
 				} else if t.Lexeme == "tarList" {

mshell/Tar.go

@@ -73,10 +73,10 @@ func openTarReader(tarPath string) (*tar.Reader, io.Closer, error) {
 	return tar.NewReader(br), file, nil
 }
 
-// createTarWriter creates a tar archive for writing, gzip-compressing when the
-// destination extension calls for it. The returned finish function must be
-// called (not deferred with a discarded error) to flush and close every layer.
-func createTarWriter(tarPath string) (*tar.Writer, func() error, error) {
+// createTarWriter creates a tar archive for writing, gzip-compressing when
+// compress is true. The returned finish function must be called (not deferred
+// with a discarded error) to flush and close every layer.
+func createTarWriter(tarPath string, compress bool) (*tar.Writer, func() error, error) {
 	if err := os.MkdirAll(filepath.Dir(tarPath), 0755); err != nil {
 		return nil, nil, fmt.Errorf("Error creating parent directory for %s: %w", tarPath, err)
 	}
@@ -86,7 +86,7 @@ func createTarWriter(tarPath string) (*tar.Writer, func() error, error) {
 		return nil, nil, fmt.Errorf("Error creating %s: %w", tarPath, err)
 	}
 
-	if isGzipTarget(tarPath) {
+	if compress {
 		gz := gzip.NewWriter(output)
 		tw := tar.NewWriter(gz)
 		finish := func() error {
@@ -117,7 +117,7 @@ func createTarWriter(tarPath string) (*tar.Writer, func() error, error) {
 // tarDirectory packs a single directory into a tarball, mirroring zipDirectory.
 // preserveRoot controls whether the directory itself appears at the archive
 // root (tarDirExc) or only its contents (tarDirInc).
-func tarDirectory(sourceDir, tarPath string, preserveRoot bool) error {
+func tarDirectory(sourceDir, tarPath string, preserveRoot bool, compress bool) error {
 	info, err := os.Stat(sourceDir)
 	if err != nil {
 		return fmt.Errorf("Error stating %s: %w", sourceDir, err)
@@ -138,7 +138,7 @@ func tarDirectory(sourceDir, tarPath string, preserveRoot bool) error {
 		SourcePath:   sourceDir,
 		PreserveRoot: preserveRoot,
 	}
-	return buildTarFromEntries([]zipPackItem{packItem}, tarPath)
+	return buildTarFromEntries([]zipPackItem{packItem}, tarPath, compress)
 }
 
 func ensureTarTargetNotInsideSource(sourceAbs string, tarPath string) error {
@@ -155,7 +155,7 @@ func ensureTarTargetNotInsideSource(sourceAbs string, tarPath string) error {
 
 // buildTarFromEntries mirrors buildZipFromEntries, reusing the zipPackItem
 // model so the tarPack dispatch parsing is identical to zipPack.
-func buildTarFromEntries(items []zipPackItem, tarPath string) error {
+func buildTarFromEntries(items []zipPackItem, tarPath string, compress bool) error {
 	if len(items) == 0 {
 		return fmt.Errorf("tarPack requires at least one entry")
 	}
@@ -165,7 +165,7 @@ func buildTarFromEntries(items []zipPackItem, tarPath string) error {
 		return fmt.Errorf("Error resolving %s: %w", tarPath, err)
 	}
 
-	tw, finish, err := createTarWriter(tarPath)
+	tw, finish, err := createTarWriter(tarPath, compress)
 	if err != nil {
 		return err
 	}