Skip to content

Support importing zipped datasets from stdin#699

Open
hartblanc wants to merge 522 commits into
mbloch:masterfrom
hartblanc:stdin-zip-import
Open

Support importing zipped datasets from stdin#699
hartblanc wants to merge 522 commits into
mbloch:masterfrom
hartblanc:stdin-zip-import

Conversation

@hartblanc

Copy link
Copy Markdown

Support importing zipped datasets from standard input (stdin)

Problem / Motivation

Currently, mapshaper supports importing zipped datasets (such as .zip containing shapefiles or geojson) from local file paths, but it does not support them when read from stdin (e.g., when using cat data.zip | mapshaper -i -).

Because stdin does not have a file extension, mapshaper falls back to reading the stream as a UTF-8 string to guess its content type (checking if it looks like JSON, KML, or SVG). When a binary ZIP stream is piped in, it fails to match these text-based formats, triggering a binary safety check (detecting the Unicode replacement character \ufffd) and halting with an "unknown file type" error.

This limitation prevents piping multi-file dataset formats (specifically Shapefiles, which require .shp, .dbf, .shx files to be imported together) directly through a single stdin stream without writing intermediate files to disk first. The ability to pipe files from stdin is useful in build environments where access to the file system is limited or simply to avoid writing temporary files to disk that need to be cleaned up afterwards.

Proposed Changes

This PR adds support for piping zipped datasets from stdin.

  1. Magic Bytes Check:
    • Modified the extension-less fallback branches in _importFile and _importFileAsync in src/io/mapshaper-file-import.mjs to read the input stream as raw binary bytes first.
    • Sniffs the first 4 bytes of the buffer for the standard ZIP signature PK\x03\x04 (50 4B 03 04).
  2. In-Memory Extraction:
    • If identified as a ZIP file, it extracts the archive's entries directly in memory to the options input cache.
    • Resolves primary files and recursively imports them (either individually or merged via importFilesTogether).
  3. Graceful Fallback:
    • If no ZIP signature is found, the raw buffer is decoded back to a string using the specified (or default utf-8) encoding, and normal text-content guessing continues.

Test Coverage

  • Added new integration tests to test/zip-test.mjs to cover both the asynchronous path (used by the CLI commands) and the synchronous path:
    • zipped shapefile from stdin (async command invocation)
    • zipped GeoJSON from stdin (async command invocation)
    • synchronous zipped shapefile import (direct synchronous importFile call)
    • All tests pass successfully (npm test and npx mocha test/zip-test.mjs).
    • I ran the built mapshaper bin against some shp.zip files piped to stdin in my own project hartblanc/anki-uk-geography and it all worked okay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants