Support importing zipped datasets from stdin#699
Open
hartblanc wants to merge 522 commits into
Open
Conversation
Wrap paths with quotes during exec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support importing zipped datasets from standard input (
stdin)Problem / Motivation
Currently, mapshaper supports importing zipped datasets (such as
.zipcontaining shapefiles or geojson) from local file paths, but it does not support them when read fromstdin(e.g., when usingcat data.zip | mapshaper -i -).Because
stdindoes not have a file extension, mapshaper falls back to reading the stream as a UTF-8 string to guess its content type (checking if it looks like JSON, KML, or SVG). When a binary ZIP stream is piped in, it fails to match these text-based formats, triggering a binary safety check (detecting the Unicode replacement character\ufffd) and halting with an "unknown file type" error.This limitation prevents piping multi-file dataset formats (specifically Shapefiles, which require
.shp,.dbf,.shxfiles to be imported together) directly through a singlestdinstream without writing intermediate files to disk first. The ability to pipe files from stdin is useful in build environments where access to the file system is limited or simply to avoid writing temporary files to disk that need to be cleaned up afterwards.Proposed Changes
This PR adds support for piping zipped datasets from
stdin._importFileand_importFileAsyncinsrc/io/mapshaper-file-import.mjsto read the input stream as raw binary bytes first.PK\x03\x04(50 4B 03 04).importFilesTogether).utf-8) encoding, and normal text-content guessing continues.Test Coverage
test/zip-test.mjsto cover both the asynchronous path (used by the CLI commands) and the synchronous path:zipped shapefile from stdin(async command invocation)zipped GeoJSON from stdin(async command invocation)synchronous zipped shapefile import(direct synchronousimportFilecall)npm testandnpx mocha test/zip-test.mjs).