Skip to content

Commit 9852599

Browse files
build: add ijson as a runtime dependency
Adds the ijson streaming JSON parser as a direct dependency so connectors that ship inside the source-declarative-manifest base image can stream-parse very large JSON response bodies without materializing the full document in memory. Motivation: source-amazon-seller-partner currently OOMs while reading GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT documents that can exceed 3 GB uncompressed. See airbytehq/oncall#12143.
1 parent 19a7083 commit 9852599

2 files changed

Lines changed: 108 additions & 1 deletion

File tree

poetry.lock

Lines changed: 107 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ cryptography = ">=44.0.0,<45.0.0" # Constrained as transitive dependency due to
8484
pytz = "2024.2"
8585
pytest = {version = "^7", optional = true }
8686
orjson = "^3.10.7"
87+
ijson = "^3.3.0" # Streaming JSON parser, used by connectors with very large response bodies (e.g. Amazon Seller Partner Brand Analytics reports).
8788
serpyco-rs = "^1.10.2"
8889
sqlalchemy = {version = "^2.0,!=2.0.36", optional = true }
8990
fastapi = { version = ">=0.116.1", optional = true }

0 commit comments

Comments
 (0)