Skip to content

Fix parallel PDF API script env handling#4385

Open
compair-steven wants to merge 3 commits into
Unstructured-IO:mainfrom
compair-steven:compair/fix-parallel-pdf-api-script-env
Open

Fix parallel PDF API script env handling#4385
compair-steven wants to merge 3 commits into
Unstructured-IO:mainfrom
compair-steven:compair/fix-parallel-pdf-api-script-env

Conversation

@compair-steven

@compair-steven compair-steven commented Jul 2, 2026

Copy link
Copy Markdown

Summary

Fixes scripts/user/process-pdf-parallel-through-api.sh so it can be run with documented environment overrides and on the default Bash shipped with macOS.

Changes:

  • Documents UNST_API_KEY as required.
  • Adds UNST_API_ENDPOINT support instead of hard-coding the public endpoint.
  • Gives STRATEGY, BATCH_SIZE, PDF_SPLIT_PAGE_SIZE, and PDF_SPLITS_DIR defaults before use.
  • Avoids unbound-variable failures when UNST_API_KEY is missing.
  • Replaces mapfile usage with a Bash 3.2-compatible loop.
  • Supports both md5sum and BSD/macOS md5 -q in the parallel script and split helper.

Validation

I tested the script against the hosted API with a valid UNST_API_KEY using a one-page split of example-docs/pdf/layout-parser-paper-fast.pdf:

PDF_SPLITS_DIR="$tmp" \
BATCH_SIZE=1 \
STRATEGY=fast \
scripts/user/process-pdf-parallel-through-api.sh "$pdf"

The script produced a valid split JSON file, combined it successfully, and the combined output contained 25 JSON elements:

jq length "$tmp/${base}-output-${md5}_split-10_strat-fast/${base}_combined.json"
# 25

I also checked the updated shell scripts with bash -n.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 2 files

Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic

Comment thread scripts/user/process-pdf-parallel-through-api.sh
Comment thread scripts/user/split-pdf.sh

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Shadow auto-approve: would auto-approve. Improves shell script portability, env var handling, and validation. Low risk, no change to core logic or API endpoints.

Re-trigger cubic

@compair-steven

Copy link
Copy Markdown
Author

Thanks for the quick review! I pushed a follow-up that validates BATCH_SIZE as a positive integer before the batching loop, tightens PDF_SPLIT_PAGE_SIZE to the same positive-integer shape, and checks that checksum generation produced a non-empty value before constructing split/output paths.

I also verified the failure paths for BATCH_SIZE=0, non-numeric BATCH_SIZE, PDF_SPLIT_PAGE_SIZE=0, and a missing PDF input. Hopefully this addresses the concerns :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant