fix: stabilize concurrent PDF parsing with LiteParse WASM by c121914yu · Pull Request #7318 · labring/FastGPT

c121914yu · 2026-07-16T09:56:15Z

What changed

Replace the native @llamaindex/liteparse runtime with @llamaindex/liteparse-wasm 2.0.8.
Initialize one independent WASM module per file-parsing worker, explicitly free each parser after a page batch, and fall back to PDF.js only when the WASM package or binary cannot initialize.
Externalize the WASM package from worker bundles, copy the runtime files into app/admin worker outputs, and verify the binary in Docker images.
Change the default PARSE_FILE_WORKERS value from 10 to 5 across runtime config, Docker Compose, Helm, tests, and Chinese/English documentation.
Deduplicate identical in-flight requests for model lists and sandbox package metadata.
Add PDF lifecycle, fallback, request deduplication, real worker concurrency, and memory-observation coverage.

The matching pro/admin worker packaging change is in draft PR labring/fastgpt-pro#1020; this PR pins that commit so CI can fetch the complete runtime.

Why

The native LiteParse/PDFium path could intermittently corrupt parser state under concurrent worker activity, producing errors such as invalid PDF format or page not found until the service restarted. A WASM instance is isolated inside each worker thread, preserving file-level concurrency without sharing native PDFium global state.

Validation

Full @fastgpt/service suite: 263 files passed, 1 skipped; 3,279 tests passed, 35 skipped.
Real 120-page PDF worker integration: 14/14 passed, including 4 concurrent workers and 12 stress tasks; the pool released about 208 MiB RSS after destroy.
LiteParse/PDF fallback unit tests: 7/7 passed.
Request deduplication tests: 22/22 passed.
pnpm --filter app build:workers
pnpm --filter @fastgpt/admin build:workers
pnpm --filter app typecheck
pnpm --filter @fastgpt/admin typecheck
ESLint, Prettier, textlint, document checks, git diff --check, and repository pre-commit hooks passed.

github-actions · 2026-07-16T10:00:43Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	18.42%	1632 / 8859
🔵	Statements	18.41%	1713 / 9304
🔵	Functions	16.2%	342 / 2110
🔵	Branches	15.14%	761 / 5026

File Coverage

No changed files found.

Generated in workflow #1494 for commit f47da82 by the Vitest Coverage Report Action

github-actions · 2026-07-16T10:04:38Z

✅ Docs Preview Deployed!

🔗 👀 Click here to visit preview

ghcr.io/labring/fastgpt-docs-pr:f47da82c2f49671c4e3a33e1908e86321f351640

🕒 Time: 2026-07-16 21:02:48 (UTC+8)

github-actions · 2026-07-16T10:07:59Z

✅ Admin Preview Image Ready!

ghcr.io/labring/fastgpt-pr:admin_dbf98b7ca658058eb3b530dee20ac14ff66d529e

🕒 Time: 2026-07-16 19:29:33 (UTC+8)

github-actions · 2026-07-16T10:09:10Z

✅ Build Successful - Preview fastgpt Image for this PR:

ghcr.io/labring/fastgpt-pr:fastgpt_27a282421f8a020cc14566cd4137d16f74632a87

🕒 Time: 2026-07-16 20:42:43 (UTC+8)

Copilot

Pull request overview

This PR stabilizes concurrent PDF parsing by switching FastGPT’s file-reading worker from native @llamaindex/liteparse (PDFium/N-API) to @llamaindex/liteparse-wasm (WASM), adds in-flight request deduplication on the web side for selected endpoints, and reduces the default file parsing worker pool size from 10 to 5 across configs and docs.

Changes:

Replace native LiteParse with LiteParse WASM in the readFile worker, including per-batch parser cleanup and a guarded PDF.js fallback only for WASM load/init failures.
Add optional in-flight request deduplication in the web request helper and enable it for model list + sandbox package endpoints, with unit tests.
Update worker build packaging (externalize/copy WASM runtime), Docker image verification, default worker concurrency, and related deployment/docs files.

Reviewed changes

Copilot reviewed 37 out of 38 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
projects/app/test/web/common/api/request.test.ts	Adds unit tests for in-flight request deduplication behavior.
projects/app/src/web/common/system/api.ts	Enables request deduplication for the “getMyModels” endpoint.
projects/app/src/web/common/api/request.ts	Implements optional in-flight request deduplication in the request wrapper.
projects/app/src/pageComponents/app/detail/WorkflowComponents/Flow/nodes/NodeCode/api.ts	Enables request deduplication for sandbox package metadata fetches.
projects/app/scripts/build-workers.ts	Updates worker runtime packaging/externalization for LiteParse WASM.
projects/app/package.json	Replaces dependency from `@llamaindex/liteparse` to `@llamaindex/liteparse-wasm`.
projects/app/Dockerfile	Adds a runtime assertion to ensure the WASM binary exists in the final image.
projects/app/.env.template	Updates template defaults/formatting relevant to deployment config.
pnpm-workspace.yaml	Pins `@llamaindex/liteparse-wasm` version in the workspace catalog.
pnpm-lock.yaml	Updates lockfile entries to reflect LiteParse WASM dependency graph.
packages/service/worker/readFile/utils/LiteParse/pdfTextPostprocess.ts	Adjusts LiteParse-related type imports to local worker types.
packages/service/worker/readFile/utils/LiteParse/index.ts	Loads/initializes LiteParse WASM in-worker and frees parser per batch.
packages/service/worker/readFile/extension/pdf.ts	Updates fallback detection logic/messages from native-load errors to WASM-load/init errors.
packages/service/test/worker/readFile/utils/LiteParse/index.test.ts	Updates unit tests/mocks for LiteParse WASM init + parser free semantics.
packages/service/test/worker/readFile/integration.test.ts	Adds memory observation logging around PDF worker stress + pool destroy.
packages/service/test/worker/readFile/extension/pdf.test.ts	Updates fallback tests for LiteParse WASM load/init failures.
packages/service/test/worker/function.test.ts	Updates expectations for new default `PARSE_FILE_WORKERS` value (5).
packages/service/package.json	Replaces dependency from `@llamaindex/liteparse` to `@llamaindex/liteparse-wasm`.
packages/service/env.ts	Changes default `PARSE_FILE_WORKERS` from 10 to 5.
document/public/deploy/docker/v4.15/global/docker-compose.zilliz.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/global/docker-compose.seekdb.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/global/docker-compose.pg.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/global/docker-compose.opengauss.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/global/docker-compose.oceanbase.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/global/docker-compose.milvus.yml	Updates default `PARSE_FILE_WORKERS` in deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.zilliz.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.seekdb.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.pg.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.opengauss.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.oceanbase.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/public/deploy/docker/v4.15/cn/docker-compose.milvus.yml	Updates default `PARSE_FILE_WORKERS` in CN deployment template.
document/data/doc-last-modified.json	Updates doc last-modified timestamps for changed docs.
document/content/self-host/upgrading/4-15/4152.mdx	Documents the LiteParse/concurrency and request-dedup changes in CN upgrade notes.
document/content/self-host/upgrading/4-15/4152.en.mdx	Documents the LiteParse/concurrency and request-dedup changes in EN upgrade notes.
document/content/self-host/config/env.mdx	Updates documented default for `PARSE_FILE_WORKERS` (CN).
document/content/self-host/config/env.en.mdx	Updates documented default for `PARSE_FILE_WORKERS` (EN).
deploy/version/v4.15/docker-compose.template.yml	Updates default `PARSE_FILE_WORKERS` in deploy template.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+  liteParseWasmModulePromise = (async () => {
+    try {
+      const wasmModule = await import('@llamaindex/liteparse-wasm');
+      const wasmPath = workerRequire.resolve('@llamaindex/liteparse-wasm/liteparse_wasm_bg.wasm');
+      const wasmBytes = await readFile(wasmPath);
+      await wasmModule.default({ module_or_path: wasmBytes });


pull-request-size Bot added the size/XS label Jul 16, 2026

pull-request-size Bot added size/L and removed size/XS labels Jul 16, 2026

c121914yu changed the title ~~fix: upgrade LiteParse to 2.0.8~~ fix: stabilize concurrent PDF parsing with LiteParse WASM Jul 16, 2026

c121914yu added 3 commits July 16, 2026 20:20

fix: upgrade LiteParse to 2.0.8

5709fd6

fix: stabilize concurrent PDF parsing

0fc6cd7

doc

27a2824

c121914yu force-pushed the fix-pdf branch from dbf98b7 to 27a2824 Compare July 16, 2026 12:28

c121914yu marked this pull request as ready for review July 16, 2026 12:30

Copilot AI review requested due to automatic review settings July 16, 2026 12:30

Copilot started reviewing on behalf of c121914yu July 16, 2026 12:30 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

fix: ts

f47da82

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: stabilize concurrent PDF parsing with LiteParse WASM#7318

fix: stabilize concurrent PDF parsing with LiteParse WASM#7318
c121914yu wants to merge 4 commits into
labring:mainfrom
c121914yu:fix-pdf

c121914yu commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

c121914yu commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Why

Validation

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

c121914yu commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading