Flaky CI: "Extract pipeline (PDF upload → run → CSV)" E2E times out on document processing

[JSv4]

Summary

The Extract pipeline (PDF upload → run → CSV) check (workflow .github/workflows/frontend-e2e-extract.yml, spec frontend/tests/e2e/extract-pdf-workflow.spec.ts:86) fails intermittently. It is NOT live-LLM gated — the LLM call is replayed from a VCR cassette (OC_LLM_VCR_MODE=replay, fake OPENAI_API_KEY), so it should be deterministic and free. PDF parsing runs against the in-stack Docling microservice (real but local).

Actual failure mode (run 26676881595, PR #1836)

All three attempts failed with a document-processing timeout, not an LLM/assertion issue:

Error: Document "USC Title 1 …" still processing (data-processing="true")
  - Timeout 480000ms exceeded while waiting on the predicate
    > 1050 | }).toPass({ timeout: timeoutMs, intervals: [5_000, 10_000, 15_000] });
Error: Row "USC Title 1 …" has no non-empty extracted cell — extract may have failed

i.e. ingest → parse (Docling) → embed did not finish within the 8-minute wait, so the extract then ran against an unprocessed document and produced empty cells.

It is flaky, not universally broken

The same workflow has recent success runs — e.g. PR #1825 (claude/canonical-caml-description-refactor) at 2026-05-30 04:58 succeeded, plus feature/in-run-history-compaction and others on 05-28/05-29. Many other history entries are cancelled (concurrency cancel-in-progress), not failures. So the spec passes when document processing happens to finish under 8 min and times out when it doesn't.

Trigger

Runs on workflow_dispatch and on pull_request touching opencontractserver/llms/**, data_extract_tasks.py, the cassette dir, the spec/helpers, or the workflow. (This is why PRs #1825/#1835/#1836 — all touching llms/ — triggered it.) It does not run on main push, so there is no green baseline on main.

Likely causes / fix directions

• Docling parse + embedding throughput on the GH runner is variable and occasionally exceeds the 480s waitForDocumentProcessed budget. Consider raising the timeout, or making the extract step wait for processing-complete deterministically rather than racing it.
• Confirm the extract VCR cassette still matches the request shape (a cassette miss could also yield empty cells).
• Add a main (or nightly) run so there is a known-good baseline + reduce flakiness sensitivity.

Context

Surfaced while babysitting PRs #1825 #1835 #1836 #1838. Fix being investigated on branch claude/extract-pipeline-e2e-fix.

[JSv4]

Refined root cause after pulling backend logs from a failing run (26687429414):

There are two failure modes, and the dominant one is a frontend grid-hydration race, not a backend/processing failure:

• The backend extract succeeds — Celery logs: Raw extraction result: TITLE 1-GENERAL PROVISIONS / ✓ Successfully extracted and saved data for cell 6 / Extract 3 marked complete.
• But the spec's one-shot cell read runs in the gap between the "Extraction in progress" overlay clearing (backend complete) and AG-Grid finishing its per-cell Apollo data fetch, so it reads an empty cell → row has no non-empty extracted cell.
• The rarer mode is the 8-min waitForDocumentReady ceiling being exceeded by cold-runner Docling parsing of the large US Code fixture.

Fix in PR #1846: (A) raise the processing-wait ceiling 8→14 min + spec timeout 20→30 min; (B) poll the cell assertion (expect(...).toPass, 120s) so it retries until the grid hydrates. The assertion is not weakened — a genuinely empty extract still fails on poll timeout.