SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

📝 Paper  |  🤗 Dataset  |  💻 Code  |  📄 MIT License

SynthDocBench is a fully synthetic, controlled benchmark for long-context visual document understanding that varies document length, layout structure, modality composition, and question type as independent axes, enabling systematic diagnosis of VLM failure modes.

Authors

Abhigya Verma^*,1  Khyati Mahajan^*,1  Amit Kumar Saha^*,1  Shruthan Radhakrishna¹  Sagar Davasam¹  Vikas Yadav¹  Sai Rajeswar^1,2,3

^*Equal contribution  |  ¹ServiceNow AI  |  ²Mila – Quebec AI Institute  |  ³Université de Montréal

---

📖 Introduction

SynthDocBench evaluates vision-language models on 1,788 questions over 200 documents containing 3,340 charts, averaging 51.1 pages per document — a fully synthetic, controlled benchmark for long-context visual document understanding.

Unlike prior benchmarks that draw on real documents and confound length, layout, and modality, SynthDocBench generates documents end-to-end with an LLM pipeline across 6 layout archetypes, varying document length, layout structure, modality composition, and question type as independent axes — enabling controlled attribution of why a model fails, not just that it does.

Key Features:

• 🧩 3 task-specific subsets — chart (24 D3.js chart types), cross_modal (evidence split across non-adjacent sections), and complex (multi-hop, difficulty L1 → L5)
• 📄 51.1 avg. pages per document, far beyond single-page chart/doc-VQA benchmarks
• 🧪 Dual-layer generation — every chart is rendered as D3.js and backed by hidden structured metadata, making ground-truth answers deterministic
• 🤖 7 frontier VLMs evaluated, cross-judge validated (GPT-5 vs. Gemini-as-judge agree within 3.5 accuracy points, r ≥ 0.94)
• 📉 Surfaces 3 failure modes invisible to existing benchmarks — length degradation, positional sensitivity (middle third hardest for 5/6 models), and chart-reading collapse in long-document settings

Benchmark

Subset	Description
chart	Chart-reading questions answerable only from a chart — probes 24 D3.js chart types (incl. dumbbell, lollipop, slope, sparkline grids) requiring exact numerical reads.
cross_modal	Questions requiring text + chart evidence from separate, non-adjacent sections (long-range cross-modal grounding).
complex	Multi-hop questions combining 2–4 evidence units across text and charts, difficulty L1 (direct lookup) → L5 (cross-section synthesis).

Source PDFs are hosted alongside the QA subsets at huggingface.co/datasets/ServiceNow-AI/SynthDocBench.

Reproduce the evaluation

See Evaluation below for the harness that scores candidate VLMs against these questions and reproduces the paper's numbers.

---

Report generation pipeline

A LangGraph-based pipeline that generates rich, themed reports (HTML + PDF, with an optional "scanned document" effect) and embedded D3.js visualizations. It supports two input paths:

• Topic-in: give it a topic and it researches (Wikipedia/Wikidata) and authors the report itself.
• Data-in: give it pre-built source text — Wikipedia groupings or arXiv papers — and it themes and visualizes them.

Pipeline overview

select_layout_archetype
  → generate_reports_from_topics   (topic-in; bypassed when reports are supplied)
  → generate_topic_summary
  → enrich_reports
  → generate_design_tokens
  → assign_blueprints_and_budget   (style DNA, chart budget, report identity)
  → generate_visualization_ideas
  → insert_placeholders
  → select_visualization_types
  → search_wikipedia_wikidata
  → structure_diagram_data
  → generate_diagram_code_with_metadata
  → validate_vizzes
  → assign_figure_numbers → insert_figure_references
  → generate_content_components
  → extract_table_metadata → validate_metadata
  → generate_qa_manifest
Post-graph: generate_html_report()
         →  convert_to_scanned_pdf()   (HTML → PDF → scanned PDF)
         →  complex question generation (→ <report>_questions.json)

After each report is built, complex/chart-reading/cross-modal QA pairs are generated from it (reusing the report's embedded JSON-LD chart metadata) and written alongside it. Opt out with --no-questions. It can also be run standalone via pipelines/complex_question_generation.py.

Features

• Style DNA & visual diversity — 7 randomized style axes plus per-component renderers so reports don't all look identical (utils/style_dna.py, utils/visual_diversity.py).
• Layout archetypes — magazine / dashboard / academic / editorial / infographic / brutalist.
• Real-world data — Wikipedia prose + Wikidata structured facts (tools/wiki_search.py).
• D3.js visualizations with figure numbering, in-text references, and a QA/metadata manifest.
• Parallelism — per-section and per-viz LLM calls run concurrently (utils/parallel.py).
• PDF export — clean PDF and a realistic scanned-document effect.
• Multi-model — GPT (Azure), Gemini (OpenAI-compatible endpoint or native Vertex AI), and Claude (Foundry / Bedrock / proxy).
• arXiv-to-report — fetch papers, extract text/figures/tables, and generate reports.
• Complex question generation — multi-hop, chart-reading, and cross-modal QA pairs generated from each finished report (pipelines/complex_question_generation.py); auto-chained into report generation (opt out with --no-questions).

Installation

uv sync          # or: pip install -e .   (dependencies are in pyproject.toml)
uv run playwright install chromium   # needed for HTML→PDF

pdf2image needs poppler on your system (brew install poppler / apt-get install poppler-utils).

Configuration — model credentials

Model configs live in utils/model_constant.py, which is git-ignored. Copy the template and fill it in (or set the referenced environment variables, e.g. in .env):

cp utils/model_constant.example.py utils/model_constant.py

The template documents which fields each provider needs (SUPPORTED_MODELS per model key; ALL_GENERATIONS is an unused stub for this pipeline).

Usage

Topic-in

# Single topic — pipeline researches and authors the report
python -m pipelines.report_generation --topic "Renewable Energy" --model gpt-4o --output output/demo

# Many topics from a JSON list, run concurrently
python -m pipelines.report_generation --topics-file topics.json --batch-size 4 --model gemini-2.5-flash

Data-in — Wikipedia groupings

python -m pipelines.report_generation \
    --input data/filtered_groupings.json \
    --sample_start_index 0 --sample_end_index 3 \
    --per_topic_limit 3 \
    --model gemini-3-pro-preview \
    --output output/trial
# or: bash run_report_generation.sh

arXiv-to-report

# End-to-end (fetch → extract → report). See the header of the script for all options.
bash run_arxiv_to_report.sh

# Autonomous mode: LLM discovers topics, fetches papers per topic, generates reports
RESEARCH_AREA="Large Language Model Evaluation" bash run_arxiv_to_report.sh

# Use a pre-fetched metadata file directly
python -m pipelines.report_generation --arxiv-metadata arxiv_dataset/metadata/papers_metadata.json

Other modes

# Regenerate from a cached state
python -m pipelines.report_generation --from-cache cache/state_file.pkl

# Convert existing HTML(s) to PDF (no LLM)
python -m pipelines.report_generation --html-to-pdf output/demo

# HTML only (skip PDF)
python -m pipelines.report_generation --topic "…" --no-scanned

Evaluation

Score candidate VLMs on the generated reports. The harness rasterizes each report PDF to page-image strips (vision-only — no HTML/metadata), asks each candidate model the report's questions, and scores answers with a judge model.

python eval/evaluate_models.py \
    --input-dir data/reports_v4 \
    --candidates gpt-4o \
    --judge gpt-5 \
    --output results/all_results.json \
    --workers 4
# --concat-num N groups N pages per strip; see the script header for all options.

Key CLI options

Option	Default	Description
--topic / --topics-file	—	Topic-in input
--input	—	Data-in: Wikipedia groupings .json/.pkl
--arxiv-metadata	—	Data-in: arXiv papers_metadata.json
--from-cache	—	Regenerate from a saved state
--html-to-pdf	—	Convert existing HTML file/dir to PDF
--model	gpt-4o	LLM model key (see model_constant.py)
--output	output	Output directory
--viz-cache	—	Directory for viz code/images
--per_topic_limit	3	Data-in: samples/papers per group
--sample_start_index / --sample_end_index	0 / 3	Data-in: slice of topic groups
--batch-size	1	Topic-in: max concurrent topics
--noise-min / --noise-max	5.0 / 15.0	Scanned-PDF noise range
--no-scanned	false	Skip PDF generation

Project structure

• pipelines/report_generation.py — the report pipeline (topic-in + data-in).
• pipelines/arxiv_pipeline.py — arXiv search / citation-reference expansion / PDF download.
• pipelines/autonomous_pipeline.py — autonomous arXiv mode (topic discovery → fetch → report).
• pipelines/complex_question_generation.py — multi-hop / chart-reading / cross-modal QA generation.
• eval/evaluate_models.py — vision-only evaluation of candidate VLMs with a judge model.
• utils/ — api_call (LLM client), html_generator, state_loader, extract_content, parallel, style_dna, visual_diversity, model_constant (git-ignored config).
• tools/wiki_search.py — Wikipedia/Wikidata API integration.
• data/filtered_groupings.json — sample Wikipedia groupings for data-in mode.

Output

1. HTML report — themed, with embedded D3.js visualizations.
2. PDF — clean print version.
3. Scanned PDF — realistic scanned-document effect (unless --no-scanned).
4. Metadata sidecar — *_report_metadata.json (style DNA, blueprints, QA global stats).
5. State cache — pickle for regeneration (with --save-cache).

Dependencies

Key deps (full list in pyproject.toml): langgraph, openai, anthropic, langchain-google-vertexai, boto3 (LLM/orchestration); beautifulsoup4, pymupdf, feedparser (data); playwright, pdf2image, pillow (PDF export).

📚 Citation

@misc{verma2026synthdocbenchcontrolledbenchmarklongcontext,
      title={SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding},
      author={Abhigya Verma and Khyati Mahajan and Amit Kumar Saha and Shruthan Radhakrishna and Sagar Davasam and Vikas Yadav and Sai Rajeswar},
      year={2026},
      eprint={2607.10400},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.10400},
}