Robot Framework DocTest library.
Simple Automated Visual Document Testing.
See keyword documentation for
- Visual Document Tests
- Web Visual Tests (Browser/Selenium)
- Print Job Tests
- Pdf Tests (very basic)
- Ai Keywords (optional)
*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Compare two Images and highlight differences
Compare Images Reference.jpg Candidate.jpgOptional LLM for image comparison
*** Settings ***
Library DocTest.Ai
Library DocTest.VisualTest
Library DocTest.PdfTest
*** Test Cases ***
Review Visual Differences With LLM
Compare Images With LLM Reference.pdf Candidate.pdf llm_override=${True}
Extract Text From Document With LLM
${text}= Get Text With LLM Candidate.pdf prompt=Return text and table contents
Log ${text}
Image Should Contain Object With LLM
Image Should Contain Candidate.png Missing product logo
Count Items With LLM
${count}= Get Item Count From Image Candidate.png item_description=number of pallets
Should Be True ${count} >= 0pip install --upgrade robotframework-doctestlibrary # core library
pip install --upgrade robotframework-doctestlibrary[ai] # + LLM-assisted comparisons
pip install --upgrade robotframework-doctestlibrary[dashboard] # + visual review dashboard & mask editor
pip install --upgrade robotframework-doctestlibrary[all] # ai + dashboardWhat each install brings:
| Install | Adds | Notes |
|---|---|---|
| (base) | VisualTest, PdfTest, PrintJobTests, WebVisualTest |
Web visual testing needs no extra dependencies — it drives the Browser Library or SeleniumLibrary you already use. |
[ai] |
LLM-assisted comparison keywords | Configure via DOCTEST_LLM_*. |
[dashboard] |
doctest-dashboard web app (review, baselines, mask editor) |
The CLI stub is always present; the extra brings its server dependencies. |
[browser] |
robotframework-browser |
Convenience: one-liner for web testing with Playwright (run rfbrowser init afterwards). |
[selenium] |
robotframework-seleniumlibrary |
Convenience: one-liner for web testing with Selenium. |
[all] |
ai + dashboard | Web drivers stay an explicit choice. |
The project is managed with uv — one pyproject.toml, one lockfile:
uv sync --all-extras # full dev environment (library, dashboard, test tooling)
uv run invoke tests # unit + acceptance suites
uv run invoke multipython # validate dependency resolution on Python 3.10–3.13
uv run pytest e2e --browser chromium # dashboard end-to-end journeys
cd frontend && npm install && npm run build # build the dashboard web UI onceYou can optionally rely on a large language model to review detected differences and decide whether a comparison should pass. This path is fully opt-in; nothing changes for users who skip these dependencies.
-
Install the optional extra only when needed:
pip install "robotframework-doctestlibrary[ai]" -
Create a
.envfile at the repository root (values here override existing environment variables):# OpenAI-compatible endpoints OPENAI_API_KEY=sk-... DOCTEST_LLM_MODEL=gpt-5,gpt-4o DOCTEST_LLM_VISION_MODEL=gpt-5-mini,gpt-4o-mini
Azure OpenAI deployments can be configured with:
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/ AZURE_OPENAI_API_KEY=... AZURE_OPENAI_DEPLOYMENT=gpt-4o AZURE_OPENAI_API_VERSION=2024-06-01 DOCTEST_LLM_PROVIDER=azure
See
.env.examplefor a combined template covering both providers. -
Use the dedicated keywords (or pass
llm_enabled=${True}to existing ones):*** Test Cases *** Review Visual Differences With LLM Compare Images With LLM Reference.pdf Candidate.pdf llm_override=${True} Review Pdf Structure With LLM Compare Pdf Documents With LLM reference.pdf candidate.pdf compare=structure
Set llm_override=${True} when an LLM approval should override SSIM/DeepDiff failures.
Without the override flag the AI feedback is logged for investigation while the original
assertion result is preserved.
Pass llm_prompt= (or the specialty variants llm_visual_prompt= / llm_pdf_prompt=) to
customise the prompt sent to the model for a particular comparison.
Python 3.10 or newer is supported. Tested with Python 3.10/3.11/3.12/3.13
pip install --upgrade robotframework-doctestlibrary
pip install git+https://github.com/manykarim/robotframework-doctestlibrary.git
or
git clone https://github.com/manykarim/robotframework-doctestlibrary.gitcd robotframework-doctestlibrarypip install -e .
Install Tesseract, Ghostscript, GhostPCL, ImageMagick binaries and barcode libraries (libdmtx, zbar) on your system.
Hint: Since 0.2.0 Ghostscript, GhostPCL and ImageMagick are only needed for rendering .ps and .pclfiles.
Rendering and content parsing of .pdf is done via MuPDF
In the future there might be a separate pypi package for .pcl and .ps files to get rid of those dependencies.
Linux
apt-get install imagemagick tesseract-ocr ghostscript libdmtx0b libzbar0Windows
- https://github.com/UB-Mannheim/tesseract/wiki
- https://ghostscript.com/releases/gsdnld.html
- https://ghostscript.com/releases/gpcldnld.html
- https://imagemagick.org/script/download.php
The executable for GhostPCL gpcl6win64.exe needs to be renamed to pcl6.exe
Otherwise it will not be possible to render .pcl files successfully for visual comparison.
Add tesseract, ghostscript and imagemagick to system path in windows (only needed for OCR, .pcl and .ps support)
- C:\Program Files\ImageMagick-7.0.10-Q16-HDRI
- C:\Program Files\Tesseract-OCR
- C:\Program Files\gs\gs9.53.1\bin
- C:\Program Files\gs\ghostpcl-9.53.1-win64
(The folder names and versions on your system might be different)
That means: When you open the CMD shell you can run the commands
magick.exetesseract.exegswin64.exepcl6.exe
successfully from any folder/location
How to solve ImportError for pylibdmtx
If you see an ugly ImportError when importing pylibdmtx on
Windows you will most likely need the Visual C++ Redistributable Packages for
Visual Studio 2013. Install vcredist_x64.exe if using 64-bit Python, vcredist_x86.exe if using 32-bit Python.
The library might return the error File could not be converted by ImageMagick to OpenCV Image: <path to the file> when comparing PDF files.
This is due to ImageMagick permissions. Verify this as follows with the sample.pdf in the testdata directory:
convert sample.pdf sample.jpg
convert-im6.q16: attempt to perform an operation not allowed by the security policySolution is to copy the policy.xml from the repository to the ImageMagick installation directory.
You can also use the docker images or create your own Docker Image
docker build -t robotframework-doctest .
Afterwards you can, e.g., start the container and run the povided examples like this:
- Windows
docker run -t -v "%cd%":/opt/test -w /opt/test robotframework-doctest robot atest/Compare.robot
- Linux
docker run -t -v $PWD:/opt/test -w /opt/test robotframework-doctest robot atest/Compare.robot
Try out the library using Gitpod
DocTest.WebVisualTest adds baseline-driven visual regression testing for web
apps on top of the comparison engine. It captures pages and elements through the
Browser Library or SeleniumLibrary session your suite already runs — no
extra dependency, the library you imported is auto-detected:
*** Settings ***
Library Browser
Library DocTest.WebVisualTest result_json=true
*** Test Cases ***
Checkout Page Looks Right
New Page https://shop.example.com/checkout
Compare Page To Baseline checkout
Compare Element To Baseline id=price-table checkout-prices- Baselines managed for you: first run stores the capture as
visual_baselines/checkout.png(passes with a warning); later runs compare against it. Update via--variable REFERENCE_RUN:Trueor by accepting in the review dashboard. - Flake resistance: failing comparisons recapture and recompare until
retry_timeout(default 3s) expires; Browser captures disable animations and normalize device pixel ratio (scale=css). - Full engine available: every
Compare Imagesoption works — masks, thresholds, move tolerance, OCR pattern masks.
See docs/web-visual-testing.md for the full guide.
The doctest-dashboard (shipped with the [dashboard] extra) is a locally runnable web app to review comparison results outside of log.html:
- browse runs and failing comparisons with diff thumbnails
- diff viewer with side-by-side / overlay / blink / swipe modes and next/previous-difference navigation
- accept a difference → the candidate is promoted to the new reference (plain file copy, git-diffable, SHA-256 audit trail)
- reject a difference → export a bug-data ZIP (reference + candidate + diffs + metadata)
- visual mask editor for coordinate, area, and text-pattern masks with live preview — including one-click create mask from diff region and instant re-comparison of past runs with the adjusted masks
Quick start:
pip install robotframework-doctestlibrary[dashboard] # wheels include the web UI
doctest-dashboard serve # open http://127.0.0.1:8008
doctest-dashboard ingest results/output.xmlRun your suites with the machine-readable sidecar enabled so the dashboard gets per-page data:
*** Settings ***
Library DocTest.VisualTest result_json=true take_screenshots=true screenshot_format=pngSee the full guide in docs/dashboard.md for the review workflow, mask editor usage, team mode, and the API.
Have a look at
for more examples.
Testing with Robot Framework
*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Compare two Images and highlight differences
Compare Images Reference.jpg Candidate.jpg*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Compare two Images and ignore parts by using masks
Compare Images Reference.jpg Candidate.jpg placeholder_file=masks.json
Compare two PDF Docments and ignore parts by using masks
Compare Images Reference.jpg Candidate.jpg placeholder_file=masks.json
Compare two Farm images with date pattern
Compare Images Reference.jpg Candidate.jpg placeholder_file=testdata/pattern_mask.json
Compare two Farm images with area mask as list
${top_mask} Create Dictionary page=1 type=area location=top percent=10
${bottom_mask} Create Dictionary page=all type=area location=bottom percent=10
${masks} Create List ${top_mask} ${bottom_mask}
Compare Images Reference.jpg Candidate.jpg mask=${masks}
Compare two Farm images with area mask as string
Compare Images Reference.jpg Candidate.jpg mask=top:10;bottom:10
Text pattern matching levels: word_pattern matches single words; line_pattern matches whole text lines (anchored, use .*….* to match anywhere); pattern matches single words — and when the regex contains whitespace it is searched anywhere within each line, masking exactly the words the match span covers. So Robot Framework masks just that phrase wherever it occurs, while .*Robot Framework.* extends the match over the whole line and masks all of it. Matching is case-sensitive; add (?i) for case-insensitive.
[
{
"page": "all",
"name": "Date Pattern",
"type": "pattern",
"pattern": ".*[0-9]{2}-[a-zA-Z]{3}-[0-9]{4}.*"
},
{
"page": "1",
"name": "Top Border",
"type": "area",
"location": "top",
"percent": 5
},
{
"page": "1",
"name": "Left Border",
"type": "area",
"location": "left",
"percent": 5
},
{
"page": 1,
"name": "Top Rectangle",
"type": "coordinates",
"x": 0,
"y": 0,
"height": 10,
"width": 210,
"unit": "mm"
}
]The companion doctest-dashboard ships a visual mask editor with live preview of coordinate, area, and pattern masks — including one-click mask creation from detected diff regions and instant re-comparison of past runs with the adjusted masks. It reads and writes the exact JSON schema shown above. See docs/dashboard.md for the full guide.
Deprecation note: the tkinter tool
utilities/mask_editor.pyis superseded by the dashboard's mask editor and will be removed in a future release. It only supports pixel-based coordinate masks.
Enable result_json=true on DocTest.VisualTest / DocTest.PdfTest to write a JSON sidecar per comparison into {OUTPUT_DIR}/doctest_results/ (statuses, SSIM scores, diff regions, resolved masks, per-page renderings). The dashboard — and any other tooling — ingests these for review, accept/reject baseline management, and mask maintenance:
*** Settings ***
Library DocTest.VisualTest result_json=true take_screenshots=true screenshot_format=png*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Accept if parts are moved up to 20 pixels by pure visual check
Compare Images Reference.jpg Candidate.jpg move_tolerance=20
Accept if parts are moved up to 20 pixels by reading PDF Data
Compare Images Reference.pdf Candidate.pdf move_tolerance=20 get_pdf_content=${true}
Accept differences if text content is the same via OCR
Compare Images Reference.jpg Candidate.jpg check_text_content=${true}
Accept differences if text content is the same from PDF Data
Compare Images Reference.pdf Candidate.pdf check_text_content=${true} get_pdf_content=${true}*** Settings ***
Library DocTest.VisualTest movement_detection=orb
*** Test Cases ***
Accept if parts are moved up to 20 pixels by pure visual check
Compare Images Reference.jpg Candidate.jpg move_tolerance=20*** Settings ***
Library DocTest.VisualTest movement_detection=template
*** Test Cases ***
Accept if parts are moved up to 20 pixels by pure visual check
Compare Images Reference.jpg Candidate.jpg move_tolerance=20*** Settings ***
Library DocTest.VisualTest movement_detection=classic
*** Test Cases ***
Accept if parts are moved up to 20 pixels by pure visual check
Compare Images Reference.jpg Candidate.jpg move_tolerance=20Take additional screenshots or reference and candidate file.
*** Settings ***
Library DocTest.VisualTest take_screenshots=${true} screenshot_format=pngTake diff screenshots to highlight differences
*** Settings ***
Library DocTest.VisualTest show_diff=${true} DPI=300*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Compare two Farm images with date pattern and east detection
Compare Images Reference.jpg Candidate.jpg placeholder_file=masks.json ocr_engine=east*** Settings ***
Library DocTest.PdfTest
*** Test Cases ***
Check if list of strings exists in PDF File
@{strings}= Create List First String Second String
PDF Should Contain Strings ${strings} Candidate.pdf
Compare two PDF Files and only check text content
Compare Pdf Documents Reference.pdf Candidate.pdf compare=text
Compare two PDF Files and only check text content and metadata
Compare Pdf Documents Reference.pdf Candidate.pdf compare=text,metadata
Compare two PDF Files and check all possible content
Compare Pdf Documents Reference.pdf Candidate.pdfBoth Compare Pdf Documents and Compare Pdf Structure now share the same helper options for masking boilerplate data:
maskaccepts inline JSON/dicts, Robot lists, or a path to a JSON file generated by the screenshot tooling. Addunit=ptwhen coordinates are already expressed in PDF points.text_mask_patternslets you provide one or more regular expressions (string or@{list}) to drop matching lines from both PDFs before diffing. This is handy for timestamps, order numbers, etc.ignore_ligatures=${True}normalises glyphs such asfi/flto their ASCII representation so small font engine differences do not fail the comparison.
*** Settings ***
Library DocTest.PdfTest
*** Test Cases ***
Compare invoices while ignoring IDs
${mask}= Set Variable {"pages":[{"page":1,"rectangles":[{"left":410,"top":36,"right":560,"bottom":78}],"unit":"pt"}]}
Compare Pdf Structure ${CURDIR}/baseline.pdf ${CURDIR}/candidate.pdf mask=${mask}
... text_mask_patterns=\\bINV-\d{6}\\b ignore_ligatures=${True} position_tolerance=5PDFs may contain special Unicode characters that look identical to standard ASCII but fail string comparisons. Common examples include non-breaking spaces (\u00A0) instead of regular spaces, or typographic dashes (\u2013, \u2014) instead of hyphens.
Use the character_replacements parameter to normalize these characters:
*** Settings ***
# Apply character replacements to all keywords in the test suite
Library DocTest.PdfTest character_replacements={'\u00A0': ' '}
Library DocTest.VisualTest character_replacements={'\u00A0': ' ', '\u2013': '-'}
*** Test Cases ***
PDF comparison with normalized whitespace
Compare Pdf Documents reference.pdf candidate.pdf compare=text
... character_replacements={'\u00A0': ' '}
Check PDF contains text with normalized characters
PDF Should Contain Strings Expected Text candidate.pdf
... character_replacements={'\u00A0': ' '}
Structure comparison with character normalization
Compare Pdf Structure reference.pdf candidate.pdf
... character_replacements={'\u00A0': ' ', '\u2013': '-'}For VisualTest text extraction keywords that use assertion operators, use the Set Character Replacements keyword:
*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Get text with normalized characters
Set Character Replacements {'\u00A0': ' '}
${text}= Get Text document.pdf == Expected text
Set Character Replacements ${NONE} # Clear when doneCommon character replacements:
| Character | Unicode | Description | Replacement |
|---|---|---|---|
|
\u00A0 |
Non-breaking space | (regular space) |
– |
\u2013 |
En dash | - (hyphen) |
— |
\u2014 |
Em dash | - (hyphen) |
‐ |
\u2010 |
Unicode hyphen | - (hyphen) |
When comparing PDF documents where font or size changes cause text to reflow across pages, use the ignore_page_boundaries option to compare only text content and order, ignoring page structure:
*** Settings ***
Library DocTest.PdfTest
*** Test Cases ***
Compare PDFs ignoring page breaks due to font change
[Documentation] Reference has 2 pages, candidate has 3 pages due to larger font
... Both contain identical text in the same order
Compare Pdf Structure reference.pdf candidate_larger_font.pdf
... ignore_page_boundaries=${True}
Compare PDF Documents with cross-page text matching
Compare Pdf Documents reference.pdf candidate.pdf
... compare=structure ignore_page_boundaries=${True}For finer control over what gets compared, use the check_geometry and check_block_count options:
| Parameter | Default | Description |
|---|---|---|
ignore_page_boundaries |
${False} |
Flatten text across all pages and compare only content and order. Automatically disables geometry and block count checks. |
check_geometry |
${True} |
When ${False}, skip line position/size comparison. Useful when layout differs but text matches. |
check_block_count |
${True} |
When ${False}, skip block count validation per page. Useful when text blocks are merged/split differently. |
*** Settings ***
Library DocTest.PdfTest
*** Test Cases ***
Compare PDF content only (ignore positions)
[Documentation] Same text content, but positions differ due to formatting
Compare Pdf Structure reference.pdf reformatted.pdf
... check_geometry=${False}
Compare PDF without block structure validation
[Documentation] Text blocks may be merged or split differently
Compare Pdf Structure reference.pdf candidate.pdf
... check_block_count=${False}
Pure content comparison (most flexible)
[Documentation] Only compare that text content and order match
Compare Pdf Structure reference.pdf candidate.pdf
... check_geometry=${False} check_block_count=${False}Note: When comparisons fail, a single summary warning is shown at the top of log.html (e.g., "Comparison failed: 5 difference(s) found"). Individual differences are logged as INFO messages within the keyword output for detailed inspection without cluttering the log summary.
Store the watermark in a separate B/W image or PDF.
Watermark area needs to be filled with black color.
Watermark content will be subtracted from Visual Comparison result.
*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Compare two Images and ignore jpg watermark
Compare Images Reference.jpg Candidate.jpg watermark_file=Watermark.jpg
Compare two Images and ignore pdf watermark
Compare Images Reference.pdf Candidate.pdf watermark_file=Watermark.pdf
Compare two Images and ignore watermark folder
Compare Images Reference.pdf Candidate.pdf watermark_file=${CURDIR}${/}watermarksWatermarks can also be passed on Library import. This setting will apply to all Test Cases in Test Suite
*** Settings ***
Library DocTest.VisualTest watermark_file=${CURDIR}${/}watermarks
*** Test Cases ***
Compare two Images and ignore watermarks
Compare Images Reference.jpg Candidate.jpg*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Get Text Content And Compare
${text} Get Text From Document Reference.pdf
List Should Contain Value ${text} Test String*** Settings ***
Library DocTest.VisualTest
*** Test Cases ***
Get Text Content And Compare
${text} Get Barcodes From Document reference.jpg
List Should Contain Value ${text} 123456789Document Testing can be run in parallel using pabot.
However, you need to pass the additional arguments --artifacts and --artifactsinsubfolders to the pabot command, to move the screenshots to the correct subfolder.
Otherwise the screenshots will not be visible in the log.html
pabot --testlevelsplit --processes 8 --artifacts png,jpg,pdf,xml --artifactsinsubfolders /path/to/your/tests/
I experimented a bit and tried to use this library for Visual Testing of Web Applications.
Please have a look at this pilot example here
Feel free to create issues or pull requests.
I'm always happy for any feedback.
In order of appearance.
- Many Kasiriha
- April Wang
This project is community driven and becomes a reality only through the work of all the people who contribute.
