Skip to content

README.md

Latest commit

 

History

History
69 lines (53 loc) · 2.34 KB

README.md

File metadata and controls

69 lines (53 loc) · 2.34 KB

looks-same Benchmark

This benchmark evaluates the performance of four npm packages for image comparison: looks-same, pixelmatch, resemblejs, and blink-diff.
It focuses on execution speed across diverse test cases, including real-world web comparisons and synthetic examples.

Test Cases

Web-Based Averages

  • Web Avg Diff (672x623):
    Aggregates failed web page comparisons from our production projects using Testplane for e2e testing.
    Represents 103,142 real-world UI mismatches (e.g., layout shifts, rendering errors).
  • Web Avg Success (656x547):
    Derived from 6,282,752 successful comparisons in production.
    Tests speed for "no difference" scenarios common in regression testing workflows.

Synthetic Comparisons

  • Equal Images (1000x1000): Baseline test with identical images.
  • 1% Visible Diff (1000x1000):
    Human-noticeable differences (ΔE ≥ 2.3 in CIEDE2000).
  • 10% Visible Diff (1000x1000):
    Larger-scale variations with the same perceptual thresholds.
  • Full Max Diff (1000x1000):
    All pixels altered (maximum difference) to stress-test worst-case performance.

Realistic Example

  • Demonstrative Example (896×784):
    Real-world UI comparison with an 8% visible difference: a button intentionally recolored to a human-noticeable shade.
    Provides visual intuition for typical failure scenarios.

    Uses the following images (from ./fixtures)
    Image 1 Image 2

Results

See results.md for execution times across all test cases.

Running the benchmark

  1. Clone the repository: 
    git clone git@github.com:gemini-testing/looks-same.git
  2. Navigate to "benchmark" directory: 
    cd looks-same/benchmark
  3. Install dependencies: 
    npm install
  4. Run the benchmark: 
    npm start