feat: Add warning threshold support for image comparisons

[simonmeggle]

This PR introduces a warning threshold that allows executing an arbitrary keyword (for example Set Tags foo:warn) when image differences fall between two thresholds.

Background

Maps often consist of multiple overlays with data from different sources. Verifying that such maps are rendered correctly is a common use case for Synthetic Monitoring.

If a map layer is missing, a large number of pixels will differ. In such cases the comparison threshold can be configured so the test fails as expected.

However, geographic maps naturally change over time (new buildings, forests, roads, etc.). These small and expected differences should not necessarily trigger an immediate alert. In many cases, a warning would be more appropriate.

A separate WARN state in Robot Framework has been discussed several times in RF PRs and Slack discussions, but has not been introduced so far.

Approach

To address this, a new parameter threshold_warn is introduced. When the comparison score exceeds this threshold but remains below the failure threshold, a user-defined keyword can be executed.

The library import argument run_keyword_on_warn_threshold allows defining which keyword should run in this situation. This gives users full flexibility to decide how warnings should be handled (e.g. logging, tagging the test, or triggering additional actions).

New Parameters

• threshold_warn (keyword-level)
  Warning threshold for visual comparison (range 0.0–1.0).
  Differences between threshold_warn and threshold trigger a warning but do not fail the test.

• run_keyword_on_warn_threshold (library-level)
  Robot Framework keyword (with optional arguments) that will be executed when the warning threshold is exceeded.

Behavior

• score ≤ threshold_warn → PASS
• threshold_warn < score ≤ threshold → PASS with warning + keyword execution
• score > threshold → FAIL (existing behavior)

Validation

• threshold_warn must be less than threshold
• threshold_warn must be between 0.0 and 1.0
• A ValueError is raised if validation fails

Compatibility

The implementation maintains full backward compatibility with existing tests.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud