Skip to content

Restore experimental Z3 SQL correctness-measurement feature#1637

Open
agusaldasoro wants to merge 4 commits into
masterfrom
dse/sql-z3-correctness
Open

Restore experimental Z3 SQL correctness-measurement feature#1637
agusaldasoro wants to merge 4 commits into
masterfrom
dse/sql-z3-correctness

Conversation

@agusaldasoro

@agusaldasoro agusaldasoro commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Overview

Restores the experimental Z3 SQL correctness-measurement feature that was removed in commit c8712da5, and reinstates the core → evomaster-client-java-sql dependency it relies on. This is a revert of that removal, merged on top of dse/memoization.

Motivation

When Z3 generates SQL INSERT data to satisfy a failing WHERE clause, there was no way to verify that the generated rows actually satisfy the original predicate. This feature adds an optional, self-checking correctness metric so we can measure — during experiments — how often Z3-generated data is truly correct.

What it does

Behind a new experimental flag measureSqlZ3Correctness (default false, depends on generateSqlDataWithZ3=true), after Z3 returns a SAT model the solver:

  1. Reconstructs a QueryResultSet from the generated SqlActions (toQueryResultSet / extractGeneValue).
  2. Runs SqlHeuristicsCalculator to compute the heuristic distance between the original WHERE query and the generated data.
    • distance == 0 → the generated insert satisfies the query.
    • distance > 0 → it does not.
  3. Reports the results into Statistics (reportSqlZ3CorrectnessDistance), which emits five new stats when the flag is on: sqlZ3CorrectnessChecks, sqlZ3CorrectnessZeroDistance, sqlZ3CorrectnessNonZero, sqlZ3CorrectnessAvgDist, sqlZ3CorrectnessEvalFailures.

INSERT statements are skipped (no WHERE to evaluate against); DELETE/UPDATE evaluation failures are counted separately and excluded from the average so they don't distort the metric.

Changes

  • core/pom.xml — re-add evomaster-client-java-sql dependency.
  • EMConfig.kt — re-add measureSqlZ3Correctness experimental option.
  • Statistics.kt — re-add correctness-distance counters and reportSqlZ3CorrectnessDistance.
  • SMTLibZ3DbConstraintSolver.kt — re-add computeCorrectnessDistance, toQueryResultSet, extractGeneValue, and the post-SAT measurement hook.

Notes

The flag is off by default, so there is no behavioral impact on normal runs; the measurement path is exercised only during correctness experiments.

Revert "Remove Z3 SQL correctness-measurement feature; drop core->client-java-sql dependency"

This reverts commit c8712da545379779d6a27512e7b37b9be953486c.
@agusaldasoro
agusaldasoro force-pushed the dse/sql-z3-correctness branch from a127e27 to 992ce99 Compare July 12, 2026 16:04
@agusaldasoro agusaldasoro changed the title Dse/sql z3 correctness Restore experimental Z3 SQL correctness-measurement feature Jul 12, 2026
@agusaldasoro
agusaldasoro marked this pull request as ready for review July 12, 2026 17:09
@agusaldasoro
agusaldasoro requested a review from arcuri82 July 12, 2026 17:09
Base automatically changed from dse/memoization to master July 20, 2026 19:37
Comment thread core/pom.xml
<dependency>
<groupId>org.evomaster</groupId>
<artifactId>evomaster-client-java-sql</artifactId>
</dependency>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding a dependency from core to a module inside driver is a MAJOR architectural change. we are in the process of reducing dependencies between the 2, not adding. adding this dependency does add all of the other dependencies it relies on, and so recursively. this is major issue that has prevented so far the upgrade of some libraries. unfortunately, we cannot have this, unless we do a major refactoring of that module

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants