Benchmark - Best Score Graph is empty

[simontiffert]

Describe the bug
When running the benchmark in version 1.22.1 the Best Score graph stays empty. If I revert to 1.21.0 I can see the Best Score Graph again.

Looking into BEST_SCORE.csv I can see it is related to the initialized column where every value is initialized: "false":
"timeMillisSpent","score","initialized"
106,"-5hard/-541soft","false"

Probably related to

timefold-solver/benchmark/src/main/java/ai/timefold/solver/benchmark/impl/statistic/bestscore/BestScoreSubSingleStatistic.java

Line 35 in 2a71f1b

.add(new BestScoreStatisticPoint(timestamp, score, subSingleBenchmarkResult.isInitialized()))));

Expected behavior
Best Score Graph (every Score type)

Actual behavior
No Best Score Graph (every Score type)

To Reproduce
Run the benchmark with 1.22.1

Environment

Timefold Solver Version or Git ref:
1.22.1

[triceo]

Hello Simon, and thanks for reporting!
Would you mind sharing what the CSV looked like before, in 1.21.0?

You are probably correct in your assessment of where the problem comes from, but the change in question should in theory be equivalent to what was there before. Therefore I am curious to understand your data.

[simontiffert]

In before the column initialized is missing as it just contains the score - I used the school timetabling of the Timefold Quickstarts to reproduce it, but it is an issue in several of my projects with the latest version:

Community Edition v1.21.0

"timeMillisSpent","score"
90,"-5hard/-541soft"
97,"-5hard/-540soft"

vs:

Community Edition v1.22.1

"timeMillisSpent","score","initialized"
118,"-5hard/-541soft","false"
127,"-5hard/-540soft","false"

With additional library:

    <dependency>
      <groupId>ai.timefold.solver</groupId>
      <artifactId>timefold-solver-quarkus-benchmark</artifactId>
    </dependency>

And the test:

@QuarkusTest
public class BenchmarkTest {
    @Inject
    PlannerBenchmarkFactory benchmarkFactory;

    @Test
    public void benchmark() {
        benchmarkFactory
                .buildPlannerBenchmark(TimetableDemoResource.generate(TimetableDemoResource.DemoData.LARGE))
                .benchmarkAndShowReportInBrowser();
    }
}

[triceo]

@simontiffert I opened a PR which should fix the issue: #1636

Would you mind double-checking?
Building the solver from main and running your benchmark should be enough.

[simontiffert]

I can confirm that the locally build main (Community Edition v999-SNAPSHOT) fixed the benchmarker issue in my project.

The written file BEST_SCORE.csv now also contains an initialized score:

"timeMillisSpent","score","initialized"
705,"-330hard/-17324medium/-5912soft","true"
739,"-320hard/-17332medium/-5912soft","true"
740,"-320hard/-17331medium/-5922soft","true"

Thank you for the fix. Looking for the upcoming release.