14.3. Benchmark Report

14.3.1. HTML Report

After running a benchmark, an HTML report will be written in the benchmarkDirectory with the index.html filename. Open it in your browser. It has a nice overview of your benchmark including:
  • Summary statistics: graphs and tables
  • Problem statistics per inputSolutionFile: graphs and CSV
  • Each solver configuration (ranked): Handy to copy and paste
  • Benchmark information: settings, hardware, ...
Note
Graphs are generated by the excellent JFreeChart library.
The HTML report will use your default locale to format numbers. If you share the benchmark report with people from another country, consider overwriting the locale accordingly: 
<plannerBenchmark>
  ...
  <benchmarkReport>
    <locale>en_US</locale>
  </benchmarkReport>
  ...
</plannerBenchmark>

14.3.2. Ranking The Solvers

The benchmark report automatically ranks the solvers. The Solver with rank 0 is called the favorite Solver: it performs best overall, but it might not be the best on every problem. It's recommended to use that favorite Solver in production.
However, there are different ways of ranking the solvers. Configure it like this: 
<plannerBenchmark>
  ...
  <benchmarkReport>
    <solverRankingType>TOTAL_SCORE</solverRankingType>
  </benchmarkReport>
  ...
</plannerBenchmark>
The following solverRankingTypes are supported:
  • TOTAL_SCORE (default): Maximize the overall score, so minimize the overall cost if all solutions would be executed.
  • WORST_SCORE: Minimize the worst case scenario.
  • TOTAL_RANKING: Maximize the overall ranking. Use this if your datasets differ greatly in size or difficulty, producing a difference in Score magnitude.
Solvers with at least one failed single benchmark do not get a ranking. Solvers with not fully initialized solutions are ranked worse.
You can also use a custom ranking, by implementing a Comparator: 
  <benchmarkReport>
    <solverRankingComparatorClass>...TotalScoreSolverRankingComparator</solverRankingComparatorClass>
  </benchmarkReport>
Or by implementing a weight factory: 
  <benchmarkReport>
    <solverRankingWeightFactoryClass>...TotalRankSolverRankingWeightFactory</solverRankingWeightFactoryClass>
  </benchmarkReport>