14.5. Statistic Per Dataset (Graph And CSV)

14.5.1. Enable A Problem Statistic

The benchmarker supports outputting problem statistics as graphs and CSV (comma separated values) files to the benchmarkDirectory. To configure one, add a problemStatisticType line: 

<plannerBenchmark>
  <benchmarkDirectory>local/data/nqueens/solved</benchmarkDirectory>
  <inheritedSolverBenchmark>
    <problemBenchmarks>
      ...
      <problemStatisticType>BEST_SCORE</problemStatisticType>
      <problemStatisticType>CALCULATE_COUNT_PER_SECOND</problemStatisticType>
    </problemBenchmarks>
    ...
  </inheritedSolverBenchmark>
  ...
</plannerBenchmark>

Multiple problemStatisticType elements are allowed.

Note

These statistic per dataset can slow down the solver noticeably, which affects the benchmark results. That’s why they are optional and not enabled by default.

The non-optional summary statistics cannot slow down the solver noticeably.

The following types are supported:

14.5.2. Best Score Over Time Statistic (Graph And CSV)

To see how the best score evolves over time, add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>BEST_SCORE</problemStatisticType>
    </problemBenchmarks>

Figure 14.3. Best Score Over Time Statistic

bestScoreStatistic
Note

A time gradient based algorithm (such as Simulated Annealing) will have a different statistic if it’s run with a different time limit configuration. That’s because this Simulated Annealing implementation automatically determines its velocity based on the amount of time that can be spent. On the other hand, for the Tabu Search and Late Annealing, what you see is what you’d get.

The best score over time statistic is very useful to detect abnormalities, such as a potential score trap which gets the solver temporarily stuck in a local optimum.

letTheBestScoreStatisticGuideYou

14.5.3. Step Score Over Time Statistic (Graph And CSV)

To see how the step score evolves over time, add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>STEP_SCORE</problemStatisticType>
    </problemBenchmarks>

Figure 14.4. Step Score Over Time Statistic

stepScoreStatistic

Compare the step score statistic with the best score statistic (especially on parts for which the best score flatlines). If it hits a local optimum, the solver should take deteriorating steps to escape it. But it shouldn’t deteriorate too much either.

Warning

The step score statistic has been seen to slow down the solver noticeably due to GC stress, especially for fast stepping algorithms (such as Simulated Annealing and Late Acceptance).

14.5.4. Calculate Count Per Second Statistic (Graph And CSV)

To see how fast the scores are calculated, add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>CALCULATE_COUNT_PER_SECOND</problemStatisticType>
    </problemBenchmarks>

Figure 14.5. Calculate Count Per Second Statistic

calculateCountPerSecondStatistic
Note

The initial high calculate count is typical during solution initialization: it’s far easier to calculate the score of a solution if only a handful planning entities have been initialized, than when all the planning entities are initialized.

After those few seconds of initialization, the calculate count is relatively stable, apart from an occasional stop-the-world garbage collector disruption.

14.5.5. Best Solution Mutation Over Time Statistic (Graph And CSV)

To see how much each new best solution differs from the previous best solution, by counting the number of planning variables which have a different value (not including the variables that have changed multiple times but still end up with the same value), add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>BEST_SOLUTION_MUTATION</problemStatisticType>
    </problemBenchmarks>

Figure 14.6. Best Solution Mutation Over Time Statistic

bestSolutionMutationStatistic

Use Tabu Search—​an algorithm that behaves like a human—​to get an estimation on how difficult it would be for a human to improve the previous best solution to that new best solution.

14.5.6. Move Count Per Step Statistic (Graph And CSV)

To see how the selected and accepted move count per step evolves over time, add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>MOVE_COUNT_PER_STEP</problemStatisticType>
    </problemBenchmarks>

Figure 14.7. Move Count Per Step Statistic

moveCountPerStepStatistic
Warning

This statistic has been seen to slow down the solver noticeably due to GC stress, especially for fast stepping algorithms (such as Simulated Annealing and Late Acceptance).

14.5.7. Memory Use Statistic (Graph And CSV)

To see how much memory is used, add: 

    <problemBenchmarks>
      ...
      <problemStatisticType>MEMORY_USE</problemStatisticType>
    </problemBenchmarks>

Figure 14.8. Memory Use Statistic

memoryUseStatistic
Warning

The memory use statistic has been seen to affect the solver noticeably.