6.2. Does Planner Find the Optimal Solution?

The business wants the optimal solution, but they also have other requirements:

  • Scale out: Large production datasets must not crash and have good results too.
  • Optimize the right problem: The constraints must match the actual business needs.
  • Available time: The solution must be found in time, before it becomes useless to execute.
  • Reliability: Every dataset must have at least a decent result (better than a human planner).

Given these requirements, and despite the promises of some salesmen, it’s usually impossible for anyone or anything to find the optimal solution. Therefore, Planner focuses on finding the best solution in available time. In realistic, independent competitions, it often comes out as the best reusable software.

The nature of NP-complete problems makes scaling a prime concern. The result quality of a small dataset guarantees nothing about the result quality of a large dataset. Scaling issues cannot be mitigated by hardware purchases later on. Start testing with a production sized dataset as soon as possible. Don’t assess quality on small datasets (unless production encounters only such datasets). Instead, solve a production sized dataset and compare the results of longer executions, different algorithms and—​if available—​the human planner.