Why suboptimisation can outperform optimality
But we should not suboptimise... That is really bad!!!
Ever heard a statement like that? Or maybe you even have said it yourself. I have faced it so many times as an Agile Coach or Product Owner.... This can be in questions like how to divide the organisation, or how to decide what to prioritise. Really complex problems where finding the optimal solution is really, really hard. So I want to share my nerdy perspective - as an old optimisation expert - on optimality and suboptimality.
The optimal solution is just an illusion
In my world, an optimal solution to a complex problem is an illusion. It can not be found. A true search for the optimal solution will make you stuck forever, and you may risk not finding any decent solution at all instead. Analysis paralysis. The fear of missing the absolutely best option. But the mathematical truth is: For complex problems you can't create a strategy for finding the optimal solution in a reasonable time. You can only find some solutions, and rely on luck for finding optimal solution.
Use (sub)optimisation to your advantage
So rely on luck??? Still some persons/organisations just seems to have more luck than others. They do better. Constantly. Why?
I think they are better at decision making, and they are not even trying to find the optimal solution. They do instead know how to (sub)optimise so they can do millions of good - not best - decisions quickly. They know how to work with (sub)optimisation techniques to their advantage, this could be:
- Divide and conquer / Bucketing techniques (Business/architectural/Organisational design supporting local decisions)
- Heuristics, like only consider few good alternatives (many local decisions and many immediate no:s)
- Iterative approaches (inspect and adapt)
Which all - unless extremely lucky - will lead to a suboptimal solution...
So a quick good suboptimal solution may not be that bad after all
The question is not if we should suboptimise - the question is how.
Most optimisation techniques are only possible to use when you drop the fear of not finding the optimal solution. So instead many (sub)optimisation techniques aims at finding the best solution - from just some few considered alternatives - within in a limited time.
So this is why I believe suboptimality outperforms optimality: quick good decisions outperforms deadly slow perfect decisions.
Therefore, finding the right (sub)optimisation techniques is not a question to me - it is to 'optimise the whole'. Especially when responsiveness and adaptivity is important. (Sub)optimisation always comes with a cost, but it pays off multiple times if you do it right.
So what is your experience: - Are we striving too much for the optimal solution? - Do you think we can move the discussion from suboptimise or not, to how we do it wisely? And maybe, if time permits, I will write about 'healthy silo:s' one day...
Great point! It is also easy to talk about optimum without explicitly agree on what the goal, or target function is. Speed of decision should for example, like you say, often be included. Defining or modeling the problem to be solved is usually more important than to find the optimum.
It depends entirely on the problem (or on the objective function). If you are trying to minimize the cost of a huge production plant, then every single variable carries a huge monetary cost. Hence, exact optimization is the goal. On the other hand, many resource planning problems have more or less fictive cost functions, where preferences are expressed as costs. Here, suboptimal fast solution methods are the way to go!
I like the perspective, it allows for a more pragmatic approach.