The Future of Decisions
We make decisions every moment of our lives – based on our values, intuition and insight. While we think carefully about important decisions, is there a better way?
Research shows that we tend to make a decision and then rationalise to ourselves – we can find flaws in our rationalisation sometimes but we are engineered to find a way to justify our choices, so our tendency is to stick with our original point of view.
In most cases our insight is fine – yet there are numerous examples where our intuition does not work. We are particularly challenged when using our intuition for complex decisions – our natural understanding of statistical significance is far from perfect.
There are many tools available to help us make decisions – for example SWOT analysis, Pareto analysis, thinking hats, decision matrices – these are aids to help us think about the decision and give us some insight that may challenge our intuition.
In this digital age we have an opportunity to make better use of data and computing to help us with our decision making. Yet this will be a radical shift in how we think about decision making – we will need to understand data, and understand how insight from data can inform our decision making. The techniques are mathematically more challenging for us to understand.
The first technique – Bayesian Inference – was developed many years ago and gives a mechanism to develop probabilities for a certain outcome based on statistical analysis. The approach is to start with an ‘a-priori’ assessment of the outcome (which is subjective). We then do research/gather data about the decision, and apply the results of the analysis to our decision probabilities to re-assess our position. The benefit of the approach is that multiple factors can be assessed and applied to our decision in a way that is likely to result in an non-intuitive answer.
The second technique – Causal Reasoning – is an evolution of both Bayesian Inference and statistical correlations. It draws on the insight that statistics does not model cause – and we all know the risk of correlation implying causation. Yet correlation may have an underlying rational causation and it is this causation that can give us deeper insight. Developing a causal network that is based in some rational deduction about the nature of cause and effect will allow us to develop better insight into data and thus inform our decisions.
There may still be some subjectivity in the assessment and indeed there can be un-intended bias in the data we use. The model we choose for the casual network may be flawed, or our choice of data for analysis maybe both flawed and biased. Hence these techniques are no panacea that take away our accountability for making decisions.
Artificial Intelligence research is starting to embrace these ideas, as is research into consciousness and our models of human intelligence - understanding how we think will no doubt improve our ability to augment our thinking.
These new approaches for making decisions will need us to embrace data and statistical analysis with casual or Bayesian models. There is still a strong human component, yet we can make better use of the data and our insight into how the world works to improve our decision making. Governments and companies have yet to embrace these techniques – but those that do are likely to make better decisions. Deep analysis of critical issues can only bring value and quality to our decisions – even if we choose to do something different we do so with knowledge and insight.
References
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3150852/
https://ieeexplore.ieee.org/document/4680669/
https://en.wikipedia.org/wiki/Decision_theory
https://en.wikipedia.org/wiki/Bayesian_inference
Carl One decision should remain with the people though - the boundary at which government agencies stop collecting and processing private citizen data to achieve better decisions which ostensibly are all for the public good. The ethical debate will lag well behind the technology and indeed some rogue operators willing to push that boundary.
What is the optimum height of a binary decision tree?
How do we integrate the human element with the technology-based models? Do we abdicate and live with the equivalent of Video assisted Referee in Football? As an adviser, do I give up my judgement to include machine assistance in decision-making? Anyone has thoughts on my dilemma?
This article got me thinking about the synergizing of casual reasoning and bayesian inference in modern reinforcement learning models. Governments (e.g. South Korea, China, Singapore) that are embracing AI or incorporating digital culture can soon reap the rewards of their efforts, with increasingly mature Machine Learning architecture and rich datasets. Excited to see how well digital solutions from the academic and private sector flow into the public sector in the coming years, or vice versa.