Predictive algorithms as assets
One thing I'm most passionate about is that the world would be a much better place if we used more predictions and if we made better predictions. We'd better manage our economies, companies, natural resources, traffic, pollution, our machines, relationships, careers… actually the list of what wouldn't potentially benefit would probably be shorter, partly because I can't think of anything right now.
I don’t mean this statement as in "if only we could make better predictions"… I mean we can make better predictions right now, but we're not investing enough of the right people, time and technology to achieve them. We're grossly underachieving, and the world is missing out, in some cases very badly.
It is of course increasingly widely known that combining the right data, domain understanding, technology and algorithms can enable the generation of new and better predictions. Domain understanding is being somewhat overlooked at present as people realise value from combing more and better data with better algorithms and technologies; remarkably, that can enable you to predict something better without necessarily improving your understanding of why you can predict it better. However that build up and use of understanding can also enable improved predictions and should be formally developed as part of any on-going effort to improve predictions.
All this leaves me predicting the next phase for prediction in society. From advances in tech came advances in the availability of data, as also came advances in our ability to analyse those data and extract insights and predictive abilities. Now it's passé to talk about data as having intrinsic value and being the new oil. As expected, with the proliferation of data and technology and algorithms, we're seeing the proliferation of organisations that specialise in mixing those things together and producing new insights, new predictions, new information. All great! I think next is the proliferation in predictive algorithms as a more widely available asset and service.
Right now the actual process of prediction is hugely hidden away. It so much depends on the data, technology, domain and those who did the magic bringing them all together. However just as now we're getting institutes for open data, for data sharing, for domain-specific data hubs, I think in the future we'll next be getting that for predictive algorithms. For example some sort of open institute for predictive algorithms would provide access to algorithms in a way that enabled the generation of new predictions to be made so much faster and better than before.
I think predictive algorithms need to be represented and recognised as an asset class at least as important as the data, technology and domain understanding used to construct them.
Thanks Martin - I love Kahneman's work but I hadn't drawn those parallels. I see the analogues though - where system 2 is like "let the data decide" whereas 1 might actually be able to throw an idea in there based on hunch and instinct and actually may work. For some process-based models there can be a good number of components that are put in there based on intuitive reasoning, but don't yet have a solid evidence basis. Actually though I think algorithms as assets is more system 2 full stop.... and therein also lies the trap - if you're overly reliant on the algorithms you may fail when you need to capture things that are outwith that. As Kahneman says, Systems 1 and 2 are convenient constructs... but they're not truly distinctive systems, just like different types of models - there are no/few truly process based and all data-driven approaches still have biases in their construction.
Enjoyed the view of algorithms becoming like our current access to Open Data. With regards to prediction models, am I correct in assuming that your viewpoint is similar to the "System 1", "System 2" thinking labelled by Daniel Kahneman? System 1 being from the gut (instinctual and liable to error), whereas System 2 depends on Data and 'experience' - both of which algorithms & ML excel at (without a human lens).