Is your data-driven prediction correct?

We often make predictions about the outcome of some discrete event based on the analysis of some data. It could be a binary outcome like the winner of an election between two candidates or a multinomial outcome like the winner of the football World Cup. We then observe a single (!) realisation of the event (Trump won the election, France won the World Cup) and then decide that a given prediction/model/data was correct or incorrect. As appealing and apparently meaningful this definition of correctness is (actual outcome = predicted outcome) it is in fact useless to determine whether the prediction, and the analysis and data employed was good and should be used again or not. It is so, because it completely ignores the element of “luck” or uncertainty. We then rush to make judgements and to scratch our heads where our model went wrong.

Consider the following game. You flip a coin and if it lands heads I give you a 100 dollars and if it’s tails you give me a 100. We play only once.

You don’t know anything about the coin, so if you assume it’s a fair coin then the expected outcome of this game is zero so depending on your mood and how much you depend on the 100 dollars you may decide to play or not. What if I let you first experiment with the coin. You flip it a million times (so we can ignore sampling error) and the coin lands heads 80% of the time. You reasonably decide that it is a loaded coin and the actual probability of heads is 80% (ignoring again sampling error since you tossed it a million times :)). Now the game has an expected outcome of 60 dollars in your favour. Is your experimenting with the coin, collecting and analysing data leading to a “correct” prediction and should you decide to play expecting to get heads? Who knows? So we play and you toss the coin and it lands tails. You are very annoyed and decide your data-driven forecast and decision was totally wrong. You start thinking whether you should have correlated the outcome of the million tosses you made with the phase of the moon.

I would say, whether the forecast and corresponding decision to play was good or not depends on other things. If you are a complete “regret-minimizer” you should never play the game as you cannot have regret of losing if you don’t play. If you are an expected wealth maximiser but you are also somewhat risk-averse it would have been rational not to play the game if you didn’t know anything about the coin as an expected outcome of zero does not compensate for the risk (in economic lingo, you require a positive risk premium to play such a game). If you are reasonably risk-averse (and not pathologically afraid of loss in particular when 100 dollars are involved), then I would say it is rational to play the game in the case where you have collected good evidence that the game is very much (with 80% probability) in you favour. Again, that doesn’t mean that you will win, and if you lose it doesn’t mean that your data collection and analysis were wrong.

Another point this raises, is that it is important to have data that is as much as possible generated from the same data generating process as the event you are trying to predict. In the above example you get to toss the exact coin which will be used to determine the game - that is a big luxury.

We haven’t observed a million Trump-Clinton elections or a million FIFA 2018 World cups. On the other hand, we do have millions of cats images. No surprise that algorithms can get good at predicting cat images but not one-shot events.