What is predictive validity, and does it matter?

Strength Scope, Belbin, Discovery Insights, MBTI, Hogan, Thomas International HPTI, Saville Wave, DISC, NEO PI-R, FIRO-B, SEI EQ, the list goes on. Personality tests, psychometric assessments, behaviour profiling tools - there are lots of names to describe these different tests. In some ways, this is the golden age of profiling tools with a rich and wide market and some massive commercial success stories.

For me there is always one key question – can the tool demonstrate predictive validity? I’m quite skeptical by nature and a qualified engineer before becoming interested in psychology, so maybe it’s the engineer in me wanting to understand how things work, or why things are done a certain way, or what the point of it all is… and it’s these questions that lead to predictive validity.

 What is it?

Research psychologists have lots of ways of looking at profiling tools. Reliability – how well a test measures the same thing when repeated. Face validity – how well the audience buys into the relevance of the test. But what makes predictive validity important? What is it? I’ve taken a few of the textbook definitions and written them in plain English:

‘The connection between what is measured and what it tells us.’

or

‘The degree to which test results can predict future outcomes.’

I also quite like thinking about it as the connection between what is tested and what is then suggested:

·      Let’s say you’ve lost your car keys, would measuring your height suggest where they could be?

·      Or perhaps your laptop is behaving strangely, would submerging it in a bath suggest if it has a virus?

Ridiculous hu? Because the tests – height or water resistance – do not predict anything relevant to the situation – key location, or virus infection.

Instinctively human beings seem to know that predictive validity is important and how it works. In everyday life we rely on selecting relevant tests to give us information about future probabilities.

Predictive validity and profiling tools 

Here’s the simplest profiling tool I can think of:

1.      Choose a number between 1 to 100.

2.      If you chose 50 or over, you think of yourself as having a sweet tooth and enjoy ice cream.

3.      If you chose 49 or below, you often notice in restaurants you pick more savoury foods than other people.

What was measured? Choice of number.

What was suggested? Sweet or savoury preference.

What behaviour is logically implied? 50+ you will eat more sweet food, 49 or below you will eat more savoury food.

Does this test predict sweet/savoury food consumption? We could do the proper analysis with control groups, observations etc. We could also apply some heavy statistical tools like factor analysis, but it’s just an arbitrary choice from 1 to 100, I’d be amazed if there is any kind of connection to food consumption, and hence any kind of predictive validity.

So, it is possible to answer questions and for those answers to have no relationship with what you are likely to do, or not do.

Unfortunately, psychologists and consultants further downstream in industry have a bit of a chequered past deploying tests and instruments that have no connection with what people are likely to do. Graphology is the analysis of handwriting with a view to making suggestions about people’s personalities, behaviours and ultimately job performance. There has yet to be any convincing empirical evidence of predictive validity. So analysing handwriting does not help when talking about the likelihood of any future behaviour, but it was popular for a while and persists in some corners of industry and consulting.

Imagine being told about areas of personal development because someone has looked at your handwriting? (Apart from the fact my handwriting is nearly illegible and being asked to tidy it up would be fair feedback.)

Conclusion

Most people might be quite skeptical towards Graphology in 2019, but would you be shocked to know that the most popular and commercially successful personality profiling tool of all time does not demonstrate predictive validity?

Or would you be shocked to know the fastest growing profiling tool in the UK does not demonstrate predictive validity?

If a test does not demonstrate predictive validity…why use it?

Enjoyed this article? Please hit like, leave a comment and share

About the author: William McKee is a consultant, trainer, and coach who helps businesses and organisations build strong teams, world-class managers and inspirational leaders.