This article was taken from the March 2016 issue of Wired magazine. Be the first to read Wired's articles in print before they're posted online, and get your hands on loads of additional content by subscribing online.
Consider the following story. A football coach in the US had heard of a great player at a local school. The player had powered his team to a strong win/loss record and his coaches thought he was one of the most talented players they had seen in some time. But the coach observed the player at a practice game and was unimpressed; the player made several errors and was not in full control of the ball. The coach told his colleagues that it wouldn't be worthwhile to try to recruit the player.
Doesn't sound quite sensible, does it? Most sports fans would think that was a pretty foolish call. Athletic performance is much too variable across occasions to base an important judgement on such a small sample. But now consider this problem: an employer gets an application from a junior executive who has to leave his company because his partner has to move city for work. The applicant has an excellent college record, had been rapidly promoted in his current firm and has strong references from his present boss. But the prospective employer interviewed the applicant and was unimpressed. He didn't seem to be terribly bright and his responses to hypothetical management situations were somewhat uninspired. The employer tells his colleagues that it wouldn't be worthwhile to try recruiting the applicant.
I find that most people regard this as a reasonable sort of decision. But it isn't. Countless studies show that the unstructured 30-minute interview is almost worthless as a predictor of school or college performance, effectiveness as a physician or corporation executive, success in army-officer training or any other long-term performance criterion. The correlations run in the vicinity of 0.10, which isn't much better than a flip of the coin. But in each of these cases, predictions based on a summary of information in the folder -- grade-point average, ability-test scores, previous job performance, letters of recommendation -- commonly are correlated 0.4 to 0.5 with performance. The difference between 0.52 and 0.65 could easily be the difference between success for an enterprise and its failure.
But we might as well interview anyway, right? Wrong. We are incapable of treating interview data as having little or no value. There's lots of psychological theory and data backing up this claim. The upshot is that my recommendation is not to interview at all -- or to devise with the help of an expert a highly structured interview in which every candidate is asked the same questions. Such interviews have some predictive validity and are worth adding into the stew of information used to make a choice.
Why do we get the athletic problem right but the employment one wrong? Because, unlike athletic performance, we haven't seen hundreds of candidates in interviews of a particular type and seen how well performance in the interview corresponds to performance over the long haul in the setting we're concerned about. We haven't seen how common it is that the guy who looks like a dunce in the interview often turns out to be a whizz on the job and the guy who aced the interview turns out to be a dud. Nor do we have systematic observations that could tell us how common it is that the person with the more impressive folder is actually more likely to turn in a good performance than someone with a weaker record.
The only way to see that the interview isn't going to be worth much is to be able to apply the law of large numbers. This principle specifies that as a sample size gets larger you get closer to true population values, such as mean or proportion. This prompts the recognition that an interview represents a very small sample of behaviour whereas the contents of the folder summarise a lot of accumulated behaviour.
How can we make it more likely that we'll understand the relevance of the law of large numbers for problems where the data is hard to code or difficult to examine systematically, such as a one-shot observation of a given event or behaviours reflecting personality traits such as friendliness or honesty? One tip: think of all evidence about an event of a given kind as a sample from a population of similar events. That can prompt the realisation that we don't really have much evidence. Even better, it can alert us to the possibility that it may be biased in some way.
The job interview is not only a tiny sample; it's not even a sample of job behaviour but of something else entirely. Extroverts in general do better in interviews than introverts do, but for many if not most jobs that's not what we're looking for.
Richard Nisbett is a professor of social psychology at the University of Michigan and author of Mindware: Tools for Smart Thinking (Allen Lane)
This article was originally published by WIRED UK
