Bias Is a Big Problem. But So Is ‘Noise.’

I have to pass on the strong dose of sanity offered by Daniel Kahneman and his colleagues in a recent NYTimes guest essay. They make some elemental distinctions that are important to keep in mind. Some edited clips:

A bias is any predictable error that inclines your judgment in a particular direction (for instance against women or in favor of Ivy League graduates, or when forecasts of sales are consistently optimistic or investment decisions overly cautious).

There is another type of error that attracts far less attention: noise. While bias is the average of errors, noise is their variability. In a 1981 study, for example, 208 federal judges were asked to determine the appropriate sentences for the same 16 cases...The average difference between the sentences that two randomly chosen judges gave for the same crime was more than 3.5 years. Considering that the mean sentence was seven years, that was a disconcerting amount of noise...In 2015, we conducted a study of underwriters in a large insurance company. Forty-eight underwriters were shown realistic summaries of risks to which they assigned premiums, just as they did in their jobs...the typical difference we found between two underwriters was an astonishing 55 percent of their average premium.

Where does noise come from? ...irrelevant circumstances can affect judgments...a judge’s mood, fatigue and even the weather can all have modest but detectable effects on judicial decisions. Another source is general tendencies...There are “hanging” judges and lenient ones...a third source is different patterns of assessment (say, which types of cases they believe merit being harsh or lenient about). Underwriters differ in their views of what is risky, and doctors in their views of which ailments require treatment. We celebrate the uniqueness of individuals, but we tend to forget that, when we expect consistency, uniqueness becomes a liability.

Once you become aware of noise, you can look for ways to reduce it. For instance, independent judgments from a number of people can be averaged (a frequent practice in forecasting). Guidelines, such as those often used in medicine, can help professionals reach better and more uniform decisions. As studies of hiring practices have consistently shown, imposing structure and discipline in interviews and other forms of assessment tends to improve judgments of job candidates.

No noise-reduction techniques will be deployed, however, if we do not first recognize the existence of noise. Noise is too often neglected. But it is a serious issue that results in frequent error and rampant injustice. Organizations and institutions, public and private, will make better decisions if they take noise seriously.