'Noise' book review

Very rarely does a book come along with the potential for genuine disruption. Even more rarely does one come along with implications for the professional services, and in particular insurance legal services. Noise is one of those books.

In the struggle to understand the world around us, arguably great progress has been in understanding the origins of the universe and the operations of subatomic particles, but not so much when it comes to the human condition. Nobody has done more to further our understanding of human decision making than Nobel peace prize winner, Daniel Kahneman. Famous for his book Thinking, Fast and Slow, Kahneman’s work on Cognitive bias has been the inspiration for many related works including Nudge “behavioral economics”, which took the thinking of Daniel Kahneman and Amos Tversky and applied it to public policy. Because of Kahneman we now know that we consistently underestimate the time taken for tasks and projects, that we are affected by reference points or anchors in negotiations, that we prefer avoiding losses to acquiring equivalent gains. Noise builds on these concepts to examine the unwanted divergence of judgement in organisations that employ “interchangeable professionals to make decisions”. That divergence is much more common than we think, it is called noise.

So much of Kahneman’s work, past and present, has been based on studies of the judicial system, partly because of the availability of data but also partly because nowhere else is the consequence of error in human judgement so consequential. Kahneman’s thinking, particularly in his latest collaboration Noise, concerns professional judgement wherever it is found. Judges, lawyers but also insurers, doctors, teachers and engineers use judgement to determine the outcomes of many aspects of our lives. The diagnosis of our illness, the grades in our exams, the cost of our insurance.

Interest from the legal system is not new. Large scale studies of noise in the judicial system were undertaken by judge Frankel in the United States in 1974. 50 judges from various districts were asked to produce sentences for defendants in hypothetical cases concluding the “absence of consensus was the norm”. Simply put, “It is not acceptable for similar people, convicted of the same offense, to end up with dramatically different sentences…”. The book has examples across many disciplines. Wine experts score only 18% of the wines identically”. The variance in underwriting was found to be 55%. For claims adjusting 43%. A senior executive in an insurer estimates the cost of excessive quotes and underpriced contracts to amounts to hundreds of millions of dollars for one organisation alone. Physicians are significantly more likely to prescribe opioids at the end of a long day. Doctors fail to diagnose melanoma from skin biopsies 36% of the time. Breast cancer radiology judgements can be incorrect 50% of the time depending on the radiographer.

The book goes on to describe in detail categories of noise. System noise, as exhibited across professional organisation can be broken down into level noise and pattern noise. Level noise describes the different leniency of judges. Pattern noise relates to the individual values of the judges - one judge might be particularly lenient on shoplifters. But pattern noise also has an occasion noise component - some judges are more lenient after their favourite football team won, doctors tend to “prescribe more opioids in the afternoon”.

Mostly noise in judgements is invisible. One of the seismic messages of the book is that professionals and the organisations that employ them maintain “an illusion of agreement”. We prefer to explain decisions that turn out to be wrong in terms of bias, not noise. And this is also in part because following a period of training professionals go on to make their judgements in isolation - experts are confident in their judgements, they expect that their colleagues will agree with them. Worse “the routines of organizations also tend to ignore or suppress evidence of divergence among experts in their midst…..noise is an embarrassment"

Ultimately, noise creates injustice. Our approach to rules and regulations is perhaps to consider them as the “final stage of the evolution of law” but instead we find that we are ruled by informal ad-hoc, “noisey” judgements putting the discretion of professionals before justice: “noise can be counted as a rights violation, and in general, legal systems all over the world should be making much greater efforts to control noise. One thing that is absolutely clear from Noise, “wherever there is judgment, there is noise, and more of it than you think”. And whilst the public debates focuses on bias in all its forms, noise sits in the background, unnoticed but as this brilliant piece of work shows, equally detrimental.

So what is to be done? Noise has a number of practical recommendations for reducing noise.

One thing to note is that algorithms beat humans when it comes to consistency. The accuracy of our predictive judgement is so low that even really basic models will consistently outperform humans. Equally powerful is the concept of the wisdom of crowds. Aggregated judgements average out the noise so long as they are independent. Equally important, “deliberating groups” actually entrench noise. So we should structure our judgements wherever possible into multiple independent tasks and despite the overwhelming urge we should resist the pull of premature intuitions, something that Noise talks about as the “internal signal of judgement”.