When experts are wrong

I read an interesting book over the holiday weekend by David H. Freedman called Wrong: Why experts keep failing us and how to know when not to trust them and it looks as if experts might be wrong nearly all of the time. But Freedman’s willing to admit he could be wrong about that.

He gives many examples of expert opinion on a wide variety of topics to demonstrate how many ways there are for experts to miss the boat (accidentally and deliberately) and then stops to sum up what people should be looking for when looking at what’s published by the magazines, journals and other media sources. To do that, he highlights the “Typical Characteristics of Less Trustworthy Expert Advice.” (pp 217-220)

The first characteristic: “It’s simplistic, universal, and definitive.” Here’s a good recent example out of the Telegraph — Want to lose weight? Turn off the lights.

Second characteristic: “It’s supported by only a single study, or many small or less careful ones, or animal studies.” If you read the article, you discover that the research done regarding weight loss and dimly lit places was done with mice.

Researchers found mice exposed to a relatively dim light at night over eight weeks had gained a third more weight than those in a standard light-dark cycle.

Laura Fonken,a neuroscientist at Ohio State University, said: “Although there were no differences in activity levels or daily consumption of food, the mice that lived with light at night were getting fatter than the others.”

The findings published online in Proceedings of the National Academy of Sciences suggests there is a “wrong” time to eat.

Weight gain is clearly not just calories in and calories out but also depends on our own body clock or circadian rhythms.

Dr Fonken and colleagues said the mice were not less active or consuming more food – but that they ate at different times.

Will this really turn out to be useful for human beings? In order to figure that out, it’d have to be decently tested with human beings, and not just a few humans beings, and not just by “On Your Honour” surveys or anecdotes, either. And if this turned out to have fuck all to do with weight gain, would anyone ever see a published article announcing that, or will this Telegraph article get quoted ad infinitum even though it’s wildly inaccurate?

Freedman explains the trouble with animal trials (pp. 50-54) when the hope is to extrapolate reliable information about human beings in the process. Mice and rats aren’t humans. Rabbits aren’t humans either. Rabbits get ill if they ingest penicillin. We don’t. How many drugs that failed their animal trials but would have been safe for human use anyway? We’ll never know. Once the animal trial fails, the drug is scrapped. And many drugs that pass animal trials wind up being dangerous or nearly useless for humans anyway.

The third characteristic: “It’s groundbreaking.” I don’t tend to watch the news at night but I caught a few minutes on Thursday because Big Bang Theory was going to be on after. The four talking heads at the desk were marveling over an novel discovery; sound makes food taste different! Holy shit, that’s amazing. No wonder airline food tastes so bad! The BBC coverage of this states,

In a comparatively small study, 48 participants were fed sweet foods such as biscuits or salty ones such as crisps, while listening to silence or noise through headphones.

Meanwhile they rated the intensity of the flavours and of their liking.

In noisier settings, foods were rated less salty or sweet than they were in the absence of background noise, but were rated to be more crunchy.

“The evidence points to this effect being down to where your attention lies – if the background noise is loud it might draw your attention to that, away from the food,” Dr Woods said.

Also in the group’s findings there is the suggestion that the overall satisfaction with the food aligned with the degree to which diners liked what they were hearing – a finding the researchers are pursuing in further experiments.

I bold a couple things in there because the small sample is in line with the second characteristic to watch for. And the other bit is highlighted because I wanted to point out how the study relies on very subjective impressions of perceived taste sensations. How does that translate into accurate testing of a causal link between noise levels and food flavour? How does that work? I’d also wonder if people were somehow primed to assume there’d be flavour differences depending on what they were going to hear or not. The mind can be tricked into bias pretty easily, something else Freedman notes in his book. If anyone wants to buy the article, they can read it and find out for me, please.

The fourth characteristic: “It’s pushed by people or organizations that stand to benefit from its acceptance.” In the book, Freedman reminds readers about research sponsored by tobacco companies and the like that winds up heavily biased toward a positive result. He also states that during G.W. Bush’s term in office, “there were widespread charges of research corruption…which by most (but not all) accounts saw research spending in part as a means to advance an ideological agenda.” (pg 219) It also put a lot of lives at risk.

On September 13, just two days after the terror attack, the EPA announced that asbestos dust in the area was “very low” or entirely absent. On September 18 the agency said the air was “safe to breathe.” In fact, more than 25 percent of the samples collected by the EPA before September 18 showed presence of asbestos above the 1 percent safety benchmark. Among outside studies, one performed by scientists at the University of California, Davis, found particulates at levels never before seen in more than 7,000 similar tests worldwide. A study being performed by Mt. Sinai School of Medicine has found that 78 percent of rescue workers suffered lung ailments and 88 percent had ear, nose and throat problems in the months following the attack and that about half still had persistent lung and respiratory illnesses nine months to a year later.

The last characteristic: “It’s geared toward preventing a future occurrence of a prominent recent failure of crisis.” I doubt I can find a decent example of that but I suppose I can at least point to a report in The American Spectator:

The most important fact to take from the September unemployment report released last week is that almost three years after the recession began the economy was still losing jobs! Almost 100,000 (95,000) additional jobs were lost last month from the economy overall. That makes 400,000 jobs lost since May. Moreover, in a regular annual benchmark revision to calibrate unemployment rates for updated data, the BLS reported a further 366,000 jobs lost for March. The total number of Americans unemployed stands at almost 15 million (14.8).

Whatever expert advice might have been floating around in the wake of the “economic downturn” (or whatever is the politically correct way to describe that fucked up business) clearly wasn’t very useful. And Freedman notes in his book as well that often excellent research and advice and correct information winds up ignored and brushed aside because the answer is a negative one, or at least a less positive end result than people might want to hear from their media sources. There must have been a few experts in relevant fields that saw this coming or guessed right or whatever but were unable to find publishers willing to print what they had to say.

To finish this up, I guess it has to be said that Freedman’s hoping to raise awareness of the risk of taking expert advice at face value. He’s not saying never trust experts; he’s saying be skeptical of what gets reported and the reasons why. From page 267:

the fact that experts may duel over this matter–and that neither side can offer iron-clad proof–doesn’t mean you are helpless to judge or even to come to at least a provisional conclusion. Look over the evidence, gauge the quality and breadth of support enjoyed by the different arguments, weigh the likely biases of the claimants, consider the qualifications and limitations to which each side admits or fails to admit, and take your best shot at deciding. No expert could do better.

It takes time and effort, but everything worthwhile does.

Advertisements

About 1minionsopinion

Canadian Atheist Basically ordinary Library employee Avid book lover Ditto for movies Wanna-be writer Procrastinator
This entry was posted in Awareness Issues, books, In the Media, skepticism and tagged , , , , , . Bookmark the permalink.