A Neuroscientist On How to Tell Real Experts From ‘Experts’

By Daniel J. Levitin, Ph.D

The first thing to do when evaluating a claim by some authority is to ask who or what established their authority. If the authority comes from having been a witness to some event, how credible a witness are they?

Venerable authorities can certainly be wrong. The U.S. government was mistaken about the existence of weapons of mass destruction (WMDs) in Iraq in the early 2000s, and, in a less politically fraught case, scientists thought for many years that humans had twenty-four pairs of chromosomes instead of twenty-three. Looking at what the acknowledged authorities say is not the last step in evaluating claims, but it is a good early step.

Experts talk in two different ways, and it is vital that you know how to tell these apart. In the first way, they review facts and evidence, synthesizing them and forming a conclusion based on the evidence. Along the way, they share with you what the evidence is, why it’s relevant, and how it helped them to form their conclusion. This is the way science is supposed to be, the way court trials proceed, and the way the best business decisions, medical diagnoses, and military strategies are made.

The second way experts talk is to just share their opinions. They are human. Like the rest of us, they can be given to stories, to spinning loose threads of their own introspections, what-ifs, and untested ideas. There’s nothing wrong with this — some good, testable ideas come from this sort of associative thinking — but it should not be confused with a logical, evidence-based argument. Books and articles for popular audiences by pundits and scientists often contain this kind of rampant speculation, and we buy them because we are impressed by the writer’s expertise and rhetorical talent. But properly done, the writer should also lift the veil of authority, let you look behind the curtain, and see at least some of the evidence for yourself.

The term expert is normally reserved for people who have under‑ taken special training, devoted a large amount of time to develop‑ ing their expertise (e.g., MDs, airline pilots, musicians, or athletes), and whose abilities or knowledge are considered high relative to others’. As such, expertise is a social judgment — we’re comparing one person’s skill to the skill level of other people in the world. Expertise is relative. Einstein was an expert on physics sixty years ago; he would probably not be considered one if he were still alive today and hadn’t added to his knowledge base what Stephen Hawking and so many other physicists now know. Expertise also falls along a continuum. Although John Young is one of only twelve people to have walked on the moon, it would probably not be accurate to say that Captain Young is an expert on moonwalking, although he knows more about it than almost anyone else in the world.

Individuals with similar training and levels of expertise will not necessarily agree with one another, and even if they do, these experts are not always right. Many thousands of expert financial analysts make predictions about stock prices that are completely wrong, and some small number of novices turn out to be right. Every British record company famously rejected the Beatles’ demo tape, and a young producer with no expertise in popular music, George Martin, signed them to EMI. Xerox PARC, the inventors of the graphical interface computer, didn’t see any future for personal computers; Steve Jobs, who had no business experience at all, thought they were wrong. The success of newcomers in these domains is generally understood to be because stock prices and popular taste are highly unpredictable and chaotic. Stuff happens. So it’s not that experts are never wrong, it’s just that, statistically, they’re more likely to be right.

Many inventors and innovators were told “it will never work” by experts, with the Wright brothers and their fellow would‑be inventors of motorized flight being an example par excellence. The Wright brothers were high school dropouts, with no formal training in aeronautics or physics. Many experts with formal training declared that heavier-than-air flight would never be possible. The Wrights were self-taught, and their perseverance made them de facto experts themselves when they built a functional heavier-than-air airplane, and proved the other experts wrong. Michael Lewis’s baseball story Moneyball shows how someone can beat the experts by rejecting conventional wisdom and applying logic and statistical analysis to an old problem; Oakland A’s manager Billy Beane built a competitive team by using player performance metrics that other teams undervalued, bringing his team to the playoffs two years in a row, and substantially increasing the team’s worth.

Experts are often licensed, or hold advanced degrees, or are recognized by other authorities. A Toyota factory-certified mechanic can be considered an expert on Toyotas. The independent or self-taught mechanic down the street may have just as much expertise, and may well be better and cheaper. It’s just that the odds aren’t as good, and it can be difficult to figure that out for yourself. It’s just averages: The average licensed Toyota mechanic is going to know more about fixing your Toyota than the average independent. Of course, there are exceptions and you have to bring your own logic to bear on this. I knew a Mercedes mechanic who worked for a Mercedes dealership for twenty-five years and was among their most celebrated and top-rated mechanics. He wanted to shorten his commute and be his own boss so he opened up his own shop. His thirty- five years of experience (by the time I knew him) gave him more expertise than many of the dealer’s younger mechanics. Or another case: An independent may specialize in certain repairs that the dealer rarely performs, such as transmission overhaul or reupholstering. You’re better off having your differential rebuilt by an independent who does five of those a month than a dealer who probably only did it once in vocational school. It’s like the saying about surgeons: If you need one, you want the doctor who has performed the same operation you’re going to get two hundred times, not once or twice, no matter how well those couple of operations went.

In science, technology, and medicine, experts’ work appears in peer-reviewed journals (more on those in a moment) or on patents. They may have been recognized with awards such as a Nobel Prize, an Order of the British Empire, or a National Medal of Science. In business, experts may have had experience such as running or starting a company, or amassing a fortune (Warren Buffett, Bill Gates). Of course, there are smaller distinctions as well — salesperson of the month, auto mechanic of the year, community “best of” awards (e.g., best Mexican restaurant, best roofing contractor).

In the arts and humanities, experts may hold university positions or their expertise may be acknowledged by those with university or governmental positions, or by expert panels. These expert panels are typically formed by soliciting advice from previous winners and well-placed scouts — this is how the Nobel and the MacArthur “genius” award nomination and selection panels are constituted.

If people in the arts and humanities have won a prize, such as the Nobel, Pulitzer, Kennedy Center Honors, Polaris Music Prize, Juno, National Book Award, Newbery, or Man Booker Prize, we conclude they are among the experts at their craft. Peer awards are especially useful in judging expertise. ASCAP, an association whose membership is limited to professional songwriters, composers, and music publishers, presents awards voted on by its members; the award is meaningful because those who bestow it constitute a panel of peer experts. The Grammys and the Academy Awards are similarly voted on by peers within the music and film industry, respectively.

You might be thinking, “Wait a minute. There are always elements of politics and personal taste in such awards. My favorite actor/singer/writer/dancer has never won an award, and I’ll bet I could find thousands of people who think she’s as good as this year’s award winner.” But that’s a different matter. The award system is generally biased toward ensuring that every winner is deserving, which is not the same as saying that every deserving person is a winner. (Recall the discussion of asymmetries earlier.) Those who are recognized by bona fide, respectable awards have usually risen to a level of expertise. (Again, there are exceptions, such as the awarding of a Grammy in 1990, which was later retracted, to lip- syncers Milli Vanilli; or the awarding of a Washington Post reporter Janet Cooke, which was withdrawn two days later when it was discovered that the winning story was fraudulent. Novelist Gabriel García Márquez quipped that Cooke should’ve been awarded the Nobel Prize for literature.) When an expert has been found guilty of fraud, does it negate their expertise? Perhaps. It certainly impacts their credibility — now that you know they’ve lied once, you should be on guard that they may lie again.

Expertise Is Typically Narrow

Dr. Roy Meadow, the pediatrician who testified in the case of the alleged baby killer Sally Clark, had no expertise in medical statistics or epidemiology. He was in the medical profession, and the prosecutor who put him on the stand undoubtedly hoped that jurors would assume he had this expertise. William Shockley was awarded a Nobel Prize in physics as one of three inventors of the transistor. Later in life, he promoted strongly racist views that took hold, probably because people assumed that if he was smart enough to win a Nobel, he must know things that others don’t. Gordon Shaw, who “discovered” the now widely discredited Mozart effect, was a physicist who lacked training in behavioral science; people probably figured, as they did with Shockley, “He’s a physicist — he must be really smart.” But intelligence and experience tend to be domain-specific, contrary to the popular belief that intelligence is a single, unified quantity. The best Toyota mechanic in the world may not be able to diagnose what’s wrong with your VW, and the best tax attorney may not be able to give the best advice for a breach‑ of‑contract suit. A physicist is probably not the best person to ask about social science.

Originally published at medium.com