Experts and Expertise
Know of a resource missing from this page?
Let
us know...
The general theme of this page is that experts are generally less expert than we (or they) think; and that we all ought be thinking much more critically about how much expertise the experts actually have. Cochran-Weiss-Shanteau (CWS) Home Page
Experts on expertise. A trove of resources, including articles, FAQ and software. Centred on a particular method of determining to what extent the experts are any good, but there is lots of more general information. [30 Sep 03] EssaysIs the artistic value of Mondrian's art testable? by Alan Lee. Target: art experts.
A clever experiment shows that art experts can't distinguish between Mondrian's paintings and randomly generated pseudo-Mondrians, despite the supposed aesthetic brilliance and profundity of the former. This suggests in turn that much of what those art experts say about the Mondrians is little more than pretentious, hollow bloviation. "My method is the visual art equivalent of the experiment in which you put a monkey at a typewriter to see how long it takes to produce a Shakespearian sonnet...I think that we should change our minds about Mondrian and recognise that his paintings have never really been anything more than aesthetic placebos." [12 Aug 02]
The Talent Myth by Malcolm Gladwell. Target: McKinsey & Co; management consultants.
Another excellent piece from Malcolm Gladwell. The talent myth is not the existence of talent; rather, it is the McKinsey-driven idea that the talent of individuals is the only thing that really matters for the success of an organization. "The broader failing of McKinsey and its acolytes at Enron is their assumption that an organization's intelligence is simply a function of the intelligence of its employees. They believe in stars, because they don't believe in systems...The talent myth assumes that people make organizations smart. More often than not, it's the other way around." [16 July 02]
Do Fingerprints Lie? by Michael Specter. Target: Fingerprint experts and those who rely on them.
Disputes the "scientific" status of fingerprinting. ""The F.B.I. calls it a science. By what definition is it a science? Where are the data? Where are the studies? We know that fingerprint examiners are not always right. But are they usually right or are they sometimes right? That, I am afraid, we don't know. Are there a few people in prison who shouldn't be? Are there many? Nobody has ever bothered to try and find out. Look closely at the great discipline of fingerprinting. It's not only not a science—it should not even be admitted as evidence in an American court of law."" [24 May 02] See also Liars Never Break a Sweat by Robert Park. Opinion For Sale: Confessions of an expert witness by Steven Moss
Interesting expose of what is really happening whenever a supposed expert gets up to testify. "As time went on, the facts started to become less important to me than winning the case. The process consisted of each side insisting that their version of the truth was the correct one, and I soon realized that what was at stake was the potency of my argument more than the reality of the story I was telling. I quickly learned not to worry whether or not I was right." [16 Mar 03]
AcademicsThe Random Insanity of Letters of Recommendation by Deidre McCloskey. Target: almost all academics.
Scathing critique of the practice, in academia, of calling for letters of recommendation in the hiring and promotion process. "In short, the letters are insane. If tables of random numbers became fashionable for deciding on hires, tenurings, promotions, I suppose you, as a serious scholar, would object. And at least a table of random numbers is what it says: random. Why haven't you objected to the system of letters, which has notably less integrity than a table of random numbers?" [1 Mar 02]
Peer
Review: Some Questions from Socrates by Christopher Martyn (pdf file)
Light-hearted yet deeply cynical introduction to scientific peer review and its
problems, cast as a dialogue between Socrates and the editor of a major medical
journal. "Socrates: Let me try to sum up what you have told me so
far. Judging the quality of manuscripts submitted to scientific journals is a
difficult and demanding task. Editors often entrust this task to people who
haven’t had any training in how to do it and, as your study showed, they may not
be very good at it..." What was the study? The editor explains: "Last year we
took a paper that we had agreed to publish but which had not yet appeared in
print. We deliberately inserted some errors of method, analysis, and
interpretation into the manuscript and sent the modified version out to 400
people on our database of reviewers to see how good they were at detecting the
faults. Socrates: And what did you discover?... Were your reviewers
astute in picking up the faults that you had embedded in the paper? Editor:
I’m afraid that they weren’t. We inserted eight errors into the paper. A few
reviewers commented on four or five, more managed to identify one or two, but
quite a lot didn’t detect any." Read the
original BMJ
paper reporting the study. The interesting part about this isn't so
much that peer reviewers are suprisingly bad at the task. After all, as
Martyn points out, they're not trained for the job. The interesting part
is the gap between peer reviewer's perceptions of their expertise and their
actual expertise. Most academics I've known have deemed themselves
perfectly competent to pass judgment in peer review. [21 Mar 04]
Medical Professionals
The Bell Curve
by Atul Gawande
As you'd expect, medical experts and programs turn out to fit a Bell Curve -
some are terrible, most are average, and some are outstanding. But how do
the truly excellent ones get - and stay - that way? Fascinating essay on
the pursuit of excellence. [4 Dec 04] Psychologists and Psychotherapists
Mind Games: Psychological Warfare Between Therapists and Scientists By Carol Tavris
Powerful critique of the psychotherapeutic "profession," arguing that therapists' scientific illiteracy can make them positively dangerous. At the heart of their incompetence is the failure to appreciate the cognitive bias known as confirmation bias, and to understand those relatively rudimentary aspects of the scientific method designed to overcome it. [3 Mar 03] The Rorschach Inkblot Test, Fortune Tellers, and Cold Reading by James Wood et al.
"Famous clinical psychologists used the Rorschach Inkblot Test to arrive at incredible insights. But were the astounding performances of these Rorschach Wizards merely a variation on astrology and palm reading?... "Excerpted by the authors from their book What's Wrong With the Rorschach? Science Confronts the Controversial Inkblot Test." [5 Nov 03]
Investment ProfessionalsStock analysts, stock brokers, fund managers, etc.
The most bogus expertise of any serious profession.
A Random Walk Down Wall Street
by Burton G. Malkiel. Target: Investment professionals.
Are Security Analysts
Fundamentally Clairvoyant? Malkiel's book is a classic debunking of the profession. [7 Sep 02]
The Complete Guide to Wall Street
Self-Defense by Henry Blodget
Series of excellent articles in Slate magazine by a savvy ex-Wall Streeter.
See especially "Smart? Skillful?
Probably Just Lucky: The (vast and unappreciated) role of luck in
investing." - one of the best discussions I've seen of the enormously important
point that any display of apparent skill there is some element of luck, and that
luck should be subtracted before evaluating or explaining the performance. [19
Sep 04] Stock market traders show signs of zero intelligence by Philip Ball, in Nature Science Update
A model making random decisions behaves just like a real stock market. So, what is the evidence that investment professionals' experience and (supposed) insight is helping them make better-than-random choices? [14 Oct 03] 
Cartoon by Nicholson of "The Australian" newspaper: www.nicholsoncartoons.com.au
Wine ExpertsThe Great and Powerful Shnoz By Michael Steinberger. Target: wine critics and connoisseurs
Critical reflections on wine guru Robert Parker. E.g.: "What has really given Parker his stranglehold over the wine industry is the 100-point rating system he employs. Other critics have embraced this approach, but Parker pioneered it, and no one has used it to greater effect. Simply put, it is a device that creates the illusion of scientific rigor. It is one thing to award a wine an A, or five stars; this leaves some room for interpretation. However, to say that a wine merits 88 points indicates a level of precision that just cannot be achieved, except in Parker's own mind." [20 Jun 02]
Cheeky little test exposes wine 'experts' as weak and flat by Adam Sage. Target: wine critics and connoisseurs
"“The truth is that you cannot define taste objectively,” said Frédéric Brochet, a researcher from Bordeaux whose study won an award from the Amorim wine academy in France. The opinions of the so-called connoisseurs are are no better and perhaps worse than that of the occasional drinker, he said. The greater the expertise, the greater the cultural baggage that prevents you from perceiving the actual taste in your mouth." See also
The Red and the White
by Calvin Trillin. An amusing, anecdotal ramble through similar territory. [13 Aug 02] Wine Buffs Tongue-Tied by Helen Pearson
"Wine connoisseurs are renowned for their elaborate descriptions of bouquet. But Wendy Parr of Lincoln University in Canterbury, New Zealand and her colleagues may have burst their pretentious babble. 11 buffs did no better than beginners at identifying single aromas typically found in wine..." [5 Nov 02]
Literary CriticsErn Malley Feature from Jacket Magazine
Can poetry critics distinguish serious poetry from nonsense dressed up as poetry? The "Ern Malley" hoax suggests they have serious problems. Poetry criticism is akin to reading horoscopes - try hard enough and you can find some apparently profound meaning in just about any babble. This website is a rich repository of resources on one of the great literary hoaxes of all time. [30 Dec 02] A Reader's Manifesto by B.R. Myers. Target: literary critics.
Scathing attack on the sloppy thinking behind the pretentiously literary style of today's acclaimed fiction, and on the goggle-eyed idiocy of critics who applaud it. "Nothing gives me the feeling of having been born several decades too late quite like the modern "literary" best seller...Clumsy writing begets clumsy thought, which begets even clumsier writing. The only way out is to look back to a time when authors had more to say than "I'm a Writer!"; when the novel wasn't just a 300-page caption for the photograph on the inside jacket." [24 Jun 03]
Futurists
Futurism Is
Dead by Hope Cristol
"It's not just that the futurist has no clothes, it's that the futurist has no
shame." See also the plaintive
response from The Futurist. [1 Jan 04]
PunditsIt's safe to predict... yes, unfortunately by Mark Halpern. Target: media experts.
"Experts and authorities are forever making predictions (many of them about the future), being proved wrong by events, and then continuing their career as experts or authorities without missing a beat; there seems to be little reason to be careful." Singles out one Ms. Judith Kipper for special attention. [requires subscription] [22 Apr 03]
Last updated:
20 Jun 2007 | 
