CIA’s insights on the psychology of analysis

Imagine someone writing a book about data analysis without even mentioning software.

“To penetrate the heart and soul of the problem of improving analysis,” writes Richard J. Heuer Jr. in Psychology of Intelligence Analysis, “it is necessary to better understand, influence, and guide the mental processes of analysts themselves.” It’s the mind that does the heavy lifting.

Though Heuer writes for CIA-type analysts, similarities to business are clear. The biggest one is that it’s always the mind that does the heavy lifting.

The two kinds of data analysts share many other aspects of their work, such as their unconscious biases. For example, they all tend to give undue importance to vivid evidence over abstract evidence.

Both also endure the after-the-fact know-it-all, the type that announces, “That analysis told me nothing new,” after confirming evidence arises. Heuer describes an experiment that suggests such beliefs are usually delusional.

Analysts’ memories, too, undermine their work, such as when asked to remember what led them to their conclusions. Also, overseers tend to regard events as more foreseeable than they actually were.

Missing data, another problem, is “out of sight, out of mind.” Two groups of experienced auto mechanics were asked for a diagnosis of a car that wouldn’t start. They were offered a fault tree with several major branches — weak battery, starter-system defects, etc. — with several themes left out and summarized as “other causes.” One group’s fault tree included all major branches. But the second group’s fault tree had several branches missing — and selected the summarized “other causes” far less often than would be expected if the summarized reasons had been listed along with all the others.

Heuer offers a few solutions. The main one seems to be his “Analysis of Competing Hypotheses.” He outlines it in eight steps.

Heuer writes the way I’d expect a CIA veteran to write: formally. Even where he has a colorful tidbit — such as any of the load of stories mentioned in the footnotes — he opts for gray.

Such drab phrasing ignores the principle he gives in Chapter 10 under “the vividness criterion.” More vividness would have made a more readable book.

The impact of information on the human mind is only imperfectly related to its true value as evidence. Specifically, information that is vivid, concrete, and personal has a greater impact on our thinking than pallid, abstract information that may actually have substantially greater value as evidence.

If there’s one thing the book reminds the reader of over and over, it’s that analysts are only human.

On tools, Heuer lists only what he considers the basics: decomposition and externalization. Vendors will have to do what they can with those clues. Unlike so much software, though, Psychology of Intelligence Analysis is free.

2 Responses to CIA’s insights on the psychology of analysis

  1. Thought provoking post Ted. A real creative “Whack.”

    It’s not unusual for my ramblings in visual analysis to lead me down different paths. Keeping an open mind about the possible causes is what leads me from “question to insight.”

    Mr Heurer looks at the world through a different lens vs the typical business analyst. Some of his insights may prove valuable.

    I’m adding his book to me reading list. My aching Amazon account thanks you for the free link.

  2. “(…) information that is vivid, concrete, and personal has a greater impact on our thinking than pallid, abstract information that may actually have substantially greater value as evidence.”

    This also explains why screaming headlines get more attention than nuanced ones and why it is sometimes difficult to gain attention when real insights are being shared.

    Your article reminds us that being an analyst also requires a splash of art: the art of shouting “Fire!” without actually causing panic. In his 14 Chapter e-book, Heuer provides plenty of ideas about how to hone this art – including ways to help us question our own assumptions and conclusions.

    I particularly like his discussion about how information is assimilated and how we have to first evaluate what is known and assumed before we can open the door to new insight. Discerning knowledge from assumption is often more difficult than it sounds.

    Thank you for sharing!

Leave a reply

Other recent posts

End of one-size-fits-all data stories

This appeared originally on the TDWI site in September behind a paywall. It’s still there, but today they’ve had the 90 days of exclusive use that I agreed to. Survey after survey reveals that about 80 percent of business users don’t use data analysis—despite all the marketing and “easy to use” tools. As if in… Continue Reading

Qlik finally set to leapfrog Tableau?

Who’s your rival? I carelessly asked a Qlik person at the company’s annual analyst reception Monday night in Miami if she hadn’t once worked for Tableau. Her revulsion was immediate. “No! Never!,” she said. We smiled. There was so much more to talk about. For one thing, how will private equity change things? Qlik wasn’t… Continue Reading

Five Tips for Better Data Stories

Originally published on September 22, 2015 in BI This Week, a TDWI publication. A “data story” sounds like such a great idea. You just mix data with storytelling and you’re done — except that most data storytellers get one thing wrong: they drown out the story with data. Such storytellers, I believe, assume that audiences… Continue Reading

Bohemian Grove a la BI

The Bohemian Grove of the BI industry convenes for the fifteenth time in just three weeks. Naturally, you ask the obvious question: Are you serious? The Grove? A summit? The answer begins with a fond recollection of the Grove. If you’ve never attended the Bohemian Grove yourself — I haven’t, though I live in the… Continue Reading