99 and 44/100ths pure data

Is the data clean enough to give us meaning? If you are one of those who insist that the war on dirty data must be won first, forgive me. I’m tired of that conversation. It sounds like backyard mechanics comparing fuel injectors when all that really matters is the commute.

I talked recently to TDWI’s just-departed education director Dave Wells. He’s been thinking, and he says, “Tweaking the details probably doesn’t make any real, significant contribution to the overall value of the insights I can gain by understanding the patterns inherent in data.”

Is it really, really going to make any difference if you can get your data from 96 percent to 98 percent pure? Will the meaning really change?

My old boss in market research had a rule of thumb: with a good sample, results usually stabilize after about 25 percent of the data have been counted.

Are we there yet? Wells says the more interesting problem these days how to soak the meaning out of what you do have. Let those who scrub keep scrubbing while the rest of us move on.

To get to the next level in BI, he says we’ll have to understand better how people think. That could involve systems thinking, how one part of an operation affects a different one. Other areas he’s looking into are design thinking and critical thinking.

Notable quotes

  • One of the real looming questions I think in BI as it becomes mainstream is what do you do when analytics contradict conventional wisdom. Then it becomes a political and religious debate.
  • The flaw in our current approach to analyzing things is best described by an analogy that says, “you cannot understand the impact of Van Gogh’s Starry Night by categorizing the brush strokes.”
  • Maybe the value of real high impact metrics, for instance, is not in the measuring of things, but in the measuring of relationships.
  • There will always be something technological to fix. Let’s go back to Starry Night. Do I need to fix that brush stroke? Or do I need to stand back and look at the big picture?

One person I’ve talked to about his ideas thinks Dave is “full of crap.” He says, “If you talk to the people who deal with data, you hear that the problem is nowhere near solved.” OK, there’s still work to be done. But rejecting Dave’s reasoning is like saying that because car design is still not perfect we can’t yet talk about driving from San Francisco to Lake Tahoe.

The full interview will be at the May 7 issue of BI This Week.

Links he suggests, with his notes:

  • Systems thinking
  • Design thinking: The weblog Functioning Form’s “Defining Design Thinking” post gives a good starting point, he says, with links to articles and papers. He writes, “Design thinking is a less mature discipline [than systems thinking], which makes it less robust but more interesting to explore.”
  • Critical thinking: “The critical thinking community is more focused on education than on business today,” he writes, “so much of the interesting stuff there is in discovering how it does apply in business. I have no doubt that it does apply because much of critical thinking is simply another perspective on decision theory.” See the Strategies for Success page and the Foundation for Critical Thinking site.

Leave a reply

Other recent posts

End of one-size-fits-all data stories

This appeared originally on the TDWI site in September behind a paywall. It’s still there, but today they’ve had the 90 days of exclusive use that I agreed to. Survey after survey reveals that about 80 percent of business users don’t use data analysis—despite all the marketing and “easy to use” tools. As if in… Continue Reading

Qlik finally set to leapfrog Tableau?

Who’s your rival? I carelessly asked a Qlik person at the company’s annual analyst reception Monday night in Miami if she hadn’t once worked for Tableau. Her revulsion was immediate. “No! Never!,” she said. We smiled. There was so much more to talk about. For one thing, how will private equity change things? Qlik wasn’t… Continue Reading

Five Tips for Better Data Stories

Originally published on September 22, 2015 in BI This Week, a TDWI publication. A “data story” sounds like such a great idea. You just mix data with storytelling and you’re done — except that most data storytellers get one thing wrong: they drown out the story with data. Such storytellers, I believe, assume that audiences… Continue Reading

Bohemian Grove a la BI

The Bohemian Grove of the BI industry convenes for the fifteenth time in just three weeks. Naturally, you ask the obvious question: Are you serious? The Grove? A summit? The answer begins with a fond recollection of the Grove. If you’ve never attended the Bohemian Grove yourself — I haven’t, though I live in the… Continue Reading