Uncharted Waters

Jan 16 2012   8:54PM GMT


Matt Heusser Matt Heusser Profile: Matt Heusser

In this classic XKCD comic strip, we see a bit of everything: Humor, enlightenment, system dynamics … real life.

The term the author uses is “citogenesis”, a clever pun on “cytogenesis”, that actual real-life term for the process by which living things are created.

It seems that on the internet, and even before that, legends are born when respectable people or organizations make respectable claims that sound like they are true – or at least, sound like they should be true.

Comedian Stephen Colbert calls this “truthiness“, a term that, ironically, has made it’s way into wikipedia, where it is described as: a “truth” that a person claims to know intuitively “from the gut” or that it “feels right” without regard to evidence, logic, intellectual examination, or facts.

Yes, some Wikipedia author put “truth” in quotes to indicate just how true it needed to be. (Hint: Not very much).

Citogenesis isn’t just for mythbusters; it can affect us at home, at the office, or out shopping.

Here’s how to spot it – and what we can do about it.

Citogenesis in the IT Shop

Consider, for example, the “Cone of Uncertainty.”  It’s a basic project management tool, it is simple, and it makes sense.  The idea is something like this:  Early in the project, you have very little idea how long the project will take.  It could be three months, it could be nine, it is most likely six. That is a 3x possible difference — and, if we are lucky, we’ll get the middle date.  (Most of the time, we get the short one, don’t we?)

Again, the idea that the cone gets shorter makes sense.  It is intuitive.  It is an improvement over classic “make up dates and define success as conformance to schedule” thinking, suggesting a range of dates that we guide to over time.

What it is not is a law.

For example, have you ever known, really known, what it will take to get a project across the finish line, only to find out that you were missing something?

The server needed an Operating Systems upgrade, the database client wasn’t compatible with the server, the application was supposed to be real-time but it runs in batch, it doesn’t matter.  Uncertainty doesn’t always go up.

Perhaps most of the time it does – on average, in something statisticians call “the aggregate.”  A fair amount of publications claim this has been proved with empirical research — meaning using actual projects, not simulations.  Empirical research is good because it’s real-world, but bad because there are too many independent variables; you need a lot of empirical research to prove things.

And that’s the rub.

One expert actually took the time to track down the data. Laurent Bossavit, who recently completed a term on the the board of directory of the Agile Alliance, tried to track down the research … and he couldn’t find any.  In a short series of Google Plus Posts, Bossavit traces the history of cone idea, pointing out that it moved from one person’s observation and intuition, to be referenced by the next paper as fact, to then be referenced by a third paper as proven fact … and that brings us to the beginning of this post with me quoting an XKCD comic.

It’s not just the cone of uncertainty.

When I was a child, I was told the George Washington Cherry Tree Story as fact, and I believed it until my adult years.  It wasn’t that I actively thought about it and had a deep faith in it, It’s just that it seemed right that a story about the personal integrity of a founding father should be, you know … true.

Likewise, while I have heard a number of leaders and executives giving pitches and presentations, often strewn with numbers, none of them ever said anything like “We’re losing money by the truckload” — yet several of them did go out of business shortly thereafter.

How to beat the fake fact

In his book How To Lie With Statistics, Darrell Huff raises some key questions to ask, including:

– Who Says So?

– How Do They Know?

– What’s Missing?

– Did Someone Change the Subject?

– Does it make sense?

Beating Truthiness

The fix for truthiness isn’t all that different.  First, take a good look at the data – is it ugly, or is it presented with nice, clean, improbably-sharp edges?  Is it presented as a generality or a law?  If it’s presented as a law, where is the data?

If the data is in a previous report, dig up that report.  If the previous report references someone else, dig that one up.

Once you’ve got three levels of someone referencing someone else, as Bossavit points out, we reach the “telephone game”, where we can expect that fair amount of nuance will be lost even to the author of the paper.

Why should we have to put these claims to test?

Well, for one thing, a great deal of modern astronomy rests on one man, Galileo Galili, challenging Arisotle’s claims that heavier objects fall faster than lighter ones.  It turns out that heavier objects do have a greater attraction to the ground (more “force” of gravity), but they a proportional increase in mass.  Since Force = Mass * Acceleration, the acceleration remains constant.

It is a great honor of my life to sit on the board of directors for the Association for Software Testing, but, you know what?

We are all called to be testers.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: