lies, damned lies and misconceptions

A reader at Columbia raised the question, what kind of novelist would be interested in statistics.

Calvino once described a city in which a man passes a girl with a puma on a leash, a sailor with a tattoo. He imagines that these are remarkable events. The comment is made: there's always some girl walking a puma for a whim. (I am paraphrasing wildly, too lazy to check.) The point is, when we observe events we can't tell whether they're unusual. Most of us are aware of this to some extent, but we start out with only a very loose sense of the ways we might go wrong; when one starts reading about data analysis, one sees how little we actually know about we actually see. Which means, of course, that the typical 'realistic' novel is not realistic in showing us a fictional analogue for what's actually in the world, it's realistic only in replicating the kinds of mistakes the naive observer makes in looking at the world. The unreliable narrator, of course, is part of fiction's stock in trade, but this is not unreliability used as literary strategy; it's just a form of unreliability that's invisible to author and readers alike.

Here's a simple example of the intuitionist's approach to probability:

We come into a room. It contains two indistinguishable urns. We’ve been told that one has 300 white balls and 700 red, the other 300 red balls and 700 white. I bet you $10 that Urn 1 has 700 red balls. Will you bet $10 that it’s not? (You may not be willing to bet with only even odds, but it’s a fair bet.)

I take a ball from Urn 1. It’s red. I put it back; the Mechanical Ball Mixer mixes the balls. I take another ball. It’s red. I put it back; the Mechanical Ball Mixer mixes the balls. After 12 draws, I have this result: RRWRRRWWRRWR (i.e. 8 red balls and 4 white). Will you still bet $10?

Almost certainly not. Most people wouldn’t. But suppose I bet $12 against your $10? $14? $16? $20? At what point would it again be a fair bet?

Most people* think something between $14 and $20 would be fair. This is not correct - I would have to bet $293 to make it a fair bet. In other words, intuition goes very quickly from being a good guide to being wildly astray.

* I do have a reference somewhere among my papers to a study with actual data, but I am not convinced that the relevant papers are not in storage in London.

Most fiction does nothing to make us aware of the gulf between cases where intution serves us well and those (surely far more common) where it does not. It does nothing to show where we should be wary, or how to think through tough cases. Most fiction is confined to the realm of false intuition; it offers us no viewpoint with a better understanding of chance. Which is simply to say that, because we live in a culture with a profound hostility to mathematics, the type of person who writes fiction is likely to be the type of person who shares that hostility and can rely on a large audience which also shares it. Among other things, this means that someone like my friend Rafe Donahue, a biostatistician at Vanderbilt, tends to be both underrepresented and misrepresented among fictional characters. Once upon a time persons of color could only get parts in films playing servants, often with amusing eccentricities which confirmed the supposed preconceptions of the audience; the sort of person who grapples with data analysis is either not seen in fiction or appears as some sort of eccentric.

As I write the Fed has cut its key interest rate by 75 points, down to 2.25%. The Fed has brokered a deal with Morgan Stanley; Bear Stearns, the fifth largest bank in the country, has been bought by MS at $2 a share, down from $160 a share last year. All this comes as a result of the collapse of the subprime market - a market dependent on financial instruments which were the pride of the finance industry only a couple of years ago. One thing fiction could have been doing all this time was enabling people to see on the page the way those in the risk business think about risk; it could have used the techniques of Edward Tufte's information design, for example, to present data in a way that did not numb the mind of the general reader.

This looks interesting from a formal point of view: fiction has made use for centuries of free indirect discourse, in which narrative is presented in the inner language of a character who is not the author, but has steered clear of the sort of inner language that helps itself to the bag of tricks of Tukey, Mosteller, Bill Cleveland and others too numerous to mention. It looks important simply because the management of risk is integral to our society; if fiction ignores the way this actually works, its view of the world is not much less primitive than one in which storms blow up because Odysseus angered Poseidon.