edit

Lessons from the anti-library

2007-05-17

No book better summarizes what I've been reading and thinking about in the last couple of years than Fooled by Randomness by Nassim Nicholas Taleb, which I reviewed in 2005. Fooled by Randomness was a work of practical philosophy, using examples from the financial world to attack the layers of illusory knowledge that we build on top of random events around us.

The financial world is a good place to find such layers, with its simplifications and retroactive explanations. Lucky traders become "geniuses", unexpected up- and downturns are found to be "inevitable", and the bell curve is assumed to be a fundamental property of reality. But this process of make-belief is not restricted to finance, it is universal. We do it to our lives, and we do it to the world we live in.

In The Black Swan - The Impact of the Highly Improbable, Taleb continues this theme of uncertainty and the illusion of expert knowledge. It is a better, deeper book, and less likely than Fooled by Randomness to be mistaken for financial advice. It is a book about skepticism, wild randomness and the power of stories. It's about how not to be a sucker to the unexpected and the unknown.

Taleb introduces the Hayekian and almost taoistic metaphor of the anti-library: A library of the books you haven't read, of the things you don't know. A massive collection of unknowledge, the anti-library contains all the books that may still change your life. Most of your favourite books are there, hidden and forgotten. It has facts you need to know, authors you'd worship. Only people who read very little can think the anti-library doesn't matter. To book-lovers, and especially generalists like me, who jump all over the place without focus, the stacks of the anti-library loom higher and darker for every book we read. Every book that changes me reminds me of the ones that still might.

The Black Swan is not about unread books as such, but about all the important, life-changing, world-changing information that we don't have. Unknowledge is an unsettling concept. Like the Death of Discworld, our eyes glaze over it and hurriedly find something more comforting to focus on. Taleb asks us to look at it, and acknowledge that it's there, a massive source of wild randomness that will eat your predictions for breakfast, and change your world again, and again, and again. As it has done so often in the past.

The title comes from a classic problem of induction, (it took only the unexpected observation of one black swan to ooverturn centuries of observations of white ones), but I like Taleb's turkey metaphor better: Imagine that you're a turkey. You've eaten well and lived in safety every day of your life. Everything in your experience tells you that tomorrow will be no different. Then Thanksgiving arrives - a Black Swan event.

A Black Swan is an unexpected event that has a massive impact and seems perfectly explainable in retrospect. It is a September 11, a stock market crash, a World War 1. Think of it as a story-breaker. At any given time we have in our heads a story of the world we live in, a story that may be bleak or rosy, but either way makes sense. All of history could be told through the stories that competed for people's attention at any given time. Medieval theology, enlightenment rationalism, marxist history. All of them stories, compelling stories that for a time, to some people, seemed to describe reality.

Then a Black Swan comes along - a new idea, a new technology, a revolution, a war, a terrorist attack - and exposes the stories for the fabrications they are. It's like when you look up at the night sky and glimpse the true, insane scale of the universe, but only for a minute, before it returns to being a roof with pretty lights again. After a Black Swan the ground seems fluid, the eternal malleable, but it doesn't last. Soon the illusion returns in the form another story: A revision of the old, or an entirely new story. The new story incorporates the Black Swan, makes it obvious, natural, inevitable. The world is again safe and rational - until the next Black Swan arrives.

We've recently lived through an event like this, a burst of insane reality followed by a return to fake normalcy. As I've described elsewhere, I was caught up in the rationalization process that followed. Taleb writes, and I agree, that we took the wrong lessons from the September 11 terrorist attacks. We learned how to avoid another attack just like it. We learned to fear the specific causes of that attack. What we should have learned was to trust our stories less. We learned that 20th century history culminated in Osama bin Laden. What we should have learned was that history doesn't march, it jumps.

Good historians know that, but the bad ones tell better stories, and their good, false stories make up the scenery the news media stages its productions against. In the hands of journalists, pundits and media experts, history becomes a mythology - a collection of moral lessons for our time. In reality, history is wildly random. While it happens it is shocking, insane and painful. The stories come later, to heal the trauma, and make-believe a sane world.

Taleb calls the world of predictable stories Mediocristan, a place ruled by mild forms of randomness such as the bell curve. Black Swans are creatures of Extremistan, a world ruled by the unlikely and the extreme. Taleb distinguishes between the popular concept of randomness, associated with bell curves, games of chance, and quantum uncertainty, and the relevant, wild kind of randomness, which comes out of anything that is unknown.

A bell curve or a game of chance is predictable because it obeys the law of large numbers. As you get more samples, the results converge. Even quantum randomness cancels out on the large scale - individually, fundamental particles are unpredictable. By the billions, they're pretty dependable.

Wild randomness does not obey the law of large numbers. It does not converge. A thousand days of not being eaten tells the turkey nothing about what to expect on Thanksgiving. Wild randomness has nothing to do with the much-abused uncertainty principle. Things don't have to be unknowable in theory to be unknown in actuality. As complexity theory shows, even a deterministic world would be unpredictable.

The Unknown is the source of randomness we should be concerned about. It's the Unknown that will change our lives, not the safe probabilites of a die throw. Taleb calls our use of game-based randomness to model real life the ludic fallacy. We mistake the world for a game of chance, when it is much more random than that.

Not all Black Swans are negative, and not all are sudden. From one perspective, all modern history is one huge, slowly unfolding Black Swan. This is reflected in the ever-changing stories we live in. Our stories used to be more stable, your children saw the same world you did. Now the stories are as shortlived as a summer blockbuster. And still we believe in them, in the next as sincerely as in the last.

Taleb admits that the Black Swan too is a story. But it's a better story. We can only understand the world through stories, there's no way to change that. What we need, (and again, Pratchett readers will see an analogy), are good stories to fight the bad ones. Stories that prepare us for the unpredictable, encourage us to doubt what we know. As Taleb writes, you can't avoid crossing the street, but you can try not to do it blindfolded. You can try not to be a sucker.

In Fooled by Randomness, Taleb pointed to Stoicism as a way to deal with the whims of fortune. The Stoics believed that you should invest your happiness in things that were in your power: The choices you make, your actions, in short, your integrity. Unhappiness comes from clinging to things that are not in your power: Fame, money, success and good health. External things come and go, only your choices are truly your own.

In The Black Swan, Taleb draws on another overlooked philosophical tradition, the Skepticism (or Pyrrhonism) of Sextus Empiricus, continued in modern times by Michel de Montaigne and David Hume, and reflected in the ideas of Popper and Hayek. The Skeptic tries to keep the mind in a state of suspended judgment, neither agreeing nor disagreeing, staying open to contrary facts and viewpoints. A Skeptic believes in reality, but not in human reasoning. A Skeptic laughs at certainty, and feels most comfortable as a Devil's Advocate.

Unfortunately, like the Stoic sage, the true Skeptic is an unattainable ideal. So we must do our best with what we have. Taleb's advise is to pick your battles. Skepticism is hard work, so reserve it for the beliefs and choices that really matter. Be practical. Let the theoreticians fret about the unattainability of perfect knowledge - in the real world we have to make decisions. Models based on mild randomness underestimate rare events, learn to exploit that. You can't estimate the probability of an unexpected disaster, but you may be able to limit its effects. You can't predict success, but you can expose yourself to lucky breaks.

Merely by thinking about the improbable, the unknown and the uncertain, you have an advantage over those who don't. Comforting simplifications make you over-confident. Uncertainty makes you curious and cautious. It's not much, but it might be all we have.