Fooled by randomness

I came across Nassim Nicholas Taleb while writing this entry, and I've been reading his book Fooled by Randomness. Taleb is a financial trader who has become a philosopher of randomness, of how our world is shaped by random events. We think that if only we have enough data about the past, and if only we look closely enough at these data, we can discover the laws our world works by, and use that to predict tomorrow.

But we underestimate the amount of data we need, and the difficulty of using it properly. This leaves us vulnerable to what Taleb calls the black swan problem, after John Stuart Mill. If all the swans you've ever seen have been white, you'll feel justified in thinking that all swans are white. And then one day you see a black swan. The black swans of our world are the events we couldn't predict, the events we thought were impossible because they hadn't happened before, and yet turned out to be the most important events of them all.

The black swan is a problem of induction, which is one of the two basic ways of gaining new knowledge. The other is deduction. With deduction you use logic to prove a statement from a set of premises. With induction you start with data and look for laws and patterns in it. Both methods come in more and less reliable forms. Deduction is useful when the premises are good, as in mathematics, and useless when they aren't. As I wrote earlier, rational people are fond of logic, too fond, so they use it too often and badly. The armchair philosophers of history have left us shelf kilometers of elaborate "proofs", but little knowledge.

In the same way, induction can be used or abused. Its unreliable form is the guesses we make about the world around us. We need these guesses to operate in everday life, we need rules of thumb that allow us to reach good-enough decisions quickly. We need to be able to take a quick glance at a situation and decide if it presents a threat or an opportunity to us. But our intuitive induction is built for speed, not accuracy, it is built for keeping our ancestors alive on the African savannah, not understanding the world, and so it also leads us to form prejudices and superstitions.

The more reliable form of induction is the scientific method, which is a kind of layer built on top of induction, (like TCP on IP). By adding formal requirements and distributing the work, science is able to use induction to find very good approximations to the laws our universe follows, theories with near-perfect predictive powers.

Scientists can then use deduction to make new predictions, which again can be tested to give us further laws, so in a way science uses the best of both methods. In the same way, bad thinking combines the worst of both methods: We use lazy induction to make guesses, and then rationalize our guesses by deducing backwards, until we find ourselves on steady ground.

Science works well when we study simple systems. The laws of physics do not appear to be changing. There's nothing to say that they won't, (which would be a major black swan), but so far they've been remarkable stable. Einsteinian physics was a black swan, but it didn't really affect the slow dynamics we usually deal with. Physics is simple, and that makes scientific induction in that area relatively easy and reliable.

Complex systems - markets, societies, minds - are different. They display unexpected emergent behavior, which just means that they appear random. Not the randomness of a die, where every outcome is equally likely, but the randomness of there always being several possible outcomes, some of which will take us by surprise. We think we can make reasonable predictions, about markets and world politics, but there's no evidence of this. Even the best qualified of us know less than they think they do. Taleb's solution is to bring back the concept of probability in our understanding of the world. Not the mathematical approach of assigning probabilities to outcomes, which has all the strengths and flaws of induction, ("if it hasn't happened before it's impossible"), but an awareness of probability as a factor in everything we do.

Why are some people rich, and others not? The popular answer is that people get rich because they're skilled. And maybe some of them are. But the randomness of market dynamics ensure that bad stock traders and CEO's will often get rich and good ones poor from chance alone. If you tell a thousand people to flip coins, a few of them will inevitably get an "improbable" series of heads or tails. A million monkeys with typewriters will eventually write a Shakespeare play. Taleb makes fun of books that explain what it is successful business people have in common, so that we can learn from them and be successful ourselves. They may be no more special than the monkey that writes King Lear, or the coin flipper who gets 10 heads in a row.

To be aware of probabilities means that we have to consider alternate histories: Both the events that might have taken place in the past, and the events that might take place in the future. To understand the world we need to look at the set of all possible histories, not just the one path we took by chance. We can't do that, but we can be aware that these alternate histories exist, and refrain from placing too much significance in the events that did take place.

With his financial background, most of Taleb's examples in this book are from the world of stock and options trading, but his ideas have a wider application. We explain the success of a CEO as a consequence of his abilities and experience, forgetting that many people share these qualities without ever being successful. We similarly explain the success of countries and cultures as a necessary consequence of their "character" or some formative event. We explain economic up- and downturns as a consequence of the policies of the current government. We judge politicians by whether their gambles paid off, not whether it was smart to make them in the first place. When interesting things happen, we look for causes, and we usually find them, even when there are none, and it all happened mostly by chance.

To fully understand our world and our lives, we would need to consider not just the history we have, but the histories we might have had. We would need to consider the history where the march of democracy was recently interrupted by totalitarianism, or where Mohammed Atta was arrested. We would need to consider the history where that lucky break you got didn't happen, or where you weren't offered that job, or didn't meet that person that day, or where you're dead, or your friends and family are or aren't alive. We would need to consider the sum of all these histories that might have happened, and all the histories that might happen in the future.

But all that we have to learn from is our own history, up to today. As Taleb puts it, we only have one sample path to study. One sample path out of many, and one far too short to say much about our future. A few thousand years of history is a disturbingly short amount of time, and tells us very little. So we're stuck waiting for our black swans, we're stuck knowing that the 21st century may surprise us as much as the 20th did previous generations. Our imagination and gut sense is of little help to us, and whatever comes to happen may violate historical "laws" you thought were absolute. And what separates what will happen from what else might happen could be nothing more than a flip of the coin.

So how do we deal with this randomness? We can't deny it, and we can't work around it, so we have to face up to it and live with it. Taleb offers one practical approach, and one philosophical.

The practical approach is to pay less attention to noise. If we accept the large role randomness plays in our world, and accept that we're easiy fooled by that randomness, and that we know far less than we think we do, then it follows that we're simply unable to deal rationally with the torrent of information that is coming our way every second of the day. Most of that information is random noise, or meaningless commentary on that noise, and if we pay attention too closely it'll fool us. We'll see big stories in random events, tie our emotions to the random fluctuations of markets and societies. We'll be fools of randomness.

To pay less attention to noise means paying less attention to daily events. There will be little of value for you in the newspaper today, just entertaining randomness and unjustified certainty. Reading it will be more likely to confuse you than to improve your understanding. Important events are rare, important subjects timeless.

Taleb also offers a philosophical approach to randomness, stoicism. Stoicism is not to abandon emotion, but to act well and with dignity independent of how chance and other people treat you. If you're dealt bad news, then, as he quotes the poet C. P. Cavafy, "just listen while shaken by emotion but not with the coward's imploration and complaints". Taleb continues:

While shaken with emotion. No stiff upper lip. There is nothing wrong or undignified with emotions - we are cut to have them. What is wrong is not following the heroic or, at least, the dignified path. That is what stoicism truly means. It is the attempt by man to get even with probability. .. The stoic is a person who combines the qualities of wisdom, upright dealing, and courage. The stoic will thus be immune from life's gyrations as he will be superior to the wounds from some of life's dirty tricks.

Control what you can control, your behavior and attitude. Play your role with dignity. Everything else, good or bad, luck or disaster, is not your achievement and not your fault, it's just part of the soup of randomness life throws your way.

Fooled by Randomness is highly recommended, but if you're uneasy about adding more books to your reading list, (and you should be), these articles by the same author cover many of the same topics:
- The scandal of prediction (we don't even know how little we know)
- Learning to expect the unexpected (randomness and black swans)
- The opiates of the middle classes (faith in scientific-sounding predictions is more dangerous than religion)




Comments

I get the feeling that everything you write these days is *really* about Islam...


Anthony: I get the feeling that everything you write these days is *really* about Islam...

In a way, but only as a trigger and an example of how not to think. The author of this book, Nassim Nicholas Taleb, learned these lessons from living in the world of finance, I've learned many similar lessons from living in the world of blog punditry. There's the same arrogance, the same certainty, the same mysticism as a replacement for serious understanding, and in addition there's not even the pretense of science, and the information filters that keep the faith alive are much bolder.


I've always been struck by how few people answer "don't know" in all kinds of polls, and have prided myself of being among their number whenever applicable. Still, I must repeat that this line of thinking should not be drawn to the other extreme - the new-age "no hard truths" cliche (even if not consciously named as such).

Sometimes we just have to go with our best guesses. Take global warming, for instance. Inaction, if the alarmists are right, may be disastrous. Curbing growth, if the alarmists are wrong, would be murder on poor countries. We need to make an educated guess about the future, and take such steps as are mandated by that guess, remembering to update our goals as new knowledge comes in.


I am sorry for killing the thread, Staerk - well, it was in fact Anthony who did that - but I would like to ask you a question concerning islam and Europe.

Given the past - which of course is the only source from which we can gather information to make predictions (good and bad) about the future - do you believe that the growth of islam in Europe will be peaceful, and what knowledge do you use as a basis for you view?


Line: do you believe that the growth of islam in Europe will be peaceful, and what knowledge do you use as a basis for you view?

I don't know ..

There are negative factors at work, but also positive ones. Negative factors:
- Lack of integration (partly caused by bad immigration laws and multicultural dogmas)
- The violent Islamist minority wants to kill us
- The conservative majority doesn't accept secular/liberal democracy
- Islamophobia (because it confuses us about what the nature of the threat is)

Positive factors:
- Free, democratic societies are adaptable, (so you can't just extrapolate the bad trends, you have to consider how we'll respond to them)
- Wealth is more attractive than poverty (hard to convince your people to obey their puritan leaders when their less pure neighbours have a higher standard of life)
- Most people, including Muslims, are pragmatic, they don't care about political ideologies or theological doctrine, they just want to live a good, God-fearing life for themselves and their family, and they'll be reluctant to sacrifice all that to bring about the visions of intellectual fanatics.
- There are Muslims who are trying to bridge the difference between Islam and Western culture

Unknown factors:
- Chance. And that's the big one. It includes all the trends that exist today but we don't know about, and all the trends that haven't started yet. Ideas, technology, people. All those things we can imagine no more than people a 100 years ago could have imagined the Cold War or the computer.

So there's really no way to tell. One argument is that since the worst outcome would be so bad, we should assume that it will happen, so we can prevent it. Pascal made a similar argument about why we should believe in God: because the penalty if we're wrong is larger if God exists (eternity in hell) than if he doesn't (just plain death). But Pascal forgot that there is more than one God to (dis-)believe in. If there are ten incompatible religions to choose from his argument doesn't make much sense.

In the same way, there is more than one really bad future we can imagine, Islam is not our only potential threat. There's old-fashioned imperialism (Russia, China), there's pandemics, asteroids, climate collapse, etc. Many of these outcomes are worse than Eurabia, and we just can't assume that they'll all happen, and act on that assumption. We have to pay attention to what's happening right now, and act on that, not do all kinds of desperate things to save the world from possibilities.


Great a topic about randomness and the unforeseeable turned into a potentially bigoted tirade against muslims. Future Predictions -

* Muslims remain a minority group in most western nations.

* Extremist attacks continue spadorically

* Domestic intellgence services increase their focus on these types of attacks and are able to prevent many/most of them

* Some nations take excessive measures against the threat of terrorism, a move which only heightens racial tensions -

- France, Ethnic exclusiveness (ie; banning head scarfes) = riots


Seems to me like you based that "I don't know..." on quite a lot of input - data which obviously could be wrong. I recall from an earlier debate someone stating that you would always "copy and paste" from your own cultural background whenever you lacked knowledge and experience. I consider that to be a splendid observation.

As regards the future - nobody knows, of course - future itself is only a hypotesis. But I believe in attempting to identify that which in future may evolve into a threat, and act in order to minimize or, ideally, completely avoid it. Granted, in this process, mistakes will be made, however thousands of years of evolution concludes that this strategy is worth holding on to.

I regard islam to be a threat which it is in Europe's best interest to attempt to minimize to the largest degree possible.


We're talking about the stoic man as opposed to the stochastic man, then?

Here's some mathemathical musings that reflect to a certain degree your post:
Input data:
We're in at a basic theme in statistics here: When do you have enough data to make a close enough approximation to the truth or the actual development? The number of data points needed for this varies with the variation in the whole, but you are only able to measure the variation in the data you have sampled, so when do you have enough data points?

Forward:
Interpolation has shown to make far better guesses than extrapolation (for those who are familiar with statistical terrain modelling: cluster your sample data at center, and see how the edges get more and more off), but to predict the future from only present and post data points we will have to use extrapolation: Any guesses into the future involving uncomplete data sets and/or any form of stochasticity and variation will therefore be suspect (and more so the farther you get from the last secure data points).

KEE


Line: Seems to me like you based that "I don't know..." on quite a lot of input - data which obviously could be wrong.

I'm not basing it on something, but on the absence of something. We don't have anything that gives us the ability to make meaningful guesses about social and cultural change. The factors I listed are illustrations of this, (if you're very concerned by the negative factors you should at least be aware of the positive ones), but the one that really matters it the last one, the big Unknown Factor.

I regard islam to be a threat which it is in Europe's best interest to attempt to minimize to the largest degree possible.

So how do we do that? I agree it's a possibility that religious conflict will become a problem in Europe again, but we have to balance those possibilities against the cost of trying to prevent them vs solving other problems, or the risk of causing a lot of harm just to prevent something imaginable. That's why I think we should focus on the problems we know we do have. Like poor integration and cultural barriers, terrorism and conservatism. Those aren't imaginary, and even if they won't lead to Eurabia they're important enough that we need to solve them.

If you try to solve Eurabia 2050 instead, and you're wrong about Islam being Evil at the core, (don't know if you believe that but many do), there are all kinds of risks. There's the risk of spending at lot of resources to solve an imaginary problem, resources which could have been better spent to solve real/more important problems, (ref the Kyoto protocol). You'd make it more difficult to solve the integration problem in a peaceful way (by deliberately making all Muslims your enemy). Many of the suggestions I hear would actually mean dismantling the liberal basis of our own culture. We'd be preemptively attacking another religion because we think they might one day attack us, which could return Europe to the values of the religious wars. Fearing the possibility of religious conflict, we'd be ensuring it.

And all that based on the fears of people who don't even understand Islam as it is today. I'm not saying any of this "will" happen, but again, if you want to consider the potential benefits of fighting Islam as a whole, you should at least also consider the dangers. And then you need to consider that all of our plans might be made irrelevant by unknowable events.

I recall from an earlier debate someone stating that you would always "copy and paste" from your own cultural background whenever you lacked knowledge and experience.

Don't we all? When actual knowledge doesn't stretch far enough, we make assumptions about the world that seem obvious to us, but maybe aren't. For instance the very popular assumption that we can make meaningful predictions for our own future. It seems so obvious to us that we can, because we're designed by evolution to build quick-and-dirty scenarios about the near future and act on them. That's the whole evolutionary point of thinking, after all. To not make guesses about the future, even when they're the kind of guessing evolution couldn't have prepared us for, goes against our instincts. I believe we should fight those instincts.


If I regard people who wish to use the state in order to oppress or enforce their ideology on others as an imminent threat to Europe and indeed the world, does this justify me taking desperate measures against all who should advocate it?

You know like you – line

We'd be preemptively attacking another religion because we think they might one day attack us, which could return Europe to the values of the religious wars. Fearing the possibility of religious conflict, we'd be ensuring it.

Indeed I believe the above situation to be the more plausible.

It seems that if you want ethnic conflict all you need to do is promote a brutally exclusive society, and sit back as they become increasingly dissatisfied and violent, with the state stepping in to protect us from the threat.


Bjoern wrote:

".....We'd be preemptively attacking another religion because we think they might one day attack us, which could return Europe to the values of the religious wars. Fearing the possibility of religious conflict, we'd be ensuring it. "


.....there you go again Bjoern , making me almost drop my cup of coffee all over my dress !...oh yeaaaah -fearing the possibility of religious conflict ? have you been hibernating in an ancient cavern on top of Mount Kanchengjunga again?

....so you would pretend that all those gajillion conflicts that involve muslims and islam has nothing to do with religion ..come by me una vez mas...i can see this is going to be an expensive blog, spilling expensive coffee everytime i read your 'oh so nonchalant and lackadaisical statements like 'fearing the possibility of religious conflict , we would be ensuring it!' . Gosh ...such excellent display of political correctness have i never encountered. You know Bjoern you would make a good ambassador for the United Nations of Dhimmitude.

Explain yourself to a dumb girl like me exactly what you mean about 'fearing the possibility of relgious conflict'. Darn , Bjoern islamoid religious conflict is the essence of Eeklam, it started from day one when the nefarious cult of Eeklam was constituted, it is their basic tenet..and their holy book is the manifesto par excellence of terror and conflict.

Sister Prasad Meenachi Bhagavatam

बिहार विधानसभा चुनाव के अब तक घोषित
परिणामों के अनुसार एनडीए ने सरकार बनाने
के लिए ज़रुरी बहुमत हासिल कर लिया है.


So much for tolerance... Kim Sook-Im, care to give any textual examples of this?

Anyway Bjørn, you wrote that "(...)we would need to consider not just the history we have, but the histories we might have had." My upper secondary school history teacher, Arne Jensen, thaught me that such a thing is called "a contra factual hypothesis." Such hypothesis are quite possible to ponder, but since they are in essence contra factual, you cannot derive any "truth" from them - as in facts or predictions that apply to the current reality. Of course you already pointed that out, but I thought it was neseccary to add some science blabber to the soup.


I think most of these postings are asking the wrong Question. The question, as I see it, is whether there is any middle ground for Islam and the West to coalesce around. The West offers tolerance and equality. Islam offers ascendency. One os right and one is wrong. Under Islam who is right may be determined by force and force is morally acceptable. The Question is can there be a middle ground. My answer is no unless Islam changes


Krister Brandser: Such hypothesis are quite possible to ponder, but since they are in essence contra factual, you cannot derive any "truth" from them - as in facts or predictions that apply to the current reality.

Yes, because we don't know what these alternative histories would have been like, not because they're contrafactual. For every alternate reality just a coin toss away from our own that we could see, we'd have that much more knowledge about our own potential futures. We'd have more sample paths to analyze, instead of just one very short one.

And even if we don't have access to these histories, the idea that they might easily have existed teaches us something important. And we can at least try to identify toss-of-the-coin forking points in history, and use that to cure ourselves of any ideas that we're "destined" to move in any particular direction.

David Elson: It seems that if you want ethnic conflict all you need to do is promote a brutally exclusive society, and sit back as they become increasingly dissatisfied and violent, with the state stepping in to protect us from the threat.

At least attempts to fight Islam itself would make integration of Muslims into European society extremely difficult. Religious oppression tends to backfire. We'd be risking a lot on a guess.


A black swan may be a different color than a white swan, but it still looks like a swan. It doesn't look like a rabbit or a giraffe.

There may be different colors of Islam, but it'll still be recognizable as Islam, and it'll still be incompatible with Western culture and values. (Actually, it'll still be incompatible with any non-Islamic culture and values.)

My considered prediction, based on 10+ years of living with and conversing with Muslims from all walks of life.


At least attempts to fight Islam itself would make integration of Muslims into European society extremely difficult

And just how much are you willing to give up inorder to make that storied integration take place? Because that's what you are advocating whether you realize it or not.

Willing to give up freedom to print salacious cartoons about Mo in
a local newspaper? The freedom to make "The Life of Muhammad" as a movie? The freedom to have your female relatives dress as they want, and move about safely as freeborn Western women? The freedom to have a secular life, without tripping over public Islamic religious expression aggressively pushed into your face every where you go? (Prayer calls blared out by loudspeaker five times a day in your neighborhood, prayer groups on your sidewalk, abolition of Christian or other religious symbols that make Muslims feel "frustrated", etc.)

None of those things are things that I am
willing to give up in order to make Muslims feel more "integrated" into the West. Thus I will continue to attack the aspects of their religion that conflict with my values.


Susan: A black swan may be a different color than a white swan, but it still looks like a swan. It doesn't look like a rabbit or a giraffe.

I think you missed the point, which is that induction can only tell you about the past. Unless you understand the elementary particles of social dynamics and can calculate the future, there's just no way of knowing that the culture and religion you've thought of as Islam in the past won't do something completely unexpected tomorrow. And if they don't, something else will. Try explaining the 20th century to a person from the year 1900. That was a whole series of dramatic, unexpected events .. and here we are, foolish enough to think this is over, that now history is back on some predictable track, (a negative track in your case.)

And just how much are you willing to give up inorder to make that storied integration take place?

Nothing. We'll see how it turns out.


And just how much are you willing to give up inorder to make that storied integration take place?

Nothing. We'll see how it turns out.

Glad to hear it. But we won't be able to hold onto what we have by simply hoping for it. . .


I like that post. It got me thinking about my alternate histories. Its not missed opportunities, or I should have studied harder, but a different destiny. And its funny to think of the different people you would be in those different histories.


well, i cant speak for bjorn here, but I know that I for one, will not base my actions today disregarding islams 1400 year of conquest, nor its rather explicit theology on the subject of war, booty etc.. Nor will I disregard the misogynic societies it created.

To do so is simply stupid. Whereas one can not know the future, the probability in this equation is rather obvious, and depressingly so.

those who dont know history, are bound to repeat it..

I do not presume to know the future, but I can only act upon current and historical data..


Trackback

Trackback URL: /cgi-bin/mt/mt-tb.cgi/1710

Post a comment

Comments on posts from the old Movable Type blog has been disabled.