The Bayesian Curse

The Dreyfus affair was a political scandal that divided France in the 1890s and the early 1900s. It involved the conviction of Captain Alfred Dreyfus for treason in November 1894. He was sentenced to life for allegedly having passed French military secrets to the German Embassy in Paris. However in 1906 Dreyfus was exonerated and reinstated as a major in the French Army. He served during World War I, ending his service with the rank of Lieutenant-Colonel.

How did Dreyfus’ fortune change? Henri Poincare, a respected mathematician, cited probability and statistics. He said undue weightage to new evidence was wrong and judgment should be based on all other available proofs. He asked the jury to rely on scientific education rather than feelings.

It was much before probability that David Hume criticized cause and effect. He said certain objects were constantly associated with each other. But, the fact that umbrellas and rain appeared together didn’t mean umbrellas caused rain. That the sun had risen thousands of times didn’t guarantee it would do so the next day. In criticizing concepts about cause and effect, Hume was undermining Christianity’s core beliefs.

With Hume’s doubts about cause and effect swirling about, Thomas Bayes began considering ways to treat the issue mathematically. In any event, problems involving cause and effect and uncertainty filled the air, and Bayes set out to deal with those quantitatively. He decided his goal was to learn the approximate probability of a future event he knew nothing about except its past, that is, the number of times it had occurred or failed to occur. As a starting point, he would simply invent a number — he called it a guess — and refine it later as he gathered more information (evidence).

Even today, investors suffer from this new evidence (information) miscalculation. In their path-breaking research paper in 1985, Werner Bondt and Richard Thaler proved how research in experimental psychology violated Bayes’ rule, suggesting that investors (most people) tend to “overreact” to unexpected and dramatic news event. In revising their beliefs, individuals tend to overweight recent information and underweight prior data.

This behavioral error violates the basic statistical principle that the extremeness of expectations should be moderated with probable considerations. Overreaction is a well-studied subject, with earlier observations made by M Keynes, who said day-to-day fluctuations in investment returns had an ephemeral and non-significant character. And, these fluctuations tended to have an altogether excessive, even absurd, influence of the market. About the same time, J B Williams noted in his theory of investment value that prices had based too much on current earning power and too little of long-term dividend paying power.

As this new evidence (information) overweight happens consistently, the portfolio of worst losers in three years always outperforms those of winners over the same period. But, why this common and profitable science does not become popular?

There is a simple answer. Humans rely on feelings more than scientific education. This is why we don’t buy losers but winners. This is why we don’t expect today’s losers to become tomorrow’s winners. This is why we suffer from biases. And, this is why ‘investing’ has become synonymous with wealth erosion rather than wealth creation. We are living under the Bayesian curse.