The Drunkard's Walk : Summary

It's Saturday morning and it's pouring in Mumbai. A perfect setting for a cup of hot tea and a book. Simple pleasures of life:) are sometimes all that one needs, I guess.

Was planning to read this book for quite sometime as it was touted as a book similar to Against the Gods and Lady Tasting Tea, books which I thoroughly enjoyed. After recovering from a brief illness this week, I thought I should spend my time peacefully with a book and managed to read this book in about 2 sittings. Let me summarize this book:

This book is a mix of historical narrative of statistics and author's own opinion about randomness. So, this book is NOT like the book mentioned above. It is more like Tableb's fooled by randomness, not as sarcastic as taleb though. It starts off with a view that randomness is not appreciated by most of us in our lives. We tend to make sense of patterns, situations, people using some notions which themselves could be based on some chance events. We do not carefully appraise our prior probabilities/ prior distributions/ prior notions based on the events we see.

Having made the point that lives, events, people, victories, defeats, success, love, emotions, relationships and pretty much everything is a stochastic process, the author goes on to give a brief historical narrative. Greeks , the pioneers of mathematics in terms of their contribution like axioms, proofs rarely ventured out to do anything on the probabilistic side of events. The same argument is presented here as that given in Against the Gods .They believed that the order/chaos is the hand of god. Romans, however were more flexible in their approach. They were interested more on the practical side of things and were the earliest people who started tinkering with the ideas of probability(BTW probabilis is a Greek word).

Cardano

Author writes at length about Gerolamo Cardano,an gambler by heart was the first to speak in terms of sample space. The concept of sample space is discussed in the context of Monty Hall problem.

Galileo too made a powerful observation that the chances of event depend on the number of ways in which it can occur. This is the frequentist view of things in the current world. This view had a tremendous influence on the developments in the field of probability.

Pascal Fermat

Pascal was hooked on the probability by thinking about the "balla problem", an example cited in almost every introductory session on probability. He also sought help of Fermat to vet his ideas. Pascals contribution among many others was the Pascals triangle, an easy way to know the number of ways in which a specific events can happen.

An interesting question which was not answered yet, at this point of time in history is ,
How to figure out the numbers in lets say 10,000th row / Millionth row in Pascal's triangle ?
Pause for a second and think about it

It was a problem during those times, but we all know that for large row number, the above pattern which was later called as binomial distribution approximates to a normal distribution..Anyways that's statistics 101..Now back to the book.

The author digresses at this point to talk about the importance of sampling and how we can be fooled by small numbers. Benford's law was a new learning for me(real-world measurements are often distributed logarithmically).

Jacob Bernoulli

The author then talks about the fusion of ideas from calculus and probability , thus culminating in to Jacob Bernoulli's Golden theorem( Law of large numbers). Until this point , the questions tackled were," What is the forecast for a set of fixed probabilities?" What are the number of trials to be fairly certain about an estimate of a distribution, given the population distribution ?"

But we all know that nobody knows prior distributions with certainty. Most of our lives we deal with data and we have to make inferences of the underlying probabilities. This is one the ways to distinguish PROBABILITY and STATISTICS. Statistics involves inferring about distributions. Probabilities that we learn in high school concerns pretty boring but important details about various things , once probabilities are known. Somehow this distinction was never made anywhere during my schooling/ college education. May be my instructors did make this point, but somehow I failed it appreciate it then. One needs to keep this distinction always in mind when dealing with real life problems. Given a poisson process with a lambda, you can do all the jazz in the world about various events and their likelihood of occurring..But unfortunately in real life, one has to face the questions like , " If I buy an option based on NIFTY, do I have use a GBM model for hedging or Poisson Jump Model ? How should I validate the hedging results and make estimates of a better underlying distribution ?. These are statistical questions!,..Anyways I am digressing from the book.

Bayes was the first person who used conditional probabilities in his work. He never publicized his work..Fortunately Laplace was also working on similar ideas and thus was instrumental in the development of this area. This is one field sadly which has still not been used widely , atleast in the finance discipline. People still use a lot of gaussian models and no wonder they fail...Life is not gaussian!. Gaussian Copula , one of the reasons of the subprime mess( Wired article here)

From here onwards the frequentist world dominated statistics. Gauss, DeMoivre, Fischer, Galton and a host of brilliant people pushed statistics in to solving real life problems. Einstein's usage of Brownian motion etc is one of the spectacular developments.

Here is where the book suddenly stops the narrative. Ideally there have been a lot many developments after this point in time, but the book kind of goes on to give author's view on randomness. These include the way we try to attribute patterns to randomness, randomness to patterns , the way we delude ourselves of being intelligent, when we are just being lucky etc. The last couple of chapters belabor this point, more than required I guess.

Overall, a nice little recount of people involved with randomness and author's view of randomness.

A trivia from the book which is kind of interesting. Much as we might think randomness is everywhere, it is pretty difficult to actually manufacture randomness. well, we all know there are host of methods to create random numbers, each with its own bias..The trivia which I found interesting is about Apple iPod Shuffle feature. Apparently there was a lot of customer feedback that shuffle was not really throwing up random numbers and listeners were under the impression that Shuffle was not developed with good randomization feature.....Here is what apple did. It DID NOT MAKE, the algo more random. it actually MADE IT LESS random and customers felt that songs were now more randomized!!!
Strange are the traits of randomness!