RK's Musings

Correlation Phantom

Day in , Day out one comes across reports, newspaper articles, business channel anchors using the word correlation. Correlation between NIFTY movement and some sector index, pollution rates Vs number of vehicles in the city, y vs x etc . If one stops to think about the word correlation and ask a few questions, opinions/arguments/citings all tend to fall apart.

In the statistical sense of the word, let me take r as a pearson correlation and bring out the first often neglected aspect of correlation. Like everything in the world of numbers, it is an estimate. This means that its got to have a sampling distribution .

If you have lets say 1000 values of X and 1000 values of Y and you want to find the correlation. well, one of the first things one usually does is to report Summation(x*y) / (n *Sx*Sy) where x and y are deviations of X and Y from their respective sample means , Sx and Sy are sample stdev of x and y respectively. Now this number is not sacrosanct. Its afterall an estimate. If you sample lets say 1000 values from X and 1000 values from Y with replacement, and calculate the correlation, you end up with another number of correlation. If you repeat this exercise umpteen number of times you get a sampling distribution for r, meaning r can take any number of values and the least you can do is to look at the frequency distribution of such numbers ...Can you form confidence bands for this ?

Well, one thing to note is that it is not a normal distribution. Fischer realized this and applied a transformation to r values and converted in to a normal dist, calculated bands in the transformed world and brought it back to the r world. Thus there is a big ram kahani to this simple task of forming confidence bands for r. Well, if r is >>0.4 , then you can assume to be normal and report the std error as (1-r^2)/sqrt(n) .

Ok. assume you have done all this and then you take 2 variables totally different in nature where you don't see any reason for them to be correlated. Let's say the # of runs scored by tendulkar since his debut in each match, # of car accidents in mumbai on each day of those matches. If the two variable are strongly(positively/negatively) correlated , common sense would say that it is totally meaningless. Spurious correlation is everywhere.

So, is there something wrong with the metric ? Or there is something missing about the way we think correlation works ? Or is there an assumption behind correlation that gets violated big time when you just use it on any two variables ? This was a question answered by a British statistician named Yule almost 80 years ago...Sadly, the results have not percolated in to everyday life...

Thus correlation has become one of the most notoriously used word to lend phantom quantitative props to whatever opinion one has to give!!

Next time you hear the word correlation being casually tossed around by someone, do try to ask a simple question - "Can you give some confidence bands for the correlation that you are talking ?" ..... In all probability they will go back and try to sincerely work out a possible answer and think through it.. OR.. they will never ever use the word atleast with you again :)

Beauty(Returns) is in the eye of beholder(Investor)

Take any asset , Every investor has a specific idea of return as it is based on the expected holding period, ability to short sell, turn over of the asset etc. If one were talk about returns, it is very difficult to give a generic metric. There are umpteen ways to calculate returns as one can imagine. For example,

• One could take daily returns and multiply by 252, daily standard deviation and multiply by sqrt(252) to get a risk return profile
• One could take weekly returns and multiply by 52, daily standard deviation and multiply by sqrt(52) to get a risk return profile
• One could take monthly returns and multiply by 12, daily standard deviation and multiply by sqrt(12) to get a risk return profile
• One could just take a mean of calendar year returns and sd of calendar year returns.

In Indian markets, there is no doubt that NIFTY, NIFTY Jnr index assets figure amongst the sought after assets. Also with Gold witnessing a dramatic rise year after year , it is attracting hordes of investors too. Lets look at these assets from a risk return perspective and understand the importance of the scale for returns calculation

Let's look at these assets over the last 5 years

Takeaways

1. Weekly annualized scale for Gold gives a completely different picture as compared to Daily annualized and Monthly annualized. The gold returns are about 5%. These numbers are important as one sees different numbers quoted in the media under different circumstances.
2. NIFTY Jnr is more risky than NIFTY. It is pretty obvious point as the mid cap is going to be more volatile than large cap.
3. NIFTY and NIFTY Jnr daily returns annualized have almost the same risk return whereas monthly annualization shows that NIFTY has a better sharpe ratio than Jnr.
4. Weekly scale - It is important to realize that distributions under weekly scale look so so different. Aggregation at this scale and metrics resulting from it paint a different picture

Let's look at the same graph after removal of 10% of outliers

Takeaways

1. Daily Scale return profile changes completely for Nifty and Nifty Jnr.
2. Weekly returns annualized for Nifty and Nifty Jnr starts making sense as the numbers are somewhat comparable to daily annualized and monthly annualized
3. One thing that stands out is that Gold gives about 15-20% returns whichever way you slice and dice daily and monthly data.

However this is just the beginning of looking at scales. For a realtime investors who want to stay put for an year, these metrics do not make sense as they would be interested in looking at returns between t1 and t2 where t1 could be any day of the year and t2 is accordingly 252 trading days away from t1. These are not rolled returns as rolled annualized returns have a different calculation. For lack of better terminology , I will use rolled returns to mean that returns between sets of ordered pairs (t1,t2) in the sample where t2-t1 is 252 days. Now amongst such ordered pairs, there are again three ways to calculate returns.

• One can take daily returns between t1 and t2 , add up , call it annualized returns , roll it on each day, , form a sample and compute stats
• One can take weekly returns between t1 and t2, add up , call it annualized returns, roll it for each week , form a sample and compute stats
• One can take monthly returns between t1 and t2, add up , call it annualized returns, roll it each month , form a sample and compute stats
• One can take point in time value returns between t1 and t2, call it annualized returns, roll it on each day, form a sample and compute stats

If one computes metrics as described above for the period 2005 to 2009 , here is how returns look

Takeaways

1. Weekly scale still paints a different picture
2. Daily returns and weekly returns look similar though the returns for each of the asset is shifted downwards by a specific amount
3. The last graph in the above illustration is probably useful as it uses point returns between ordered pairs(t1,t2) for every data point . If one thinks about it, this is what an investor with an expected holding period of 1 year would get .

There are umpteen other scales at which the data can be aggregated.

So, sharpe ratio of the assets reported in the media with out mentioning the proper definition of the scale used , is very dicey to interpret. Depending on what one wants to say, one can combine risk return metric at the appropriate scale and can use to further an opinion about the asset.

Why average is not a better estimate ?

Charles Stein stunned the statistician community when he first published a paper which showed that average of past events is not the best predictor of future event.

Here is the original article which uses baseball stats to give an insight in to the paradox -- Paradox in Statistics

Box Percentile Plot

This is a new addition to my diagnostic tool kit. I had never used Box Percentile plot before. However this week, I had to look at outliers in a data. Histograms are not that good for outliers. Box Plots, same story(though whiskers in the box plot are useful in getting an idea about the data points 1.5 IQR away from the boxes). I usually prefer Box plots for the simple reason that it gives a sense of the data from a quartiles perspective. . However most of the times you are interested in knowing more about them from a diagnostic perspective.

Box-Percentile plots build on the widely popular box plots and add a twist. One of the dimensions of the box in the box plot is not used. Look at the following figure of samples from Normal, Uniform w/outliers, Trimodal. All the box plots show the same illustration.
Box Percentile plots tries to use one of the box to the convey the same data, this giving a sense of outliers. They combine percentile plots which show ALL the data and box plot which is focused more on IQR.

Now whatever is written above is angrezi...Here's the math behind it.
If x_1,x_2.....x_n are ordered data points , then the data point x_k is plotted at a height wk/(n+1) if x_k is less than median, else it is plotted at a height (n+1-k)w/(n+1), where w is the desired width of the box....width is proportional to the percentile of the height

So, by construction , box-percentile plots give more information about the univariate distributions.

sos package : Google for R based search

While using R, there are umpteen instances one has to browse for the relevant package for a dataset / function / class. It is a time consuming activity....But now, thanks to the authors of sos library its an effortless exercise!...

Let's say you want to use robust estimator for a sample and you want to know whether R community has it already or not. With just three lines of code, you get a ton of summary relating to what you want to search

library(sos)
PL <- findFn("lmRob")
summary(PL)

Hearty thanks to all the authors behind this awesome package :Spencer Graves, Sundar Dorai-Raj, and Romain Francois

Package Link : sos

Work of a data analyst

Recently I came across a nice illustration which captures the work of a data analyst. Each of the tasks mentioned below requires a specific set of skillsets and if one can work on learning at least 30-40% of skills from each of the disciplines mentioned, then data analysis becomes really interesting. One can walk through the entire task list and subsequently hope to understand things and probably build some useful models.

If one lacks skills in any of the areas, then one has to depend on others for expertise. Depending on others means , sometimes, time lags, sometimes coordination problems etc etc. If one want to avoid that route , then one has to spend a LOT of time in gaining at least working knowledge in all the areas mentioned and be really really good at one of the aspects, be it acquiring data / parsing / filtering / mining / representing / refining / interacting with data.

It can easily be a life time effort to achieve expertise in each of the areas but at the end of it , the whole process might turn out to be a very satisfying journey.

The Drunkard's Walk : Summary

It's Saturday morning and it's pouring in Mumbai. A perfect setting for a cup of hot tea and a book. Simple pleasures of life:) are sometimes all that one needs, I guess.

Was planning to read this book for quite sometime as it was touted as a book similar to Against the Gods and Lady Tasting Tea, books which I thoroughly enjoyed. After recovering from a brief illness this week, I thought I should spend my time peacefully with a book and managed to read this book in about 2 sittings. Let me summarize this book:

This book is a mix of historical narrative of statistics and author's own opinion about randomness. So, this book is NOT like the book mentioned above. It is more like Tableb's fooled by randomness, not as sarcastic as taleb though. It starts off with a view that randomness is not appreciated by most of us in our lives. We tend to make sense of patterns, situations, people using some notions which themselves could be based on some chance events. We do not carefully appraise our prior probabilities/ prior distributions/ prior notions based on the events we see.

Having made the point that lives, events, people, victories, defeats, success, love, emotions, relationships and pretty much everything is a stochastic process, the author goes on to give a brief historical narrative. Greeks , the pioneers of mathematics in terms of their contribution like axioms, proofs rarely ventured out to do anything on the probabilistic side of events. The same argument is presented here as that given in Against the Gods .They believed that the order/chaos is the hand of god. Romans, however were more flexible in their approach. They were interested more on the practical side of things and were the earliest people who started tinkering with the ideas of probability(BTW probabilis is a Greek word).

Cardano

Author writes at length about Gerolamo Cardano,an gambler by heart was the first to speak in terms of sample space. The concept of sample space is discussed in the context of Monty Hall problem.

Galileo too made a powerful observation that the chances of event depend on the number of ways in which it can occur. This is the frequentist view of things in the current world. This view had a tremendous influence on the developments in the field of probability.

Pascal Fermat

Pascal was hooked on the probability by thinking about the "balla problem", an example cited in almost every introductory session on probability. He also sought help of Fermat to vet his ideas. Pascals contribution among many others was the Pascals triangle, an easy way to know the number of ways in which a specific events can happen.

An interesting question which was not answered yet, at this point of time in history is ,
How to figure out the numbers in lets say 10,000th row / Millionth row in Pascal's triangle ?
Pause for a second and think about it

It was a problem during those times, but we all know that for large row number, the above pattern which was later called as binomial distribution approximates to a normal distribution..Anyways that's statistics 101..Now back to the book.

The author digresses at this point to talk about the importance of sampling and how we can be fooled by small numbers. Benford's law was a new learning for me(real-world measurements are often distributed logarithmically).

Jacob Bernoulli

The author then talks about the fusion of ideas from calculus and probability , thus culminating in to Jacob Bernoulli's Golden theorem( Law of large numbers). Until this point , the questions tackled were," What is the forecast for a set of fixed probabilities?" What are the number of trials to be fairly certain about an estimate of a distribution, given the population distribution ?"

But we all know that nobody knows prior distributions with certainty. Most of our lives we deal with data and we have to make inferences of the underlying probabilities. This is one the ways to distinguish PROBABILITY and STATISTICS. Statistics involves inferring about distributions. Probabilities that we learn in high school concerns pretty boring but important details about various things , once probabilities are known. Somehow this distinction was never made anywhere during my schooling/ college education. May be my instructors did make this point, but somehow I failed it appreciate it then. One needs to keep this distinction always in mind when dealing with real life problems. Given a poisson process with a lambda, you can do all the jazz in the world about various events and their likelihood of occurring..But unfortunately in real life, one has to face the questions like , " If I buy an option based on NIFTY, do I have use a GBM model for hedging or Poisson Jump Model ? How should I validate the hedging results and make estimates of a better underlying distribution ?. These are statistical questions!,..Anyways I am digressing from the book.

Bayes was the first person who used conditional probabilities in his work. He never publicized his work..Fortunately Laplace was also working on similar ideas and thus was instrumental in the development of this area. This is one field sadly which has still not been used widely , atleast in the finance discipline. People still use a lot of gaussian models and no wonder they fail...Life is not gaussian!. Gaussian Copula , one of the reasons of the subprime mess( Wired article here)

From here onwards the frequentist world dominated statistics. Gauss, DeMoivre, Fischer, Galton and a host of brilliant people pushed statistics in to solving real life problems. Einstein's usage of Brownian motion etc is one of the spectacular developments.

Here is where the book suddenly stops the narrative. Ideally there have been a lot many developments after this point in time, but the book kind of goes on to give author's view on randomness. These include the way we try to attribute patterns to randomness, randomness to patterns , the way we delude ourselves of being intelligent, when we are just being lucky etc. The last couple of chapters belabor this point, more than required I guess.

Overall, a nice little recount of people involved with randomness and author's view of randomness.

A trivia from the book which is kind of interesting. Much as we might think randomness is everywhere, it is pretty difficult to actually manufacture randomness. well, we all know there are host of methods to create random numbers, each with its own bias..The trivia which I found interesting is about Apple iPod Shuffle feature. Apparently there was a lot of customer feedback that shuffle was not really throwing up random numbers and listeners were under the impression that Shuffle was not developed with good randomization feature.....Here is what apple did. It DID NOT MAKE, the algo more random. it actually MADE IT LESS random and customers felt that songs were now more randomized!!!
Strange are the traits of randomness!

Stats & Probability - Need of the hour

Here is a TED talk by Dr.Arthur Benjamin ( Mathemagic ted talk person) who says, that stats & probability should be elevated to the status that is given to calculus

 

Meeting PCA again

I was familiar with this technique, had used it in a couple of places. This week I had a chance to apply this technique to a problem at hand. However, this time I did not want to rush in . Had in my archives an old book called "User's guide to Principal Components". Took some time out to know a bit more about the technique. As always, there is always some new learning by reading old stuff.

A bit of history, to begin with. This technique was first introduced by Karl Pearson in 1901 (Refer to his other contributions here). The general procedure to carry out the technique had to wait until 1933 when Harold Hotelling introduced it in his paper. Thus 1930s and 1940s had a great deal of activities relating to PCA.Then things subsided and the development took place as a poisson process :). With the advent of computers, things changed. People were now able to find factors, invert matrices with ease and thus the application of PCA became widespread in various fields. An unfortunate development was that many social scientists extended PCA to factor analysis and interpreted factors by their own whims and fancies.

The author of this book , Mr. Edward Jackson has put in his thoughts from his working experience at Eastman Kodak for over 20 years. Books from practitioners are gems for they show how to marry theory and application , and more so, make you feel that there is no single answer to any problem. Smartness lies in using a specific technique to a situation with adequate mix of intuition and quantification. Another book, Super Crunchers, makes the same point

Principal Component Analysis, abbreviated as PCA, is a data exploration technique as well as an inferential technique. The methodology is same for population as well as samples( a greatly comforting aspect). Also most of the aspects relating to PCA are distribution free.

Let's say your are a police officer in charge of keeping a tab on the militant threats in Mumbai. Lets put some numbers to the situation. Imagine you track 10 variables at different points in the city.. You figure out that these 10 variables should to be present in a specific bands so that you are fairly confident of no militant threat in Mumbai. You have put in place police infra, informer infra etc in the city to get 10 dimensional vector from all the potential threat points in mumbai, let's say 5000 points, ending up with a 5000*10 matrix of observations.All the variables can be quantified on a scale of 1-10.Lets say the first variable x1 can be quantified on a scale of 1-10 and the mean is 5 and the bands are at 4.5 and 5.5. Any observation beyond this band might make you suspicious. Lets say you keep a track of all these 10 variables on a big chart. So, basically you add daily observations to your dashboard .

If all the 10 variables are with in the bands(lets say these are 95% confidence interval bands), would you be 95% confident that there is no militant threat ?.

At the outset, it might seem correct. You are tracking 95% intervals for each of the variables. However , on thinking a little bit, you realize that the type I error shoots up. Reason, the joint probability of threat,thanks to Type I errors is (1-0.95)*(1-0.95)*....10 times = 0.4. This means that if you control all 10 variables , there is 40% chance that one of the variables is outside the bands!!! Now this is assuming that the variables are uncorrelated. But think abt it, variables are often correlated , especially in a militant threat kind of situation. So , your Type I error goes up. What do you do ? Well , there are a couple of things you can do. One is see to it that the bands are designed in such a way that Type I and Type II error is built in to bands mechanism OR, do something which is relatively easier,Do not track these variables x1,x2,x3,.....x10 for 5000 sites, but a different linear combinations of these variables call then X1,X2,X3,X4 which have one great property , which is, the transformed vectors are uncorrelated.

Basically this is my pathetic attempt :) at explaining PCA with out any matrix notion. PCA helps you view the 10 variables sets in a different perspective which gives a lot more clarity on the relational structure in the data. By looking at the bands and Hotelling T-square you can act on the variables realization.Ok, now to say the same thing in matrix notation. Whatever be your input matrix, Let S be covariance matrix then one can find an orthonormal matrix U such that U'SU =L , a lower triangular matrix. The diagonal elements are eigen values which can also be obtained by solving |S-m*I|=0 .Characteristic vectors are obtained by (S-m*I)Xp = 0 where Xp is the characteristic vector.Thus one can transform the original data to z where z=U'[x-xbar). ( With out latex support, writing math is particularly difficult on typepad ). One use of this transformation is that L matrix is very useful becoz trace of L is the sum of original variances and det|S| is just the product of eigen values.

Another important aspect that I have learnt from this book is about the stopping criterion. I was using proportion of variance explained as a stopping factor for the number of components selected. Why ? I don't know....I just accepted somebody's word. I was completely dumb in not even thinking about it.However this book revealed to me that there is nothing great about that rule. Bartlett's test provides a better way to scientifically create a stopping condition. This was new to me. Somehow , it is never mentioned in the usual 10,000 ft view read of PCA. There are at least a dozen stopping criterion and I guess at least crunching half a dozen would bring upon a healthy dose of skepticism in your views.

What I could summarize above is just 5% of the book. The title is definitely an understatement. It should be have been "all encompassing guide" becoz in whatever field you are, for whatever problem you want to use PCA, there is a guideline in the book. This is one of those good books which focus less on abstract theorems and more on practical applications.

Theorizing Vs Empirical analysis

With easier access to raw computing power, one might think that all it takes is to analyze data and let the data throw hypotheses.

Is theorizing over in this data intensive world ? Has the world become so complicated that theorizing is useless? May be not.

I read this short note(Tom Swift and His Electric Factor Analysis Machine) couple of years back but I always reread it frequently so that I dont make the mistake of living only in an empiricist world. Some amount of apriori assumptions/ hypotheses are required. In the financial world, traders provide this knowledge. It is easier sit at a desk and crunch data and build models. However it might be a complete useless exercise if you do not marry apriori assumptions with your empirical work.

Here is a Link to this beautiful article: Tom Swift and his electric factor analysis machine

R : The Linux for Statisticians

Via NYTimes:

 

Robert Gentleman & Ross Ihaka (people behind R)

“R is a real demonstration of the power of collaboration, and I don’t think you could construct something like this any other way,” Mr. Ihaka said. “We could have chosen to be commercial, and we would have sold five copies of the software.”

“The great beauty of R is that you can modify it to do all sorts of things,” said Hal Varian, chief economist at Google. “And you have a lot of prepackaged stuff that’s already available, so you’re standing on the shoulders of giants.”

I guess once you try R, you will be hooked forever.

Data Vs Pattern

Often times most of the data / variables that we come across appear random and we start taking decisions based on gut. If gut works , its great. However before bringing in the gut part , it might be sensible to stretch a bit and look at data carefully and examine the details.Some time, just by changing the our view, we might see patterns. well PCA is all about that. I don't have to go that far to make this point. For example, if one looks at the current graph of sin(x) for some values of x, it looks like this:

The data points appear completely random. However just by tweaking the aspect ratio(x:y), the same data looks like this:

I tend to believe to that there are patterns everywhere and it all boils down to spending time with data and trying to understanding it carefully.However one also needs to be skeptical about all the patterns that one finds and have a null hypothesis that pattern is random, back test the pattern and then form an opinion. Instead of having a null hypothesis that data is random, it is better to have a null hypo that data pattern is random.

Quote for the day

In God we trust; all others bring data.

—Dr. W. Edwards Deming

> rnorm(15) Thoughts

A Random sample of my thoughts

1. If Nifty falls / Nifty rises , does the implied vol move in the same direction of the underlying or does it move the opposite direction. This becomes a very important question in the context of choosing a local volatility idea or chucking such an idea ? Local vol models assume that underlying and smiles move in different directions!!
2. As there is a short sale restriction on stocks, what exactly is a tradable underlying for Nifty option, for valuation purposes ? Yes you do hedge a long call option with whatever is a cheap, a future/ETF. But then is it sensible to treat a long call option as option on futures and put option as an option on index, the reason being that that they are the respective hedges for the options available
3. Does vol of nifty determine the atm imp vol level for nifty options? If so, how does it move ?
4. Is there a curvature for vol across strikes ? If so, does the curvature remain quasi-stable?
5. What is the functional/deterministic/stochastic relation between the comovement of nifty futures and nifty options in a 20,100 seconds(9:55AM - 3:30PM) trading day ?
6. What is the average trade volume per moneyness in a trading day ? This is easy...got to crunch a few numbers to get an idea
7. World-over the realized vol is explained by atm variance largely(80%)? Is it the case in Indian markets ? Are skew and convexity merely jazz in Indian markets which one need not worry too much ?
8. If you look at an option trade that happens at time t1, what would be the corresponding future price that you would consider for valuation? the instantaneous futures price, average of future prices in the interval (t1-delta, t1)? Can you choose delta with some rationale, meaning, some metric which justifies choosing last 5 min / last 10 min / whatever that one thinks is meaningful ?
9. Parameters for Vol of Vol for Nifty - Are they stable and more importantly what is their impact on Imp vols
10. Is imp vol in Indian markets a completely random animal ? Is Indian market completely a punter's market , meaning every one is a momentum player ?and analyzing numbers beyond a point is a useless activity ? ( I hope not )
11. Are circuit breakers, fancy as they might sound, properly designed? Shouldn't financial markets be there for the price discovery to happen? How can the third busiest exchange in the world trade for 30 seconds (Monday,after election verdict) and let the order flow go to other stock exchanges ?
12. Are Indian investors sensible ? Nobody wants to buy when nifty was 2600 level and now when it has reached 4300 , every one wants to buy? Shouldn't it be the other way round ?
13. Who makes money in the entire game ? Are the only people who make money are Custodians, Fund account management fees, brokerage houses, financial advisors? Is "extracting decent alpha consistently", a fantasy and there is survivorship bias amongst the successful fund managers that we see around ?
14. How do people trade reliance options or for that matter any of the American options on the Indian market? Are there good pricing and hedging tools for these options available with traders ?
15. Gap up and Gap down!!! , the biggest problem in the Indian markets -- How do you sleep well when you know that you have got to deal with such an issue the next day morning ?

................................May be I will find clues to the above stuff, someday!

Evidence Based Technical Analysis : Summary

 

This is a book which I have been planning to read for quite some time, may be since an an year . Finally found sometime to spend on this book.
However the book turned out be just about OK, not engaging, considering the amount of hype around it .Having said that, this book has to be read by anyone who is in to back testing trading rules / strategies and has only a vague idea of stats!

Anyways, Here are my brief takeaways :

Chapter 1 - Objective Rules and Their Evaluation

Takeaways

• One needs to formulate a basic idea of a rule. Rule is an objective statement that can be implemented by a computer program and that generates unambiguous long/short/neutral signals
• Position bias needs to be tested in a trading rule. Randomize + Equivalent position would serve as a good control group
• Detrending is very imp in back testing rules
• Avoiding look-ahead bias and accounting for trading costs are very important!

Chapter 2 - The Illusory Validity of Subjective Technical Analysis

Takeaways

• Important to understand the various bias that can creep in validating a claim
• Overconfidence,Optimism bias
• Hindsight bias
• Confirmation bias
• Illusory correlation
• Sample size neglect
• Representative bias
• Biased second hand knowledge
• Attribution bias
• Knowledge Illusion
• It is important to recognize it is easy to fit a pattern to a time series. However it could be only wishful thinking unless it is tested vigorously with scientific rigor.
• Good stories well told can make people misweight or ignore facts.
• People are not naturally rigorous logicians and statisticians. A need to simplify complexity and cope with uncertainty makes us prone to seeing and accepting unsound correlations. We tend to overweight vivid examples, recent data and inferences from small samples.
• Scientific method is a reliable path to validity, mitigating the misleading effects of our cognitive biases

Chapter 3 - The Scientific Method and Technical Analysis

Takeaways

This chapter summarizes the development of scientific method. Aristotle's syllogisms, Bacon's contradictory thinking, Popper's falsification method, Null Hypothesis-Alternate hypothesis development. This chapter leaves the reader with a powerful impression that advancement in any field needs the 5 stages of work

• Observations
• Hypothesis
• Predictions
• Verifications
• Conclusion

Unless these become a part of TA, practitioners of TA are nothing but astrologers, alchemists and folk healers

Chapter 4 - Statistical Analysis

Takeaways

Nothing much ...For a person who doesn't know abt stats, this chapter might be useful...Definitely skimmable for someone who uses stats for rozee roti

Chapter 5 - Confidence Intervals

Takeaways
Nothing much...Except that one can use resampling method Or Montecarlo method for testing hypothesis and establishing confidence intervals.
Most of the content in this chapter is Stats101. One takeaway for me,is that one has to zero-center the sample for conducting resampling method. This is done to align with the universal null hypothesis of a trading rule -Ho- Returns from the trading rule are 0.

Chapter 6 - Data mining Bias - Fool's gold of Objective TA

Takeaways
For me, this section is the meat of the book. I took quite some time to read slowly what this section talks about and here are my takeaways.

• After the fact probability tends to much higher before the fact. A monkey coming up with a master piece is very less, but given a history, the probability of a masterpiece being produced is very high
• Be aware of data mining bias - Random component + Genuine predictive power of rule , the random component dominates the data mining route. You are just lucky with the rule in the history and out of sample is bound to under perform.
• Data mining - Always the best performing rule is chosen...Well, the rule was just lucky in that period
• Five Factors one needs to think about Number of rules being tested, Data size, Correlation between rules, variation in the expected returns amongst the rules, Presence of outliers
• Three ways to cut DM Bias - Out of sample testing(walk-forward testing), Markowitz scaling down method, Resampling methods and Montecarlo methods

Chapter 7 - Theories of Non Random Price motion

Takeaways
This section looks at the shaky foundations of Efficient Market Hypothesis. All forms of EHM are analyzed. Arguments based on logic as well as empirical evidence are put forth in order to show that EMH is crap and there is enough scope for Arbitrage opps in a market

S&P Case study:
The last part of the book deals with testing about 60,000 binary signals to S&P data. If you can read through relatively dry chapters 1-7, you are bound to enjoy this part of the book. Once you start thinking about the ways in which trading rules can be built , this section is priceless as allows to zoom in to various aspects of back testing..

Personally, I think that the S&P case study at the end is far more valuable than the first 400 pages of dry read. However if you haven't used stats for a while, you might like the first 400 pages too.


My takeaway from the book is : There are a ton of insights from the S&P case study towards the end of the book!, the rest is jazz.

The Lady Tasting Tea : Visual Summary

This post is going to serve as a good visual narrative of the development of frequentist world. If you love stats and have worked on identifying patterns in data, it is but obvious that you would have met a host of tests, with different names from different fields. Statistics is one field where the contribution has been made from all kinds of fields ranging from agriculture, clinical psychology, math, finance etc.

WHY do you think so many people contributed to this field ? Just pause for a few seconds and think about it.

I had never paused and thought about it.... and the opening remarks of this book provided that insight,though very obvious in the hindsight. Statistics, a significant part of it deals with the way experiments need to be conducted ,and experiment/test/learn is the way of life for any scientist irrespective of the domain he/she is working. So ,the contributions to the field of stats are going to come from all possible fields.

Ok,let me attempt to do a visual summary of this book, for the simple reason that this book is a fantastic narrative of the history of statistics, the people who contributed, a peek in to their idiosyncrasies, their likes,their dislikes, is something that is a delight for the readers. I will try to cover most of the personalities mentioned in the book and their contributions.

Karl Pearson
Pearson was the first to look at the world from the eyes of a "distribution". What we are see are nothing but realizations of a distribution. If we collect more data, then we know more about the distribution. Under this assumption , he went on to create families of distributions based on mean, standard deviation , skewness and kurtosis. Experimental results are thus a distribution of numbers and distribution equations tell the probability of occurrence of these numbers. Measurements themselves have a probability than the errors in measurements which was the prevalent thought ( Galton , Abraham De Moivre, Carl Gauss).

Pearson also needed a tool to fit the measurements to distributions and he came up with a powerful tool called "Goodness of fit" which is used till date. (An example, if you have to select amongst a host of ARIMA models , for a given realization, one uses goodness of fit like AICC and chooses the model)

Gosset

Gosset was a classical empiricist. He felt Pearson theory is good but difficult to practice in reality as one needs to deal with small sample sizes. He worked on this problem after his work hours( classic success of "sex and cash" theory) . He said two estimates mean, standard deviation are enough to say something about the distribution. He published under a pseudonym - "Student t". This was very useful ..think about it. With pearson's case, you estimate 4 parameters, then you estimate the estimate of 4 parameters, ..it is an infinite loop. But with Gosset's insight, you stop at the first computation....wow!! It is a marvelous achievement.

Ronald Aylmer Fisher
Fisher, a personality, has had tremendous influence in the development and usage of statistics. His experiments, "Studies of crop variation", produced gem of results. ANOVA, MANOVA,ANCOVA, etc. Separating the main effects of the experiment was the underlying philosophy behind randomization. "Statistical Methods for Research Workers", a book stripped of complex math equations and made available for easy usage was instrumental in the adoption of Fischerian thought everywhere. Fischer was the person who first introduced the words "Degrees of freedom". may be becoz, he was always inclined to think geometrically.

The fundamental difference between Person's philosophy and Fischerian view is that : Pearson believed that if data represented distribution. Fischer believed that distribution is an abstract concept and he believed that all one can do is find a statistic describing the abstract concept. This statistic can be anything, mean, median, iqr etc. All these statistics will be random and hence one needs to study these estimates as such. He was also instrumental in coming up with "Maximum Likelihood estimates" , a way to iteratively figure the best values for the estimates give the data and a distribution .Today with the advent of computers, mle is a command away to this powerful and time consuming procedure which was very mathematical and laborious to do by hand.

Fischer also introduced p values, which went on to become the basis for hypothesis testing

Tippett Gumbel, Emil Julius

100 year flood prediction is difficult using Fischerian concepts . Hence Tippet and Gumbel studies this aspect and made a significant contribution. Today we know the work by the name Gumbel distribution.

Neyman

Neyman was the first person to ponder on the question, of connecting p values to hypothesis statements. He figured out that one needs to have a null hypothesis and alternate hypothesis for an experiment to use p values. One can only reject null hypothesis or fail to reject null hypothesis. No causality is being talked in here. This was his biggest contribution. He also developed and gave the interpretation for the word "confidence intervals"

Bayes

Bayes , a priest by profession, made an outstanding contribution to the world of stats. He dealt with the world of inverse probability. Look at the data and adjust the prior hypothesized distribution was the thought, a thought which was very very radical. For some reason, Bayesian stats is still not taught properly in MBA courses, Finance courses across the world, in spite of the fact that Google made tons of money using baye's fundas. There are umpteen disciplines that are crying for the application of bayesian principles . Risk management, for certain!

LEBESGUE, Henri Léon - Lebesgue measure

Lebesgue measure -- Key to the development of Real Analysis

Kolmogorov, Andrey Nikolayevich - MOZART of Mathematics Kolmogorov's contribution to probability and statistics is pivotal.

He was the first mathematician to use measure theory and lift probability from a step sister treatment in math, to the grand status of what it is today. With out Kolmogorov, it is very unlikely that development would have happened at this rate. He pondered on 2 questions and spent his entire life on them

• What are the mathematical foundations of probability ?
• If there is a set of data in a time interval, how does one interpret them ?
  
  He also coined the term "stochastic process" which is the bread and butter of quants

Florence Nightingale David

Lot of people think that she was founder of nursing profession. That is just one part of story.But for stats guys, her contribution to the theory of statistics is path breaking. Any paper you pick on stats, with in 2 -3 handshakes, you will find a reference to F.N. David's work

Wilcoxon, Mann of Mann-Whitney test

Why not do away with parameters, a revolutionary thought, in the hind-sight was first envisaged by Wilcoxon, Prof Mann and his student Donald Whitney. Their efforts gave rise to an entire branch on non parametric statistics.

Prasanta Chandra Mahalanobis
Mahalanobis is credited for coming up with "Randomized sampling", instead of "opportunity" Or "judgment" sampling. Under the then Indian PM, Nehru, he went to create economic indices which became crucial for tracking the performance of various five year plans.

Jerome Cornfield

The first example of inverting a 24*24 matrix and helping a noble prize winning economist are some of the few contributions of Jerome Cornfield. In a way one can say that he helped in bringing out the popular Input-Output economic analysis.

Many of Professor Cornfield's numerous contributions to both biostatistics and public health grew out of his research on the health effects of smoking. He became interested in the use of case-control studies after reading seminal papers by Doll and Hill (1950) and Wynder and Graham (1950), which used this methodology in early discoveries of the association between smoking and lung cancer. Professor Cornfield then demonstrated that case-control studies can be used to estimate the risk of disease as a function of smoking status so long as the rate of disease in the population is known. He also showed that the odds ratio, an approximation to the relative risk for rare diseases, can be estimated either prospectively or retrospectively. These results form the basis for much of the modern era's epidemiologic research.

Gertrude Cox

Cox is the first woman to be elected into the International Statistical Institute.In 1950 Cox and William G. Cochran wrote the book Experimental Design that became a classic in the design and analysis of replicated experiments.

Stella Cunliffe

Another towering woman in the statistical world

Samuel S Wilks

Samuel Stanley Wilks was an American mathematician and academic who played an important role in the development of mathematical statistics, especially in regard to practical applications.Wilks worked with the Educational Testing Service in developing the standardized tests like the SAT that have had a profound effect on American education. He also worked with Walter Shewhart on statistical applications in quality control in manufacturing.

J Tukey - The Picasso of Statistics

Fast Fourier Transforms, Exploratory data analysis, box plots, stem-and-leaf plots, rootgram instead of histogram are some of the stellar contributions of this versatile scientist.

George Box Gertrude Cox

Famously remembered for the Box-Cox transformation, a technique used to reduce data variation, make the data more normal distribution-like, improve the correlation between variables and for other data stabilization procedures.

Deming

Deming is credited to have brought quality movement. His contribution to the quality was from a statistical standpoint where he looked at variation in output as a combination of common cause variation and special cause variation. He championed for the reduction of common cause variation and that changed the face of Japan

LÉVY, Paul Pierre

Levy, was dissatisfied with counting methods in probability. He developed and applied Martingales to clinical trial studies and today Martingales are used in so many domains, finance, modeling, you name it...Martingale has entered the common vocabulary, thanks to Levy

David Dickey ( Dickey-Fuller unit root test - Stationarity test)

Stationarity of residuals , one of the methods to check is the dickey fuller test.

BAHADUR, Raghu Raj

India born mathematical statistician considered by peers to be "one of the architects of the modern theory of mathematical statistics". He is popularly known in the context of Anderson-Bahadur algorithm

Wald

Inverse Gaussian distribution, also called Wald Distribution , came from Abraham Wald.Also popularly known for Wald Chi square Test which is a statistical test, typically used to test whether an effect exists or not. In other words, it tests whether an independent variable has a statistically significant relationship with a dependent variable.

Brad Efron

Credited for discovering "Resampling method", one of the greatest breakthroughs in field of statistics

Finally, the author who has has put a fascinating account of the above personalities and many more, all in one book - Dr. David Salsburg

Factor analysis needs a priori theory

While hacking some trading techniques, I came across an author trying to link Factor analysis and Arbitrage Pricing Theory(Ross). I was wondering the use for it . My thinking was a classic case of an empiricist. Today I came across a reference to a old paper which says the link provided has some meaning to it. It is always better to have some theoretical model before pca/factor analysis/ eigen value decomposition.

Just because there is computational power and software doesn't mean that your hypotheses(read crap, most of the times) are true.

In the words of the J. Scott Armstrong

"The cost of doing factor analytic studies has dropped substantially in recent years. In contrast with earlier time,. it is now much easier to perform the factor analysis than to decide what you want to factor analyze. It is not clear that the resulting proliferation of the literature will lead us to the development of better theories. Factor analysis may provide a means of evaluating theory or of suggesting revisions in theory. This requires, however, that the theory be explicitly specified prior to the analysis of the data. Otherwise, there will be insufficient criteria for the evaluation of the results.
If principal components is used for generating hypotheses without an explicit a priori analysis. the world will soon be overrun by hypotheses."

Link to the paper : Tom Swift and His Electric Factor Analysis Machine

An edge

Where do we use conventional stats in the real world ? Taleb says there is an edge beyond which it does not make sense. Here is an illustration from his recent article

 

Link : An Edge
Appendix is worth going over too - Technical Commentary

A reread of Black Swan

It's close to 2 years since I read "Black Swan" for the first time. It had a definite effect on me for most of the arguments the author puts forth are very strong. One thing that propelled me in rereading this book was an article by one of my Prof's who said that the teaching system for MFE courses should radically transform itself for it to have any use in the times to come . I was immediately reminded of Ludic Fallacy which Taleb mentions in his book. So, managed to reread the book.

Let me list my takeaways from the book

• There are two kinds of worlds - Mediocristan and Extremistan . Dynamics are completely different in these two worlds.
• In Mediocristan, Law of Large Numbers works. In Extremistan, you cannot estimate the probabilities of rare events
• Many domains in our lives are increasingly falling under Extremistan
• Beautiful example of Turkey which is fed 1000 days before it is killed .For turkey , 1001 day is a Black swan.For a butcher it is not
• Do you want to be speculator or a prostitute in life ? This question is similar to what Mr. Nama had asked me years ago when I was doing my internship at GE ? He guided/suggested me to be a speculator rather than a prostitute. The principle is same as the cash flow quadrant idea brought out Roberto Kiyosaki
• Beware of the Ludic Fallacy
• Keep in mind the various types of bias - Confirmation bias, Narrative Fallacy, Silent Evidence, Tunneling in to Future
• Beware of EXPERTS in domains where things move.
• Melting Icecube analogy to show how complex the estimation of probability is, in real life
• Barbell Strategy - Invest 80-90% of money in risk free securities and 10% of money in extremely high risk securities. If one thinks of this strategy and applies to real life, I think it will have tremendous implications.
• An entire chapter which bashes Gaussian thinking.
• Introduction to Mandelbrot's fractal way of thinking . Such thinking makes black swans atleast gray swans , in taleb's words
• History Jumps, It doesn't crawl...Just forget the geometric brownian motion, arithmetic brownian motion, etc etc...It's all a bunch of crap as at its fundamental level, it believes in a gaussian world
• AVOID BEING A SUCKER , by exposing yourself to a lot of blackswans instead of avoiding blackswans
• Tinkering is the only way to know what works and what doesnt
• Missing a train is only painful if you run after it
• It is more difficult to be a loser in a game you set for yourself. - ( Brilliant point..Set up your own game and have fun , rather than living somebody else's life)

The reread was as entertaining as the first time I read it 2 years back...Books are indeed man's best friends!

Resampling Methods

When I came across this statement ," Bootstrap method has now been hailed by an official American Statistical Association volume as the only “great breakthrough” in statistics since 1970 " , I was humbled by the fact that I had never come across this method earlier.

The only kinds of bootstrapping I am aware of is

1. Bootstrap a venture :)
2. Bootstrap method to build yield curve using a set of fixed income instruments.

I came across bootstrapping as a method to find out the holding period of arb trading strategy . Apparently it is used widely in calculating the holding period of a stock/stock pair using one single realization. Naturally I was curious to know more about it and stumbled on to this beautiful online resource Resampling - Julian L Simon

If you can spare 10-12 hrs on this online resource, I am certain that you will see that resampling methods are intuitively far superior way of dealing with statistical problems as compared to fischerian formulaic non-intuitive approach. Overall the learning from the online resource has been good. It has motivated me to look in to one of the popular books on this subject : Resampling Methods

My takeaway : If you do number crunching for rozee roti,resampling methods would no doubt enhance your intuition about statistics . Afterall, the difficulty with statistics is not with the math, but with the kind of thinking one needs to do , in order to cull out patterns.

Bayes Vs Fischer

Another difference which one can keep in mind to understand the philosophical difference between Bayes and Fischer's way.

In the context of parameter estimation,

Bayes World : Parameter is random and credible intervals are fixed and one talks about the probability that estimator is present in that interval.Probability being referred to the posterior probability of the parameter

Fischer's world : Parameter is constant and confidence intervals are not fixed. For example 90% confidence interval means, if a large number of independent 90% confidence intervals are constructed, then approximately 90% of them will have the true parameter.

For some reason which I can't understand, Baye's is not taught in a lot of schools around the world and one gets too much of Fischer fundas that one fails to appreciate Baye's approach.

Carol Alexander in her new book on risk management mentions that very few people in the risk mgmt profession use Baye's and says that it is rather unfortunate that risk managers rely on Fischerian stats and make a mockery out of risk mgmt.

Fischer Vs Bayes

Couple of folks have emailed me to post the link of the article that I was mentioning in one of my previous post. The document attached with this post is perhaps a remarkable view from a person who understands both sides of the world, Fischerian and Bayesian, and proposes an alternative view for design inference.

One can't speed read this document. The article needs to be read slowly to understand it thoroughly...It is certain to make any reader wonder about the effective use of stats for drawing inference

Link to the document - Fischer Vs Bayes

Design of Experiments

I am in to designing a modeling software in my current work. In relation to that, I had to review some concepts relating to Design of Experiments. This can be a potent concept while carrying out real life experiments in the lab.

I just dont remember any of the experiments  I did in my Under grad. May be they were just single run experiments and hence did not require a DOE as such. Sometimes when I look back, I really feel that the experiments and the education related to it that I received was was far removed from the real world scenario. May be surveying that I learnt in my engineering came any where close to making one understand the real difficulties of experimentation and drawing conclusions.

The next time I came across DOE was during a training session , when I was working in the Quality department of an MNC. However I never got a chance to carry out the DOE in terms of a real life application. Today when I review the fundas, the concepts are definitely robust and take in to  consideration a lot of aspects. One thing I am hungry of is to work on a real life experiment and use the concepts of DOE.

Anyway, here are some basic aspects of DOE :

Model a continuous Dependent variable Y which depends on a set of continuous and discrete variables. Uncontrolled factors such as machines, week of the day, etc can have an effect on the experiment outcome. Hence blocking designs are incorporated so that blocking factor has minimal impact on the experimental outcome.

• Y is continuous
• Objective is to fit a linear or quadratic form of equation between Y and X’s.
• Linear Model generates main effects and interaction effects graphs
• Quadratic form generates response surface graphs which can be used to study the dependence
• Common Designs
  • Comparative Designs
  • Completely Randomized Designs
  • Blocked Randomized Designs
  • Screening Designs
  • Full Factorial
  • Fractional Factorial
  • Plackett-Burman designs
  • Advanced Modeling
  • Response Surface Modeling
  • Regression Modeling

Main Steps in a DOE

1. Set objectives
2. Select process variables
3. Select an experimental design
4. Execute the design
5. Check that the data are consistent with the experimental assumptions
6. Analyze and interpret the results
7. Use/present the results (may lead to further runs or DOE's)

Step 1: Objectives

The first step is to decide is the type of DOE. Is it going to be a sequential / Iterative DOE? It is not necessary to complete the experiment in one go. Iterative approach is the one which is followed widely

Assumptions in DOE:

• Are the measurement systems capable for all of your responses ?
  • Measurement systems in place for Y
• Is your process stable? 
  • Use run charts to gauge this
• Are your responses likely to be approximated well by simple polynomial models?
  • If not, its useful to break up the experiment in to separate experiments
• Are the residuals (the difference between the model predictions and the actual observations) well behaved?
  • Residuals need to be (roughly) normal and (approximately) independently distributed with a mean of 0 and some constant variance.
• Tests for Residual Normality
  • Histograms
  • Normal Probability plots
  • Independence of residuals over time
  • Independence of Residuals from Factor Settings
  • Plot of Residuals Versus Corresponding Predicted Values

Step 2: Select Process Variables

Process variables include both inputs and outputs - i.e., factors and responses. The selection of these variables is best done as a team effort. The team should

• Include all important factors (based on engineering judgment).
• Be bold, but not foolish, in choosing the low and high factor levels.
• Check the factor settings for impractical or impossible combinations - i.e., very low pressure and very high gas flows.
• Include all relevant responses.
• Avoid using only responses that combine two or more measurements of the process. For example, if interested in selectivity (the ratio of two etch rates), measure both rates, not just the ratio.

Step 3: Select Experimental Design

There are a whole host of designs that one can choose from. It depends on a number of aspects like factors, levels, budget and time constraints

• Completely randomized designs
• Randomized block designs
  • Latin squares
  • Graeco-Latin squares
  • Hyper-Graeco-Latin squares
• Full factorial designs
  • Two-level full factorial designs
  • Full factorial example
  • Blocking of full factorial designs
• Fractional factorial designs
  • A 23-1 half-fraction design
  • How to construct a 23-1  design
  • Confounding
  • Design resolution
  • Use of fractional factorial designs
  • Screening designs
  • Fractional factorial designs summary tables
• Plackett-Burman designs
• Response surface (second-order) designs
• Central composite designs
• Box-Behnken designs
• Response surface design comparisons
• Blocking a response surface design
• Adding center points
• Improving fractional design resolution
• Mirror-image foldover designs
• Alternative foldover designs
• Three-level full factorial designs
• Three-level, mixed level and fractional factorial designs

Description:

Completely randomized designs
As the name suggests, randomize the number of runs which equals = Factors * levels*Replications

Randomized block designs
When there is some nuisance factor which affects the experiment, one can use the randomized block design to over come the interference of the factor. There is a single factor of primary interest, typically called the treatment factor, and several nuisance factors. For Latin square designs there are 2 nuisance factors, for Graeco-Latin square designs there are 3 nuisance factors, and for Hyper-Graeco-Latin square designs there are 4 nuisance factors.

Full factorial designs
If there are k factors, each at 2 levels, a full factorial design has 2k runs. The design consists of replication, randomization and center points

Fractional factorial designs
Choose a fraction and then decide to run that fraction of total number of runs possible. The way to go about deciding which runs to include is of prime importance. Whatever the strategy one chooses in the definition phase, it is better to choose a design which is balanced and orthogonal.

Plackett-Burman designs
Screening Design where only main effects are of importance


Step 4: Analysis of DOE Data

• Look at the data. Examine it for outliers, typos and obvious problems. Construct as many graphs as you can to get the big picture.
  • Response distributions (histograms, box plots, etc.)
  • Responses versus time order scatter plot (a check for possible time effects)
  • Responses versus factor levels (first look at magnitude of factor effects)
  • Typical DOE plots (which assume standard models for effects and errors)
  • Main effects mean plots
  • Block plots
  • Normal or half-normal plots of the effects
  • Interaction plots
• Sometimes the right graphs and plots of the data lead to obvious answers for your experimental objective questions and you can skip to step 5. In most cases, however, you will want to continue by fitting and validating a model that can be used to answer your questions.
• Create the theoretical model (the experiment should have been designed with this model in mind!).
• Create a model from the data. Simplify the model, if possible, using stepwise regression methods and/or parameter p-value significance information.
• Normal plot of all the effects: All the effects thrown out by regression are clearly seen to be away from the normal distribution graph. This is another way to validate the effects that one is considering in the model
• Test the model assumptions using residual graphs.
  • If none of the model assumptions were violated, examine the ANOVA.
  • Simplify the model further, if appropriate. If reduction is appropriate, then return to step 3 with a new model.
  • If model assumptions were violated, try to find a cause.
  • Are necessary terms missing from the model?
  • Will a transformation of the response help? If a transformation is used, return to step 3 with a new model.
• Use the results to answer the questions in your experimental objectives -- finding important factors, finding optimum settings, etc.

Software that can be used to get one's hands dirty in using and applying DOE: camo.com

Hope some day I will utilize this knowledge that I have gained over the last one week.

Tic - Tac - Toe : PCA

Have you ever wondered why X and O are used in the popular game Tic-Tac-Toe ?
Why not use a , b for the game ?

If one looks at the english alpabets , one can fit an alphabet in 5 by 5 squares with each dark square as 1 and white square as 0, then what we have done essentially is to represent each alphabet in a 25 dimensional space.

 

Let's say we now plot a diagram where each alphabet can be drawn on a 2 D plan , which means reducing the dimension of the data from 25 to 2. The way it is done in mathematics is called Principal Component Analysis. The objective of principal component analysis is to reduce the dimensionality (number of variables) of the dataset but retain most of the original variability in the data. The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible.

Let's look at what happens when we reduce the 25 dim data of english alphabets to 2 dim.
Clearly we see that X and O fall in different clusters and hence can be easily diff.

Intuitively though it seems that X and O seem to be correct choices for a game, there is some math basis for their selection !

I Propose - Cmaps fulfills

Yesterday while I was reviewing a few concepts in statistics, I felt a need for a software which could help me draw a concept map for the entire subject. I felt this way becoz I realized that the entire branch of statistics can be distilled down to a few concepts , the knowledge of which should be more than enough to play with the statistical packages.

Somewhere, someone seems to have heard this demand .

Voila, Today I see a news item in Wired that Cmaps has developed a software exactly similar to my proposal. I am on my way to download and use it.

Bayes theorem revisited

Its been almost ages that I had last read anything about Bayes theorem , perhaps the last time I stumbled on the theorem was when one of my friends Tarun sent a email forward with the subject line" How to choose your spouse using Bayes theorem". Though much of the article was only meant for mathematician's world, It never occurred to me that there are a lot of commonalities between the statistical world and Bayes theorem.

Anyway , today I had to go through Bayes theorem for some work related issue:

Here's a basic intro:

Bayes' Theorem is a theorem of probability theory originally stated by the Reverend Thomas Bayes. It can be seen as a way of understanding how the probability that a theory is true is affected by a new piece of evidence. It has been used in a wide variety of contexts, ranging from marine biology to the development of "Bayesian" spam blockers for email systems. In the philosophy of science, it has been used to try to clarify the relationship between theory and evidence. Many insights in the philosophy of science involving confirmation, falsification, the relation between science and pseudosience, and other topics can be made more precise, and sometimes extended or corrected, by using Bayes' Theorem. These pages will introduce the theorem and its use in the philosophy of science.

Begin by having a look at the theorem, displayed below. Then we'll look at the notation and terminology involved.

In this formula, T stands for a theory or hypothesis that we are interested in testing, and E represents a new piece of evidence that seems to confirm or disconfirm the theory. For any proposition S, we will use P(S) to stand for our degree of belief, or "subjective probability," that S is true. In particular, P(T) represents our best estimate of the probability of the theory we are considering, prior to consideration of the new piece of evidence. It is known as the prior probability of T.

What we want to discover is the probability that T is true supposing that our new piece of evidence is true. This is a conditional probability, the probability that one proposition is true provided that another proposition is true.

My thoughts:
At the core of bayes theorem lies the fact of testing a hypothesis ,given a piece of new evidence. It will not change the original hypothesis by a large amount, but only slides it down or up depending on the evidence. Over a period of time, we see that initial bias of the assumption gets evened out.

Links :
Intro to Bayes theorem
Intuituve expanation of Bayes Theorem