RK's Musings

Stats & Probability - Need of the hour

Here is a TED talk by Dr.Arthur Benjamin ( Mathemagic ted talk person) who says, that stats & probability should be elevated to the status that is given to calculus

 

Meeting PCA again

I was familiar with this technique, had used it in a couple of places. This week I had a chance to apply this technique to a problem at hand. However, this time I did not want to rush in . Had in my archives an old book called "User's guide to Principal Components". Took some time out to know a bit more about the technique. As always, there is always some new learning by reading old stuff.

A bit of history, to begin with. This technique was first introduced by Karl Pearson in 1901 (Refer to his other contributions here). The general procedure to carry out the technique had to wait until 1933 when Harold Hotelling introduced it in his paper. Thus 1930s and 1940s had a great deal of activities relating to PCA.Then things subsided and the development took place as a poisson process :). With the advent of computers, things changed. People were now able to find factors, invert matrices with ease and thus the application of PCA became widespread in various fields. An unfortunate development was that many social scientists extended PCA to factor analysis and interpreted factors by their own whims and fancies.

The author of this book , Mr. Edward Jackson has put in his thoughts from his working experience at Eastman Kodak for over 20 years. Books from practitioners are gems for they show how to marry theory and application , and more so, make you feel that there is no single answer to any problem. Smartness lies in using a specific technique to a situation with adequate mix of intuition and quantification. Another book, Super Crunchers, makes the same point

Principal Component Analysis, abbreviated as PCA, is a data exploration technique as well as an inferential technique. The methodology is same for population as well as samples( a greatly comforting aspect). Also most of the aspects relating to PCA are distribution free.

Let's say your are a police officer in charge of keeping a tab on the militant threats in Mumbai. Lets put some numbers to the situation. Imagine you track 10 variables at different points in the city.. You figure out that these 10 variables should to be present in a specific bands so that you are fairly confident of no militant threat in Mumbai. You have put in place police infra, informer infra etc in the city to get 10 dimensional vector from all the potential threat points in mumbai, let's say 5000 points, ending up with a 5000*10 matrix of observations.All the variables can be quantified on a scale of 1-10.Lets say the first variable x1 can be quantified on a scale of 1-10 and the mean is 5 and the bands are at 4.5 and 5.5. Any observation beyond this band might make you suspicious. Lets say you keep a track of all these 10 variables on a big chart. So, basically you add daily observations to your dashboard .

If all the 10 variables are with in the bands(lets say these are 95% confidence interval bands), would you be 95% confident that there is no militant threat ?.

At the outset, it might seem correct. You are tracking 95% intervals for each of the variables. However , on thinking a little bit, you realize that the type I error shoots up. Reason, the joint probability of threat,thanks to Type I errors is (1-0.95)*(1-0.95)*....10 times = 0.4. This means that if you control all 10 variables , there is 40% chance that one of the variables is outside the bands!!! Now this is assuming that the variables are uncorrelated. But think abt it, variables are often correlated , especially in a militant threat kind of situation. So , your Type I error goes up. What do you do ? Well , there are a couple of things you can do. One is see to it that the bands are designed in such a way that Type I and Type II error is built in to bands mechanism OR, do something which is relatively easier,Do not track these variables x1,x2,x3,.....x10 for 5000 sites, but a different linear combinations of these variables call then X1,X2,X3,X4 which have one great property , which is, the transformed vectors are uncorrelated.

Basically this is my pathetic attempt :) at explaining PCA with out any matrix notion. PCA helps you view the 10 variables sets in a different perspective which gives a lot more clarity on the relational structure in the data. By looking at the bands and Hotelling T-square you can act on the variables realization.Ok, now to say the same thing in matrix notation. Whatever be your input matrix, Let S be covariance matrix then one can find an orthonormal matrix U such that U'SU =L , a lower triangular matrix. The diagonal elements are eigen values which can also be obtained by solving |S-m*I|=0 .Characteristic vectors are obtained by (S-m*I)Xp = 0 where Xp is the characteristic vector.Thus one can transform the original data to z where z=U'[x-xbar). ( With out latex support, writing math is particularly difficult on typepad ). One use of this transformation is that L matrix is very useful becoz trace of L is the sum of original variances and det|S| is just the product of eigen values.

Another important aspect that I have learnt from this book is about the stopping criterion. I was using proportion of variance explained as a stopping factor for the number of components selected. Why ? I don't know....I just accepted somebody's word. I was completely dumb in not even thinking about it.However this book revealed to me that there is nothing great about that rule. Bartlett's test provides a better way to scientifically create a stopping condition. This was new to me. Somehow , it is never mentioned in the usual 10,000 ft view read of PCA. There are at least a dozen stopping criterion and I guess at least crunching half a dozen would bring upon a healthy dose of skepticism in your views.

What I could summarize above is just 5% of the book. The title is definitely an understatement. It should be have been "all encompassing guide" becoz in whatever field you are, for whatever problem you want to use PCA, there is a guideline in the book. This is one of those good books which focus less on abstract theorems and more on practical applications.

Theorizing Vs Empirical analysis

With easier access to raw computing power, one might think that all it takes is to analyze data and let the data throw hypotheses.

Is theorizing over in this data intensive world ? Has the world become so complicated that theorizing is useless? May be not.

I read this short note(Tom Swift and His Electric Factor Analysis Machine) couple of years back but I always reread it frequently so that I dont make the mistake of living only in an empiricist world. Some amount of apriori assumptions/ hypotheses are required. In the financial world, traders provide this knowledge. It is easier sit at a desk and crunch data and build models. However it might be a complete useless exercise if you do not marry apriori assumptions with your empirical work.

Here is a Link to this beautiful article: Tom Swift and his electric factor analysis machine

Quote for the day

Parsimony is desirable but not always obtainable

- Edward Jackson (PCA Practitioner)

I guess it's true sometimes, about of our lives too !.

R : The Linux for Statisticians

Via NYTimes:

 

Robert Gentleman & Ross Ihaka (people behind R)

“R is a real demonstration of the power of collaboration, and I don’t think you could construct something like this any other way,” Mr. Ihaka said. “We could have chosen to be commercial, and we would have sold five copies of the software.”

“The great beauty of R is that you can modify it to do all sorts of things,” said Hal Varian, chief economist at Google. “And you have a lot of prepackaged stuff that’s already available, so you’re standing on the shoulders of giants.”

I guess once you try R, you will be hooked forever.

Data Vs Pattern

Often times most of the data / variables that we come across appear random and we start taking decisions based on gut. If gut works , its great. However before bringing in the gut part , it might be sensible to stretch a bit and look at data carefully and examine the details.Some time, just by changing the our view, we might see patterns. well PCA is all about that. I don't have to go that far to make this point. For example, if one looks at the current graph of sin(x) for some values of x, it looks like this:

The data points appear completely random. However just by tweaking the aspect ratio(x:y), the same data looks like this:

I tend to believe to that there are patterns everywhere and it all boils down to spending time with data and trying to understanding it carefully.However one also needs to be skeptical about all the patterns that one finds and have a null hypothesis that pattern is random, back test the pattern and then form an opinion. Instead of having a null hypothesis that data is random, it is better to have a null hypo that data pattern is random.

Quote for the day

In God we trust; all others bring data.

—Dr. W. Edwards Deming

On Google

Paul Graham on Google :

Their hypothesis seems to have been that, in the initial stages at least, all you need is good hackers: if you hire all the smartest people and put them to work on a problem where their success can be measured, you win. All the other stuff—which includes all the stuff that business schools think business consists of—you can figure out along the way.

> rnorm(15) Thoughts

A Random sample of my thoughts

1. If Nifty falls / Nifty rises , does the implied vol move in the same direction of the underlying or does it move the opposite direction. This becomes a very important question in the context of choosing a local volatility idea or chucking such an idea ? Local vol models assume that underlying and smiles move in different directions!!
2. As there is a short sale restriction on stocks, what exactly is a tradable underlying for Nifty option, for valuation purposes ? Yes you do hedge a long call option with whatever is a cheap, a future/ETF. But then is it sensible to treat a long call option as option on futures and put option as an option on index, the reason being that that they are the respective hedges for the options available
3. Does vol of nifty determine the atm imp vol level for nifty options? If so, how does it move ?
4. Is there a curvature for vol across strikes ? If so, does the curvature remain quasi-stable?
5. What is the functional/deterministic/stochastic relation between the comovement of nifty futures and nifty options in a 20,100 seconds(9:55AM - 3:30PM) trading day ?
6. What is the average trade volume per moneyness in a trading day ? This is easy...got to crunch a few numbers to get an idea
7. World-over the realized vol is explained by atm variance largely(80%)? Is it the case in Indian markets ? Are skew and convexity merely jazz in Indian markets which one need not worry too much ?
8. If you look at an option trade that happens at time t1, what would be the corresponding future price that you would consider for valuation? the instantaneous futures price, average of future prices in the interval (t1-delta, t1)? Can you choose delta with some rationale, meaning, some metric which justifies choosing last 5 min / last 10 min / whatever that one thinks is meaningful ?
9. Parameters for Vol of Vol for Nifty - Are they stable and more importantly what is their impact on Imp vols
10. Is imp vol in Indian markets a completely random animal ? Is Indian market completely a punter's market , meaning every one is a momentum player ?and analyzing numbers beyond a point is a useless activity ? ( I hope not )
11. Are circuit breakers, fancy as they might sound, properly designed? Shouldn't financial markets be there for the price discovery to happen? How can the third busiest exchange in the world trade for 30 seconds (Monday,after election verdict) and let the order flow go to other stock exchanges ?
12. Are Indian investors sensible ? Nobody wants to buy when nifty was 2600 level and now when it has reached 4300 , every one wants to buy? Shouldn't it be the other way round ?
13. Who makes money in the entire game ? Are the only people who make money are Custodians, Fund account management fees, brokerage houses, financial advisors? Is "extracting decent alpha consistently", a fantasy and there is survivorship bias amongst the successful fund managers that we see around ?
14. How do people trade reliance options or for that matter any of the American options on the Indian market? Are there good pricing and hedging tools for these options available with traders ?
15. Gap up and Gap down!!! , the biggest problem in the Indian markets -- How do you sleep well when you know that you have got to deal with such an issue the next day morning ?

................................May be I will find clues to the above stuff, someday!

Filehash

Handling large amounts of data in R is tricky as R typically loads the entire dataset in to RAM. While this means that computations are going to be very fast, it also means that dataset that can be used for analysis is dependent on your RAM. One solution which I stumbled on was Filehash which seemed to offer me a solution than the usual way that I was going about. I was getting the entire stuff in postgreSQL and then doing computations using R.

Filehash seemed to be great as I don't have to bring another RDBMS in to the work loop. However after spending 1-2 hours on it, I realized it was not that useful for me. It is great if you have key value pairs, but not really great if you have too many columns with dependencies and there is a lot of subsets that you need to work on. In any case, filehash was nothing but DB2 sitting at the back of R. Why should I put my data in to DB2, when a powerful postgreSQL is at my disposal. I have ditched Filehash. There is another package called ff , but sadly it is more useful if you want to sample stuff from a big file. In summary , it appears to me that the only way out is Hadoop-R/Mapreduce-R. That's a steep learning curve for me.

Why should one think of fancy systems like hadoob?
Fact :On a typical day, 6 lakh NIFTY futures are traded in 20,100 seconds of a trading day ,meaning, 30 trades per second.Now each of these 30 trades happen at various price points and quantities. If you want to track the average futures price(quantity weighted) every second for a SINGLE day , it is easy. If you want to do it for a MONTH, it might take some time. If you want to do it for an YEAR and also calculate a 5 minute moving average of futures price, taking in to consideration gapups and gap downs across days, we are talking abt lots of computations on lots of data points. Even in such a simple application using RDBMS systems is a PAIN. I am guilty of using them now, but it will be of no use as things scale.

Turning a Boring task in to Fun

As we all know, every kind of work involves a definite amount of tedious, unpleasant,low-mid IQ work.I don't subscribe to these labels though.

In the life of a data analyst, the data preparation and data cleansing is something which has to be done before he/she can even think of doing any sane analysis. So, how does one approach data prep and data cleansing? Is it something to be looked down upon ? Well, I was under that impression many years back. My view changed completely when I took the effort to understand the complexity of data prep and data cleansing phases. Also, I began to develop this feeling that working on these phases gives a tremendous amount of insight that no modeling on the cleansed data would give. Here is a fact which is interesting : Renaissance hedge fund employs around 25 top notch PhDs to work on data treatment such as missing and capping treatment, outlier effects etc. No wonder this amount of care at the initial stages translates to tons of insights in to the arb opportunities in the market. Many years working on databases, I have kind of developed great love towards it. One of the things which is a pre-requisite to successful modeling is data prep phase. Working on data, which to this day, many people have an aversion is a fascinating place to start the analysis. However I am amateur in terms of the possibilities that lie in DBMS world. I have to understand and implement lots of things in this space, at least in relation to fin world. KDB+, Hadoob(TP keeps reminding of this fascinating world that I have not explored at all). Well, I guess it is a life long process.

Recently I was asked to quickly develop a database of brokers and sub brokers associated with NSE, as a side-activity when I am not doing my regular work. This is just one of those requests you get when people know that you program . Well, at the first sight of the task, it does seem as though it is a boring task. Well, all it takes is to screen scrape NSE site , dump the content in to delimited text file, read it back in to whatever dbase you want to . Well, if the task is to hand over a xls, just import the delimited text file and be done with it. This task of writing a program to screen scrape NSE was easy. However preparing the data from badly formatted html files is a little painful. As mundane as this task sounds to be, it was a pleasant experience. HOW?

• Haven't screen scraped NSE before,But took this little task as an opportunity to do just that..As they say, all it takes to recognize a good programmer from a bad one is to look at 100 lines of code . Some companies conduct rounds of interview, Phone interview, this type of interview, that type of interview, HR round (what do you want to do in life type of questions, which the interviewer himself would have no clue what he himself would want to do in life, 5 years down the lane questions , how do you see yourself ,,kind of questions for a PROGRAMMING job!! give me break!. In 5 years the skill sets I have could become completely obsolete and may be I will teach math:).. Any ways the point is, IT companies should just ask the interview candidate to code for 15 minutes and examine the code.It gives far better insights in to the candidate's productivity than asking "what are your 3 strengths and 3 weaknesses". Till this date I have never understood how these questions make sense in any interview( For heaven's sake, I am not at a shrink's place where quizzing abt my psyche is imp!)
  
  NSE definitely needs to improve its IT coding standards for the site is not being programmed properly. This is not some casual statement made by hear say. Screen scraping the relevant files shocked me to the kind of coding standards that were being followed!. Investigation of this sort is always fun.

• Had used C++ earlier to screen scrape but it failed becoz the library that I was using was built for HTTP and not HTTPS. Well, I could rebuild it ..but gave it a thought and ditched it to favor C# library. It worked like a charm. C# is a language for people who just want to get some things done with no particular great concern about efficiency, performance etc. At least thats what I believe.There are tons of libraries that are build and you just have to use them. How much time does it take to get up to speed on C# screen scraping?.Not more than a couple of hours.

• Thanks to the poor formatting of the files, had to resort to manual search of relevant tokens. Made it fun by using regex which I had extensively used probably a year ago. But you know, programming is like going to gym. You go to gym regularly and you are fit. You stop it for a month, you become lazy and you stop it for an year,forget about running for even 15 min on a treadmill.Your body will not cooperate. Programming is exactly like that. If you want to just type in with out looking too much in to the reference section of the language, with out Ctrl C+ Ctrl V(which most of the people indulge in, they call it programming, i call it effective copying), you got to program daily. If you don't enjoy it, get out of programming and do something else. So Regex became something which I had stopped using a year ago. I mean I had conveniently avoided it , becoz..well, lets face it...doing it with the dumb Contains() or searching manually appears easier that regex. Using Regex needs you to think..Searching plain text for tokens appeared correct becoz I did not want to invest time in creating regex in the first place. A mundane activity like parsing a ill formatted html file gave me an opportunity to go back to gym( use regex which i had not used since a year) . Needless to say, it was like going on a treadmill after an year. However after spending 3-4 hours on regex, I was back to normalcy :).

• So, here I was with a broker and sub broker database and was ready to hand it over and be done with it. I had spent on this task already close to 20 hours. Then I just looked at the data and thought , what would I want to think about this data and answer some subway ride questions, meaning, if you look at data on a subway ride, and you have nothing else to do but to wait till the train takes you to the destination, what questions would you ask about the data ? This is type of mindset where you think furiously whatever random/rational/senseless questions that you have about the data and quicky answer them. Well, I had so many other things to do ..Should I spend any more time on this broker/sub broker dbase..Then I thought, why not ? This is probably the only thing that I will recollect from this brief task , apart from just the usual hacking recollections. So, I spent an additional hour looking at Data. Just an additional hour greatly helped me answer some questions which will help some back of the envelope calculations about ALL BROKERS that are associated with NSE. Now that's some useful info

• Should I develop a rails interface ? That was a stretch..Have other important things to work on and hence decided to leave this task at this stage and handover the dbase.
  
  Sometimes, I do not know whether thinking in detail about any task hinders productivity or enhances productivity. Had I have not thought in detail about this task, may be I would have done it in lesser time and probably would have not had fun in the process.But one thing I do believe is that thinking about details at a micro^micro level makes any boring job a fun activity.
  
  
  

English Mania

Jay Walker @ TED shows evidence of the english mania. This year, China is going to be the country with the largest number of english speaking people. Mass chanting of english sentences is something which I have never heard of, till date!
But in China...