RK's Musings

Elements of Hindustani Classical Music

This book is written by Shruti Jauhari, a noted Hindustani Classical Singer and a disciple of K.J.Yesudas. The book is an accessible introduction to the various elements of Hindustani Classical Music. To read a language, you need to know the alphabets and the grammar associated with it. In the same way, to listen to Hindustani music, you need to have a basic knowledge of Svar,Raag and Taal   One often comes across terms like bandish, gharana, alap, vilambit , taal, etc in the context of Hindustani Music. Unless one has some understanding of these terms, it is difficult to be a discerning listener.

There are very few books written in English that give a sufficient knowledge with out entering the intricacies of the subject. This book belongs to such a category, where a beginner gets a good 1000 ft. view of the various aspects of Hindustani Music.  It starts off with a basic history of Hindustani music in India to give a reader some idea about the way music and musical ideas developed . The book then goes in to explaining terms such as Sruti, Svar, Musical Scale, That , Raag, Taal, Laya in an intuitive manner showing the basic math behind it.

There is a chapter that gives an intro in to the basic singing process and tries to provide simple answers to questions like, what constitutes a rendition, what are the parts of a typical rendition, where does an artist get to innovate, what is the role of tabla player in a rendition etc. The book then goes in to a basic description of various gharanas. In the modern world where learning is from a guru whoever is accessible in the city or town that one lives, gharanas might not be relevant. But it is always nice to know the various gharanas and the artists associated with various gharanas. The book also lists some eminent Musicologists and their profiles.

Perhaps the most useful chapter to me in the book is a full fledged description of the prominent raagas and their attributes. Well listening to raagas is one thing but singing or playing is a completely different thing. Unless you are thoroughly aware of the structure of a raag, it is very difficult to play beyond the known composition. Your guru might teach you a few compositions in a specific raag. Playing the given composition is a matter of practice. But soon one realizes the real fun in playing lies in improvisation. I play about 7 raagas on Sitar and  I maintain a chart that gives the attributes of these raagas. Sometimes the way that I maintain the chart for a specific raaga is not very systematic. This chapter has taught me a nice way to structure the chart. It has a chart for each of the 25 most popular raagas in Hindustani Music. I doubt I will ever play so many raagas in my life time!, but I guess I will refer to the structured chart given in this book from time to time.

Born Standing Up : Summary

 

Getting good at any craft takes a long time, be it deal making or trading or mastering a game or programming. However books/articles,etc. out in the world are stufffed with "get rich quick", "get smart quick","get skills quick" kind of messages. I can't speak of other fields but programming books, that implicitly carry these messages, are always in majority, be it in the offline and online bookstores. ‘Learn XYZs language in 30 days', ‘Become an ace coder in X weeks' etc. For all such coders who believe such books and coding something for 30 days is going to transform you, make your venture the next instagram, this book has a lot to offer. Steve Martin gives a biographical account of his life where he claims his achievement is not a result of some sudden random event that got him to spotlight and has since then remained in spotlight. Far from it, its a story of sweat, tears, rejections and occasional acceptances, smiles and rewards . Steve starts off his book saying

I did Stand-Up comedy for eighteen years.Ten of those years were spent learning, four years were spent refining, and four were spent in wild success. I was seeking comic originality, and fame fell on me as a by-product. The course was more plodding than heroic. I did not strive valiantly against doubters but took incremental steps studded with a few intuitive leaps. I was not naturally talented. I didn't sing, dance, or act, though working around that minor detail made me inventive. I was not self-destructive, though I almost destroyed myself.

The book gives a background of Steve's father whose career ambition to be an actor made the family move from Texas to Hollywood to California. Amidst the various road trips, car radio introduced Steve to comedy. Later TV shows hooked him on to comedy. Meanwhile his father's acting gig was going nowhere and he reconciled to working in real estate, yielding to the pressures of raising a family. He was forever against Steve's interest in comedy as he thought it was flippant job. So, Steve's childhood , despite a doting mother was a troubled childhood. Somewhere in the book he remarks,

I have heard it said that a complicated childhood can lead to a life in the arts. I tell you this story of my father and me to let you know I am qualified to be a comedian.

Thanks to his father's decision to move to Inglewood, California, Steve finds himself a job at Disneyland, the fairy tale place for any kid. His job was selling guide books. His job used to get over by noon and he had a free admission in the park. During this free time he came in contact with magic and he was enthralled by it. At the age of 10 being a regular employee at Disneyland, he learnt magic tricks from a fellow named Eddie Adamek. He was completely fascinated by one, Wally Boag, a performing artist at Disneyland. Slowly he starts attending every show of Wally Boag and tries to absorb all aspects of his performance. This stint in Disneyland comes to an end, thanks to poor sales of guide books. Steve then works on an ad-hoc job for an year before knocking the doors of local magic shops and he gets a job at one such shops. Thus, Steve began his show business career at the age of fifteen in 1960. He stood behind a counter eight hours a day, shuffling decks, manipulating Wizard decks , Mental Photography cards, and performing other magic tricks. This practice for 8 to 12 hours each day made him better and better at magic. He also developed a keen interest towards comedy after seeing Wally Boag's performance. He also starts reading various magic books and devours for magic tricks that he can use at work.While performing his magic tricks, he realizes that audience loved it when some tricks didn't work. This is when he faces a choice in his life, either be a magician or do comedy. Given that advanced magic tricks required more money, stage lighting infra etc, he takes a decision of somehow mixing comedy and magic so that he stands out.

Steve remarks,

At age eighteen, I had absolutely no gifts. I could not sing or dance, and the only acting I did was really just shouting. Thankfully, perseverance is a great substitute for talent. Despite a lack of natural ability, I did have the one element necessary to all early creativity: naïveté that fabulous quality that keeps you from knowing just how unsuited you are for what you are about to do.

He also starts learning banjo. Magic by itself might be a common place. Comedy by itself might be a regular event. But mix magic , comedy and banjo, you get a combination that will make a person stand out in any performance. That's exactly what Steve aimed for. When he picked banjo he had no teacher, he literally learnt by slowing down banjo records. When asked in a Charlie Rose interview, about his thoughts on banjo, Steve remarked,

In high school, I couldn’t  play an instrument, I remember getting my first banjo, and reading the book saying saying his is how you play the C chord, and I put my fingers down to play the C chord and I couldn’t tell the difference. But I told myself, just stick with this, just keep playing, and one day you’ll have been playing for 40 years, and at this point, you'll know how to play.

He indeed got better as he kept playing and practicing.In 2009, Steve released his first album, 'The Crow' and it won a Grammy…Ok, Back to the story now.

Steve started working at ‘Bird Cage Theatre’ at the age of eighteen with no specific skill set, except an ambition to become good at comedy. He worked at Bird Cage Theatre four years mastering the craft.He worked steadily on his magic act, six minutes at a time, four times a day, five on Sunday, for three years. That's a ton of practice. Even though he relied less on magic in the later years, he says that the ability to judge audience reaction from all those magic shows was priceless. At Bird Cage Theatre, Steve starts using banjo in his act and tweaks his show sometimes to rely purely on comedy.While working at Bird Cage, he also starts doing part time at another theatre, ‘Prison of Socrates'. About these formative years of experience, Steve remarks

My act was eclectic, and it took ten more years for me to make sense of it. However, the opportunity to perform four and five times a day gave me confidence and poise. Even though my material had few distinguishing features, the repetition made me lose my amateur rattle.

Steve consciously starts restricting magic to one specific act so that he doesn't carry of risk of being called a magician. He wanted to be a comedian and not a magician. Really Amazing, the kind of clarity he had at such a young age. Well , he had a parallel life going on too. He got himself enrolled in a state college and started reading about Philosophy, ethics , logic. Not surprisingly, he found content for his comedy acts in these courses. Like they say, if you want to buy a car, you suddenly start noticing cars on the street. In the same way, Steve's liking for comedy made his courses on syllogism , philosophy etc, a wonderful source for his comedy acts. During these years where he was doing multiple things, he had two illuminating moments, one he decided he would write his own comedy and second, he met Nina Goldbatt who had a significant impact on his professional life. Courtesy Nina, he later got a break in the then popular show, Smother brothers show. It is not a fairy tale story after this break at Smother brothers show. Steve met with a lot of failures, experimented a lot as the material was getting obsolete very quickly. He continued to study, despite the hectic schedule and that kept his mind afresh. He could rely on the study time to energize him to come up with creative content for his acts. He improved on the comedy format where he cut off the reliance on punch lines to make the audience laugh. He improvised and developed his unique style where the act free flowing in nature.

Steve summarized his eight years on this job,

My goal was to make the audience laugh but leave them unable to describe what it was that had made them laugh. In other words, like the helpless state of giddiness experienced by close friends tuned in to each other's sense of humor, you had to be there. And for the next eight years, I did that

After this eight year stint, Steve left television writing and hit the road, literally. He went on a string of road shows designed by a local agency and here is where he honed his skills as a comedian. The anonymity made him brave enough to experiment in colleges and small towns. This was also a time when he was away completely from his parents and sister. This act of doing, experimenting, getting feedback, learning from it and improvising is the classic "deliberate practice act", that he went through, during his years on the road. He also emphasizes the role of solitude in those years and says that it was a crucial element for becoming good at his craft.

He remarks,

The travel isolated me. Friends were available only through costly phone calls, and contact with my parents and sister spiraled down to a pinpoint. In this netherworld, I was free to experiment. These out-of-the-way and varied places provided a tough comedy education. There were no mentors to tell me what to do; there were no guidebooks for doing stand-up. Everything was learned in practice, and the lonely road, with no critical eyes watching, was the place to dig up my boldest, or dumbest, ideas and put them on stage. Because I was generally unknown, in the smaller venues I was free to gamble with material, and there were a few evenings when crucial mutations affected my developing act. THE CONSISTENT WORK enhanced my act. I learned a lesson: It was easy to be great. Every entertainer has a night when everything is clicking. These nights are accidental and statistical: Like lucky cards in poker, you can count on them occurring over time. What was hard was to be good, consistently good, night after night, no matter what the abominable circumstances.

So, these years spent on creating, refining, experimenting the craft made the Steve Martin that he is, today. Starting from a job at ‘Coffee and Confusion' where he had to talk to empty chairs just to attract passerby, he went on to doing comedy act for as large as 45000 jam packed auditorium. I have not been able to touch upon a lot of other subtle aspects that are mentioned in the book, that one can apply to developing skills in any craft. I guess one needs to read this book to get all the flesh and bones of the story.

Takeaway :
The basic theme that one can infer from Steve Martin's story is Diligence. His diligence played a pivotal role in his success. Diligence not just in terms of persistence, but also in the ability to ignore unrelated pursuits.

The Razor’s Edge : Summary

 

It has been a long time since I have read any fiction book. So, thought of reading one, on this weekend to take a break from stats, programming and the usual routine. It took me about 9 hours to read the entire book and I must say that there was not a single instance during those 9 hours that I felt like taking a break. So, the book has a smooth flow of prose with just enough characters that you don't lose the story or get bored anywhere. The plot has about 15 characters , including one played by the author himself, Somerset Maugham but not all 15 characters get the same footage(obviously) .The protagonist of the story is Laurence Larry Darrell who goes on a spiritual quest and the book tries to weave a story around that quest. The other 13 characters come and go in with varying periodicity through out the book. Besides Larry, the main characters in the story are Isabel who loves Larry but ends up marrying Gray Maturin, Elliott Templeton, Isabel's uncle whose sole purpose in life is to socialize and be at parties, Sophie MacDonald, a poetess turned whore turned dope addict, whom Larry almost ends up getting married, but thanks to a Isabel's devious plan, never does so.

The story starts off with Larry being engaged to Isabel. After a stint in the army, Larry is a changed man. He wants to understand about God, Evil, role of knowledge in salvation and whole lot of things that any spiritually inclined person would seek. The difference though is that many of us, answer or seek answers to these questions, while keeping a day job, holding the responsibilities of a spouse, a parent etc. Not Larry. He has no intention to work and he tries to argue with Isabel, that in his meager income they can live a decent life. Despite Isabel deeply in love with Larry, she rejects him. I guess for some people, economic considerations weigh far more than matters of the heart, when it comes to settling down with somebody. Isabel gets married to a business magnet,Gray Maturin. Larry then goes on a quest that takes him to Germany, Spain and India. He finally gets some clarity after staying in India at an Ashram for a few years and meeting some spiritually inclined people in India.

The book does have a dose of aspects from Hinduism like karma, renunciation, rebirth, etc. But the author makes it clear that he never intends of summarizing or even talking at length about such aspects in his book. He is clear about his role, i.e a storyteller. I must say he has played his role as a character and as an author with perfection , as he make spiritual elements, the various locations , a dozen characters as props to show the conversion of an atheist Larry to a devout spiritual person by the end of the book. No wonder this book is rated as one of the finest works of Somerset Maugham. However the movie adaption met with a commercial failure.

Some quotes / conversation that I found interesting amongst the various characters in the novel are :

• I have been reading Spinoza the last month or two. It fills me with exultation. It's like landing from your plane on a great plateau in the mountains. Solitude, and an air so pure that it goes to your head like wine and you feel like million dollars
  
• What are you do going to do with all this unnecessary knowledge which you call wisdom ? It I ever acquire wisdom I suppose I shall be wise enough to know what to do with it
  
• I wish I could make you see how much fuller the life I offer you is than anything you have a conception of. I wish I could make you see how exciting the life of spirit is and how rich in experience. It's illimitable. It's such a happy life. There's only one thing like it, when you are up in a plane by yourself, high, high and only infinity surrounds you.You are intoxicated by the boundless space. You feel such a sense of exhilaration that you wouldn't exchange it for all the power and glory in the world.
  
• Unfortunately sometimes one can't do what one thinks is right without making someone else unhappy
  
• What's the good of knowledge if you are not going to do anything with it ? Perhaps it will be sufficient satisfaction merely to know, as it's a sufficient satisfaction to an artist to product a work of art. And perhaps its only a step towards something further
  
• Love isn't a good sailor and it languishes on a sea voyage. You will be surprised when you have the Atlantic between you and Larry to find how slight the pang is that before you sailed seemed intolerable
• Most people when they are in love invent every kind of reason to persuade themselves that its only sensible to do what they want. I suppose that's why there are so many disastrous marriages
  
• I had not the heart to laugh at Elliott any more, he seemed to me a profoundly pathetic object. Society was what he lived for, a party was the breath of his nostrils, not to be asked to one was an affront, to be alone was a mortification; and, an old man now, he was desperately afraid. It made me sad to think how silly, useless and trivial his life had been. It mattered very little now that he had gone to so many parties and had hobnobbed with all those princes, dukes, and counts. They have forgotten him already.
  
• Art is triumphant when it can use convention as an instrument of its own purpose
  
• Advaita doesn't ask you to take anything on trust; it asks only that you should have a passionate craving to know Reality. It states that you can experience God as surely as you can experience joy or pain.
  
• In later ages the sages of India in recognition of human infirmity admitted that salvation may be won by the way of love and the way of works, but they never denied that the noblest way, though the hardest, is the way of knowledge, for its instrument is the most precious faculty of man, his reason
  
• We are all greater than we know and the wisdom is the means to freedom. For Salvation, it is not essential to retire from the world , but only renounce the self. Work done with no selfish interest purifies the mind and that duties are opportunities afforded to man to sink his separate self and become one with the universal self.

Steal like an Artist

Austin Kleon’s book is essentially a blog post turned in to a book .The basic theme of the book is that in art as in other forms, you have to steal. The word steal has a negative connotation and hence the book differentiates between two kinds of thefts:

Basically theft in this sense means that you are free from the burden of trying to be completely original. Most of the book is common sense stuff but how often do we really follow commonsense ? 

There are some nice visuals in the book. Some of them that I found interesting are :

Very often, ONLY when you have multiple ideas,
that you are capable of coming up with a perfect idea that works

 

It is better to separate your workspace in to analog and digital mode

Paul Graham in one of his essays says that he follows a similar configuration where he uses two computers, one connected to internet and the other offline, so that while working on something that requires concentration, the urge to browse randomly, check email, tweets, blogs is eliminated.

 

As Steve Jobs says, you cannot connect dots going forward.
However, dots you create in fact make you what you are.

 

All you need is a little space and a little time—a place to work, and some time to do it;
a little self-imposed solitude and temporary captivity.

“Be regular and orderly in your life, so that you may be violent and original in your work.”

— Gustave Flaubert

Matplotlib for Python Developers : Summary

For any number crunching work, be it making a report or developing a model or doing data diagnostics, visualization of the data is imperative. Be it univariate or multivariate data, visuals help us to look beyond the summary statistics or test statistics. For someone working in finance, there is an entire discipline of `Technical Trading' where buy , sell , stop-loss decisions are made based on visuals. Whether one believes it or not is a different question altogether. Keeping Technical analysis side, there is an obvious need to look at data, be it histograms, density plots, contour plots, barplots, boxplots, etc. Tools that churn out these graphics are compulsory in any data analyst's toolbox. My toolbox contains ggplot2, lattice and base-R. I had started using ggplot2 package, 4 years ago, and since then I have been using it regularly in my work. Since the output is usually publication ready, one of the real life situations where I had used ggplot2 visuals was in an annual newsletter to investors that reported their portfolio performance. I don’t think anybody cared about what the visuals were saying as long as the portfolio was making money. But using ggplot2 definitely lent a professional look to the newsletter.

Well the quality of output was certainly one of the motivations behind learning and using ggplot2. However there was a bigger reason that I got attracted towards ggplot2. ggplot2 is not merely something that is helpful in drawing some visuals. It is much more than that. It has an underlying grammar based on Wilkinson's Grammar of Graphics. Once you take an effort to understand the layered model, you look at a graph from a completely different viewpoint. When I encountered ggplot2, I was totally thrilled because it taught me how to map data to a visual. I can safely say that the grammar model gave me new eyes to look at a visual. Before using ggplot2, I really had a very cursory understanding of statistical graphics. Anyways, I will try to write a separate post on ggplot2 at a later date.

This post is about Matplotlib. I have started liking Python for data analysis. However I have realized that my toolkit was empty for doing visualization in Python. Any book that I start reading on data munging , data cleaning using Python modules, I could not move past a few chapters as there was always a reference to Matplotlib. There was no other option but to slog through Matplotlib and understand the details. I am coming to Matplotlib after working with ggplot2. So learning Matplotlib was an interesting experience where I constantly asked questions like, “How does ggplot2 do the samething ?” “Is the graphing capability for this particular problem better supported in Matplotlib or ggplot2” etc. This kind of constant comparison with ggplot2 and base-R packages helped me understand Matplotlib better.

In this post, I will try to summarize only 50 percent of the book and there's a reason for it. The first 4 chapter and the last chapter are most relevant for an analyst doing interactive data analysis. There are four other chapters dedicated to embedding Matplotlib in GUI libraries like GTK+ , Qt 4, wxWidgets and the web interfaces. These are more specialized tasks that I don't think I will be doing in the near future. For a number cruncher like me, the ability to quickly map data to visuals in an interactive manner is the key to doing a good job. Keeping that requirement in mind, I have read only five chapters from this book and have ignored four chapters that deal with embedding Matplotlib. So, in a sense, this summary is a partial summary of the book. Let me try to summarize these five chapters:

Introduction to Matplotlib

Firstly, what's Matplotlib ? It is a python package for 2D plotting that generates production-quality graphs. It supports interactive and non-interactive plotting, and can save images in several outputs. It can use multiple window toolkits and provides a wide variety of plot types. But the biggest USP in my opinion is the ease of use. Who's the lead developer for Matplotlib? It is written and maintained primarily by John Hunter.

 

John Hunter started off Matplotlib as a result of patch rejection from iPython. It was modeled on MATLAB because graphing was something that MATLAB did very well. The high degree of compatibility between them made many people move from MATLAB to Matplotlib. The current version that is available for download is Matplotlib1.1.0. To get an idea in to the massive effort that has gone in to this library, have a look at some of the statistics of Matplotlib 0.8480 that were made in presentation by John Hunter.

 

There are

• 80 python modules
• 21 files of extension
• 110,000 lines of code
• 189 examples
• 2-5 active developers at any time
• 25 contributors
• 5000 downloads/month
• 500 mailing list subscribers
• 18 backends

These numbers are as of 2006. You might have to extrapolate this data 6 years to get an idea of the massive open source visualization package is out there NOW. No wonder it is used in almost all scientific disciplines.

The book starts off by mentioning the key advantages of Matplotlib

• It uses Python.
• Its open source
• Its a real programming language
• It is very customizable and extensible
• It has LaTeX support
• Its cross platform and portable

Matplotlib gives output in both forms, raster images and vector images. There are two types of backends, Hardcopy backends (output is raster /vector images) and user interface backends. A backend that displays the image on screen is called a user interface backend. Matplotlib introduces two layers , renderer, that does the drawing and canvas , the destination of the drawing. The standard renderer is the Anti-Grain Geometry (AGG) library, a high performance rendering engine which is able to create images of publication level quality, with anti-aliasing, and subpixel accuracy. AGG is responsible for the beautiful appearance of Matplotlib graphs. The canvas is provided with the GUI libraries, and any of them can use the AGG rendering, along with the support for other rendering engines. So to get going, one needs to install NumPy, the Python bindings for the UI you are going to work with, python-dateutil module, pytz - the timezone module. One doesn't have to worry about all these dependencies if iPython is being used. iPython already has a Matplotlib mode in it. For most of the research purposes, I guess iPython pylab mode is enough to get going. The chapter ends with brief instructions on installing Matplotlib on various platforms.

Getting Started with Matplotlib

This section takes the reader through simple line plots. Through “This-is-the-code-and-this-is-the-output” kind of instruction, the chapter takes the reader through a series of steps , by the end of which, he/she is equipped to draw line plots in python, change the axis labels, reset the axis ranges, add title and legend to the graph and save the plots. The advantage of working with iPython is that whatever changes you make to the figure, they appear instantly on the figure ( sometimes by invoking draw() function). This is a massive advantage because most GUI libraries need to control the main loop of execution of Python, thus preventing any further interaction, i.e you can't type while viewing the image. The only GUI that plays nice with Python's shell is Tkinter. iPython in pylab mode starts two threads, one to execute the GUI library code and another thread to handle user command input and hence is so much more useful for a newbie. In fact I found iPython to be tremendously useful while learning NumPy and other packages as it gives instantaneous feedback on what you type. The chapter ends with some explanation to change the default Matplotlib settings. Now this is similar to par() settings in R. Till date, except for a few things I have never really changed them in R. Depending on the graph, I tend to change them in the graphing function itself. From that experience in R and extrapolating it to Matplotlib, I don't think I will ever change the default settings and meddle with the configuration files.

Decorate Graphs with Plot Styles and Types

For any plot, there is an obvious need to customize stuff. In Matplotlib , the syntax for customization is very simple. If you have to change the color, linestyle and marker style, you can basically change it using just one addition string input to the plot function. Handling xaxis ticks and yaxis ticks also has an intuitive syntax.

There is an interesting pic in the chapter that summarizes the various readily available plots available in Matplotlib

The chapter then goes on to explain the code fragments for generating histograms, pie diagrams,error bar charts, bar charts,scatter plots, polar charts. There is also section on ways to annotate the visual. Overall a very hands on chapter that gives one good enough functions to plot data.

Advanced Matplotlib

This is the really interesting chapter of the book as it explains the OOPS behind Matplotlib. The first three chapters seem almost magical with all plots coming out of Matplotlib blackbox by typing in a few commands. This chapter goes in to the blackbox, explains the OOPS structure in detail. This sort of introduction is invaluable for a newbie Matplotlibber as it equips one with a decent understanding of the various classes and subclasses. This understanding can then be used to go over the documentation of Matplotlib which, by the way, is close to 1100 pages long. Books like these are like the life jackets in the ocean of functions and objects in massive libraries like Matplotlib. I started off with Matplotlib document and it was overwhelming to say the least. This book gave me confidence to go over Matplotlib documentation , not the entire 1100 pages but the most prominent sections of it.

Matplotlib can be used in three ways

• pyplot : pyplot provides a MATLAB-style, procedural, state-machine interface to the underlying object-oriented library in Matplotlib.Matplotlib.pyplot is stateful because the underlying engine keeps track of the current figure and plotting area information, and plotting functions change that information. No need to use any object references while plotting
• pylab - a module that merges NumPy and Matplotlib in to one common namespace so that it gives an environment closer to MATLAB
• OOPS way - This is the most powerful way to write Matplotlib code because it allows for complete control of the result however it is also the most complex. This is the Pythonic way to use Matplotlib, and it's highly encouraged when programming with Matplotlib rather than working interactively.

The chapter then shows a sample graphic using the first  two ways, pyplot and pylab. It subsequently introduces the basic objects in Matplotlib that gives one the control on the plot, i.e , Figure Canvas, Figure and Axes. The flow of objects is as follows : You start off with a figure object, add subplots to it and work with Axes objects. Once you have figure object and axes objects , you can invoke ton of functions and customize the look and feel . You have much more control over the graphic like setting the scale of the axis, sharing axes etc. The chapter has an important section on dates. This is very useful for plotting financial data. By using locators and formatters, the section shows various ways to show a time series graphic with legible date formats on the x axis. Another interesting things about Matplotlib is the use of LaTeX annotation. Matplotlib includes an internal engine to render expression, mathtext. The mathtext module provides style mathematical expressions using FreeType 2 and the default font from , Computer Modern. As Matplotlib ships with everything it needs to make mathtext work, there is no requirement to install a system.

The chapter ends with contour plots. Ideally this should have appeared in chapter 3 along with other plots. Clubbing it with OOPS concepts makes it a little out of place. In any case, the contour plots are highly useful. As an example, lets say you do a singular value decomposition of a matrix. Linear algebra tells us the resulting matrices form an orthogonal basis. Now you can use contour plots to check that. Lets say I simulate a 20 by 20 random matrix, do an svd and get the u and v matrices. If I take a dot product of u and u transpose , get a 2d matrix, plot the 2d matrix as a contour plot, you get the following visual

All the 1 are on the diagonal and the rest of matrices are populated with 0 values. This tells us that u forms an orthonormal basis.This kind of visualization of matrices is just one of the many things that one can do using Matplotlib

Matplotlib in Real World

This is an interesting chapter of the book that gives a half-a-dozen real life case studies where Matplotlib is used as an effective visualization tool.

After going through this book on Matplotlib I think I have to work with this library and see where it stands with respect to ggplot2. For univariate data, Matplotlib looks like it has the same power as ggplot2 but what about multivariate data. For example, I have used a sample dataset(diamonds) to plot a histogram of one of the data variables using ggplot2 and Matplotlib

Here are the result of Matplotlib and ggplot2(enables to slice based on a different dimension)

Can one replicate the ggplot2 graphic in Matplotlib ? No idea. Need to figure out whether Matplotlib supports faceting and aesthetic grammar layers like ggplot2. But I guess even if Matplotlib does not support grammer of graphics , I think it is worthwhile to learn it as it becomes an invaluable tool , in doing interactive data analysis in python environment.

 

 Takeaway

This book is an accessible introduction to Matplotlib that  equips a reader enough familiarity so that he/she can go over the mammoth 1100 page documentation of Matplotlib.

NumPy 1.5 – Summary

Programming in Matlab / R exposes one to vectorized way of thinking. One doesn't usually write loops often and one tends to think in terms of vectors,array, matrices etc. R for instance is designed to facilitate vectorized input and output. Almost all the functions in base R support vectorization. Most of the functions in the packages on CRAN are equipped to take vectorized input. In fact the program design itself makes vectorizing easy. Recycling rule in R for instance , sometimes makes your function automatically handle vectorized input, even though you never meant to handle such an input. For a R newbie , the fact that his/her function does much more than expected is a happy side effect. However once you have logged in decent number of hours in R, you realize that it is your responsibility that whatever code you write, it handles vectorized input . Hence these days, one of the first unit tests that I write is to check whether my code breaks down for vectorized input.

If you stick to only Python Standard Library, you will not get to see the power of thinking in vectors. At least that's my impression. If you have programmed in R like me and then exploring Python, you will in fact be eagerly , thirstily, looking for stuff that facilitate vectorization. Thanks to Travis Oliphant, we now have Numerical Python(NumPy) library that gives a lot of R equivalent functions in Python.

Historically , Numarray and Numeric were Python libraries for matrix computations. Numeric was first released in 1995 . In 2001, a number of people inspired by Numeric created SciPy - an open source Python scientific computing library that provides functionality similar to Matlab, Maple and Mathematica. Around this time, people were growing increasingly unhappy with Numeric. Numarray was created as an alternative for Numeric that was better in some areas. Soon, there were a lot of developments around Numarray and SciPy that depended on Numeric could not take advantage of these developments.

In 2005, Travis Oliphant, a professor and an early contributor to SciPy decided to something about the situation. He tried to integrate some of the Numarray features in to Numeric. A complete rewrite took place that culminated in the release of NumPy 1.0 in 2006. Originally NumPy was a part of SciPy but today it exists as an independent library. This book is meant to teach some basic skills to work with NumPy and it does so with a good balance of theory and practical examples.

I feel NumPy does not have a steep learning curve if you are already exposed to Matlab or R. The reason is obvious. Once you are familiar with vectorization, you can easily spot the functions and understand them. I have thoroughly enjoyed reading this book as it has equipped me with skills to translate R code in to Python and at the same time get all the powerful features of NumPy. As such NumPy is a package that is usually a part of the basic tool kit for any researcher using Python. The fact that it has been around for quite sometime means that it has matured as a library. Its first release was NumPy 1.0 in 2006. As of today, you can get NumPy 1.6, i.e you have the advantage of using a very stable library that has gone through a life of almost 6 long years. NumPy is used by scientists, statisticians, researchers , academicians, quants who typically need to deal with huge data sets and are looking for quicker computations. NumPy’s USP is that it is the closest that you can get to run your code using C with out actually writing C code.

The author lists some key points of NumPy that make it appealing in the context of Big Data  :

• Much cleaner than straight Python code
• Fewer loops required because operations work on arrays
• Underlying algos have stood the test of time
• NumPy's arrays are stored more efficiently than an equivalent data structure in base Python
• Array I/O is significantly faster
• Performance improvement scales with the number of elements. Really pays off to use NumPy for large datasets.
• Large portions of NumPy are written in C. That makes NumPy faster than pure Python.

The book is organized in to 10 chapters in such a manner that it takes a first time NumPy reader systematically over all the major features of NumPy.

Chapter 1 : NumPy Quick Start

The first chapter starts off with a set of screen shots relevant to installing NumPy on Window , Mac and other platforms. The highlight of the chapter is a simple example of summing two vectors and comparing the speed of NumPy code with that of plain vanilla Python code. I ran a single run of “generate two separate million dimensional vector  , do some operations, add them” and found that NumPy was 11 times faster than Python code and was 1.5 times faster than using list comprehensions. Instead of relying on this single run, I ran a similar code 1000 times to get some summary statistics of the speed. The code compared the speed of NumPy and List comprehensions with that of plain vanilla Python loop. I found that the speed of NumPy with Python was on an average 14 times faster than plain vanilla Python code and speed of List comprehensions was on an average 2 times faster than plain vanilla Python code. So all in all, NumPy is a clear winner over both simple python loops and list comprehensions.

This chapter also introduced me to iPython. There was some learning curve for me. But once I figured out working with the magic commands, I realized that iPython was fantastic for interactive analysis. There was some inertia in going over iPython but soon I realized that magic commands were indeed magical. I found a few videos on iPython  that were helpful  in understanding iPython. After a brief encounter with iPython, I stumbled on to a chapter in a book titled,`Python for Unix and Linux System Administration` that goes at length in describing various iPython commands. Now when I look at R programming, I realize that R has a gui but probably needs something similar to iPython. May be it will be just a matter of time that some open source enthusiast will develop iPython equivalent in R. Some of the features of iPython that I find very useful are

• Searching history and executing them
• Repeating specific lines from history and executing them again
• Tabbing feature
• psearch
• bookmarking paths
• Ability to call external scripts.

Obviously there are many more magic commands that are available in iPython that I haven't explored. I think it is hard to leave iPython once you start using it.

Chapter 2 : Beginning with NumPy fundamentals

At the heart of NumPy is the the object ndarray, that comprises two parts, actual data and meta data. ndarray contains homogenous data described by the metadata object, dtype. The first things that one needs to learn is to create an ndarray for various numerical types. Firstly the data in the ndarray can be obtained from an existing Python object like list or using the arange function or from an existing array using I/O or using linspace. The dtype object is used to manipulate the default numerical type of the ndarray data. This chapter starts off with a list of constructors for populating data and meta data. It then talks about ways to slice and dice the array. All R equivalent functions of cbind and rbind are mentioned. There is an important point that is not highlighted well enough in this chapter, i.e the slicing and dicing the ndarray does not give you a new ndarray but gives a pointer to the specific memory location. This means that if you extract a subset of data and update it, the original ndarray gets updated too. To avoid this inconvenience , a slicing function can be put through a copy function that creates a new ndarray from the existing ndarray. Very much like the unlist function in R, there are two functions , i.e flatten and ravel that can applied to an ndarray to retrieve the data elements in the form of a 1 D array. The difference between the two is that flatten allocates new memory whereas ravel doesn't. So, based on whatever your requirement is, you might have to use flatten or ravel. By the end of this chapter, a reader gets a good idea of creating an ndarray, stacking/resizing/ reshaping/splitting ndarrays.

Chapter 3 : Get in to Terms with Commonly Used Functions

The first thing that any data analyst needs to do is to get the data in to the working environment. Much like the read.csv and read.delim functions in R ,you have loadtxt and genfromtxt function in Python. The syntax is more or less similar. genfromtext has arguments where you can define your own converters while extracting data. My guess is genfromtxt is far more useful than loadtxt as real world data is always messy and needs to be treated in someway or the other before getting it to a analytic environment. A sample list of ndarray functions such as sum , cumsum, cumprod, mean, var, std, argmax, argmin are mentioned .The author illustrates these functions by using some finance related examples like calculating stock returns, plotting SMA, Bollinger bands, Trend lines etc. By the end of this chapter, a reader gets a good idea of the doing I/O operations in NumPy and using various inbuilt functions for a ndarray object.

Chapter 4 : Convenience Functions for your convenience

This chapter gives a taste of basic statistical functions in NumPy such as covariance, correlation, polynomial fitting and smoothing functions. The best thing that I came across in this chapter in the vectorize function. Write whatever function you want to write, pass it to vectorize function in NumPy and your function is all set to take vectorized input. Coming from an R environment, where it is the coder's responsibility to ensure vectorized input handling, this function is like a boon. You write a normal function and use numpy.vectorize and your function is ready. Wow! It’s so neat.

Chapter 5 : Working with Matrices and ufuncs

This chapter covers matrices, a subclass of ndarray object. All basic operations of matrices are covered like transpose, inverse, solving linear equations etc. The highlight of this chapter is ufuncs, an abbreviation for universal functions. Universal functions are not functions but objects representing functions such as add. Now here is where things get interesting. You have methods for these functions namely reduce, accumulate, reduceat and outer. accumulate is the NumPy equivalent of rowSums, ColSums in R. All these four methods of functions are very useful in data munging.

Chapter 6 : Move Further with NumPy modules

This is a chapter in book that made me feel that I was reading some code in R. Except for a few cosmetic changes in function signature , all the object and classes signatures of NumPy’s linalg module and random module look similar to R code. This chapter starts off with describing linalg package functionality.All decompositions are supported in linalg package, i.e cholesky, svd, pseudo inverse, eigen value decomposition etc. Subsequently the chapter talks about various statistical distributions and random number generators relating to those distributions. Overall a very easy read for a reader exposed to R.

Chapter 7 talks about special functions like Sorting, searching, financial utilities etc. For testing ndarrays, NumPy has a testing package. Chapter 8 describes the main functions in the testing package that can be used to test equivalence of two ndarrays/ matrices. I have skipped chapter 9 that gives a 10000 ft view of matplotlib, the graphics package in Python. I thought I would go over carefully and learn it properly rather than getting some cookbook kind of learning. The last chapter of this book talks about SciPy where there are many more modules for a researcher looking beyond NumPy. The last chapter has made me curious about Scikits.statsmodels. I am certain that the models covered in Scikits.statsmodels would be far less than the ocean of models that are available in R. But I am curious to know what sort of models are available in Scikits.statsmodels. Will go over it someday.

 

 Takeaway :

This book gives a good working knowledge of Numerical Python library. By explaining the nuts and bolts of the major functions of the 400 odd functions in the NumPy package, it gives the reader good enough ammunition to crunch data.

A Byte of Python

This book gives a quick intro to Python and one does not need any prior programming knowledge to go over the contents. I quick read the book in an hour’s time and it served as a syntax F5er.

• Some of the GUI libraries mentioned that I need to explore are PyQT, PyGTK, wxPython, TkInter
• Need to check out Charming Python link mentioned in the book
• There is no char datatype in Python. Never realized this actually.
• Statements which go together must have the same indentation.
• Singleton tuple can be created by specifying it this way  (a,). Why does one require them though ?
• os.system(command) will execute the command. Good for creating your own backup routines
• All class members are public and all the methods are virtual in Python.
• Python does not automatically call the constructor of the  base class, you have to explicitly call it.
• A variable starting with double underscore is effectively a private variable
• Python provides a standard module called pickle using which you can store any Python object in a file and then get it back later intact. One can dump and retrieve objects in to a file that has an extension .data using pickle and cpickle module. This is similar to load and save command in R
• You can pass multiple arguments to a function use tuples or lists.

Python : Summary

I like books that explain things visually and this book falls in that category. I am reading this book after reading and understanding the basics of Python from “Think Python” and “Learn Python the Hard Way”. This book serves a nice visual recap of Python 101.  I have listed down some of the points in various chapters mainly to ruminate over the learning's from the previous two books.

Chapter 1- Getting Started

• Python environment variables

  • PYTHONPATH : Has a role similar to PATH. This variable tells the Python interpreter where to locate the module files you import into a program. PYTHONPATH should include the Python source library directory and the directories containing your Python source code.
  • PYTHONSTARTUP : Contains the path of an initialization file containing Python source code that is executed every time you start the interpreter
  • PYTHONCASEOK : Used in Windows to instruct Python to find the first case-insensitive match in an import statement. Set this variable to any value to activate it.
  • PYTHONHOME : An alternative module search path. It's usually embedded in the PYTHONSTARTUP or PYTHONPATH directories to make switching module libraries easy
  • IDLE (Integrated DeveLopment Environment) is similar to RGUI. You can use it to type in programs, code modules, debug them
• use import sys , sys.argv to access the arguments passed to Python script
• Like R, you can programs in interactive mode or script mode

Chapter 2 - Expressions and Statements

• The dir() function prints a list of currently defined names
• del deletes only a variable, not the object to which it points, because another variable may be pointing to the same object. Python keeps track of objects internally and deletes them automatically when they're no longer in use.
• Every object has a unique identity, which is the address of the object in memory, expressed as an integer or long integer. An object's identity doesn't change; it is constant during the object's lifetime. Use the id() function to determine an object's identity.
• In assignment of one variable to another, there is a crucial point that needs to be noted. If x is a string and you assign x to another variable y. Now y is also pointing to the same string. Now if you make changes to y, the value that x is pointing to remains and a new value is created and the name y gets binded to this new value. So, there is no problem when you assign one variable to another and then go and make changes to variables. Instead let's say x is a list variable and you assign it to y. You make changes to y, it gets reflected in x. Why? List is mutable and hence both x and y point to the same data.
• From highest to lowest priority, the operators are not, and, and or. The expression x and not y or z, for example, is equivalent to (x and (not y)) or z.
• Python doesn't have an exclusive-OR (XOR) Boolean operator or a built-in Boolean type
• An is comparison matters only for mutable objects such as lists and dictionaries
• You can chain comparisons to create expressions such as this: x<=y<z .Only adjacent operands are compared

Chapter 3 - Working With Numbers

• Python has four numeric types , Integer, Long, Float and Complex Number. Python rounds floats to 17 significant digits.
• print sys.maxint gives the maxint allowed on your machine
• You can use the coerce() function to see how Python promotes numbers
• int(x) to convert x to a plain integer , long(x) to convert x to a long integer, float(x) to convert x to a floating-point number,str() function to convert a number to a string
• Useful functions from random module - seed, randrange, choice, shuffle, random, uniform

Chapter 4 - Working With Strings

• Python implicitly constrains indexes that are too small or too large to sensible values. Reversed indexes return empty strings
• s.isalpha() to test whether all the characters in s are letters. s.isdigit() to test whether all the characters in s are digits.
• string.maketrans(from, to) creates a transition table and then one can use s.translate(tablename) to get all the translation done
• int(s) , long(s),float(s),complex(s),list(s),tuple(s) are all useful for converting string in to different datatypes
• repr(expr) converts object to string
• the string returned by repr() can be passed to eval() to (re)create an object with the same value
• There are a ton of functions related to strings. Probably this is the reason why Python is used in a lot of bioinformatics research

Chapter 5 - Working With Lists and Tuples

• Why both Lists and Tuples ? Tuples are faster to access and consume less memory than lists. Python also requires tuples in certain situations, such as string formatting, dictionary keys, and multiple function arguments. Tuples also are handy when you want to be sure that your data can't be modified by an alias
• To convert list L to tuple T , use (L,)
• The reverse of tuple packing is sequence unpacking: x, y, z = a
• s[i][j] used to index tuple or list
• If a is a mutable object such as a list, however, a = b doesn't create a copy of a but instead creates an alias. a and b will refer to the same object, so changes in one are reflected in the other
• [0] * n is an efficient way to initialize a list with n zeroes
• Repetition creates shallow copies.
• in, count, index can be used to test membership
• s[:0] = t to add at the beginning of list, s[len(s):] = t to add at the end of list, s[i:j] = [] to remove elements from the list, s[i:i] = t to insert elements
• reverse() modifies the original list. Reverse a copy if you want to maintain the original.sort() modifies the original list. Sort a copy if you want to maintain the original.
• The bisect module contains functions for manipulating sorted lists. The insort() function inserts an item into a list, and maintains the sort order
• The value returned by bisect() can be used to partition a list
• cmp(x, y) returns -1 if x < y, 0 if x == y, and 1 if x > y.

Chapter 6 - Working With Dictionaries

• d.popitem() returns a random item from dictionary
• You can embed a dictionary in a dictionary by using the key as id(dict_object)
• d.keys() , d.values(), d.items() to retrieve stuff from a dictionary
• clear() method to delete all key-value pairs from a dictionary
• d1.update(d2) to update d1 with all the key-value pairs of d2. If d1 and d2 contain matching keys, the d2 items prevail.

Chapter 7 - Control Flow Statements

• In an expression surrounded by (), [], or , press Enter, and continue the expression on the next line
• You can use the -t and -tt command-line options to detect inconsistent use of tabs and spaces in your programs
• Equivalent to seq function in R – range([start,] stop [,step]) to return a list of integers from start to, but not including, stop.
• One common use of range() is to assign consecutive values to a group of variables to create an enumeration

Chapter 8 - Functions

• Difference between parameters and arguments
• In R, you type the function and you get to see the entire code. In Python, one can use print lassname.__doc__ to see the documentation of the function
• Parameters with default values must come after parameters without default values
• When you call a function, Python passes the object references of the arguments from the caller to the function. For immutable objects (numbers, strings, and tuples), this behavior effectively creates a local, temporary copy of the argument inside the called function; the function cannot change the original object. The story is different for mutable objects (lists and dictionaries). Passing a reference creates an alias of the original object inside the called function. aliases are variables that share references to the same object. The original object (in the caller) will change if you modify the passed-in object in place (in the function). On the other hand, if you reassign the aliased parameter a value, the reference to the original object is lost, and changes in the function won't be reflected in the caller.
• To avoid modifying a shared mutable object in a function, pass in a copy of the object
• It's good practice to make local copies of mutable arguments in a function to ensure absolutely that there are no side effects. It should be the responsibility of the function, and not its caller, to prevent side effects.
• Functional Programming tools
• Lamda :Creates small, anonymous functions
• apply() : Indirectly calls a function and passes it positional and keyword arguments . The apply() function takes a function, a tuple, and a dictionary as arguments. It calls the function by using the tuple items as positional arguments and the dictionary items as keyword arguments, and returns the result of the function call.
• map() :Does the same thing as apply family in R.Applies a function to sequence items and returns a list containing the results.However in R, there are many more variations of apply. I hope that someone writes a plyr , reshape equivalent modules in Python. In R, I use them so often that i have put them in the startup script .
• zip() : Takes a variable number of sequences and returns a list of tuples, where the n-th tuple contains the n-th item of each sequence
• filter() : Returns a list containing the items of a sequence that satisfy a given condition
• reduce() : Applies a function to sequence items, starting with the first two items and successively using the result of a function call together with the next item, ultimately reducing the sequence to a single value
• Differences between lambda and def
• lambda is an expression, whereas def is a statement. You can use a lambda expression anywhere you'd use a normal expression: as an argument, as a list item, as a dictionary value, and so on.
• The def block permits multiple statements, whereas the lambda body must be a single expression.
• Nonexpression statements such as if, while, for, and print are forbidden in the body of a lambda expression, which restricts the number of operations you can cram into a lambda function.
• You can use a lambda expression without assigning it a name.
• A list comprehension is a concise and often clearer alternative to creating lists by using lambda, map(), and filter(). [expr for var in seq if cond] This appears like a very interesting way to cut down unnecessary code. This is something I have not seen much in the previous two books on Python that I have read. So , I guess this is one new learning for me after slogging through 250 pages of this book is list comprehension. Yet to use list comprehension in a code fragment

Chapter 9 - Modules

• Flow of content is the module is as follows : documentation string, import statements, global variable definition, module's classes, module's functions, and finally if statement that checks whether the module is being imported in to another program or is it run on a standalone basis
• Because Python doesn't rerun module code after the first import, a variable created in a module won't revert to its initial value on subsequent imports
• dir(object) is similar to str function in R. It shows the list of important attributes of object
• from module import doesn't import names that begin with an underscore, which are termed private names
• Python executes module code only once, during the first import. You can force Python to reload and rerun a module's code by using the built-in reload() function. This function is useful when you want to see module changes without stopping Python.
• You can append a directory in the sys path using the function sys.path.append(dir)
• You can determine how a module has been loaded by inspecting its built-in attribute
• sys.modules.keys() gives the list of all modules loaded
• There are three types of namespaces
• Local namespaces. Python creates a new local namespace whenever a function is called. The local namespace contains a key-value pair for each function parameter and each local variable defined in the function's body. Variables in functions that are declared global are not in this namespace (they're in a global namespace). Each function has its own local namespace, which Python deletes when the function returns a value or raises an exception.
• Global namespaces. Python creates a new global namespace when a module is first loaded via import. The global namespace contains a key-value pair for each top-level (global) variable, function, class and imported module. Each module has its own global namespace, which Python deletes when the interpreter halts.
• Built-in namespace. Python creates the built-in namespace when the interpreter starts. The built-in namespace contains a key-value pair for each of Python's built-in functions (len, str, and so on) and exception (TypeError, SyntaxError, and so on) names. There's only one built-in namespace, which Python deletes when the interpreter halts.
• locals() to return a dictionary corresponding to the current local namespace
• globals() to return a dictionary corresponding to the current global namespace.
• vars(object) to return a dictionary corresponding to the local namespace of object.
• compile() is used to compile a string into a code object. eval() returns the result of the evaluated expression. exec used to run a file, object or code object

Chapter 10 - Files

This chapter contains a list of functions that one would typically use to work with files stored on the disk. The note on pickle and cpickle module was something that I haven't tried till date. Have to work on it soon.

Chapter 12 - Classes

The following gives a schematic diagram of Python in-built types

 

• In the above diagram , Slice objects represent extended slices, which are used in Numerical Python (NumPy). Extended slices don't work with standard sequences (lists, strings, and tuples) .May be once I go over NumPy, I will learn about Slice objects in greater detail.
• Python provides several predefined class attributes. The class's attribute holds the class's attribute names and values in a dictionary
• __class__ gives the name of instance class
• A class variable exists independently of class instances. The variable is available no matter how many class instances are created, even if none is created.
• If you change a class variable's value at run time by qualifying it with the class object (class.attr = value), Python dynamically changes the value of attr to value in all existing (and future) class instances
• You can access the attribute of a class before instantiating the class. This is so different from the other languages I have coded.
• You can do functional overloading, Operator overloading by writing __XYZ__ functions where XYZ is the relevant operator or function that you are trying to overload
• pt = Point(3, 9) is equivalent to Point.__init__(pt, 3, 9)
• set the truth value of an instance by writing your custom your custom __nonzero__()
• To make instance callable , use instance callable, use the special method __call__(self [,args])
• Names in a class's namespace (__dict__ attribute) aren't visible in the class's methods, so you must
  fully qualify names that refer to class attributes outside a method
• Private names begin __ but don't end__ with a double underscore
• isinstance() ,issubclass() functions used to check class membership

 

 Takeaway :

With abundant screenshots scattered through out , this book indeed offers a quick recap of all the basic elements of Python programming.

Learn Python The Hard Way : Summary

 

Prof. Hadley Wickham, the creator of ggplot2 and other useful packages like plyr, reshape etc. has one strong advice to R programmers – “Read other’s code”. This comes from a person who has developed 30 packages till date. We all have an immense urge to program, code up something, view the results, tweak our code to make it work etc. However pausing to read somebody else’s source code requires a certain amount of hard work, willingness to learn from others . In R particularly, where all the functions are documented really well, one hardly NEEDS to go in to the code. But that’s exactly what Hadley Wickham recommends.

In that sense, this book by Zed A Shaw has a similar message for Python. You have to read code on a regular basis and it is typically hard work to read what other people have written. I guess that’s the reason why this book is titled, “Learn Python The Hard Way”. This book introduces Python step by step in 52 exercises where the author gives pointers to various modules, websites for the reader to figure out stuff. So, all the exercises have one common structure – “ introduce a topic and make the reader curious to check out things from other sources”.

As a newbie, I found this book interesting for a couple of reasons. Firstly, the author urges the reader to type out every single line of code in the book. No copy pasting allowed when you learning something new. The other thing I liked about the book is about author giving clear instructions to the reader to follow a directory structure for a Python project. For a long time I never followed any specific directory structure funda for many projects in whatever languages I have coded. However once I learnt Ruby on Rails, I understood the advantage of following a nice standardized directory structure for any task/ project/ library. Not all frameworks make strong recommendations like Ruby on Rails. So, the programmer has to figure out something that works. That’s usually a trial and error process. Starting from a well thought out directory structure in Python is going to be helpful in the long run when you want to go back, review or commit the project to Version control system.

Let me list down the things that I learnt from this book.

• %d , %s, %r  are used for substituting stuff in the string
• %() is the used to substitute the respective variable in the string
• Learnt about close , read , write , readline , truncate functions
• raw_input is used to get input from the user
• Functions appear similar to functions defined in ruby.
• Functions should start with def
• You need not put an open parenthesis right after the function name
• You can leave spaces after the parenthesis (
• You can leave spaces after the closing parenthesis ) and colon
• You go back to usual code environment from a functional environment by writing with no indent. Unlike ruby there is no need to put an end at the end of every def
• At the end of def statement , there is a need to put colon
• You have to indent all the lines of code in a function with 4 spaces.
• Duplicate argument names are not allowed
• Variables in script are not connected to variables in the function
• The variables in the function are not connected to variables in the script.
• f.seek(0) takes you back to start of the file
• f.readline() reads a specific line in the text return at the end of function can be be used to return something from a function.
• Exercise 23 was very interesting as it asked me to go and visit bitbucket.org and then browse a random python project, click on source and write about whatever I could find interesting about the project. This exercise says this “ When you do this exercise, think of yourself as an anthropologist, trucking through a new land with just barely enough of the local language to get around and survive.” Despite hardly knowing any aspects of Python, I looked up bitbucket.org and started randomly browsing a source program stumbled on bootstrap-py3k.py file from pyquery. This is what I could make out from the file.
  • You can import a ton of libraries by listing them down separated by comma
  • import X imports the module X, and creates a reference to that module in the current name space. Or in the other words, after you've run this statement, you can use X.name to refer to things defined in module X
  • from X import * imports the module X, and creates references in the current name space to all the public objects defined by that module. X in itself is not defined. So X.name will not work but name will work
  • from X import a, b, c imports the module X and creates references in the current namespace to the given objects
  • try Except. Similar to the try catch in java and other languages
  • Unlike R, If loop has no bracket and has a terminating colon
  • for x in list() - This is similar to what you find in R.
• The program that I randomly browsed was overwhelming. Goes on to say the distance I need to walk before I can code properly in Python.
• beans, jars, crates = secret_formula(start_point) - This is very different from the usual assignment that you get to see in other languages. In R something like this, c(x,y,z) <- test() does not work.
• I learnt the function pop that can be used on words
• Any block of code needs to have 4 spaces before the actual statement begins
• function definition or an if definition should terminate with colon
• if - elif - else , elif is equivalent to else if
• Format strings --  %f for floating decimal point, %d for single integer decimal %r for string %s for string
• Some of the functions associated with lists - append(x) appends the elements of x , extend(x) extends x by adding all items in the list. Some of the other functions are insert, remove, pop, index, count, sort, reverse
• You can use lists as stacks, use append and pop function to mirror LIFO principle
• Python lists start from index 0
• Wrote a program that incorporates all the 100 key words mentioned in the first  36 chapters.I think the author’s suggestion write such a program is pretty useful in getting up to speed with the syntax
• In Python, any number that begins with 0 is considered as Octal number
• The built-in number objects in Python support integers, floating-point numbers and complex numbers.

• All numbers in Python are immutable objects, meaning that when you perform any operation on a number object, you always produce a new number object
• Long integer has no predefined memory and its minimum and max values are dictated by machine architecture
• floating point in python is similar to double in C - 53 bits precision
• Always use “ ” for strings so that you can use single quotes with in the string
• Use triple-quoted string for a bigger string. Line breaks in the literal are preserved as new line characters
• Tuples are like lists, except they are immutable
• Tuples may contain immutable objects
• join is an interesting keyword. If you say stringvar.append(list[3:5]) , it adds stringvar between each of the list items
• dir(li) returns a list of all the methods of a list.
• dir(d) returns a list of the names of dictionary methods
• Tuples may contain immutable objects
• Lists and Dictionaries are the power horses in Python. They are best utilized as iterators.Lists are ordered collection whereas Dictionary is an unordered collection
• Understood the importance of map operators.
• Lists can have functions embedded in them.
• Each of the functions in the class takes self
• Indentation is very appealing. I don't have to worry about the painful flower brackets
• You can assign functions to any variable , you can put functions in a list. 
• __init__ sets up all the initial variables of the class. It is the constructor function for the class
• self is a key word that is used in the context of the class. it is similar to `this' function in C++
• To instantiate a dictionary , either use X = dict() or x= {}
• Use str function to convert numbers to strings
• Another use of dictionaries is that you can assign some data to a key and be certain that there will no be duplicates. If at all you try to add a duplicate to dictionary, the data for the last entered item would be taken as the relevant data for the key.Let's say you are reading a lot of items and you want to find the last encountered data, you can use dict for that purpose
• Names starting AND ending with double underscores work differently.
• __doc__ when used in this sense print x.__doc__ prints the comment in the function.
• __dict__ contains attributes in a class instance
• Learnt about pass keyword. pass is a null operation - when it is executed, nothing happens. It is useful as a placeholder when a statement is required syntactically, but no code needs to be executed for example
• Learnt the way to use setattr  and getattr 
• The takeaways from PEP8 , the python coding style guide are,
  • Arguments on the first line forbidden if you are not using vertical alignments
  • Use spaces or tabs. Don't mix both. If possible use only spaces
  • Maximum line length is 74 for code and 72 for comments and doc string.
  • Separate top-level function and class definitions with two blank lines
  • Method definitions inside a class are separated by a single blank line
  • Use blank lines in functions ,sparingly, to indicate logical sections.
  • Imports should be on separate lines
  • Imports should be at the top of the file
  • Imports should be grouped in the following order : standard library imports, related third party imports, local application/library specific imports
  • Avoid whitespaces
    • Immediately inside parentheses, brackets or braces
    • Immediately before comma, semicolon, or colon
    • Immediately before the open parenthesis that starts the argument list of a function call
    • Immediately before the open parenthesis that starts an indexing or slicing
    • More than one space around an assignment operator to align it with another
  • Always surround binary operators with a single space on either side
  • Use spaces around arithmetic operators
  • Don't use spaces around sign when used to indicate a keyword argument or a default parameter value.
  • Multiple statements in a single line are generally discouraged
  • Module names should have short all-lowercase names
  • Class names use CapWords convention
  • function names should be lowercase
  • Always use self for the first argument to instance methods
  • Always use cls for the first argument to class method
  • Method names inside the class - use lowercase words separated by underscores
  • Leading underscore in a variable name for private variables
  • Two underscores to invoke Python's name mangling rules
  • Constants are CAPS_LETTERS
• Here are the steps that I have followed to install python packages
  • Download ez_setup.py in to Scripts folder
  • Set the System Path, Windows path variable to contain this directory
  • Double click on this ez_setup.py module
  • RPy needs R.dll and hence you need to give to append the relevant folder name in the path variable
  • easy_install executable and other scripts get populated in Scripts folder
  • Installed pip using  easy_install pip
  • Installed numpy .
  • Download win32all, windows extensions from Mark Hammond .
  • Downloaded RPy installable
  • Installed scipy using easy_install
  • Currently I am using R 2.13. Rpy is built for R 2.12. Hence had to download R2.12 , give its bin in windows path and then invoke the python script.
  • Some description of the packages that I have installed
    • pip - used for installing packages which are present in the PyPy package index
    • distribute - Is a lower level tool for building, installing and managing Python packages
    • VirtualEnv - Is a tool to create isolated environments for Python
    • Nose - Extends unittest to make testing easier
    • NumPy/SciPy : This pair of libraries provide array and matrix structurs, linear algebra routines, numerical optimization, random number generation, statistics routines, differential equation modeling, Fourier transforms etc. Basically you get the entire MATLAB toolkit for free
    • Need to check out these modules sometime – IPython, Cython, PyTables, PyQT, TreeDict, SQLAlchemy, Sage, Enthought, Sphinx. Loooooong way to go !
  • Understanding setup.py 
    • What is it ? This is used by Python distutils as a standard way for installing third party Python modules. Before distutils, module creators would have to create install files for all the different platforms. That activity is made redundant ,thanks to setup.py.
    • What does this file contain ? It contains all the calls Python makes to distutils
    • What happens when we run it ? There are two things that happen. First is the build step which puts all the files which need to be copied in to the distribution root's build directory. Second step is the install phase where all the files are copied to the install directory for installation
• The steps I carried out in order to install a custom build python module are :
  • Make a copy the default skeleton package and rename it to project name, XYZ
  • Rename the NAME module to the XYZ
  • Rename NAME in the all the relevant files
  • Ceate a rk module and put it in XYZ folder
  • Edit the setup.py to contain
  • Run python setup.py dist - This will create a zip file in dist folder
  • Extract the dist folder wherever you want and run pythonsetup.py  install
  • The module is installed in python and you can import in the code and start using it
  • If you have to uninstall this egg, you have to run pip uninstall “name given in the setup.py”
• Came across HitchHiker's guide to Packaging on the net. Will go over it someday at leisure
• Exercise 46 has a nice introduction to Python packaging. Somehow I find packaging in Python much easier than R. Well I should not comparing two different things , but as a amateur programmer, I think that, it is easy to install R packages , at least there is no additional stuff to install to install packages. a command like install.packages() is all that is needed. For Python, it is not as straight forward as it seems. You need to install pip, or an executable and then use easy_install command. For packaging though, I found steps in python to be crystal clear and easy to do. However in R, for some reason I found it a bit difficult to learn this stuff. R CMD makes it easy though, but if you want to put in documentation for function, tests, etc , it will take some time to learn packaging in R. May be I am just rambling here. R enthusiasts will dismiss my statement that it is far easier to package stuff in R than in any other language. May be ..But somehow after going through this chapter, creating an installable in Python appears very intuitive and easy. The fact that Dropbox, a startup that became a fantastic years in the recent times uses Python for everything says a lot about the versatility and usefulness of Python packaging.
• Some gyan on unit testing
  • Write one test file for each module you make
  • Keep test cases short
  • Reference to doctests and nosetests Have to read more about them someday
• Learnt about isdigit function that can be invoked on a string to check whether it is a string
• I quickly browsed Chapter 50-51-52 as I don't think I will doing any webdev work in the times to come. If at all I need to do something I will probably use ruby on rails and get it done. 

The author concludes the book with a superb reminder to any programmer

Which programming language you learn and use doesn’t matter. Do not get sucked into the religion surrounding programming languages as that will only blind you to their true purpose of being your tool for doing interesting things.

Think Python : Summary

If there is a lot of data parsing and cleaning that needs to done before modeling, I tend to follow one of the three paths :

• Path 1: Use Python to clean the data, export the data structure in to a file/database. Leave Python environment and move in to R to do the modeling.
• Path 2 :  Use Python to clean the data, Stay in Python environment and invoke R to do the modeling. Rpy is the go to module in this context.
• Path 3 :  Painfully do the data cleaning in R, despite R hogging memory, and then model stuff in R

Path 3  is something I take very often. However Paths 1 and 2  are also interesting as they give a ton of modules that one can use from Python. A few years back I had used some data types of Python, mainly the dictionary and had worked on something I don’t even remember properly. It was more of an ad-hoc task and had since then never used Python in a big way but for some basic data cleaning tasks. Over the years I have slowly graduated to performing the entire data cleaning exercise in R itself and completely avoid Python. Lately I have realized that I have followed a convenient path instead of a hard but worthwhile paths(1&2). So, I picked up this book to get a decent understanding of data types and modules in Python . In this post, I will list all the points that I found relevant in this book for a newbie like me :

• The three common elements of natural languages, i.e Ambiguity, Redundancy and “Not literal in meaning” , do not apply to programming languages. They have exactly the opposite attributes. They are Non Ambiguous, Non Redundant and Literal.
• Syntax rules come in two types, tokens and structures
• Python is an interpreted language. The error with Python code could be Syntax error, Run time error or Semantic error.
• >>> symbol is called chevron.
• invalidSyntax and invalidToken are the usual exceptions that you come across.
• Python variables are case sensitive.
• PEMDAS - Useful mnemonic for remembering python order precedence.
• ^ is not an exponent operator in python.
• type() is a function that is useful to know the type of the object in Python. It is similar to class function in R.
• Semantic errors are tough to catch.
• There are at least 31 keywords in Python.Type import keyword ; print keyword.kwlist to get the list.
• import math will fetch all the math functions from the python standard library.
• Function definition has to be executed before the first time it is called
• Stack diagrams are used for understanding function environments
• void functions - nice way to call functions that don't return anything.
• To see where Python searches for libraries, type the command import sys; sys.path. The output is a list and the first item in the list is a null string symbolizing the current directory.
• Passwords are never stored as plain text , be it in a file or a database. They are converted to a hash code and stored. When a user enters the password it is checked with the hash code stored internally. The best thing about hash code is it involves a mathematical one way process between password and code. It is very unlikely that you will be able to crack the password given a hash code
• There can be compiled python code also. These days there are some apps that are distributing compiled python code instead of the usual py files
• Set environment variable PYTHONPATH to a colon separated list of directories so that python searches for the relevant folders while execution
• The trick if __name__ = “__main__” exists in Python so that Python files can act as either reusable modules, or as standalone programs.
• How to draw a fractal using Python ?.
• Functions that return something are given a  catchy name, Fruitful functions
• If a function returns no value and you try to print, it will print None
• Some examples of recursion mentioned are gcd function, palindrome function
• Interesting use of the word bisection : Debugging by bisection. Well, basically apply the bisection method to finding out the bug.
• eval function can be used to evaluate python commands in a string
• Functions that can be used with string are len, for, slicing, upper, lower, find
• Index range in string means including the first index to second index , excluding the second index
• The word “in” is a boolean operator that takes two strings and returns True if the first appears as a substring in the second
• Python does not handle uppercase and lowercase letters the same way that people do. All the uppercase letters come before all the lowercase letters.
• Strings are immutable
• split function takes a third argument x[a:b:c] means from a to b in steps of c
• The following code is to reverse a string :x[::-1]
• Program testing can be used to show the presence of bugs, but never to show their absence!
• There are some string related functions built in to Python that can be used for text processing stuff.
• zfill() function can be used to pad zeroes for a number
• int() is equivalent to as.numeric() in R
• Saw similarities between map operator and functions in R. May be this is the reason why people world over love python.
• The list object stores pointers to objects, not the actual objects themselves. The size of a list in memory depends on the number of objects in the list
• The time needed to get or set an individual item is constant , no matter what the size of list it
• The time needed to insert an item depends on the size of the list, or more exactly, how many items that are to the right of the inserted item(O(n)) . In other words, inserting item at the end is fast than at the beginning
• The time needed to reverse a list is proportional to O(n)
• The time needed to sort a list varies, worst case is O(n logn)
• Useful functions that go with lists “in”, “extend”, “append”, “sort”, “sum”,+,*,[a:b]
• A nice analogy between map, filter and reduce functions applicable to lists. Capitalize is like map where you apply a function to each element, filter is like selecting only some items from the list , reduce is like summing up all the elements or counting all the elements in a list
• list(s) creates a list out of the elements of the string
• split is a function that returns list
• The most dangerous aspect of python unlike R is that , if you assign a list to a variable X, then you say Y=X , if you make changes to Y , the changes are reflected in X. Basically it is pass by reference and not pass by value.In R, the external object is not changed , pass by value happens . In Python code, the external object is changes as pass by reference happens. This means most of the functions in R are pure functions.
• in operator to check the presence of an element in a list. One can also use index function to check for the presence of the element.
• To reverse a word, use the following code x[::-1]
• One can use sorted function for list
• append modifies the list and returns none
• always use append instead of a = a +[x]
• This is a fantastic thing in Python. “in” operator in dictionary takes the same time irrespective of the size of the dictionary.
• While checking for an word in a list of words, the “in"" function is slower than bisection method , which is slower than the access through hash table in a dictionary.
• For dictionaries, python uses an algorithm called “hash table” that has this remarkable property that “in” operator takes the same amount of time irrespective of the size of the dictionary
• Hash tables are apparently used to create a 2 dim array where you store keys as hash values and use these hash values to map to the actual values. Basically hash table is extremely useful when doing stuff with a large number of strings
• You cannot use lists as keys in dictionary as lists are mutable. Any mutable object cannot be used as keys for a dictionary - Mutable objects give rise to duplicate hashes. Similarly a dictionary cannot be used as a key
• A previously computed item that is meant for later use is called memo
• Any variable defined outside the scope of a function is treated as global variable. You can happily use them in a function. However you cannot set them in a function as any setting operation introduces a new variable in the function whose scope is only limited till the function is running.
• To set a global variable in a function, you have to define the variable as global.
• You can add, remove or replace elements of a global list but if you want to reassign the variable, you have to declare it
• Learnt about a way to check duplicates using “set”  in Python.
• Tuples are a sequence of values. The values can be of any type, and they are indexed by integers. They are immutable. This is the key to understanding tuples. Unlike dict and lists, tuples are immutable.
• You cannot modify a tuple but you can replace one tuple with another
• Tuples are good for swapping operations and return values
• There are certain functions like divmod where you enter the input by scatter
• zip is another feature of tuples that is very useful
• items function on a dict returns tuples
• You can compare tuples
• Decorate, Sort, Undecorate pattern is useful in many situations like sorting, counting, etc
• You can use tuples or dictionaries to pass parameters. To pass tuple as parameter , append * . To pass dictionary as parameter, append ** for the parameter value.
• A few days back I stumbled on to Ziff's law in an NY Times article.  I managed to use dict,tuples, lists to empirically check ziff's law on Jane Austen's novel, Emma
• Also I have started exploring the Rpy module to invoke R from python. There is some initial pain in learning how to install Rpy. Once that is done, R can be used seamlessly in Python.
• Computed a Markov Chain for Phrases in Jane Austen’s novel Emma
• Tuples are very useful in sorting stuff or creating results similar to table in R
• You can sort a list if the elements are arranged in the form of tuples
• After reading the chapters on text parsing and string handling, I have this feeling that Python is the king for text parsing. No wonder it is used in Google and other places where they have to deal with a ton of text.
• To randomly select items from a histogram, one can create a list where each element is repeated x number of times, where x is frequency of the word
• Random is a useful module which has functions like random(), randint() , choice()
• If you want to remove punctuation from strings , you can use string.punctuation to check the elements that need to be removed.
• os module has many useful functions like os.getcwd(), os.path.getabspath(), os.path.exists(), os.path.isdir(), os.path.isfile(), os.path.listdir(), os.path.isfile()
• There is a mention of pipe in python that is useful in reading very big zip files.
• repr(s) can be useful for debugging.
• Learnt to screen scrape using Python
• Reorganized my iTunes folder with Python.My iTunes folder had duplicate files and I had to remove those duplicates. Obviously manually going over them was a nighmare. Firstly I removed some obvious files like video files and other non music files from the 1500 files from the folder. Then I used MD5 function, used the function to find duplicates in my music folder and removed them programmatically. Now I have about 978 music files in total that I will categorize someday in to various playlists. 
• The classes are pretty peculiar in Python. These classes have no attributes but you can generally assign instance.x, instance.y to some value. I mean this is weird for me as I have always thought that classes having attributes and functions
• I am coming from R to Python and I am alarmed at the fact that Python passes objects by reference. In R objects get copied. In Python a reference to the object is sent. This means that function can totally change the object that you are passing
• You can check whether two objects alias to the same data by using the `is' operator.
• copy module has two types of copy functions, one is the shallow copy and one is the deep copy.
• hasattr is a function that can be used to check the attributes of the object
• Learnt new terms like invariants, pure functions and modifiers.
• Functional programming is a style of program design in which the majority of the functions are pure. Pure meaning that whatever input is received by the function, it is not modified.
• Came to know about datetime module , probably the most important module that I will use.
• There is a strange thing about invoking functions in Python. When I first came across functions, I was left wondering why there is a need to pass self in to each of the function. At least in the languages that I have coded I have never passed self object. This chapter made me realize that the major reason for passing self is that there are basically two types of invocations in Python. Let's say I have a class X that has a method test. Either I can invoke it via X.test(obj) or via obj.test() . If the method takes an argument, then you code the function with the first argument as self and give the rest of the arguments in the usual way
• use __init__method for default constructor
• use __str__ for giving a string representation of the object
• use __add__ for operator overloading of +
• Think of overwriting some operator, look at python docs to find the exact string to use , let's say it is YYY , then write a method with the name __YYY__ and your class has the operator overloading set.
• use __dict__ to get the attributes of the class
• use getattr function to get attributes of the class
• Learnt about the use of __radd__
• pass statement has no effect in a class definition. It is only necessary because a compound statement must have something in its body.
• The concept of deep copy and shallow copy is something that I came across it in the context of Python, after my initial encounter with them in C++.
• Default values get evaluated ONCE, when the function is defined; they don't get evaluated again when the function is called.
• Even though two lists or tuples are distinct objects with different memory addresses, Python tests for deep equality. In some other instances ,shallow equality is tested.
• There is a difference between instance attributes and class attributes.
• At the time of class definition itself, the inheritance structure is defined in Python
• With the help of cards, deck and hand, the chapter on inheritance gives a good introduction to the various concepts related to inheritance.
• I have ignored chapter 19 as I am planning to use R for visualization. Only if I cannot do something in R, would I probably come back to this book and learn about GUI capabilities of Python.
• Appendix talks about debugging. Typically there are three types of errors that one comes across in programming. First are the syntax errors. Second are the run time errors such as Name error, Type error, Key error, Attribute error, and Index error. The third type of errors is semantic error and is often difficult to crack as compared to the first two types of errors.

The Pleasures of Reading in an Age of Distraction: Summary

Alan Jacobs, the author of this book is an English Professor at Wheaton College, Illinois. Given his position as a professor, his students and other people often ask him, “What are the 10 best books on literature that every educated person must read", "Dear Prof, Can you suggest some books to read this summer?", This book is written to answer all such questions. So one might think this book is basically a recommendation type / instructional / didactic guide to reading. Far from it, this 150 page long essay on reading at Whim, with no fixed pattern, with only one objective in mind, "Pleasure".

The book starts off with the author noticing that many people including his son are put off by books such as, “How to Read a Book?”, “ How to Read Literature like a Professor ?” , “ The New Lifetime Reading Plan” , etc. The premise behind all these kind of books is that reading needs to be systematically carried out and there are certain books that need to be read to appreciate and become good at understanding literature. Most of these books smell of Responsibility, Obligation and Virtue, the very attributes that make people make run away from reading. So, he says, reading needs a model that works, i.e "Read at Whim". The people who look out for such "10 best books to read" recommendations actually don't really want to read a book, but want to check things off from a mental bucket list. They want to say,“Yes, now I am done with this book”. Reading at Whim means reading something that gives you pleasure,i.e there is nobody that we are signaling to , nobody that we are trying to impress.It is really out of pure enthusiasm that one reads. One usually sees this in children when you give them a book. They read it for the pure joy of it. There are tons of authors out there who feel that reading must not be frivolous, meaning, Harry Potter is not serious book, in their opinion. In fact they have this assumed checklist of books that,`Ought to be read' by a serious reader. The author states that this model is broken, and says, "Read at Whim" should be the new model.

Ok, fine. You should read at Whim, So pick up whatever you feel like reading and the one that you think will give you pleasure. Done deal. 20 pages in to the book, the author makes this abundantly clear.So, Is there any point in going over the 130 odd pages in the book ?

Well, the rest of the book is NOT reiterating this message over and over again. `Reading at Whim' is the foundation of the model that the author talks about in the book. If this were the only principle that we follow, soon we will be facing with situations as these

• Let's say you like Jane Austen novels and you get joy/pleasure reading her words. However you soon hit a limit. She wrote only six novels. So, once you read these books, you would want to go back and read these books again. But,as we all know, too many rereading squeezed into too narrow a time frame will drain the books' power and leave them forever inert on the shelves. So you face this Law of Diminishing returns when all you want to read are a few select books.
• Let's say you like Lord of Rings and really loved reading it. Soon, you might start reading books that carry stories similar to Lord of Rings.Given the amount of books published in a year, it is certain that there will be enough imitators of hits. You might get frustrated reading those books as they fail to match up to the original. You might accept them for what they are, imitators of the original and keep reading. The second behavior can be dangerous as you might start accepting plots less clever, characters less vivid, prose less dynamic and thoughts less insightful.

So, you see reading at Whim can take us only so far. In this context, the author talks about the second element of the model, i.e self-knowledge and discernment. These are crucial to develop while reading. These will help you chuck the books midway, if you think prodding through the text doesn't give you pleasure. This also makes you aware of your tastes and preferences. “ Self-knowledge and Discernment” are precisely the things that you will not develop if you tend to follow somebody else's recommendations, maintain a list of books to be read, etc.

“How to Read a Book” and similar guides offload accountability for our reading: they say, implicitly, that self-knowledge and discernment aren't needful because experts can take care of that for us. But if we reject that implicit claim, the next question that needs to be addressed is,“ How to move from `blind propensity' to `informed consent' to `Whim's sovereignty' ” ? One of the suggestions by the author is to “Read Upstream”, i.e read books that your favorite authors have read as they give a peek in to your favorite books' characters, plots and imagination. This kind of upstream reading is also useful in math. You might come across a good application of a technique, but if you read upstream you might get to read all the trials and tribulations that went behind the technique etc. For example,Baire's failure in categorizing functions helped Lebesgue in defining measurable and non-measurable functions. If you read ONLY about Lebesgue and don't look in to the development made by Baire, you are likely to miss a lot of action. Reading upstream need not be only be about historical developments behind a technique. It might be about things that make you wonder, curious about life in general. If you look at Cantor's math and read about the Cantor's life, what shaped his ideas about infinite infinities, What drove him mad, what made him die alone in an asylum, What made his story tragic but his achievements a mathematical breakthrough, you will forever look upon George Cantor in a completely different light.

The author then makes a strong case for annotating the text/ reading with a pencil. By turning our passive reading style in to an active one, the book tends to offer more than what it might seem in the beginning. There is also warning against highlighting, as Highlighters allow you very quickly and easily to mark a text, but only by covering it with a bright color; and the very quickness and easiness of the process are inimical to the kind of active reading that is needed. This point is similar to Dr.Medina's finding mentioned in his book Brain Rules. By making the initial contact about an idea/phrase/character more elaborate , it is likely that one remembers better. By reading fast we miss on the opportunity of elaborate encoding. Obviously this does not apply to every book. One should not read Harry Potter with a pencil , such books are good when the reader goes with the momentum, the less stoppages the better. This means that as a reader, the decision to annotate or go with the flow of the book is important.

Reading Slowly is the next aspect that author focuses on. Most of us read fast because of the implicit thought that “ Time is too short to read all the books”. Yes time IS short, but one crucial aspect that gets neglected by making reading . `a race' ,is, “Books become better when they are reread. Unless you annotate , read slowly, your re-read would be equivalent to a new read". Reading fast - It's like you have the content uploaded in your working memory, feel good about it, check off that item from the list, move on to the next book." Considering the short term nature of working memory, its like all the content is in RAM. Once the application shuts off, RAM is erased. If you want the stuff to get stored in long term memory, you have to read slowly, annotate and MOST IMPORTANT part is to re-read. Whenever you have the urge to read a set of blogs / books in quick succession, pause and ask yourself, “ Do you want to read' ? ” or “Do you want to have read ?”. An honest answer will keep you off the speed track.

Via a Poem from W.H.Auden, the author makes a case for `eye-on-the-object' look that is needed for getting pleasure from a book, i.e we must cultivate attention while reading. We need to be attentive of words , phrases, characters, etc. so that we can lose ourselves in the process of reading. The poem mentioned in this context is very beautiful and goes like this,

You need not see what someone is doing to know if it is his vocation.

You have only to watch his eyes; a cook mixing a sauce, a surgeon

making a primary incision, a clerk completing a bill of lading,

wear the same rapt expression, forgetting themselves in a function.

How beautiful it is, that eye-on-the-object look.

There is a section that talks about a 12th century Abbot, Hugh's advice to his monks. Even though it belongs to advice centuries ago, it is equally relevant for people whose motives for reading are far from monastic. Hugh's advice on humility is relevant to the book as it says the reader should keep in mind three aspects,

• Hold no knowledge or writing whatsoever in contempt
• Should not blush to learn from any man
• When he has attained learning himself, he not look down upon anyone else.

These lessons mean that one should not only be attentive to what one studies, but also positively disposed towards it: friendly,even affectionate.

Amidst all this discussion about reading, the author takes a radical view point , i.e Schools can never teach students to deep-read. Irrespective of which class a student is in, there is always this feeling that, ` I will be graded' lurking in his mind. So, the kind of attentiveness that is proper to school is more of `hyper attention' than `deep attention'. Look at any kids curriculum, you will amazed at the QUANTITY that is covered as a part of syllabus. With grades and the competitive pressure, Can a student deep-read ? No , says the author as reading textbooks and the like-does not require extended unbroken focus. It requires discipline not raptness.I don't agree to this point. Yes, a student probably can't deep-read all subjects but I think focusing on a few subjects and understanding them really well, might be better than knowing a bit about all the subjects. Yes, the student might fall behind on the average grade across subjects, but he will graduate from a school or a college with a better frame of mind. However looking at the way the educational system in India, I think the author might be right as LOT is taught and tested from the young minds that there is no choice but to cram.

One of the most important points that I found relevant to my reading habits is : Reread. I tend to read math /stats books a lot and I find it imperative to reread them.Well, one aspect of summarizing and posting them to a blog is that, these summaries serve as a starting point when I reread a book. The author makes a strong case for reading and I quote the author ,

If most of us read too fast, most of us also read too many books and are unwisely reluctant to return to something we think we already know. I use "think" here advisedly, because , as my examples show, a first encounter with a worthwhile book is never a complete encounter and we are usually in error to make it a final one. But those who want to have read, who are checking books off their bucket list , will find the thought of rereading even more repulsive than the thought of reading slowly and ruminatively. And yet rereading a book can often be a more significant dramatic and new experience than encountering an unfamiliar work

This visual broadly gives the structure.model explained in the book

 

Takeaway :

We usually read for information or understanding or entertainment. Dismissing all the so called expert recommendations that one receives on reading, the book has one central message , "Read at Whim". It warns the reader from making reading in to a 'have read' activity.

Creativity Now : Summary

This book contains most of the productivity hacks that one comes across in various articles/blogs/books.  In one sense, this book is a laundry list of hacks that one can try out to increase productivity. A big font size for the text and rich images scattered through out the book, makes it a coffee table book.

Some of the hacks that I found interesting are,

• On a daily basis, Try to use pen and paper at least a few minutes to work something out , be it a math problem or a back of the envelope calculation of something or simply draw images that capture whatever you are working on. Writing makes one’s relation to work intensely personal and more so using a pen and paper.
• Keep a Swipe file – Your swipe file should contain good ideas and examples from your field of work or interest and from other fields. This hack is spot on for a programmer. You got to keep a folder that lists the tasks and the most efficient code that you have figured out for the task.
• Force a image or word association with a number and vice-versa. This little hack is very useful for remembering stuff.
• Declare a MAD – Massive Action Day :  Pick a day and focus on only one task. Switch off TV, Mobile, RSS alerts, email alerts, Internet etc. and work on something for 8-10 hours at a stretch.
• Keep an “IDEAS BOX” – Twyla Tharp mentions this as a key to her successful career.
• Make a NOT-TO-DO list. We always seem to know what we need to do and we don’t care or think about what not to do, in a day
• Use time pods – 45 min time period where you focus on only one task. More on the lines of Pomodoro technique where you work on 25 min time slots.

Stop What You’re Doing And Read This! : Summary

The preface to this book gives a reason behind the title for this book,

This year we learnt that there are many thousands of children across Britain who cannot read competently, that there are thousands who leave primary school unable to put together basic sentences. One in three teenagers reads only two books a year, or fewer, and one in six children rarely reads books outside of the classroom. Many parents do not read stories to their children, and many homes do not have books in them. Stories and poems, for these thousands of children, are not a source of enchantment or excitement. Books are are associated with school, or worse – they are associated with acute feelings of shame and frustration.

The ten people who have contributed to this book are from very different backgrounds. Some grew up with a multitude and variety of wonderful books within their reach; some had parents who imparted to them a fierce desire for books and for learning; for others, books were hard to come by, or even illicit. But all ten are united here in a passionate belief in the distinctive and irreplaceable pleasures and powers of reading. They describe a poem as a lifeline, a compass, or literature as the holding place of human value.They each contend that books are not just for the classroom, but must be made easily available beyond it, because great books are essential to a richer quality of life. These writers know that learning to read transformed their very brains, and that literature has helped them to express their questions and ideals, and molded their imagination and sense of self. This book is a manifesto. In a year of rude awakenings to low levels of literacy and a widespread apathy towards books and reading, this book demands an interruption. Stop What You’re Doing and Read. Read these essays, because they aim to convince you to make reading part of your daily life.

In this post, I have tried listing down some of the statements/comments/questions/remarks that I have found interesting in the ten essays penned in the book.

  Zadie Smith

• I have spent a lot of time in libraries. To go somewhere to study, because you have chosen to, with no adult looking over your shoulder and only other students for support and company – this was a new experience for me.
• I came to understand why silence is necessary for serious study, and what the point of coffee is.
• You might discover how quickly an afternoon with a toddler passes in a local library, quicker than practically anywhere else. You might realize that giving up smoking or writing a novel is easier to do when you are one of a group of people all seated on some fold-up chairs in a circle. It might strike you that what you really want in life is silence. Despite the many wonders of the Internet, you might suddenly long for the smell of the old books.

 

  Blake Morrison

• The best books give us a lifeline, a reason to believe, a way to breathe more freely.
• The existence of the text is a silent existence, silent until the moment in which a reader reads it. Only when the able eye makes contact with the markings on the tablet does the text come to active life. All writing depends on the generosity of the reader.
• It takes courage to own up dark thoughts and dangerous feelings. But poetry – the most intimate yet public of forms – is the ideal place.
• Poetry is a serious business, but it isn't solemn or funereal. All it insists on is that we read carefully, with concentration. The shortening of lines is a signal to slacken our pace.
• Reading catatonically can become a way of coping with travails. To read so as to limit the power that the real world has over us.
• When you are deep into a book, it's for yourself alone, But once you've finished, if it's any good, you want to share it with others – rehearse the story, assess the characters, discus what makes the book special. Children do this instinctively, and the growth of book clubs, reading groups and blogs reflects a continuing and perhaps increasing need for adults to share books in this way.
• Don't build a canon( reading list) just for the heck of it. Read at whim and don't be intimidated by experts
• Sometimes we find places that are very comforting just like books. We tend to keep these places a secret so that it doesn't get ruined when a lot of people discover it. Books are NOT like that. We have no investment in keeping it to ourselves. Share them. The world will be the better for it.

 

 Carmen Callil

• “Live always in the best company when you read”.This my and mother I did: we read for company
• To this day, I do not move outside the house without a book; inside my house they are my paintings, my decorations, my fellow travellers and my comfort.
• Books are shields against a terror of boredom, that curse of most childhoods
• Books are like gardens; a Kindle or an iPad is like a supermarket- it makes life easier, but one doesn't want to loiter in it. You can fiddle with books. Like gardens, they can be wonderful to look at.

 

 Tim Parks

• Only the sequence of signs matter. The writing is in the sequence of the signs. This is the one thing that we can't change. The experience is the sequence. The experience is not in any one moment of perception, but in the movement through the sequence from beginning to end, at our own speed, with interruptions.At beginning of each sentence we are projected towards the end. At the end we have the momentum of the beginning. Same with the paragraph, same with the chapter, same with the whole book, may be the trilogy. The beginning requires the end, the end the beginning. We are locked into a journey.
• It's learning how to take intense pleasure in reading that makes it so useful. There are two pleasures in reading, one enchantment and second , awareness of the same. Two seemingly diverse experiences make any reading a pleasure.
• Life is simply too short to read the wrong books, or even the right books at the wrong time.
• So, this novel, which was definitely not what you were looking for, now turns out to be exactly what you needed. It has allowed you to discover something new about yourself, because you were watching your reaction as you read it.

 

  Mark Haddon

• It also occurs to me that whilst I read different books these days, my reasons for reading have changed very little. It's the thrill of being transported to another world.
• Select the right words and put them in the right order and you can run a cable in to the hearts of strangers. Strangers in China, strangers not yet born.
• Books are like people. Some look deceptively attractive from a distance, some deceptively unappealing , some are easy company, some demand hard work that isn't guaranteed to pay off. Some become friends and some stay friends for life. Some change in our absence - or perhaps it's we who change in theirs - and we meet up again only to find that we don't get along any more. Unlike people, one can at least dump them or hand them to a friend without causing offense or feeling guilt.
• I am always in search of novels that understand and articulate precisely what it feels like to be a human being.
• Lay the novel alongside film and its specialness becomes obvious. Film promises everything. Ancient Rome, dinosaurs, talking dogs, car chases, sex, Mars, vampires... Such a boundless cornucopia that we forget what it can't do. It can't do smell or taste or texture. It can't tell us what is is like to inhabit a human body. Its eyes are always open. It fails to understand the importance of the things we don't notice. It can't show those long stretches of time when we are seeing nothing at all, just drifting in our own minds.
• The sense of being inside looking out, of seeing a world that belongs to everyone, but is nevertheless yours alone. It is this uncrossable gulf between me and not-me, between my private experience and yours, which lies at the heart of being human and which no other medium can touch, and this border is where the novel lives and moves and has its being.
• Talking about reading as the cause of anything is to get things back to front. Reading is primarily a symptom. Of a healthy imagination, of our interest in this and other worlds, of our ability to be still and quiet, of our ability to dream during daylight.
• If we want more people to enjoy better books, whatever that means, we should concentrate on the things that prevent people reading. Poverty, poor literacy, library closures, feelings of cultural exclusion. Alleviate any of these problems and reading will blossom. These are real threats, not technology, not the pervasive and risking fear that readers are being tempted else where by the shallow pleasures of Britain's got talent.
• Novel is just the right words in the right order.
• Films and television programs, plays and paintings and sculptures never really become friends in the way that novels do. We can admire, we can be impressed, we can be moved and consoled, but we rarely feel that peculiarly personal attachment we feel to a loved novel, because whilst writing novels is a long and solitary business, reading them is always a collaboration, and a good writer gives the reader space and encouragement to play their part so that when we close the final page, we have had an experience that is partly of our own making.

 

  Michael Rosen

• I can't really sort out who's who, real or imaginary, and I think this is how we all read when we have time and space to think about books.
• Dickens told us about a Miss Havisham whom he created, but when many of us read about that Miss Havisham, we bring her to life with the Miss Havishams we know in our lives.
• My father's performance had given such life to the characters that their vocabulary became ours, and they could now live with us on the campsite, and , it turned out, beyond, for years after.
• Part of the power of stories is the way in which we can see facets of this or that fictional person in the people we know, and scenes from the fictional world have echoes in the events of the real world.

 

  Dr.Jane Davis

• You think of reading as an individual, even a solitary activity, one that you want to defend as such, because usually, for devoted readers, the act of reading is deeply private. I'm going to argue, though, that even highly proficient readers might want to try shared reading, which is in equal measure about books and people. It isn't just about getting non-readers in to reading; it is about building relationships out of communal meanings. Sharing a books is a multiplier, as anyone who has ever read, night after night, to a story-besotted child will know.It is about mutual recognitions, sharing of selves.
• Read the poem alone and you have your own experience and imagination to touch the poem in to life. Read it with six others and you have six lives and six imaginations with which to inhabit this flexible human-shaped space.
• Over the last 100 or so years, the loss of religious as a reputable discourse in common life has led to a poverty of language and this to a poverty of contemplative thought and feeling about what we are, and what we need. We need some inner stuff, scaffolding to help us get around our inner space, something to help us map, explore and even settle those places where we are still primitive.
• What is that part of being human which is touched by silence, which recognizes an intense atmosphere when people are moved, which gets scared or exhilarated when alone in a big space, or when faced with a newborn baby? Science may gradually work this out that is our mainstream model these days for accredited seriousness, for what we can be confident in believing. But literature- too often now dismissed or misplaced - has always known that buried part, and in thousands of ways.
• What people instinctively know, and science is beginning to understand, is that what makes people happy, above all, is a network of supportive fellow creatures, a sense of purpose, challenge and meaningful occupation. Shared reading can provide all this. Get a few people together, pick up a good book and try it.

 

  Jeanette Winterson

• There are two dominant modes of experience offered to us at present - actual( hence our appetite for reality TV, documentaries and “true-life” drama) and virtual- the Web. Sometimes these come together as in bizarre concept of Facebook : relationships without the relating. Reading offers something else; an imaginative world.
• To cross the threshold of a book is to make a journey in total time. I don't think of reading as leisure time or wasted time and especially not as downtime. The total time of a book is more like uptime than downtime, in the way that salmon swim upstream to get home.
• Reading is becoming a casualty of the surf-syndrome of the web. Reading is not skimming for information. Reader is a deeper dive. Or a high climb. Nan Shepherd talks about the exhilaration of altitude. The air is thinner. The body is lighter. But you have to acclimatize. You have to acclimatize yourself to books.
• The consequences of homogenized mass culture plus the failure of our education system and our contempt for books and art, mean that not reading cuts off the possibility of private thinking, or of a trained mind, or of a sense of self not dependent on external factors.
• A trained mind is a mind that can concentrate. Attention Deficit Disorder is not a disease; it is a consequence of not reading. Teach a child to read and keep that child reading and you will change everything. And yes, I mean everything.

 

  Nicholas Carr

• For Emerson, the best books – the “true ones” – “take rank in our life with parents and lovers and passionate experiences, so medicinal, so stringent, so revolutionary ,so authoritative”.
• Books are not only alive; they give life, or at least give it a new twist.
• The groups of nerve cells , or neurons, that are activated in the brains of readers closely mirror those involved when they perform, imagine or observe similar real-world activities
• When we open a book, it seems that we really do enter, as far as our brains are concerned, a new world – one conjured not just out of the author's words, but out of our own memories and desires – and it is our cognitive immersion in that world that gives reading its rich emotional force.
• In our day-to-day lives we are always trying to manipulate or otherwise act on our surroundings, whether it's by turning a car's steering wheel or frying an egg or clicking on a link at a website. But when we open a book our expectations and our attitudes change drastically. Because we understand that “we cannot or will not change the work of art by our actions', we are relieved of our desire to exert an influence over objects and people and hence are able to “disengage our cognitive systems for initiating actions”. that frees us to become absorbed in the imaginary world of the literary work. We read the author's words with “poetic faith”.
• It is only when we leave behind the incessant busyness of our lives in society that we open ourselves to literature's transformative emotional power.
• Although early versions of popular e-readers like the Kindle and the Nook did a pretty good job of replicating the tranquility of a simple page of text, it now seems likely that the page's calm, and the immersive reading it encourages, will be broken as a book's words are made to compete for a reader's attention with a welter of on-screen tools, messaging systems and other eye-catching diversions. The very form of a book seems fated to change as the written word shifts to a new means of production and distribution.
• Reading is more than visual decoding of alphabetic symbols. It is a state of mind, a dream of life, and a book, if it is going to be a true book, needs to be more than a container of words; it needs to be shield against busyness, a transport to elsewhere. Stevens put it simply:”The house was quiet because it had to be”.

  Dr. Maryanne Wolf

Dr.Maryanne Wolf and Dr. Mirit Barzillai ask a set of questions that probe in to the “future of reading”. With the onslaught of information overload, increasing distractions ( twitter, instant messaging, etc.) there are many possible ways the “the reading brain” is going to shape up in the future; “shape up” is the word used as we are never born to read or write anything. Unlike vision or language, reading has no generic program that unfolds to create an ideal form of itself. Rather, learning to read lies outside the original repertoire of the human brain's functions and requires a whole new circuit to be built afresh with each new reader. The brain changes itself by building a versatile “reading circuit” out of a rearrangement of its original structures, such as visual, conceptual and language areas. In the last essay , the authors ask a lot of questions in this context,“How will the next generation of kids build a 'reading circuit' ”. Will the twitters/IM make the brain circuitry devoid of 'deep-reading' and instead build something completely new that is equipped to handle the kind of info overload that we are seeing ?

• We may already have within our grasp the tools to conceptualize that the “new readers” of the twenty-first century need: a differently evolving reading circuit, one that connects the existing expert deep-reading skills to the evolving information-processing skills in order to be able to use the resources of the twenty-first century external platforms of knowledge wisely and well. The task is to figure out how to get there. We will need many minds to plot this progression.

 Takeaway:

The book is a nice collection of essays about “reading” by ten authors ; all of the essays have one common theme – Make time to deep-read in your busy schedule.

Speed Traders : Summary

In the introduction, the author tries to define HFT using the voices of leading HFT players. What are the characteristics of HFT? There is no agreement on a common definition of HFT, but there are some elements that one can at least attribute to HFT. They are

• Low latency trading.
• High turnover strategy.
• Trader goes home `flat' with no open position.

The goal of any HFT firm broadly is to have a set of uncorrrelated trading strategies that has statistically more winners than losers for all the trading positions of the day. The introduction mentions the fundamental market driven factors that gave rise to HFT:

• RegNMS.
• Order Handling Rules leading to ECNs and hence a proliferation of venues.
• Explosion of the universe of instruments; Human market maker is giving way to computerized market maker.

The author ends his intro clarifying terms such as program trading, quant trading, algo trading, automated trading, prop trading, stat arb, ultrahigh frequency trading. I found the explanation of these terms better explained in Barry Johnson's book ,`Algorithmic trading and DMA'.

The Emergence of High-Frequency Trading

This section lists down the significant events from 1969-2007 that were a precursor to todays' HFT

• 1969 Feb - Instinet Launched.
• 1971 Feb - NASDAQ starts electronic trading in 2500 OTC stocks.
• 1975 May - SEC bans fixed commission rates.
• 1976 May - NYSE introduces DOT.
• 1984 July - NYSE Approves SuperDOT.
• 1987 - Launch of ITG Posit.
• 1994 August - BNP purchases Cooper Neff Options trading firm.
• 1997 - Order Handling rules approved.
• 1998 - Launch of Attain, NexTrade, Strike Technologies, Bloomberg tradebook.
• 1999 April - Regulation ATS becomes effective.
• 1999 July - Goldman buys Hull for $530M.
• 2000 - BRUT/Strike merger, Archipelago and PSX electronic exchange launched.
• 2001- Launch of Liquidnet, Launch of Direct plus, Archipelago/REDI Book merger.
• 2001 April - Decimalization rule hits stock exchanges.
• 2002 - Launch of NASDAQ Super montage, Instinet - Island merger, Tradebot trades 100 million shares a day for the first time.
• 2004 - Nasdaq acquires BRUT, Launch of Pipeline.
• 2005 - RegNMS comes to force, NASDAQ acquires INET, NYSE acquires Archipelago, Knight acquires Attain and renames to DirectEdge ECN, Citi acquires NexTrade.
• 2006 - NYSE buys Euronext, CME-CBOT merger.
• 2007 - Chi-X Europe launched by Instinet. It becomes largest Multilateral trading facility in Europe. A graphic for the above events should have been there to make the content easier to read and recall.

A few visuals like the would have made the chapter more readable:

The Path to Growth

This chapter traces the developments through the period 2007-2010 June.

• 2007 April - General Atlantic invests $300M in GETCO.
• 2007 May - NASDAQ announces its plans for acquiring Nordic Stock exchange.
• 2007 July - Citi buys Automated Trading desk for $680M.
• 2007 Oct - NASDAQ buys Boston Stock exchange.
• 2007 Nov - MiFID goes live in Europe , NYSE announces its plans for a data center Mahwah, NJ.
• 2008 June - `Flow traders' receives funding.
• 2008 June - GS Sigma X and Credit Suisse ATS platform log in record volumes of 406M and 210M shares respectively.
• 2008 Nov - BATS becomes an exchange with its USP as speed with maker-taker model. As of today, BATS logs in 12% of the daily US equity trading volume.
• 2008 - Citadel's Tactical Trading fund earns $1.19B in 2009 with a team of 55 people!
• 2009 Mar - GSET launches Sigma-X in Hongkong. It soon becomes a big hit.
• 2009 June - NASDAQ introduces two types of flash orders , BATS introduces flash orders.

High Frequency Trading Goes Mainstream

This chapter traces the developments during 2009-2011 , the time frame when HFT entered the parlance of media, journalists, politicians, economists etc. It all began with a NYTimes article by Charles Duhigg on July 24, 2009 on HFT. That was the first time when HFT became known to wider audience and most importantly lead to false notion amongst people that flash orders were used to manipulate market.Click here  to read the full article that was published in NY Times.

Soon after this article got published, NASDAQ and BATS stopped offering flash orders. Amidst the falling economy, HFT was making money and this was met with harsh reactions from everyone. Meanwhile BATS and DirectEdge were eating in to the market share of NASDAQ and NYSE. So, to stay competitive in the game, NASDAQ launched INET in late 2009, an HFT enabled platform. NASDAQ also asked SEC to look in to various data centers that were mushrooming in the country. These data centers were heavily used by hedge funds and HFT shops. Meanwhile Chi-X was launched in Europe and Asia. Chi-X was very successful in both the venues. In Japan, it was launched under the name, `Arrow Head' and the latest numbers from Arrow Head - 100 Million shares a day - shows that it is a massive success. There is another market development mentioned here, i.e the ban on naked access. Naked access goes under various names, 'Sponsored Access', 'Unfiltered Access', etc. All those names boil down to one type of access - Allowing hedge funds to participate in the market using broker's market participant Id. The broker does not put any kind of controls to the orders and merely offers the platform to the clients to trade with broker's participant Id. According to Aite group, 38% of the equities volume is from naked access. Banning this was a big jolt to all the broker-dealers.

The chapter also features the success story of GETCO, an options trading firm started by 2 floor traders in 1999, that went on to become one of the biggest success stories from Chicago. There are some interesting facts about the GETCO's story like , it hired skilled video gamers from Illinois Institute of technology for its trading team. Well, to think of it, HFT is a kind of video game played with sophisticated technology. In Feb 2011, GETCO became a NYSE designated market maker along with the biggies like Goldman, Kellogg group, Bank Am and Barclays.The success of GETCO is also an indication of what a powerful financial and technology hotbed Chicago had become.

The Technology Race in HFT

Based on a few interviews with HFT shops, the author lists some basic requirements for a HFT shop to get going

• A reliable and sufficiently fast data input source.
• A robust and sufficiently fast market access.
• Low enough transaction, clearing and processing costs as well as sufficiently effective and efficient processing and verification capability.
• Strategy monitoring tools.

Well, if these things appear obvious, Kumiega and Ben Van Vliet of Illionois Institute of technology have developed a step-by-step methodology( using machine control theory) that addresses the needs of institutional trading and hedge fund industries for development, presentation and evaluation of high-speed trading and investment systems. They call it methodology and it says that the following stages of development should serve as a road map for running HFT shop.

• Before even firms even start building, they should ask themselves how they are going to monitor a successful algo.
• Build a customized database of historical data and purchase or build a tool for proper back testing of a strategy.
• Data cleaning methods are a key component. Writing your own scrub methods is better as it gives control over the data.Procure dirty data and see whether your system works.
• When algos go out of range,let's say 3-4 stdev , stop the strategy and fix it, just like a plant shuts off a machine if it goes 3 stdev from the normal.
• Backtesting is critical. However there are certain aspects you can't back test and be sure. Latency, Market data and timing are always going to be things that are difficult to check in backtesting.
• Fully define the functionalities and performance requirements of the trading/investment system and develop a system. Don't tradeoff robustness for the sake of speed.
• Put in place a reporting mechanism to capture relevant metrics for the HFT strategies.

The chapter ends with saying that, low latency is essential, but it is the strategy that needs to be robust. This is a very different view from what I heard from one of my friends last week. He is of the opinion that there are less than 10 strategies that are basically used in all the HFT shops and the speed is the ONLY thing matters. This book and other people mentioned in the book says `otherwise'. May be both matter equally!.

The Real Story Behind the “Flash Crash”

I found this chapter to be most interesting in the entire book as it goes behind the events that lead to NY Times article in July 2009 . It was in June 2009 that NASDAQ and BATS introduced flash orders to compete against DirectEdge. DirectEdge had given its participants a special order type called Indication of Interest , that helped DirectEdge add 10% to its existing market share. So, this tactic by NASDAQ was to make SEC ban such orders on the entire US market including CBOE that had roaring flash order business. SEC did not make regulatory changes as far as options are concerned. But it did make changes and subsequently flash orders were completely removed from the market.

One typically associates flash crash with some kind of HFT trading gone bad. But this chapter makes it very clear that HFT traders hate flash orders as the only entities that are profitable in a flash order are the exchanges and the guy putting the order. Infact reading this chapter makes one realize that flash crash actually goes on to show why HFT firms are needed in the first place becoz most of the HFT firms shut off their computers during the flash crash. Since there was no liquidity from HFT firms,volatility increased so much that Dow tanked 600 points and bounced back. Liquidity and Volatility have an inverse relationship and thus 'HFT firms providing liquidity lessen the volatility' is the argument provided `for' HFT case. When SEC report came out investigating flash crash, it was clear that a massive market sell order of $4 Billion E-Mini contract was the tipping point for the crash in the already nervous market. Why would a trader put in such a big order as a market order and not a limit order is surprising ? Somehow the press has been flogging HFT for the flash crash , when infact, there were firms who were providing liquidity even during the flash crash. Rumor is that there is a firm in Chicago which made $100M in one day. Now given that HFT profits are close to $2B-$3B, that's a huge amount of money.

Will there be more flash crash type of events in the future ? Certainly says the book and it says this has got nothing to do with HFT strategies.

Life after the “Flash Crash”

This section describes the various events post May 2010 flash crash

• May 12, 2010 - Talk of HFT tax as they were falsely blamed for flash crash
• May 14, 2010 - The first piece of investigation revealed that it was a traditional money manager who placed a $4B sell trade with out price limit, that triggered the crash
• May 24, 2010 - Talk of eliminating stub quotes
• May 26, 2010 - Goldman launches DMA in Latin American markets
• June 4, 2010 - SGX announces $250M “Reach Initiative”, the fastest platform in the world that has a 90 microsecond door to door speed. NASDAQ has a door to door metric of 177 micro seconds. This proposed platform at SGX is 100 times faster than the existing infra
• June 10, 2010 - SEC approves rules that require exchanges to stop trading in specific stocks if the price goes up by 10% or more in 5 min interval.
• June 11, 2010 - Revolving door practice is seen at GETCO. GETCO hires Elizabeth King, a personnel from SEC. This practice of hiring people from regulatory bodies is called 'Revolving door practice'
• June 22, 2010 - GETCO launches its dark pool
• July 21 2010 - DirectEdge launches two platforms, one for blackbox stat arb guys and second for passive agency order guys
• Aug 24 2010 - Chi-X receives a takeover enquiry from BATS. Chi-X becomes the second largest trading venue after London Stock exchange in Europe
• Sep 1, 2010 - BATS and NASDAQ stop flash orders. DirectEdge continues with Indication of Interest orders, a flash order in disguise. From the flash crash to the Waddell & Reed trade that started it all, from the discussion about stub quotes to circuit breakers, from the launch of new exchanges in the United States to GETCO's dark pool in Europe, by November 2010, high frequency trading was on the minds of everyone in the financial world.

The Future of HFT

This sections lists the events that have happened in 2010.

• Oct 21, 2010 - Chi-X Global announces that it will start operating in Australia by Mar 2011
• Oct 21, 2010 - SGX makes GETCO a trading member in its securities operations
• Oct 25, 2011 - SGX makes a $8.3 B takeover offer for the operator of Australian Bourse
• Nov 3, 2010 - SEC voted to bar brokers from granting HFT unfiltered access to an exchange, a move aimed at imposing safeguards meant to prevent bad trades
• Nov 8, 2010 - SEC decides to ban `stub quotes'
• Nov 11, 2010 - Chi-East, a joint venture between Chi-X Global and SGX was launched and this became the first pan-Asian independent dark pool to be backed by a regional exchange.Chi-East is widely regarded as a landmark in the development of Asian Equities
• Nov 24, 2010 - DirectEdge announces that it was changing the way it handles flash orders. It also indicated that it will slowly phase out flash orders
• Dec 1, 2010 - Equiduct Systems Ltd, announced that it traded more than 1 billion euros in Nov 2010 for the first time as ATS aimed at retail brokers gathered momentum. Equiduct is an MTF operated by Berlin bourse.
• Dec 6, 2010 - ASX that was selling itself to SGX said that the decision to merge with SGX was in the best interests of Australia's market micro structure development. Collectively the combined SGX - ASX entity would list companies worth $1.9T, fourth in Asia behind Tokyo, HongKong, and Shanghai.

The book is interspersed with interviews with 6 High frequency traders.

Meet the Speed Trader : John Netto

In this chapter, John Netto , a high frequency trader talks about his trading experiences and HFT in general. Here are some of the aspects he mentions,

• The creation of HFT strategy was the easy part. Issues such as how one will raise capital and how one will handle dealing with compliance, regulatory, reporting, and developing infrastructure are all questions not asked at first but which will become a critical part of the equation of profitability
• Low latency is very important for HFT but the degree of importance varies based on strategy.
• In terms of infra, I would say buy , than build so that you can focus on strategies to make money.
• Sophisticated systems can be thought of as the assistants to the traders, not replacements to the traders, because after all, strategy, structure and people are key to success.
• HFT is going to get bigger, stronger and more prevalent. More traditional investment managers will explore HFT in the times to come.
• Always have a few strategies that you think , make money on paper at least.
• I work out daily. Fitness and being active are a very important part of success . Trading requires you to be mentally alert and hence you have to be be fit.

Meet the Speed Trader : Aaron Lebovitz

Aaron Lebovitz spent two decades in the industry and then started his own firm, Infinium Capital management, a prop shop in 2003 and made it in to an exceptional force in the industry. Here are some of the aspects he mentions,

• Hook up with a firm that has a proven track record, and make sure to know how to write code.
• Integration of formal models in to trading decisions is the kicker you get in applying quant.
• Started off with pairs as his first high-freq strategy.
• His mantra is, `Developing new markets, increasing global market liquidity, and driving efficient price discovery'
• Infinium trades 23.5 hours 6 days a week. It did shut down its machines on the day of flash crash
• It is not all about speed. Having subtle or good strategies will get you alpha.
• The future of HFT is still very much in doubt, hinging on an uncertain regulatory environment.
• I don't foresee traditional investment managers shifting their focus to becoming market makers or developing statistical arbitrage strategies

Meet the Speed Trader : Peter van Kleef

Peter Van Kleef has spent the last 15 years running automated trading ops. Here are some of the aspects he mentions,

• It never really crossed my mind why someone would want to trade any other way than HFT. It is just so much more efficient
• Start writing your own trading strategies and see whether they make money on paper. Think about fancy infra later.
• Trying to apply solutions from other fields of science and research is also a start when exploring HFT.
• Whenever possible, I try to balance work with some sort of physical exercise. It is MUST for being a better trader.
• Low latency is important but is not the be-all and end-all.
• It is time for people to stop competing for speed and start producing alpha.
• A lot of the profits that are sustainable usually will come from the strategy and not necessarily from technology.
• Leverage is the key driver because returns are quite low.
• The overall profit largely depends only on the number of trades that can be done because the average profit is nearly certain. I don't know many other strategies that, if run well, consistently range about 50% per year and above. We are not in the business to rate ourselves; high-frequency is very profitable, if done right. That much is sure.
• Sophisticated retail investors are already running HFT strategies.
• HFT of the yesteryears is becoming a commodity as you can buy all the infra that is needed.
• Part of what will determine the future of the industry is what regulatory changes legislators and SEC will introduce.

Meet the Speed Trader : Adam Afshar

Adam Afshar's views on HFT are little different from the traders covered earlier. He is of the opinion that it is the low latency that is extremely critical. Here are some of the aspects he mentions,

• HFT is a method or a tool. It is not a strategy. HFT methods are applied to three types of strategies - Market Making, Stat arb and Algorithmic execution.
• My HFT shop uses models where there is absolutely no human intervention in the execution. I believe this should be the way, as human order execution does not allow you to collect data and you can't back test. If you use an algo for execution, one can collect all types of data and then improve it. Your reliance of specific individuals is also less
• High-speed data management is the linchpin of a success HFT shop.
• You can't avoid latency issue. Its a prerequisite for whatever you do in HFT world.
• The biggest revenue generating idea in my shop is , one that scours 110 million news items and takes positions intra-day
• HFT popularity will slowly fade away. It will soon become a commodity once tech makes the landscape flat.
• My advice to students is to take science and liberal arts subjects as they teach how to think. You can figure what you want to do later.

Meet the Speed Trader : Stuart Theakston

Stuart Theakston runs GLC, an HFT firm. Here are some of the aspects he mentions,

• High frequency traders simply replace specialist/jobbers in providing liquidity in a much more competitive framework.
• HFT traders are not getting a free lunch by installing heavy duty infra. Infact all they are doing is eat each other's lunches.
• An unfortunate way in which flash crash was handled by the exchanges : Some of the trades were canceled. This gives all the more a reason for HFT traders to be away from the market during crisis.
• I spend 70% of the time with quants, traders and programmers and 30% of the time watching market and doing research myself. I read a lot of academic papers and journals, etc.for they provide valuable source of ideas.
• Most of the people in the media, political circles, policy makers are clueless about the function of HFT. Volatility reduction demands Liquidity and HFT guys play a crucial role here. The policy makers have the potential to kill this entire HFT crowd if they don't understand issues properly.
• Order-anticipation and cross-venue arb are sensitive to latency, liquidity provision is less so.
• Prop trading firms can spend 99 cents on infra to make 1 dollar. Hedge funds can only spend 19 cents of the dollar given the structure. So these participants cannot compete in the same space
• HFT is nearing the bottom in terms of competition to be the fastest. The opportunities are outside equity asset classes like Forex, Exchange traded CDS and developed markets.
• Suppliers will spring up who try to sell HFT trading in a box to the masses. But the retail users won't make any money, at least on a properly risk-adjusted basis
• Most of the development in HFT have already happened in developed-market equity and equity derivative markets, and these are approaching maturity. These markets are likely to be the domain of increasingly specialist outfits. The low hanging fruit has been picked. The next wave is likely to be in emerging markets and esoteric ETFs. This is where people developing their careers in this space should focus.

Meet the Speed Trader : Manoj Narang

Manoj Narang started his firm in 1999 that was in to providing financial toolbox. In the last few years or so, he has transformed his firm in to a profitable HFT firm.Here are some of the aspects he mentions,

• Its the law of large numbers at play in HFT. If you have a strategy that works 55 percent of the times, you can't do a few trades. You have to do a ton of trading to translate that edge in to sustainable profits.
• We generally are buyers of hardware and builders of software and systems.
• We have never had a a losing week since we started in the business.
• A successful high frequency strategy will have a Sharpe ratio higher than 4 and a successful HFT operation that runs multiple strategies generally will have a double-digit Sharpe ratio. Such high Sharpe ration are unheard of in traditional investment places.
• Back of the envelope calculation shows that the profitability of the entire HFT industry is around $2B. Not much in relative terms. There are many other corners of the financial industry such as derivative trading, that generate hundreds of times this amount of profit in one year.
• The main use of outside capital in the world of HFT is to fund research and development, operations, not to actually trade the capital. Very little capital is required for trading purpose.
• Once you are an electronic market maker, it is skills that become important and not status or power or connections.
• Market making function in equities is decentralized. It doesn't mean everyone should be a market maker. Just because everyone needs food, doesn't mean everyone becomes a grocer. Similarly individuals should seek to be investors and not liquidity providers.
• Wall street has little to do with HFT.
• I think the main risk in the financial markets is that the capital is getting increasingly concentrated, which makes herd like behavior even more prevalent. The frequency with which bubbles inflate and exploder is getting more and more rapid as a result because massively capitalized investors jump from one asset class to another asset class, leaving devastation in their wake
• HFT does not connote systemic risk.The only systemic risks to the market are the ones posed by herd-like behavior.
• Tradeworkx started HFT in 2009 and this 50 member shop account for 3% of overall volume in SPDR ETF.
• One must keep in mind that modeling systemic correlations is VASTLY different from modeling structural correlations.
• How do I see our fund in five to ten years? I have no idea; five or ten years are an eternity when you are immersed in a world where microseconds matter.

Takeaway :

If you take all the events that have happened relating to HFT world and list them in a chronological order, spice it up with some HFT trader stories and viewpoints, what you get is, this book.

How to Write a Lot : Summary

The title is a marketing ploy so that someone struggling to write, be it  a graduate student. a researcher or a  professor , buys the book thinking that they will get some magical advice from it. The author himself admits this towards the end of the book that the title is not exactly what the book is all about. He says the actual title should have been, “How To Write More Productively During the Normal Work week with Less Anxiety and Guilt ?”. 

This book is targeted towards people in academia who need to publish their research / write research grants/ write books etc. This book isn’t about cranking out fluff, publishing second-rate material for the sake of amassing publications, or tuning a crisp article in to an long winding exposition. The author makes it clear that the book is not targeted towards the creative/ artistic kind of writers.  He jokingly remarks, “The subtlety of your Analysis of Variance(ANOVA) will not move readers to tears, although the tediousness of it might”

The suggestions by the author can be applied to any domain, any activity I guess. Personally I could relate the suggestions given in the book to programming. Its better to be a “disciplined/scheduled programmer” than a “binge programmer”.  Came across the word “Binge Writers”, in this book. It’s a nice word to describe those who wait for a break, for a 3 day weekend, for a vacation, for a sabbatical, etc. to write their inventory of thoughts. Sadly when a break does come by, they tend to accomplish less than what they had expected. I find this aspect particularly relevant to programming.

Though there are various aspects mentioned in the book, there are some core ideas around which the content is organized. They are as follows :

• There are some specious barriers to writing
• I can’t FIND time
• As though we run through our weekly/monthly planner and figure out that we have absolutely no time to spare. Crap…The author suggests that  you need ‘ALLOT’ time, not ‘FIND’ time. Make a schedule and stick to it
• Need more analysis to do 
• Binge writers tend to be binge readers and binge statisticians. meaning Binge writers more often than not tend to do not so great analysis too.
• Write what you have analyzed , whatever partial work you would have done, at frequent intervals.  It will only help you in doing better analysis/research.
• I need a new desk/new computer/new ___( fill in the blank with whatever fancy gadget that you need) / new printer / a study room to start my writing
• All of them are mere excuses and the person who gives the above reasons also knows that they are mere excuses.
• Let me wait for my inspiration. Will write when I feel like
• Another barrier which crumbles under close examination. There are tons of experiments that show that  “scheduled writing” always triumphs “writing whenever you feel like”.  Schedule automatically brings in creativity sooner or later
• Set specific goals, prioritize and track them religiously.As they say, anything that gets measured gets done.
• The author shows his own system of tracking where he uses a simple file to track the following
• Date
• Duration of writing
• Project worked on
• # of words written
• The utility of the above data is that when you sit down and see the metrics like average #  of words you have written over  day / month/ year , the number of days you managed to write, your whole outlook toward writing is improved. These simple stats will make you aware of the limitations / short sightedness in your planning + writing process.
• Writer’s block is an utterly crappy notion. There is nothing like that. It is a dispositional fallacy(A description of behavior can’t explain the described behavior). Its like saying you can’t write because you are not writing. Prolific writers follow their writing schedule regardless of whether they feel like writing.
• Start your own Agraphia ( pathological loss of the ability to write ) group
• The author mentions about an interesting development in North Carolina’s faculty group. They have started a group where they meet regularly to discuss specific writing challengers, to understand others’ ideas and insights in to writing etc. Well, it is basically a “weight watcher” community approach to writing I guess.

Stephen King, the prolific science fiction writer says (about your writing room )

“It needs only one thing, a door which you are willing to shut”

This book, in way says, the same thing with a tweak , “a DOOR which you are willing to SHUT & a TIME that you are willing to ALLOT each day.”

Pricing the Future : Summary

Any book that promises a journey spanning 300 years is bound to focus on events that / people who made the maximum impact for the development of option pricing formula. If one were to pause and think about the option pricing formula, one would come up with questions like

• Pretty naïve question to start with, Why is there a need for options, in the first place? How were the traded historically? Were they precursor to some other instruments?
• What factors drive option prices? Can one mathematically pin down a few factors, assuming others are constant?
• Option by default depends on the movement of the underlying security. So , how to mathematically describe the underlying security?
  • Should one form an equation for the price?
  • Should one form an equation for price increments?
  • Should one assume a discrete process or a continuous process, after all traded price series is a discrete time series?
• Given a specific process for the underlying security, How does one go about figuring out the price of option?

This book in a way traces all the developments leading to Black Scholes equation like the Brownian motion, Ito’s calculus, Kolmogorov forward and backward equations,etc. and leading up to the most important idea of option prices, “replication”. Each of these ideas are described chapter wise. Let me summarize briefly, the chapters in this book.

Flowers and Spices

The book starts off describing Tulip mania of 1630’s and the reason it talks about Tulip mania is this : It was the first instance when government, in order to come out of a crisis, converted forward contracts to options contracts. The chapter then talks about Dutch East India company that dealt in Spices. The history of this company is closely linked to the emergence of the first formal market for options. Dutch East India company (VOC) became a powerful company in Netherlands just a few years from its inception. The shares of the company became valuable and a futures market emerged. Subsequently to cater to various kinds of demands, options market emerged for VOC shares. VOC finally got bogged down in corporate politics, corruption and became bankrupt. The period of 1680s was also the time when there was a need for communicating to general public about the ways to trade and understand options. To clarify the various terms and mechanics of options, Joseph de la Vega wrote extensively in his book,“Confusion of Confusions”. De la Vega was the first to describe in writing the workings of the stock market, in particular the trading of options. His work is held at such a high esteem that, since the year 2000, the federation of European Securities exchanges sponsor an annual prize for “outstanding research paper related to the securities markets in Europe”.

In the Beginning

This chapter contains some important developments that happened in the financial markets between 1694 and 1885.The chapter starts off in 1694 with John Law advising French king on restoring the financial stability of the country. John Law started a firm in France that printed paper money. It was the first attempt in Europe to replace metal coins with paper money as legal tender. The bank, forerunner of France central bank, guaranteed that the notes would always be worth as much as the metal coins on which they were based. Law also convinced the king to grant him powers for a natural resource trading company so that he can bring in more revenues in to the country. Law's company became very popular and there was a mad scramble amongst people to buy shares of the company. Like any bubble, the printing machine idea flopped by 1720 and France was facing an imminent financial disaster again. However the taste of trading and speculation activity that Law gave to French citizens was still in full force. Unofficial trading of various other instruments increased. So, finally in 1724, the government decided to bring some order in to this situation and an official Bourse was created. The whole system ran well until the French revolution in 1789 after which chaos ensued. There were few more developments that lead to the reopening of Paris Bourse and this time everybody was allowed to trade. Again forwards were outside the regulation but it did not stop the increasing volumes in the instruments and soon became the hub of speculators. Also with the collapse of a big investment bank, France went in to a recession. Out of all these developments, there was one positive development,legalization of forward market in 1885.

From Rags to Riches

This chapter talks about the life of Jules Regnault. It is a classic rags to riches story. Why is the story relevant to the book or options pricing ? Well, Jules Regnault was the first person at least as per the book who deduced square root of time law. He not only tried proving it using math, but also used the law in the stock markets to retire rich. He started his work life at the age of 23 as a broker assistant. His living conditions were miserable but he managed to improve his living conditions by working hard in the broker's office. Regnault was the first person to try to understand the workings of the stock exchange in mathematical terms, and his explanations had all the trappings of a scientific theory. Regnault managed this with hardly any formal education system. After a full day's work at the Bourse, he would sit in a small room under the attic and do quant. Truly inspiring life. Remember this was in 1860s and he was investigating concepts such as random walks, role of information in stock markets, useful/harmful sides of speculation, insider information, factors that drive the valuation of an instrument, ways to calculate fair value of an instrument etc. An admirable quality about Jules Regnault's life is that he never shied away from applying things in the stock market. He used all his principles at the Bourse and retired at the age of 47 after making a ton of money. In one sense, Jules Regnault can be called the first quant who made a fortune in the market.

The Banker's Secretary

In any introductory book on options theory, you are likely to see payoff diagrams for various options and option strategies. This chapter talks about the first person to use these diagrams for communicating option positions, Henri Lefevre. Lefevre was a personal secretary to business tycoon James de Rothschild. Lefevre did not participate in speculation or trading activities but was a keen observer and educator of markets. He published various books and articles, thanks to the fact that he was a secretary to Rothschild and could influence the publishers. He made two main contributions to options theory. First contribution was his mental model of comparing economy and the flows of capital and goods with the human body and its organs. In Lefevre model of economy, stock exchange is like the heart that keeps blood moving through the veins, government is like a brain that thinks, combines and regulates the flow, Speculation is like the nervous system that provides the momentum that keeps commodities and capital in continuous motion. His second contribution was in 1873 and 1874 through his books. In those books he introduced a method of analysis that we are all familiar with, the pay off diagrams. The pay off diagrams for individual options might be very simple and a use of such a diagram to explain things could be a stretch. The real power of payoff diagrams comes in to play when you are analyzing a set of option positions. The final payoff diagram for a set of option positions, at once provides the price ranges where the entire position makes or loses money. Lefevre's method and his extensive publications in the form of books , papers, articles, etc. helped the common masses understand options and options based investing in a better way.

The Spurned Professor

Louis Bachelier

The study of financial markets began in earnest at the turn of twentieth century. The first underpinnings of a mathematical theory were developed in the doctoral thesis of a thirty-year-old French student , Louis Bachelier. In 1900 Bachelier defended his PhD thesis and the committee composed on Poincare, Appell and Boussinesq awarded the thesis as “somewhat better than okay”. This evaluation haunted Bachelier through out his life as he could not secure a faculty position without a “real distinction” on his PhD. One of the reasons for Bachelier's theory not getting attention was that it was incorrect. He analyzed price movements as a specific random walk ( an Arithmetic Brownian motion ) which allowed for stock prices to take negative values. So, the thesis that was probably the first step towards a mathematical treatment of financial markets lay dormant for a long time.

Botany, Physics and Chemistry

The first observation about the jiggle movement of particles was by a Dutch Scientist Jan Ingenhousz. However the credit goes to Robert Brown for the name. I find Robert Brown's life very inspiring. He made use of his free time and did all the observations and work after work hours. He never socialized or dined out. Basically he was a guy who kept to himself and did outstanding work. He observed that Brownian motion never ceases though he never knew the reason for Brownian motion. Remember this was 1827 and molecular/atomic theory was not established. Then came along Einstein , aptly named as the father of atomic theory. He hypothesized and predicted the Brownian motion behavior. He also formulated the square root law by clarifying that one must not analyze a single drunkard's walk but an ensemble of drunkard walks. In 1906 Marian Smoluchowski , another scientist made important contributions to understanding Brownian motion when he postulated that , the observed jittery motions are the displacements due to the unobservable zigzag movements that are a result of huge number of impacts. He concluded this after analyzing the speed of the particles at various resolutions. He saw that the velocity of the particles increased at higher resolutions. This made him conclude that whatever action that is seen in the microscope is basically the displacement. This book provides a fantastic analogy to Brownian motion which one will never forget after reading once. The books says it in a beautiful way:

Imagine the dance floor of a studio. Flashing strobe lights highlight the seemingly jerky movements of the dancers, Of course, even the coolest dancers do not disappear in to thin air between the flashes. They move contiguously through time and space, but their movements are hidden during the dark periods between flashes. In fact, there could be different dance steps that give rise to identical positions during the strobe flashes. This is the key point. Particles pushed around by the molecules in a liquid describe a continuous path, and the jerky motion we see under the microscope is just a sample of the entire movement.

In the next chapter, the book gives an account of Norbert Weiner's life who proved that even for the jerkiest observations, it is practically certain that continuous paths exists which give rise to them. Typically these historical developments give so much meaning to what one gets to read on a pure mathematical text on Brownian motion. Brownian motion is continuous. “Oh! ok! ” was my first reaction when I studied stochastic calculus. However books such as these give so much context to mathematical properties that learning math becomes that much more interesting.

Disco Dancers and Strobe Lights

It was in 1880 that physicist John William Strutt discovered random walk in a completely different context, super imposition of waves of equal amplitudes, equal periods but random phases ? He came to the conclusion that amplitude was proportional to the square root of number of vibrations( variant of square root rule). Thorvald Nicolai Thiele, another scientist also worked on the random walk , while developing concepts in computational techniques. He used Gauss method of least square to conclude that an ensemble of particles following random walk would have an average displacement proportional to the square root of number of steps. Some 80 years later, this was picked up by another scientist Kalman and today we know a ton of applications of Kalman filter. Karl Pearson, the famous statistician added in a bit with his article in “Nature” magazine. Despite all these efforts, Brownian motion was not on sound mathematical foundation. It was Norbert Weiner who formally defined the process, proved its existence in a 40 page paper in 1923. In 1948, Paul Levy, considered as the founding father of probability theory , defined and constructed another class of random walks called Levy processes. In the last 60 years, more and more scientists studied various classes of random walks such as reflecting random walks, loop-erased random walks, biased random walks, random walks with drifts etc. In 2006, the prestigious Fields Medal, the Nobel prize equivalent in mathematics, was awarded to a French mathematician, Wendelin Werner, for his work on random walks.

The Overlooked Thesis

Regnault had brought Statistics , Lefevre Geometry and Bachelier Calculus to understanding options. This chapter highlights some of the important elements of Bachelier's PhD thesis that shows how the thesis was so far removed from the traditional way of analyzing finance. The section mentions the following from the thesis

• The thesis referred to the efficient market hypothesis and formulates the mathematical equivalent of the same.
• It departs from the tradition of describing economic phenomenon verbally and uses mathematical equations to describe things.
• It showed that fluctuations of the stock price around true price follows Gaussian distribution.
• It showed two justifications for the square root of time law.
• It used Fourier methodology to derive heat equation in the context of price probabilities
• It showed that Options value depends on the security's volatility and time
• It used reflecting principle to reduce the complexity involving Brownian motion calculations.

One of the offshoots of Bachelier thesis was Chapman-Kolmogorov equation that ties in the probability distribution of a variable at the end of a discrete Markov chain to its intermediate probabilities. Books such as these should be made required reading for a student before he/she gets exposed to math relating to Brownian motion. If you learn square root of time law using math and with no prior exposure to the rich history behind the development, you might learnt it but not appreciate the beauty that lies behind it. Similarly you might prove that Brownian path is nowhere differentiable but you will feel the result in a completely different way after reading Theoder Svedberg experiments about calculation of velocity of particles. All said and done, I strongly believe that historical developments about concepts/formulas are very important, sometimes more important than merely knowing a few ways to derivation of Black-Scholes equation.

Another Pioneer

Developments in science tend to be non-linear and controversy laden. Examples of plagiarism accusations are rampant. In such a context, it is but obvious that option pricing formula also has , in its history , some people whom we can only speculate that, they knew about a way to value options much before anyone else. One such individual mentioned in this chapter is Vincenz Bronzin. His work was accidentally discovered by Swiss historian Wolfgang Hafner, who later sent it to Prof Zimmermann. Both concluded that Vincenz Bronzin work in early 1920s had all the necessary math and concepts relating to pricing an option and in fact Bronzin’s work ends up deriving the same formula as Bachelier's. The fact that his work was never popular/ recognized shows that historical development of any concept is tough to attribute to individuals. You never know that some person who never published stuff might have known the concept right through. Vincenz Bronzin work on option pricing was so advanced that some of the concepts looked similar to what the final Black-Scholes formula looked like, decades later.

Measuring the Immeasurable

I loved this section where various historical figures are mentioned in the development of probability and stochastic processes. Firstly, what has a stochastic process got to do with pricing of options. What's wrong with using / trying to use a deterministic process ? Well, the basic difference between a stochastic process and deterministic process is, in the latter case you can pin point the result and in the former case, you can only get a probability distribution for the result. So, all diffusion processes, arithmetic random walks and geometric random walks are all nothing but a way to summarize the particle movement and any computations on it will likely result in a probability distribution. Ok, let me get back to summarizing this chapter. This chapter talks about Kolmogorov, the father of modern probability. Picking on one of the Hilbert's 23 problems that were announced in 1900, Kolmogorov developed a full-fledged axiomatic theory of probability in 1933. In doing so, he heavily relied on the works of Henri Lebesgue, George Cantor and Bachelier. Henri Lebesgue is credited for his revolutionary method of integration called the Lebesgue Integration that is applicable to a wide range of functions and sets of points. Lebesgue benefited from Cantor's work on real line and introduced the concept of measure. Well , the development of Lebesgue integration is  in itself is a fantastic story and “The Calculus Gallery” covers it in a splendid way. Kolmogorov also derived Fokker Planck equation in his monograph, unaware that the PDE was developed in 1913 and 1914 by two physicists Adriaan Fokker and Max Planck to describe the time evolution of a variable's probability distribution. The variable could be a particle's position in a fluid or distribution of stock price. I liked this section because of a nice analogy that gives the difference between Chapman Kolmogorov and Fokker Planck equation. I love analogies as they are the first things that come to your mind than the raw equations. I paraphrase the author's words here

The Chapman-Kolmogorov equation gives the probability of jumping from A to Z by investigating the probabilities for two consecutive jumps, say from A to P and then from P to Z, or from A to Q followed by a jump from Q to Z, and so on. It is like driving from New York in any direction and ending up in San Francisco. What are the chances of that happening ? Well, you could drive via Dallas, Texas, Via Witchita, Kansas, via Rapid City, South Dakota or via any other route. The probability of ending up in San Francisco is then computed by adding the probabilities of all possible routes. This is what the Chapman-Kolmogorov equation does. The Fokker-Planck equation goes one step further. It develops the whistle stop tours in to a distribution of probabilities of ending up anywhere on the West Coast, be it San Francisco, Santa Barabara, Los Angeles, or Tijuana, Mexico

After formulating the PDE, Kolmogorov found that the solution to the PDE was Chapman-Kolmogorov equation. He then turned the situation around: starting with a parabolic PDE, he pondered on the question whether it had a solution. If the answer was yes, then the solution could be none other than the Chapman-Kolmogorov equation. Thus he proved that Chapman-Kolmogorov equation indeed existed and was not merely a figment of imagination. Further development was needed in this area as there were strict conditions that one had to impose on Kolmogorov's PDE describing Brownian motion so that it had a solution.

Accounting for Randomness

Kiyoshi Ito

This section talks about the contribution of Kiyoshi Ito. Since the Brownian motion is jagged and jittery at any resolution, one can't calculate the velocity of the particles, as Marian Smoluchowski concluded in 1905. So how does one go about analyzing any Brownian path if the usual Riemann and Lebesgue Calculus cannot be applied ? Karl Weierstrass was the first mathematician who came up with such a nowhere differentiable function. How does one work with such functions? There is no way to slice the paths and handle it in the usual way. Whatever slice you look at , there will be jaggedness. What's the way out? Here is where Ito comes in. In 1942 Ito published a paper that contained the mathematical machinery to handle such functions or paths. He used Taylor series expansion and found that the first three terms of the expansion were all that mattered for a stochastic process and thus ignored the rest to come up a forecast of probability distributions. Today Ito's calculus is synonymous with “framework for handling Stochastic Differential equations”. Ito lived for 93 years and contributed immensely to field of Stochastics. In his Kyoto-Prize address, he says that he devised stochastic differential equations, after painstaking , solitary endeavors. For those who think that Solitude drives people crazy and must be avoided, Ito is a classic example of the tremendous possibilities of Solitude in life.

The Sealed Envelope

Wolfgang Doblin

This section talks about Wolfgang Doblin, a young mathematician who commits suicide while serving in the army to avoid getting killed by Germans. Before his death, he sends a sealed envelope to Academy of Paris. There is a big history behind the sealed envelopes that one can read from this section. The sealed envelope was supposed to be opened in 2040 but thankfully it gets opened in 2000. The math that Doblin sent to the academy had a framework to deal with Stochastic PDEs in such a way that one could lessen the restrictions imposed on it. Mathematicians wonder that if Doblin had not served in the army and had not met the fatal outcome, option pricing would have developed much earlier. In any case, a reader gets a glimpse in to this young genius,thanks to this book. Doblin's life revolved around math and it served as a way to get him out of his gloomy and depressing environment. Though he only had the rare hour to focus entirely on math, usually during the night shifts hidden away in the telephone station that he was guarding and that provided some heat, his preoccupation with mathematical problems alleviated the dreariness and kept him from falling in to depression In one of his letter to his professor he writes,

Fortunately, my mathematical work helps me fight despair. As I am not interested in alcohol, I do not have the luxury of getting drunk like others.

The Utility of Logarithms

Paul Samuelson

This chapter talks about Paul Samuelson who relooked at Bachelier's thesis and improvised on it. Instead of considering plain vanilla Brownian motion, Samuelson insisted on geometric brownian motion because of couple of reasons

• Share values are always positive in GBM scenario
• Price jumps, in percent, are equally distributed
• Model is in accordance with human nature, as displayed by Weber and Fechner's psychology
• Data observed on stock exchanges fit GBM than Arithmetic Brownian motion

The chapter also mentions M.F.M.Osborne, a Physicist, who claimed in 1958, that log values and not the changes in stock prices are normally distributed. He was motivated to analyze the behavior after reading the works of psychologists Weber and Fechner. This observation is the same utility argument made by Daniel Bernoulli in 1713 while solving St.Petersburg Paradox.

The Nobelists -  The Three Musketeers

  

Fischer Black – Myron Scholes – Robert Merton

The next two chapters talk about Fischer Black, Myron Scholes and Robert Merton. MIT was one of the common denominators for all the three people. Fischer Black and Myron Scholes worked together on consulting projects for financial firms and thus were aware of markets and the places where quant techniques could be applied. They developed a good working rapport as Black's forte was theory and Scholes was good at implementation. Robert Merton had worked on pricing warrants along with his MIT professor Samuelson. Warrants were similar to Options but for some technical differences. Before the trio cracked the formula, there were quite a number of scholars who came close to cracking it. In 1961, a graduate student in economics, Case Sprenkle made some headway. He assumed GBM for prices and allowed drift. But his approach was faulty as he posited that investor's utility function would be revealed in prices. Also he ignored time value of money. In 1962, James Boness another student from Chicago also attempted and improvised on Sprenkle's model by incorporating time value of money. However there were some problems with his theory as he assumed, all stocks on which the options are traded are defined to be of the same risk class and all investors are indifferent to risk. So, all attempts of Thorp, Case Sprenkle, James Boness, Robert Merton and Samuelson had shortcomings. But they all carried seeds of the eventual solution.

Somewhere in 1968 Fischer Black tried formulating the option price as PDE but faced a tough time solving the PDE. So, he filed it in his cabinet and went on with his work. However in 1969 , while conversing with his sole associate of his company, Myron Scholes, Black got a chance to start the work again. The key idea that was used in cracking the formula was “Portfolio hedging and replication” . If the option is deep in the money, then delta or the hedge ratio is 1. This means you can perfectly hedge a long call option by shorting one share of the underlying contract. If the option is not deep in the money but near ATM, then the hedge ratio is definitely not 1. It must be less than 1. If you represent the hedge ratio as x, then the combination of call and hedge position is basically a risk free portfolio growing at risk free rate. With the unknown hedge ratio as a variable, Black and Scholes wrote an equation describing option pricing.

With the PDE in hand, Black-Scholes made an educated guess that turned out to be the solution for option pricing formula. The solution to Black-Scholes PDE surprisingly did not have stock return component at all. This was totally unexpected from the prevailing notion that option price should somehow contain the return of the underlying instrument as one of its components. But Black-Scholes proved it otherwise. Robert Merton arrived at the same solution using Ito's calculus. Initially Black had a lot of difficult publishing the paper in academic journals. It was only in 1973 that Black and Scholes managed to publish it under the title,”The Pricing of Options and Corporate Liabilities”, in Journal of Finance and Journal of Political Economy. According to a study undertaken in 2006, the paper garnered 2589 citations, making it the sixth most cited paper in all of economics. In the paper they acknowledged the fact that Merton had worked out a different approach and arrived at the same solution, thus kind of validating their final solution to option pricing.

The Higher They Climb ...

This section highlights the lives of the three musketeers before the Nobel Prize award in recognition to the work. There are some people who believe that Texas Instruments was solely responsible for the tremendous marketing of Black Scholes formula by incorporating it in the module in its calculator. Traders did not need to know anything about the formula except the inputs. So, they happily traded away the options at CBOE that is today one of the biggest derivative exchanges in the world. Black joins Goldman at the age of 46 , becomes a partner with in two years and dies at the age of 56 (1995) because of cancer. Scholes and Merton land up at LTCM a fund using quant stuff to manage money. The principals get their recognition in the form of a Nobel Prize in 1997.

...The Harder They Fall

This section talks about the fall of LTCM. The LTCM bust is basically a `leverage' lesson for all money managers.If one needs to read in detail about the demise of LTCM, the book by Roger Lowenstein is spot on. This section does not fit the flow of the book.

The Long Tail

The last chapter talks about the problems behind Black-Scholes formula such as assuming volatility constant, assuming GBM that has normal distribution at it's core. The books ends by saying

So, there are pitfalls and they can and do lead to spectacular failures like the LTCM crash, crisis of 2007, demise of Lehman Brothers. But to fault Black, Scholes and Merton and their equations for such aberrations would be utterly wrong. This would be like accusing Issac Newton and the laws of motion for fatal traffic accidents.

 

Takeaway

”What risk premium should be used for option pricing” was a stumbling block for developing the option pricing formula. The answers given by Black-Scholes-Merton surprised everyone : no risk premium at all. This book traces the historical developments leading to option pricing formula and in doing so, weaves an entertaining and a gripping narration of all the mathematical developments that were a precursor to Black-Scholes formula.

Books read in 2011

Tabulated the list of books read in 2011, for quick reference.

 

#	Title		Summary
1		Linear Algebra Done Right	Summary
2		In Code	Summary
3		Numerical Linear Algebra	Summary
4		The Cauchy-Schwarz Master Class	Summary
5		Lebesgue Measure and Integration	Summary
6		Inside the Black Box	Summary
7		Probability Essentials	Summary
8		Probability through Problems	Summary
9		The Lebesgue Stieltjes Integral	Summary
10		Music Room	Summary
11		Measure Integral and Probability	Summary
12		Poke the box	Summary
13		The War of Art	Summary
14		Do the work	Summary
15		Finite Markov Chains	Summary
16		The Housekeeper and the Professor	Summary
17		Probability Theory and its Applications ( Volume I )	Summary
18		Fifty Days of Solitude	Summary
19		R Cookbook	Summary
20		Accidental Genius	Summary
21		R Graphics	Summary
22		Pomodoro Technique	Summary
23		Measuring the world	Summary
24		The theory that would not die	Summary
25		Untitled	Summary
26		How an economy grows and Why it doesn’t ?	Summary
27		How an economy grows and Why it crashes ?	Summary
28		Moneyball	Summary
29		Doing Bayesian Data Analysis	Summary
30		Markov Chains	Summary
31		The Practicing Mind	Summary
32		We are all Weird	Summary
33		Practical .NET for Financial Markets	Summary
34		Introduction to Probability Simulation and Gibbs Sampling with R	Summary
35		Models Behaving Badly	Summary
36		Robust Portfolio Optimization and Management	Summary
37		R Inferno	Summary
38		Listening below the noise	Summary
39		Blah Blah Blah – What to do when words don’t work ?	Summary
40		Getting Good with Git	Summary
41		Brain rules	Summary
42		Boomerang	Summary
43		LaTeX	Summary
44		How I became a Quant	Summary
45		Moonwalking with Einstein	Summary
46		Algorithmic Trading & DMA	Summary
47		Weighing the Odds	Will summarize someday
48		Flatland	Will summarize someday
49		The Creative Habit	Will summarize someday
50		Statistical Inference	Will summarize someday
51		A First Look at Rigorous Probability Theory	Will summarize someday
52		The Art of R Programing	Summary

The Art of R Programming : Summary

This book is written by Normal Matloff , a professor who has worked both in the Computer science department and Statistics department at UCLA. Hence this book is markedly different from the books that are available on R. You get a nice blend of views about R. Also the author clearly states in his preface that this book is essentially a book for those “who want to develop software in R”.

If you want to write some adhoc code for doing some adhoc analysis, this book is definitely a stretch. However if you consider using R for doing your day to day work as well as doing research, then this book is an awesome reference. I have been programming in R for sometime and it is likely that I can remember ONLY specific points / libraries / packages that come up in my work. It is difficult to keep a lot of stuff in working memory. At the same time when you read such books , they help you recall some of the obvious things that you would have internalized but wouldn’t have cared to pause and think about them.

In this post, I will list down all such points(some of them are very basic things)  that I came across in this book, that I had internalized but never gave a second thought.

• a single function can take in variety of input classes(R is polymorphic).
• instances of S3 classes are lists with an attribute class.
• if class is a just a list with additional attribute, why do we need them ? Well, because genericFunctions in R take classes as input and invoke functions specific to that class.
• to do a google search in all your packages installed use help.search("multivariate").
• use sos package to get help. Obviously missing from the list of help resources mentioned in this book is www.stackoverflow.com
• there is no scalar in R.
• ˆwhen we assign something to x, x is basically a pointer to a data structure.
• try to use all() any() functions whenever possible.
• sapply gives out a matrix when the entry is a vector.
• NULL is a special R object with no mode.ˆ It is better to assign a variable NULL and then use it in a loop to grow the variable /vector.
• sign function is a damn useful function when used appropriately.
• : operator produces integers whereas c() produces double( remember this to do avoid so painful bugs in condition checking)
• names(x) <- NULL removes the name of the elements in a vector.
• sapply converts an input vector to an output matrix.
• we can ask the function to skip over NA values.
• understand the importance of NULL. If you assign a variable as NULL, then you can dynamically grow the vector in a loop. NULL is a special object with no mode.
• R uses column-major order.
• learnt about pixmap package that gives the grayscale of the image in a matrix format.
• apply will generally not speed up the code. It makes for a compact code, that is easier to read and modify
• when you subset a matrix, you get a vector, the original properties of matrix go missing. Hence use drop = FALSE , then the matrix nature is preserved in subsetting.
• component names in a list are called tags.
• names of the list can be abbreviated to whatever extent is possible with out causing ambiguity.
• list with single bracket means you are working on another list. list with double brackets means you are working with the elements of the list.
• you can remove a component from the list by setting it to NULL.
• use length function to get the size of the list.
• R chooses least common denominator for unlisting operations.
• There is an unname() function that can be used to remove names from a vector. I had never had  a chance to used this function till date.
• lapply() gives you back a list whereas sapply() gives a vector or a matrix.
• lists are heterogeneous counterparts to vector, the same way data frames are heterogeneous counterparts to matrix.
• there are usually three ways to access a data frame df$a , df[,1] and df[[1]].
• use drop = FALSE so that extracted elements have data.frame attribute.
• ˆnever had a chance to use complete.cases() function which basically removes all rows whichever has NA.
• learnt how to create dataframes dynamically in a loop using the assign function.
• remember lapply on a dataframe will sort each of the individual columns. That is disastrous as all the data gets mixed up.
• factors can be thought of vectors with a bit more information added like levels.
• ˆoutput of split is a list.
• ˆuse table argument to get cross tabs.
• ˆcut() function is used to generate factors from data.
• one can use ls() in different ways to print objects from various environments.
• ˆreferences to local variable actually go to the same memory location as the global one, until the value of the local changes. In that case, a new memory location is used.
• ˆget() is one of the most useful functions in R
• you can access the environment using the function parent.frame()
• ˆuse <<- to assign values to global variables.
• ˆeven though global assignment of variables is scorned by many , R uses a ton of global variable assignments in its internal code.
• sweep function is used to add a specific vector to all rows or columns.
• any subsetting function in vectors or matrices is nothing but a replacement function. Whenever you write classes of your own, it is always better to write replacement functions for the same. Basically it’s the same thing as operator functions for a class in C++.
• use anonymous functions wherever possible.
• closure consists of a function's arguments and body together with its environment at the time of the call.
• use assign function to manipulate non locals.
• R promotes encapsulation, allows inheritance , classes are polymorphic
• R has two types of classes S3 and S4. I have a good pneumonic to remember how S3 works. S3 is like a general manager in a company. Basically he does nothing . He merely delegates work. so a print
  function in R is like a general manager which uses dispatch function to call the relevant function in the input object and make it to do the work. These are called genericFunctions that have nothing internally
  but a dispatch feature.
• always remember that , by having an access to the object name, one might get an error by invoking objectname.print, objectname.summary as these objects might be different name spaces. So, you must prefix the object name by namespace and then invoke the generic function.
• getAnywhere is a function to get all the namespaces and objects for which a specific function is de need
• S3 classes are written by specifying a list and then assigning a class attribute to that list. That's about it. So , in one way this simplicity has a  flip side. Lot of errors might creep in. This is one of the reasons for the existence of S4 classes which have a richer structure
• If you want to specify inheritance, a simple vector of names can be assigned to class attribute.
• S4 classes are considerably richer than S3 classes
• S4 classes are defined using setClass function and functions are defined using setMethod
• John Chambers advocated S4 classes where google R style guide advocates S3 classes.
• If you are writing a general purpose code, then exists might be a very useful function.
• Try to use cat instead of print as the latter can print only one expression and its output is numbered, which may be a nuisance.
• came to know about snow package and Rdsm package that can be used for parallel computation in R.
• There are ton of functions for gleaning directories, files etc. Infact I revamped all  my chaotic iTunes collection using file and directory functions in R
• grep(), grepl(), sub(), gsub(),nchar(), paste() , sprintf(), substr(), strsplit(), regexpr(), grepexpr() are basic functions that are needed for text analysis.
• starting from R 2:10, there is a new function , debugonce() that can be used.Its very useful as you don't
  need to type debug and undebug always.
• you can put conditional breakpoints also browser(s>1).
• starting from R 2:10,you can use setBreakpoint( linename,linenumber) to directly place breakpoints
• trace(gy,browser) is another way of placing a browser() at the start of the function with out modifying
  the function.
• option(errors=recover) is very useful to trace back the error for the first time.
• need to check out debug package by Mark Bravington.
• StatET new version has debugging facility. All my R code is written on an older form of StatET version. Looking at the number of people hitting the support list of StatET with questions, I have an inertia to move to the new StatET version that promises debugging.
• If you are writing very important code /algo, it is better to keep options(warn=2) so that it instructs R to convert them to errors and makes the location of warnings easier to fin

For an experience R user, the last three chapters of the book cover very important topics like

• How to enhance performance of the code ?
• How to make code efficient ?
• How to make R talk to C++, Python and other languages ?
• How can one perform parallel computations in R ?

Takeaway:

This book is a valuable addition to the R literature and has something new to offer to any R programmer, be it a beginner or a seasoned programmer.

The theory that would not die

 

This is a talk by Sharon Bertsch McGrayne , the author of a fantastic book titled, “The theory that would not die”. 

My book summary is here . The video lecture @ CMU is worth watching and might motivate someone read the book.

 

Algorithmic Trading & DMA

This book by Barry Johnson clarifies a lot of terms that we get to hear in the context of trading using computers( program trading , DMA, algorithmic trading, high frequency trading, systematic trading, statistical arbitrage, quantitative trading ). Most of the times, the articles / papers / academic literature don’t make an attempt to clarify what these terms are. So, the reader is left to imagine whatever is convenient to him/her based on the context of the material.

I have tried writing a summary of this book but then realized that there is actually no point in doing so. I have learnt something new from every sub-section/sub-sub-section/sub-sub-sub-section, of this book that I feel a blog post can never do justice to my learning's from this book.

Even though the book is organized in 4 parts spanning 550 odd pages, the first three parts explain the heart of algorithmic trading. The last part of the book is just meant to give a flavor of strategies implemented by hedge funds that convert real time news to machine readable format and subsequently use it to generate buy and sell signals intra-day( more on the lines of twitter based hedge fund). So, leaving the last part of this book aside, the book has 11 excellent chapters in total  and these contain stuff that one probably learns/gets to know, only by working in a hedge fund that employs quant strategies OR working for a LONG time in an algo-trading outfit. There is hardly any mathematical prerequisite to read this book. The book would definitely give any reader excellent breadth and depth of knowledge about Algorithmic execution and Direct market access.  Once you read this book, my guess is, this book will give you crystal clear context , whenever you read any article/paper/ book on this subject.

Moonwalking with Einstein : Summary

“Moonwalking with Einstein” is a book recounting the experiences of US Memory Championship winner Joshua Foer, whose day job is NY Times journalist. The book is easy on eyes and can be read in a few hours time. One might shy away from books that deal with memory assuming they are of “self-help” type books . But that's not the case with this book. This book is a result of curious journalist who happens to cover National and International Memory championships for a couple of times and starts wondering about the participants. He starts pondering over a few questions:

• Passive questions like (When I say passive, any memory championship event spectator would have these questions)
• How do people memorize a long list of random numbers ?
• How do people memorize a deck of cards ?
• How does one memorize a poem ?
• Curiosity driven questions like
• Are the participants’ brains wired differently?
• Are they gifted people and average Joe can never perform such feats ?
• How does brain remember things? Memory competitions definitely check on working memory. So, Is long term memory involved at all in improving short term memory ?
• Does memorizing some arbitrary collection of random things be useful at all ?
• How do the mathletes practice for these memory events ? Is there something that one can learn from these people so that we can apply to anything that we want to become good at.

In fact all the people whom the author interviews say that their feats are achievable by anybody. All it takes is to learn the right technique and master it. This makes the author very curious and so begins his quest for learning about memory, memory techniques, deliberate practice, exploring the common myths surrounding memory etc. Well, the first question that any reader gets by reading a few pages in the book is : There are so many devices/gadgets/services that help us remember things. Basically our culture is an edifice built on external memories. So, Why develop internal memories ? Is there any point in developing them?

I have listed down a few points against the various sections of this book that I found interesting.

• There is something about mastering a specific field that breeds a better memory for the details of the field
• Experts process enormous information flowing through their senses in more sophisticated ways.
• Chunking is key to remembering and learning better. Chunking is key step in memorization techniques.
• At the root of chess master's skill is that he or she has a richer vocabulary of chunks to recognize
• Expertise is vast amounts of knowledge, pattern-based retrieval, and planning mechanisms acquired over many years of experience in the associated domain.
• You need a stockpile of images / places to form visual images of words. You sequence these images to remember better.
• We are exceptional at remembering visuals and pathetic at remembering words, numbers,etc. and this is key aspect driving all the memory techniques. The author describes his learning process with his mentor Ed. He learns about a technique called memory place, a method where you represent the content you want to remember using images and then populate the images in a familiar location.Humans are amazingly good at gobbling spatial information.So, instead of our memory that is non linear in nature, you are making an effort to linearize, put a pattern, put a sequence to these images. By laying down elaborate, engaging, vivid images in your mind, it more or less guarantees that your brain is going to end up storing a robust, dependable memory.
• We read, read and read, then we forget, forget and forget. Frankly most of us read a lot of stuff and forget almost instantly. So, why read at all ? is a valid question as so much of info is readily externalized. You want any fact about any thing, you get it on net. You want to keep track of things, there are to-do list reminders The rate of externalizing memories has become faster. Anything you would want to commit to memory ,there are devices/gadgets/tools to make the memorization unnecessary. So, is there going to me a need for memorization at all ? This question I guess has to be answered by each one of us individually. As for me, I think there is a value in memorization as it instantly helps you connect the new material you are reading with the stuff that is already in your brain.

In the entire book, I found two sections particularly interesting.

First one is where deliberate practice is tied to memorization techniques. The author hits a plateau in his preparation for the US memory championships and this leads him to explore  `deliberate practice', a concept popularized by Dr. K. Anders Ericsson.

What's deliberate practice ? Well, we all go thorough various stages of learning, right from Cognitive stage where we are intellectualizing the task, to Associative stage where we are concentrating less, making fewer errors , to Autonomous stage where we operate in autopilot mode. It’s fine if we are in autopilot mode for some mundane stuff but it is not ok when we are want to gain expertise in something.

The trick is to get out of autopilot mode as much as possible by practicing hard. Practicing hard need not be practicing for long hours. It means to design the work in such a way that it stretches you and there is a possibility of learning from failures that ensue. So, according to Ericsson,the way to achieve expertise in any field is to practice failing so that there is a constant feedback for your work. In one word, we need to be attentive and choose hard tasks to become damn good at something. The author finds that mathletes apply these same scientific principles for memorization too. They develop hypotheses about their limitations, conduct experiments and track them. This reminds me of Atul Gawande's book, “Better” , where he makes a strong case for tracking something, whatever be your interest.Tracking gives you an instant internal feedback. 

The other section of the book I liked was about a particular school in NY.The author visits a school in South Bronx, more specifically a class taught by a teacher who trains 43 kids in the class in developing memory. Whenever we hear memory development, there are only negative images that come in to our minds, like rote learning, creativity crushing etc. However this teacher at this school firmly believes that memory needs to taught at school level like any other skill. These memory techniques involve methods where kids are inculcated ways to visualize things. Every fact is turned in to an image. Well, do these techniques help ?The class is a case in point where all the 43 kids do extremely well in the school and in the entire region of South Bronx.

The author also manages to meet Tony Buzan. Though the author is extremely sarcastic about Buzan's goals, he does seem to agree one of Buzan’s core principle that, memory and intelligence go hand in hand.

There is a nice little fact mentioned in this book : “Inventio” is the root word for the words “Inventory” and “Invention”. So in one sense , until one does not have some inventory of thoughts, invention seems unlikely, as more often than not, invention is nothing but clever combination of seemingly unrelated ideas/facts.

Takeaway :

Memory is like a spider web that catches new info. The more it catches the bigger it grows. If you are curious about `memory' and its role in every day learning / awareness, you will find this book very interesting.

How I became a Quant : Summary

The purpose of this post is to summarize the various stories mentioned in this book. This write up is a result of a marathon writing on this weekend. Actually it took more time to write than read the entire book!. In the process of summarizing, I have tried listing down some of the statements/remarks/suggestions(under the heading, learning's) made by these quants that are worth pondering upon.

David Leinweber

David Leinweber graduates from MIT and Harvard with a bachelors and PhD respectively. He lands up at RAND corporation to work on AI. After working many years on classified stuff, he moves to LISP machines Inc. Once the market turned harsh towards hardware firms, David moved to Inference Corporation, a software-only AI firm where he starts working on AI applied to finance. LISP, with its garbage collection mechanism is not really suited to real time financial trading applications. So, David goes on to start his own firm, Integrated Analytics where AI is used for stock picking. He launches, Market Mind, an AI based tool for equity trading. Soon after this startup success, he joins First Quadrant, a Quantitative investment management firm where his group starts managing $6 Billion. After First Quadrant, he starts another firm Codexa which doesn't take off. Subsequently he lands up as a faculty at Berkley.

Learning's:

• Markets are not instantaneously and perfectly efficient. Insights, and the ability to execute them rapidly in ever-faster electronic markets, will continue to be rewarded.
• There are secretive firms that consistently show up on lists of the highest-volume traders reported by the exchanges. Founders of these firms show up on lists of billionaires. Are they just the lucky typing monkeys? Are they the investment equivalent of the lady in Jersey who won the lottery three times? Probably not. They make too many separate bets, thousands every day. And they do too well, too consistently. To attribute their success purely to chance strains credulity.
• Stocks are stories, Bonds are mathematics
• The PhD quants think the technicians are essentially examining tea leaves and goat entrails. The technicians think that PhDs are hopeless geeks who wouldn't know a good trade if they sat next to it on a bus.

Ronald N Kahn

This story is about the quant Ronald N . Kahn who is generally known amongst practitioners through his popular book, Active Portfolio Management. Kahn graduates from Harvard with a PhD and joins BARRA in 1970s , a firm that was revolutionizing the area of finance at that time. Kahn learns every thing about finance and quantitative applications to finance from BARRA. It was an ideal play ground for an academic as the team mostly comprised PhDs who walked away from their own field and came to finance. One of the leaders at the firm, Richard Grinold offers Kahn to sit through his course at Berkley. Soon, they start collaborating on many projects and one of their outputs is the book on active portfolio management. After doing tons of quant work at BARRA, Kahn joins BGI, following his boss steps. His experience at BGI managing money turned out to be an equally a rewarding experience. So, in one sense Ronald N. Kahn followed the ideal path for a quant, i.e , do research for a couple of years and apply quant techniques in managing money.

Learning's:

• Undergraduates learned from their professors. But the role of a graduate student is to teach their advisor new things. I wonder if such a thing should be applied to our usual work place reviews. Imagine that a yearly review is based on, not only the contributions that XYZ makes to the firm, but also on what his/her colleagues learn from him/her over the past one year. Imagine there was strong incentive system in firms to impart skills/knowledge to coworkers. Indeed such a place would be great place to work.
• Important communication should always be in writing and limited to one page
• Success in finance would require one to acquire much more intuition and insight into the basic problems faced by investors and to build up a toolkit for analyzing those problems.
• My approach to quant finance was very different from what a formal education in this area would involve. Basically, I tried to master specific areasâlike interest rate optionsâand then branch out and master new areas, through new projects. It was a patchwork quilt approach. I hoped I could build enough sufficiently large patches that they would eventually form a substantial quilt.
• Is quant revolution over ?. Certainly much of the relevant theoretical underpinnings are in place. At the same time, there exist even today significant issues in need of rigorous quantitative analysis. Present-day debates over liability-driven investing prove that fundamental issues remain unresolved among investment professionals Other areas worthy of further research include hedge funds and, more generally, optimal leverage and shorting. Between these remaining fundamental issues and the never-ending competition amongst active managers for out performance, the need for quants has never been greater. There is plenty more to do in this challenging, interesting, and rewarding field.

Gregg E. Berman

This is the story of a quant who doesn't want to be called a quant. Quant is sometimes used pejoratively on Wall Street as in, `He is ONLY a quant'. After graduating from Princeton with a PhD , Berman realized that he was not cut out for academics. His first few jobs were at hedge funds where he wrote automated trading strategies for commodity futures and trades those strategies.In the first few years he was so fascinated by coding that he programmed even on 2 hour train daily commute. The years at hedge fund helped him understand markets, psychology of trading, the type of statistics that can be used, etc\dots He traded futures for a couple of years and did not particularly enjoy it during a bear market phase. He felt that trading was an activity where luck played a significant role in the success and his academic bent of mind, where hard work and smartness fetch results, was not accepting the vagaries of trading. He remarks,

I also found that that I had less of a stomach for the huge ups-and-downs of trading than I had previously believed. This was a hard lesson for me to acknowledge. After all, many people claim that if they just had some money to start with they could use it to make a fortune. Well, I had millions in potential capital at my discretion and was asked to do just that. But I couldn't. I very much wanted to, but I just couldn't come up with any systems or strategies that would make fortune in the commodities markets. The more I wracked my brain, the more I realized I liked the detailed analytical parts of my job much more than the trading parts

The big change in the career came when he moved to RiskMetrics and did a lot of quant work at JPMorgan. At JPMorgan he donned quit a few hats and became a true quant in Wall Streetish sense. The big takeaway from Berman's story is that quant is not a job but an attitude that one must develop, to do well in finance, and more so in today's tech driven finance.

Evan Schulman

This is the story of a quant, Evan Schulman, whose belief in efficient markets got strengthened in his initial years of work at a trust company analyzing pension accounts. After developing quant tools that helped portfolio managers focus on their stock selection , he came to believe that quantitative skills can improve trade processes, if not generate alpha. He subsequently joined Keystone group at Boston to develop quant stuff like `Implementation shortfall 'models.

After his stint at Keystone, he moved to Batterymarch and that was a key step in his career. At Batterymarch, the entire firm was working hard to lower commissions and to keep the market impact of trades to a minimum.Schulman executed the first program trade while at Keystone Funds. Lot of people on the Street know Schulman because of this first program trade thing. He not only helped computerize the firm-front, middle and back office-but also introduced an innovative trading system at Batterymarch. Schulman let brokers access its orders via computer to trade. He left Batterymarch to start Lattice Trading, a firm that offered an electronic trading product that was a forerunner to today's algorithmic trading.

What was Lattice core idea ? It was multi-broker system to allow for the fact that institutions tended to use several brokers in the conduct of business. Lattice became very popular and was bought by State Street. Evan then left to start another firm , Upstream technologies in 1999.In his earlier firm, the work was mostly for the institutional side. At Upstream, he and his co-founder Mark Hoffman decided to apply the tools, discipline, quality control to individual accounts, even the small ones. Upstream uses quant stuff to work on individually managed accounts that dominate mutual funds. As Evan remarks

Optimization is ideal for the accounts of individuals.

Today, Upstream technologies is a success story on Wall Street.

Learning's:

• A quant who believes in efficient markets can still add tremendous value to improving the investment process itself, be it on the institutional side or the retail side.
• Compound returns equal a function of the average return less half the variance. Meaning, the longer the time horizon, the greater the difference between arithmetic return and geometric return on investment
• Not all problems can be solved by throwing more bodies in to a mix. The performance and productivity of quants varies a lot. Sometimes just one or two quants can bring in radical changes to the business. I am reminded of Dragan Skoko and his tremendous contribution to the success of Batterymarch

Leslie Rahl

Leslie Rahl graduates from MIT in 1972 with a degree in Electrical engineering and realizes that she has no aptitude for circuit boards, electricity, or the mechanical facets of electrical engineer's trade. In a totally random combination of events, she lands up in Citi and spends 19 years doing trading and risk management. Starting from a job of trading options for a prop book, she starts handling interest rate trading book and eventually swap business. In 1994, she leaves Citi to start her own company, Capital Markets Risk Advisors and since then has grown her company to a decent size on Wall Street.

Her belief is that models are useful for trading and not valuation. This comes from a quant who has traded for 19 odd years , so there could be a bias in her statement. What surprises me is that she starts a risk management company where you are typically valuing something under extreme events. So, how is that she has spent the last 15 years on something she doesn't believe. Anyways, when asked how she became a quant ? , she replies ,`I was born a quant' and adds that being a quant helped her not only with solving quantitative problems, but has taught her an analytic framework for problem solving that applied equally well to non quantitative problems.

Learning's:

• It is better to run your own business / work for a smaller company/ startup than working in a big firm . When you count everything, Positives outnumber the negatives .

Thomas C. Wilson

Thomas Wilson, a graduate from Berkley and Stanford is not sure whether he would consider himself a quant or not. He believes in `asking the right question' as being crucially important than any mathematical technique. You ask the right question and then go about looking/ creating/developing/copying the mathematical techniques to solve the question. He remarks

I put on the other end of the continuum individuals whose contributions were driven more by âtheâ question, or the intuitive interpretation of the observed economic and financial phenomena, rather than by the quantitative techniques that were used to represent their intuition. For individuals at this end of the spectrum, phrasing the question seemed more important than the techniques used to find the answer. In this camp, I put such individuals as Akerlof, Stiglitz, Lucas, Diamond, and Dybvig, individuals whom I judge to have contributed more through the intuition behind the question then the actual quantitative techniques they used.Who can argue that the intuition and insights behind Akerlof âs market for lemons outshadows the relative simplicity of the algebra usedf to prove the point?

He is extremely honest in confessing that he has lesser mathematical expertise than the quants who get cited in finance papers, media. In his career, he has stumbled on three questions and has learnt/developed mathematical tools to answer the questions. In this book, Thomas Wilson lists some of the most important questions he has answered in his career.

Market Risk Era ( Early 1990's)

• Is it possible to calculate VaR for nonlinear portfolios from local risk information \verb#(e.g., deltas and gammas)#, which are readily available on the trading floor?
• How many independent factors are practically required to capture the risk of a multicurrency fixed income trading book?
• What happens to the tails of our VaR calculations if we have only estimates of volatilities and correlations and not their exact values?

Credit Risk Era ( Late 1990's)

• Do equity markets alone provide sufficient information to characterize the contemporaneous correlations between average sector default rates and migrations?
• How useful are other factors such as the level of real economic behavior as measured by GDP, interest rates as a surrogate for debt coverage, or foreign exchange rates for sectors with international competitors in explaining sector defaults and their correlations?
• Do equity markets, which react instantaneously to new information, effectively describe the autocorrelated time series properties of credit cycles?
• How can retail portfolios and large corporate exposures be integrated into the same portfolio model consistently?
• How can default only portfolio models be expanded to a mark-to-market approach comparable to market VaR?

Strategy Debate ( Late 1990's to Today)

• What are the inherent theoretical biases, and is it possible to make adjustments to correct for the biases?
• What is the appropriate cost of capital, by line of business, if capital is allocated based on the standalone risk of each underlying business? And, what role does earnings volatility play in the valuation of a bank or insurance company?

Learning's:

• Build your intuition before building your model
• Trust your intuition
• Challenge your intuition
• Be as good at your communication as you are at the theory
• The model is always wring - but that doesn't make it useless.

Neil Chriss

This is the story of Neil Chriss, known for this book `Black Scholes and beyond'. Neill Chriss received his PhD from Chicago University. In the initial few pages of this story, he gives a few reasons for leaving academia, especially leaving pure math for industry. He compares pure academic mathematicians to Explorers whereas applied mathematicians/practitioners to Mountain Climbers. Explorers take risk with out any guarantee that it will pay off. Mountain Climbers have a fixed goal, i.e the top of a mountain and thus spend their time getting there. Explorers have a different personality than Mountain Climbers, though in one's life it is very difficult to sometimes clearly demarcate one from another. In the case of Neil Chriss, he clearly says that he did not have that `passionate,ever thinking,every imagining' mathematical interest.

After his PhD from Chicago, he moves to University of Toronto to join the math department. Here he develops an interest in derivatives and writes a paper on `Option Pricing formula with volume as variable'. This paper gives way to interactions with Goldman team, mainly Derman and Kani from Quantitative Strategies group. He then ends up doing a summer job at Goldman and then receives a post doc at Harvard. However Wall Street appears more exciting to him and he lthus eaves Harvard for a job at Morgan Stanley. At Morgan Stanley, he starts collaborating with leading quants like Robert Almgren,Peter Muller etc.He finally moves to GSAM( Goldman Sachs Asset Managment) and starts managing money for the firm. He also becomes NYU Courant's math fin program director. This unique place offers him to meet many more quants like Jim Gatheral, Steve Allen, Peter Fraenkel, Nassim Taleb etc.

He ends his story with a prediction about the usage of quantitative techniques in finance. This book was written before the crash of 2008. So , the tone of the author is overtly optimistic in one sense where he predicts that quantitative asset management will dramatically change asset management business. Well, one might take that statement with a pinch of salt , after the massive beating that most of hedge funds took , in the past few years. Over all , I found the story of Neil Chriss very interesting as he has managed to do things in academia as well as Wall Street, despite declaring himself as a mountain climber. Well, he seemed to have climbed a lot of mountains to make up for not taking up a explorer's job

Learning's:

• You have to deeply care about math to become a full time mathematician( Explorer)
• Came to know of this book titled,`The Glass bead game'. Should read it someday
• You have to enjoy working with computers to become a productive quant
• Neil Chriss learnt probability , stochastic processes from scratch after joining as a faculty in math. Its never too late to learn stuff in one's life .You just have to have the right attitude.
• Take regular feedback of your work from other people
• Historically mathematicians had more interaction with the real world. Hence there is nothing to be embarrassed about leaving academics to pursue work in industry.
• Having a strong background in technology , having had the experience of writing tens of thousands of lines of code, was an advantage. To do anything largescale in quantitative finance requires good technology. Everything from the storage and retrieval of trade data to the execution of trades increasingly relies on technology that is efficient and reliable. Knowing how to build systems that work well now and can be maintained and grown is an important asset because as a business grows in complexity, the technology that drives it has to grow with it. It is all too easy to build a small system that works well but find that as it grows it grows out of control in complexity and becomes difficult to manage.

Peter Carr

After pursuing an undergraduate degree in accounting and economics from University of Toronto, Peter Carr heads to UCLA for a Phd in Finance and then follows it up with a Postdoc at Princeton.He mentions that his choice of UCLA was particularly helpful as it gave him time to pick up math and make up for his lack of formal training in mathematics.Obviously what he means by lack of enough math skills, is a relative statement. His time at Princeton proved important. His meeting with Dilip Madan changes his course of life and he heads to Wall Street and spends a successful quant career at Morgan Stanley, Bank of America and Bloomberg.His remarks about quants and quant career in general , are particularly encouraging. He remarks

Learning's:

• Although many academics think otherwise, mathematical finance is in its infancy with lots of low-hanging fruit to savor. If you want to be a quant, my advice is to see it as a calling, not a job. The money is incidental. It comes and goes. A good idea never goes away and if you are lucky, you’ll have a few before we all fade away into obscurity.
• Academics use English words to describe mathematical concepts and that one should never confuse the everyday meaning with its mathematical definition.
• Read and work voraciously to make up for a lack of formal training. Sometimes you end up doing a better job than people who are trained formally.

Mark Anson

With a PhD from Columbia University , Mark Anson proceeds to law school and then begins his career. While practicing law he realizes that his first love was always direct application of quantitative techniques and gets back to doing a quant role. One of his remarks he makes in this book, that needs to be kept in mind I guess by any aspiring quant is,

Quantitative skills are like a foreign language.The more you use the skills, the more honed they become. Conversely,failure to apply the skills on a regular basis leads to a slow dissipation

It is often that we learn some skill set/techniques in school or in a specific project and , we then move on. It is like storing something in the working memory and deleting it. Unless specific skills are put to repeated use in one form or the other, quants run a risk of losing them.

Mark Anson sees the uncertainty in the markets as a good breeding ground for sophisticated, innovative quant solutions. In the US context, he mentions that quant skills need to applied to avert the pension crisis, that will be a certainty by 2050, if nothings done in a proactive way.

Learning's:

• Be less concerned with dazzling audiences with quantitative skills or technical jargon and more concerned with understanding the empirical relationship between economic variables
• Math is a tool to apply to financial markets. You have decide whether you have to make it blunt or refined, based on the context, problem, etc.
• Traditional fundamental research will be less important in the future because the goal will not be to find undervalued stocks or bonds, but instead, to focus on financial outcomes.
• Love to embrace uncertainty as that is the reason quants are employed in the first place.

Bjorn Flesaker

Bjorn Flaseker starts off his quant story with a reference to Garbage Can Model and its relevance in a sell-side derivatives quant group. He remarks

I have personally found the Garbage Can Model to provide a useful framework for understanding and interpreting the behavior of organizations where I have worked, including, but not limited to, the importance of solutions looking for a problem, the importance of decision making opportunities, and the significant degree of randomness in actual choices made.

After doing Bachelors at Norway, he is told by his faculty members that he should look to do a PhD.Subsequently Flaseker joins UCLA for a PhD in finance. A chance attendance of a seminar on HJM model gives him a problem to work on , as a part of PhD thesis and so begins his quant journey. In his 5 years at Berkley, he meets a lot of other quant researchers (including his future wife).He subsequently moves to Illinois at Urbana-Champaign to take up a faculty job. During his stay at Urbana-Champaign, he works on various derivative models, early exercise models etc. In 1992, he gets a chance to present his work at Merill and subsequently is offered a job. His quant career begins at Merill and takes off.

Learning's:

• Collaboration matters. People matter in quant work (unlike the popular perception that quant is some lonely creature working on some arcane models)
• Revisit basics from time to time
• Make a business impact with your model from time to time. It need not be some sophisticated out of the world model. A straight forward model that solves a business problem is better than complicated pie-in-the-sky model.
• You never know when a model will become useful . In this context, the binomial trees that Flaseker worked on in the university was immensely helpful in making the first contribution at Merill

Peter Jäckel

Peter Jackel graduates with a PhD from Oxford, then ends up in a post-doc role at which point of time , he realizes that he had enough of short-term employment phase( post-doc). Typically as the post-doc position nears, one writes proposals for research grants, or looks for another post-doc position in some other university. Peter Jackel instead opts out of this race and looks out for a job in the industry. At the time of applying for these jobs, he claims that he had no knowledge of stochastic calculus, derivatives ,market knowledge etc. After a few interviews at various places, he lands up a job at Nikko securities where he is taken in as a model validation quant. He gets lucky here as he gets to meet Bruno Dupire and thus his real learning begins. In a matter of weeks, he learns about finite differencing solvers, forward Kolmogorov equations, Monte Carlo simulation techniques, and general derivatives replication theory..all the techniques that is today offered as a part of 18 to 24 month MFE program!. It is truly amazing that he learnt all of this in just a few weeks time.

Then after 17 months, Nikki securities goes through a major restructuring and he loses his job. This turned out to be another fortunate random event, as lands up in NatWest where he gets to meet Riccardo Rebanato. He learns a ton of stuff from him and implements a whole lot of model validation code. It is during this phase of life that he starts thinking about a proposal from John Wiley and sons , that he write a book on MonteCarlo methods. Well, as it turns out he was not all that interested in it except that he wanted to dispel one myth that was prevalent amongst every one using monte carlo methods, i.e Sobol numbers. There was a opinion that was taken for granted that Sobol numbers are suitable only for low dimensional problems. However Peter Jackel found that he could easily scale the dimensionality problem by using suitable initial conditions . Sobol numbers were perfect for his back testing and this lead him to write a book on MonteCarlo, that is now the staple diet of any MFE program.

After another rejig of teams at NatWest, Peter Jackel is moved to front desk where he is a part of team which builds a structured product quant library from scratch. His 3.5 years of work of coding the library taught him many more things from an implementation perspective. This is kind of reminder to any quant who doesn't want to program. The real learning of quant fin comes from programming, is the biggest takeaway from this story.

Learning's:

• Build your models transparently and to the highest standards so that you don't have to be afraid to be open about them; and bear in mind that you don't own the models you build for the company you work for- in fact, as quants, we own nothing, not even the paper we write on. We deliver a service to our employers. The company may have to share part of what we develop with other teams in the company and is at liberty to do so. The company may be obliged to share what we are doing with regulators. Always be aware that what we develop and implement may be subject to scrutiny.
• Anyone who wants to make it as a quant to spend some time in a model validation group: you get a lot more time to dedicate to analysis and learning than in the front office where there is a constant and relentless pressure for delivery of directly usable analytics
• Collaboration matters a lot in quant work
• Get the right balance between a general and a bespoke approach for any individual project
• Every thing you do as a quant is cumulative. No piece of code that you write is trivial. It will be used in some future work of yours. This also means that you should not be sloppy in your coding . Doing proper documentation matters. Following the right design pattern matters.
• Coding something from ground up is a wonderful experience and one must never hesitate to take up such a job, despite the risks that it might entail.

Andrew Davidson

This story of Andrew Davidson, is filled with the theme, `I am not a quant' in almost every other sentence that he writes. Having an undergraduate education from Harvard and MBA from Chicago, Andrew Davidson's first job was Exxon. The bureaucracy got to him and he left the firm to work at Merill and finally started his own company Andrew Davidson\& Co. The story is about Andrew's skepticism about

models used in the mortgage industry. He cites specific instances where disregarding theory sometimes better. He points at a number of places where he does not possess the skills of a true quant and infact revels in the fact that it is not needed to survive and make a successful quant career

Learning's:

• How traders and other participants describe the operation of the market, and how it truly operates, are two different things.
• A clear conceptual picture often is better solution than a mathematical model that misses a key component

Andrew B. Weisman

After graduating from Columbia with a PhD, Andrew B Weisman's takes up a consulting job with Lehmann where he immediately realizes the big gap between theory and practice.

In this book he remarks

I had my first epiphany as a quant researcher; it consisted of the fundamental realization that most of the trading that occurs on an interbank trading desk takes place on a timescale that, for the most part, defies fundamental economic analysis.In truth, longer-term market convictions based on well-structured,defensible analysis are apt to get in the way. Such speculative activity is more akin to playing Pong.1 It is primarily a reflexive activity,with the trader operating under no imperative to distinguish Margaret Thatcher from Terry Hatcher. In many cases, doing so would have been a tall order.

There is one statement that he makes which left me little surprised. Andrew remarks

The market tends to place a significant premium on complexity. Basically, one should never explain a simple concept or procedure in simple terms, or it will be robbed of its marketability. Keep the curtain firmly in place and crank the devil out of your thunder machine

Why have a curtain? Why should one not be transparent about the model ? Somehow makes me feel little uncomfortable as the above statement asks you to remain secret about your model. As long as the strategy is making money, shouldn't it be good enough to market it to clients ?

Learning's:

• Mean Variance Optimization in its plain form is dangerous and should not be used for portfolio allocation
• VaR is an ordinal measure and not a cardinal measure. It is easy to confuse between the two and sleep in peace until one day when the portfolio evaporates. Much like the thanksgiving turkey mentioned by Taleb, which until the day of its death feels good about the fact that it is being fed healthy food on a regular basis.

Clifford S. Asness

At the outset, Clifford Asness , says `Luck' has played a significant role in his success. After graduating with a PhD from Chicago, he joins GSAM and starts

working on building portfolio models as well as manging money. With in a few years with 4 models, his group starts managing $7 billion. Humbly, he attributes the good chunk of initial success at GSAM to luck. Subsequently he leaves GSAM to start AQR with a $1 Billion. The first two months of his startup are some of the worst times for the fund. The billion dollar fund loses 60% and has a huge drawdown. Subsequently Asness reins in the risk exposure of the fund and makes it a sustainable venture during the troublesome times. Rest is history. Today AQR is a huge success story with assets close to $40 Billion. In this book, Asness dispels a lot of myths surrounding quants such as

• Quants are data miners
• Quants use blackboxes
• LTCM's blowup shows the limit of Quantitative investing
• Quants are driving with rear-view mirror

Learning's:

• Being in industry, managing money does not mean, you don't publish stuff. Keep publishing articles / opinions / models regularly.
• Never be afraid to call a bluff a bluff.

Stephen Kealhofer

This is the story of a quant and an entrepreneur. Like other stories mentioned in this book, there are random events that shape this quant's story. Stephen Kealhofer's story starts off with a decision to move to Berkley as a faculty where he starts working on `default probability' models. The second significant event that happens in his life is a consulting assignment that involved developing a statistical rating model that Oldrich Vasicek had developed. This project helped him in getting in a good understanding of the practical difficulties of implementing models.The third significant event happens when he is consulted for analyzing Junk bond cases. This assignment resulted in creating a good database with variables that could be used to estimate the empirical default probabilities. The fourth significant event was meeting Mac McQuown who became the co-founder of Stephen's startup , KMV corporation. The other co-founder was the legendary Vasicek himself.

There is sometimes a myth that quants are these nerdy people who work with formulas and not good business people. Vasicek is a brilliant counter case for such a notion. Vasicek's model is something that any quant learns in his 101 course. The fact that the team was instrumental in building a company like KMV that ultimately got sold off to Moody's speaks volumes about the success quants can have in entrepreneurial ventures. Nowadays with most PhDs directly going in to quantitative money management, we might see many more quentrepreneurs . I loved this story because it actually talks about the hardships a quant startup goes through in explaining the work they do to , to established institutions and adding practical value to the business.

The fact that the trio built a company based on a model is something that any quant can take inspiration from and remind himself from time to time, that ,”One has to integrate whatever quantish stuff that he/she does with the business problem”. Pie-in-the-sky work is good but one should also focus on doing quant that is relevant to the business.

Learning's:

• Always keep in mind that model /quant work has to deliver business improvement / business value. One cannot divorce the modeling aspects of the problem from the nature of data available , nor from the business context the model would be used
• Even if you are not completely certain about it, take a position and start exploring the position. It is similar to "having a map is better than no map" situation where having a map will motivate people to explore.
• Finance is a beautiful field. It touches everything, so it is a license to explore the world
• There is more data in finance than any field I can think of. One of the many great things about markets is that they generate large amounts of high-quality data about the actions of people.
• Finance is a beautiful field. It touches everything, so it is a license to explore the world
• The path from data analysis to the implementation of business ideas is incredibly fast. In biotechnology, it is measured in decades; in finance we do it in months or even weeks.
• If you can deduce something useful, you get paid.

Julian Shaw

I found the story of Julian Shaw as one of the best quant stories in the book.

After completing a PhD from University of Toronto, Julian Shaw loses interest in pure mathematics. He takes up job at Gordon capital as he needed a better job than teaching math or computer science in his university. He has a very unique experience at Gordon capital which was populated by market makers, option traders, basically risk takers. Nobody was interested in quant stuff like risk management. Julian Shaw applies his quant skills and brings to the attention of the management of a trader using wrong vol estimates to price and subsequently trade. His work is not so well received in the beginning. However when the trader loses his money, then people start taking notice of him. But Julian Shaw sadly remarks , `Risk management's role should come before the trade goes haywire and not after'. In any case, Gordon Capital blows up and he moves CBIC, another firm where nobody wanted to listen to quant saying the positions carry a huge risk. So, Shaw leaves CBIC very quickly and through a random head-hunter call, lands up at Barclays. He then writes about a few war stories at Barclay that make an interesting read.

Learning's:

• Finance is more than just pricing models and arbitrage, more than picking up dimes in front of steamrollers and sweeping up golden crumbs. Finance is concerned with the way people organize production and distribution, which is a prerequisite for human welfare.
• The hardest part of quantitative analysis is framing the problem in the first place.Many quants aren't good at this, partly because they don't think it's their job. They want the PDE or whatever to be served up to them on a plate. Solving PDEs is a useful skill, but there is a lot of good software to do that. There is no good software for identifying options embedded in deals defined by verbose legal documents
• Most of JP Morgan quants have ditched complex multivariate stuff to simple historical simulation to capture VaR. This makes it easy to communicate to the trader. However there is an entire debate about whether VaR is good absolute measure of risk at all. Consensus is that it is merely a relative measure
• A crude but transparent risk system is more effective in practice than a sophisticated but opaque system!
• The quantitative firepower applied to problems in finance is inversely proportional to the amount of money at stake.
• Many quants from the physical sciences are too lazy or arrogant to learn some econometrics
• Unlearn two attitudes to become a good quant.
  • The first attitude is that specialization is good. Bad quants pretend they are still physicists or mathematicians or whatever; they can't be bothered learning new tricks from other disciplines, and they never learn much finance. 'If all you've got is a hammer everything looks like a nail.'
  • The second attitude to unlearn is the assumption that the failure of people to understand what you are saying proves how smart you are. In academia, one only cares what the other specialists in one's own field think. By contrast, in a financial institution nothing you do counts for anything unless you can convince the (usually) intelligent but nonmathematical people in charge to do something different from what they would have done otherwise. Despite Poincare's famous maxim that one doesn't really understand a mathematical theorem until one can explain it to the first person one meets on the street, most mathematicians can't even explain their theorems to other mathematicians! And they are proud of it!
• Quantitative finance is a craft and a trade. It is barely engineering, and it is certainly not a science. Good quantitative finance can be summed up as the art of leaving things out, plus the art of selecting the right tools.
• The test of a model is its ability to hedge a real portfolio in the markets
• One can't be an expert at everything but one does need to understand the essence of a wide range of mathematical and statistical techniques. It is rare that one needs to invent completely new techniques; usually you can find a problem with the same mathematical form, often in an apparently unrelated discipline. The art of the quant is to find appropriate tools and bolt them together to create an effective solution to the problem at hand.
• If you want to be more than a technician, you will have to learn some finance; finance is not a corollary (trivial or otherwise) of any mathematical science.
• Interview Tip : think up one money-making idea and be prepared to defend it. This will generate a lot more interest at your interview than the Malliavin calculus!
• Go through Peter Carr's FAQ's in Option Pricing Theory to see where you stand as far as understanding options theory is concerned.
• if you want to make your mark, find a really big financial problem and create a quantitative framework for dealing with it. You will need some imagination.

Steve Allen

After graduating from Columbia and NYU Courant, Steve Allen joins Chase Manhattan bank to work in their operations research department.Till then he had never worked with computers and so he taught himself languages needed for the job.The first year at Chase was frustrating for his team as there was a distinct lack of management interest in their quantitative skills. Despite the obvious discontentment in the environment, Steve Allen stuck to his job and started utilizing the time effectively. He learnt to code, take up special classes in the evenings to learn bond mathematics , interact with traders to suggest tweaks in their existing formulas, etc. Basically he kept his mind active.

Soon, things started changing at Chase when a new management comes in that is more interested in quantitative techniques.Steve then gets to use MonteCarlo simulations etc not for their finance department but to improve their operations. After spending quite a number of years in Operation research, he moves on to building an analytic and modeling support for the firm's traders and this he does it for 10 years. He self-learns differential equations, option theory and a lot of math that he never had a chance to learn. These 10 years proved to be extremely useful in Steve's career as he developed modeling + reporting skills that became valuable to Chase traders. These 10 years of effort(it reminds me of Malcom Gladwell's observation of the time to become an outlier) helped him in launching a successful risk management career.

Steve Allen's attributes his success to the environment as he says that it was environment that changed where quantitative techniques became more prominent in the area of trading and risk management. After a successful 35 year stint in the industry, Steve Allen became the program director for MFE at NYU courant. Does he think the field has become matured and there is no opportunity to be creative ? . He quips

If you want to have a good career, you had better find a way to be creative; it's unlikely that your personality will be charming enough to induce an employer to pay you well for routine performance. But I add that creativity takes many forms- it consists not just of finding some new mathematical solution but also of discovering new ways to communicate results and build consensus.

Learning's:

• Keep learning . Steve Allen's attitude can be inferred from this remark `I enjoyed an advantage in having to struggle to come up the curve mathematically. It put me in a good position to figure out how to educate others who were finding the new concepts difficult.'
• Always have a vague sense of outcome so that you don't become too theoretical, to the point that you miss the fact that you are modeling in finance-a social science.
• Crunching numbers is a good thing and it is better to show results to traders than talk about some arcane mathematical concept that they would be least likely to be interested in.

Mark Kritzman

Mark Kritzman graduates with a degree in economics and joins investment advisory department that managed asset allocation of pension fund clients. Here he comes across mean variance optimization technique that appeals to him as a method for asset allocation. He then gets an MBA from NYU to sharpen his quant skills. Post this,he moves on to the investment department of AT&T. This was a great place for Mark as he could interact with top notch people in the research team of AT&T. He moves to Banker's trust and then eventually starts his own firm.As of the book's publication, his firm Windham Capital Management , managed $30B of assets. He cites three main things that helped him develop quant skills.

1. First is AT&T research team where his one-one conversations / discussions with Bill Sharpe, Harry Markowitz, Bob Merton, Steve Ross, Jack Treynor, and Fischer Black helped him immensely in developing financial intuition.
2. Starting up a new firm gave him a ton of learnings as he had develop many applications himself
3. Contributing editor for Financial Analyst Journal role helped him get better at communicating difficult concepts in easy terms , devoid of jargon and symbolism.

Mark gives a list of problems that he solved in his quant career. The readings gives an idea of the kind of problems that one need to tackle in the industry to make a mark.

Learning's:

• Mathematical proficiency will not be sufficient to guarantee a successful career as a quant, especially as these skills become more commoditized. The successful quant should combine mathematical proficiency with an appreciation for economic and financial theory, and he or she must know which questions are really important

 

Bruce I. Jacobs and Kenneth N. Levy

This is a unique story of 2 quants who are managing billion dollar funds. These funds have close to 83 accounts totaling $9B as of 2010. If you read this story it does not sound like as you are reading the story of businessmen. If you ingore the fact that they are managing money, the activities the two quants mention in the book could as well be of 2 social scientists in a research lab.

Bruce gets a PhD from Wharton and ends up as a faculty at Wharton. He meets Ken while Ken is doing his doctoral education at Wharton. During 1970s , almost everyone in academia believed in `Efficient market hypothesis' and so Bruce ends up choosing PhD topic on the same. Bruce gets a little let down by the ivory tower thinking at academia and joins industry. He joins Prudential where he meets Ken again and this random event changes both their lives forever. Neither of them believed in Efficient market hypothesis and soon they started Jacobs Levy Equity Management in 1986. At one point in time before the crash of 2008, they were managing $20B. That's quite an achievement. From 0 to $20B in 24 years with focus on quant research is amazing. Their investment approach was based on market complexity. Their focus on Kaizen approach to quant can be inferred from their remark

Once modeled, return-predictor relationships are likely to change over time. The world is constantly evolving, and old inefficiencies can disappear, giving way to new ones. Merely tilting a portfolio toward historical anomalies does not produce consistent performance. It takes ongoing research on new inefficiencies, new sources of data, and new statistical techniques to keep an investment approach in synch with evolving opportunities.

While most of the then investment managers were using historical data, these quants believed that history was merely one realization from many that could have happened. So, they build a simulator that takes in to account possible alternate histories, alternate correlations etc and start managing their fund. They call this process `disentangling'. Well you can call it i guess, sophisticated bootstrapping, in todays world. First three years of the firm, they earned hardly anything.They did research , wrote papers and got them published. So, this is a kind of startup where founders wrote papers instead of thinking about money, sales, marketing etc. Why did they do it ? Why couldn`t they do it in academia itself? I have no answers but I guess the fact that you are in a startup makes one that much more focused on what they are doing things,than let`s say a rather relaxed university setting.

Basically they spent their first three years doing multi-variate analysis. Its actually very praise worthy that they started off doing focused multivariate analysis before even thinking of the usual stuff in a startup. Wow. these guys rock...I mean , you come across some people, who just write some ill researched trading strategy/portfolio management strategy and then start a firm, and then morph in to sales people. They come with fancy brochures and lot of ENGLISH. So, for all such people, this story should be a telling reminder that `Doing homework matters in Quant' . You just can't start a firm in a garage , build a fancy site and start pitching. Might work for other areas but definitely not for a quant shop.

In this case, the quants got their initial clients based on their published research articled on portfolio disentangling. It was difficult to convince the normal investors to put in money and so they found few pension managers willing to take that risk. Today their average account size runs in to 100 million dollars. They did a lot of research and tried them out in the market. I guess thats a big advantage of being in a smaller organization where things are nimble. They were among the first pension funds that shorted assets and soon they did a ton of quant stuff on long-short strategies and incorporated the same in to their investment strategies. With $20b in AUM, they also took time out to write three books that are popular amongst investment practitioner community. After reading this story, I was amazed what these two quants have done.

Learning's:

• Spend time in understanding concepts / techniques clearly
• Always simulate and backtest. Historical data is merely one realization
• It is possible to do extensive research even when you are managing money. So, this notion that sometimes we carry as traders being sharp , quick and quants being slow is not that universally applicable. These two quants show that managing money and doing research can happily coexist.

 

Tanya Styblo Beder

After graduating from Yale, Tanya Syblo Beber takes up a job in the M&A department at First Boston. She tries returning to Yale for a PhD but that never materializes. Instead she gets involved at a swap trading desk . After a few years at the desk, she starts a risk management consulting start up and runs it for about 13 years. Somehow this story sounds like an detailed CV rather than giving any pointers/cues for aspiring quants.

Allan Malz

Allan Malz story can be summarized in one phrase,` Being at the time place at the right time'. He was at the Fed during the Golden Age of Supervision, at RiskMetrics during and after the technology bubble, and at a hedge fund during what may prove the heyday of hedge funds. So, as one would guess, he attributes his quantdom to the way he responded to random events than anything else.

Peter Muller

Peter Muller is a star quant on Wall Street, who striked it big with his team, `Process Driven Trading' at Morgan Stanley. After graduating with a math degree from Princeton, he heads to California and ends up at BARRA, a startup that was doing innovative modeling stuff headed by Barr Rosenburg , a Berkley Professor. He meets Richard Grinold at BARRA and gets the financial intuition needed for any math nerd to make sense of the markets. Peter Muller wants to do a PhD , while at BARRA. However after attending a few classes at Berkley decides that industry provided many more interesting problems and challenges than academia. Once BARRA became huge, the start up culture was lost and Peter Muller developed Poker skills. He gets an idea of starting a fund based on BARRA's models, but applied to trading. Luckily he finds himself at Morgan Stanley and gets the authority to start a prop trading desk. And they rest is history. Peter Muller's PDT is one of the most successful trading desks in the world and nobody knows how they operate to this day. In this book, though, Peter promises to write a book someday titled,`How I became a trader ?'

Andrew J. Sterge

Andrew J. Sterge graduates from Princeton with a Phd in mathematics and heads out for his first job at CoreStates Financial Corp in Philadelphia, where he learns the first and probably the most important lesson for trading,"Cut your losses". He then joins Cooper Neff, an options trading firm where he interacts with Richard Cooper and Roy Neff, whom he calls the most important people in shaping him as a quant. Over a period of time Cooper Neff morphs in to a HFT firm and this is the place where Andrew finds the work challenging , fun and thus ends up contributing massively to the P\&L of the book. Cooper Neff gets acquired by BNP and Andrew spends another successful career in quant trading at BNP. Currently Andrew works at AQR as VP. One of his statements in the book, captures his focus and drive in becoming a successful quant trader

A big part of becoming a quant was the hunger I had to improve my life,
both literally and figuratively

This quant story is interspersed with a ton of lessons and pointers to aspiring quants and traders. Here are a sample

• Quant is not a destination but a process
• You have got to be a voracious reader of market microstructure , trading strategies , everything that is related to markets. This will give you an immense edge in trading.
• Read `The Microstructure of Securities Markets' book
• Start working and start playing with tick data. You have got to store, clean , manage, view, analyze, visualize tick data. These skills are essential to become a good hft trader.Also, It is a prerequisite for building market microstructure models.
• You cannot make yourself a successful quant with out implementation. The greatest research in the world does no good if it cannot be implemented
• Effective implementation requires a lot of courage on the quant's part.A quant must take responsibility for his work and live the P&L day to day. Handing a model off to a trading desk to then go work on another project is a subtle way a quant tries to gain a free option: Take all the credit if the model works, blame it on traders if it does not.Working every day with traders, making their pain and the firm's pain your own, and putting yourself at risk for possible failure, is your only hope if you want to be really great in this business.
• You cannot just take your backtests to a bank and ask them to give you the money. Implementation takes a Herculean effort from many people.
• Becoming a quant is most of the times a team effort
• Prices are determined by agents acting in their singular self-interest. The only way anyone knows a fair, equilibrium price has been established is for prices to overreact in both directions. I have set out to profit by making this process more efficient. In the two areas I really know anything about, options pricing and market microstructure, I read virtually every book and article published on those subjects and then find things to exploit that are not written about. Lastly, no model or mathematical insight can make money on its own. One needs a team that collectively understands every aspect of running a trading business. Becoming a quant is not an individual sport.

 

John F Marshall

This is the story of a quant who is deeply influenced by his childhood incident of visiting commodity futures trading floor. So, he does his bachelors at Fordham and then works as a commodity trader for 8 years. Subsequently, he gets a PhD and lands up in St Johns University. The highlight of Marshall's career is founding IAFE ( International Association of Financial Engineers) . They say that one wrong doing can erase all the good things that one does. In this case, even though this story is presented in this book, John Marshall was accused of insider information scam by Securities and Exchange Commission. He was accused of passing inside information about a multibillion-dollar corporate takeover to a professor at Pace University. The Pace professor, Alan L. Tucker, made more than $1 million trading on the tips in 2007, according to the S.E.C. The Justice Department convicted John Marshall and put him behind bars. Such a horrendous ending to an illustrious quant career.

---

Finally the authors of this nice collection of quant war stories:

Richard R. Lindsey

Richard R. Lindsey is president and CEO of the Callcott Group, LLC, where he is responsible for directing research activities and advisory services. He is the Chairman of the International Association of Financial Engineers. Dr. Lindsey served as the Director of Market Regulation for the U.S. Securities and Exchange Commission (SEC) and as the Chief Economist of the SEC. He was a finance professor at the Yale School of Management before joining the SEC.He has a BS in Chemical Engineering from Illinois Institute of Technology, an MS in Chemical Engineering from Berkeley, an MBA from the University of Dallas, and a PhD in Finance from the University of California, Berkeley.

Barry Schachter

Barry Schachter is Director of Quantitative Resources at Moore Capital Management, where he is responsible for risk management, financial engineering and trade analysis. He is also a Fellow of the Program in Mathematics in Finance at the Courant Institute of New York University.Dr. Schachter received an MA and PhD in Economics from Cornell University, and a BS in Economics from Bentley College.

---

Here is the summary in pdf format - Download

 Takeaway:

This book gives a nice collection of quant stories , people who have transitioned from academia to Wall Street and have made a strong case for applying quantitative skills to finance, be it managing money/ risk/ transaction costs.

LaTeX – Summary

Nowadays I start most of my work with a .Rnw document that I convert later in to a LaTeX file. Since .Rnw keeps a running log of the trials / mistakes encountered in a project, it serves as a good summary document of the time spent on data analysis , model building , visualization , back testing etc. One big limitation that I was facing in preparing .Rnw documents was that I had forgotten most of the LaTeX syntax that I had used 4 years back. I somehow got by till date by knowing only a handful LaTeX commands. But my “memory” never fails to embarrass me!..and so had to go over LaTeX syntax.

So, after a brief search in the gigantic ocean of LaTeX literature, I picked up this book to get up to speed on the syntax. In the hindsight, this book was perfect to me as I could read the entire book in one sitting. The book is organized in to 13 chapters and has a cookbook kind of layout. The author explains a requirement, does something, shows why/how it works and moves on. In all, I picked up some 100 odd commands from this book that should more than suffice my requirement for now. I have listed down these commands in this post and I guess these are the most widely used commands in preparing basic LaTeX documents. Now that I have spent some time on this book , I hope to make my .Rnw files more readable so that it ultimately leads to better analysis.

Here is a list of some important commands (105 to be specific ) that can be used based on the type of task 

Formatting Words, Lines and Paragraphs

Designing Pages

Creating Lists

Creating Tables and Inserting Pictures

Cross Referencing

Listing Contents and References 

Using Fonts

Trivia – What is a sentence called if it contains all the alphabets (a-z) ? It is called pangram and in the context of fonts, pangrams are best to check how various fonts display them.

Developing Large Documents

In the above list, I have deliberately avoided mentioning math related commands as they constitute a universe in themselves. Hence the commands needed are completely dependent on the type of math that is being described in the document.The above list is also missing bibliography related commands.

Takeaway :

The book is written in the form of a cookbook. I think it covers ~ 90% of the situations that one comes across while preparing a TeX document. This book teaches basic commands of LaTeX and then suggests to pick and choose any package from the plethora of open source packages available, based one one’s requirement.

Boomerang : Summary

This book introduced me to a new term, 'disaster tourism' . People who visit places where financial disasters occur. Michael Lewis goes on one such tour to Iceland, Greece, Ireland and Germany. This book recounts his experiences of the visits and in turn gives a reader some idea of “What the hell is going on in these countries that is causing stock markets to gyrate wildly, investors to panic and making world leaders increasingly edgy?”

The book starts by talking about Kyle Bass, a successful hedge fund manager from Texas who minted money by shorting during the mortgage crisis. After tasting blood in the mortgage market, he then starts buying CDS on countries from Goldman and other Wall Street firms for as little as $1100 for an insurance of $1 Million Iceland bond. Basically a $700,000 return on investment of $1100 ( assuming that Iceland defaults and pays 70 percent of the bond value). When Michael Lewis goes to interview Kyle Bass during his preparation for the book,`The Big Short', he is amazed by the hedge fund manager's strategy. How does a guy who has never gone beyond US know things about Iceland and more importantly how is so sure that he is shorting the entire country ? This makes him curious enough to investigate first hand the situation in these countries. In a sense, this book is a sequel to the book 'The Big Short' and talks about the disasters happening in four countries, Iceland, Greece, Ireland and Germany.

Iceland

A country with 0.3 million population( think of a medium sized town in India) gets in to $140B debt, 8.5 times the GDP of the country. This happened so quickly between 2003 and 2007 that it is really unbelievable. How did a country whose primary occupation was fishing and smelting aluminum become a hedge fund ? Michael Lewis narrative mentions some key turn of events that changed the fate of the country. Firstly, the introduction of fishing quota and fish securitization( I had never heard of such a thing before..Last time I read about it was in some graphic novel that talked about the downfall of a hypothetical economy).This quota made fishing activity concentrated in a few hands and the other people were left exploring for alternatives. Some of them joined smelting aluminum work. The others headed for education and soon Iceland was minting PhDs. With so many PhDs who obviously did not want to fish nor spend time in aluminum factories,the situation was odd.They wanted to work on something new and sadly they latched on to the most hyped up job in US markets and world wide, “The Investment Banker”. They all wanted to do investment banking.

Carrying the traits of fisherman, i.e aggressive risk taking to banking to finance was a disaster. The most profitable trade was the carry trade and every Tom Dick and Harry became traders. Most of these I-bankers created artificial inflation by traded assets amongst themselves. This had to end as all the price inflation and growth was actually a result of a few people who managed to borrow insane amounts of short term foreign capital and infused it in to the country. All the signs of prosperity were bogus. In 2007 Icelanders owned 50 times more foreign assets than in 2002. The emperor was finally naked in Oct 2008 , when the country was declared bankrupt. After the subprime crisis all the hedge funds and FIIs who invested in Iceland to get eye popping returns actually got their eyes popped literally and withdrew all their money. Where is Iceland headed ? God knows. May be they will go back to fishing!

Greece

A country with 11 million population( think of one of the Indian cities) gets in to $1.2 T debt. That's right. It is a T with a capital T…We are talking about Trillions here. Basically it is equivalent to GDP of India in 2008. If you read about the bail out packages that are being doled out, they are paltry compared to the mess Greece is in . So, what made such a small country in to debt ridden country ? Michael Lewis meets with accountants, lawyers , monks , politicians and weaves an interesting narrative exploring the reasons for the current situation. He says the government is the culprit and quotes a few numbers that are insane. For example National Rail Road earns 100M euros and has 800M euros as expenses.Govt employed personnel earn thrice that of a private sector employee. The author's meeting with accountants and lawyers convinces him that the extent of corruption and malpractices that are practiced by EVERYONE in the country can fill libraries. Tax evasion was rampant and the politicians cooked up numbers to gain in to entry in Euro zone. All numbers quoted by the Govt were fraud numbers. The party goes on for years.

Finally the music stops in Oct 2009 when the then prime minister Kostas Karamanis gets involved in a scandal and the nation goes to polls. The scandal in itself an alarming story that involves Vatopaidi Monastery in Greece. Amazing what simple living monks can do to a country. In a matter of few years these so called simpletons become the real estate emperors in Greece. The political changes in Oct 2009 brought George Papandreou to power. He immediately realized there was nothing in the Govt coffers to spend and everything was a fraud. He had no choice but to come clean about the govt. In the recent past he has taken quite a few measures like job cuts, compensation reductions, etc. applicable to the Govt employees. Obviously this has resulted in massive unrest. From reading this story it is evident that is not a question of whether Greece will default, it is only a question of when will it do so.

Ireland

This is the story of a country which went from a budget surplus in 2007 to Budget Deficit of 32% of GDP, from an unemployment rate of 4% in 2006 to 14% in 2010. Real estate bubble meant $106B dollars of losses for the three biggest banks in Ireland. As recently as 1980s the country was one of the poorest countries in the world and it became one of the richest countries by 2007 . How did this miracle happen ? Michael Lewis answers this question and in turn gets an answer to his question, `What the hell is wrong with Ireland' Till 2007 if you had asked any academic, he would give amazing explanations for turn around of Ireland which were good as sound bytes in the media but were actually shallow arguments. So, How did Ireland become rich? Well, as the author finds out from the eyes of a skeptical  Dublin University professor , Morgan Kelly, that there was no miracle in the first place. People just went berserk. 20% of the workforce was employed in building houses. 25% of the GDP came from construction industry. Basically the Irelanders followed this philosophy,

“We are going to get rich by building houses for each other”.

Alas! the housing bubble crashed and so too the banks that lent money in pursuit of higher growth rates. They were a few people who alarmed everybody about it, for example a research analyst in Morgan Stanley wrote a scathing report on Ireland Banks. However since the banks were clients of Morgan Stanley, the report was thoroughly massaged, or in the other words fabricated to death so that the fee keeps coming from these banks. It was elaborate Ponzi Scheme played by banks,government at the cost of Irelanders. As of today, the situation is : the government that has bailed out the banks , are living on life line , i.e , the short term loans from European Central Bank. When the lifeline stops, Ireland will collapse ? Since the time most of Poles( People emigrated from Poland in large numbers to Ireland ) have left Ireland, people have lost faith and are silently witnessing the failure of the government, Where will growth come from ? After spending time with academics, politicians, lawyers, banker, the author seems to be saying, “Nobody Knows'”

Germany

Germany had no real estate bubble in the country. Prices were very much in line. There were no crappy domestic loans. So, why did Germany get affected by the subprime crisis ? The author weaves a nice spin on the German culture saying that Germans are a bunch of people who are obsessed about combination of clean & dirty , i.e clean exterior-dirty interior, clean form-dirty content. They are actually mirror images of Greece, Iceland, Ireland stories. Other countries used foreign money to fuel various forms of insanity. The Germans, through their bankers , used their own money to enable foreigners to behave insanely. They were on buy side of all the crappy subprime mortgages issued on wall street by Goldman, Morgan Stanley etc. Though they kept their external image clean, their internals were crappy. They bought bonds like crazy always trusting the ratings that were assigned to the mortgages.

With the bubble bursting ,their banks have taken a beating. However there is a crucial difference between the cultures here. Unlike US banking CEOs who are handed hefty bonus despite the crisis, some of the German CEOs were imprisoned. They were looked down upon the society. All said and done, Germany has a crucial role to play in Europe's future mainly because of its relationship with ECB. If the struggling European Nations are downgraded to Junk, then there will be a serious problem with ECB and there is a chance that ECB might itself default. Can Germany suck up all the losses of its neighbors ? Economically the best solution but Politically it won't fly. So where does this lead to ? Will there be Euro Currency at all, if all the countries fall like nine pins? It's a very uncertain future in Europe and I guess these stories mentioned in the book makes any reader shudder at the question ,”What's future for European Finance ?” It looks like it just a matter of time when things will apart ?

California

Michael Lewis focuses on California to bring out the systemic issue in various cultures that is causing problems. Even though he talks about various countries and their financial problems, he is pointing towards a larger malaise afflicting various countries, i.e the problem of failing to self regulate.

The situation facing various countries and California is summarized by the author using a nice analogy.

Entire countries were told,”The lights are out. You can do whatever you want and no will ever know.” What they wanted to with the money in the dark varied. Americans wanted to own homes far larger than they can afford.Islanders wanted to stop fishing and become investment bankers. Germans wanted to be more German, the Irish wanted to stop being Irish. Greeks wanted to turn their government in to a stuffed animal with fantastic sums so that the Govt can give it to as many citizens as possible.

In all the cases, there is a pattern here. Self regulation is thrown to the winds and in the longer term every one suffers. The story about California was new to me as I had never known that Vallejo, a city in California has gone bankrupt.Based on what the situation is right now, it is predicted that many more cities are going to be bankrupt. The most affected state as such is supposed to be California. The author starts off describing the situation in California by narrating an incident relating to Meredith Whitney where she predicts that municipal markets are going to be next victim of the subprime crisis. Indeed as the situation unfolds, as Vallejo's bankruptcy makes it clear that there is a systemic problem lurking here and if things don't change, it is probably only a matter of time before there would be many more cities going bankrupt.

Takeaway :

The book gives cultural, economic, political reasons for the bleak situation in Iceland, Greece, Ireland, Germany and California state . Its a real page turner and probably gives a better picture of Europe's situation than what one can infer from various confusing articles in the media.

Brain Rules

Books such as ”Brain Rules” are basically meant to be a bridge to understanding some very basic things. A layperson like me does not know much about the brain. I have completely forgotten whatever I have learnt in high school about the anatomy of brain . So, why read this book ? In fact why should anyone read this book? Most of our survival in modern day world depends on our brains. So, a little knowledge about it might help us in performing better and probably living better too. The author states in his preface that the rules that he talks about in the book are a collection of ideas that he wants others to explore in the field of education, business, entertainment, marketing and in general , every aspect of our lives.

My intention to read this book was mainly to understand some very basic aspects like

• After an early morning workout, why am I more alert while reading something ? Why am I able to grasp better?
• How do I remember stuff that I read ? I mean there is so much stuff that's happening around? How do I read / understand / recall them in a better way ?
• I am pathetic in remembering stuff. Is there something called short term memory and long term memory ? If so, how does one go about improving long term memory ?
• Why are pictures based learning emphasized heavily instead of word based learning everywhere? Yes, We remember pictures better, but Why ?
• Once we hear an song, even a 5 sec beat of the some part of a song, we instantly recollect the lyrics/the song and probably different memories associated with the song ? Why doesn't the same happen with passwords ? I can't get to remember passwords on websites. If by mistake, I delete the browser form history + cache+ cookies, then in all probability I have to use `forgot my password' link to retrieve all the passwords ? Is there some different kind of memory that is associated with remembering passwords, as compared to sounds / visuals etc. .?
• Why do I code better after playing an instrument ?

Obviously I don't have the time nor the inclination to become an expert in understanding above aspects. However I felt I needed a little bit of understanding. Had read a reference to this book somewhere a few years ago, i think it was related to some behavioral economics article where the article was talking about "Blind Spots".Recently I came upon quite a few references to this book. So, I guessed the book might have something to say and hence picked up this book with a firm resolution that I won`t be spending more than an hour or so.However I ended up spending more than that, thanks to the easy flow of words and ideas in the book. The book is cleverly written, cleverly because, books such as these can be terribly boring. But Dr. John Medina does a wonderful job of using analogies, simple language to show the workings of the brain. I got some kind of answers to my questions. In fact the book explores lot more aspects.

I guess this book will be useful to anyone who wants to understand a little bit about how brain functions, and what can one do in one's daily life , so that one learns and remembers stuff better.

Getting Good with Git : Summary

This book helps one “get up to speed“ with git. With github being the goto repository-master for most of the R projects out there, it has become imperative for any R programmer to have a decent knowledge of git and the know about ways to interact with github. As of today github has 1.1 million programmers hosting close to 3.2 Million repositories. This has surpassed sourceforge long back. So, an R programmer cannot be ignorant of this distributed version control system. Personally I found it a stretch to have a version control system. But once I started using it for a few projects along with ProjectTemplate directory structure , I found that it helped me in carrying out analysis in a better way. So, how do you go about using it? Well, there are many plug-ins to get going on git. Before you start using the plugin, you need to get the vocabulary of git.

There are a lot of terms that might be overwhelming to begin with for a newbie. Here is a sample
tag, stage, tracked, untracked, branch, working tree, master, head, treeish, SHA-1,carrot parent, tilde parent,patch,stash,diff,clone,push, rebase

Even if you use some plug-in like Egit( that I use), you got to understand the basics of git and terminology that goes with git. Git is supremely elegant for use but at the same time, understanding the internals is takes some time. My objective was mainly to use effectively git and hence this book was kind of ideal to begin with. The book starts off by giving some basic principles of distributed version control system( git is an example of DVCS). One of the main use of Git for me is that I am tired of maintaining v_1, v_2 files and my working directory has become very huge. Git forces me to think in versioning and thus relieves me of maintaining the versions manually. Git works based on snapshots , meaning you have the entire repository on your local machine. You no longer check in and check out files. This is a crucial difference between git and other systems. Here’s some history on Git. Git was developed by Linus Torvalds after relationship broke down with Linux community and BitKeeper, a proprietary DCVS. The purpose of developing was to create an open source DCVS with the following goals.

• Simple design
• Strong support for non-linear development (thousands of parallel branches)
• Fully distributed
• Able to handle large projects like the Linux kernel efficiently (speed and data size)

The first version was released in 2005 and since then, in the last 7 odd years, its popularity is growing wide and far.

So, what are the advantages of using Git

• Git is fast ( you can do everything from a simple command line)
• Easy to learn( I don’t know about this as it depends on what you want to learn about git!)
• Git offers a staging area ( this is immensely useful feature if you have gone through other versioning systems where you can a file is checked in or checked out by a user)
• GitHub is available for sharing – I guess this is one of main reasons for its growing popularity.

The book then gives a quick tour of commands for DOS and LINUX. It then dives in to the bare minimum stuff that you need to know about git. They are the following

• git init
• git add
• git commit
• git status
• git diff : shows the difference between staged files and rest of files
• git log
• git diff SHA-1-a SHA-1-b
• git describe
• git tag
• gitk
• gitbranch

The book then takes to some advanced level git commands that will be helpful in big multi-user projects. The following illustration (Via Scott Chacon’s Pro Git ) gives an idea of the life cycle of a file in git.

Each file in your working directory can be in one of two states: tracked or untracked. Tracked files are files that were in the last snapshot; they can be unmodified, modified, or staged. Untracked files are everything else - any files in your working directory that were not in your last snapshot and are not in your staging area. When you first clone a repository, all of your files will be tracked and unmodified because you just checked them out and haven’t edited anything.

The book shows the various stages a file can be , and the appropriate git commands that need to be used. One way to explore the above life cycle is to add files in interactive mode. In the interactive mode, git shows options like status, update, revert, add untracked, patch, diff, quit and help. Each of these options are useful in a specific scenario depending on the file status.

Some of the commands explained in this section are

• git diff : shows the difference between staged files and rest of files
• git diff –cached : shows the difference between files in original repository and working directory
• git commit –a –m , -a : lets you avoid the staging area and commits every file that was tracked earlier
• git reset
• git stash
• git push
• git rebase

The book ends with a nice tutorial for a) setting up a repository on github, b) cloning the repository on your local machine and c) working with using various git commands to synch your working directory with the remote repository.

Takeaway :

With screencasts and a large font size, this book is a quick read. Perfect for a newbie who wants to understand just enough, to start experimenting with git.

Blah Blah Blah: What to do when words don’t work - Summary

Dan Roam, the author of popular books “The Back of the Napkin” and “Unfolding the Napkin”, has written a new book with a message that one must fuse our linear wordy based thought process with pictures/images/visuals to comprehend things in a better way. Well, this message is not something new. Presentations with visuals catch our attention rather than the boring bullet point slides. To some extent my book summaries might also be boring as they are merely words, although I try to put in some visuals to make it interesting.

This book has three parts. The first part talks about the fact that we are drowning in blah-blah-blahs and in the process losing our clarity of thought. The second part of the book talks about vivid thinking, a jazzy word for using visual and verbal thought process to get to the message/ideas that we ought to understand and the stuff we ought to filter out as noise.

The author uses an analogy of Fox and Hummingbird to distinguish between visual and verbal skills necessary to think. Fox is linear, analytic, patient, clever and smug( representing our verbal mind). Hummingbird is spatial, spontaneous, synthesizer and easily gets distracted (representing our visual mind). This book argues for a balance between Fox and Hummingbird traits.

Why should we care about developing these skills? Well, there is a lot of blah-blah that we get to hear in our lives, be it in corporate presentations, conference calls, customer meetings etc. and most of the times the idea is either complicated, missing or outright misleading. The intent of blah-blah-blah could also be obfuscate or deviate from the original intent. Hence it is imperative to understand the right message from all the blah-blah-blah, and that is where these Fox and Hummingbird skills are useful.

We have all been drilled with Fox Skills/ Verbal skills right from our childhood. However Hummingbird skills have not been taught consciously to most of us and hence we develop a phobia to use visual skills/ images/drawings to understand things. It is merely a phobia, says the author, and tries to offer a framework, i.e visual grammar,to get going on developing these skills. The framework has 6 elemental pictures

• Portraits are visual representation of nouns and pronouns ( WHO and WHAT )
• Charts are visual representation of quantity ( HOW MANY)
• Maps are visual representation of preposition and conjunctions ( WHERE )
• Timelines are visual representation of tense ( WHEN )
• Flowcharts are visual representation of complex verbs( HOW )
• Multivariable plots are visual representation of complex objects – Synthesis of all the above elements answering WHY

The last part of the book shows the reader a way to see the forest from the trees by distilling 7 attributes of a vivid idea. It uses an acronym FOREST to show the essential attributes of an idea. F == form, O == Only the Essentials, R == Recognizable, E == Evolving, S == Span Differences, T == Targeted.

Whether you believe in this structured framework or not, this book will make you realize that visual imagery is imperative for better understanding of the ideas you get to read/write/talk in the verbal/wordy world.

Calculus Gallery : MindMap

I was going over some old folders when I stumbled on to this mind map of the book “Calculus Gallery”.  
Indeed, a picture is worth a thousand blah blah blahs.

 

Listening below the noise : Summary

Yesterday was a rather unpleasant day for me. Whenever I am in a bad mood / I become restless I pick up this book by Anne D LeClaire, titled, “ Listening below the noise”. For some reason this book has been my best companion over the past few years. I reread parts of this book from time to time. This time around, I thought I should blog about this and write a few words about the book.

The author starts off by saying, “Like too many of us, I mistook a busy life to a rich one”, and cites her mid-life emptiness as one of the reasons for exploring “silence”. While strolling on the Cape Cod Beach on a warm afternoon in January 1992, she suddenly realizes that “silence” is the missing element in her life and decides to consciously inculcate it in her daily life. She then begins, as a whim, a unique ritual, i.e, maintaining a day of complete silence every other Monday.

Entry :

On the first day of silence, she realizes that her energy levels and productivity levels are far better than a normal day. She begins to wonder whether turning away from outside world is the first step to nourishing the inner world that is so very vibrant in every one of us. The first thing she realizes that silence and mindfulness reduces the habit of multi-tasking things in life. When we are doing something, i.e mindfully doing something, we cannot multi-task. Silence makes one realize that single-tasking is the way to do any work and is a far better and a peaceful alternative to multi-tasking. Is silence necessary for creativity? The author’s first brush with silence makes her feel that silence/solitude is probably a necessary condition to do creative work. In Silence, Our minds are more tuned to the creative energy that lies within us.

Staying in complete silence is not an easy practice, as everyone would agree. The author reflects on one such day when she has an immense urge towards breaking her “silence” tradition. She reveals one of her tricks, i.e, strike a bargain with oneself that it would be only a couple of hours of silence instead of maintaining one full day of silence. This little trick  sometimes becomes useful in centering her. However these bargains are only helpful for a temporary period of time. The quieter we become, the more we hear. What we hear might be unpleasant at times and hence there is a tendency to fill up the silence with noise, i.e songs/TV/Soap-opera/movies/empty chatter. On some level, we are afraid of what we will hear.

When we are in silence, there is one thing we definitely become aware, at least most of us I guess. There is a muddy messy space our inner garbage heap where we toss scraps too painful to consider or confront, the loss, pain, grief and disappointment that are all too real. When we weed out extraneous stimulation and let go of the reins of control, these things claim our attention. And it is silence that allows us the space and stillness in which to think about our motives, to examine our behavior, to see where we’ve fallen short. It is only when we drag our smallest, shabbiest parts into the light that we can move toward becoming whole. We are massively afraid of dealing with this compost. So, in that sense this is the reason we fill our lives with distractions so that we don’t get to meet this compost. However the wise thing to do is to actually confront this compost, to turn it over. For within the same mounds lies the fertile matter out of which new life arises and is nourished.

Cultivation :

It is also sometimes necessary to create boundaries in life for silence to walk in to our lives. May be cutting down on TV/ avoiding the idle chatter with someone, consciously weeding out distractions, etc are bare minimum steps that one must take for the silence to make presence in our lives. There is no guarantee that these restrictions on external stimulation would help one to remain silent as there is sometimes that internal chatter that drives away the silence ruthlessly.

Silence is Janus-faced. Like the Roman god Janus, silence holds two faces. To be silenced is not at all the same as choosing not to speak.A chasm lies between the two, as wide as that between fasting for a purpose and starvation. To be silenced is crippling, belittling, constricting, disempowering. Chosen stillness can be healing, expansive, instructive.

Let’s say you decide to have one day of complete silence. What if others in your family pick up a fight for some trivial reason ? The fact that you have chosen not to respond might fill you up with more resentment as you cannot vent out feelings. So , is the silence painful in such situations ? Well, not necessarily says the author narrating one such incident when silence helped in not messing up a situation and thus was invaluable in providing time and space to defuse a situation.

Silence enhances listening ability, i.e, listening to things that are left unsaid. Bird-watchers, Naturalists typically develop a keen sense of listening. One colleague of mine is so excited whenever he sees tigers/butterflies/birds etc. whenever he goes for a Jungle safari. To observe/spot/infer the kind of bird/butterfly, it demands a sense of stillness from a person. Unless you are fully present in the NOW and develop a keen ear, going on a Jungle safari is practically useless. Ability to remain Silent is a great skillset to have in such activities. Similarly even in our daily lives unless we are silent we cannot truly listen to others. We are merely talking and waiting to talk. We engage in chatter, not conversation, and our chatter reveals our egos’ needs: Love me, admire me, envy me, fear me, help me, see me. There is little space for truly hearing others. Silence in that sense enables true conversations to take place.

Fertilization :

Silence is the first essential for most of the creative endeavors. Even a creative act like choreography might need the choreographer to visualize the steps in silence and then make his/her troupe follow those steps. Personally I find silence very stimulating , be it for programming something, be it for writing a blog post , be it for practicing music. The attention we can give to work, complete and undistracted concentration does enable us to bring the best with in all of us. Like the solitary spider who busily weaves her web in perfect silence, we need to be alone and quiet for our subconscious to spin its creations. Stillness focuses the brain. And like the tensile strength of the spider’s strands, it buttresses and strengthens creativity. Another beneficial aspect of silence is it produces marked improvement in physical and psychological health. At what age, do we develop a low tolerance for quiet time? When do we begin to call it boredom? When do we begin the excessive yearning for entertainment and diversions? The prospect of loneliness can make us fear solitude and silence. The paradox is that those are the very measures that can heal us. The author concludes this section by pondering over her experiences of remaining silent for an entire week.

Harvest :

After 9 years of practicing this ritual, the author develops sudden resistance to this ritual. She has this intense urge to put an end to this ritual but finally bargains another 2 months with herself. During these two months, she reflects on the benefits of silence through the years . She also remembers her yoga teacher’s words:

Sometimes we go through a time  that looks like a setback but in reality that time is a place of preparation. A resting space. A gathering of energy. Like an archer pulling back the bowstring so the arrow can shoot forward.

She also realizes another benefit that silence bestows on people who practice it, i.e, the inner strength that it provides.Trees growing in a forest are fundamentally weaker and less able to weather wind and storms than ones that stand alone, because the solitary trees, without the shelter provided by the others, develop stronger, deeper roots. After two months of the bargain period, the author realizes that silent-day experiences have helped her develop a  spiritual way of living and is no longer a mere ritual. So ,she decides to continue the practice(till date).

Sowing :

In the final section of the book, the author provides a few ways to incorporate silence in to our daily lives.

I have read this book at least half a dozen times till now and every time it has made me appreciative of the importance of silence and solitude in one’s life. Sometimes when I am restless, I just pick up this book and read a random paragraph. The writing is so beautiful that it brings my restless mind to calmness and allows me to focus on my work.

R Inferno : Summary

       

 

The author of “R Inferno”, Patrick Burns, starts off by saying, “If you are using R and you think you’re in hell, this is a map for you”. Well, this fantastic book needs to be read by any R programmer, irrespective of whether he thinks he is in hell or not. The metaphor used in this book is that of journey through concentric circles, each circle representing people (programmers) who are suffering in pain because of “violating the proper programming conduct”. Using this metaphor, the author makes an amazing list of items that one need to keep in mind while programming in R. There is a good discussion on each of the items too. My intent of this post is to merely list down the main points of this book. 

Circle – 1: Falling into the Floating Point Trap

• Be careful with floating point representation of numbers. There will always be numerical errors which are very different between logical errors
  

 Circle – 2: Growing Objects

• Preallocate objects as much as possible
• Try to get an upper bound of the vector you will need and allocate the vector before you run any loop. Limit the number of times rbind is used in a loop
• If you do not know how many elements will get added in each loop, populate the data for each iteration in to a list and then collapse the list in to a data frame
• R does all the computation in RAM. It means quicker computation but it means that if you are not careful it will eat up all your RAM
• Error: cannot allocate vector of size 79.8 Mb. This should not be interpreted as “Well, I have X GB of memory and why can’t R allocated 80 MB”. The fact is that R has already allocated the memory efficiently and it has reached a point where it cannot allocate more memory
• To check the memory that is being used up, generously scatter the code
• cat('point 1 mem', memory.size(), memory.size(max=TRUE), 'nn')
• memory.size() and memory.limit() gives an account of memory used up and memory that can is still left that can be used
  

 Circle – 3 : Failing to Vectorize

• Write functions / code which inherently handles vectorized input
• Vectorization does not mean treating collection of arguments as a vector.
• min, max, range, sum, and prod take the collection of arguments as the vector. Mean does not adhere to this form mean(1,2,3,4) gives 1 as output whereas min(c(1,2,3,4)) gives the right answer as 2.5
• Vectorize to have clarity in the code construction.
• Subscripting can be used as a vectorization tool
• Use ifelse instead of if to help vectorize your code; vector is not a welcome input in if condition.
• Use apply/tapply/lapply/sapply/mapply/rapply etc have inbuilt vectorized functions instead of writing loops

 Circle – 4 : Over Vectorizing

• apply function has a for loop inside. lapply function has a for loop inside. Hence mindless application of these functions is skirting with danger
• If you really want to change NAs to 0, you should rethink what you are doing – you are introducing fictional data

 
Circle – 5: Not Writing Functions

• The body of a function needs to be a single expression. Curly brackets convert a bunch of expressions in to single expression
• Functions can be passed as argument to other functions.
• do.call allows you to provide the arguments as an actual list
• Don’t use a list when atomic vector will do
• Don’t use a data frame when matrix will do
• Don’t try to use an atomic vector when list is needed
• Don’t use a matrix when data frame is needed
• Put spaces between operators and indent the code
• Avoid superfluous semicolons that you would have been carrying from the old programming languages
• Rprof can be used to explore which functions are taking most of the time
• Write a help file for each of your persistent functions.
• Writing a help file is an excellent way of debugging the function.
• Add examples while writing a help file and try to use data from the inbuilt datasets package

 
Circle – 6 : Doing Global Assignments

• Avoid Global assignments ( <<- ). The code is extremely inflexible when global assignments are used.
• R always passes through value. It never passes by reference.

 Circle – 7 : Tripping over Object Orientation

• S3 methods make the “class” attribute. Only if an object has “class” attribute, do S3 methods really come to an effect.
• If Generic functions take S3 class as an argument, it searches the S3 class with the function which matches the name of the generic function and executes it
• getS3method(“median”,”default”)
• Inheritance should be based on similarity of the structure of the objects , not based on similarity of concepts. Dataframe and matrix might look similar conceptually, but they are completely different as far as code reuse is concerned. Hence inheritance is useless between matrix and dataframe
• There is multiple dispatch in S4 objects
• UseMethods creates an S3 generic function
• standardGeneric creates S4 function. More strict guidelines for S4 class object
• In S3 the decision of what method to use is made in real-time when the function is called. In S4 the decision is made when the code is loaded into the R session. There is a table that charts the relationships of all the classes.
• Namespaces : If you have two functions with the same name in two different packages, namespace allows you to pick the right function.
• A namespace exports one or more objects so that they are visible, but may have some objects that are private.

Circle – 8 : Believing it does as intended

 

In this circle there are ghosts, chimeras and devils that inflict the maximum pain

Ghosts

• browser(), recover(), trace(), debug() are THE most important functions in R debuggin
• always use prebuilt nullcheck functions such as is.null ,is.na
• objects have one of the following as atomic stogarge modes:logical, integer, numeric, complex, character
• == operator and %in% operator – Their importance and relevance
• Sum(numeric(0)) is 1 and prod(numeric(0)) is 1
• There is no median method that can be applied to data frame.
• match only matches first occurrence
• cat prints the contents of the vector . while using cat you must always add a newline as by default it doesn’t have one.
• cat interprets the string whereas print doesn’t
• All coercion functions strip the attributes from the object
• Subscripting almost always strips almost all attributes
• Extremely good practice to use TRUE and FALSE rather than T and
• sort.list does not work for lists
• attach and load put R objects on to the search list. Attach creates a new item in the search list while load puts its content in the global environment, the first place in the search list. source is meant to create objects rather than loading actual objects
• If you have a character string that contains the name of an object and you want the object, then one uses get function
• If you want the name of the argument of the function, you can use deparse(substitute(arg_name))
• If a subscript operation is used on an array , it becomes a vector not a matrix. If you use drop=FALSE , the attribute is kind of preserved
• Failing to use drop=FALSE inside functions is a major source of bugs.
• The drop function has no effect on a data frame. Always use drop=FALSE in the subscripting function
• rownames of a data frame are lost through dropping. Coercing to a matrix first will retain the row names.
• If you use apply with a function that returns a vector, that becomes the first dimension of the result. I came across this umpteen number of times in my code and I just used to have the result transposed.
• sweep function is a very useful function that is not emphasized much in the general r literature floating around
• guidelines for list subscripting
  • single brackets always give you back the same type of object
  • double brackets need not give you the same type of object
  • double brackets always give you one item
  • sungle brackets can give you any number of items
• c function works with lists also
• for(i in 1:10) i does not print anything . The problem is that no real action is involved in the loop. You must use instead print(i)
• use of seq_along or seq(along=x) is always better
• iterate is sacrosanct. Never knew about this earlier. This statement means that if you have a for loop with index on i and then you change the value of i in the loop, it does not effect the global counter of the loop
• R uses dynamic scoping rather than lexical scoping

Chimeras

• factor : Factors are an implementation of the idea of categorical data, Class attribute is “factor” , “levels” attribute has a character vector that provides the identity of each category
• factors do not refer to numbers. as.numeric() typically gives numbers that has nothing to do the factors
• subscripting does not change the levels of the factor . Use drop=TRUE to drop the levels that are not present in the data.
• Do not subscript with factors
• There is no c for factors
• Missing values makes sense in factors and hence there can be level NA for a factor
• If you want to convert data frame to character, it is better to convert to a matrix and then convert to a character
• X[condition,] <- 999 Vs X[which(condition),] <- 999. What’s the difference ? The latter treats NA as false while the former doesn’t
• There is a difference between && , &. Similarly || , |. The latter is used in vector comparisons and former is used for a single element. Use & | in ifelse condition and && || in if condition.
• An integer vector tests TRUE with is.numeric. However as.numeric() changes the storage mode to double
• Be careful to know the difference between max and pmax
• all.equal and is.identical are two different functions altogether.
• = is not a synonym of <-
• Sample has helpful feature that is not always helpful. Its first argument can be either the population of items to sample from, or the number of items in the population.
• apply function coerces a dataframe in to matrix before the application. Its better to use lappy instead of apply to keep the attributes of dataframe intact.
• If you think something is a data.frame or a matrix, it is better to use x[,”columnname”]
• names of a dataframe are the column names while names of a matrix are the names of the individual elements
• cbind with two vectors gives a matrix , meaning, cbind favors matrices over data.frames
• data.frame is implemented as a list. But not just any list will do – each component must represent the same number of rows.

Devils

• read.table creates a data.frame
• colClasses to control the type of input columns that are imported
• use strip.white to remove extraneous spaces while importing files
• scan and readLines function to read files with irregular data format
• Instead of storing data in a file, retrieving the file back to R , it is better to save the object and attach/load the object as and when required
• Function given to outer must be vectorized
• match.call can be used to access ... in the argument of the function
• R uses lazy evaluation. Arguments to functions are not evaluated until they are required
• The default value of an argument to a function is evaluated inside the function, not in the environment calling the function
• tapply returns one dimensional array which is neither a vector nor a matrix
• by is a pretty version of tapply
• When R coerces from a floating point number to an integer, it truncates rather than rounds
• Reserved words in R are if , else , repeat , while , function , for , in , next , break , TRUE , FALSE ,NULL , Inf, NaN, NA, NA_integer, NA_real, NA_complex, NA_character
• return is a function and not a reserved word
• Before running a batch job, it is better to run parse on the code and check for any errors.

Circle – 9 : Unhelpfully seeking help

This circle gives some guidelines in the context of posting queries in various R help forums.

 

Takeaway:

This is my favorite book on R. Any R programmer at whatever level of expertise he/she is at, journey through these circles, would certainly make them a better programmer, and their present / future pain of debugging their R code less traumatic.