RK's Musings

Speed Traders : Summary

In the introduction, the author tries to define HFT using the voices of leading HFT players. What are the characteristics of HFT? There is no agreement on a common definition of HFT, but there are some elements that one can at least attribute to HFT. They are

• Low latency trading.
• High turnover strategy.
• Trader goes home `flat' with no open position.

The goal of any HFT firm broadly is to have a set of uncorrrelated trading strategies that has statistically more winners than losers for all the trading positions of the day. The introduction mentions the fundamental market driven factors that gave rise to HFT:

• RegNMS.
• Order Handling Rules leading to ECNs and hence a proliferation of venues.
• Explosion of the universe of instruments; Human market maker is giving way to computerized market maker.

The author ends his intro clarifying terms such as program trading, quant trading, algo trading, automated trading, prop trading, stat arb, ultrahigh frequency trading. I found the explanation of these terms better explained in Barry Johnson's book ,`Algorithmic trading and DMA'.

The Emergence of High-Frequency Trading

This section lists down the significant events from 1969-2007 that were a precursor to todays' HFT

• 1969 Feb - Instinet Launched.
• 1971 Feb - NASDAQ starts electronic trading in 2500 OTC stocks.
• 1975 May - SEC bans fixed commission rates.
• 1976 May - NYSE introduces DOT.
• 1984 July - NYSE Approves SuperDOT.
• 1987 - Launch of ITG Posit.
• 1994 August - BNP purchases Cooper Neff Options trading firm.
• 1997 - Order Handling rules approved.
• 1998 - Launch of Attain, NexTrade, Strike Technologies, Bloomberg tradebook.
• 1999 April - Regulation ATS becomes effective.
• 1999 July - Goldman buys Hull for $530M.
• 2000 - BRUT/Strike merger, Archipelago and PSX electronic exchange launched.
• 2001- Launch of Liquidnet, Launch of Direct plus, Archipelago/REDI Book merger.
• 2001 April - Decimalization rule hits stock exchanges.
• 2002 - Launch of NASDAQ Super montage, Instinet - Island merger, Tradebot trades 100 million shares a day for the first time.
• 2004 - Nasdaq acquires BRUT, Launch of Pipeline.
• 2005 - RegNMS comes to force, NASDAQ acquires INET, NYSE acquires Archipelago, Knight acquires Attain and renames to DirectEdge ECN, Citi acquires NexTrade.
• 2006 - NYSE buys Euronext, CME-CBOT merger.
• 2007 - Chi-X Europe launched by Instinet. It becomes largest Multilateral trading facility in Europe. A graphic for the above events should have been there to make the content easier to read and recall.

A few visuals like the would have made the chapter more readable:

The Path to Growth

This chapter traces the developments through the period 2007-2010 June.

• 2007 April - General Atlantic invests $300M in GETCO.
• 2007 May - NASDAQ announces its plans for acquiring Nordic Stock exchange.
• 2007 July - Citi buys Automated Trading desk for $680M.
• 2007 Oct - NASDAQ buys Boston Stock exchange.
• 2007 Nov - MiFID goes live in Europe , NYSE announces its plans for a data center Mahwah, NJ.
• 2008 June - `Flow traders' receives funding.
• 2008 June - GS Sigma X and Credit Suisse ATS platform log in record volumes of 406M and 210M shares respectively.
• 2008 Nov - BATS becomes an exchange with its USP as speed with maker-taker model. As of today, BATS logs in 12% of the daily US equity trading volume.
• 2008 - Citadel's Tactical Trading fund earns $1.19B in 2009 with a team of 55 people!
• 2009 Mar - GSET launches Sigma-X in Hongkong. It soon becomes a big hit.
• 2009 June - NASDAQ introduces two types of flash orders , BATS introduces flash orders.

High Frequency Trading Goes Mainstream

This chapter traces the developments during 2009-2011 , the time frame when HFT entered the parlance of media, journalists, politicians, economists etc. It all began with a NYTimes article by Charles Duhigg on July 24, 2009 on HFT. That was the first time when HFT became known to wider audience and most importantly lead to false notion amongst people that flash orders were used to manipulate market.Click here  to read the full article that was published in NY Times.

Soon after this article got published, NASDAQ and BATS stopped offering flash orders. Amidst the falling economy, HFT was making money and this was met with harsh reactions from everyone. Meanwhile BATS and DirectEdge were eating in to the market share of NASDAQ and NYSE. So, to stay competitive in the game, NASDAQ launched INET in late 2009, an HFT enabled platform. NASDAQ also asked SEC to look in to various data centers that were mushrooming in the country. These data centers were heavily used by hedge funds and HFT shops. Meanwhile Chi-X was launched in Europe and Asia. Chi-X was very successful in both the venues. In Japan, it was launched under the name, `Arrow Head' and the latest numbers from Arrow Head - 100 Million shares a day - shows that it is a massive success. There is another market development mentioned here, i.e the ban on naked access. Naked access goes under various names, 'Sponsored Access', 'Unfiltered Access', etc. All those names boil down to one type of access - Allowing hedge funds to participate in the market using broker's market participant Id. The broker does not put any kind of controls to the orders and merely offers the platform to the clients to trade with broker's participant Id. According to Aite group, 38% of the equities volume is from naked access. Banning this was a big jolt to all the broker-dealers.

The chapter also features the success story of GETCO, an options trading firm started by 2 floor traders in 1999, that went on to become one of the biggest success stories from Chicago. There are some interesting facts about the GETCO's story like , it hired skilled video gamers from Illinois Institute of technology for its trading team. Well, to think of it, HFT is a kind of video game played with sophisticated technology. In Feb 2011, GETCO became a NYSE designated market maker along with the biggies like Goldman, Kellogg group, Bank Am and Barclays.The success of GETCO is also an indication of what a powerful financial and technology hotbed Chicago had become.

The Technology Race in HFT

Based on a few interviews with HFT shops, the author lists some basic requirements for a HFT shop to get going

• A reliable and sufficiently fast data input source.
• A robust and sufficiently fast market access.
• Low enough transaction, clearing and processing costs as well as sufficiently effective and efficient processing and verification capability.
• Strategy monitoring tools.

Well, if these things appear obvious, Kumiega and Ben Van Vliet of Illionois Institute of technology have developed a step-by-step methodology( using machine control theory) that addresses the needs of institutional trading and hedge fund industries for development, presentation and evaluation of high-speed trading and investment systems. They call it methodology and it says that the following stages of development should serve as a road map for running HFT shop.

• Before even firms even start building, they should ask themselves how they are going to monitor a successful algo.
• Build a customized database of historical data and purchase or build a tool for proper back testing of a strategy.
• Data cleaning methods are a key component. Writing your own scrub methods is better as it gives control over the data.Procure dirty data and see whether your system works.
• When algos go out of range,let's say 3-4 stdev , stop the strategy and fix it, just like a plant shuts off a machine if it goes 3 stdev from the normal.
• Backtesting is critical. However there are certain aspects you can't back test and be sure. Latency, Market data and timing are always going to be things that are difficult to check in backtesting.
• Fully define the functionalities and performance requirements of the trading/investment system and develop a system. Don't tradeoff robustness for the sake of speed.
• Put in place a reporting mechanism to capture relevant metrics for the HFT strategies.

The chapter ends with saying that, low latency is essential, but it is the strategy that needs to be robust. This is a very different view from what I heard from one of my friends last week. He is of the opinion that there are less than 10 strategies that are basically used in all the HFT shops and the speed is the ONLY thing matters. This book and other people mentioned in the book says `otherwise'. May be both matter equally!.

The Real Story Behind the “Flash Crash”

I found this chapter to be most interesting in the entire book as it goes behind the events that lead to NY Times article in July 2009 . It was in June 2009 that NASDAQ and BATS introduced flash orders to compete against DirectEdge. DirectEdge had given its participants a special order type called Indication of Interest , that helped DirectEdge add 10% to its existing market share. So, this tactic by NASDAQ was to make SEC ban such orders on the entire US market including CBOE that had roaring flash order business. SEC did not make regulatory changes as far as options are concerned. But it did make changes and subsequently flash orders were completely removed from the market.

One typically associates flash crash with some kind of HFT trading gone bad. But this chapter makes it very clear that HFT traders hate flash orders as the only entities that are profitable in a flash order are the exchanges and the guy putting the order. Infact reading this chapter makes one realize that flash crash actually goes on to show why HFT firms are needed in the first place becoz most of the HFT firms shut off their computers during the flash crash. Since there was no liquidity from HFT firms,volatility increased so much that Dow tanked 600 points and bounced back. Liquidity and Volatility have an inverse relationship and thus 'HFT firms providing liquidity lessen the volatility' is the argument provided `for' HFT case. When SEC report came out investigating flash crash, it was clear that a massive market sell order of $4 Billion E-Mini contract was the tipping point for the crash in the already nervous market. Why would a trader put in such a big order as a market order and not a limit order is surprising ? Somehow the press has been flogging HFT for the flash crash , when infact, there were firms who were providing liquidity even during the flash crash. Rumor is that there is a firm in Chicago which made $100M in one day. Now given that HFT profits are close to $2B-$3B, that's a huge amount of money.

Will there be more flash crash type of events in the future ? Certainly says the book and it says this has got nothing to do with HFT strategies.

Life after the “Flash Crash”

This section describes the various events post May 2010 flash crash

• May 12, 2010 - Talk of HFT tax as they were falsely blamed for flash crash
• May 14, 2010 - The first piece of investigation revealed that it was a traditional money manager who placed a $4B sell trade with out price limit, that triggered the crash
• May 24, 2010 - Talk of eliminating stub quotes
• May 26, 2010 - Goldman launches DMA in Latin American markets
• June 4, 2010 - SGX announces $250M “Reach Initiative”, the fastest platform in the world that has a 90 microsecond door to door speed. NASDAQ has a door to door metric of 177 micro seconds. This proposed platform at SGX is 100 times faster than the existing infra
• June 10, 2010 - SEC approves rules that require exchanges to stop trading in specific stocks if the price goes up by 10% or more in 5 min interval.
• June 11, 2010 - Revolving door practice is seen at GETCO. GETCO hires Elizabeth King, a personnel from SEC. This practice of hiring people from regulatory bodies is called 'Revolving door practice'
• June 22, 2010 - GETCO launches its dark pool
• July 21 2010 - DirectEdge launches two platforms, one for blackbox stat arb guys and second for passive agency order guys
• Aug 24 2010 - Chi-X receives a takeover enquiry from BATS. Chi-X becomes the second largest trading venue after London Stock exchange in Europe
• Sep 1, 2010 - BATS and NASDAQ stop flash orders. DirectEdge continues with Indication of Interest orders, a flash order in disguise. From the flash crash to the Waddell & Reed trade that started it all, from the discussion about stub quotes to circuit breakers, from the launch of new exchanges in the United States to GETCO's dark pool in Europe, by November 2010, high frequency trading was on the minds of everyone in the financial world.

The Future of HFT

This sections lists the events that have happened in 2010.

• Oct 21, 2010 - Chi-X Global announces that it will start operating in Australia by Mar 2011
• Oct 21, 2010 - SGX makes GETCO a trading member in its securities operations
• Oct 25, 2011 - SGX makes a $8.3 B takeover offer for the operator of Australian Bourse
• Nov 3, 2010 - SEC voted to bar brokers from granting HFT unfiltered access to an exchange, a move aimed at imposing safeguards meant to prevent bad trades
• Nov 8, 2010 - SEC decides to ban `stub quotes'
• Nov 11, 2010 - Chi-East, a joint venture between Chi-X Global and SGX was launched and this became the first pan-Asian independent dark pool to be backed by a regional exchange.Chi-East is widely regarded as a landmark in the development of Asian Equities
• Nov 24, 2010 - DirectEdge announces that it was changing the way it handles flash orders. It also indicated that it will slowly phase out flash orders
• Dec 1, 2010 - Equiduct Systems Ltd, announced that it traded more than 1 billion euros in Nov 2010 for the first time as ATS aimed at retail brokers gathered momentum. Equiduct is an MTF operated by Berlin bourse.
• Dec 6, 2010 - ASX that was selling itself to SGX said that the decision to merge with SGX was in the best interests of Australia's market micro structure development. Collectively the combined SGX - ASX entity would list companies worth $1.9T, fourth in Asia behind Tokyo, HongKong, and Shanghai.

The book is interspersed with interviews with 6 High frequency traders.

Meet the Speed Trader : John Netto

In this chapter, John Netto , a high frequency trader talks about his trading experiences and HFT in general. Here are some of the aspects he mentions,

• The creation of HFT strategy was the easy part. Issues such as how one will raise capital and how one will handle dealing with compliance, regulatory, reporting, and developing infrastructure are all questions not asked at first but which will become a critical part of the equation of profitability
• Low latency is very important for HFT but the degree of importance varies based on strategy.
• In terms of infra, I would say buy , than build so that you can focus on strategies to make money.
• Sophisticated systems can be thought of as the assistants to the traders, not replacements to the traders, because after all, strategy, structure and people are key to success.
• HFT is going to get bigger, stronger and more prevalent. More traditional investment managers will explore HFT in the times to come.
• Always have a few strategies that you think , make money on paper at least.
• I work out daily. Fitness and being active are a very important part of success . Trading requires you to be mentally alert and hence you have to be be fit.

Meet the Speed Trader : Aaron Lebovitz

Aaron Lebovitz spent two decades in the industry and then started his own firm, Infinium Capital management, a prop shop in 2003 and made it in to an exceptional force in the industry. Here are some of the aspects he mentions,

• Hook up with a firm that has a proven track record, and make sure to know how to write code.
• Integration of formal models in to trading decisions is the kicker you get in applying quant.
• Started off with pairs as his first high-freq strategy.
• His mantra is, `Developing new markets, increasing global market liquidity, and driving efficient price discovery'
• Infinium trades 23.5 hours 6 days a week. It did shut down its machines on the day of flash crash
• It is not all about speed. Having subtle or good strategies will get you alpha.
• The future of HFT is still very much in doubt, hinging on an uncertain regulatory environment.
• I don't foresee traditional investment managers shifting their focus to becoming market makers or developing statistical arbitrage strategies

Meet the Speed Trader : Peter van Kleef

Peter Van Kleef has spent the last 15 years running automated trading ops. Here are some of the aspects he mentions,

• It never really crossed my mind why someone would want to trade any other way than HFT. It is just so much more efficient
• Start writing your own trading strategies and see whether they make money on paper. Think about fancy infra later.
• Trying to apply solutions from other fields of science and research is also a start when exploring HFT.
• Whenever possible, I try to balance work with some sort of physical exercise. It is MUST for being a better trader.
• Low latency is important but is not the be-all and end-all.
• It is time for people to stop competing for speed and start producing alpha.
• A lot of the profits that are sustainable usually will come from the strategy and not necessarily from technology.
• Leverage is the key driver because returns are quite low.
• The overall profit largely depends only on the number of trades that can be done because the average profit is nearly certain. I don't know many other strategies that, if run well, consistently range about 50% per year and above. We are not in the business to rate ourselves; high-frequency is very profitable, if done right. That much is sure.
• Sophisticated retail investors are already running HFT strategies.
• HFT of the yesteryears is becoming a commodity as you can buy all the infra that is needed.
• Part of what will determine the future of the industry is what regulatory changes legislators and SEC will introduce.

Meet the Speed Trader : Adam Afshar

Adam Afshar's views on HFT are little different from the traders covered earlier. He is of the opinion that it is the low latency that is extremely critical. Here are some of the aspects he mentions,

• HFT is a method or a tool. It is not a strategy. HFT methods are applied to three types of strategies - Market Making, Stat arb and Algorithmic execution.
• My HFT shop uses models where there is absolutely no human intervention in the execution. I believe this should be the way, as human order execution does not allow you to collect data and you can't back test. If you use an algo for execution, one can collect all types of data and then improve it. Your reliance of specific individuals is also less
• High-speed data management is the linchpin of a success HFT shop.
• You can't avoid latency issue. Its a prerequisite for whatever you do in HFT world.
• The biggest revenue generating idea in my shop is , one that scours 110 million news items and takes positions intra-day
• HFT popularity will slowly fade away. It will soon become a commodity once tech makes the landscape flat.
• My advice to students is to take science and liberal arts subjects as they teach how to think. You can figure what you want to do later.

Meet the Speed Trader : Stuart Theakston

Stuart Theakston runs GLC, an HFT firm. Here are some of the aspects he mentions,

• High frequency traders simply replace specialist/jobbers in providing liquidity in a much more competitive framework.
• HFT traders are not getting a free lunch by installing heavy duty infra. Infact all they are doing is eat each other's lunches.
• An unfortunate way in which flash crash was handled by the exchanges : Some of the trades were canceled. This gives all the more a reason for HFT traders to be away from the market during crisis.
• I spend 70% of the time with quants, traders and programmers and 30% of the time watching market and doing research myself. I read a lot of academic papers and journals, etc.for they provide valuable source of ideas.
• Most of the people in the media, political circles, policy makers are clueless about the function of HFT. Volatility reduction demands Liquidity and HFT guys play a crucial role here. The policy makers have the potential to kill this entire HFT crowd if they don't understand issues properly.
• Order-anticipation and cross-venue arb are sensitive to latency, liquidity provision is less so.
• Prop trading firms can spend 99 cents on infra to make 1 dollar. Hedge funds can only spend 19 cents of the dollar given the structure. So these participants cannot compete in the same space
• HFT is nearing the bottom in terms of competition to be the fastest. The opportunities are outside equity asset classes like Forex, Exchange traded CDS and developed markets.
• Suppliers will spring up who try to sell HFT trading in a box to the masses. But the retail users won't make any money, at least on a properly risk-adjusted basis
• Most of the development in HFT have already happened in developed-market equity and equity derivative markets, and these are approaching maturity. These markets are likely to be the domain of increasingly specialist outfits. The low hanging fruit has been picked. The next wave is likely to be in emerging markets and esoteric ETFs. This is where people developing their careers in this space should focus.

Meet the Speed Trader : Manoj Narang

Manoj Narang started his firm in 1999 that was in to providing financial toolbox. In the last few years or so, he has transformed his firm in to a profitable HFT firm.Here are some of the aspects he mentions,

• Its the law of large numbers at play in HFT. If you have a strategy that works 55 percent of the times, you can't do a few trades. You have to do a ton of trading to translate that edge in to sustainable profits.
• We generally are buyers of hardware and builders of software and systems.
• We have never had a a losing week since we started in the business.
• A successful high frequency strategy will have a Sharpe ratio higher than 4 and a successful HFT operation that runs multiple strategies generally will have a double-digit Sharpe ratio. Such high Sharpe ration are unheard of in traditional investment places.
• Back of the envelope calculation shows that the profitability of the entire HFT industry is around $2B. Not much in relative terms. There are many other corners of the financial industry such as derivative trading, that generate hundreds of times this amount of profit in one year.
• The main use of outside capital in the world of HFT is to fund research and development, operations, not to actually trade the capital. Very little capital is required for trading purpose.
• Once you are an electronic market maker, it is skills that become important and not status or power or connections.
• Market making function in equities is decentralized. It doesn't mean everyone should be a market maker. Just because everyone needs food, doesn't mean everyone becomes a grocer. Similarly individuals should seek to be investors and not liquidity providers.
• Wall street has little to do with HFT.
• I think the main risk in the financial markets is that the capital is getting increasingly concentrated, which makes herd like behavior even more prevalent. The frequency with which bubbles inflate and exploder is getting more and more rapid as a result because massively capitalized investors jump from one asset class to another asset class, leaving devastation in their wake
• HFT does not connote systemic risk.The only systemic risks to the market are the ones posed by herd-like behavior.
• Tradeworkx started HFT in 2009 and this 50 member shop account for 3% of overall volume in SPDR ETF.
• One must keep in mind that modeling systemic correlations is VASTLY different from modeling structural correlations.
• How do I see our fund in five to ten years? I have no idea; five or ten years are an eternity when you are immersed in a world where microseconds matter.

Takeaway :

If you take all the events that have happened relating to HFT world and list them in a chronological order, spice it up with some HFT trader stories and viewpoints, what you get is, this book.

Choosing Workflow applications

I stumbled upon a very interesting write-up that captures the necessary software needed for managing one’s projects. The paper is written by Kieran Healy of Duke University. The takeaways from the paper are as follows :

• Three principles of work flow :
• Keep a coherent record of actions. Instead of doing a bit of statistical work and then just keeping the resulting table of results or graphic that you produced, for instance, write down what you did as a documented piece of code. Rather than figuring out but not recording a solution to a problem you might have again, write down the answer as an explicit procedure
• A second principle is that a document, file or folder should always be able to tell you what it is. Beyond making your work reproducible, you will also need some method for organizing and documenting your draft papers, code, field notes, datasets, output files or whatever it is you’re working with.
• DRY principle. Repetitive and error-prone processes should be automated if possible.
• Learnt about Dropbox feature. Found it very interesting as it gives an instant back up of everything that you consider important.
• Git can be used for version control
• Sweave your files so that you always have a track of the code/syntax/libraries/datasets used for a specific research project.
• I guess one can use ProjectTemplate library to generate a rails like structure for statistical projects
• Create a beamer presentation of any project you do, no matter how trivial it is. The discipline will all be useful when you need to produce non-trivial presentation
• After reading this paper, I guess the following can be a rough guideline for a project using R
• Use Project Template library to create a directory structure
• Start recognizing the datasets you would want to work with and document the datasets
• Code your scripts / functions and document them
• Do your analysis – Decide wisely between R scripts and Rnw files. May be all the trials you can do in .R files and mini finalized steps in .Rnw files
• Create Rnw documents that capture the essentials of your project and use Sweave to convert them in to reproducible research documents like pdf
• Use git to do version control of the project
• Use beamer to produce a sample presentation of the project/purpose/ learnings/ applications etc
• If need be, pick out the functions, datasets and convert in to R package and install it on your machine.
• As one can see, the learning curve is steep here. Each step in the above process will take time and effort to learn and execute. But the effort would be worth it as the projects will be there with you for as long as you code in life. If ever you want to look back at a project, you can start with the beamer presentation then proceed to sweave output, and then to the package developed. It will all make sense very quickly. Doing work without this workflow will make it difficult to revisit your projects/research work. I guess its like the difference between short term and long term memory. Following this workflow gives you a schema of the project and hence gets stored in your long term memory. Doing adhoc work stores stuff in your short term memory and then it just vanishes from there.

Practical .NET for Financial Markets

I went over this book to get some idea on STP. I was completely clueless on the things that happen at the back end, once the trade is placed with a broker. STP was introduced in Indian Markets by FT and few other vendors almost a decade ago. It has lead to a clear improvement in operational efficiency. Even though this book talks about .NET platform, I liked the chapter on STP that explained things using business case and .NET code.

STP

• STP(Straight Through Processing) providers have the responsibility of carrying and delivering STP-related instructions and messages. They provide connectivity between brokers, custodians and fund managers, and bring them to a common network. Once the entities enter the common network, they communicate with each other using standard messaging protocols like Swift 15022.
• In India FT is the leader in this game, with STP gate solution being used by everyone.
• ISO 15022 is an acceptable communication protocol for achieving STP interoperability in several countries
• STP was initiated as a part of T+1 initiative by the equities market
• Swift 15022 terminal is the source through which all the market participants get on to this platform.

• Cryptography is critical in this entire STP layer
• Substitution Cipher can be used
• Transposition Cipher can be used
• Symmetric algos and Asymmetric algos are used to protect the confidentiality of the data. I think asymmetric algo is used in FT case
• Digital Signature addresses non-repudiation issues encountered in high risk situation.

STP Interoperability

• If there are multiple STP platforms, SOA can be used as a model for different STP platforms to work.
• Web services are being used so that STP platforms can talk to each other.SOAP is the messaging communication framework used by Web services to send or receive messages

The entire structure of the book can be summarized using this visual.

Will go over the contents in greater detail at sometime in the future.

Project Euler

Colin Hughes shares his story on “why he built Project Euler? ”

• It should be Playful, bottom-up learning
• Programming is addictive and you can get feedback on customizing other’s code.

So, the code for becoming good at a language is –:)

         do ( 

        i  <- 1
        for ( i in 1:N )  print( “ You will fail ” ) ;    
        i <- N+1
        if ( i ==  N+1 )  print ( “ You will crack it ” )

             ) while(  you breath / live   ) 

“Is there a way of extending this type of learning programming to any kind of learning” ? , is something that his story leaves you thinking .

Code Quality Measurement :)

Via CleanCode :

 

The Shallows : Summary

What is common between Michael Lewis, Malcolm Gladwell, Paul Graham , Clay Shirky ? If you have read any of the articles in the media written by the above authors, their articles are distinct. They are long, hyperlink free, engage our minds for a long period of time using arguments / interesting examples , make us a bit contemplative . In short we can DEEP READ their articles. They have the power to change our thought process too. However the other 99.9999999 % of the stuff that we come across on the net are basically short articles heavily laden with  hyperlink , video links, multimedia etc. Exposure to such media is doing something to our brains and Nicholas Carr discusses the issue thread-bare in his new book , “ The Shallows”.

Medium is the message , goes the popular saying and Nicholas Carr argues that medium has greater impact than the content , on the way we think and act. The author begins his book by admitting that his capacity for concentration and contemplation has come down, after years of exposure to internet. He chose to ask around a few people , formally and informally , and found that this condition to be true for a large number of people. A wide range of people including, faculty from premier medical schools, students from various Ivy league colleges,corporate personnel increasing admit that they are in the same boat. They seemed to have lost the ability to read and absorb a longish article on the net. Some admit that they no longer read on the internet, they merely skim/ scroll. Few even go to the extent of justifying this change in the mindset by saying,”One no longer needs to read books, as the relevant content one is looking for is available as a blurb, short article, paragraphs in Google books, wiki pages , etc”. “Why waste reading a full length book when one can just skim a few articles / facts about whatever one is looking for ? “, goes the usual stance.

Basically for an increasing majority, net is rewiring their brains by making them think that the very idea of reading a book is old fashioned.  There is absolutely no debate about the riches that internet can bring along. Its stupendous growth is a testimony for its utility. You can get to amazing amount of information with in a few clicks. However it looks like what we are trading away is “our old linear thought process” , Calm, focused, undistracted, the linear mind is being pushed aside by a new kind of mind that wants and needs to take in and dole out information in short, disjointed , often overlapping bursts- the faster the better! . Do we all realize it that net is rewiring our brains ? When we do a ton of things all at one go, check our email/gmail- allow the you tube video to load,-monitor the tweets-get notified of blog posts by RSS reader, what exactly is happening to our brains ? Most of us do seem to relate to the situation described above as we constantly multi-task on net. We might kid ourself and think that we are in control of the media. Are we truly in control ? Or is the Net that is controlling us ?

The book delves a bit in to the structure of brain. It traces the initial assumption of scientists and researchers who held a view that brain structure was akin to a single continuous fabric of nerve fibres which was kind of fully determined by DNA and the childhood experiences. The prevalent view was that structure develops early in a human’s life and there after it is pretty much fixed. Subsequent discovery of neurons, synapses, neuro-transmitters in the brain , and research shows that brain is massively plastic in nature. The space between neurons where the signals are transmitted changes constantly based on our life experiences, the way we think, the way we do things, etc. Umpteen number of examples are quoted in the book which drives home the point that mind is plastic but not elastic, meaning the mind does not snap back to the old structure after a while. It is constantly changing and massively elastic based on the way we think/do stuff.  Ok, so if the brain  is work in progress forever, can internet , which is basically a tool in our hands , change the entire structure of our brains ? 

Yes says neuroplasticity research. The book traces out a bunch of tools starting from maps, clocks, ploughs, machines etc, and shows that in each case, the tool took us from the practical everyday usage to a situation where mind developed higher level of abstraction. Maps for example transformed experience in space to abstraction of space. Clocks which were just a tool to record the flow of time have had an unimaginable impact on our mind. We have managed to divide time in to greater precision , and have cut days in hours minutes and seconds, thus virtually changed the meaning of the word “time”. “I don’t have time for ---- ” is a very common sentence you get to hear and if you pause and reflect on that statement, you are bound to realize that time controls us , a tool which we invented to control stuff. Same is the case with any tool which we start using. Internet is no different.

The book takes a stance that “The World of a screen is different from the World of a page”. Pages in the book can be deep read. Arguments , sometimes lengthy and winding can be read and pondered for a while. This is almost next to impossible while reading stuff on the screen . Do you remember any content that you have read for let’s say 1/2 – 1 hr  hr on the net ? Unlikely. Why ? Because the medium itself is designed to give you short bursts of disjointed information. So, if you are constantly reading stuff ONLY online and not engaging yourself on deep reading, brain will get rewired soon and it is likely that your attention span is going to reduce in the activities not connected to net. That’s a little scary because true innovation comes from deep thought and something that robs deep thought is definitely not desirable.

The Very image of a book is changing. ebooks have hit the market which are being marketed at a deep discount price as compared to books/hard covers. The distribution of ebooks has also changed dramatically. Ipad, Kindle and other ebook readers, now enable us to get a book with in no time. But what is further dramatic is what we see on the screen. Inbuilt dictionaries, hyperlinked content, access to internet, blogs, magazines at now a click away. In the earlier times, if you had to disengage from a book, you had to atleast put the book away but now the distractions are built right in to the book!. So, will you be able to deep read a book ? Well, you can if you turn off the wireless and all the other distractions that come with ipad / kindle ? Economics will no doubt push the books to be published in ebook format, kindle format and the reading habits are bound to change. Infact I think we should welcome all these technologies but need to be AWARE that there are tons of distractions that the ebook readers come along with. Unless one turns off those distractions, you might not be able to read books in a silent and contemplative way. 

Last week, my friend gifted me Kindle. It was a delightful surprise. The first thing I noticed that it had an awesome interface and easy access to tons of books and was extremely light. The moment I started reading I could not resist accessing kindle store, visiting hyperlinks in the book etc. However once I turned off the wireless and other distractions, Kindle was a charm. I could read for a long stretch of time with absolute ease.  I think the new Kindle is going to be a game changer in the e-book reader business.

Another nice argument put forth in the book is that of memory. Research confirms that there are two types of memories, short term / working memory and long term memory. Long term memory usually comprises schemas, linkages , ideas and short term memory is like a scratch pad which gets rewritten quickly. However for something to go in to long term memory , it has to pass from the working memory. The problem with reading stuff on the net is that since it is disparate,hyperlink heavy , the information is tough to get in to long term memory. On the other hand , linear reading facilitates the passage between working and long term memory. Various experiments have shown that hyperlink , hypermedia text increases the cognitive load and most often times never gets to the long term memory. So, net by its very nature, distracts us. In other words, we get distracted from distraction by distraction. 

An often quoted view is “ Why remember anything at all , if I can access stuff at the click of a button?”, The book gives a fitting reply to such a view by stating that

“We don’t constrain our mental powers when we store new long term memories. We strengthen them. With each expansion of our memory comes an enlargement of our intelligence. The web provides a convenient and compelling supplement to personal memory, but when we start using the web as a substitute for personal memory, by passing the inner process of consolidation, we risk emptying our minds of their riches”

The book draws upon various empirical and experimental research to prove that Net is subverting our capacity for concentration and contemplation , a by-product of linear reading.  It does make one aware that conscious disengagement from the web is imperative in our daily lives.

  Takeaway :

In the choices we have made, consciously or not, about how to use computers and net, we have rejected the intellectual tradition of solitary, single-minded concentration, the ethic that the book bestowed on us. We have cast our lot with the juggler. Unless we consciously disengage from this medium for a few hours daily , we will become more distracted in our thoughts and actions.

Finally!!!

Have been waiting for this kind of plugin since ages. Finally I stumbled on to one today.

Readability : Cuts the crap while reading articles on the net. I think it is pretty useful if you don’t want to get distracted while reading stuff on the web.

Caret Browsing

I find Caret Browsing  , an annoying feature in Firefox . Thankfully there is an easy hack to control it.

“ Caret Browsing is turned on (and off) by pressing F7.”

Rework

 

I am a great fan of 37 signals. I have been using backpackit service for the last 5 years and have always loved the web app for its simplicity and UI design. The fact that Ruby on Rails got a major push because of the founders , is one of the best things to happen for people who believe in rapid prototyping,  as they  could  use Ruby on Rails to quickly test the traction for their web app ideas.  

This book is from the founders which summarizes their learning's from running 37 Signals gig. 

One specific thing which  I found particularly nice which echoes with what I believe, as far as statistical tools are concerned. One can have the biggest and the mightiest of servers, the most expensive statistical software with 24*7 support and still churn out  plebeian data analysis. One the other hand a nimble guy with basic open source components which might cost nothing , and who PUTS in hours working on stuff , can actually churn out kick ass stuff. The authors in the book name one of their essays as, “TONE IS IN YOUR FINGERS”, which is my favourite from the collection of thoughts in this book. It goes something like this:

Guitar Gurus say,”Tone is in your fingers”.You can buy the same guitar, effects pedals, and amplifier that Eddie Van Halen uses. But when you play that rig, its still going to sound like you. Likewise, Eddie could plug in to a crappy Strat / Pignose set up at a pawn shop, and you ‘ll still be able to recognize that its Eddie Van Halen playing. Fancy gear can help, but the truth is your tone comes from you.

Many amateur golfers think they need expensive clubs. But it’s the swing that matters, not the club.

Give Tiger Woods a set of  cheap clubs and he‘ll still destroy you.

- Jason Fried & Co

 

Useful tool

I have often wanted to underline statements / words in a pdf but the security settings of most of the online documents prevented me from doing so. I like the content being in a word doc .

Found a nice software (free) which converts a pdf in to word doc. I am pretty happy with the quality of the output.

Link : HelloPdf

WHY ? ..

NYT article says :

The F.T.C. said that beginning on Dec. 1, bloggers who review products must disclose any connection with advertisers, including, in most cases, the receipt of free products and whether or not they were paid in any way by advertisers, as occurs frequently. The new rules also take aim at celebrities, who will now need to disclose any ties to companies, should they promote products on a talk show or on Twitter. A second major change, which was not aimed specifically at bloggers or social media, was to eliminate the ability of advertisers to gush about results that differ from what is typical — for instance, from a weight loss supplement.

I am definitely against this!. Shouldn't crap reviews/content written , be killed by the "wisdom of crowds" funda. Why is there a need for a regulation ?

Trivia : Why choose PNG over GIF in an app ?

Never bothered to think about this aspect until today, when the choice of the format made a ton of the difference to the output!

Via C# Corner :

First off, why use PNG instead of GIF? Probably the most important reason is that GIF supports a maximum of 256 colors. The second reason is Unisys has a lockdown on the rights to LZW compression which the GIF format can, and usually does use. PNG overcomes the color depth hurdle by providing up to 48Bpp (bits per pixel). Just as importantly, the PNG format is patent-free and available for use by anyone.

PNG has other advantages over GIF as well:

• Alpha Channels: GIF supports binary transparency, that is a pixel can either be opaque or transparent. PNG allows full variable transparency with 254 levels of partial transparency. I should also say that it is also possible to obtain 65,534 levels of transparency, but that is out scope for web graphics.
• Gamma Correction: This allows authors the ability to correct the differences in an image that arise from being viewed by different monitors. Most notably is the difference between PC and Macintosh.
• Two Dimensional Interlacing: Also known as Progressive Scan. While GIF does support this, PNG allows a better representation of the image to displayed as the data is transferred.

Filehash

Handling large amounts of data in R is tricky as R typically loads the entire dataset in to RAM. While this means that computations are going to be very fast, it also means that dataset that can be used for analysis is dependent on your RAM. One solution which I stumbled on was Filehash which seemed to offer me a solution than the usual way that I was going about. I was getting the entire stuff in postgreSQL and then doing computations using R.

Filehash seemed to be great as I don't have to bring another RDBMS in to the work loop. However after spending 1-2 hours on it, I realized it was not that useful for me. It is great if you have key value pairs, but not really great if you have too many columns with dependencies and there is a lot of subsets that you need to work on. In any case, filehash was nothing but DB2 sitting at the back of R. Why should I put my data in to DB2, when a powerful postgreSQL is at my disposal. I have ditched Filehash. There is another package called ff , but sadly it is more useful if you want to sample stuff from a big file. In summary , it appears to me that the only way out is Hadoop-R/Mapreduce-R. That's a steep learning curve for me.

Why should one think of fancy systems like hadoob?
Fact :On a typical day, 6 lakh NIFTY futures are traded in 20,100 seconds of a trading day ,meaning, 30 trades per second.Now each of these 30 trades happen at various price points and quantities. If you want to track the average futures price(quantity weighted) every second for a SINGLE day , it is easy. If you want to do it for a MONTH, it might take some time. If you want to do it for an YEAR and also calculate a 5 minute moving average of futures price, taking in to consideration gapups and gap downs across days, we are talking abt lots of computations on lots of data points. Even in such a simple application using RDBMS systems is a PAIN. I am guilty of using them now, but it will be of no use as things scale.

Future of Search

Via Googleblog:

Marissa Meyer on the future of search

Postfix and Prefix Operators

C++ is a language that requires sustained effort of programming for a years , say, the gurus of the language. May be that is the case..Personally , the language is fascinating in terms of design. Just by looking at why some things are done the way they are in C++, the learning can be immense.

For example - Postfix and Prefix operators -- i++ and ++i . How do you think such a thing is implemented? As you know that a function is overloaded based on the input parameter types. Did you realize that there is no input parameter , at least, on the surface of it ? Then how does it work ? How does compiler know the difference between ++() prefix and postfix..Well, here is something that one can understand and appreciate how things are done. In the case of postfix, compiler silently passes int 0 as an input and in the case of prefix, no parameter is passed.

In our hurry to program away to glory, sometimes, these little inner circles are missed out by us. Do you know that the return type of postfix is a const type and return type of prefix is a reference type..Why in the world was C++ designed like that ?

Well, answer lies in a simple statement , i++++ is illegal , which most of us agree. Double increment doesn't work. WHY ? because i++ returns a const and nothing can be done on const

So, you see, there are some differences in the way postfix and prefix is designed : Return values are different, Compiler passes an input silently in one of the operators and there is a creation of temporary object in postfix.

What's the use of knowing the above stuff ? When you write your own classes, you will keep all these issues in mind when designing your prefix and postfix operators for your custom classes.

Java - typesafe ?

Do you think Java is a type-safe language ? Read Upenn's professor's take on it

Quantitative Finance C++ Library

CODE LIBRARY

Have just started organizing the code that I have written over the last 12 months. Sourceforge is the best thing to use for such purposes. Here is the link to 100 odd classes that I written for pricing derivatives. Need to comment Finite Difference code and upload it . May be I will do it over the labor day weekend.

Link to my QuantFin Library : Source Forge:Quantitative Finance C++ Library

Gruff

If you ever want to analyze data using charts , and , excel -data interaction becomes overwhelming, then one needs to write one's own charting infra..I was in a similar position where excel based charts were not enough for my analysis. There was simply too much data to look at and hence ended up writing my own diagnostic tool using rails-postgres . Gruff a charting library for rails proved to be INVALUABLE.

Link : Gruff . However if you want a little more jazz to the graphs you can look at this link Vertical Labels

Overall, I am happy with gruff for it met my requirement. Don't want anything fancier than what gruff can provide.

Ruby in 30 minutes

I have a problem with Ruby. It is a great language , extremely good for fast prototyping. However this is not the language I use it daily. However, there are times when I have to use ruby where the effort of doing something in a different language is not worth it.

My problem stems from the fact that I forget the syntax as I don't use it regularly. In the last 1 year , I would have probably worked on 4 RoR mini-projects and in every instance I had to pain stakingly go through and review the syntax . This time again I have to develop a diagnostic tool, very very fast and I don't have time to go through the syntax at leisure. I did something different. Picked up this book "Making use of Ruby" and put myself a time limit of 30 minutes for reviewing the relevant syntax..All I can say, is that, the result is pretty good...In 1/2 hr time, one can review the syntax of loops, classes, namespaces, mixins, file operations , instantiations, arrays, hash etc and I must say this book is pretty well organized in that sense. First 120 pages is that would be needed for quick prototyping.

My next task is Rails recap. I just want a quick way to look at price-s score-long/short position data all in one web page and the data is in postgres.The only thing left to figure out is : How to dump the data from postgres in to a chart ? I just hope I figure this out quickly so that I can investigate the results. Sometimes I guess one needs to have patience and develop a diagnostic tool for simple excel based charts are of no help in understanding what's going on...

Quote for the day

"Ease yourself into the language. Don't feel you have to use all of the features, and don't try to use them all on the first day."
- Bjarne Stroustrup on C++

Splendid team effort

Mitchell Baker on his experience at Firefox(record for the highest no of downloads in 24 hrs - 8 Million):

"Every once in a while — for those people who are really lucky — we get to experience a moment where everything comes together. A period where dreams and hard work merge together with remarkable results.

This is such a time for Mozilla.

It’s based on hard work and execution of course. The number of people who have done something unexpected in the last few months, something that changes the outcome, is very high. But that’s only part of it. And there are plenty of times in life — most of life for most people, in fact — where people work hard and pour themselves into their effort but don’t experience the lift and buoyancy of sense of validation.

The periods that are so memorable often involve a team of people, and something that makes that group of people cohesive and satisfying. Sometimes these periods involve working on something that seems giant, hard to achieve and meaningful. Often then involve many things coming together in a way almost didn’t seem possible. And they involve a response from the world at large that demonstrates all the work and energy was worth it."

Blog graph

Analysis of Blog graph by CMU researchers : Interesting observations

Temporal patterns: For the two months of observation,we found that blog posts do not have a bursty behavior;they only have a weekly periodicity. Most surprisingly,the popularity of posts drops with a power law, instead of exponentially, that one may have expected. Surprisingly, the exponent of the power law is -1.5, agreeing very well with Barabasi’s theory of heavy tails in human behavior .

Patterns in the shapes and sizes of cascades and blogs: Almost every metric we measured, followed a power law. The most striking result is that the size distribution of cascades (= number of involved posts), follows a perfect Zipfian distribution, that is, a power law with slope =-2. The other striking discovery was on the shape of cascades. The most popular shapes were the “stars”, that is, a single post with several in-links, but none of the citing posts are themselves cited.

Generating Model: Finally, we design a flu-like epidemiological model. Despite its simplicity, it generates cascades that match several of the above power-law properties of real cascades. This model could be useful for link prediction, link-spam detection, and “what-if” scenarios

Link : www.cs.cmu.edu/~jure/pubs/blogs-sdm07.pdf

Inspiration behind Design Patterns in Software

The Timeless Way of Building

The Timeless Way of Building is a 1979 book that ties life and architecture together, written by Christopher Alexander.

It is considered by many to be one of the major, seminal texts of the twentieth century.[citation needed] It has had a huge influence on creative thinking, especially in the areas of architecture and software design. It could be classified as one of what the Estonian researcher of the Orient Linnart Mäll calls the humanistic base texts.

In the book, Alexander introduces the concept of the "quality without a name", and argues that we should seek to include this nameless quality in our buildings. This quality is hard to pin down in words. Alexander uses the technique of surrounding it with existing concepts that reflect a part of the quality with no name but are not sufficient to define it individually.

One central concept in The Timeless Way of Building is that of creative unfolding. Alexander uses the extended metaphor of the growth of a human embryo - with illustrations - to take the reader through a creative complexification process.
This starts with a single orientation (in building, say, where the new building will be and which way it will face; in the embryo metaphor a single fertilised cell) and gradually unfolds (in building through the thoughtful addition of new features - where the entrance will be, what sort of root it will have; in the embryo metaphor how the spine forms and where the arms and legs will be) into a fully featured, complex yet somehow naturally appropriate design. A completed building; a new born child.

This book popularised the concept of the pattern language.

Link : Book

Bridge Pattern

Via Bridging the gap between Interface and Implementation :

A nice article explaining the importance of Bridge Pattern

Useful Tools

Via : 15 -hot- tools that made me a coding Paris Hilton.

Tivo for India

Why shouldnt there be a Tivo for India ? Why hasn't been there one ?

• Is it because of purchasing power of consumers is limited and would not be willing to invest in an instrument which records their programs of interest ?
• Out of overall 200 Million TV set penetration in India - Rural folks dont mind TV ads
• Marketers still rely on TV for promotional activities and customers are often hooked on to ads. I have seen a lot of folks discuss some of the quirky, catchy ads being discussed over the  lunch table. Does this mean TV as a mode of ad promotion is going to stay for a lot more years
• What will happen to the mobile devices ? Additional connections are 6 million per month . How can one leverage this platform for promotion
• Recent development in terms of sms promotion from theatre services , airline ticketing services, insurance, banking services show various sectors are trying to grapple with the challenges of promoting their product in the 160 characters of an sms

So, can one rule out Tivo type of product in India ?

Mobile Search $20B Mkt in the coming 15-20 years

Mike McCue , cofounder of TellMe networks on Voice Recognition and Mobile search..

Via  Fast Company:

FC: What's the potential market for voice-powered applications?
McCue: In the enterprise segment, it's a multibillion-dollar market. Directory assistance is another billion or two. That could grow by $10 billion or $15 billion in 5 to 10 years. Personally, I love the iPod (NASDAQ:AAPL); I have about 30,000 songs on mine. But it's impossible to navigate with that wheel. It's crying out for speech interface. That could be huge.

FC: What about mobile search?
McCue: That market is now basically zero. Over the next 5 to 15 years, it will be a $15 billion to $20 billion market.

FC: How will that be commercialized?
McCue: It will be ad supported. It's going to take time to develop, but it will happen. We will handle the design, but Microsoft is doing a lot of work with ad engines, ad inventory, and the relationships with advertisers

Emerging Culture of short attention spans

Here’s another evidence of how our attention spans have reduced and firms are catering to that demand.

Sony Television is retooling old series like “Charlie’s Angels” into episodes of under six minutes.

Flickr Id

Flickr Id has immense use in web apps. If one needs to have a gallery that has to be displayed on the website, embedding Flickr slide using Flickr Id is the easiest way . Use idGettr - Find your Flickr ID  to  find your Flickr user or group id.  I am developing a website for an NGO in Bangalore. Displaying their past events using flickr was immensely useful to me. No longer one needs to copy all the images in public/images folder and create gallery layouts in your webpages..Thanks Flickr!

Here's the code for embedding a slide show
<iframe align="center" src="http://www.flickr.com/slideShow/index.gne?
user_id=12345678@N00&tags=YOUR_TAGS" frameBorder="0" "width=500" height="500" scrolling="no"></iframe>

Funding numbers

Internet and Mobile Startup fundings in India
2005 - VC investment in India - 7 deals - $35M
2006 - VC investment in India - 30 deals - $200 M
2007 - ??
any guesses ? Travel, Matrimony and Jobs are the regular ones that are making the splash..Whats the big thing that will kick off in India ? food for thought