Wall Street and the Mismanagement of Software 267
CowboyRobot writes "Last week, a bug in high-frequency trading software from Knight Capital Group resulted in erroneous trades costing almost a half-billion dollars. So, what went wrong and how can they, or any other software developer, prevent something similar from happening again? In hindsight, it's clear that the developers did not verify the code under enough conditions. But the real issue is how these high-frequency trades work in the first place. Robert Dewar at Dr. Dobb's suggests the financial industry needs to take a page from the avionics rulebook, which has very strict guidelines about what code can be implemented due to the high cost of failure in that field. 'High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week's incident. It doesn't come for free, but it certainly costs less than $440M.'"
Not a bug, but a test harness (Score:5, Informative)
Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:
http://www.theregister.co.uk/2012/08/08/knight_capital_analysis/
http://www.nanex.net/aqck2/3525.html
http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill
Completely wrong (Score:5, Informative)
It wasn't HFT software, it was regular trading software. The developers created a build of the software that included a module that generated lots of silly trades as test data. The software was hooked up to the exchange as a live test to ensure it would talk to the exchange correctly. Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.
It wasn't actually a bug; everything worked perfectly. It was more of a configuration management problem.
The only relevance it has to HFT is that if the NYSE limited the rate of trades then a lot less money would have been lost.
One loss of $500m is another's gain of $500m (Score:5, Informative)
This isn't the equivalent of $500m of infrastructure burning down. The money was lost by some investors, but gained by other investors.
Re:That's What We Did (Score:5, Informative)
From the article I read recently, they had just put a new system into place, but they weren't even watching what it did, and had no "kill switch" in place to stop it immediately if it started acting up. Crazy. Maybe the programmers screwed up, but whoever was supervising the installation and running of the thing was a moron.
Re:That's What We Did (Score:4, Informative)
Someone pointed me toward this article on another /. thread about the incident. Apparently, their testing program got packaged in with the code when it was deployed, and it's that test program that wreaked havoc.
The Knightmare Explained from Nanex Research: http://www.nanex.net/aqck2/3525.html [nanex.net]