Forgot your password?
Businesses Bug Programming The Almighty Buck

Wall Street and the Mismanagement of Software 267

Posted by Soulskill
from the of-barn-doors-and-horses dept.
CowboyRobot writes "Last week, a bug in high-frequency trading software from Knight Capital Group resulted in erroneous trades costing almost a half-billion dollars. So, what went wrong and how can they, or any other software developer, prevent something similar from happening again? In hindsight, it's clear that the developers did not verify the code under enough conditions. But the real issue is how these high-frequency trades work in the first place. Robert Dewar at Dr. Dobb's suggests the financial industry needs to take a page from the avionics rulebook, which has very strict guidelines about what code can be implemented due to the high cost of failure in that field. 'High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week's incident. It doesn't come for free, but it certainly costs less than $440M.'"
This discussion has been archived. No new comments can be posted.

Wall Street and the Mismanagement of Software

Comments Filter:
  • That's What We Did (Score:3, Insightful)

    by crrkrieger (160555) on Friday August 10, 2012 @08:51AM (#40944573)

    Back in the late 90s when I was system admin for a trading company, they recruited me from a place that did 911 computer aided dispatch software. My shop, at least, recognized that some of the same reliability issues were at stake, so some people get it.

    • by gl4ss (559668)

      loss stop.

      trading is done against other people or other peoples computer algos, so no matter what I guess you had some sort of loss stop? or pre-allocated funds which were manually accepted to go into trading? puzzling thing about knight capital is that they seemingly had neither and the robot had access to whole company credit as if were..

      • by somersault (912633) on Friday August 10, 2012 @09:48AM (#40945173) Homepage Journal

        From the article I read recently, they had just put a new system into place, but they weren't even watching what it did, and had no "kill switch" in place to stop it immediately if it started acting up. Crazy. Maybe the programmers screwed up, but whoever was supervising the installation and running of the thing was a moron.

        • Maybe they were not a moron. Maybe they were just technical people detached from the business needs--equally as dangerous.

          • by somersault (912633) on Friday August 10, 2012 @10:52AM (#40945983) Homepage Journal

            I'm one of the least business interested type techies you can have, but even I would know that the most important thing when doing stock exchange software is making sure that it's not doing anything retarded, and cutting it off immediately if it does. That's the type of thing that terrifies the hell out of me, and why I wouldn't really want to get involved in financial programming without a lot of supervision. Then again, maybe that means that some of the programmers who go to work for these places are idiots who don't think about consequences..

            • by malkavian (9512) on Friday August 10, 2012 @11:14AM (#40946323) Homepage

              That's the thing about "not doing anything retarded". There's a lot of things that can fit that description. The frame problem is what killed classic AI, and it's exactly the core of this problem. And it's probably more a problem for financial trading than it is for avionics.
              To get around that, you need a base set of heuristics from the experts. That's what a spec document is for, to determine the limits and boundaries along with the exact operation. I suspect that a fair bit of this gets rushed through in the attempt to get an algorithm out that's better able to play your opposition before your opposition gets their own one out that'll toast yours.
              Political pressure comes from on high to "get things moving now, what's the hold up?", and pressure is applied to the front lines to move it.
              Which comes back to a management failure. Some things take time to get right, and you have the option of managing the environment to allow for the latency until things come out right (which is a fairly meaty task, but means you largely go from stable state to stable state), or you can utilise politics to speed things up (and this frequently means corners are cut; always a risk, hopefully calculated, frequently not). This often means going from a stable state to an uncertain one, with the hope that things won't go bad enough, and you can fix stuff on the fly.
              Programmers aren't the ones in complex enterprises that should decide what's sane and what's not. That's for the people who have the experience in the field. If that info doesn't get passed on, it's pointless blaming a programmer (hey, go program exactly what I'm thinking of without telling you, and get it right!).
              Doing things the right way takes time and money. Financials are usually willing to spend money, but they're very used to getting things "now".
              That's something they may need to re-evaluate, and go back to the more old fashioned way of doing things, and taking time to ruminate, and double check.. They'll lose the maximum possible profit point, but keep things stable and still profitable.. Alas, many of them don't consider that acceptable, and want it all, and want it now.

              • by somersault (912633) on Friday August 10, 2012 @11:35AM (#40946651) Homepage Journal

                But the real problem issue here isn't even the buggy software IMO. It was the way the software was put into place, not monitored, and nobody was ready to just shut it off if it started going haywire. According to the article I read, Knight hadn't even noticed a problem themselves - it was pointed out to them by the stock exchange that they were doing a very high amount of trades compared to usual, and it still took them half an hour or more to shut everything down. There should have literally been a big red STOP button in place to shut things down if they went wrong.

        • by superflippy (442879) on Friday August 10, 2012 @12:46PM (#40947643) Homepage Journal

          Someone pointed me toward this article on another /. thread about the incident. Apparently, their testing program got packaged in with the code when it was deployed, and it's that test program that wreaked havoc.

          The Knightmare Explained from Nanex Research: []

      • by fatphil (181876)
        The weird thing is that I remember hearing about Knight Capital a few weeks before this in a similar context - so I think they must have fucked up the same way before (or this is bad reporing, and not "news" any more).
    • by Anne Thwacks (531696) on Friday August 10, 2012 @09:30AM (#40944961)
      You seem to assume that they want to stop the "accidents". You misunderstand the senario - the whole thing is a distributed Ponzi scheme. It works like a sort of "reverse pass the parcel". Each transaction takes a small percentage in brokerage fees as the underlying "commodity" gains or loses value. However, over time, the brokerage fees exceed the true worth of the commodity has in real life.

      The spectacular "losses" are the path by which the money leaves investors, and is syphoned off to pay the disproportionate brokerage fees.

      As a point of reference see gold - the value of gold traded between speculators each day is 1,000 times the value actually sold into industry/jewelery/etc - so a 0.25% brokerage fee on both buyer an seller is worth 5* the value of the gold that enters or leaves the market each day. Obviously, the additional margin actually paid by real life users for the benefits of a liquid market is a tiny fraction of the value of the gold - sometimes there has to be a "software problem" or "rogue trader" or, to use the colloquial term "scapegoat", to provide the money distributed on brokerage fees.

      I am not suggesting any one person is responsible for this - the banks have colluded to look the other way while this system has developed in an ad-hoc manner. In all probability, many of the so called "masters of the universe" are too stupid to understand what is going on, as it requires an understanding of the laws of maths, which they generally don't have. The few "whizz-kids" who could understand are "highly motivated" to look the other way.

      • It seems we need an electronic system that levies no fees, as fees are becoming a dark incentive. Maybe they always have been.

        • by mspohr (589790) on Friday August 10, 2012 @12:34PM (#40947509)

          I think the problem is that Knight doesn't pay any fees so is free to do as many trades as possible at no cost. This makes programmed trading possible.
          If there was even a very small fee (as some people have proposed), then the high speed programmed trading would not be profitable.
          Knight and others doing this type of trading are profiting from very small changes in price which they can see from their order book. They are "front running" the market and this has been illegal but I guess it is so profitable that the authorities are encouraged to ignore the man behind the curtain.

    • Reminds me of the axiom that software correctness cannot be verified/proved with 100% accuracy. Don't know if applies. Also, it takes me three days to settle trades; during that lag time do all trades act as a batch--with each subsequent stock price evaluated in order when they do settle? So all these trades fly around in a manic flurry, and three days later the piper gets paid based on the net outcome?

    • by timeOday (582209) on Friday August 10, 2012 @11:17AM (#40946363)
      I find the comparison to the 911 hotline or avionics (in the summary) almost offensive. This $400,000,000 "loss" is just ownership changing hands in a zero-sum game; nothing was actually destroyed. Even individual investors (who spread their bets) were probably on both the winning and losing sides.

      It's NOT the same as airplane full of people being destroyed or an ambulance failing to show up. In fact, all the money the summary suggests pouring into perfecting HFT software would be a waste and a loss to the economy overall. The real question is how can we fix the incentives to get those HFT developers back working on avionics or 9/11 call centers or something else with real value?

  • by Anonymous Coward on Friday August 10, 2012 @08:52AM (#40944589)

    First 100 trades in a day: free
    Next 1000, taxed at 0.02%
    Next 1000, taxed at 0.1%

    And so on.

    This would do wonders for the problem.

    • by sycodon (149926) on Friday August 10, 2012 @09:08AM (#40944745)

      Outlaw HFTs They just pervert the market.

      • by ShieldW0lf (601553) on Friday August 10, 2012 @09:17AM (#40944835) Journal

        If code like this works better than human judgement does, doesn't that mean we've made rich people obsolete?

        • by Shavano (2541114)
          They long ago decided that the middle class was obsolete.
      • ...or as others have suggested in the past, put a miniscule tax on each trade. That alone would be enough to make it go away. Either way, you're totally correct, there's just no place in the market for that BS.

        • I thought that basically already existed through SEC fees. at least there is a few cents tacked on to every trade i make (on top of brokerage comissions). if the hft's aren't paying this, why not?

        • by gorzek (647352)

          A random delay attached to each trade would also work. I'm not against the transaction taxes, either, though.

      • HFT by itself does not pervert the market ... it creates liquidity which typically reduces volatility as well

        There are 2 main concerns though:
        1. algorithm errors which can cause runaway price movements
        2. front running

        in the first case, it has no impact on long-term investors and typically little impact on short-term traders but can act as a catalyst for wider sentiment moves .. but then so can many things (major bankruptcies, terrorist attacks, move in debt markets, etc) when the human traders are looking f

        • by sycodon (149926)

          I admit I don't know the capabilities of the HFT software. What does it consider? What does it know? What does it value?

          But it seems to me that the act of placing a value on a company is more than just spreadsheet calculations. Apple is a good example of a company that on paper, would not seem like a good bet. Miniscule market share, higher than average prices for it's products, etc. But Steve Jobs was the intangible asset. Does the software understand these things?

          If not, and it is merely reacting to other

          • by tnk1 (899206)

            HFT doesn't care about companies, that's all long term thinking (relatively). HFT cares about whether the graph line of a particular stock rises above or beyond a certain calculated value against itself or an index or perhaps against other stocks in a segment, as very, very basic examples. In short, they are riding market trends, not tracking actual company value.

            It's like playing cards with cards made out of a variety of different materials: gold, platinum, toilet paper, plastic. The card you are holdin

    • I would be enought to tax them at 0.0001% each, it wouls still seriously discourage HFTs, as they work with very small margins, but high volumes.

    • by Shavano (2541114)
      That would make being a large broker unprofitable.
  • by Anonymous Coward on Friday August 10, 2012 @08:54AM (#40944609)

    Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:

    • Running a test system in a production environment sounds like a "buggy system"(albeit a system one level higher in the chain) to me...

      • by gorzek (647352)

        Yeah, it should be damned difficult to "accidentally" run your test system in a production environment. If it's not, then the separation between your test and production environments isn't nearly strong enough.

    • Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:

      That sounds like the time in 1979 when they mistakenly loaded a nuclear attack simulation tape into NORAD's actual online defense system and nearly started WWIII.

  • by G3ckoG33k (647276) on Friday August 10, 2012 @08:57AM (#40944631)

    So, sue the developer for the cost he caused.

    That should teach him a lesson. ;)

    • by kraut (2788)

      So, sue the developer for the cost he caused.

      That should teach him a lesson. ;)

      Good luck recovering $400 million from an unemployed coder

      • by 140Mandak262Jamuna (970587) on Friday August 10, 2012 @09:33AM (#40944981) Journal
        They will sue the unemployed coder for 400 million dollars. Some CXO will certify in good faith he hopes to collect the money. S&P will accept the certificate and rate the credit worthiness of the company AAA. Goldman Sachs will use S&P rating to sell the company to some poor smuck, your 401K mutual fund or my pension fund or our municipalities long term fund at 400 million over the true worth. Everyone involved in the racket will award themselves huge bonus, consultation fee and commissions. That is how 400 million dollars of profits are created, transferred to private individuals and the corresponding 400 million dollar loss for the counter parties are socialized. Either small investors, or government institutionalized investors, or straight forward government bail out. We are always the counterparties who are on the losing end of every such gigantic whoppers.

        America once had a great capitalism. Now we have the system where no matter what risk the rich insiders take, all the profits are theirs and all the losses are ours. A system where the ruling elites are protected from the consequences of their actions, where they can rig the game so that they win no matter what, is how societal collapse begins. Jared Diamond's book "Collapse" discusses specific case studies showing how it collapses. Greenland colonization from Iceland, Pueblo Indians, Easter Island were what he discusses in great detail.

    • I assume this was tounge-in-cheek, but realistically, we're actually headed that way. Licensing boards for software developers, etc. The problem is, as a software developer, it's much more costly to fix 100% of bugs than it is to fix 98% and not worry about that last 2%. If we held every developer to task for every bug in software, we'd have a whole lot less software written, and what would be written would cost 10x as much.
      • by BVis (267028)

        So it's OK that out of 100 planes, 2 crash?

        Yes, bug-free software is essentially impossible. But you can get 99.99% of the way there. Yes, it costs a ton, in the short term. In the long term, you save/make money. But, like every other business in America, they're focused on quarterly earnings, not how the company does over 5 years.

    • by dcw3 (649211)

      Do you know it was the developers fault? It could have been poor requirements. Maybe the customer was offered several choices, and decided to go with the cheap one that didn't include all of the needed error checking.

      This is a lesson in risk management, not only for the financial industry, but any company. Whenever your dealing with the possibility of a large financial loss, or potential harm to life, or property, a risk mitigation plan should be implemented. We do this for every project we work on.

  • Completely wrong (Score:5, Informative)

    by Anonymous Coward on Friday August 10, 2012 @08:57AM (#40944633)

    It wasn't HFT software, it was regular trading software. The developers created a build of the software that included a module that generated lots of silly trades as test data. The software was hooked up to the exchange as a live test to ensure it would talk to the exchange correctly. Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.

    It wasn't actually a bug; everything worked perfectly. It was more of a configuration management problem.

    The only relevance it has to HFT is that if the NYSE limited the rate of trades then a lot less money would have been lost.

    • Re: (Score:2, Funny)

      by Anonymous Coward

      Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.

      And no one immediately caught the "real" vs "silly" trades bug because NO ONE CAN TELL THE DIFFERENCE!

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      You are correct, this was not HFT. But the test software theory that Nanex put out is I believe wrong. First, it does not match the facts, there were not enough trades and enough spread to justify the losses.

      From parties involved in the matter I heard that their agency router had a latent bug, triggered by the changes introduced to support NYSE's RLP. The agency router incorrectly handled out-of-the-money limits as pegged orders, creating principal positions. The desk saw the principal positions, sca

    • by DarthVain (724186)

      LOL somehow to me that makes it worse.... WHOOPS!

      I mean it is one thing to get some complicated trading algorithm slightly wrong that takes advanatge of some arcane finincial construct and have it go out of wack making bad trades... That is bad, but I can understand how it might happen.

      Hooking a testing script to production and pressing the "GO" button is just stupid. I can just see the the sysadmin "ohshitohshitohshit, unplug it unplug it now!" ....670,000 trades completed... you made -420,000,000 dollars!

  • by knapper_tech (813569) on Friday August 10, 2012 @08:59AM (#40944653)
    It's already such a waste that so much talent is getting thrown at problems that seek to make money while producing absolutely nothing. HFT is cleverly sanding in the middle of a river in an eddy and dipping your hand in to tap power without getting pushed downstream. What does Wall Street actually produce? What is their product? Why should we care that they periodically lose their minds and shirts? If anything HFT should be taxed into oblivion so that excellent minds aren't recruited to deliver nothing of social value.
    • Re: (Score:2, Insightful)

      by dkleinsc (563838)

      In theory, what Wall St is supposed to produce is investment directed at useful activity - for instance, if making solar panels is useful, and making fake cold fusion devices is not, Wall St is supposed to ensure that the solar panel company gets investment capital to make more solar panels while the cold fusion company does not.

      In practice, this doesn't happen as well as it should because many investors are stupid and believe the hype (e.g. Facebook IPO), and even more try to profit off of other people bel

      • by Sique (173459)

        It's just the "law of the great numbers fallacy" in disguise. Yes, long term the coin will flip to each side with about the same rate. But for the next coin flip, it's 50%, whatever the current rates are. Each coin flip is completely independent from all the coin flips that happened in the past.

        Yes, long term, Wall Street will funnel investments to the right companies, and if you calculate an average over all trades you will find that with 99.% certainity it works. But the next trade is more or less random

      • by fuzzyfuzzyfungus (1223518) on Friday August 10, 2012 @09:30AM (#40944959) Journal

        The really troubling thing (to my mind) is not so much the 'what does Wall St. produce?' question(which, as you note, is ostensibly 'capital allocation'; but the 'how efficiently do they actually produce it?' question.

        In a non-pathological market situation, you would hope to see Wall St.'s share of the economy as a whole be static or declining(as newer technology makes allocating capital easier and less expensive) and the demand for 'capital allocation' exist only so far as other business sectors find that more efficient capital allocation makes them more efficient and productive(in the same way that you would want to see any other support function of a business kept in line with the business overall. You wouldn't want your IT group consuming a greater percentage of your total economic output every year). Trouble is, that isn't what the numbers reflect.

        Instead of acting as other suppliers do, and having their health and size depend on the success of the customers, the financial services sector has managed to capture a steadily increasing share of the value relative to other sectors. Absolute growth would be one thing, if the economy as a whole is growing; but relative growth, in terms of percentage of total output captured, suggests a substantial increase in the market(and regulatory) power of financial services without necessarily any increase in the value of their product to their customers. That is bad.

        • by dkleinsc (563838) on Friday August 10, 2012 @11:00AM (#40946093) Homepage

          This increase in the relative growth of the financial sector was one of the predictions in Karl Marx's Das Kapital: He saw bond markets and stock markets as the natural and predictable outgrowth of the role of a capitalist, which in his view was somebody who made their living not by producing stuff but by buying the means of production and making other people produce stuff. Bonds in particular simply abstract the concept completely away from any actual work: The capitalist now doesn't even do the buying and managing himself, but buys bonds allowing somebody else to do that work and demands a portion of the proceeds of the firm in return. As the capitalist class gains more and more wealth, the trade in bonds and other financial instruments goes up as a percentage of the economy, while the industrial and agricultural production becomes less important.

          (And no, I'm not arguing that workers of the world should unite and revolt, just that Marx was a serious economist who made some good points about how capitalism works.)

  • Want to stem the flood of HFT software into the Wall Street environment? Pass a law that requires any such software to be written in Ada. Think that one through...
    • You did notice the article was written by the president of AdaCore - developers of GNAT, The GNU Ada complier?

      For situations like this, where there's a fixed day the system has to be ready to (due to the opening of a new market) it may actually not be the dumbest thing you could do... but I don't think that is a common constraint for HFT trader programmers. In general, I think Jane Street Capital is more on the right track with their focus on functional programming. It seems you can get a lot of speed, corr

    • Given the nature of finance these days, PLEASE DO write the software in Intercal. Or in Brainfuck.
  • High frequency trading programmers can't just talk to the government to try to convince them to give a bigger budget for safety. HFT programmers work under time pressure unlike anything in Avionics, because it doesn't just have to be ready by a certain fixed date, millions in profits may be gained for every day you can shave off development time.

    It's interesting, from a programming technical point of view, that functional languages like Haskell and OCaml are used in this domain. They don't offer the correct

    • Re:Not comparable (Score:5, Insightful)

      by MickyTheIdiot (1032226) on Friday August 10, 2012 @09:04AM (#40944693) Homepage Journal

      Is it me or are we continuously using "profits" as a excuse for bad *anything* and pushing that idea to an extreme?

    • The problem isn't the software but the culture. Frankly most code has bugs when released. The problem was there was a deadline and instead of delaying the release until the software was fully verified, they went with the release anyways to meet the deadline. The culture of Wall Street rewards risk taking and abhors rules even if the rules are best practices. I firmly believe one of the leading causes of the meltdown of 2008 was the repeal of Glass-Steagall in the 90s. Commercial banks and investment ban
  • Warning to execs (Score:4, Insightful)

    by jerpyro (926071) on Friday August 10, 2012 @09:02AM (#40944679)

    Maybe it should serve as a warning to executives to not release buggy software. I know a lot of shops that push things out the door before they're fully baked.

    In terms of the stock market, I don't see a problem. The long-term market wasn't affected, no value was created or destroyed, and those who played the game improperly lost out big time. Short term trades on the exchange are gambling. Anyone who tells you otherwise just wants your money. Don't forget, there's always a buyer *and* a seller. Just because Knight lost $450m doesn't mean other people didn't gain $450m.

  • maybe some like FAA code reviews there they just don't let any piece of code go in to a autopilot system.

  • by fastgriz (1052034)
    The SEC usually gives them a mulligan when a software boo-boo costs a big institution a lot of money. I never get a do-over when I make a bad trade though...
  • by Tackhead (54550) on Friday August 10, 2012 @09:05AM (#40944711)

    A "safety culture" has infused the entire industry, with hazard/safety analysis a key part of the overall process. Until the software has been certified as compliant with the standard, the plane does not fly.

    Blair K., Certified Master of the Scrum, responded: "Well, that doesn't sound like a very agile process to me! "Certified" and "compliant with a standard" sound pretty waterfallish. Why not just have a 15-minute standup and decide to launch the plane? At last the aerospace industry could deliver aircraft on time and under budget."

    Customer wants their plane painted hot pink? We can totes do that, bros! Shouldn't take more than 24 hours to get to Home Depot and get a few cans of spraypaint. Delivered! And if bits of paint peel off at altitude and get sucked into the engine, gluing themselves to the turbine blades until catastrophic failure of an engine, well, we can just patch the paint recipe in the next sprint! Paint that's "hot pink" is part of this sprint. The user story about engines that don't fail is part of the next sprint.

    The real problem with aircraft design is that all our little user stories are in a big clunky database. If we printed out the database's contents (by hand!) on little 3x5 index cards, then we'd be using the best practices of both Scrum and Kanban. Our planes would be so damn agile they'd have turning radii measured in inches.

    When a senior engineer piped up that an aircraft with a turning radius measured in inches would kill everyone on board due to G-forces measured in the thousands of Gs, and would likely tear itself apart because the centripetal force far exceeds the tensile modulus of steel, titanium, carbon fiber, or anything else available, he was terminated because "switching from traditional tube-construction to blended-wing-body design made of unobtanium" was part of the next epic.

    • Very funny, but the avionics industry is based on real specs. As in, non-junk specs from people who actually know what they're talking about. In a perfect world, business software would have that too.

      But since it doesn't, doing enterprise systems in formally verified Ada would be equally much of a disaster in practice.

      • by Rob Riggs (6418)

        Very funny, but the avionics industry is based on real specs. As in, non-junk specs from people who actually know what they're talking about. In a perfect world, business software would have that too.

        Exactly. First, imagine a typical American MBA. Now imagine the requirement specs from that MBA.

        If you imagined written specs, you failed.

    • You don't work for Fluent, or do you?
  • by slashmydots (2189826) on Friday August 10, 2012 @09:06AM (#40944713)
    What they really mean is they were too lazy to write "catch code." They spend all that time making it hyperintelligent then lazy out and don't write any checks and stops and tests into it. Just a tiny bit of heuristics would have detected that as a trade that maaaaybe might need a human to review for a second first. Even every version of Halo is the same way. Yes, is "shouldn't" happen but put in a check in the movement engine to see if the player is currently moving faster than running or falling should allow. Tada, no "superbounce" glitch. For MW3, if someone gets a score of 15 kills and 0 deaths with 100% accuracy, MAYBE they might be cheating and should be booted from the game. A little AI goes a long way, people. But nope, programmers are just too lazy. Once the product is "done," they're out of there!
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      You probably aren't thinking enough about the 'problem area' here. Note that I personally think HFT should be taxed into obscurity as it produces nothing and has the potential to cause real damage. But setting that aside, the whole point of these algorithms is to be HIGH FREQUENCY. Ie: fast. If you can get a quote (order) into the market, one ms faster than your competitor, you can make the profitable trade and your competitor won't. Any checks and balances you put in - will result in slower algorithms. It

    • What they really mean is they were too lazy to write "catch code." ...

      Hardly the case.

      Trading systems make money by being the first to respond to incoming bids and offers. Add some "catch code" and the system will lose that race and make no money. Trading system developers routinely examine the code in system calls and libraries to see which calls will execute the fastest and if they can write custom code that will execute faster. It's not laziness but rather a design requirement.

  • When HFT "works," the trading company makes tons of money. When a "bug" hits -- and said bug causes a loss, rather than an unintended gain --, the trading company writes it off its taxes or gets TARP-III to cover. Why worry about bugs? Or, more accurately, it's like being chased by a bear. You only need to run faster than the other guy. Fewer bugs in your HFT code than Other_Big_Trading_Co and you're ahead of the game.

  • by dutchwhizzman (817898) on Friday August 10, 2012 @09:09AM (#40944753)

    It should be called high frequency gambling and taxed as such. It has nothing to do with the (perceived) value but only with a gamble on what the sentiment and competing algorithms will produce as the next stock or derivative price.

    Any derivative trade and any stock trade that is done within 28 days of purchase should be taxed as gambling. It's nothing more or less than that so it's fair if these big online casino's get their profits taxed so the rest of us can profit too.

    • by ed1park (100777)

      Warren Buffett suggested a 100% short term capital gains tax to eliminate all market volatility to foster real growth and investment. So simple, it's genius. Short term capital gains is all investments less than 1 year old. We could start off with just securities.

      Occupy Wall Street should make this their top goal.

    • What I'd like to see, just for giggles, would be a rewrite of the rules for various games commonly gambled on(eg. poker) into the forms and terms of 'legitimate' financial transactions, followed by some test cases in jurisdictions where gambling is tightly regulated; but derivatives trading(that, um, just so happens to be based on the future outcomes of certain playing card distributions...) is not. You might even be able to treat your winnings as capital gains, rather than income, and write off your 'capit

  • by Anonymous Coward on Friday August 10, 2012 @09:10AM (#40944771)

    This isn't the equivalent of $500m of infrastructure burning down. The money was lost by some investors, but gained by other investors.

  • Test People (Score:4, Interesting)

    by stuffduff (681819) on Friday August 10, 2012 @09:15AM (#40944823) Journal
    "Anyway, no drug, not even alcohol, causes the fundamental ills of society. If we're looking for the source of our troubles, we shouldn't test people for drugs, we should test them for stupidity, ignorance, greed and love of power." -- P. J. O'Rourke
  • Cost the company half a bil, the rest of the market says Thanks :)

  • I don't think the problem was a software bug at all, or that they deployed without enough testing. Another article mentioned they deployed their test system with the production software. I think this was probably a packaging issue. Or even a network issue, where they plugged their test system in to the live network accidentally. It's entirely possible the problem is too much software testing.
  • Under normal circumstances you can build a test-suite that will present the software with a large number of scenario's and see if it still gives the 'correct' answer after a code update. However, here the 'correct' answer is not really clear. You are trying to test the results in a highly chaotic system of which you do even understand the mechanics, cannot simulate correctly and cannot measure. On top of that, your software is going to actively influence the dynamics of the very system you are working in, m

  • It makes no sense for everyone to be so concerned about the survival of companies like Knight -- especially people opposed to algorithmic trading in the first place. Just let firms like Knight blow up! Their loss is others' gain, after all.

    There was half an hour of wacky behavior in certain stock prices during Knight's whole blowup process, but that affected essentially zero long-term investors. Long-term investors don't need protection from this sort of incident.

    Now, a flash crash is a bigger deal si

  • Avionics and trading systems differ in a fundamental way.

    Once the plane leaves the ground (or even reaches significant speed on the runway) any malfunction becomes catastrophic. And it becomes catastrophic to third party participants. This warrants the extreme measures taken to vet avionics S/W, H/W, pilots, manufacturing, maintenance and so on.

    Financial systems have no such characteristic. A problem can result in losses that can run up to catastrophic levels only if allowed to run unchecked. And the losses

    • by ceoyoyo (59147)

      Ah, someone who thinks that making software perform correctly involves putting in lots of little band-aid... er, sanity checks.

      It's quite possible to write code that is correct in itself. No checks necessary. In the aviation industry "oh, it fails sometimes but the sanity checks catch it" isn't good enough. In this case, if they'd been a little more careful, they wouldn't have had a problem.

      Software failed in a situation where it shouldn't have. Therefore, it is a software development problem.

  • by torkus (1133985) on Friday August 10, 2012 @09:32AM (#40944973)

    Sorry but this whole idea doesn't 'fly'. Avionics are *NOT* the type of cutting edge technology used in stock market matching engines or HFT engines. Avionics are designed to be utter reliable to such a degree that they wind up using older tech. The deployment and approval cycle is also long enough that 'new' for a plane is probably years out of date.

    HFTs on the other hand, are bleeding edge systems with essentially a cost-is-no-factor approach. You're talking about a world where microseconds are very litterally counted. 10G and higher network connections - not for data throughput but because it lowers latency but a small but appreciable amount. No, they obviously don't want the FUBAR situaion Knight had because of pushing tech but to assume the stock market is using tech with any resemblance to what's in the DreamLiner shows a lack of understanding of both worlds.

    Let's use a car analogy! Sure you can make a race car utterly reliable and safe - it's called a Volvo.* It will undoubtely get you to the finish line for race after race after race with no maintenance while the cars meant for the race break down, crash, need maintenance and so on. Just like race car accidents, you don't usually hear about trading mistakes unless they're spectacular.

    *substitute your own preferred car mfg

    • by ceoyoyo (59147)

      You're right, but that doesn't mean they couldn't take a few hints from other industries.

      Hooking the new software up to run in parallel with the old for a while, without actually executing trades, might be a good place to start.

  • Assess actual damages to other investors on the exchange caused by high frequency trader actions to the high frequency traders that caused the meltdown. Since nobody can absorb a loss of that size without being destroyed, traders will exercise reasonable caution or other investors will wind up owning all of their assets.
  • by ilsaloving (1534307) on Friday August 10, 2012 @09:34AM (#40945005)

    Stop bailing these bastards out every time they screw up.

  • by Remus Shepherd (32833) <> on Friday August 10, 2012 @09:37AM (#40945025) Homepage

    The purpose of avionics is to get a plane from one point to another without incident.

    The purpose of automated stock trading software is to make as much money as possible while screwing the other guy if you get the opportunity.

    You'll never make automated stock software 'safe'. Its purpose is inherently risky and combative. You're not up against the laws of physics and the occasional thunderstorm; you're up against other people who have similar software and an urge to hurt you. This is Wall Street PvP (that's Prick-versus-Prick). It's unsafe by its nature.

    You cannot make competitions entirely 'safe'. What you can do is pen them in so that they do not hurt bystanders. Just like putting crash walls around a NASCAR track, we need to put up regulations around Wall Street so their blood combat does not spill out and harm the larger economy. Re-implementing Glass-Steagall is the least that we can do to keep Wall Street's fiery crashes from hurting the common people. There are probably more reforms we could make to wall them off properly.

  • So, what went wrong and how can they, or any other software developer, prevent something similar from happening again?

    There one sure fire way of preventing this from happening again, and it is a quite obvious one too. Stop.

  • The software was told to buy stock, and buy stock it did. The problem is not in the software per se, it's in failing to recognize an anomalous situation and continuing blindly on. Those who use the software have to sit down and work out why they would need a program that did what it did, and clearly work out parameters so that the software determine when it needs to switch behavior and go into sleep mode or something. Most biological pathways have some sort of feedback system built in that limits the initia

  • Some guys who don't contibute to the world lost a whole lot of money. Well Boo Hoo for them.

    Don't expect me to care that people who were getting something for nothing lost a load of cash.

  • FTS...
    "... very strict guidelines about what code can be implemented due to the high cost of failure..."
    It's other people's money, after all. Who gives a shit if we pissed a bunch of it away with one of our toys? It's not like we're writing software that controls airliners or something.
  • by iplayfast (166447) on Friday August 10, 2012 @02:43PM (#40949513)

    High-frequency automated trading are destroying the stockmarket. With transactions on a stock happening at thousands of times a second, when things go bad they go bad very quickly. When things go well, the normal trader gets reamed.

    IMHO High-frequency automated trading should be outlawed, stocks should be bought and sold at a rate comparable to human interaction. Say 1 per second. Then if things go bad, it goes bad over the course of a day and people can react. Brokers normally buy and sell bulk amounts of stock, so this would be no different.
    It would level the playing field, and put the normal investor at less of a disadvantage compared to the big companies. If a stock is particularly hot, then some trades won't get made by the end of the day. This is in other words reverting to the stock market of old, where the market could be looked at, and expected to stay the same over the course of a minute.

"Text processing has made it possible to right-justify any idea, even one which cannot be justified on any other grounds." -- J. Finnegan, USC.