Forgot your password?
typodupeerror
Businesses Programming IT Technology

Software Defects - Do Late Bugs Really Cost More? 125

Posted by Cliff
from the effects-of-the-software-lifecycle dept.
ecklesweb asks: "Do software defects found in later phases of the software development cycle REALLY cost THAT much more than defects found in earlier phases? Does anyone have any empirical data (not anecdotal) to suggest that this logarithmically increasing cost idea is really true? That is the question I use whenever I want to tick off a trainer. Seriously, though, it seems an important question given the way this 'concept' (or is it a myth?) drives the software development process."

"If you're a software engineer, one of the concepts you've probably had driven into your head by the corporate trainers is that software defects cost logarithmically more to fix the later they are found in the software development life cycle (SDLC).

For example, if a defect is found in the requirements phase, it may cost $1 to fix. It is proffered that the same defect will cost $10 if found in design, $100 during coding, $1000 during testing.

All of this, to my knowledge, started by Barry Boehm in papers[1]. In these papers, Mr. Boehm indicates that defects found 'in the field' cost 50-200 times as much to correct as those corrected earlier.

That was 15 years ago, and as recently as 2001 Barry Boehm indicates that, at least for small non-critical systems, the ratio is more like 5:1 than 100:1[2].

[1] - Boehm, Barry W. and Philip N. Papaccio. 'Understanding and Controlling Software Costs,' IEEE Transactions on Software Engineering, v. 14, no. 10, October 1988, pp. 1462-1477

[2] - (Beohm, Barry and Victor R. Basili. 'Software Defect Reduction Top 10 List,' Computer, v. 34, no. 1, January 2001, pp 135-137.)"

This discussion has been archived. No new comments can be posted.

Software Defects - Do Late Bugs Really Cost More?

Comments Filter:
  • by Pogue Mahone (265053) on Tuesday October 21, 2003 @06:00AM (#7268617) Homepage
    The bugs might be cheaper to fix, but they cost a lot more to find.

    At any stage, you can only find bugs that are introduced at or before that stage. So while fixing a requirements bug in the coding phase might be more expensive than fixing it during the requirements phase, fixing a coding bug during the requirements phase is a tricky operation that I'll leave as an exercise for the reader :-)

    Of course, if you omit some of these phases completely, you won't introduce any bugs during them. That's why the JFDI(*) methodoloy is so popular.




    (*)Just F*cking Do It

    • Replying to self ... sad.

      s/Thigs/Things/
      s/methodoloy/methodology/

      Oh, and I saw JFDI originally in an article in Computer Weekly, I think. It's also here [orangeandblack.co.uk]

      • IMHO the JFDI methodology probably doesn't work very well for large projects (50 people * 2 years).

        But then again, what methodology does work for those cases?

        --
        Real computer scientists don't program in assembler. They don't write in anything less portable than a number two pencil.
        • Well, I've just f*cking done it couple of times. There was big discussion at my previous firm that "something needs to be done". I wrote cool library in two weeks, went to boss and said that "something has been done". He almost jumped throught the roof... I got a nice little raise in salary shortly after that and everybody was happy :)

          Oh, the whole project was 50 people * 3 years.
          • The reality to this is that you already had a concrete design. Not on paper but in your mind. You did not have to describe it to anyone to get it built correctly. So this is not strictly JFDI.

            Get a phone call at 17:00 from a section that you have never heard of and get a project dumped in your lap to be ready by 10:00am, now that is a JFDI.

    • by penguin7of9 (697383) on Tuesday October 21, 2003 @09:17PM (#7277221)
      The bugs might be cheaper to fix, but they cost a lot more to find.

      Not if you are a company like Microsoft or Sun and you let your customers do your bug hunting for you, for free.

      Of course, that's also how bug hunting works for open source software, but with OSS, at least I don't pay anybody for the privilege of finding their bugs for them.
    • (*)Just F*cking Do It

      it's called 'Incremental Development' :)
    • by Anonymous Coward
      That's beside the point. The notion that defects are more expensive if found later is about mistakes of a stage which are found in subsequent stages. In essence it means that if you're at the requirements stage, get it right, because missing or wrong requirements are expensive to fix later (mostly because you either end up with unmaintainable code or have to throw a lot of work away and redo it). You're not supposed to try and fix off-by-one loops at the requirements stage. It usually takes deep insight int
      • Of course, if the design and requirements shift with the wind, you've got another problem entirely. It wasn't too long ago that the company I work for couldn't lock down a set of requirements for the design and code to be built from.
    • At any stage, you can only find bugs that are introduced at or before that stage.
      That's misleading. While it's important to find bugs in each phase, it's even more important to follow practices in each phase that help prevent bugs in later phases, for exactly the reason you gave.
  • Yes they do. (Score:5, Informative)

    by Karora (214807) on Tuesday October 21, 2003 @06:03AM (#7268629) Homepage


    There's plenty of proof out there. Even "ancient" but worthy texts like "The Mythical Man Month" discuss this one.

    The size of the project and the nature of the bug really combine to drastically affect the outcome.

    For me personally we have just spent about a year tracking down a particular set of bugs (probably not all nailed yet) which showed up post-live. When we were pre-live these would undoubtedly have been easier to fix, but something else that we could have done at that point would have been to improve our design, which would have nuked most of the bugs completely. Once we are in production however we have this forward/backward compatibility heuristic tying one hand behind our backs, and redesigning the thing gets much much bigger.

    But that's just anecdotal, of course.

    • by Frobnicator (565869) on Tuesday October 21, 2003 @07:56AM (#7269024) Journal
      Similar experience for me, too. It is anecdotal evidence and not proof of the costs(let us not confuse the two). Now some questions to add to your observations: Should the company be liable for an engineer's errors (as is normally done in business)? Or should the individual or team be liable?

      Most recently I've been tracking down an error in our system. After nearly a month of trying various things, I found the problem of an error. In this case, two years ago the hardaware engineer building the FPGA and DSP programs didn't bother to fix the [relatively simple] design problem. Rather than give all communications the same format, a few commands differ substantially from all others (different responses in certain circumstances, for example).

      The problem made it into the PC software that interfaces with the board. The problem is documented in several [maybe 20?] bugs of the software that works between the PC and the external device. The problem is documented in at least 50 bugs in a port of that PC software. It has been in production for several years, and implemented by external companies (which I feal sorry for, due to the complexity of the communications bug).

      Now we're working on a completely new FPGA/DSP board to replace the earlier board. Design changes prevent us from directly implementing the bug in the new design, although otherwise the communication protocols are the same. Implementing the same malformed communications will mean breaking the simple straightforward design and carefully implementing a set of 'design exceptions' (read: 'bugs').

      It would have taken one engineer an hour or so to fix this thing when they first saw it. It would have taken both teams a few days to fix it when writing the PC to DSP interface (~1 FTE month). It would have taken a few weeks to fix it when writing the port, requiring changes to the PC software and the DSP (~1 FTE year). If we choose to fix the error now, it will probably result 2+ FTE years of work to just fix everything, and more time for regression testing every old peice of software for this one bug. If we choose to leave it in, we will devote at least that much time in evaluating, implementing, and testing the old errors. Not to mention the continued maintenence work when the eventual bugs are found in the new board.

      Now we're forced with a tough financial decision: do we spend a month or more carefully re-creating and testing the 'design exceptions', (probably 3-5 FTE years in total) or do we do it 'the right way' and break both our own and our customers' software? (again, several FTE years, but potentially loosing faith with the customers.)

      This particular bug could have been prevented by about $50 of work. It has now cost the company tens of thousands of dollars, and will probably cost a few hundred thousand before all is said and done.

      Now, lets throw some financial ethics into the $50 --> $5,000 --> $50,000 --> $500,000+ problem: The engineer was in a hurry to fix the problem before a company imposed deadline. Is that engineer responsible for the enormous financial cost? If so, how much? If not, why not? It can be argued that his negligence cause a half-million dollars in damages. It can be argued that the engineer was responsible for $50 but the team was responsible for allowing it to grow. It can be argued that this is a regular business cost due to falibility of engineers' designs.

      This begs the question:

      How responsible are any of us for the errors we introduce?

      frob

      • Depends on the circumstances.

        If the engineer was rushed through the design and nobody had the time to check his designs, the company gets what it deserves.

        However if the engineer skimped his work (for whatever reason) and the team failed to check his work, I think the team would share some kind of responsability.

        If the engineer made the mistake and willfully let it in, he and the team should both partialy liable (he because he left an obvious flaw in and his team for not checking it)

        This is also why I n
      • I have some problems with this way you have reported this story (and maybe I'm taking it personally cause I do hardware.).

        You say the protocol has exceptions instead of always being the same. Do you KNOW that the exceptions were put there to get around a bug? How do you KNOW that a fix for the bug existed - maybe that fix was the addition of the protocol exceptions because for technical reasons there wsa no other solution available to the engineer. Do you KNOW that the hardware engineer saw the bug - or e
        • You say the protocol has exceptions instead of always being the same. Do you KNOW that the exceptions were put there to get around a bug? How do you KNOW that a fix for the bug existed - maybe that fix was the addition of the protocol exceptions because for technical reasons there wsa no other solution available to the engineer. Do you KNOW that the hardware engineer saw the bug - or even defined it as a bug?

          Yes there are two sides, but in this case the other side does not fit any of your reasons. I kno

      • Should the company be liable for an engineer's errors (as is normally done in business)? Or should the individual or team be liable?

        That's an interesting idea that goes right to the heart of how software development is done today. Realistically, at present, a company will have to be accountable for the errors, because if it were all pinned on an individual developer, no-one would risk taking on the job.

        In a better world, software engineering (currently a rather offensive term to real engineers, and on

    • I have to echo this. One of my assignments right now is to fix a post production bug. It happens to be a particularly nasty bug that we haven't been able to reliably reproduce. I think if you factor in the things needed to work around the fallout of this bug its easily the 50-200 ratio described. Even if you don't its still big. There can be a lot of soft costs involved in fixing a post prod bug. One big one is opportunity cost. If I didn't have to spend my time fixing a bug I could be off doing othe
    • I at the time those numbers were calculated, the software development process was very different from today. It was harder to distribute software, harder to deploy updates, harder for developers to get information about errors in the field. Testing the next release was a lot more critical because if a bug did exist it might not be possible to fix for several months until the next release could be sent out via floppy or mag tape to each customer.

      Today most people download their software throught the Interne
      • I think the factor depends a lot on the specific environment the software runs in. When this idea was first proposed, replacing software in the field meant shipping hard copies to users, and for embedded software, replacing PROMs, hence the 50-200 factor. These days the distribution medium is much more likely to be the internet, and at worst even upgrading an embedded system is a matter of plugging in a laptop and reflashing, so the factor might be much lower (5-50 maybe), but the principle still stands.
      • Today most people download their software throught the Internet, and can get patches just as fast, even automatically as they are posted.

        But that's the cost of updating the software. What we're all missing is the cost of the bug. Not the cost of finding and fixing the bug, but the cost of the bug itself.

        Let's say that you're working for a bank and their wire transfer software delvelops a bug during the end of month or (even worse) the end of quarter period. There may only be 5000 to 10000 transactions

    • I've not read the particular references sited and I've not seen anyone care to speculate on the nature of the cost vs time curve (i.e. exponential or logarythmic) but my take on this effect isn't so much about bugs in code as it is about errors throughout the entire SDLC.

      Here is an example. Let's say that the architect for project X decided to use a file based ISAM instead of a relational database to persist the data. Think how costly the fix would be if this error wasn't discovered until the part of th

  • Trade off (Score:2, Interesting)

    by Koos Baster (625091)
    Defects are easier to find in a concrete product than in a conceptual design. Also, many bugs will be introduced in later stages. Therefore, even a full proof design may evolve into a buggy implementation. So surely: there is a trade-off between looking for "bugs" too early and fixing bugs too late.

    Nevertheless a trainer is correct in stressing the golden think-before-you-code rule - especially when instructing unexperienced coders.

    --
    Every program has two purposes -- one for which it was written and anoth
    • Thats not right. You can apply the exact same quallity ensurance measures at any stage of the development, except tests.
      It is equally easy to have a formal review on your design as it is to have a formal review on your code.
      While you "plan" software you should have a clue how you want to esnure that you have: a) what you want, and b) how you want it, in every single stage.
      A standard way is:
      Use Use Cases to capture requrements
      Use Scenarios to refine Use Cases and lay the foundation for later test cases
      Use Wa
    • I think the problem here is that bugs and defects are possibly two different things (not that you missed this distinction). Obviously it's going to cost a lot (either money or time or whatever) to fix a defective requirement that gets fully coded. You'd quite possibly have to start entire pieces of the project from scratch. On the other hand, a buffer overflow error that is introduced early on in the process isn't going to be any harder to fix later than it is if it is caught right away.

      I wonder if the b
      • On the other hand, a buffer overflow error that is introduced early on in the process isn't going to be any harder to fix later than it is if it is caught right away.

        Consider the code:

        void foo(char *input) {
        int a; char buf[25]; int b; /* do stuff w/ a & b */
        strncpy(buf, input, 25); /* a and b no longer used, do stuff w/ buf */
        }

        Let's consider the case of having an input string 26 bytes long including the NULL terminator. If strncpy has an off by one error it'll NULL terminate the string (
    • Bugs are when the software doesn't fulfill the specification; defects are when the specification doesn't fulfill the requirements. These problems are introduced at different stages of development. As one professor put it there are two questions "Did I implement the thing right?" and "Did I implement the right thing?" Early on in the software design it's important to make sure that the specifications that are written for the software actually meet what the customer wants. These are the problems that can
    • Some pro and con points to this discussion:
      1) A lot of times, a "coding bug" is actually the result of an undocumented assumption in the design that conflicts with an undocumented assumption in the requirements. Code bugs can be prevented in the earlier phases of a project by providing as much detail as possible in requirements and design.
      2)Also, a lot of times people call it a code bug because they are too lazy (or don't have the audit trail available) to determine at what point the bug was actually introd
  • Logarithmic (Score:4, Informative)

    by gazbo (517111) on Tuesday October 21, 2003 @06:19AM (#7268665)
    Looks more exponential to me.
    • Re:Logarithmic (Score:1, Informative)

      by Anonymous Coward
      Logarithmic progressions are exponential.

      • Logarithmic progressions are exponential.

        No, they aren't. Think about it. For any n > 0, base > 1, log(n+1)-log(n) < log(n)-log(n-1), whereas exp(n+1)-exp(n) > exp(n)-exp(n-1).

        The parent was correct; the examples were 10^n for n=0,1,2... This is exponential, not logrithmic.

        -- MarkusQ

        • Exponential up, logarithmic down. They are opposites, like plus and minus. If a progression is exponential, it is logarithmic, and vice versa.

          That said, such progressions are generally said to be logarithmic. It is just a convention. Like the fact that geometric and arithmetic series can be thought of as division and subtraction respectively, but we generally don't.
          • by MarkusQ (450076)

            Exponential up, logarithmic down. They are opposites, like plus and minus. If a progression is exponential, it is logarithmic, and vice versa.

            No, they are inverses [2dcurves.com]. If the mapping from a the range to domain is exponential, the mapping from the domain to the range is logarithmic. This does not mean that there is some value A' for every value A such that A^B is the same as log-base-A'(B) for all B. The functions are not the same.

            -- MarkusQ

          • If a progression is exponential, it is logarithmic, and vice versa.

            No, if a progression is exponential then the reverse of that progression would be logarithmic.

            You're confusing "opposite" with "inverse". The next step after +n vs. -n and then n vs. 1/n might be e^n vs. logn.

            Look at it another way. Exponential series grow faster than any polynomial. Logarithmic ones grow slower than any polynomial one. The two cannot be the same: arguably the most fundamental properties they have are opposites.

  • More references... (Score:3, Informative)

    by Bazzargh (39195) on Tuesday October 21, 2003 @06:36AM (#7268712)
    As I recall there was a conference paper in Extreme Programming Perspectives [aw-bc.com] which describes an "infection" model for bug creation, fixing, etc. They were trying to model exactly the effect you describe to see if they could (in a model) find any justification for XP's argument against the increasing cost of bugs through phases. Again, just from memory, they do try to validate the model against figures from real studies.

    There's also material in Watts Humphrey's book on the Personal Software Process [amazon.co.uk] (about as far from XP as you can get). That book is illustrated throughout with statistics about students who tried to complete the exercises in the book, including in Chapter 13, where there's a section on "The Costs of Finding and Fixing Defects.".

    • Uh, maybe bugs found earlier in the development process are cheaper and easier to fix because they're much more obvious bugs! I don't know about you, but as a developer I always fix the easy problems first, and leave the "pie in the sky" enhancement requests for future development...
  • Yes (Score:3, Interesting)

    by Anonymous Coward on Tuesday October 21, 2003 @06:38AM (#7268723)
    Compare the cost of testing, then over-the-air updates to a set of mobile phones & associated risk management
    to
    the cost of just building and shipping new code
    that has yet to undergo testing or launch.

    To give you an idea, managing the testing and upgrading over-the-air softare in mobile phones can become a new project in its own right with all the associated monitoring and overheads.

    Fixing the bug of a pre-launch project can be a 1 minute job.
    • Re:Yes (Score:2, Informative)

      by archilocus (715776)

      Agreed. I've always thought Barry's estimates were on the money.

      Comes down to what you consider a bug ? If you think a bug is a spelling mistake on a web page then 1:5 is probably not far off (but bad enough!!!) If a bug is bad design decision in a mass market product then 1:1000 might be a bit on the light side...

      Don't look back! The lemmings are gaining on you!
    • If you have to create a new project to deal with upgrading code in a distributed environment, you incurred the bug way back in the requirements / design stage of the system. Implementing a fix then would have been pretty expensive, too. Does the cost of that project really change all that much between the beginning and the end of the lifecycle?

      Posit: It costs more because where one massive design problem has been built into the system, there are likely to be several (dozens?) more just as bad or worse. Fin
    • THey actually test cell phone firmware? Have you used a Nokia phone lately???
  • Costs accumulate (Score:5, Insightful)

    by David Byers (50631) on Tuesday October 21, 2003 @07:19AM (#7268871)

    Never forget that complexity accumulates. Fixing the bug itself probably costs about the same at every stage, but other costs are introduced as the project moves along, and peak after the software has been deployed.

    A bug found after deployment has costs associated with it that a bug found during coding does not:

    • Cost of running integration and system tests again.
    • Cost of recertification (if you're in that kind of environment).
    • Cost of deploying the software again.
    • Support costs when only half your customers deploy the new version.
    • Indirect costs caused by using resources to fix bugs rather than implement revenue-generating features.
    • Liability for damages caused by the bug.

    The cost of finding and fixing the bug may be negligible compared to other costs.

    Another aspect of the issue is the nature of the bugs you find late. In my experience, bugs that survive testing and deployment tend to be either bugs in requirements or pretty subtle bugs that slipped through testing, and both are more expensive than the type of bugs commonly detected early on during development.

    • by SnowDog_2112 (23900) on Tuesday October 21, 2003 @08:25AM (#7269170) Homepage
      If I could mod that post up, I would, but my Magic Mod Points are empty today, so I'll just post a little "me too" post.

      I can't point you at any studies, but I think it's common sense. In anything but a fly-by-night shop, the later in the cycle you are, the larger the ripple-effect is from making any change.

      If I can fix a bug in my code before it gets to QA, QA has never seen the bug. There's no bug in the bugtracking database, there's no need to review the bug at a weekly cross-functional bug triage meeting, and there's no need to write specific regression tests that specifically make sure that bug is fixed. There's also no need to perform those specific regression tests on every build that follows to make sure it's still fixed. There's no need to hold a meeting to justify the cost of fixing the bug versus the cost of simply leaving it in and documenting its presence and its workaround. Just there, I've saved a ton of time.

      The costs explode even higher once the software is in the field. Once it's in the field, it hits support every time the bug is reported in the field (multiple times, usually, as of course level 1 support is usually going to blow off the report, tell them to reboot, or whatever the "filter out bogus complaints" method of the week is). Finally it might bubble up through support but only after it gets seen multiple times, costing us money every time. Then it gets argued about by who-knows-who (more time/money) until finally someone tells development it's a bug, and then we have to hold meetings and decide whether it's important enough to fix immediately, whether it should go in a service pack or just the next version, etc. We have to write up a technical bulletin and distribute it, that bulletin has to be reviewed by documentation, product management, QA, and who-knows-who else. Then QA has to specifically add test cases to make sure the fix is there in future versions, etc.

      The costs explode. Again, in any sort of large corporate environment, a cost difference of 100:1 seems completely reasonable to me.
      • The costs explode. Again, in any sort of large corporate environment, a cost difference of 100:1 seems completely reasonable to me.

        I think the key point behind your post is that you're considering the overall costs to a large scale outfit, including all the supporting infrastructure the company has to create to handle bugs from QA and customers. Perhaps a lot of people here are only considering the time spent directly by the development and QA teams themselves.

        OTOH, with a smaller scale outfit the cos

    • by dabraham (39446)
      There's also the question of definition. How much does it cost when
      • some customers who were thinking about buying your product decide to buy CompetitorCo's because they heard that you've had three bug fixes already?
      • you annoy your partners by telling them "Hey, the stubs we sent you to start working against has just changed. Here's the new version."?
      • you burn out your geeks by calling a meeting Friday afternoon and telling them that MajorCustomerCo just found a big bug, and it needs to be fixed by Monda
    • That was a great post, but I do disagree with this one part:

      Fixing the bug itself probably costs about the same at every stage,

      I think one of the reasons fixing bugs later on is more painful is that the developers who were intimately familiar with the code when they wrote it will have moved on, and will take time to get back up to speed. Worse yet, the original developers may have left, and unless they were of the rare breed who both write good comments and leave compact but complete high level docume

  • by oliverthered (187439) <olivertheredNO@SPAMhotmail.com> on Tuesday October 21, 2003 @07:36AM (#7268941) Journal
    The guy in fight club worked out if the cost of a recall and fixing the fault was going to be greater than the cost of litigation.

    I would expect the same kind of factors come into play when the product is software instead of hardware. So why not try google [google.com]

    Sometimes it costs less to pay a person to manually correct data that is incorrect due to a fault in the core of a product, sometimes it's cost less to do a re-write.
  • There are XP proponents that claim that the cost of change curve is still valid. [xprogramming.com](Summary: If you perform work based on a fault assumption, it will always cost you). The real question that XP poses, though is: How do we deal with this fact?

    "The exponential cost curve is mostly in detecting and communicating the Mistake and naming the change that is to be made. XP cannot change that curve, and indeed, XP takes that increasing cost curve neatly into account. So the first lesson I get is that one should not

  • Even disregarding the direct developer cost of finding and fixing the bugs, take a look at a recent example. Enter the Matrix seems to have been plagued with bugs (although I must admit that I haven't played it yet myself). It's "reputation" is now far from good, and so the bugs has caused a large loss in potential sales.

    Now a hypothetical example. You work for a large company, and you're looking for some enterprise-level database software (HypotheticalDB). You get some that has been hyped, and spend a fa

    • OMG, so much for my reputation, too ("it's reputation")... How did I miss that? Now I have to make up for it; Fixing bugs after is much more expensive than if I'd corrected it then and there. :)
    • by joneshenry (9497) on Tuesday October 21, 2003 @09:09AM (#7269540)
      From what I have read [oracle.com], Oracle's founders had the best solution to the problem of customers holding off buying until version 2.0: "This first Oracle was named version 2 rather than version 1 because the fledgling company thought potential customers were more likely to purchase a second version rather than an initial release."
    • Let's further assume that the vendors of HypotheticalDB insist that your server be connected to the internet "for support purposes". When their first customer encounters a bug, they rush to the customer site, analyze the problem, fix it, and patch all the other customers systems over the net without telling the customers. Now they haven't lost any sales, and they've saved themselves the cost of doing massive testing in a live enviroment. (I.e. look at an airline reservation system with tens of thousand of c
  • If you have a good logical design that compartmentalizes each functional unit of your code (what I'll call "well-factored"), how long should it take to fix any one bug? For a typical app, even of pretty hefty size, you should, in theory, be able to run to the exact object, swap out what's broken, and *poof*, every place that functionality is needed is good to go. XP et al really do lose a lot of time in the overhead it takes to keep two people on any programming task, unit test, and the rest. You might be nearly guaranteed nice code, but what's your opportunity cost? In short, it's having two coders hacking about twice as much on what, if they're mature enough, should be well-documented, modular code!

    Now we all know *poof* is not the case, and we all know that a well-factored system is about as hard to come by as nirvana (which means each fix requires ripping out a chunk of code), but the argument is still a valid one. Unless you have a huge system, where perhaps someone's "fixed" a bug by hack on top of hack ("Hrm, Bob's addFunction always returns a number one too low. Instead of bugging Bob, I'll just add one to the result in my function."), bugs today aren't like bugs in pre-object oriented days. If coders in the 80's had the debug tools and langauges we have today... Let's face it, it's much easier to create an Atari 2600 game today [io.com] than it was when you had to burn to an EPROM to test on hardware each time and print out your code to review it.

    The bottom line is whether it's more cost-effective to prevent 99.44% of bugs up front than it is to fix the extra 10% that slip through. I believe the original post is simply suggesting that the cost of fixing on the backside is dropping considerably, especially compared to what the same results would've required decades ago, and that is, honestly, a good point.

    (Remember, this isn't upgrading code -- might be awfully tough to make code that's slapped together change backends from, say, flat files to an RDBMS; this is just bug fixing to make what you've got work *now*. But XP tells us not to program thinking that far down the road anyhow [extremeprogramming.org], so future scalibility is another topic altogether.)
    • bugs today aren't like bugs in pre-object oriented days

      Why? I think bugs today are exactly like bugs in "pre-OO days". Modular code was a good idea then, and is still a good idea now, but I fail to see why OO has anything to do with it...

    • how long should it take to fix any one bug? For a typical app, even of pretty hefty size, you should, in theory, be able to run to the exact object, swap out what's broken, and *poof*, every place that functionality is needed is good to go.

      But often:

      1. The interface(s) will have to change. Bad or poorly documented interfaces are a big problem.

      2. Other parts actually relied on the bad unit behavior and compensated somehow. "Fixing" it may break other things that assumed bad behavior.

      Hiding behind a bl
  • by Technician (215283) on Tuesday October 21, 2003 @09:28AM (#7269769)
    If POP3 could have looked forward and seen the SPAM and Forged header abuses, security could have been part of the standard. Now that POP3 and IMAP mail is everywhere and forged headers are also everywhere, changing the de-facto standards is a big thing. Making the switch to something more robust will be a long and painful transition. Everything will be incompatible for a while.

    It will be as easy as getting the US to switch to the metric system or transition with the rest of the world to driving on the left side of the road. Both would be much cheaper if they were implimented in the beginning instead of attempting a transition later.
    • > It will be as easy as getting the US to ... transition with the rest of the world to driving on the left side of the road.

      I was under the impression that the side of the road on which people drove was split fairly evenly between left / right? Britain, parts of the Commonwealth, and Japan on the left, the USA and mainland Europe on the right, and India straight down the middle except when there's a cow in the way.
    • Very true, but you meant SMTP. POP3 and IMAP are somewhat secured with login methods.
  • Uh.. that depends on the bug. A bug where the grammar and spell checker are switched has a small initial cost to the user, but once they figure it out, it's fine. Fixing it should be near minimal cost. Something in an errata to the manual.. if there was a printed one, and poof. A software bug that costs near nothing.

    If it's a bug where something is off by a dollar per 100 or so transactions, that is hella costly. Both to find if it's not consistent, customer support and all of the other efforts to fix
  • The rule is, 90% of the bugs take up 90% of your budget, and the remaining 10% take up the other 90% of your budget. This goes the same for time before deadline.
  • Yeah, it is so cheap having those MS Windows worms running around.
    I can't see how it would only cost 5 times as much to get millions of users to patch their system, account for their lost time, and write the patch.
    • Uh, Micro$oft just posts the patch to their servers. Patching those millions of systems doesn't cost Micr$oft a cent! And Micro$oft assumes no liability whatsoever for problems caused by their security holes. On the other hand, if they delay a release, their revenue slips. Now do you begin to understand their business model?
      • Yes, it does. It probably costs them 1,000s for each patch (more like 100,000s), so you have to divide that by the number of systems to get a cost-per-patch, which may be minimal but will *always* be > 0. Also, there are bandwidth costs, promoting costs, PR people who spread the word about the patch, techs which post detailed articles on the patch, and probably customer service techs who stand by to deal with their very important customers.

        Since a patch goes out to (hopefully) so many systems, it isn't
        • Actually, customers click on "Windows Update" or have automatic "Windows Update" notification, plus most IT publications announce whenever there are new patches available, so PR costs should be negligible. Yes, there are fixed development costs associated with each patch, but these are probably only twice as costly as doing the fix before release (does anybody really beleive they go through a full test cycle on every patch?) Agreed, the cost per patched system is non-zero, but it may be so low as to be less
  • Depend on the bug (Score:2, Insightful)

    by emmenjay (717797)
    Coding bugs are generally not to tough to fix (though sometimes hard to find). Design bugs are the killer. If you discover a design bug after implementation, you might need to change or even rewrite big slabs of code. The logarythmic estimate is probably a worst case analysis, not an average case. But without a doubt, design bugs that make it into production are bad stuff. That's why sofwtare engineers are either grey-headed or bald. :-)
    • Design bugs are the killer.
      And yet, every place I've worked has given only lip service to design reviews, refusing to spend the time to actually analyze the designs or fix flaws pointed out...
  • What type of bugs? (Score:4, Insightful)

    by Mesozoic44 (646282) on Tuesday October 21, 2003 @10:34AM (#7270432)
    Years ago I worked with a bunch of economists in the US Federal Government - they categorized 'bugs' in their memos into three types:
    Typos: Simple misspellings of words. Infrequent, easy to detect, easy to fix.
    Writos: Incoherent sentences. More frequent, hard to detect, harder to fix.
    Thinkos: Conceptually bonkers. Very frequent, subtle and hard to detect; almost impossible to fix.

    Most 'late' bugs that I've seen in software projects belong in the last category - a lack of design or the failure to make a working mock-up leads to 'thinkos' which are only obvious when the application is nearly completed. These are expensive to fix.

  • by photon317 (208409)

    If define "later phase of the project" as a point further down the line on a scale of code complexity, then it's obviously true. Rooting out a bug is much easier when the codebase is smaller and simpler than when it has grown into a huge complex behemoth.
  • I think that the accumulated cost factors for late bugs is exaggerated by lumping in major design flaws with simple bugs. Surely a bug found during the coding process itself is cheapest to fix (thus XPs pair programming and test first methods) because the hunt isn't going to be as large. However, having worked on the same project for three and a half years, we stumble over hidden bugs from time to time, and they really are not much more difficult than recent bugs to fix.

    Where we find ourselves paying a pre
  • I used to do embedded programming where it was really costly to fix in the field.

    Here's a similar project's repair estimates (pdf). [nasa.gov] Mind you, this product cost 1000x our product, but since they were at similar customer sites, the repair cost wouldn't be significantly different.

    Service trip #1 = $413 million
    Service trip #2 = $497 million
    Service trip #3 = $547 million
    Service trip #4 = $400 million
    (note: these prices don't include airfare)

    In fact, it would be far cheaper to just toss out our old hardware
  • Back when you had to physically send an employee to fix the problem after it shipped, or to send replacement ROMs, it would have been 100:1. Now everbody assumes bugs found after ship are par for the course and builds in software/firmware upgradability over the 'net, it's probably more cost effective to ship with bugs and fix them later, when you factor in the opportunity cost of delaying shipment to be absolutely sure there are no bugs. Many companies appear to operate that way these days (cough, microsoft
    • Now everbody assumes bugs found after ship are par for the course and builds in software/firmware upgradability over the 'net, it's probably more cost effective to ship with bugs and fix them later

      Depends on your context. The project I am currently working on doesn't have any possibility of firmware upgradability, short of replacing the main circuit board (since the firmware is in a mask-programmed surface-mount microcontroller). So not only will a re-mask be required (expensive in itself), but also my c
  • by Anonymous Coward
    I think a little common sense can show that the cost of finding defects in the field is higher, although it depends on the nature of the product. Each time a customer calls support, it'll cost money. If multiple customers call support for the same bug, it will cost even more. In addition, the lost reputation may cost future sales.

    Would you buy a car from the same company again if your current car had a lot of recalls? Is it cheaper for the car company to fix a defect before the car is made, or perform

    • Would you buy a car from the same company again if your current car had a lot of recalls?

      My current car has had two recalls in the three years I've owned it, yet that wouldn't stop me buying another car from the same manufacturer. What WILL stop me by buying another car from the same manufacturer is the difficulty I have had getting them to fix relativly minor problems under warranty.

      Recall problems generally affect a large number of vehicles of the same type - so think of those as "design" problems. The
  • Contractors (Score:4, Insightful)

    by pmz (462998) on Tuesday October 21, 2003 @12:55PM (#7272173) Homepage

    How do you fix a bug cheaply when the contract has ended and all the people working on it are gone? Enter training costs for new staff.

    How about needing a whole new contract just for the bugs? Enter the immoble bureaucracy.

    How about a year later, when, even if someone from the project is still around, it takes them a few days just to remember what they did 14 months ago? Enter seemingly wasted time.

    Anecdotal evidence is viable evidence for the undeniable fact that late bug fixes are very expensive.
  • The same question was asked in an [google.com]
    XP usenet post. In fact, in Kent Beck's book on eXtreme Programming, he discusses
    the cost of changing software in the various stages of the development process,
    and a recognition that this "10-100-1000" progression is not necessarily true is
    a fundamental part of the XP philosophy.
    For what it's worth, I believe there are many domains where that cost escalator
    does not apply. If you have a well-designed application rolled out to a manageable
    userbase with access to a helpdesk o
  • Minor bugs might be okay, but show stopper bugs are definitely more expensive.

    Here's why: when it's early in the development cycle, the team working on the product is small and intimately familiar with it. Finding and fixing the bug takes less time and costs less money in idled staff-hours affected by the bug.

    As the product grows, more developers are added, additional staff (let's pretend this is a commercial effort) comes in, sales, marketing, customer support, et cetra... Now, the same show-stopper bu
  • It's expensive when you have to trash your CD stock because they're unshippable, or when you have to ship CDs to all of your customers three times in two weeks after you release. Try it and I bet you will have all the empirical data you and your wallet need.

    Fix bugs early; it's less expensive that way. =)
  • Sounds like you meant the cost grows exponentially, not logarithmically. But, yes, that makes perfect sense and probably applies to much more than software. Imagine you are a "developer" building a house. You put the foundation one foot too close to the property line, violating zoning laws. Imagine the costs if someone noticed the flaw in the blueprint before concrete was poured vs. after the foundation was poured, vs. after the whole house was built on top of it.

    It certainly depends on the bug, but w

  • You are comparing apples and oranges.

    A bug found in the requirements phase is not the same as someone's misplaced semi-colon.

    Say you were told to develop an inventory management style. You deliver a curses-based terminal app and the customer says "wait, I was expecting a web interface!"

    That is a requirements-gathering bug that will require substantial work to correct after release!

  • "Corporate Trainers" are only mouthing conventional wisdom.

    FWIW, many of the original studies are for software development projects for which (1) the system is unique (e.g., flight control); (2) has never been built before; and (3) "the right people" for the job exist but are too expensive (so you have to do with folks that are less skilled.)

    With these constraints, the figures of 10x, 100x, 1000x help predict life-cycle cost given a larger economic constraint (i.e., scarcity of talent). Estimation is a

  • Ok, the scientific thing to do would be to 1) fix a bug in development, and compare the cost against 2) leaving the bug as is, then trying to fix it while you are in live production....

    Any takers? ;)
  • by crazyphilman (609923) on Tuesday October 21, 2003 @04:23PM (#7274633) Journal
    The cost of a bug isn't in cash per se. Whether a programmer is in-house or a contractor, they're going to be at your shop for the standard work-week at least, right? So they're either fixing your bug or they're browsing slashdot. You pay the same either way.

    The REAL cost of a bug while the project is being coded is in delays to your project, which could push you past deadline. The cost of a bug after the project rolls out is the embarassment of getting caught with your pants down, and of having the inconvenience of pulling people off of other work to fix it.

    So in my opinion, bugs are "cheapest" to fix during the initial design and prototype phase, where you're probably not that close to your deadline and you have some wiggle room.

    They're more "expensive" to fix when you're closer to a deadline and the delay screws you up (for example, find a bug during user acceptance testing and you've got to go back and code, then start the testing all over again).

    They're most "expensive" to fix when you've rolled out the project, the users come to depend on it, and something goes wrong. This embarrasses you and makes your code look untrustworthy, and forces you to scramble to deal with the problem, rolling out a patch, etc, all while dealing with hot-under-the-collar users.

    I think this three-level way of looking at it is a lot more useful than any kind of imaginary mathemagical flim-flam. Forget the numbers, worry about the egg on your face. ;)

  • I worked at a company that produced Telco products (switches, base stations, etc) and we had an estimate of $10,000 per bug found in the field. This was assuming it was a relatively straightforward problem that could be fixed within a few days. If it took longer for the developers to invetigate/fix the bug, the cost went up. The cost was so high because we had multiple levels of support who would investigate before passing the info onto a developer who would confirm the bug's presence and fix it. A new buil
  • by cookiepus (154655)
    This question is idiotic, and in fact given the kind of code I get to work with every day, I should like to punch you in the nuts for asking. But since I cannot do that, I am going to give you a real answer ;-)

    BTW, if you only read one part of my post, read the last paragraph.

    For small, non critical projects the difference is indeed smaller because the complexity is much more manageable. Let's say you're building a house for your dog and your design forgot to specify which way the door should be facing. I
  • This is more complex than just a simple metric. I would agree that the cost of that coding bug could have been 20 minutes to test, 10 minutes to fix and after implementation it two weeks to placate the client, prove my software, etc. So the management of people costs a lot of time hence money. XP does NOT fix this "people perception" problem. Now look on the counter side. How long would it have taken me to construct the testing regime that would have eventually found that bug on an existing product wit

  • There are plenty of dinky little bugs that can get unearthed in the few days before release. These aren't necessarily more expensive to find or fix.

    Really bad late bugs don't happen if you get your requirements straight, and test early and often.
  • Again as mentioned in "The Mythical Man Month", bugs found during the later stages of development (and more so post deployment) are more costly, because:

    • The cost of regression testing
    • Other (perhaps subtle) bugs may well be introduced by fixing another
    The book goes on to say that, post deployment, fixing bugs (and the introduction of new ones in the process) causes the system itself to atrophy. In other words become more unstable, not more stable.
  • Unlike this guy [cnn.com], you aren't likely to survive going over a waterfall these days. A more recent discussion of the cost of change [agilemodeling.com] and a further examination by Alistair Cockburn [xprogramming.com] might be better than reviewing Boehm again.
  • by AJWM (19027) on Wednesday October 22, 2003 @12:47PM (#7282044) Homepage
    ...the cost of the wasted effort down the wrong path.

    For example, if you get a requirement wrong and spend X developer-months designing and coding a subsystem around that requirement, the cost to fix it includes that already sunk cost plus the cost of reworking the design and code to make it conform to what the spec should have said.

    Or consider the case where section II.3.iv of the spec conflicts utterly with the requirements detailed in section IV.2.iii. If you don't catch that early (and assuming its a large project, given the size of the specs), you'll have two different subproject teams off designing, coding and testing to cross purposes and you'll only discover the problem at integration time.

    Sure, some requirements or design bugs are trivial to fix even after coding is almost complete (you got the color of some GUI feature wrong, say). Others aren't (you missed some key requirement that radically affects the way the data should be represented and you have to change all your data structures and database tables).
  • Every method guru agrees that bug will creep into any development effort. The thing you need to do when designing and writing your code, is that it may all be wrong from the design up. I have seen few methods that emphasize on planning your bugs ahead. XP will even declare that YAGNI!
    There will be some points in your design that will have grave consequences if there are bug in it, it is up to the designers/programmers to identify those points and plan the repairs ahead.

    How? I dont know really, I'm supprise
  • by RodgerDodger (575834) on Thursday October 23, 2003 @01:09AM (#7287980)
    Or you can read Alistair Cockburn's proof [xprogramming.com]

    What it boils down to is that if I do something wrong, then at the minimum, the cost of correcting the mistake is:
    cost of doing the wrong thing first + cost of changing it do the right thing - cost of doing the right thing first

    As the cost of doing the wrong thing + the cost of changing it is always going to be larger than the cost of doing it right, you'll always end up with a positive number.

    The rest is about momentum; the earlier the mistake was made in the cycle, the more subsequent decisions were made that are also wrong.

    Note, however, this has nothing to do with the cost of adding new features later. Here, you've got nothing done wrong to start with, and the cost of changing it is equal to the cost of doing it right. What you lose is the opportunity cost, which can be iffy.
  • by renoX (11677) on Thursday October 23, 2003 @01:45AM (#7288110)
    On one end you have Ariane5 exploding because of a software error, on the other end, you have 10 clients within your enterprise which loose time because of software's bugs.

    With such huge range of differing costs for finding the bug before or after the shipping of your product, the "average cost" of bugs is meaningless.

    I think that the only thing to remember is:
    - bugs found late cost more to fix than bugs found earlier (any specific number is invalid)
    - finding bugs early is difficult and can be expensive.

    Of which you can deduce that:
    - if late bugs can cost you very much (Ariane5 for exemple), you want to spend a lot of money on software testing|review at each level.
    - otherwise if tests can cost more than the fix (a small number of internal users with a non-critical software), then maybe you can use the clients as testers, but it must be managed well (tell the users, be in close contact with the users, don't let them wait the fixes too much).

  • For example, if a defect is found in the requirements phase, it may cost $1 to fix. It is proffered that the same defect will cost $10 if found in design, $100 during coding, $1000 during testing.


    In the above example, you're talking about a bug that is an error in getting the requirements down correctly and letting it live all the way out into the field. Such a bug would indeed be quite costly to repair! Most likely it would require a new version of the product. This type of bug might be that the prod

"Don't worry about people stealing your ideas. If your ideas are any good, you'll have to ram them down people's throats." -- Howard Aiken

Working...