Properly Testing Your Code? 472
lowlytester asks: "I work for an organization that does testing at various stages from unit testing (not XP style) to various kinds of integration tests. With all this one would expect that defect in code would be close to zero. Yet the number of defects reported is so large that I wonder how much testing is too much? What is the best way to get the biggest bang for your testing buck?" Sometimes it's not the what, it's the how, and in situations like this, I wonder if the testing procedure itself may be part of the problem. When testing code, what procedures work best for you, and do you feel that excessive testing hurts the development process at all?
the best way to test code... (Score:4, Insightful)
Re:best way to test is to use automated testing (Score:3, Informative)
My company has been successful implementing automated functionality tests with Rational Robot (part of teamtest). If you just take the time to define proper test scripts you can easily redo all functionality tests on various platforms (if you use VMWare or similar sw to simulate different platforms) at the click of a button.
This saves time every release as the developers can focus on finding the really tough bugs instead of running boring functionality tests again.
Re:best way to test is to use automated testing (Score:2, Funny)
-flamesplash
Re:best way to test is to use automated testing (Score:4, Insightful)
I beg to differ. This is how most developers test their code as well, though manually.
If you're just testing to make sure your code does what it is supposed to do you are likely in BIG, BIG trouble. Users (and black hats) do just the opposite.
Focus just as much on making sure your code doesn't do things it WASN'T designed to do. Or risk a CERT or Security Focus advisory...
Re:the best way to test code... (Score:5, Insightful)
The flaw in the thinking is the assumption that all bugs are inevitable. You accept as given the idea that the bugs have to be found and corrected. It actually is possible to avoid introducing the bugs in the first place.
The sad thing is, it is likely true in every case that *avoiding* the bugs is cheaper than *correcting* the bugs. Yet we keep introducing bugs & assuming they will be found & corrected later.
Re:the best way to test code... (Score:5, Insightful)
The question is what type of mistake. Is your program crashing a lot? Then see the above poster. Is your program generating the wrong results? Then the problem is that you have not specified rigorously enough. With good engineering specs, the actual code is just data entry.
When was the last time... (Score:3)
Re:When was the last time... (Score:5, Insightful)
Re:When was the last time... (Score:5, Insightful)
Then you aren't writing particularly complex software. If your users need software that does sophisticated processing, mathematical or otherwise, then the programmer probably isn't the best person to work out how it should do it. This is true whether you're working on software for pricing derivatives, or for tracking shipments in a supply chain, or for controlling manufacturing machinery. That's why there are notations like UML, so that functional experts can communicate unambiguously to the software developers what a system should be doing. A good programmer knows about programming, a good analyst knows about business processes, some people are both, but only with years of experience, and even then, only within a single industry.
The requirements, specification and alanlysis process is what separates software engineering from "hacking".
Re:When was the last time... (Score:2)
But I was referring more to the actual user interface and db support. The formulas are standard and fairly easy to write and debug. But any time a user requests that they have a button somewhere I try to cut them short and just ask them what they want the end result to be. But if they want some custom analysis then yes they need to let us know what the formula is though.
Re:When was the last time... (Score:3, Insightful)
Re:When was the last time... (Score:3, Insightful)
Boss: Go write some code that does some stuff
Me: Well what about this? I need this info.
Boss: Well just start working and we will fill that stuff in laster.
How the heck can I write good when I am hardly told what the application is supposed to do? So I write something, it doesn't take into account the missing details that I asked about. Those get defined two weeks after the thing is supposed to be done. The app turns out terrible and then the powers that be want to know why it has problems. It is incredibly frustrating.
Re:the best way to test code... (Score:5, Interesting)
I'm currently on sabbatical, but I consult 1 day a fortnight for a couple of small local companies who can't afford me full time - all I do there is code review, and they are of the opinion that I more than double the effectiveness of their less experienced programmers.
THL.
Re:the best way to test code... (Score:5, Informative)
Identify the people in your company that have the best "critical eye" for each language you develop in -- and see to it that they get the time they need to really critique code, either during implementation or at least before starting integration testing (preferably before unit testing, actually). It may be hard to convince management the first time around, but if you account your hours precisely enough, the results WILL speak for themselves, in terms of hours spent overall and on testing/integration.
Xentax
Re:the best way to test code... (Score:3, Interesting)
But the way to make solid code is to get each bug out as soon as you put it in.
Over my thirty-five years of professional programming I developed a coding/testing method that produces extremely solid code - to the point that one of my collegues once commented that [Rod] is the only person he'd trust to program his pacemaker. B-) It goes like this:
Design ahead of coding.
Of course! If you haven't spent more time designing than you eventually spend coding, you likely haven't yet understood the problem well enough.
This doesn't mean you have to understad every nitty-gritty detail before you write your first line of code - you'll discover things and change your design as you go. But you should know where you're goiong. And as you go, map the road ahead of you.
Not coding until you understand where you're going is VERY scary to administrators. But it gets you to your destination MUCH sooner than striking out randomly.
Get the modularity right.
Think of the final solution as a knotted mass of spaghetti threaded through meatballs inside an opaque plastic bag. Squeeze it around until you find a thin place, where few strands (representing flows of information) connect the halves of the solution on each side of the bag. Cut the bag in half here and label the strands: You've defined a module boundary and interface. Repeat with each of the two smaller bags until the remaining pile of bags each represent an understandable amount of code - about a page - or a single irreducable lump of heavily-interconnected code (one "meatball"). Then tear them open one-by-one and write the corresponding code.
Debug as you go:
This is the key!
Program top-down like a tree walk, stubbing out the stuff below where you're working. As you write the code, also write one or more pieces of corresponding test-code that produces output that is a signature of the operation of every bit of the code you write, and an expected-output file that is hand-generated (or computer-generated by a different tool, preferably written by someone else, or at least in a different language and style if you're alone).
Use a system tool (like diff or cmp) to compare the results (in preference to writing programs to "check" it, so you don't have to worry whether the test is passing beacuse the code is right or the test is broken.)
Run the test(s) every time you make a change or add code. Make a single change at a time between test runs, get it working and tested before you move on. (This is easy for procedural modules and subroutines. For instance: You can build the running of the test into your makefile, and fail the make if the test fails. GUI stuff is tougher, and I didn't have to deal with it myself. But tools are now available to perform similarly there.)
The result is that your bugs will generally be confined to the changes you just made, drastically limiting your search space and shortening your debugging time.
Do COVERAGE testing, and TRACK it.
...} with a cross across the right if the branch case is untested, across the left if the through case is untested. Switch to vertical bar when fully tested (including hitting the edge from both sides if applicable)
Don't move on until you have exercised every bit of code where you were working. "Exercise" means your tests have been updated to execute every line or component of a line, driving it to its edge and corner cases and extracting a signature that shows they're doing their job correctly.
Automated coverage tools are an inadequate kludge to try to do this after the fact. Unfortunately, they pass code once all the branches have been executed, but have no idea whether they did the right thing. They may test that you hit the edge conditions - but can they tell if the edge is off-by-one? Human intelligent, with its knowlege of intended function, is required.
I developed a style of marking listings to document coverage, which is why I use hardcopy even in this day of glass terminals.
- a vertical bar beside a line that is completely working. Cross-mark across its top (T) at the first of a set of working lines, across the bottom of the last of a set.
- an "h"-shaped mark beside one that represents a partially-tested branching construct (if, for, do, "}",
- For compounds (i.e. "for( ; ; ) {" or " ? : " underline the portions fully tested, put an "h" with crossbar through those partially tested.
- Declarations "pass" when you've checked that the type is right for the job and at least one hunk of client code uses them.
- Comments "pass" pretty much automatically when you think they're right.
- A place where code is not yet present, or where the code above and below is tested but the flow across the gap is not, gets a break in the vertical line, with crossbars, as if there were an untested blank line "in the cracks". (But there should be a comment in there mentioning it. I start such comments with "MORE", so I can find any that are left with an editor. Similarly a MORE comment marks where I've stopped coding for now.)
When the code is done-and-coverage-tested there's a vertical slash beside all of it. (Sometimes you have to add test code temporarily to make something visible externally, but #ifdef or comment it out rather than removing it when you're done.)
The result is like growing a perfect crystal, with flaws only in the growing surface. When you've tested a part you're usually DONE. You never have to return to it unless you misunderstood its function and have to change it later, or if the spec changes.
DOCUMENT!
Co-evolve a document as the project develops if there's more than one on it, or if it has to be handed off to others later. If you're alone, you can get away with heavy comments.
Put in the comments even if you have the document.
Keep the comments up to date.
Comment heavily even if it's just you and always will be. When you come back to the code (especially if you're following this methodology and only get back to it MUCH later) you'll have forgotten what you were thinking. So put it all down to "reload your mental cache" when you get back to it.
The document should be a complete expression of the intended operation of the code - but in a very different and human-understandable form. (Especially not just pseudo-code for the same thing, or "i = i+1; add one to i". Use pseudo-code only for an ilustration, not an explanation.) Remember: Testing can NEVER tell if it's RIGHT. It can only tell if two different descriptions match. "Self-documenting code" is an untestable myth - all that can be tested is whether the compiler worked correctly.
There's more but I have to go now. I'll try to continue later. The above contains the bulk of the key stuff.
Re:the best way to test code... (Score:5, Interesting)
The appropriate quote is "It's just what we asked for, but not what we want!"
I don't think this kind of 'bug' can ever be removed. Despite an understanding of the 'business' side of things, my experience has been that the overwhelming majority of specs suck ... whether it's incomplete definitions, contradictions, or questions about what order various rules things should be in. Coding errors should be few and far between. To have them occur generally means that the writing went too fast ... although, to be fair, given the "I want those changes yesterday!!!" attitude of the modern business world, this situation seems to occur with more frequency now than it did a decade back.
Re:the best way to test code... (Score:3, Insightful)
That is, assuming you have a spec at all beyond some vague handwaving and perhaps some marketing viewgraphs (er, power point presentations).
Specifications almost invariably need to go through a step that I refer to as "debugging the spec" -- looking for internal inconsistencies and contradictions, vagueness, ambiguity, etc. Of course many projects aren't given the time to do that -- meaning that additional time is wasted on the back end fixing all the design and code problems that lack of a good spec caused.
The projects I've been involved with that went fastest and smoothest all started with a good spec that had gone through several iterations of "debugging" and review with the end users. In one case that "spec" was the Nth draft of the user manual -- we "prototyped" the thing on a whiteboard in front of a small group who would be the first users. (Of course you need a pretty experienced designer/developer involved so as not to commit to something that can't be implemented in the available time.)
Of course you need to make sure that the business side signs off on the spec so that they can't later say that it wasn't what they asked for. (It may not be what they meant, but that's their problem -- although it is the analyst/designers responsibility to double check if that's what they really meant if there seems to be an obvious problem.)
And a good spec means that the test cases can be written in parallel with the actual code.
Re:the best way to test code... (Score:3, Insightful)
Re:the best way to test code... (Score:3, Funny)
The most effective way... (Score:3, Funny)
Just use the Microsoft way... (Score:3, Funny)
boom boom
cLive ;-)
Testing (Score:2, Informative)
Testing is good to verify that your code does exactly what you think it does; a lot of the time I produce code that I "think" works, using JUnit allows me to verify that it actually does.
Check out junit.org [junit.org].
For those of you who are sceptic about unit-testing, you should try it. Setting up the tests are not as tedious as one might think, they force you to think your problem through, and maybe most of all: they make your build look cool
Trying to fit in implicit restrictions (Score:3, Insightful)
One wonders how your development has been organized. Everybody here should know the basics of software engineering, including but not limited to:
1) document APIs exactly, including definitions of legal and illegal data sets
2) separate test group from programmers
3) separate quality assurance from both API testing and programmers.
Well, that's the theory. I've never worked in a place where that would have been implemented. Instead, people trying to bring this in have been kicked out. In practice, maybe one should try to get a feeling of each API: how is it supposed to be used? Use each piece of software only in the implicit limits of it's programmer's idea to keep the number of bugs down. Not to mention the obvious coding style mantras.
Re:Trying to fit in implicit restrictions (Score:5, Insightful)
This is excellent advise. In my experience, the most stable code comes from pragmatic design followed up by pragmatic coding.
Design your system thoroughly. Identify every component, and the minimum interface required for that component. Carefully document that interface (API) - use Design By Contract (preconditions, postconditions and invariants) if possible.
Moving targets mean that the API will almost certainly have to be extended - documentation on the design and intent of the component/API as a whole will reduce the pain of this process. The responsibility for this documentation is shared between the design and implementation phases. Pay careful attention to documenting assumptions made within the code, e.g. ownership of member/global variables.
When it comes to coding, start with a skeleton. Put in the API function/method as defined, then check/assert every pre/post condition. Think about how any parameter could be out of range, or violate the assumptions you make. Once you are happy you're checking for all illegal use, you can go on to code the internals.
When coding internals, remember that you cannot trust anything (with the possible exception of other code in the same component). Check/assert the return values (and in/out parameters) of all calls you make. Have a well-defined system-level design for error handling, that doesn't allow the real error (or its source, if possible) to get lost.
As for testing, I'm all for the XP method: write your test cases first. This helps you to think about what you API is doing, how you are going to actually use it, and what you can throw at it that may break it (helping you to lock down the pre/postconditions).
You must use regression tests! Testing is useless if its done one, but the code is modified afterwards. Have a library of test cases, and use all of them. Every time a bug is found, add a test case for that bug, and ensure it is regression tested every time.
Code audits can detect and solve a lot of common implementation bugs. Use them to look for unchecked pointer/buffer use, assuming return values or success/failure of functions, and that asserts are correctly and accurately used.
In my experience most bugs do NOT come from implementation errors, but from developer misunderstanding, especially late in a project or in maintenance, or even during bug fixing! A developer must fully understand the code (s)he is working on, and all the assumptions it makes. Never adjust a non-local variable without first checking all other functions that use or modify that variable, and understanding the implications. Never use a function or method without understanding all the side effects (on parameters and scope/global state). This is why all of this information should be documented, and audits performed to ensure that the documentation is accurate.
Book Recommendations? (Score:2)
Re:Trying to fit in implicit restrictions (Score:3, Informative)
The problem is that very few bugs occur at the purely code stage, and most of them are easy to trace. The real problem is design bugs - they're the killers. The solution at our company is to (a) review, (b) separate development, and (c) follow a V-model quite strictly. We have one group of engineers, where everyone writes, codes, reviews and tests, and for each section, everyone will have a different role so no-one gets stuck just being the tester.
Someone writes a requirements spec, and also writes the system test spec by which they'll prove that the thing works as expected. Writing the test spec forces you to put numbers to your requirements, basically making you self-review the requirements for errors, and it find a lot of bugs. Someone else checks that the requirements spec makes sense, and also that the system test spec matches up with the requirements spec. And typically we spend over a quarter of our time at the requirements stage, bcos it's dead easy to edit a single line in a Word document but it's a helluva lot more difficult to change a zillion C files, test specs, etc.
If the system is simple, we'll go straight to code. But if the system is complex, we do a detailed design (usually in Simulink and Stateflow these days, since we're coding for embedded control systems). The person who does the design MUST NOT be the same person who wrote the requirements - this effectively gives us another review of the requirements, to catch any oddities which can't be implemented. And someone will review that the design is actually meeting the requirements.
The same person who's done the detailed design will also write a test spec to say how to test the code. This will cover all boundary conditions, so any time there's a comparison, for instance, we'd check that it gets ">=" instead of just ">". We've got an in-house tool which allows us to write test specs in Excel and run them on code automatically. And someone will review this to make sure it matches the design and covers all cases.
Then someone writes the code. The coder MUST NOT be the designer - as with the requirements/design separation, this gives us a free review. They'll put that through Lint to check for obvious problems (we follow the MISRA coding standard, with a few exceptions where MISRA didn't get it right), and then run their code through the test spec. If it fails, they'll look for the bugs in the code. Sometimes they find bugs in the test spec; in that case the test spec gets modified. Having an automated test spec means we can run tests on code with zero overhead for repeating the tests.
And then the code will be run against the system level tests, and hopefully everything passes. If it doesn't, the system level test has found a bug in the design, where the design isn't meeting the requirements. Rinse and repeat.
It's worth saying that we're writing code for automotive control systems. How much testing to do is really a trade-off of the cost of testing against the cost of failure, and in our case, failure is not an option!
Grab.
Need more data... (Score:4, Insightful)
And the answer to that is of course: "No, you should test more, and fix the bugs". And of course, looking over your development model to see why you have so many errors might be a good idea (such as formalizing who can commit code, if you've got lot's of programmer at various skill-levels).
But in real-life, many bugs are not that important, and time-to-market and development cost is more important.
So unless you provide us with more data, such as ...
...I don't think anyone will be able to give any good advice either.
Code Review, Code Review, Code Review (Score:5, Informative)
For more information, the developers bible (IMHO) Code Complete (available on Amazon and elsewhere) has some good information on testing strategies and some hard numbers on effectiveness of testing. Good luck.
Re:Code Review, Code Review, Code Review (Score:5, Insightful)
Study after study has confirmed: Code reviews find more defects, per staff hour, than any other activity, including all kinds of testing.
Aside from that, the benefits of having more than one person aware of each change to the code are significant. If George is sick, or quits, or wants to go on vacation, it's not just George who can make the next change.
Re:Code Review, Code Review, Code Review (Score:3, Interesting)
What do you do in that case? Self-reviewing the code is of questionable value -- since you tend to skim over the parts you wrote "because you know it works!".
Re:Code Review, Code Review, Code Review (Score:2, Interesting)
Requirements review and Design review find more defects per person hour than code review, especially when you consider that a single requirement defect will result in multiple design defects, and a single design defect in multiple implementation defects.
But yes, code review is a good plan all the same.
Re:Code Review, Code Review, Code Review (Score:3, Interesting)
Re:Code Review, Code Review, Code Review (Score:5, Informative)
Again, violent agreement. Why? Testing is basically just writing the code again, only in a more restricted form. You take a known input, and then program the output expected (rather than derive it another way) and then compare the two implementations.
Inspection, on the other hand, compares the program with it written in another form: human language. Since human language is generally too vague to execute automatically, the only way to test the equivalence of the two is to inspect.
By far the best inspection book is Software Inspection by Tom Gilb [amazon.com]. His very generous web site [result-planning.com] contains a ton of supplementary material [result-planning.com].
Remember, proving two statements are the same is the halting problem, and is NP-complete (i.e. you must check all possible solutions). Testing is a measure of code against code, inspection a measure of code against requirements. Together they kill a lot of bugs because they find different discrepancies between three statements of the same problem.
Re:Code Review, Code Review, Code Review (Score:2)
Designers should rely on their tools. We're a Java shop here, and we employ everything from JUnit testcases for whitebox testing to JProbe for memory leak testing. We also use JTest for lint-like output. This, combined with selective code reviews of trouble spots gives you the most bang for the buck.
You must remember that code reviews will NOT catch the most insidious and costly mistakes, such as general architecture design flaws, race conditions, and memory leaks. You may catch a few, but peoples' eyes glaze over pretty quickly, and these are tricky bugs to catch.
In short: USE THE TOOLS! That's what they're there for. They've come a long way. They're very good at what they do. And there's no substitute for firing the product up and just plain using it for a while.
Start at the beginning (Score:4, Interesting)
If you look at the guys with really low bug rates, like the NASA guys running the Shuttle control software, they have very separate test and development teams, and a competitive attitude. The test team "wins" if it finds a bug, and the devlopers don't want to look silly.
Some Extreme Programming techniques, such as paired coding may help too.
Re:Start at the beginning (Score:2, Insightful)
For instance, I was reviewing some code a few months ago that said to compute an array of three vectors. Well, we were actually only going to use the third element of each array, so the developer only stored the third elements.
That was OK, but we also ended up documenting that in the detailed design spec, which is thousands of pages in itself.
Most desktop apps can't afford our process and probably don't need it. Embedded software in avionics equipment probably do need this process and can't afford not to use it (I don't know if they all do).
I see a pattern (Score:2)
Thats why they can't design a modular windows. Too busy testing....
Quality is Designed In... (Score:2, Insightful)
Wrong problem.. (Score:4, Insightful)
The main thing: testing does absolutely nothing to minimize the number of defects in a particular application.. There are lots of other things that are as important.. ie: are these defect reports being seen by the appropriate developers and are they being acted on, what types of procedures and communication actually exists between the developer and the QA persons (assuming that they are not the same folk)..
The last point isn't as bizarre as it sounds, I've seen lots of places where a QA person enters bugs, but the developers silently reject them ("its not a bug, that's how the program works")
Testing just tries to discover the presence of defects, by itself, it cannot ensure that your product works perfectly (for an application of even moderate complexity, there may be an exponential number of cases and paths to check, most test cases are written for a percentage of those only).. Because of this, if you feel that you're spending too much time testing, perhaps you need to check if your test cases are appropriate to the situation and stage of development..
Another point is that tests can be automated to some degree or the other, perhaps a scriptable tool might assist in lowering some of the drudgery associated with actually assuring the quality of your software...
rant mode = on...Excessive testing ONLY hurts if it takes people away from development at the early or even middle stages of a project and forces them to run tests on incomplete sections of code.. otherwise, there is NO such thing as too many things...
Irrational users (Score:2)
Well, it's never to much, until you've done every possible thing. Thats one of the advantages of open source development. A lot of people are working, playing, coding with the beta version wich means a lot of things get tested. In my experiance most errors show up with unexpected things, like missing error checks etc. A good point to start testing is to test agains the developer. Ask him everything, what if i do this, what if i do that. If you think crazy enough (like normal users), you will recieve a dozen of 'euh, don't know' answers. Thats where your errors are
Testing is just part of good design/implementation (Score:2, Insightful)
If you have a well thought out and well structured design... and if every single function is documented atleast mostly before coding begins... and if all of the theory of interactions fit in... then only a small amount of testing is required
The main problems which require more testing come when corners are cut over the initial design stages, or their not done as fully as they should be doing, or when people dont actually think about what input users can give
For one of my projects I did ages ago for college, half of the class (which was split to give the same coding ability spread in both halfs) were each asked to write this program and the other half were asked to design it, review the design, make a proper testing strategy, document and THEN start writing it
The second half took a little longer to get their program working, but the first half had more bugs, and spent more time testing
For me (Score:2, Interesting)
I suppose its just a case of using the product like an actual user who knows nothing about it would, right from the first step.
Its been said before, but... (Score:3, Insightful)
When you have a problem with bugs, you need to figure out where in the process the problem happened. Was the unit spec wrong? Documentation? Implementation? Unit testing procedures? Was it a correctable problem caused by the engineer involved?
If you really want to be bug-free, every time one shows up you have to figure out why it happened, and change things.
Personally, I think the biggest one is to make engineers work 9-5. Not 9-7, or 11-9. Tell them to go home at 5, even if they're in the middle of something. Software engineering is a very complex task that takes a lot of energy and concentration to do right. Just like Doctors who work long hours make mistakes (resulting, often, in people dying), engineers who work too long make mistakes too.
Being in the "zone" is often the death of good code. You get lots of cool code written, but none of it is double-checked, none of it is verified to match spec, and it often ends up afterwards difficult to understand.
Now don't get me wrong, I don't do any of this, and my crap is FULL of bugs, but thats what you need to do if you really want to help it. Writing buggy code is like a public works program for QA people. Who wants a hundred thousand unemployed anal-retentive QA people nitpicking something else like your car's inspection or your tax forms? Better to keep them in the software where they can't do any harm
Design by Contract (Score:2, Interesting)
State using Assertions what is expected of the code. Pre Conditions and Post Conditions.
If any of these fail, then throw an appropriate Exception.
Eat your own dogfood.... (Score:2, Insightful)
Mathmatical proofs and unit testing asside make sure that your program dosent crap out when its being used or abused. So get some regular people to bang on it..
Never.. (Score:2, Insightful)
Dedicated testers OTOH are perfect in letting a program die in the most obscure ways. Did you know that you can crash a Commodore 64's BASIC interpreter by just typing PRINT 5+"A"+-5 ?
programming and debugging are the same thing (Score:5, Interesting)
IMHO, programming and testing should be done at the same time in the development stage.
While programming and "bugging" happen at the same time, programming and de bugging/testing should happen at the same time too.
It is very well explained in Bruce Eckel's Thinking in Java [bruceeckel.com]. You should just test everything in the code itself, even if it happens to add some overhead. Once called that function, you want that <something> happens.. so check it in the code.
I know this is not the usual way procedural programming happens. It seems much more straightforward to drop the code as it comes and then check if it behaves correctly.
But if you do so you will often discover that that tests made afterwards ara not comprehensive of all possible situations.
And so you discover that testing and debugging are just unfinished tales, and it is even worst if testers are not the programmers who did the work.
Plus, I hate testing, so I force myself to do the work well and let the code (as long as possible) test itself, even if it makes development slower and boring.
Umhh... i'll preview this post 10 times, hoping it's free from bugs :)
Obviously my code contains no ewwows ;)
Re:programming and debugging are the same thing (Score:3, Interesting)
Re:programming and debugging are the same thing (Score:3, Interesting)
Testing a solution for correctness, or probable-correctness, is usually easier than figuring out the solution in the first place. This means the test code will be smaller and less bug-prone than the code that does the actual work.
An example: sorting a list of numbers. An efficient sorting implementation can be rather complex, but making sure it works is pretty simple, even in C: min = list[0]; for(i = 1; i < length; i++) assert(list[i-1] <= list[i]);
Literate Programming (Score:3, Interesting)
When a programmer is simultaniously coding and documenting thier code, at both the high and low levels, the larger "thought" bugs will decrease in number and severity.
Even if you don't use a literate programming system, often documenting the system before you write it can help make the code more clear.
- Serge Wroclawski
Too many people perhaps? (Score:2, Interesting)
I vaguely remember something similar being said about the space shuttle disaster.
Simple sata test and regresion (Score:2)
But for testing it depends what you testing,
a good general test process for data processes (most functions can be though of as dataprocesses) is
Generate some input test data,
Work out what the results should be by hand.
this is your first stage regression.
Now run the input test data through the application/function
Diff the results against your hand generate file.
Any descrepancies should be resolved as,
A bug in the application/function
or
A Bug in the hand generated files.
Fix all the hand-job problems
Repeat until you test files are perfect.
You now have a second stage regression test,
Known good inputs, and known good outputs
Use the correct test files to fix the application bugs.
If a bug is found that isn't in you second stage regression, then generate test files for it.
Fix the bug in the application(using the test files).
Then run you second stage regression and check that any differences are down to the bug that was found (and been corrected).
Following this process you application should always get better, and a you should soon be able to build up a fairly large sample of test data.
The test harnis is simple enough (just a diff on the files and a bit of code to wrap up the functions.), to prevent artifacts caused by the testing process.
Any-how that's more-or-less what I do for most of my testing, bug fixing.
Re:Simple sata test and regresion (Score:2, Interesting)
very large quantity of data (Score:2)
You could have somthing that generats loads of test data , and use a simple script to check that it's data is ok, and even see what happens when bad data is fed in.
If someone one finds a bug, they generate test cases(and variants).
UAT testing should generate loads of data.
An live envoriemts give you loads of data for full-cycle developments.
Understand your real goals (Score:3, Informative)
This strategy works - lots of shops use it all the time. However, the real premise of the process is that you want to get through client acceptance testing as soon as possible, as long as the result is not dissatisfaction on the part of the client with the software after they've accepted it. As you have noticed this strategy doesn't actually produce bug-free code.
This is not surprising. What you achieve is after all pretty much determined by what your goal was. You (shops in general) need to think hard about what your actual goal is. If your goal is nearly-zero-defects, then the traditional process isn't doing the right things for you. If however, your goal is to obtain milestone payments from your client, then it's pretty good. This is an area where the business goals determine the software engineering processes.
Let's put another hat on and think about what the negative affects of this strategy might be (negative is really defined in terms of what your goals are, but let's be vague about that for a moment).
All of the above factors are unpleasant for those left to maintain the code. Many of them also limit the longer term flexibility of the product and hence the useful life of the software. This feeds back into development processes because limited product lifetimes mean that there is less incentive to change your process to produce software which can persist (i.e. why make the effort to ensure that the system is flexible enough to last through 20 years of changing requirements when you expect the system to be retired after only 7 years?)
You mentioned XP - it offers a lot of techniques that resolve these problems:-
However, XP is best adapted to projects where a single team makes multiple frequent deliveries of code, can work closely with the client, and where the development project continues in the medium to long term. These characteristics allow many of the XP techniques, and this means that techniques taken out of XP may not help projects of a different style.
Having said this, the automated testing angle is a real strength. If testing is done manually, it's time consuming and expensive. Hence people don't do it as much as they might otherwise thing is appropriate. Maintenance deliveries often just undergo regression testing, and faults can creep in which might have been caught by the original unit or integration tests. Automated testing has many advantages :-
Just as a data point, I work on some software that has an automated test suite. The suite contains between 500 and 1000 test cases; the test suite conducts those tests in under 5 minutes on a very old machine. To do these tests manually would take one full-time person at least a week.
The summary is :-
Missing in the discussion (Score:3, Interesting)
Biggest bang per buck: case specific testing (Score:2)
Get back to programming basics. (Score:5, Interesting)
Even when using one of these app builders I read through all the code and put tests and logging into the generated code. Funny that these tools are supposed to make us more productive. My coding and testing every line still beats total time spent on a project since I don't have to go back and redo it later. When it's done, it's done. Next project. I've had comm programs run for over 5 years error free servicing 1000s of users per day. One specialized delivery, billing, and inventory system I wrote was used over 6 years error free and caused the owner to stay with hardware that was compatible with the software (not M$) because the programs always worked. And not a damn bit of it was OO or came from some automated builder tool.
In short, the closer you get to the metal and the more familiar you are with the code that is executing, the better your chances of producing error free programs. Takes longer to market, but then you don't have to redo it forever until the next bug ridden version comes out. Saves time and coders to work on the next version and the customers are always pleased. Get back to the basics. Try it, you'll like it.
Re:Get back to programming basics. (Score:3, Insightful)
The only perverted problem with this approach is that in case you are selling your code to some 3rd party company for example, and you are competing with a set of other companies you might have terribly hard time trying to make the customer understand why your development takes 3 times longer than what is promised in other proposals. This is not so much problem when you have a proven track record clearly stating that the complete time wasted is less using your approach. Still, in some cases, the customer might be stuck to some bizarre outsourcing process with inherently excludes all proposals that exceed the shortest estimate by some magic percent.
The point being that many of current customers for software support this bizarre approach of first jumping to lake and then seeing if you hit a stone or not. Ofcourse its not easy job to decide to pay $100 for a candy, if you can get the same thing with just some dog dung added for $33. Especially if the software provider does not say that the dung is included in the offering.
Hurt yourself? (Score:2)
Testing should always be a part of the development process. The wording here implies that testing somehow is considered to be outside the scope of development and I suspect this mindset is causing a lot of bugs to remain undetected.
It's just like documentation or support, those are also (or damned well should be) integral parts of the development process. Sometimes I think that most programmer's believe that the development process consists of the steps hack, compile, ship instead of the tedious iterative process of analyze, design, code, test.
So what, then, is excessive testing?
Well, as long as you find bugs doing it, it's not excessive.
If your projections predict a bug or two in a specific piece of code and your tests fail to find them, then testing (provided that the test method isn't flawed) gave you a much desired quality assessment of that piece of code - meaning that the testing still wasn't excessive.
Running the same tests over and over on the same code with the same data, now that's excessive, not to mention stupid.
Wanna know a secret? (Score:2)
They write test cases to test A, unit, system, etc. Their team leader approves it, and of course all the tests pass before it goes out.
In the end, there's always some obscure way feat A interferes with feat B, but you're not going to write tests for every combination of keystrokes possible.
If you have user testing (u should), they'll find a score of bugs u didn't. Of course, the users too have a deadline to get stuff shipping and they too want to do the minimum possible.
In the end, nobody involved has a personal incentive to make it perfect. With proper testing procedures in place, everyone has a piece of paper with someone's signature on it which says they passed. They don't feel (too) guilty when there's a bug, (hey, my TL approved it!).
Anyway, how are you going to ask the customer for $20k extra so you can test for an extra week?
Suggestion (Score:2, Insightful)
Close the loop (Score:5, Informative)
The object of finding bugs isn't to result in fewer bugs by fixing them. It's to result in fewer bugs by not writing them in the first place. The developers need to review found bugs on a regular basis, with the objective of changing development methods to avoid them in the future.
It's all fine and good to say "don't write buggy code in the first place," but this sort of feedback is the only way to get there. What makes this so hard in many organizations -- aside from the usual disrespect many developers have for QA people -- is that developers fear that this process is some sort of performance evaluation. As soon as this happens, the focus shifts from finding better processes to defending existing processes: "It's not really a bug," "There isn't really a better way of doing that," "We just don't have time to do it the 'right' way," and so on.
This is why the feedback needs to be direct from QA to the developers, who are then tasked to categorize bugs and develop recommendations for avoiding them. It's the latter that is the "product" required by management, not a list of bugs with developer's names on them. Management should otherwise get the hell out of the way.
Re:Close the loop (Score:3, Interesting)
I agree. For some reason (maybe it's just me) developers are nowadays too full of false pride as well, thinkin: I am the lead coder, analyzing bugs is the job of trainees. In my opinion the situation should be (atleast in some cases) completely opposite, only veteran coders can make correct assumptions and define pre-cautions for future and fix this particular case in the correct way. Otherwise it might just lead to a decline of the original code - making things even worse.
Working with bugs is a tough job, do it with pride! *with the allbugfixers unite anthem playing gently on background*
Re:Close the loop (Score:3, Insightful)
I don't know who your company is hiring but where I work things are as they should be. It's the college kids who are hired for the experience are the one's with cocky attitudes while the advanced developers are trying to push management to get better processes in place rather than just "Fly to the moon by next thursday."
... Then wednsday comes and it's "No wait, fly to Mars instead. But still do it by tommorow".
At the moment our company isn't as structured as it should be. We don't have a QA team or a testing team. It's just management pushing the developers to get thigns done. But the developers are pushing back to say "hey, we need a process here. It's not just writing code. We need to design it first. And that takes time. We also need to implement code review and pick someone who's got the experience to decide what goes in CVS."
So my point is that in my experience it's the inexperienced developers who want to just jump in and write code thinking "it's not my job to fix bugs". I think this has a lot to do with wanting to get that advanced status. But as they grow they realize that they're going about it wrong and smarten up with regards to processes.
--
Garett
Buggy degsign and planning (Score:3, Insightful)
If a system is designed well then you should have far fewer bugs, even if you are using code monkies who don't know a quick sort from a n^2 bubble.
Design you systems well, know your people, Bill's good at that kind of thing and likes it(but crap at ui's say),
Jess loves doing data imports, (may not be that quick, but always does them well).
Fread always designes and produces good/fast systems cores.
Get your developers talking and sharing knowlage, 'I'm, having a bit of a problem' , or 'Who knows how to', are good things for people to be saying, so incorrage them to own up to the inadiquacies, and they won't have them for long.
If you can manage that then your productivity and bug counts should drop dramaticly, and the bugs you do have should be easier to fix.
Developer testing... (Score:5, Insightful)
Developers must test their code.
With a test team backing you up, it becomes too easy to change something, run it once (if at all), and then push it into the next build so the test team can catch your errors. I've found that as a tester, a huge proportion of bugs are simply features implemented where the developer just forgot something stupid. I end up wasting 5 minutes writing a report, my manager assigns the bug back to a developer (hopefully the one who made the mistake but not always), and the developer comes back to the code a week later, spending 20 minutes just trying to figure what s/he wrote a week back.
My point: this wastes 30 minutes of people's time for every little stupid mistake. Pressure your developers to really give a thorough test to the code they write before the check it in, especially if you have a test team, because you just end up wasting more people's time.
I would go one step further... (Score:3, Insightful)
...and say, "Developers should write their test suites BEFORE they write their code."
We have a fairly large open source project [faeriemud.org] with contributors coming in and going out all the time (well, not a lot going out; but any number is a problem there). Our experience shows that if you can't write a test suite you're not ready for anything more than a crude prototype. The problem with test-after-coding regimes is the testing gets short-circuited. You've already got working code. You "know" it works. You're just proving it works. So you test the obvious stuff that proves this.
Since we have instituted this policy, coding efficiency has actually improved. Coders who have tried to devise a complete set of tests have formalized their understanding of the requirements in a sense which the most complete requirements doc will never do. We include the test suite in CVS. Nobody commits until their update passes the entire test suite. This results in an enormous (but complete) test of everything done so far. But you can't imagine the thrill of seeing your patch pass that many tests the first time.
All of which is completely separate from what a QA process is for.
Speaking from a non programmer point of view (Score:2)
I think many people would be happy to wait a bit longer for better products but the industry has brainwashed them into thinking that its almost easy to bung a new version out.
The best way to avoid mistakes is not to make them in the first place but thats not so easy when working on a compressed cycle with management on top of you - its not just programmers who deal with it - its network designers, SOE architects etc etc
Bugs exist - they always will - but minimising them requires time and time is a commodity not readily found. I dont know what the solution is - as i say its just my thoughts.
Product or process problems ? (Score:3, Interesting)
First thing to do : look in your bugtracking software ( you DO use bug tracking software, right ?) , and try to isolate hot spots. Is there a particular piece of code that generates more bugs than others ? Is there a common pattern to the bugs (ie. memory not being freed, of-by-one errors etc.) ? Are they _really_ bugs or mis-interpretations of the requirements or the design ? In my experience, the 80/20 rule applies to bugs in spades - it is just hard to find the patterns.
If you need to, make the bug categorisation in your bug tracking software more specific. Once you get an idea of what your hotspot is, you can work at fixing the cause of the bugs.
If it's a particular piece of code, make sure it's reviewed by the best developers/architects you have, and consider refactoring it. At the very least, insist that it is reviewed and tested thoroughly before chec-in to the source code control system, and consider adding a hurdle to jump prior to check in (e.g. get the manager to sign it off).
If the code was written by one developer, consider swapping them out and giving it to someone else - it may be they're in over their head.
Make sure you increase the number of test cases for this piece of software, and check for "edge cases" religiously - if the code is broken at all, it is likely to be broken in more ways than you realized.
If it turns out that the problems tend to have a common cause (memory leaks, of-by-one errors,etc.) consider a structure which forces developers to concentrate on those issues before checking in code; again, consider the hurdle ("software must be signed off by the off-by-one guru prior to check in"), and hone your tests to check for these kinds of errors if possible.
If the bugs stem more from misunderstood requirements or designs, beef up those areas. Work on your requirements and analysis processes; consider training courses for the developers to get them up to speed on interpreting these nebulous documents, and look at improving the review process by having designers present. Frequent "mini-deliverables" (another concept stolen from XP) will help here too - get your team to deliver a working piece of code - it need only be a minimal sub-system - and get it reviewed by designers and analysts. If the bugs tend to occur on the boundaries - i.e. invalid API calls, invalid parameters etc. - consider design by contract or aspects.
Finally, there's a bunch of hygiene stuff
N
no one size fits all process (Score:3, Informative)
That being said, here are some remarks that make sense for any project:
In general a testing procedure that gives you no defects just indicates your testing procedure is bogus: defect free code does not exist and no test procedure (especially no automated procedure) will reveal all defects.
The XP way of determining when a product is good enough: write a test for a feature before you write code. If your code passes the test it is good enough. This makes sense and I have seen it applied successfully.
A second guideline is to write regression tests: when you fix a bug, write an automated test so you can avoid this bug in the future. Regression tests should be run as often as possible (e.g. on nightly builds). All large software organizations I've encountered do this. Combined with the first approach this will provide you with a growing set of automated tests that will assure your code is doing what its supposed to do without breaking stuff.
Thirdly, make sure code is reviewed (by an expert and not the new script kiddie on the block) before it is checked in. Don't accept code that is not up to the preset standards. Once you start accepting bad code you're code base will start to deteriorate rapidly.
Testing is a start (Score:2)
Testing/debugging is like finding and putting out existing fires. If the organization can use test results to prevent future fires, then you're a step above. And probably more advanced than 90% of the software houses, too :-)
How I Test (Score:2)
Hold on! Don't use testing to fix a poor process (Score:2, Insightful)
Testing is necessary but not sufficient. There must be a way to capture requirements, convert requirements to design, convert a design to implementation, and finally test. At each transisition it is a good idea to make an assessment of how well you accomplished your task.
Skipping any of these steps and putting if off until test is pure folly. An extremely false economy.
Assault it with random data (Score:4, Insightful)
Just generating random data and trying to load it caught a lot of bugs, but even more effective was to take a valid image and modify the bytes in it at random, and then try to load it.
Of course, the reason this was so effective, is that the loaders would get mostly what they expect, and then suddenly something illegal. This is the kind of thing you tend to forget about when you write code.
Since it is so easy to attack your program with random data, this kind of testing gives you a lot of bang for the buck, but on the other hand, the bugs it find may not always be those that are likely to occur in practice.
proper design + inspections ? (Score:2)
The disadvantage to testing is that you detect errors, but need to spend time finding the source. If you avoid the error by detecting it at design-time or during code review, you will spend less time dealing with it, since you will know more about the root cause to begin with.
101 little bugs in the code (Score:2, Funny)
Hundred and one little bugs in the code
Hundred and one little bugs in the code
Fix the code, compile the code
Hundred and two little bugs in the code
What kinds of bugs are you finding? (Score:5, Insightful)
Programming is the same way. What kinds of bugs are you finding? Are they just stupid bugs, like buffer overflows or off-by-ones (good design, bad implementation), or are they unhandled errors, or are they API mis-matches or faulty algorithms (bad design)?
Have you made any effort to go back and say "Gee, we are getting a lot of off-by-one errors. OK folks, we need to think about our loops."?
And when you find one type of bug, do you go back and identify anyplace else a similar bug may exist?
If you are hitting high and right, and you never adjust your sights, you will NEVER hit the target consistently. If you never feed back the CAUSE of the bugs, you will never eliminate them.
Re:What kinds of bugs are you finding? (Score:4, Insightful)
Evaluate your test suite for coverage (Score:2, Informative)
It's amazing how poor coverage can be with a naively written set of tests. Ideally you want to write the tests so that the coverage comes out good, but in practice you may have to patch the tests with more tests to cover the parts you missed. You may also have to change the code to make it easier to cover.
Rare error cases (like malloc failures) can be hard to cover.
PC board testing analogy (Score:4, Interesting)
A number of years back I wrote test programs for printed circuit boards. First you created a model for the board that simulated the logic circuits. You then wrote test patterns that were applied to the board's inputs, and the simulator model predicted the board's outputs. The inputs together with the predicted outputs were applied to a real board that you wanted to test, and if this test program passed you assumed that the PC board was good with a high degree of probability.
One mode of the simulator allowed you to simulate faults that might occur on the board. The simplest kinds of faults were physical IC pins "stuck-at-zero" and "stuck-at-one" (these were the most common faults in real life), and if you wanted to be thorough you could also simulate "internal" faults down to the gate level.
I worked in a contract test programming house, where the contract with the customer required us to produce a test program with a specified minimum level of fault coverage, usually just at the physical IC pin level to minimize cost of developing the program. This ranged from say 90% for cheaper commercial work to 99%+ for certain government contracts. With >95% coverage, the "real life" fault coverage was maybe one or two "dog pile" boards out of 1000 would pass the test program but fail a system test.
The point of this is in that business, there was a clear objective measure of a test programs "quality". The measure wasn't perfect, but it was far better than just blindly writing a test program based on a "gut feel" for how the board should work. In addition, the test programmer had a clear, objective goal.
I think a useful tool in the software business would be a measurement of the percent of lines of code that were actually run during the QA process, along with a log of those lines that were not run and not run. Often there are big chunks of code that only get triggered by very special conditions, and there is no way QA can guess those strange conditions. The standard QA process is very subjective; there is no objective measure of any kind as to how thorough the testing was, other than just documenting a list of features that were (often superficially) exercised.
A more sophisticated tool could go beyond lines of code and into log the various logic combinations exercised in "if" statements, etc.
Several years ago I wrote an experimental tool that did this for a specialized database programming language. Basically it rewrote the program with a logging call after each statement (and yes, the "QA version" ran very slowly). The results were quite eye-opening, revealing chunks of "dead code" and conditions no one ever thought of testing. Unfortunately the project kind of died.
Many languages have "code profilers" that are mainly intended to analyze performance, but many of them could be easily adapted to become QA quality measurement tools.
Do these kinds of tools exist, and if so why aren't they more widely used?
Evolve the Tests (Score:2)
Sure, build a big suite of tests to run and check for things to go wrong. Every bug fixing process suggests it own test.
Then you find out that you don't have the time and resources to run all the tests everytime someone makes a change to the codebase.
So, use smaller suites of the faster tests and weed out some of the ones that have been ironclad passes for the last 5 dozen code checkins. For frequent testing it makes sense to only shake what's new and rickety, not what's stood through 10 hurricanes.
Run the exhaustive complete test suite infrequently, say when a release is imminent, or as often as you can afford to spare the resource cycles.
Architecture Problems (Score:2)
Code which is "forced" into a paper architecture is sometime worse that code with no architecture at all. In many of my projects, parts of my architecture change part way through so that the code will work better. Sometime not everything can be thought of before hand. OO programs have a lot of information to fit in a human barin at one time, problems are bound to show through. I don't have any "high eng tools" to help with the architecture either, which doesn't help.
Also, the architecture itself may suck.
What kinds of problem are you having? I think you need to design test routines geared towards not letting the types of problems you currently have through. It is hard to have any specifics, since the post was so vague.
-Pete
Methodology and testing (Score:3, Insightful)
With the waterfall model, you really need to know way ahead of time that what you are coding is what will be desired in the end product. It forces you to have a clear picture in your model of what you are trying to build and with each step in the process, you must develop testing procedures that address that level of the code. For example, at a high level, you may say, let's build a compiler, and following that decision, you need to devise a test that proves that the compiler works. The next phase, you may say, let's build an assembler to produce machine code for the compiler. Then you need to build tests that prove that the assembler works. This methodology continues right down to the smallest module of code, and when all of the pieces have been written, integration testing begins, and you make sure that each larger piece can correctly function based on the output of the smaller piece.
However, in the spiral model, it allows for a well-defined core code to be produced with tons of modules that evolve as the spiral expands. Integration is a function of the spiral, and testing occurs within each iteration of the spiral loop. Code produced with the spiral model also tends to be somewhat more difficult to test in later stages, IMHO, due to the nature of the testing that occurs at each cycle in the loop. Testing becomes more critical in later stages as the previous stages become more nested into the core of the program.
Well, enough Software Engineering for one day. Back to work....
Let the User's Do It (Score:5, Funny)
Make sure it compiles and runs and then upload it to Debian/unstable.
(Yes, I'm joking).
"...and do you feel that excessive testing hurts the development process at all?"
If didn't hurt why would you label it "excessive"?
testing throughts (Score:2)
Larger tests don't test more. What the large tests do is make sure everything works together. You need the small tests to make sure each piece actually works.
The bigger the test, the more likely that your testing platform doesn't resemble production.
There is a big difference between getting your test to run sucessfully and having bug free code.
Chances are the test cases have nothing to do with hiow the users actually use the program. Chances are the programmer has never actually seen how a user uses the program. Chances are, the first time he does he'll go back to his computer and start cursing the user for not doing things the "right" way.
If something breaks and you don't add that something to the test case you're asking for it to break again.
Testing deserves powerhouse machines and sadistic maniacs who like to break code. People who want the tests to be sucessful and don't run them that often are obviously not going to be as nasty as the real world. Even the sadistic manic has a hard time being as nasty as the real world.
Tests are less expensive than that production errors. But only if they find errors. Tests that prove the code works perfectly usually don't.
No programmer likes to be told he made a mistake. On the other hand they love a challege. Make testing fun and brutal and it'll be much more productive than if you make it boring and painless.
The point of testing is to FIND the bugs (Score:2)
I find that.. (Score:2, Interesting)
double-blind testing is when you give the code to a willing party and just let them work with it like they normally would for business purposes, without letting them know it is a beta testing. You have to also include some type of bug report that people can fill in if they wish, but try to encourage them not to cause bugs, and just work with the program as if it was normal. This allows you to see if any of the normal functions that people use everyday would be buggy.
Black-box testing works great to Just test the programs function calls and modules. When I do BBTesting I usually give it to another party with instructions as to how the functions are called and utilized. This party knows how to test the extremes and the common values and give me the best testing.
White-box testing is testing that involves intricate knowledge of the code. When I do this it is usually in development. At the end, if I feel like I enjoy pain I will do a through white-box testing suite for the program, but that has only happened once or twice.
In expenses, the cheapest form of testing is BB testing, followed by Double Blind, and then WB. Since white box testing takes a long time to design run and analyze the results I find.
There's some thoughts for you though.
More testing != Better Quality (Score:5, Interesting)
The Jelly Bean model goes like this: Let's suppose you have a big vat of red and blue jelly beans. Your objective is to remove all the blue beans. You do this by reaching in, grabing a hand full of beans, throwing away all the blue ones, and dumping the red ones back in.
At the begining, it will be very easy to find the blue beans (assuming the blue-bean density is high), and towards the end, it will be very difficult (since the blue-bean density will be low). If you graph the cumulative number of blue beans you remove each day, you'll get a exponential curve; quite steep at the begining (high rate of discovery) and which flattens out as you approach total bean removal.
Software defect discovery follows this model exactly. Defects are easy to find at the begining if there are a lot of them, and hard to find towards the end. This means that if your defect discovery rate is pretty much constant (with respect to the number of hours of testing you've done) then you're probably still way down in the very first part of the curve, and your number of defects is probably very high.
Here's the important thing to remember though; the quality of your product has nothing to do with how many defects you find and fix during testing. The quality of your product is determined by the number of defects remaining! If you find and fix 10,000 problems, you might think you're doing very well, but if there are 10,000,000 defects remaining, your product is still crap.
You can estimate the number of defects remaining by trying to fit the number of defects you've found so far onto that exponential graph I mentioned above. The most popular method to use a Weibull curve, or Quadradic Regression.
Now, why is testing to improve quality a bad plan?
Let's say you worked at Ford, and roughly 50% of the cars you turned out had something wrong with them. You get lots of unhappy customers demanding their money back. Is your problem:
a) That you have a design defect in your car.
b) That you are introducing defects in production.
c) That you are testing cars insufficiently.
Most people realize that to test every car as it comes off the line is futile. There's too many of them, with too many potential points of failure. There's no way you can test them all. The root cause of the problem has to be in either a or b, and if you're looking to improve the qulaity of your cars, this is where you would spend your money. This isn't to say that Ford doesn't test their cars, I'm sure they do, but testing should be a means of verifying quality (IE, 1/1000 cars tested had a defect, our goal was 1/500, so therefore we can stop spending money on finding design and production faults), and not a means of improving it.
It's so easy to see this when we're talking about cars. Why does everyone get it backwards when we start talking about software?
Not only is it impossible to test every possible combination of inputs to most software, it's also very expensive to find and fix problems this way. If you find a problem in design review, or code inspection, then you have your finger on it. You know EXACTLY where the defect is, and how to fix it. On the other hand, when you say "Microsoft Word crashes when I try to select a paragraph and make it Bold", you have no idea where the fault is. Any one of several thousand lines of code could be the problem. It can take literally days to track down and fix the defect.
Your testing should not be a means of finding faults, but a means of verifying the quality of your product. Testing is not part of the development process.
Automated testing does indeed work (Score:2)
You write test cases as you go. You make sure you can run an automated regression test at any time. If you don't do this, then any time you change code you might break old code and you won't realize it. Just doing spot checks at the keyboard isn't good enough. And the programmers need to be writing these test cases first, and they need to be kept separate from tests written by external groups.
Tried Cleanroom? (Score:2, Informative)
My personal recommendation is the "Cleanroom" methodology. You create a functional specification with a mathematical guarantee of completeness and consistancy. Auditable correctness is also a part of the process. Then when it comes to testing you generate test cases that cover all states, all arcs and then do statistical test case generation based on a usage model. The overall cost of this process is a bit more up front, but studies have shown that the process far more than pays for itself in greatly reduced maintenance/debugging costs.
So to answer you question is that to generate a decent set of test cases, you really have to understand the problem space and have mapped out the state-space in some manner. Trying to derive this without a methodical approach and ones testing will be spotty. The worst I've seen so far was a random state-space walker (ala Brownian motion). Statistically this approach avoids all the difficult cases in the far corners of the state-space.
Now for the bad news: Cleanroom is quite tedious for the programmer. The enumeration phase takes seemingly forever and can be mind-numbingly boring.
Here's the amazon link on the layman's book on Cleanroom: Cleanroom Software Engineering: Technology and Process by Stacy J. Prowell, Carmen J. Trammell,Richard C. Linger, Jesse H. Poore [amazon.com]
And now for the shameless self promotion bit with a long winded sales pitch for executives on Cleanroom: my own Cleanroom company: eLucidSoft [elucidsoft.net].
Just chant over and over: "Hire eLucid, play golf."
Document your thought process (Score:2, Insightful)
Any consideration you have during the writing of a single line of code is gold. And like a great dream, if you don't get it down when you think it, you will lose it in a day or two.
My 2 cents.
zerg (Score:2)
If you're just looking for regular bug testing, assume it's a given that the user will not report bugs to you and have the program automatically email you a core dump and/or stack trace and/or any appropriate data if an unhandled exception occurs.
What's your methodology (Score:2)
I am shocked at how frequently our developers don't have a good understanding of their architecture, or sometimes even the problem that they are trying to solve. As a result, when they go do do "testing" they are frequently performing tests that are not valid.
For example, they might create a new build and test that build only on their development workstation before full deployment of the application.
Naturally the development box has different resources from that of a standard production machine. Many developers don't seem to understand this.
Another example - frequently boundary conditions, or interfaces to other applications are not fully tested.
Using bad methodology, all of the time that you spend testing is wasted.
Management tends to feel that testing time is wasted because their experience is that the time that they have invested in the past has been fruitless.
Please develop:
valid test cases,
valid test plans, then
execute them,
find gaps, then
use the gaps to learn how not to make the same mistakes in the future!
Phooey.
Anomaly
PS - God loves you and longs for relationship with you. If you would like to know more about this, please contact me at tom_cooper at bigfoot dot com
bebugging (Score:5, Insightful)
Actually you can do this by adding more bugs, yes adding them, The technique is called bebugging and the is basicly:
1) Produce code, it contains an unknown number (N) of bugs.
2) Programmer (or bebugger) seeds the code with a number (B) of known new bugs, the number and type of bugs should be determined from bugs found in previous debugging cycles.
3) Code is submitted to testing and some bugs are found (F).
3) The bugs found are examined and categorised as either real bugs (FN) or bebugs (FB).
4) Number of real bugs (N) can be found as the ratio of found bebugs (FB) to unfound bebugs (F).
5) Don't forget to remove all the bebugs.
Requirements (Score:3, Insightful)
I'm certain someone has already said this, but over 80% of defects come from crappy requirements. Forget about your design & analysis, your coding practices, inpsection techniques, debugging and testing abilities - if your requirements are not CLEAR, CORRECT, ATOMIC, UNAMBIGUOUS, and CONSISTENT, you might as well start burning money.
NASA correlated a $1 cost to correct a "defect" in the requirements stage (here a defect can be any requirement that does not meet all 5 attributes I listed above) to several hundred to thousands of times over when addressing the same defect at the testing stage. Crappy requirements and crappy specifications are a big part of what makes your code buggy and expensive.
LA Times posted a study last year that showed that the average US programmer only coded for 51 days a year. 51 days!! One fifth of your working year spent writing new code. The rest of the time? DOING REWORK.
Biggest cause of rework?
UNCLEAR AND AMBIGUOUS REQUIREMENTS.
Spend the time and effort to beef up your requirements gathering and management processes. You'll get your ROI in ONE project cycle.
Correctness... (Score:3, Funny)
Re:testing... (Score:2)
That's mostly true, but it gets better when you let developers test eachothers code. They will do anything to make it go wrong
And they are very descriptive testers most of the time.
Fascinating! (Score:2, Insightful)
Thank you for your interesting and insightful treatise.
Clearly, with your keen grasp of the history of programming, and your understanding of VB and HTML, you will someday make a fine salesman or mid-level manager. You already exhibit the skill and insight used by management at most companies to plan their projects, procedures, timelines, and budgetary requirements.
Many programming issues would clearly benefit from simplification, and perhaps you are on to something. By reducing the number of tools a language attempts to implement, it clearly decreases the number of distractions for the programmer. If you wish to pursue this concept further, you may perhaps wish to research another foundational language called Logo.
Good luck in your next class!
Billbert, PhB, TFIC
Redmond, WA
P.S. - A side note regarding the original article:
A firm understanding of your process for code development, and a clear design for your testing procedure are essential.
If you spend some time in advance on planning, your code will benefit. Also, a good test plan should include a measure of the relative impact of a 'bug' or 'defect' to help determine priority of response by your programmers.
By focusing on your programming objectives from the beginning, and maintaining that focus throughout your entire design lifecycle, you should be able to identify underlying problems in your current development model and use them to improve your entire process, with a goal of helping prevent errors in design, and catching errors in code before going to test.
Peer review at each stage prior to testing (planning, functional design, algorithm design, coding, and test design) will also help catch errors in advance, and lead to the development of much better code. It may sound like it will take longer and cost more, but it saves time and money in the end in terms of not having to rewrite and maintain poorly implemented code.
Cheers.
Re:Metrics and Code Reviews (Score:2)
Neither programming or software assurance are areas I have knowledge in, so take the above with a grain of salt and check it out yourself. It all seemed rather interesting.