Not All Bugs Are Random 165
CowboyRobot writes "Andrew Koenig at Dr. Dobb's argues that by looking at a program's structure — as opposed to only looking at output — we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on an aspect of its input such as size, we should verify that it works properly as close as possible to the decision boundary on both sides. I've seen quite a few programs that impose arbitrary length limits on, say, the size of an input line or the length of a name. I've also seen far too many such programs that fail when they are presented with input that fits the limit exactly, or is one greater (or less) than the limit. If you know by inspecting the code what those limits are, it is much easier to test for cases near the limits.'"
The 80's called... (Score:1)
They want their structured programming back...
Re: (Score:3)
And always test for the null set.
I once tried to get the following into some coding standards, unfortunately I failed: Every program and function should do nothing correctly
Re: (Score:3)
And always test for the null set.
I once tried to get the following into some coding standards, unfortunately I failed: Every program and function should do nothing correctly
You didn't fail, you just failed to notice that you correctly did nothing!
Obvious, but worth restating. (Score:5, Informative)
Fairly obvious statement, but one some newer programmers don't know. Koenig is talking about white-box testing, which is well understood.
Re: (Score:3)
Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"
Re:Obvious, but worth restating. (Score:5, Insightful)
Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"
It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.
We are so busy giving new names to old concepts, and barfing up new programming languages that we end up re-inventing and re-naming long known concepts with shiny new trendy names that we fail to notice this was well known decades ago.
Re: (Score:3)
Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"
It would make more sense, because there was never an assertion that bugs were random.
Furthermore limit testing is not a new concept.
I haven't read TFA, but the newer programming paradigms, such as agile/XP, are heavily test-based. Instead of writing a design, implementing it, and then testing it in production, we start with writing test cases (unit & integration tests, and hopefully some behaviour tests). When they are all fulfilled, we can stop programming.
But how do you know which tests -- beyond functional ones -- to write? Writing and evaluating is already non-trivial for realistic quality tests (throughput, security). Surprisin
Re: (Score:2)
If you are designing an API, there are dozens of different error codes (invalid value, invalid enumator, run out of memory, invalid handle). Sometimes one bad input could generate three or more errors. But it is the specification which states which error takes priority.
To make sure everything works as expected, you do all sorts of things:
Positive testing - making sure things do what they are supposed to do. You look up all the input parameters, work out which combinations are critical and need to be tested
Re:Obvious, but worth restating. (Score:4, Informative)
Are any bugs truly "random"? I always thought computers were deterministic machines.
OK, maybe process scheduling bugs are fairly random...the point is that the headline makes it sound like bugs just happen all by themselves.
Re: (Score:3)
Are any bugs truly "random"? I always thought computers were deterministic machines.
If your code contains a state machine and the code has a bug where it in some conditions references invalid or uninitialized memory to determine it's next state, then the bug may appear quite random. In fact, it may appear to be several different bugs.
This is particularly fun when doing cross platform programming, where some platforms always zero initializes memory upon allocation causing programmers who develop on those platforms to think everything is peachy, only to have the code promptly crash or worse
Re: (Score:2)
the seemingly random bug is in fact deterministic and only triggered under certain unintentionally well-defined conditions.
The faulty code is of course fully deterministic. The resulting output may be completely arbitrary.
Re: (Score:3, Interesting)
Yep, fairly obvious, but where I work there's this whole class of people who hide their incompetence behind the phrase "black box testing". At first I didn't understand why they insisted so much on doing black box testing, until one day we basically ran into what the article describes: a subsystem behaved poorly when one of its inputs was exactly on the boundary that that very subsystem had defined. In that case I saw the black box bunch crash and burn because they weren't able to even design a good test c
Re: (Score:2)
It is not the ob of black box testers to diagnose a problem, just report the failure. Hence the term "black box testing". They are not debuggers, but rather testers.
Re:Obvious, but worth restating. (Score:5, Insightful)
I think in common usage, white box testing does not mean "debugging" or "diagnosing" the code - it means more like "testing that takes into account the internal design and implementation of the system under test rather than just the inputs and outputs".
There's a need for both types of testing. However, as long as white box testers don't get too wrapped up in the details to step back and view the system as a whole (as developers sometimes do), they can do both. But there are so few good testers in the business, you don't want to waste a good white box tester on developing black box tests.
Sadly, I suspect there are quite a few "average" developers out there who, had testing as a career been given more respect by the industry over the decades, would have been "great" white box testers. While there is significant overlap between the skills/personality attributes needed to be a great developer and those needed to be a great tester, there are a bunch of skills/personality attributes that are quite important in one field and not nearly as so in the other.
Re: (Score:2)
Exactly. What test your code with values on eitehr side of the branching points in the code. Wow never would have thought of that one.
Is this news for anyone? (Score:5, Insightful)
Re: (Score:3)
Now... there are exceptions, too. For example if you know your input will always be less than X characters, you have only one boundary condition to check (no characters). If you know it will always be exactly Y characters you have no boundary conditions. On length, that is... using the same kind of example OP was using.
Re: (Score:2)
Re: (Score:2)
A problem is, a fair % of programmers don't care for code maintenance and unit tests.
Documenting expected behaviour (e.g. javadoc) as to what exceptions will be thrown and writing unit tests to verify these trivial cases may seem pedantic to some.
Re: (Score:3, Insightful)
Even if you think you "know" the input will always be a certain length or less than x characters you had better check it anyway and test what happens if it isn't. Otherwise one day you can guarantee it won't match your expectation.
About 60% of the random, occasional failure bugs I fix are people making these kinds of assumptions. Clocks never run backwards, sensors never fail, data never gets corrupted. In reality they do with alarming regularity.
Re: (Score:3)
In many cases, the "data corruption" case is just impractical to test for very well unless every datum has a hash or something associated with it or another copy exists and every function verifies the hashes or for both copies matching on entry.
Also, I've seen code that was so filled with validation of stuff that was already validated or produced by code we wrote that the bugs were mostly in the validation code not the logic associated with the core functionality.
There is a happy medium. It's important that
Re: (Score:2)
Data corruption in my books should be handled by ending execution cleanly so as to not cause further problems.
Re: (Score:2)
Data corruption in my books should be handled by ending execution cleanly so as to not cause further problems.
Just shutting down may be an option if you are writing an iPhone App. If your mission critical flight control system gets corrupt data off the ARINC-429 bus, you better have a plan to deal with it and continue.
Re: (Score:2)
When it's easily detectable, I agree.
I've developed in systems where a single data corruption problem that wasn't caught as such (for example, a triple bit error in memory that wasn't detected, let alone corrected, by ECC) would bring down millions of dollars worth of hardware. That quickly got the unwelcome notice of CIOs, and sometimes CEOs, of some large companies. And, I thought that was the best answer as, in those systems, the data integrity was more important than uptime (and we got a nice, if enormo
Re: (Score:2)
That was essentially my point -- in cases where all of a sudden your processing realizes the data makes no sense at all, stop trying to parse it, you're going to end up doing more damage (and possibly create an attack vector). This is one of many reasons I still subscribe to the Unix-style "many small specific programs" concept. Let the piece that failed core dump, and with it the data that caused the error and let everything else move on if necessary.
Re: (Score:2)
The only (albeit large) exception is text values I think. I think a lot of people don't write tests to see that FirstName can be 40 chars but not 41 if for no other reason then creating a 41 char test string is a pain. I think unit testing suites should provide easy ways to test bounds like this. Something like Assert.MustBe(Firstname = 40); no generating of random strings, or how is often done someone pounding on a keyboard and counting manually that the string matches the criteria they are trying to impos
Re: (Score:2)
If you know the input will always be less than X characters, you have TWO boundary conditions, not one:
1. No characters
2. X-1 characters
Re: (Score:2)
Yes, but if its supposed to be verified by class/module A, which calls class/module B with clean input, then you don't need to test B with bad input, only A.
Re: (Score:3)
In large, long-lived project, these assumptions are all hard to make.
Re: (Score:2)
"The software, written in Ada....."
That says it all.
Not the important part, no. The problem was bad engineering practice, not Ada.
Ariane 5 : Notable launches [wikipedia.org]
The software, written in Ada, was included in the Ariane 5 through the reuse of an entire Ariane 4 subsystem despite the fact that the particular software containing the bug, which was just a part of the subsystem, was not required by the Ariane 5 because it has a different preparation sequence[21] than the Ariane 4.
Ada is well proven software engineering technology that can help you produce better software, but it isn't magic. You still have to follow sound procedures for it to help you.
For anyone that is interested, there is more useful detail in these links:
I heard tell that Ada was to blame for the Ariane V disaster. Is this true? [adapower.com]
Put it in the contract: The lessons of Ariane [irisa.fr]
Re: (Score:2)
It's called integration testing. That is why unit tests are considered insufficient.
Re: (Score:2)
That is what I thought as well.
Re: (Score:2)
We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.
How do your testers about those boundary conditions? And how do they know about all of them? And how do they know that they know about all of them?
Re: (Score:2)
We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.
I remember an amusing one: user reported a print re-run registration page failing on a few particular orders among thousands during the history of the (subcontracted) application. I found the common factor to be that all failing orders were for 142 copies. It turns out that the handling code for some absurd reason assigned page*count/(amount-142) to the total_page_count (or whatever), and overwrote it with the proper "amount * page_count" in the next line*. The rest of the code worked fine for all positive
This is not new (Score:2, Informative)
http://en.wikipedia.org/wiki/Boundary-value_analysis
Bounds test? (Score:5, Insightful)
Has testing degraded so far that people don't now what a bounds test is?
Re: (Score:2)
You're more likely to hear "What's testing?" from a lot of programmers nowadays.
Testing was never even mentioned in my university courses, much less formalized.
Re: (Score:2)
With Software Engineering courses more prevalent so is testing. What is a much bigger problem is coder schools who teach languages but not methods.
Re: (Score:2)
What is a much bigger problem is coder schools who teach languages but not methods.
They do teach methods! Classes and inheritance too...
Re: (Score:2)
Re: (Score:2)
unfortunately... this.
i'm not a programmer. i can't code anything. but when i see people wiring code and not bothering with any tests for it, i feel sad, because i am seeing way too many regressions in basic functionality that could be caught by a couple of simple tests.
in addition to passing these problems to users, it is also demotivating for other participants of a project that does not do proper test development. and i do not see a solution to this problem.
Re: (Score:2)
Re: (Score:2)
support (verb): The act of charging for fixes to the code you didn't do testing on... It is a win-win for both sides...lol
Re: (Score:3)
Who might know about this? (Score:5, Interesting)
That depends. If you're are a die-hard TDD fan, for example, then you'll be writing your (unit) tests before you necessarily know where any such boundaries are. Moreover, there is nothing inherent in that process to bring you back to add more tests later if boundaries arise or move during refactoring. It's not hard to imagine that a developer who leaves academia and moves straight into an industrial TDD shop might never have been exposed to competing ideas.
(I am not a die-hard TDD fan, and the practical usefulness of white box testing is one of the reasons for that, but I suspect many of us have met the kind of person I'm describing above.)
Re: (Score:3)
you can't write the tests before knowing what the program is supposed to do first and for that you need to know the bounds it needs to adhere to...
or well, you can write them, but then you're doing them to just fill a tickbox and to input hours into timetracking. which is fucking useless. still happens a lot though, because someone on the management chain is a tdd fan but is also an agile fan. furthermore, tests for the ui could be testing against boundary values xb while the back end was developed against
Re: (Score:2)
you can't write the tests before knowing what the program is supposed to do first and for that you need to know the bounds it needs to adhere to...
I read the context, as given in the summary and TFA, as switching between different algorithms at certain bounds, not necessarily limited to only accepting input at all if it's within certain bounds. In other words, the bounds and algorithm switching could be implementation details, not necessarily exposed in the relevant interface.
Re: (Score:3)
FWIW, what you're describing seems more practical to me. However, if you start from a position that "You should not be doing almost any programming until the design is done" then you're already a long way from how many TDDers practise and from the process that many who advocate TDD, including big name consultants and authors, describe.
Re: (Score:2)
I don't think you can credibly claim a big name consultant like Robert C. Martin is not a TDDer, but when you watch interviews like this [infoq.com] it seems pretty clear that he would object to your claim. See also the infamous Sudoku contrast between Ron Jeffries' TDD attempts and Peter Norvig's version, assorted comments by Kent Beck ("It's an analysis technique, a design technique, really a technique for structuring all the activities of development."), etc.
Re: (Score:2, Insightful)
Has testing degraded so far that people don't now what a bounds test is?
It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense. It may be sufficiently complex so as to appear random, but it isn't. There's very little "random" about a computer, or a computer program. Every now and then a single bit error creeps into the I/O but the rate this occurs in computers not in space and thus subj
Re: (Score:2)
The headline and article do not come close to matching.
It's as bad as the one that makes you think that cheese was being spread on roads when it was really salty water.
Re: (Score:3)
It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense.
In the 68020 processor manual, it says that determining the exact execution time of an instruction is very difficult, even if you know all the details of the processor implementation. That was 20 years ago. Today, precise timing is basically unpredictable. Then you have multi-threading, where the order of operations is basically unpredictable. Then your code is waiting for external events that are totally unpredictable. Anything that has anything to do with timing is essentially random.
not to a developer, programmer, or comp scientist (Score:2)
> The word "random" also includes the definition of "odd, unusual, or unexpected."
In everyday conversation, people may say "random" when they mean "unexpected". Hopefully they wouldn't WRITE that in a published article, but they may say it in informal conversation.
A computer programmer who passed Programming 201 will distinguish between random, unexpected, and arbitrary in informal conversation, because in our world those words have COMPLETELY different meanings. Among programmers, saying "random" when
Re: (Score:3)
Even in the computer science sense the article is using random correctly. It's arguing bugs from programming errors are NOT randomly distributed through the code base but cluster around code that does certain implementation tasks.
There article is IMHO vapid, but does use it's vocabulary correctly.
Re: (Score:3)
I actually read the article. You know what it is? It is a simplified tutorial on white box testing for people who don't know, that is, amateur programmers. Yeah, that would have once been new and interesting to me, when I was learning to program. Not when I was doing it for a living though. Bad title and summary.
No shit Sherlock (Score:5, Insightful)
You know, I remember writing test plans to to test input that were one below, at, and one above, some arbitrary limit when I was a trainee programmer coding up COBOL on mainframes back in the mid 80s.
How on earth does this drivel make it onto Slashdot? This is 30 year old news at least (which makes it straight out of the 17th C in internet years)
Re: (Score:2)
Re:No shit Sherlock (Score:4, Insightful)
Afraid? Less afraid than not paid for. 'Write your code correctly, then you don't need to test'. A bit simplified, but this is more or less what I heard more than once from a customer.
Re: (Score:2)
In the case of test inputs, we're largely talking unit tests here, which form part of the code base to verify the functional code is correct. The 'customer' shouldn't have such fine grained control over the development process.
(As distinct from a black box tester who never looks at the source code etc)
As far as not using xUnit/TDD, that's often a generational thing where development practices stem from the previous millenium before Kent Beck and others popularised its widespread adoption. I've worked on sev
Re: (Score:2)
It is not really a question of fine grained control. More a question of time and money. Suppose you are a consultant, who is asked how much time a certain project needs. You say: Four weeks. You know how to write tests. You like to write tests because you know that requirements evolve and even with a proper design changes in one place might have unexpected consequences in another place. But then comes another consultant, wh
Re: (Score:2)
As far as not using xUnit/TDD, that's often a generational thing where development practices stem from the previous millenium before Kent Beck and others popularised its widespread adoption.
Yepp, and now it is an ideology. Most people / developers don't understand for what you need a unit test. A "unit test" is a set of tests for _one single_ "compilation unit".
Example: I'm the developer of the Java Collections library (the stuff in java.util, List/ArrayList etc.) As I shift classes back and forth in the
Re: (Score:2)
No human has ever written 100% correct code 100% of the time.
Once I found a human, taught him to write a single hello world program, which he did correctly, then I forbade him from ever programming again, ever, on pain of death.
Ha! I just proved an anonymous coward wrong. I feel so accomplished.
Re: (Score:2)
Wait a minute. I know that guy! I hired him because 100% of the time, the code he had written was 100% correct.
Let's just say that you should have killed him.
QA needs to be testing coders are poor QA / tester (Score:2)
QA needs to be testing coders are poor QA / testers and we need real live testers not some scripted system that can be coded to pass but still not be right and or well it passes but if any one looks at the out put they will say that does not look right.
Re: (Score:2)
There's a scope for both and shared knowledge can be beneficial.
Part of the automated build process can be running QA scripts via a robot. e.g. QA person files a bug report with steps to reproduce the issue. A VNC script is attached to the bug report, which is then translated to, say, a junit script. Programmer inserts asserts in the junit code to verify expected conditions. In this way regressions are fewer and don't escape to the QA people in the first place.
Re: (Score:2)
Who passed that load of unpunctuated garbage?
Re: (Score:3)
QA ignorance is not a bad thing. QA should not be working from designers POV but from a requirement/user story POV. If it does not satisfy the requirements or the user story, it is a bad design. Period. The less QA knows of the design the better.
I'm afraid of yet another test framework (Score:2)
> Sadly, I still see developers not testing, and are practically afraid of writing test scripts.
I'm a little bit afraid of some tests I've been asked to write, and I've been programming professionally since 1997. My credits include WordPress, Apache, testing the Linux kernel raid, and other well known software.
I fear it because tests for this project involve learning at least one and probably two new unit testing frameworks which each try to approach testing in an innovative new way. One has tests that
Of course not (Score:1)
Those are features
. . . (redux) DUH (Score:1)
And some people cannot be programmers, some cannot hammer a nail, and some cannot wear their wife's clothings in public. You aren't a real man until you can do all three.
Re: (Score:2)
And some people cannot be programmers, some cannot hammer a nail, and some cannot wear their wife's clothings in public. You aren't a real man until you can do all three.
... at once.
Look at commit time stamp. (Score:5, Insightful)
If I am a columnist with some modest name recognition I could have converted this mildly amusing (to me at least) observation into a column. But alas, I am not one. So he gets to repeat the age old advice given by old Prof Mahabala, teaching Intro to Computing 201, (Fortran programming, in IBM 365/155 using punch cards no less ) back in 1980 into a column, and all I get to do is to bitch about it.
Re: (Score:2)
Something without bread, pasta, or cakey desserts.
Bread and pasta are good for maintaining a stable blood sugar, and the brain likes a stable blood sugar.
Well Duh (Score:2)
we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on
If you have an MVC architecture with 10000 models, controllers and views; all of your domain data is stored as Object in maps, keyed with strings. If the keys for your middleware and your maps are built up on the fly and type casts are used at the point where your domain data is actually used, then you are probably going to have problems.
Oh and here's another good one: you have a module blah with an internal namespace com.company.blah but now you want a newblah so you copy blah and change two classes but le
It's good to test the corner cases (Score:4, Informative)
News at 11.
Re: (Score:2)
Poster meant "3"
Look at the comments (Score:3)
no bugs are random (Score:2)
Re: (Score:2)
Some bugs arise from such a complicated set of conditions that they appear to be random. These are the most interesting bugs, and the most infuriating.
Other bugs can even be truly random; a race condition that depends on whether one thread gets scheduled on a processor that's running just slightly faster (or slower) than another, for instance.
race bug is not random, output is undetermined (Score:2)
> Other bugs can even be truly random; a race condition that depends on whether one thread gets scheduled on a processor
The bug exists before the processor is even purchased.
The bug is not random, though the output may be influenced by a random event. The output isn't even random - I debugged one of those in the kernel and the result was spinlock. The only random part is WHEN the problem becomes visible to the user.
That reminds me of a Heisenbug we had once. Completely off topic, but we once had a bug
How is any bug random? (Score:2)
Re: (Score:2)
Don't look at the bugs individually, but at the distribution of bugs in the application. Bugs do not have an equal chance of appearing anywhere in a codebase, so their distribution is not random. There is much you can learn about the quality of your system and your development process depending on where the bugs are found, and this knowledge is actionable. Maybe your team needs to take unit tests more seriously. Maybe your developers have no idea of what are realistic scenarios. Maybe you are building on a
Worst headline of 2013? (Score:3)
Bugs are never RANDOM. Bugs are, by definition, an error (human blunder ... incorrect design, improper code, etc.)
Bugs Never Random (Score:2)
No, a select few are random. One of my coworkers from a decade ago randomly copied portions of if-then statements trying to get them to work. "This branch works so I'll copy it, paste it, and fiddle with the conditions until something else happens." Never mind those two branches required completely different decisions. Looking at one part of his program, a third of his if-then branches could never be run because they would always evaluate to false. I pointed that out to him and he couldn't offer a shre
Randomness and information (Score:2)
Re: (Score:3)
Apparently you've never dealt with race conditions and multi-threaded code.
Re: (Score:2)
Good point, though it doesn't actually refute GP. Computers are deterministic*. On the other hand, programs, functions, etc may not be deterministic.
* Except devices specifically designed not to be deterministic, such as hardware random number generators that rely on quantum, electron phenomena, etc. Then again I'm not sure it is correct to call such devices computers.
Re: (Score:2)
Computers are not deterministic. Interrupts are raised at any time by interface hardware. Users generate keystrokes and mouse events at random. Internet packets arrive at random times. Pre-emptive multi-tasking operating systems suspend and resume processes according to the load at the moment, not according to a fixed schedule of n clock cycles.
If you want to go back to something like a TRS-80 where you were programming the hardware, not an OS, then yes, they approached deterministic, but with random
Re: (Score:2)
Computers are deterministic.
That's an over-simplification due to the widespread prevalence of multi-core CPUs and operating systems with preemptive (i.e., clock-driven and environment-driven) multitasking and interrupts. These things make even things that are clearly computers be non-deterministic. That "computers are deterministic" is merely a useful model of the world, not reality.
Re: (Score:2)
Let's see you force one to happen in the debugger, then.
Re: (Score:3)
It is not random. If you have enough knowledge and the ability to comprehend that knowledge, you can predict what will happen. Nothing is random.
Sure, as long as you start a program and let it run all by itself without touching anything. As soon as you introduce human input, the system may still be deterministic, but the output of the program is in effect random because the behavior of the operator cannot be predicted. The kind of "knowledge and the ability to comprehend that knowledge" that you describe is known as omniscience, and most IDEs currently don't support it.
Re: (Score:2)
Oh.
Excuse me.
"Stochastically Variable".
Re: (Score:2)
When exactly will that bit be flipped by the stray subatomic particle hitting your PC from the sun?
Some bugs really are random.
Re: (Score:2)
Yeah no. Not sure if this applies to software testing (although some cases can be so unpredictable that you might as well refer to them as random), but the results of some quantum events are indeed random. Even with perfect knowledge of all variables from some moment in the past, some future events are impossible to predict. There are no hidden variables.
Re: (Score:3)
Not if the different threads are clocked from different PLLs.
Re: (Score:2)
Apparently you've never used Perl.....
Re: (Score:2)
Re: (Score:2)
For me, what made me swear off Perl was back when I was doing backend work for web development in 1998....
I had been struggling with some fairly extensive code I had put together. But I decided to tag along with the coworkers/bosses for a few beers. Intoxicated as I was, I decided that "hell, I need to go fix that code" and went back to the office... And I understood the code better while intoxicated...
The day after I loudly proclaimed that that would be the last Perl project I'd ever work with....
Re: (Score:2)
Then I saw my name on it.
You can do astonishing things to text in a single line of Perl but then may have to write six lines of comments to give a clue as to what the hell is going on
Behaviour defined arbitrarily by whitespace (and not in a python way) is far worse since there is nothing to read to work out what is going on.
Re: (Score:2)