Forgot your password?
typodupeerror
Programming Bug

Not All Bugs Are Random 165

Posted by timothy
from the especially-not-bees dept.
CowboyRobot writes "Andrew Koenig at Dr. Dobb's argues that by looking at a program's structure — as opposed to only looking at output — we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on an aspect of its input such as size, we should verify that it works properly as close as possible to the decision boundary on both sides. I've seen quite a few programs that impose arbitrary length limits on, say, the size of an input line or the length of a name. I've also seen far too many such programs that fail when they are presented with input that fits the limit exactly, or is one greater (or less) than the limit. If you know by inspecting the code what those limits are, it is much easier to test for cases near the limits.'"
This discussion has been archived. No new comments can be posted.

Not All Bugs Are Random

Comments Filter:
  • by Anonymous Coward

    They want their structured programming back...

  • by Animats (122034) on Sunday December 29, 2013 @06:37PM (#45814005) Homepage

    Fairly obvious statement, but one some newer programmers don't know. Koenig is talking about white-box testing, which is well understood.

    • Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

      • by icebike (68054) on Monday December 30, 2013 @01:32AM (#45816163)

        Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

        It would make more sense, because there was never an assertion that bugs were random.
        Furthermore limit testing is not a new concept.

        We are so busy giving new names to old concepts, and barfing up new programming languages that we end up re-inventing and re-naming long known concepts with shiny new trendy names that we fail to notice this was well known decades ago.

        • Yes, we can retitle from "Not All Bugs Are Random" to "White-Box Testing Is A Real Thing"

          It would make more sense, because there was never an assertion that bugs were random.
          Furthermore limit testing is not a new concept.

          I haven't read TFA, but the newer programming paradigms, such as agile/XP, are heavily test-based. Instead of writing a design, implementing it, and then testing it in production, we start with writing test cases (unit & integration tests, and hopefully some behaviour tests). When they are all fulfilled, we can stop programming.

          But how do you know which tests -- beyond functional ones -- to write? Writing and evaluating is already non-trivial for realistic quality tests (throughput, security). Surprisin

          • by mikael (484)

            If you are designing an API, there are dozens of different error codes (invalid value, invalid enumator, run out of memory, invalid handle). Sometimes one bad input could generate three or more errors. But it is the specification which states which error takes priority.
            To make sure everything works as expected, you do all sorts of things:

            Positive testing - making sure things do what they are supposed to do. You look up all the input parameters, work out which combinations are critical and need to be tested

      • by Joce640k (829181) on Monday December 30, 2013 @04:52AM (#45816713) Homepage

        Are any bugs truly "random"? I always thought computers were deterministic machines.

        OK, maybe process scheduling bugs are fairly random...the point is that the headline makes it sound like bugs just happen all by themselves.

        • by Lord Crc (151920)

          Are any bugs truly "random"? I always thought computers were deterministic machines.

          If your code contains a state machine and the code has a bug where it in some conditions references invalid or uninitialized memory to determine it's next state, then the bug may appear quite random. In fact, it may appear to be several different bugs.

          This is particularly fun when doing cross platform programming, where some platforms always zero initializes memory upon allocation causing programmers who develop on those platforms to think everything is peachy, only to have the code promptly crash or worse

    • Re: (Score:3, Interesting)

      by Anonymous Coward

      Yep, fairly obvious, but where I work there's this whole class of people who hide their incompetence behind the phrase "black box testing". At first I didn't understand why they insisted so much on doing black box testing, until one day we basically ran into what the article describes: a subsystem behaved poorly when one of its inputs was exactly on the boundary that that very subsystem had defined. In that case I saw the black box bunch crash and burn because they weren't able to even design a good test c

      • by plopez (54068)

        It is not the ob of black box testers to diagnose a problem, just report the failure. Hence the term "black box testing". They are not debuggers, but rather testers.

        • by uncqual (836337) on Sunday December 29, 2013 @11:09PM (#45815643)

          I think in common usage, white box testing does not mean "debugging" or "diagnosing" the code - it means more like "testing that takes into account the internal design and implementation of the system under test rather than just the inputs and outputs".

          There's a need for both types of testing. However, as long as white box testers don't get too wrapped up in the details to step back and view the system as a whole (as developers sometimes do), they can do both. But there are so few good testers in the business, you don't want to waste a good white box tester on developing black box tests.

          Sadly, I suspect there are quite a few "average" developers out there who, had testing as a career been given more respect by the industry over the decades, would have been "great" white box testers. While there is significant overlap between the skills/personality attributes needed to be a great developer and those needed to be a great tester, there are a bunch of skills/personality attributes that are quite important in one field and not nearly as so in the other.

    • Exactly. What test your code with values on eitehr side of the branching points in the code. Wow never would have thought of that one.

  • by JeffOwl (2858633) on Sunday December 29, 2013 @06:38PM (#45814015)
    We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.
    • I don't even understand why this is being considered as news. If you don't know this already, you aren't the kind of programmer I've ever worked with.

      Now... there are exceptions, too. For example if you know your input will always be less than X characters, you have only one boundary condition to check (no characters). If you know it will always be exactly Y characters you have no boundary conditions. On length, that is... using the same kind of example OP was using.
      • by Sique (173459)
        I made it a routine to always check for all boundaries, even for those I believed to be covered already, and to throw an exception if the check fails. Even so often there are other undiscovered bugs that allow the function to be called with invalid data, or someone (often me) later reuses the function at another place and forgets the boundary checks.
        • A problem is, a fair % of programmers don't care for code maintenance and unit tests.

          Documenting expected behaviour (e.g. javadoc) as to what exceptions will be thrown and writing unit tests to verify these trivial cases may seem pedantic to some.

      • Re: (Score:3, Insightful)

        by AmiMoJo (196126) *

        Even if you think you "know" the input will always be a certain length or less than x characters you had better check it anyway and test what happens if it isn't. Otherwise one day you can guarantee it won't match your expectation.

        About 60% of the random, occasional failure bugs I fix are people making these kinds of assumptions. Clocks never run backwards, sensors never fail, data never gets corrupted. In reality they do with alarming regularity.

        • by uncqual (836337)

          In many cases, the "data corruption" case is just impractical to test for very well unless every datum has a hash or something associated with it or another copy exists and every function verifies the hashes or for both copies matching on entry.

          Also, I've seen code that was so filled with validation of stuff that was already validated or produced by code we wrote that the bugs were mostly in the validation code not the logic associated with the core functionality.

          There is a happy medium. It's important that

          • Data corruption in my books should be handled by ending execution cleanly so as to not cause further problems.

            • Data corruption in my books should be handled by ending execution cleanly so as to not cause further problems.

              Just shutting down may be an option if you are writing an iPhone App. If your mission critical flight control system gets corrupt data off the ARINC-429 bus, you better have a plan to deal with it and continue.

            • by uncqual (836337)

              When it's easily detectable, I agree.

              I've developed in systems where a single data corruption problem that wasn't caught as such (for example, a triple bit error in memory that wasn't detected, let alone corrected, by ECC) would bring down millions of dollars worth of hardware. That quickly got the unwelcome notice of CIOs, and sometimes CEOs, of some large companies. And, I thought that was the best answer as, in those systems, the data integrity was more important than uptime (and we got a nice, if enormo

              • That was essentially my point -- in cases where all of a sudden your processing realizes the data makes no sense at all, stop trying to parse it, you're going to end up doing more damage (and possibly create an attack vector). This is one of many reasons I still subscribe to the Unix-style "many small specific programs" concept. Let the piece that failed core dump, and with it the data that caused the error and let everything else move on if necessary.

      • The only (albeit large) exception is text values I think. I think a lot of people don't write tests to see that FirstName can be 40 chars but not 41 if for no other reason then creating a 41 char test string is a pain. I think unit testing suites should provide easy ways to test bounds like this. Something like Assert.MustBe(Firstname = 40); no generating of random strings, or how is often done someone pounding on a keyboard and counting manually that the string matches the criteria they are trying to impos

      • If you know the input will always be less than X characters, you have TWO boundary conditions, not one:
        1. No characters
        2. X-1 characters

    • by LWATCDR (28044)

      That is what I thought as well.

    • We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.

      How do your testers about those boundary conditions? And how do they know about all of them? And how do they know that they know about all of them?

    • We absolutely test all boundary conditions, on both sides. This is standard practice where I work, for just that reason.

      I remember an amusing one: user reported a print re-run registration page failing on a few particular orders among thousands during the history of the (subcontracted) application. I found the common factor to be that all failing orders were for 142 copies. It turns out that the handling code for some absurd reason assigned page*count/(amount-142) to the total_page_count (or whatever), and overwrote it with the proper "amount * page_count" in the next line*. The rest of the code worked fine for all positive

  • This is not new (Score:2, Informative)

    by Anonymous Coward

    http://en.wikipedia.org/wiki/Boundary-value_analysis

  • Bounds test? (Score:5, Insightful)

    by jklovanc (1603149) on Sunday December 29, 2013 @06:41PM (#45814037)

    Has testing degraded so far that people don't now what a bounds test is?

    • by msobkow (48369)

      You're more likely to hear "What's testing?" from a lot of programmers nowadays.

      Testing was never even mentioned in my university courses, much less formalized.

      • by jklovanc (1603149)

        With Software Engineering courses more prevalent so is testing. What is a much bigger problem is coder schools who teach languages but not methods.

        • by dkf (304284)

          What is a much bigger problem is coder schools who teach languages but not methods.

          They do teach methods! Classes and inheritance too...

      • by richlv (778496)

        unfortunately... this.
        i'm not a programmer. i can't code anything. but when i see people wiring code and not bothering with any tests for it, i feel sad, because i am seeing way too many regressions in basic functionality that could be caught by a couple of simple tests.
        in addition to passing these problems to users, it is also demotivating for other participants of a project that does not do proper test development. and i do not see a solution to this problem.

      • testing (noun): the action of releasing your program in the wild and cross fingers.
        • by penix1 (722987)

          support (verb): The act of charging for fixes to the code you didn't do testing on... It is a win-win for both sides...lol

      • by dbIII (701233)
        It was one of the points of difference between the engineering programming classes and the maths department ones. On the engineering side we were expecting things to go wrong and out of bounds while on the maths side it was all just supposed to work first time.
    • by Anonymous Brave Guy (457657) on Sunday December 29, 2013 @08:09PM (#45814613)

      That depends. If you're are a die-hard TDD fan, for example, then you'll be writing your (unit) tests before you necessarily know where any such boundaries are. Moreover, there is nothing inherent in that process to bring you back to add more tests later if boundaries arise or move during refactoring. It's not hard to imagine that a developer who leaves academia and moves straight into an industrial TDD shop might never have been exposed to competing ideas.

      (I am not a die-hard TDD fan, and the practical usefulness of white box testing is one of the reasons for that, but I suspect many of us have met the kind of person I'm describing above.)

      • by gl4ss (559668)

        you can't write the tests before knowing what the program is supposed to do first and for that you need to know the bounds it needs to adhere to...

        or well, you can write them, but then you're doing them to just fill a tickbox and to input hours into timetracking. which is fucking useless. still happens a lot though, because someone on the management chain is a tdd fan but is also an agile fan. furthermore, tests for the ui could be testing against boundary values xb while the back end was developed against

        • you can't write the tests before knowing what the program is supposed to do first and for that you need to know the bounds it needs to adhere to...

          I read the context, as given in the summary and TFA, as switching between different algorithms at certain bounds, not necessarily limited to only accepting input at all if it's within certain bounds. In other words, the bounds and algorithm switching could be implementation details, not necessarily exposed in the relevant interface.

    • Re: (Score:2, Insightful)

      Has testing degraded so far that people don't now what a bounds test is?

      It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense. It may be sufficiently complex so as to appear random, but it isn't. There's very little "random" about a computer, or a computer program. Every now and then a single bit error creeps into the I/O but the rate this occurs in computers not in space and thus subj

      • by dbIII (701233)

        It's worse than that. They're using the word "random" to describe

        The headline and article do not come close to matching.

        It's as bad as the one that makes you think that cheese was being spread on roads when it was really salty water.

      • It's worse than that. They're using the word "random" to describe the behavior of a digitally based, computationally deterministic system. One. Zero. Off. On. Yes. No. This is all a computer understands. It cannot be "random" in any meaningful sense.

        In the 68020 processor manual, it says that determining the exact execution time of an instruction is very difficult, even if you know all the details of the processor implementation. That was 20 years ago. Today, precise timing is basically unpredictable. Then you have multi-threading, where the order of operations is basically unpredictable. Then your code is waiting for external events that are totally unpredictable. Anything that has anything to do with timing is essentially random.

    • by Evil Pete (73279)

      I actually read the article. You know what it is? It is a simplified tutorial on white box testing for people who don't know, that is, amateur programmers. Yeah, that would have once been new and interesting to me, when I was learning to program. Not when I was doing it for a living though. Bad title and summary.

  • No shit Sherlock (Score:5, Insightful)

    by cruachan (113813) on Sunday December 29, 2013 @06:41PM (#45814047)

    You know, I remember writing test plans to to test input that were one below, at, and one above, some arbitrary limit when I was a trainee programmer coding up COBOL on mainframes back in the mid 80s.

    How on earth does this drivel make it onto Slashdot? This is 30 year old news at least (which makes it straight out of the 17th C in internet years)

    • The more interesting article is this one [drdobbs.com]: "Our survey of more than 3,000 developers and managers shows...salaries are on the rise once again"
  • Those are features

  • by Anonymous Coward

    And some people cannot be programmers, some cannot hammer a nail, and some cannot wear their wife's clothings in public. You aren't a real man until you can do all three.

    • by mooingyak (720677)

      And some people cannot be programmers, some cannot hammer a nail, and some cannot wear their wife's clothings in public. You aren't a real man until you can do all three.

      ... at once.

  • by 140Mandak262Jamuna (970587) on Sunday December 29, 2013 @06:52PM (#45814129) Journal
    Talking from personal experience, all the bugs I ever committed to production were coded in between 1 PM and 2 PM, my sleepiest time, when my stomach digesting the lunch was competing with the brain for blood supply.

    If I am a columnist with some modest name recognition I could have converted this mildly amusing (to me at least) observation into a column. But alas, I am not one. So he gets to repeat the age old advice given by old Prof Mahabala, teaching Intro to Computing 201, (Fortran programming, in IBM 365/155 using punch cards no less ) back in 1980 into a column, and all I get to do is to bitch about it.

  • we can sometimes predict circumstances in which it is particularly likely to fail. 'For example, any time a program decides to use one or two (or more) algorithms depending on

    If you have an MVC architecture with 10000 models, controllers and views; all of your domain data is stored as Object in maps, keyed with strings. If the keys for your middleware and your maps are built up on the fly and type casts are used at the point where your domain data is actually used, then you are probably going to have problems.

    Oh and here's another good one: you have a module blah with an internal namespace com.company.blah but now you want a newblah so you copy blah and change two classes but le

  • by Black Parrot (19622) on Sunday December 29, 2013 @07:34PM (#45814423)

    News at 11.

  • by tomhath (637240) on Sunday December 29, 2013 @08:04PM (#45814589)
    A study by the company I worked for a few years ago only found one thing that correlated with the number of bugs - the number of comments. It wasn't so much that the algorithm was complicated and needed explanation; more often it was just bad programmers tossing in a bunch of commented out print statements and comments that didn't accurately describe what the code was doing.
  • In my mind all bugs arise from an unexpected set of conditions and/or poor coding. They may be unanticipated but that does not make them random.
    • by russotto (537200)

      Some bugs arise from such a complicated set of conditions that they appear to be random. These are the most interesting bugs, and the most infuriating.

      Other bugs can even be truly random; a race condition that depends on whether one thread gets scheduled on a processor that's running just slightly faster (or slower) than another, for instance.

      • > Other bugs can even be truly random; a race condition that depends on whether one thread gets scheduled on a processor

        The bug exists before the processor is even purchased.
        The bug is not random, though the output may be influenced by a random event. The output isn't even random - I debugged one of those in the kernel and the result was spinlock. The only random part is WHEN the problem becomes visible to the user.

        That reminds me of a Heisenbug we had once. Completely off topic, but we once had a bug

  • Besides when it's caused by interference from background radiation?
    • by hibiki_r (649814)

      Don't look at the bugs individually, but at the distribution of bugs in the application. Bugs do not have an equal chance of appearing anywhere in a codebase, so their distribution is not random. There is much you can learn about the quality of your system and your development process depending on where the bugs are found, and this knowledge is actionable. Maybe your team needs to take unit tests more seriously. Maybe your developers have no idea of what are realistic scenarios. Maybe you are building on a

  • by khb (266593) on Sunday December 29, 2013 @11:44PM (#45815811)

    Bugs are never RANDOM. Bugs are, by definition, an error (human blunder ... incorrect design, improper code, etc.)

    • No, a select few are random. One of my coworkers from a decade ago randomly copied portions of if-then statements trying to get them to work. "This branch works so I'll copy it, paste it, and fiddle with the conditions until something else happens." Never mind those two branches required completely different decisions. Looking at one part of his program, a third of his if-then branches could never be run because they would always evaluate to false. I pointed that out to him and he couldn't offer a shre

  • True random data has maximal information content (or close to that at least).  Meaningful software does not.  It may be incomprehesible, but it is not random in nature, and there will always be many points of view from which it is far from random.  Find one of these, and you have a toehold into how it works.

Luck, that's when preparation and opportunity meet. -- P.E. Trudeau

Working...