Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Distributed Statistical Debugging

Posted by michael on Thu Oct 09, 2003 12:49 PM
from the data-crunching dept.
Bug Programming IT Technology
Luis Villa writes "The Cooperative Bug Isolation Project at UC Berkeley and Stanford is working on statistical debugging techniques to report, find, and fix the bugs that drive the most users crazy every day. A handful of outside bug volunteers have been running the project's special feedback builds for a few weeks, and that has generated some really interesting data. But for strong results they need more runs. /. has been known to generate those kinds of big numbers ;) Their site has feedback builds of several open source applications, and the entire project is open sourced. Read more about it, then install some applications, and help them make our free software better for everyone. I'm really looking forward to the end results."
+ -
story

Related Stories

[+] Debugging Expert Wins ACM Dissertation Award 83 comments
An anonymous reader writes "The Association for Computing Machinery (ACM) is reporting that Ben Liblit has been awarded the 2005 Doctoral Dissertation Award for his study on understanding and fixing software 'bugs' in the real world. From the article: 'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers. His approach uses sparse random sampling rather than complete data collection for gathering information from the experiences of large numbers of software end users. It also simultaneously ensures that the observed data is an unbiased, representative subset of the complete program behavior across all runs.' Slashdot broke the story on this research back in 2003. Apparently the project is still going strong."
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Distributed Statistical Debugging 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • if you want to track a lot of bugs try to hook it into windows :)
  • How can they tell which bugs are the culprits driving the users crazy every day? Do they monitor large numbers of keys being pressed and map out the physical keyboard layout to determine how hard the users head is smacking into the keyboard?

    • Re:Bugs.... (Score:2, Informative)

      by Benoni (132028)

      cybermace5 asked:

      How can they tell which bugs are the culprits driving the users crazy every day?

      It's a numbers game. We're looking for statistical trends in large numbers of runs. That means we will learn the most, the most quickly, about the bugs that are happening most often to the most users. Bug triage falls out as a natural consequence of the sparse sampling and statistical modeling approach.

      That said, you suggestion about measuring head-into-keyboard smacks isn't half bad. There are some gr

      • The bugs that drive me nuts don't result in crashes. Mozilla, for example, stops handling keystrokes when you open a tab on an unresponsive page.

        This project does nothing at all to deal with the bugs which are most annoying and difficult to report.

        Perhaps it should be called it the Cooperative Crash Isolation project.

        • Re:Bugs.... (Score:2, Informative)

          by Benoni (132028)
          In the future we may add the ability for users to manually report non-crash application misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.

          But in the current public deployment you are quite right that we only pick up on crash bugs. Consider that the more general project name gives us something to aspire to. :-)
          • That sounds spectacularly cool, but difficult to implement. I can think of a bunch of ways to do it, none perfect.

            The highest profile example is Mozilla's terrible "voting" system, where I must already know the bug's number before I can be annoyed by it.

            The most promising example that I've seen so far is bugs.kde.org, wherein a user attempting to report a bug is presented with a list of likely preexisting matches. If users were encouraged to add a noiseless "me too" to a bug database of that sort, it mi
      • I still don't understand. When I use your modified Nautilus and can't copy and paste text into rxvt, how do you know there's an issue? It seems to me that sparse sampling is exactly the wrong way to find something like that -- you need to look at blocks of time and observe repeated efforts to GET WORD TO FORMAT MY *$^&$* TABLE CORRECTLY or PASTE THIS *^$#$* THING FROM KONQUEROR TO XCHAT!

        Naturally, as you're merely CS doctoral student at Cal and I'm some ignoramus raving at you on Slashdot, my opinion ne

        • Re:Bugs.... (Score:2, Informative)

          by Benoni (132028)
          If you're a raving ignoramus, you're a raving ignoramus who asks good questions. :-)

          In our current deployment, we only learn about problems that crash the application. These are important bugs, but they are certainly not the only bugs.

          We are considering ways to let users manually report non-crash misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging technique
  • They patented error/bug feedback software or something last month.

  • debugging techniques to report, find, and fix the bugs that drive the most users crazy every day.

    Can you do something about these ladybugs that are driving Midwesterners nuts [google.com], then?
  • personally i think that projects such as this are a good thing. the more people test software the small the chance of bugs being in the final release. now if only microsoft would follow suit...
    • MS does - in XP when an app crashes, a debugger loads up and offers to send a minidump to MS.
    • They don't follow suit.

      They had this idea first.

      MS has a database of all the crashes reported by its bug reporting tool(in XP & WS03), these crashes are analyzed, sorted and categorized and help them fix the bugs.
  • It's a feature

    Sorry...had to be said
  • When I boot my computer at work, it loads up Windows. Make it stop!
  • I didn't see any of the really interesting data referred to in the article on the website... anyone care to share a link?
  • "statistical debugging techniques to report, find, and fix the bugs"

    A debugger that FIX BUGS?

    What's next? A compiler that brews beer?

  • This might be really cool in combination with things like distributed genetic programming. =) Are there any open source projects that combine various distributed programming/bug tracking techniques with a central web interface to track stats/view results, etc..?

  • Other architectures? (Score:3, Interesting)

    by gouldtj (21635) on Thursday October 09 2003, @01:28PM (#7174839) Homepage Journal

    From my reading of this, it sounds like the data is going to be architecture specific (i.e. x86, PPC). So that means that those hundreds of thousands of samples that are needed, would be needed for every architecture.

    Hmph, mean they probably won't figure out why some programs are seemingly less stable on PPC :( But, I guess many bugs effect every architecture, so I can be happy about that ;)

    • Re:Other architectures? (Score:5, Insightful)

      by Ed Avis (5917) <ed@membled.com> on Thursday October 09 2003, @02:11PM (#7175210) Homepage
      Yes, but if you fix a crash-causing bug (such as a buffer overrun) on one architecture, it's likely that the same bug was present on others. Or rather, would also cause a crash on other platforms - since a bug is a property of the source code, the meaning of the program, and the fact that a program happens to work on Tuesday doesn't mean the bug is not there.

      Consider: projects such as openssh and Apache do not issue separate vulnerability warnings for different platforms. If a memory-corruption bug exists, it is probably exploitable on most platforms given the right conditions.

      Still, it's very likely that some particular bug cause crashes often on PPC, but is less likely to be tickled on x86. Then it might not get fixed, because it's less likely to be reported. But if a PPC user does report it, the x86 users benefit from the fix too, even if for them it was a fairly obscure and not-often-noticed bug.
      • I was getting to respond with the same answer. Someone mod the parent up! Good job!

      • Yes, but if you fix a crash-causing bug (such as a buffer overrun) on one architecture, it's likely that the same bug was present on others. Or rather, would also cause a crash on other platforms - since a bug is a property of the source code, the meaning of the program, and the fact that a program happens to work on Tuesday doesn't mean the bug is not there.

        Yes, but I was thinking more like endianness bugs and casting pointers to integers on 64-bit machines (yeah, there are still people who do this).

    • The instrumentation is injected early during compilation, at roughly the source code level. We're looking at things like variables, function returns, branches, etc.: source-level constructs. Thus, the data is not architecture specific.

      Of course, if the bug is architecture specific, then identifying the vulnerable architecture is part of what we'd want the system to produce as analysis results.

      By design the system learns the most about the most frequent failures. So yes, that probably means that we wi

  • Number 1 on the list (Score:3, Funny)

    by one9nine (526521) on Thursday October 09 2003, @01:30PM (#7174857) Journal
    Duplicates on Slashdot. ;-)
  • I wager this is violating Microsoft's newly issued patent [slashdot.org] on crash reporting.
  • Microsoft has been using this feature in XP and it apparently does not work. One of our applications crashes frequently for no reason at all. I have told all of our users (1600+) to just click the "SEND" button when it asks to send the "encountered a problem report" back to Microsoft.

    In my calculations Microsoft must have gotten at least 30,000 reports of this bug and it is still not fixed yet...

  • so /. will stop losing comments into thin air like it did two of my replies to this article. :-|
  • From reading the FA it appears that this is for RPM based distros only. Any plans for alternate distos?
  • I really hope slashdotters go out of their way to use this software, because I know the people working on it and they are great, and due to the complexity of the problem, it works best when there's insane amounts of people running it. The best part is that you don't have to do anything, just run your favorite programs to help out the study. And as for extending this to other architectures and stuff like that, you have to realize this is just a bunch of researchers who are starting to work on this issue, a
  • What - no ebuild distribution? Just RPM?

    I guess they don't want the thousands of gentoo users to participate... Shame - they're the sort to be likely to contribute something to said bug reports since they are more likely to know something about programming...
      • There isn't any reason they can't have both.

        Personally, I like the gentoo portage system. I'd really prefer not to mix rpms and ebuilds on my system - my guess is that it will eventually come back to haunt me. I'd probably sooner just download a source tarball and stick it in usr/local where I can get rid of it later if I put in the protage-supported version.

        That being said - I have no religious devotion to portage. If somebody else wants rpms or to use apt-get, all the more power to them...
  • The idea is interesting, but the current implementation can only handle C - not C++ or other languages. That means it can't handle Open Office currently.

    That's too bad, because the open source application most needing high reliability is Open Office. Don't get me wrong - Open Office is quite useable, and I experience fewer reliability problems with it than with Microsoft Office. However, making Open Office extremely reliable (far beyond most software) would help far more people than getting the GIMP mor

    • RTFA (Score:3, Insightful)

      by jbellis (142590) *
      this isn't the same thing at all.

      the "stanford bug checker" is a static source analysis program. this is something else entirely, and arguably more useful.
      • the "stanford bug checker" is a static source analysis program. this is something else entirely, and arguably more useful.

        I think they both have their use. This system takes samples of the state of a running program and does a statistical to see wich state is likely to be the cause of a crash. Not really something that can't be done by a programmer using a debugger, but automatic and given sufficient input i might point you to exactly what condition causes a crash. The stanford bug checker try to prevent
        • either you're not a programmer, you STILL haven't read the article, or you're an idiot. they have NOTHING IN COMMON beyond the name "Stanford."

      • Russians won WWII (Score:1, Offtopic)

        by melted (227442)
        Not Americans. Americans only chimed in when it became clear that the Russians are kicking the crap out of Hitler. You can't imagine how seriously all this talk about Americans "winning" the WWII pisses off the people who fought with Wermacht for 5 years and lost 35 million brothers and sisters in combat to actually win the war.
          • More correct figure is about 35mln. Most of Stalin's repressions occured in late 30's, and they're not a part of 35 mln. There was no "alliance" with Hitler. There was, hovewer a pact that USSR and Germany won't attack each other which made a lot of sense, because in early 40's Germany had more military power than any single other country on the planet, and it was pointless for Russia to even try to defeat them alone.

            Nobody turned over Jews, SS troops were good at finding them themselves. Nobody protected
              • What you'd read in high school history textbook wasn't true. :0) And no, Americans did not win WWII. They only chimed in in the end when it was already won.
                • Oh yeah, why was Stalin demanding a second front (i.e. Italy, but the Germans stalled the American and British progress by "heading for the hills", Dieppe, where Canadian commandos valiantly failed, and finally Normandy)? I guess he could have handled the whole thing a alone?

                  Never mind, we asked for Stalin's help to fight Japan. Which he did. After the atomic bombs were dropped.

        • Re:TO America: (Score:2, Insightful)

          by 91degrees (207121)
          Freedom of speech should not be extended to those who seek to curtail the freedom of speech

          Why the hell not!? Freedom of speech is a right. Not a luxury good to be traded.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.