Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Bug Programming IT Technology

Distributed Statistical Debugging 103

Posted by michael from the data-crunching dept.
Luis Villa writes "The Cooperative Bug Isolation Project at UC Berkeley and Stanford is working on statistical debugging techniques to report, find, and fix the bugs that drive the most users crazy every day. A handful of outside bug volunteers have been running the project's special feedback builds for a few weeks, and that has generated some really interesting data. But for strong results they need more runs. /. has been known to generate those kinds of big numbers ;) Their site has feedback builds of several open source applications, and the entire project is open sourced. Read more about it, then install some applications, and help them make our free software better for everyone. I'm really looking forward to the end results."
This discussion has been archived. No new comments can be posted.

Distributed Statistical Debugging More

Distributed Statistical Debugging

Comments Filter:
  • if you want to track a lot of bugs try to hook it into windows :)
    • Microsoft has been doing this for a few years now, and they've used the resulting data to fix the "worst offending" bugs.

      The results are encouraging: Office XP programs with Office XP Service Pack 2 basically never crash, at least in our organization of more than 100 desktop and mobile users.

      I certainly can't say the same for OpenOffice 1.0... it seemed to crash every time I moved the mouse, so I eventually went back to Office XP on Windows.

      I find it disingenous that /. fails to mention Microsoft's wide

  • How can they tell which bugs are the culprits driving the users crazy every day? Do they monitor large numbers of keys being pressed and map out the physical keyboard layout to determine how hard the users head is smacking into the keyboard?

    • Re:Bugs.... (Score:2, Informative)

      by Benoni ( 132028 )

      cybermace5 asked:

      How can they tell which bugs are the culprits driving the users crazy every day?

      It's a numbers game. We're looking for statistical trends in large numbers of runs. That means we will learn the most, the most quickly, about the bugs that are happening most often to the most users. Bug triage falls out as a natural consequence of the sparse sampling and statistical modeling approach.

      That said, you suggestion about measuring head-into-keyboard smacks isn't half bad. There are some gr

      • The bugs that drive me nuts don't result in crashes. Mozilla, for example, stops handling keystrokes when you open a tab on an unresponsive page.

        This project does nothing at all to deal with the bugs which are most annoying and difficult to report.

        Perhaps it should be called it the Cooperative Crash Isolation project.

        • Re:Bugs.... (Score:2, Informative)

          by Benoni ( 132028 )
          In the future we may add the ability for users to manually report non-crash application misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.

          But in the current public deployment you are quite right that we only pick up on crash bugs. Consider that the more general project name gives us something to aspire to. :-)
          • That sounds spectacularly cool, but difficult to implement. I can think of a bunch of ways to do it, none perfect.

            The highest profile example is Mozilla's terrible "voting" system, where I must already know the bug's number before I can be annoyed by it.

            The most promising example that I've seen so far is bugs.kde.org, wherein a user attempting to report a bug is presented with a list of likely preexisting matches. If users were encouraged to add a noiseless "me too" to a bug database of that sort, it mi
      • I still don't understand. When I use your modified Nautilus and can't copy and paste text into rxvt, how do you know there's an issue? It seems to me that sparse sampling is exactly the wrong way to find something like that -- you need to look at blocks of time and observe repeated efforts to GET WORD TO FORMAT MY *$^&$* TABLE CORRECTLY or PASTE THIS *^$#$* THING FROM KONQUEROR TO XCHAT!

        Naturally, as you're merely CS doctoral student at Cal and I'm some ignoramus raving at you on Slashdot, my opinion ne

        • Re:Bugs.... (Score:2, Informative)

          by Benoni ( 132028 )
          If you're a raving ignoramus, you're a raving ignoramus who asks good questions. :-)

          In our current deployment, we only learn about problems that crash the application. These are important bugs, but they are certainly not the only bugs.

          We are considering ways to let users manually report non-crash misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging technique
      • Benoni,

        I have been taking part in this for a while, (gnumeric) however frequently have it switched off, simply out of bandwidth concerns (I pay through the nose if over a rediculously low limit.) Mia Culpa I've not read all the documentation at your site (only so many hours in a day, etc.) Does this only send back reports when gnumeric crashes or is it sending back reports on a constant basis? How much bandwidth does it use? Can I just leave it on knowing it is trivial usage?

        Cheers,
        H.

        • All good questions, Harry8.

          Does this only send back reports when gnumeric crashes or is it sending back reports on a constant basis?

          It sends back a report when the application exits. Crashes include some extra crash information, but successful exits upload a report too. (That way we can look for the differences between good runs and bad.) See the web site [berkeley.edu] for more details on exactly what's in those reports.

          There is no continuous reporting while the program runs: it's just one blob sent when the pro

          • Cheers Benoni,
            That level of bandwidth use is fine with me & well worth it, I'll just switch it on for all users of that machine & forget about it. I guess I was only concerned if it was constantly sending back information - which I thought might hurt the hip-pocket after a few good 6-7 hour sessions of usage. Turns out there is no cause for concern - which is all good.
            I'll be interested to see what results it all brings.
            All the best.
  • They patented error/bug feedback software or something last month.

  • debugging techniques to report, find, and fix the bugs that drive the most users crazy every day.

    Can you do something about these ladybugs that are driving Midwesterners nuts [google.com], then?
  • personally i think that projects such as this are a good thing. the more people test software the small the chance of bugs being in the final release. now if only microsoft would follow suit...
    • MS does - in XP when an app crashes, a debugger loads up and offers to send a minidump to MS.
    • They don't follow suit.

      They had this idea first.

      MS has a database of all the crashes reported by its bug reporting tool(in XP & WS03), these crashes are analyzed, sorted and categorized and help them fix the bugs.
  • It's a feature

    Sorry...had to be said
  • When I boot my computer at work, it loads up Windows. Make it stop!
  • I didn't see any of the really interesting data referred to in the article on the website... anyone care to share a link?
  • "statistical debugging techniques to report, find, and fix the bugs"

    A debugger that FIX BUGS?

    What's next? A compiler that brews beer?

  • This might be really cool in combination with things like distributed genetic programming. =) Are there any open source projects that combine various distributed programming/bug tracking techniques with a central web interface to track stats/view results, etc..?

  • Other architectures? (Score:3, Interesting)

    by gouldtj ( 21635 ) on Thursday October 09, 2003 @01:28PM (#7174839) Homepage Journal

    From my reading of this, it sounds like the data is going to be architecture specific (i.e. x86, PPC). So that means that those hundreds of thousands of samples that are needed, would be needed for every architecture.

    Hmph, mean they probably won't figure out why some programs are seemingly less stable on PPC :( But, I guess many bugs effect every architecture, so I can be happy about that ;)

    • Re:Other architectures? (Score:5, Insightful)

      by Ed Avis ( 5917 ) <ed@membled.com> on Thursday October 09, 2003 @02:11PM (#7175210) Homepage
      Yes, but if you fix a crash-causing bug (such as a buffer overrun) on one architecture, it's likely that the same bug was present on others. Or rather, would also cause a crash on other platforms - since a bug is a property of the source code, the meaning of the program, and the fact that a program happens to work on Tuesday doesn't mean the bug is not there.

      Consider: projects such as openssh and Apache do not issue separate vulnerability warnings for different platforms. If a memory-corruption bug exists, it is probably exploitable on most platforms given the right conditions.

      Still, it's very likely that some particular bug cause crashes often on PPC, but is less likely to be tickled on x86. Then it might not get fixed, because it's less likely to be reported. But if a PPC user does report it, the x86 users benefit from the fix too, even if for them it was a fairly obscure and not-often-noticed bug.
      • I was getting to respond with the same answer. Someone mod the parent up! Good job!

      • Yes, but if you fix a crash-causing bug (such as a buffer overrun) on one architecture, it's likely that the same bug was present on others. Or rather, would also cause a crash on other platforms - since a bug is a property of the source code, the meaning of the program, and the fact that a program happens to work on Tuesday doesn't mean the bug is not there.

        Yes, but I was thinking more like endianness bugs and casting pointers to integers on 64-bit machines (yeah, there are still people who do this).

        • I'd say relying on a particular endianness is a bug even if you only run the program on one platform. Likewise assuming that an int and a pointer are the same size is always a bug. In both cases you are relying on something that's undocumented.

          I guess this illustrates the difference between a bug and a defect. Assuming sizeof(int) == sizeof(void *) is a bug in code, but it may not be a defect in the final executable if you compile for a platform where they do happen to be the same size.

    • The instrumentation is injected early during compilation, at roughly the source code level. We're looking at things like variables, function returns, branches, etc.: source-level constructs. Thus, the data is not architecture specific.

      Of course, if the bug is architecture specific, then identifying the vulnerable architecture is part of what we'd want the system to produce as analysis results.

      By design the system learns the most about the most frequent failures. So yes, that probably means that we wi

  • Number 1 on the list (Score:3, Funny)

    by one9nine ( 526521 ) on Thursday October 09, 2003 @01:30PM (#7174857) Journal
    Duplicates on Slashdot. ;-)
  • I wager this is violating Microsoft's newly issued patent [slashdot.org] on crash reporting.
  • Microsoft has been using this feature in XP and it apparently does not work. One of our applications crashes frequently for no reason at all. I have told all of our users (1600+) to just click the "SEND" button when it asks to send the "encountered a problem report" back to Microsoft.

    In my calculations Microsoft must have gotten at least 30,000 reports of this bug and it is still not fixed yet...

    • That assumes that windows is the problem, and not your application
    • In my calculations Microsoft must have gotten at least 30,000 reports of this bug and it is still not fixed yet...

      Yes, but what makes you think the feature doesn't work?
    • I would click SEND more often myself if the app would give you back a "support ticket number", or a bug ID in the DB so that you could look back later and see how many other entries have been logged for the same error and if a patch is in the making ...

      Now that would be cool! (and very much like an automated bugzilla interface compiled w/ the code ...)

    • Re:Microsoft's Distributed Statistical Debugging (Score:1, Informative)

      by Anonymous Coward
      That isnt how watson works, the bugs are contained in what is called "Buckets", once the bucket for that particular stack trace (that particular bug) has reached its limit for enough information to debug, you can press send all you want, the bucket limit has been reached, they dont see 30,000 of the same crashes, maybe 5 or 10, they dont need more.

      Click all you want. Its pointless once they got enough crash info.

      Now they should open the watson buckets for developers on MSDN so we can pull the info from th
      • Click all you want. Its pointless once they got enough crash info.

        It's not pointless. They don't store every crash dump, but the total number of crashes is recorded. Higher numbers mean higher chances of a fix. Of course, that's assuming that the app didn't wipe out its stack so the fault goes into some generic "we have no idea what's going on" bucket, and that all crashes are actually identical or similar enough, and that the problem is actually in Windows and not the app etc.

        As for making the crash info

    • I'd wager that they see one report from one IP.
  • so /. will stop losing comments into thin air like it did two of my replies to this article. :-|
  • From reading the FA it appears that this is for RPM based distros only. Any plans for alternate distos?
  • I really hope slashdotters go out of their way to use this software, because I know the people working on it and they are great, and due to the complexity of the problem, it works best when there's insane amounts of people running it. The best part is that you don't have to do anything, just run your favorite programs to help out the study. And as for extending this to other architectures and stuff like that, you have to realize this is just a bunch of researchers who are starting to work on this issue, a
  • What - no ebuild distribution? Just RPM?

    I guess they don't want the thousands of gentoo users to participate... Shame - they're the sort to be likely to contribute something to said bug reports since they are more likely to know something about programming...
    • I guess they don't want the thousands of gentoo users to participate.

      Hey now, don't take it personally. That's not what it's about. It's just a matter of prioritizing our limited resources. I would love to add Gentoo to our list, and in fact foser [gentoo.org] has expressed an interest in teaming up. We just need to find the time to get all the pieces put together.

  • The idea is interesting, but the current implementation can only handle C - not C++ or other languages. That means it can't handle Open Office currently.

    That's too bad, because the open source application most needing high reliability is Open Office. Don't get me wrong - Open Office is quite useable, and I experience fewer reliability problems with it than with Microsoft Office. However, making Open Office extremely reliable (far beyond most software) would help far more people than getting the GIMP mor

  • Moooo.
  • I downloaded MozillaFirebird in spanish and I get the netscape quality feedback agent each and every time mozilla dies, I try to answer the questions but I never get the feeling something good is actually working
  • Actually... First time I remember seeing anything like this was from a company called FullCircle, who had a bug reporting tool that Netscape had packaged up into its client and called it the Netscape Quality Feedback Agent - but it was indeed FullCircle. Those crash reports got logged, and people actually did go through on a regular basis, especially after beta releases and milestone releases, and go looking through at all of the top crashers; top crash analysis days were regularly held, and people did ho

Slashdot Top Deals

Understanding is always the understanding of a smaller problem in relation to a bigger problem. -- P.D. Ouspensky

Close