Debugging Expert Wins ACM Dissertation Award 83
An anonymous reader writes "The Association for Computing Machinery (ACM) is reporting that Ben Liblit has been awarded the 2005 Doctoral Dissertation Award for his study on understanding and fixing software 'bugs' in the real world. From the article: 'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers. His approach uses sparse random sampling rather than complete data collection for gathering information from the experiences of large numbers of software end users. It also simultaneously ensures that the observed data is an unbiased, representative subset of the complete program behavior across all runs.' Slashdot broke the story on this research back in 2003. Apparently the project is still going strong."
dissertation: (Score:1, Funny)
blargh (Score:1, Offtopic)
'leveraging'
*tears out own hair and screams*
Comment removed (Score:4, Funny)
Re:blargh (Score:3, Informative)
a. To provide (a company) with leverage.
b. To supplement (money, for example) with leverage.
2. To improve or enhance: "It makes more sense to be able to leverage what we [public radio stations] do in a more effective way to our listeners" Delano Lewis.
So listen and listen good all you academic paper writers: unless what you really mean by "leverage" is "improve", don't use it.
"Liblit's dissertation proposes a method for leveraging the key strength of use
Re:blargh (Score:4, Interesting)
The money aspect you refer to has to do with debt financing whereby you manage to use your equity to finance something larger than your equity. I don't think the article is referring to corporate finance.
In a perfect world you would use a few people who would recognize and fix the bugs. These people would never talk to the users. They would have no need to and neither would gain from the experience.
In the world that I exist in, users are the ones who spot the bugs, specifically the circumstances under which the bugs exhibit themselves. I use my user's eyes to leverage {user's eyes, my skills}.
If all you mean is "improve", you would not use a word which essentially demands a discrepancy in the metrics between cause and effect.
b. To supplement (money, for example) with leverage.
If you add money to an account because of a margin call, does this increase or decrease your leverage? That is a horrible excuse for a definition.
Sounds like Doc Watson (Score:4, Insightful)
The current idea seems to be tracking the same termination events in the same way as Doctor Watson and sending the relevant data back to UWisc without informing the user. It sounds like a good idea, but I doubt it is in Liblit's power to fix Windows OS bugs.
Re:Sounds like Doc Watson (Score:1)
My Grandfather was having trouble running FlightGear under windows and sent one of those to Microsoft. Even got a response telling him it was a driver problem.
I send Firefox errors, just to give them some hope
Re:Sounds like Doc Watson (Score:5, Informative)
Informed participation is a really big deal for me. No user should ever find themselves participating in the Cooperative Bug Isolation Project without their knowledge. Opt-in is explicit and revokable, and if the opt-in system runs into trouble of any kind, the fallback position is no data reporting at all.
The whole thing collapses if users don't trust me. So I've taken every measure I can think of to ensure that they can. Please see the relevant project page for more details about privacy matters [wisc.edu].
Working on it! Check back in with me in a few years ... maybe less. :-)
Request for more information (Score:3, Interesting)
However, asking us to read 170-odd pages of your dissertation is a little much. Would it be possible to describe the data collection system, how reports are generated and if the reports are sent automatically or as in the case of Dr. Watson sent with user approval. Also, what types of bugs you found using your statistical methods, as well as what types of bugs you
Re:Request for more information (Score:5, Informative)
Hey, it's a real page-turner. Well, it has pages and they turn, at least.
The other questions you ask are all good ones, but a bit much to address in a Slashdot comment. Please see the project home page [wisc.edu] for more information. The "Learn More" [wisc.edu] page may answer some of your questions, and there are additional drill-down pages from there with even more technical material on selected topics.
Please understand that I don't mean to brush off your insightful questions. They are just questions for which satisfactory answers are hard to give in a sentence or two.
Re:Request for more information (Score:2)
Why? If you're really interested in that area, that's not much material and it's important research. I know that I'm going to read it just because it's interesting.
But if you're not deeply interested, you will be able to pick the most interesting bits by looking at the table of contents, won't you? Or is that too much effort, too? Besides, the project has a Web site that is even referenced in the /. blurb -- did you even bother to lo
Shaka, when the walls fell (Score:1)
Doesn't it make sense to ask him while he's here to discuss the topic in a simplified manner since he's the world's leading expert on the topic? This is a discussion forum, so being able to hear him express the concepts allows us to participate with him in a two way transfer of ideas.
The alternative is to write everything down and simply refer to documents instead of engaging people in conversation.
Re:Shaka, when the walls fell (Score:2)
If you would have co
Re:Shaka, when the walls fell (Score:1)
I'll keep that in mind the next time I have an interest in something and have the opportunity to have direct access to an expert and have essentially carte blanche to ask anything. Read up on it first, then hope that the expert hangs around while I'm busy informing myself.
As for the thrust of my main post, it was not about
Re:Sounds like Doc Watson (Score:5, Interesting)
Plenty of users do. There's a great blog posting by Raymond Chen called There's an awful lot of overclocking out there [msdn.com] where he talks about investigating some of these "Watson" crashes.
The crashes were impossible - instructions like
Turns out unscrupulous vendors were selling overclocked computers without informing buyers. Pretty cool article.
Re:Sounds like Doc Watson (Score:2)
xor eax, eax
Watch out -- you may have just reverse-engineered Sony's latest DRM enforcement mechanism!
Re:Sounds like Doc Watson (Score:1)
The crashes were impossible - instructions like
How is that impossible? It's a single-byte, single-clock, u/v pipe op to zero eax... How do you (or the original author of the comment) do it? mov eax,0? *snicker* That'd certainly explain some of the Windows bloat...
Not really (Score:1)
Re:Sounds like Doc Watson (Score:2, Interesting)
Not that I mind sending back data when it can be useful; if someone is going to look at the error logs, memory, etc., and try and make it so that it won't crash aga
Re:Sounds like Doc Watson (Score:3, Interesting)
Automated dump analysis is an old idea in the mainframe world, but almost unknown outside it. The microprocessor world grew up with interactive debuggers and an early user-as-programmer assumption. This hasn't translated well to the modern software world.
In the mainframe world, there have even been mainframes that recorded the last 64 or so branches using dedicated hardware, so that afte
Doctor Brown, you mean. (Score:2)
Re:Sounds like Doc Watson (Score:1)
Re:Sounds like Doc Watson (Score:2)
Check out FindBugs for finding bugs in Java (Score:5, Informative)
http://findbugs.sourceforge.net/ [sourceforge.net]
Re: (Score:1)
Re:Check out FindBugs for finding bugs in Java (Score:1)
Re:Check out FindBugs for finding bugs in Java (Score:2)
Re:Check out FindBugs for finding bugs in Java (Score:1)
Heh, you're right, and thanks for the mention!
Re:Check out FindBugs for finding bugs in Java (Score:2)
I guess I'm just grumpy beca
Re:Check out FindBugs for finding bugs in Java (Score:1)
Re:Check out FindBugs for finding bugs in Java (Score:2)
Since they have a different field of analysis (directory vs. file), you get completely different findings...
Re:Check out FindBugs for finding bugs in Java (Score:1)
Now I just have to push it to the other people in the company.
Re:Check out FindBugs for finding bugs in Java (Score:2)
You can try, but you'll succeed anyway using it yourself. I used it on a piece of the codebase and in the next teammeeting, said I'd "found some potential bugs with this new tool". People will get interested. That's actually how I found out about FindBugs, because of a colleague who was impressed with me using PMD.
Standup Fight? (Score:2)
Seriously, congradulations, Ben!
Thank you, open source community (Score:5, Informative)
This research has been a wonderful collaborative effort, and many people deserve to share the credit. To quote from part of the Acknowledgements section of my dissertation [wisc.edu]:
So thanks, Slashdot, for helping me find those users (or helping them find me). The exposure was invaluable. And thanks, open source community, for your participation. I've benefitted greatly from standing on your massed shoulders. This could not have happened without you.
Re:Thank you, open source community (Score:2)
Re:Thank you, open source community (Score:2, Funny)
We will now debug your dissertation.
Now at the University of Wisconsin-Madison (Score:2, Informative)
Closed versus open source comparison (Score:1, Interesting)
Re:Closed versus open source comparison (Score:1)
Heh... (Score:5, Funny)
Re:Heh... (Score:5, Informative)
Yes, exactly. The users are beta testers; we may as well admit it. I want to make them better beta testers. :-)
Re:Heh... (Score:2)
Re:Heh... (Score:2)
(hi, Ben!
Re:Heh... (Score:1)
Re:Heh...users are the Beta (Score:1)
Re:Heh...users are the Beta (Score:1)
If you mean incentives from the users' perspective, I like to pitch it this way. My statistical methods naturally tend to "learn" the most, most quickly, about the failures that happen most often. So the more a user participates, the more the developers' attention will be swayed to the bugs that user cares about. Thus, users can help steer bug triage.
Open source bug trackers can work the same way. When you report bugs in some project's Bugzilla s
Almost like infinite monkeys writing Shakespeare (Score:2, Interesting)
Drinking Club (Score:1)
Re:Drinking Club (Score:1)
Alternative (complementary?) approach (Score:2)
http://www.agitar.com/products/20051101-agitator.h tml [agitar.com]
They only work with Java. Here is
Re:Alternative (complementary?) approach (Score:2)
You do better if you actually force the de
Dammit, wrong article (Score:1)
Re:Speaking of debugging through sampling... (Score:1)
Liblit. There's an off-by-one-error in your count of L's. (It doesn't matter what they say about you, as long as they spell your name right.)
Yeah, that's me along with a truly fantastic team of collaborators. And there's more where that came from [wisc.edu].
reminds me of my rules on reporting list outages (Score:3, Funny)
I told people we were switching to new software (Mailman)- and that if they got an error message or similar, to flip a quarter X times (I forget how many) and ONLY email me if they got all heads. I didn't want to get a couple dozen reports of the same problem, and I figured that if there were any problems, they'd affect a large set of the 1000+ users of the list.
It worked brilliantly.
Guardian doesn't think so (Score:1, Interesting)
Where are the results? (Score:2)
I learned my lesson with the cddb disaster: don't submit your own data unless you can mirror it yourself. Otherwise, if someone gets bored or greedy, everybody's hard work gets lost forever. Or, worse, it gets subverted to make profit for Gracenote.
Mirroring is easy enough even if you have almost no bandwidth: use bittorrent. So, where are the submitted results?
Real world bugs are not about a language (Score:2)
The QA reports that there is an error on the screen while they are trying to save some data. The error is intermittent, it happens for some data sets but not for others. Investigation shows the following:
1. The data records that need to be saved must be first compared to existing data records for the same primary keys, if there are records, then the data is up
Minor Rants (Score:2)
I always think it's weird, something about the art and craft of getting better code out gets little notice, but some FireFox alpha (which is feature INcomplete, really only for extension developers) gets 200,000 messages. Mabe a third of them will be flames "(IE/FireFox/Whatever) is so buggy, you suck unless you switch to (whatever)". But tools to debug these get ignored. How much work is going into KDE vs GNOME, and even a group that wants to fork KDE (deity() help us).
RANT #2
I really think we're