Microsoft Fuzzing Botnet Finds 1,800 Office Bugs 111
CWmike writes "Microsoft uncovered more than 1,800 bugs in Office 2010 by tapping into the unused computing horsepower of idling PCs, a company security engineer said on Wednesday. Office developers found the bugs by running millions of 'fuzzing' tests, a practice employed by both software developers and security researchers, that searches for flaws by inserting data into file format parsers to see where programs fail by crashing. 'We found and fixed about 1,800 bugs in Office 2010's code,' said Tom Gallagher, senior security test lead with Microsoft's Trustworthy Computing group, who last week co-hosted a presentation on Microsoft's fuzzing efforts at the CanSecWest security conference. 'While a large number, it's important to note that that doesn't mean we found 1,800 security issues. We also want to fix things that are not security concerns.'"
xkydgtufhlofhil (Score:5, Funny)
ghulkgiplgbvihlnk luioguilgil.bjohj110-o; Huto;bn
Re: (Score:3, Interesting)
ghulkgiplgbvihlnk luioguilgil.bjohj110-o; Huto;bn
I don't understand this Score:4 Insightful comment. Can someone explain?
Re:xkydgtufhlofhil (Score:5, Informative)
don't understand this Score:4 Insightful comment. Can someone explain?
Even though your name does look quite suspicious, I'll try to explain anyway.
:-)
The parent is showing how fuzzing works:
Using random 'data' to test the various functions of software, so we can find out if a certain piece of input triggers undesirable behavior.
In this case you could say that he's not only giving an example, but is testing the slashdot user comments code as well.
But it's perhaps more an attempt at humor.
Re: (Score:2, Funny)
Windows: "It's not a bug, it's a feature."
GNOME: "It's not a bug, it's a design decision."
Re: (Score:2)
Re:xkydgtufhlofhil (Score:4, Insightful)
Re: (Score:2)
Re:xkydgtufhlofhil (Score:5, Informative)
The most obvious of these are assumptions like "a newline can't occur in a single-line field" (a mistake web developers often make, because they assume the data are coming from an HTML input element that only allows single-line data; but an attacker can in fact send anything they want in an http request), or "nobody's going to have a single-quote character in their name" (hello, SQL injection attack). This sort of thing is probably not a major factor in Office, because it's common for documents to have those kinds of characters in them. There might be a couple of weird old control characters (like the ASCII NUL, 000), but those bugs were probably found aeons ago.
A second major category of problematic assumptions assumptions has to do with languages and code pages and character sets. When software that was written to assume a particular character set (like ASCII, or Latin-1) or even just one code page at a time (like, whichever one is the system default) has to be extended to support more (like, especially, Unicode), you run into all kinds of nasties. Again, though, Office probably had to deal with these issues a couple of versions ago. They may have found a few more, but at this point it's probably not the most fertile ground any more.
When you're dealing with file formats, however, there are also things like "the value at offset 0x003C from the beginning of the object header contains the size of the object, which can never be more than 0xFFFF" and "an object can embed another object by referencing it, but there are never any circular references, because the software doesn't allow the user to put an object inside itself". These sorts of assumptions pop up every time you write or change code that reads a file format, so they never go away really. This sort of thing is probably *most* of what the Office team found, I suspect.
Re:xkydgtufhlofhil (Score:4, Informative)
Hey, I resemble that remark! And yes, it's resulted in chuckles over the years. Microsoft, DevelopMentor, random e-commerce sites... many have fallen to the Irish. When talking to security professionals, I introduce myself as "the woman whose name is a SQL injection attack", and it seems to help them remember me.
Re: (Score:2, Interesting)
Re: (Score:1, Offtopic)
somewhat relevant xkcd:
Exploits of a Mom
http://xkcd.com/327/ [xkcd.com]
Re: (Score:1)
Even though your name does look quite suspicious, I'll try to explain anyway.
Thank you for your explanation.
And your benefit of the doubt.
Re: (Score:3, Funny)
>In this case you could say that he's not only giving an example, but is testing the slashdot user comments code as well.
It's testing not just the user comments code, but also the moderation system code and the moderators themselves. In this case, it appears that he found a bug which causes the comment to be moderated Insightful by providing a certain combination of random characters as input. I will now attempt to replicate this problem.
______TEST DATA FOLLOWS______
TvaHokVAwgZGLrzPnDsIzHnKwuOOQEgaFskFJx
Re: (Score:3, Informative)
Fuzzing is a technique where you modify the data sent to a file, protocol or data parser (e.g. code that reads an xml file) by changing random bits. Thus, if you have a 'text' command, a fuzzer could change that to 'next', or if you have a quoted striing "test", the fuzzer could change the end quote to something else, e.g. ' "tests '.
Hence, what you can end up with is something that looks like random garbage.
Or user a sales. (Score:2, Interesting)
It is an alternative to the monkey test: Take a sales person from across the ahlloway and let him click on your application. If it does not crash or give absurd error messages you can do the actual testing.
GIGO!
Re: (Score:1)
Last time I tried that, it took me forever to get the feces out of the keyboard.
Re: (Score:1)
Re: (Score:3, Insightful)
Well color me red, here I thought this kind of testing should have been done prior to release. Guess the new model of software development is to have the users discover the bugs (can I get a smiley on this) instead of paying a QA team to test.
No, color you stupid. Office 2010 hasn't been released yet.
Nice try though.
Re: (Score:1)
Hey, Microsoft! (Score:5, Funny)
"We also want to fix things that are not security concerns."
It's 5AM EST. April Fools' day is over everywhere but a few pacific islands. Give it up already.
Re: (Score:3, Funny)
While a large number, it's important to note that that doesn't mean we found 1,800 security issues
Don't worry, we all know that you haven't fixed any security issues.
Re:Hey, Microsoft! (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2, Funny)
It would actually be believable except for the "also". :P
Re: (Score:2)
There were only 1800 bugs. Not like it was anything serious.
Elsewhere in this "issue" of slashdot, we have the story of Microsoft's OOXML failing the standards test because they've only bothered to address about 5% of the issues there.
You just had another "emergency" security update for IE a few days ago.
Microsoft wants the world to run on its software and most of the business world does, not to mention the government. But they are willing to sell it to you BEFORE they bother to take a look and see how well
New bugs (Score:5, Insightful)
I wonder how many "new" bugs they'll create by fixing the found bugs.
Anyway, nice to see that they're performing fuzzing tests, not enough people/companies do that. There's also quite little tool support for it.
Re: (Score:2, Interesting)
Yeah, just like the numerous regressions I see in the Linux kernel, WINE, Ubuntu releases etc.
Re: (Score:1)
Yeah, just like the numerous regressions I see in the Linux kernel, WINE, Ubuntu releases etc.
Why is this modded offtopic? It's cool and popular to poke fun at Microsoft but heaven forbid you point out Linux, WINE, and Ubuntu have regressions?
Re: (Score:2)
> Why is this modded offtopic?
Because it is.
> It's cool and popular to poke fun at Microsoft but heaven forbid you point
> out Linux, WINE, and Ubuntu have regressions?
Why is the fact that other software also has regressions relevant? Do you think that is news to anyone here?
Re:New bugs (Score:5, Funny)
Re: (Score:1)
Hmmm (Score:1, Flamebait)
(I'm coming from a bitter place, I've been stuck going through idiotic publisher files for the last 3 days and I'm certain it was designed by monkeys(or for them))
If only this was easier... (Score:3, Interesting)
This is a great methodology of testing but to be honest I'm not sure it is within the scope of most software firms. While I'm sure we could all drop entirely random data into a parser and see if it fails, to REALLY conduct a test you have to do the same thing broken down by data element in the file format and then for each of those test both realistic and unrealistic test cases.
Then you throw on top of that UI and Web-Page fuzzing and you now have to somehow hook every element on a site and throw in random data which is not realistic with a large rich application.
Re:If only this was easier... (Score:5, Informative)
The whole point of the data is that it's unrealistic. There are a few tools out there for doing this type of testing, or easily modified to do it. I haven't used many testing tools but you could take something like Skipfish [google.com] and add in some fuzz testing pretty easily.
Re:If only this was easier... (Score:5, Insightful)
As with all testing tools, the more of them you use, the better. There are many reasons why you don't want to employ all tests, e.g. lack of knowledge, lack of manpower, lack of money or lack of time. The good thing is that if you can get them automated, then they quickly become affordable.
For an example: I was thinking if it was wise to put findbugs (which works on compiled byte code) next to checkstyle (which works on source code level) in my Java project. Obviously I put them both in; they duplicate bugs but who cares ? I'll just look at checkstyle first and findbugs second. If I can put in a pre-build fuzzing component I probably will.
But fuzzing tools are different than unit tests. Fuzzing can never cover every nook and cranny. They will produce reports that are much less readable, and that cannot be directly tied to particular events (e.g. during regression testing). If anything, they'll put some pressure on developers to put in more unit tests; if the fuzzing tool finds many bugs in a component, it should be a good indicator that even the basic unit tests have not been created.
Re: (Score:3, Interesting)
Unit tests don't find everything either (Score:2)
But fuzzing tools are different than unit tests. Fuzzing can never cover every nook and cranny.
Neither will unit tests [c2.com].
Re: (Score:3, Informative)
A fuzzer isn't really hard to write.
Pick a word-based variant of Dissociated Press [wikipedia.org] that requires similarity a random number of words back/ahead and allows split on special characters (separators) besides whitespaces. Feed it a lot of your actual files. Actually, the amount of data it can produce may be vastly bigger than the amount of data it takes in, because it can jump back and forth in the input files recombining their fragments multiple times.
Of course then you need a test unit that feeds the fuzz to y
Re: (Score:1, Informative)
This is a great methodology of testing but to be honest I'm not sure it is within the scope of most software firms.
Microsoft runs huge (and I mean huge) server farms for all kinds of internal testing - unit tests for rolling builds, automated functional tests for the same, performance regression tests, compatibility tests (what if we run it on Vista without SP1 and with Office 2003 with latest updates installed?..) - you name it, it's there.
But, even with all the servers, it still takes hours for a complete test run.
One would think that this is the case... (Score:3, Interesting)
What you describe is "smart" or "generational" fuzzing, where you have a detailed knowledge of the target that you are fuzzing. The thing is, dumb (mutational) fuzzing is still effective. Very effective. Check out Charlie Miller's CanSecWest presentation - An analysis of fuzzing 4 products with 5 lines of Python
http://securityevaluators.com/files/slides/cmiller_CSW_2010.ppt [securityevaluators.com]
In 3 weeks of (really) dumb fuzzing, 174 unique crashes in PowerPoint were discovered.
Re: (Score:2)
In 3 weeks of (really) dumb fuzzing, 174 unique crashes in PowerPoint were discovered.
The fuzzing was dumb, but the picking of files as basis for the fuzzing was smart. Unfortunately Charlie Miller doesn't present a tool for doing that.
Re: (Score:3, Interesting)
Its only a great model for testing if you've exhausted the extensive list of known bugs that people hit every day under common circumstances.
Finding bugs in the file format is great and all, but fixing the bugs that users actually see every day is far more important and you can reset assured it will be released with a bucket load of very obvious bugs that should have been fixed rather than dicking around throwing random data at it.
I know there are potential security issues to deal with and those are import
Re: (Score:3, Interesting)
I wouldn't knock what they're doing. As we've recently seen with Adobe, exploits in the payload format can be used to manipulate users and even launch code. And remember how we used to be all panicky about Word macro exploitations until the defaults were changed to shut them off? "Good times", indeed.
Consider that Microsoft dominates the market, and that the ".DOC" format is widely accepted across companies. Nowadays .DOC files are readily passed by email filters, web filters, etc. Office workers open
"Botnet?" (Score:1)
Re:"Botnet?" (Score:5, Funny)
FTFA:
Microsoft was able to find such a large number of bugs in Office 2010 by using not only machines in the company's labs, but also under-utilitized or idle PCs throughout the company. The concept isn't new: The Search for Extraterrestrial Intelligence (SETI@home) project may have been the first to popularize the practice, and remains the largest, but it's also been used to crunch numbers in medical research and to find the world's largest prime number.
"We call it a botnet for fuzzing," said Gallagher, referring to what Microsoft has formally dubbed Distributed Fuzzing Framework (DFF). The fuzzing network originated with work by David Conger, a software design engineer on the Access team.
Odd that they would call it that publicly, given the negative connotation of the word. I would have called it "fuzzy clouds grid computing" or something like that.
Re: (Score:2)
Re:"Botnet?" (Score:5, Funny)
"Cluster Fuzzed" would be much better, specially when somebody finds a remote exploit in their cluster code, then Microsoft will be cluster fucked.
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:1)
or Cluster Ftck?,
its only one bit they're flipping...
Re: (Score:2)
If you are flipping only one bit in a "Cluster Fsck", you are missing the point and could be having a lot more fun.
Re: (Score:3, Informative)
Odd that they would call it that publicly, given the negative connotation of the word. I would have called it "fuzzy clouds grid computing" or something like that.
Developers tend to name things that are used internally in a way that is short and more to the point, which is not necessarily something perfect for marketing/PR.
Sometimes these things slip through.
Re: (Score:2)
Well, at least now the marketing drones can advertise MS products as "Botnet Optimized"
Re:"Botnet?" (Score:4, Funny)
Let me explain: Microsoft discovered that all of their desktop computers were zombied with malware, and after wresting control from the botnet C&C, decided to take advantage of this increased ability to remotely administer their computers to run QA tests, on the off chance there might be some need for it.
</joke>
Re:"Botnet?" (Score:5, Funny)
They had to infect the computers with Office 2010.
Re:Speaks to the complexity (Score:5, Insightful)
Your point being? In 10 years since I started using it, I still don't know all the Vi commands and Emacs is so daunting I never even attempted it.
Re: (Score:2)
Nothing really new, you just want your OS to be 'Unix' like when one app or new networked lifestyle cloud is compromised.
You really hope your fav 'application' does not open up your OS and start pumping your personal data out.
Apple seems itoy distracted, Windows seems Win7 PR happy.
Re: (Score:3, Interesting)
1800 down, 10,000,000 to go (Score:2, Troll)
Re: (Score:3, Funny)
The same as I thought. Tip, meet iceberg.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
ME was based on '95/'98 and XP was not based on ME.
Re: (Score:2)
I use Word 2007 at work and it is very buggy.
Undoubtedly true.
If I had my way, I would not use it, even OOo is better.
"Better" in what sense? It's certainly not less buggy... hell, some components of OpenOffice.org (ok, I'll name it: Impress) seem to have never been actually tested.
Re: (Score:2)
> I use Word 2007 at work and it is very buggy. If I had my way, I would not use it, even OOo is better.
I agree Word 2007 is buggy. But OOo is NOT better. It's far buggier.
FWIW, it's not so much the bugs in Word 2007 that annoy me than the way it does formatting and selection of certain stuff. Yes I know you can customize the behaviour but I still find I have to "battle" with it a bit too often.
OpenOffice on the other hand has rather blatant bugs like these:
Launch openoffice writer.
Type three lines of: "
Re: (Score:2)
Any other time you might have been modded funny for a lame joke like that, but - as you - mention, it is 2nd of April, we're still trying to recover from yesterday. No lame jokes today, please.
..they were all sitting in front of the screen.... (Score:1)
Software firm test his software? (Score:2)
*pat in the back* good one Microsoft, now you test your software. What about now to change and respect standards, PLEASE.
Re: (Score:1, Troll)
Re: (Score:1)
Aren't you a piece of work.
Ah, reminds me of the old days (Score:2)
Of Imprise Delphi 4 and Corel WordPerfect 9.
Just kidding. Seems like a good initiative on Microsoft's part.
This is pretty standard in Haskell (Score:2)
But for some reason random data testing is less popular for the other languages I'm familiar with.
Did they fix... (Score:2)
Did they fix that bug where the useful menus get replaced by that horrible ribbon thing?
I know there are downloads to revert to the menus, but can't do that at work.
Re: (Score:2)
Yes, they called it the ALT key, and it was fixed in 2007 too.
wow imagine that (Score:1)
Coming from a programmer's point of view, this is like saying a nurse used an alcohol swab before plunging a needle into your arm, to avoid you contracting anything....i think in today's world this is all pretty standard even for the smaller budget companies, most are unit test driven and have intensive test environments to stress test their apps. I have seen this even in a company as small as 3 programmers. Do they want a medal, seriously....i am thinking of saying something....maybe someone would care to
Re: (Score:2)
Programming 101: TEST YOUR GODDAMNED INPUTS!
Programming 102: If you missed the lesson in Programming 101 where you have to test your inputs, you fail and have to repeat Programming 101. Some people never get out of Programming 102. I've worked with a few of them. "It ain't that pretty at all".
This is clumsy after-the-fact testing at best, just throwing random garbage at the program and hoping to hit a condition. Having said that, I do want to applaud Microsoft for at least, finally, taking some steps
Re: (Score:2)
Computer Science 395 (Software Engineering): Remember how back in Programming 101 we told you how to perform testing on your code? Turns out we grossly oversimplified our discussion of how to go about doing that for pedagogical reasons. While it's nearly trivial to perform the test/debug cycle on code that you wrote for a class project that does one well-defined thing and will only ever be seen by you and your grader/professor, the scope of testing and debugging transforms radically when you attempt to s
Re: (Score:3, Insightful)
Yes, I've taken that class.
I'm not talking about testing, I'm talking about design. If you expect a URL in a field and someone puts executable code in there, you should not be executing the code - you should be rejecting the URL. Data of that nature should not be put in a memory area where an instruction can be sent to run it.
Stack overflows, buffer underruns, and things of that nature are not things that should be caught in testing. They are things that should be prevented in the first place. If your c
Re: (Score:1)
Yet so many people take this class and then some, to learn UML, Unit testing,
variants of stress testing, even how to stand their ground against unwanted elements in the environment, but one thing will never change, if the boss says to do it his way, you can either finally do it his way, or hit the highway, and usually this is were the problem lies.
I have come across a few times in my life that the necessary steps were avoided on purpose and left many with critical bugs in their apps, only because of some hi
Re: (Score:2)
You clearly don't understand what fuzzy testing is.
Fuzzy testing and unit testing are only related in that they are tests.
Unit tests are well defined tests for known conditions.
Fuzzy tests, if done perfectly, would appear to be random and thrown entirely random data at the applicaiton in order to increase the chances of finding out what happens when the app gets something the develop
Re: (Score:1)
Funny i read the word "buffer overflow" and to my knowledge this is
considered part of the unit testing and not the fuzzy.
As for many not doing fuzzy testing, well i never considered the 2 seperate
issues in my unit testing, so you will have to pardon my lack of
empathy for the situation
Re: (Score:3, Interesting)
I don't know of anyone who does regular fuzzy testing. Everyone that matters does unit testing.
Just FYI, Microsoft does fuzz testing in all areas of business, not just Office. The "news" here is really that the Office fuzz testing is done with a cluster of the developers' own computers. (Although it's definitely a good story to get out to all the shitty software houses out there that don't already do fuzz testing.)
When I worked in Xbox game testing back when the Xbox 360 was shiny and new, we had a large pi
that doesn't mean we found 1,800 security issues (Score:3, Insightful)
it's important to note that that doesn't mean we found 1,800 security issues.
"...we have absolutely no idea where THOSE are."
No surprise, with that "format"! (Score:4, Insightful)
Have you even seen the “specification” that MS tried to make a standard. It’s a horribly convoluted mess, that can only be described as an upside-down pyramid of always patching new stuff onto the old framework, while never doing a needed complete re-design. Like Windows ME.
Hey Microsoft! If there are more bugs than features in your file format, maybe you should do a re-design, hm? ;)
Re: (Score:1, Flamebait)
i'm sorry, I must have missed something ...
Where is your example of some successful software product without backwards compatibility?
Contrary to popular belief, a 'complete rewrite' is almost universally a retarded idea, and any developer with more than a couple years experience knows this.
When you're programming to get something done, its a little different than sitting in mommies basement rewriting your python script because you don't actually have anything else to do.
Dumbest possible way to not find errors (Score:3, Insightful)
Remember the very obvious maxim of Dykstra: testing can only tell you there ARE errors, it can't tell you there AREN'T errors.
Randomly poking at data only find you the very dumbest errors. It takes some real thinking and mulling to realize, hey, if a xml field crosses this buffer boundary, and the last 4-byte Unicode code was cached, it's going to get bashed by the next 3-byte escape code. Or 255 bytes of code-page Yen symbol (255) followed by a 254 will lead to sign-extension and access to an address in the kernel trampoline DLL. Those kind of combinatorial errors are not going to be discovered by random poking at the data.
So they're going to (and have) given everybody a false sense of security, when the basic method can do nothing of the sort. it can only fin errors of the most trivial sort. It can't find errors that thousands of unemployed Russian hackers can dream up of testing for, and it can only FIND errors, not tell you there aren't an unlimited number of remaining errors.
Re: (Score:2)
Yeah, well, they find bugs and fix them, regardless of all the other testing they perform. If they are not deprecating other tests that (may not be fully covered) then yes, I can see a problem here. And don't forget, even the dumbest bug can become a vulnerability.
Re: (Score:2)
You just don't get it. Think. Fuzzing can find errors, if they're simple. But people are exponentially better at finding (and exploiting) complex combinatorial errors of the type I described. Fuzzing is never going to find, even in 2^32 passes, those kinds of errors.
It's a losing game, for losers.
Re: (Score:2)
The thing is that fuzz tesing finds a staggering number of bugs in LOTS of components. Way more than the bugs found by people investigating "complex combinatorial errors".
Why spend months investigating in writing the tests to find all of those permutations when Charlie Miller can write a 6 line python script and turn it loose for a couple of weeks and find dozens of exploitable security holes in products from every vendor out there?
It's not a botnet. (Score:3, Insightful)
Wait, I suppose it could be a botnet, if MS's IT department distributed the required software by exploiting security holes in the victim OS instead of just using admin rights to install the new app. Come to think of it, that might be easier
There must be some funny counting here (Score:2)
Even though some of us would easily believe MS office has 1800 security issues that need fixing.(and in my opinion every crash due to malformed input is a security issue)
I find it hard to believe they found 1800 of these by generating random data, what is far more likely is that they recorded 1800(or more) crash events
and after fixing two or three programming errors(problematic hidden assumptions about the input) 1800 of them were not reproduced.
This hardly counts as solving 1800 bugs.
The technique itself i