Microsoft Bots Effectively DDoSing Perl CPAN Testers 332
at_slashdot writes "The Perl CPAN Testers have been suffering issues accessing their sites, databases and mirrors. According to a posting on the CPAN Testers' blog, the CPAN Testers' server has been being aggressively scanned by '20-30 bots every few seconds' in what they call 'a dedicated denial of service attack'; these bots 'completely ignore the rules specified in robots.txt.'"
From the Heise story linked above: "The bots were identified by their IP addresses, including 65.55.207.x, 65.55.107.x and 65.55.106.x, as coming from Microsoft."
I've seen it before (Score:5, Interesting)
Re:So how do we DDoS Microsoft? (Score:2, Interesting)
Bing? ...But that would only help them to DDoS Bing.
Fixing Bing's poor indexing (Score:2, Interesting)
http://arstechnica.com/microsoft/news/2010/01/microsoft-outlines-plan-to-improve-bings-slow-indexing.ars [arstechnica.com]
Re:Probably just a bug. (Score:5, Interesting)
By the way a philosopher once told that "evil" did not exist. That it was most of the time just a kind of hidden stupidity.
Comment removed (Score:4, Interesting)
Re:MS ineptitude? (Score:2, Interesting)
Re:I've seen it before (Score:4, Interesting)
It's interesting to read this, as I've had some random and somewhat incomprehensible port scans coming from an IP address identified as one of theirs. If you're just an insignificant slob, you can't write to their abuse address, either; you'll get bounced. I simply blocked that particular IP address. Let them worry about who's gotten to them.
Complain to Upstream Providers (Score:4, Interesting)
The CPAN folks could complain to their ISP and have them drop the traffic that's coming in to their boxes.
Most ISP's will work with you to correct DDOS problems.
Re:Probably just a bug. (Score:4, Interesting)
Re:So how do we DDoS Microsoft? (Score:5, Interesting)
As said below, never ascribe to malice that which can be adequately explained by stupidity. (Insert lame joke about MSFT being full of stupidity here).
Yeah, though this particular sort of stupidity has been going on for a long time, and not just at Microsoft (though they seem to be the worst culprit).
I run a couple of sites that, among other things, has links to return the "content" in a list of different formats (GIF, PNG, PS, PDF, ...). Periodically, the servers get bogged down by search sites hitting them many times per second, trying to get every file in every format. The worst cases seem to come from microsoft.com and msn.com, though it happens with other search sites, too. Actually, the first attempts I saw at "deep search" like this came from googlebots around 10 years ago, though they quickly backed off and haven't been a serious problem since then. MS-origin "attacks" of this sort have been happening every few months, for nearly a decade.
I've generally handled them with a couple of techniques. One is to check the logs for successive requests from the same address, and insert sleep() calls with progressively longer sleeps as more messages arrive. The code prefixes the "content" with a comment explaining what's happening, in case a human investigates.
Another technique is to look for series of "give me this in all your output formats" requests, verify that it's a search bot, and add the address to a "banned" list of sites that simply get a message explaining why they aren't getting what they asked for, plus an email address if they want to get in contact. So far nobody at any search site has ever used that address. I did once get a response from a guy who was studying sites with such multi-format data, for a school project, to see how the various output formats compared in size and information content. I took his address off the banned list, and suggested that he add a couple-second delay between requests, and he finished his project a few days later.
I suspect that the googlebot folks may have read my explanation of the delays and added code to spread their requests out over time, since that's what their bots seem to do now. But I never heard from them. They must have gotten complaints (and bans) from lots of web sites when they started doing this, so they probably realized quickly that they should add code to prevent such flooding of sites.
Re:MS ineptitude? (Score:4, Interesting)
I remember reading that the MSNBOT reads the "Robots.txt" file, but cpantesters has a lowercase filename:
http://static.cpantesters.org/robots.txt [cpantesters.org]
http://static.cpantesters.org/Robots.txt [cpantesters.org] doesn't exist, so basically MSNBOT only respects the robots.txt on case insensitive operating systems.
Re:So how do we DDoS Microsoft? (Score:5, Interesting)
Every once in a while, I still see sites that don't serve up unless you include "www." in the address - but it's like I said - a dufus.
Looks like someone hasn't read RFC 1178 and enjoys breaking interoperability.
Your method also breaks email by redelegating MX records one sub domain above where the control should be and MX's point to, thus breaks delegation of sub domains.
Re:MS ineptitude? (Score:4, Interesting)
The standard clearly specifies lower case. However, if you are correct there's a simple way to send bingbots one way and all other bots another: create Robots.txt and robots.txt with different contents.
download locally first and then test indexing (Score:1, Interesting)
Bing should have used Wget first to download the articles to a local hard drive, and also to add a 2 to 3 second wait. Let it run over the weekend. Then test the search indexing algorithms on the local HTML files. They were probably performing indexing tests. I know they have smart people working for them, so it probably involved a contractor who didn't think about performance issues.
Re:So how do we DDoS Microsoft? (Score:3, Interesting)
Instead we have Slashtroglodytes screaming about conspiracies by MSFT.
Just for the record, since you're commenting under a thread I started, I do not believe that there was a conspiracy to attack CPAN. I think there is a conspiracy to continue accidentally attacking CPAN. The information provided ought to be more than sufficient to figure out what is going on. Remember, any time two people work to screw a third out of something, it's a conspiracy by definition.