Netgear Routers DoS UWisc Time Server 447
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
So who got fired? (Score:4, Interesting)
Yah right. Some hapless low level programmer probably got all the blame for putting test data in there in the first place.
I wonder what NetGear's liability is. (Score:5, Interesting)
Now did NetGear get permission (Score:4, Interesting)
Analysis Tools used in this article.. (Score:5, Interesting)
RRGrapher, FlowScan and Cflow being ones I have never messed with..
Cool.. new tools to play with!
Re:Err why ? (Score:5, Interesting)
Home centric routers do not tend to have their clocks set before shipping as there is no assurance that a battery keeping that clock powered will be doing so ver the entire span of time from manufacture to customer plugging it in. Even if it did the drift involved would give some inaccuracy as well.
There are two correct solutions. One is that Netgear should operate their own time server and hard code that server as a secondary or fallback time server. The primary time server should be aquired from the internet service provider when they get their network ip address via dhcp.
-Rusty
Re:So who got fired? (Score:3, Interesting)
About once a month a link to my company goes up on the MSN home page (about 3 links down in the top news section). It's like a firehouse and that peaks at an insane 14MBits/second.
Expecting a public service to handle 100 MBits is ridiculous. It was an erroneous mistake by netgear and there should be severe reprecusions.
Our usage graph...You Jerks! (Score:5, Interesting)
http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cg
Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.
It generated costs on the other side too (Score:5, Interesting)
Aparently there are a lot of Netgear users in Germany who are stuck with horrendous bills now. I wonder if Netgear is going to pick those bills up?
Re:I wonder what NetGear's liability is. (Score:5, Interesting)
I mean, we're talking 150+ Mbps here, for months on end. That's $15K/mo in bandwidth, assuming they have a really good deal and pay only $100/Mbps/mo.
Re:Think Strata (Score:3, Interesting)
Just my $0.02
Re:Who pays? (Score:3, Interesting)
Just a suggestion.
Re:So who got fired? (Score:2, Interesting)
when i worked for a ISP over here in
300mbit/s out, on old cisco 7500s....had to get new gig cards for it...the original GEIPs started borking....
heh, everyone needed their vouyerism fix.
They originally thought it was an IT Dept! (Score:4, Interesting)
They thought that maybe somewhere someone had published a net time server in a document or whatever and that an IT department was deploying it on workstations or there was a document floating around telling people to set it up as their time server...
Looks like they finally got to the bottom of it!
Comment removed (Score:3, Interesting)
They're not the only ones (Score:4, Interesting)
I took a Unix course at the University of Colorado in Fall 2001, I think. We had a guest lecture from Evi Nemeth [amazon.com], who is a professor emeritus at CU.
She had done some work on a couple of the DNS root servers, G and H if memory serves. She showed a rate of query graphs for those servers. There was a huge jump in the middle of the graphs that corresponded neatly with the release of Windows 2000.
Turns out Win2000 had it hard-coded to consult the DNS root servers every time it wanted to run a nslookup!
NetGear's Customer Support (Score:4, Interesting)
After a lot of research on the internet, I discovered that this was a well known problem with the MR814, fixed with an update to its firmware. It was strange because I asked the user if he had updated his firmware, which he said he did.
It turns out that the firmware was only released on the Austrilian version of the NetGear website. Downloading and installing that version fixed the users problem.
I sent a polite note to NetGear technical support informing them of this on April 7th. I got back a note on 4/8 saying that it would be forwarded to the appropriate people. On April 17th I sent a more harshly worded note. On April 20th I got back a note saying again that my request would be forwarded to engineering.
I gave up. It wasn't worth it.
Just for fun on May 13th I checked their site again. They had finally updated the software.
This runaround was all to just make a solution to a problem that they had already fixed available. Imagine the hassle trying to get them to actually fix a problem?
Windows Time Service (Score:4, Interesting)
One would expect millions of XP boxes phoning home daily would overload a time server. For myself, I've changed the NTP server to a different server (which I will not name) and had somewhat more reliable time syncing.
The commands are net time
Wow, that solves my little blinkenlites mystery (Score:2, Interesting)
As a side note, one thing that frustrates me about the RP614, although I'm otherwise happy with it, is that even though I can choose an option to allow ping to function, it still wont allow other types icmp traffic through and renders traceroutes out from my workstation useless.
Re:Our usage graph...You Jerks! (Score:2, Interesting)
My hat is certainly off to you folks, it's so refreshing to see somebody facing a serious problem, and actually go about the course of identify and deal with it, with no mention of 'sue them' etc etc. Instead, the problem was identifed, tracked, and eventually the root cause discovered. At that point, they stayed on the high road, and went thru the company to address it, even though initial contacts were 'problematic'. My expectation from most americans after that root cause was discovered, would be for them to get a bidding war going between various law firms as to who could garner the largest settlement, and only then make contact with Netgear, via whichever law firm was bidding highest.
I sympathise with the problem, and I can sure see how something like that slipped thru various pre-release testing cycles (or possibly the lack thereof). The article has definitely made me step back and think about how 'accidental' things like this can slip thru, and possibly consider a new set of release testing parameters to catch such accidents. The /. boys (and girl) are having fun screaming for the head of the folks that caused the problem, but I think there's a valuable lesson in this, made much more valuable by the paths taken towards resolution. It's so refreshing to see non confrontational co-operation in a case like this. That's the kind of spirit that makes the open source world thrive, and it can apply to more than just 'lines of code'.