Netgear Routers DoS UWisc Time Server 447
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
NTP should be responsibility of network server (Score:5, Informative)
Re:Now did NetGear get permission (Score:5, Informative)
Check out the NTPd man pages- I believe this server is a second echelon mirror.
Think Strata (Score:5, Informative)
Unfortunately, the code droids seem to think that there's something magical about being at Stratum 2 instead of Stratum 3 or Stratum 4; also, they seem perfectly willing to take advantage of a nonprofit consortium (the owners/operators of public Strat 1 clocks) instead of spending the $500 or so on hardware to service their own customers, who presumably paid them for something.
Anyone else remember the Good Old Days when it was considered polite to ask first before using someone else's clock?
[Truechiming since 1987...]
(Geography)Re:And then, on friday august 22 2003.. (Score:2, Informative)
Mentioned on ntp.org mailing list a while ago.. (Score:5, Informative)
David L. Mills wrote on 2003-06-26 10:55:
> Guys,
>
> I find myself on the review team for an incident taking place at U Wisconsin/Madison. Apparently, the Netgear folks have manufactured some 700,000 routers with embedded SNTP clients configured to use the public U Wisconsin NTP server. The server address is unchangeable and the client cannot be disabled. If that isn't bad enough, if the client gets no replies, it starts sending packets at one-second intervals until forever and without backoff.
>
> The U Wisconsin folks determined some 285,000 different IP addresses are now sending between 300 and 700 packets per second requiring between 150 and 400 megabits per second. Apparently, the principal eason for this flux is misconfiguration of the firewall component of the router. This is costing them $266 per day.
>
> The Netgear folks were slow to respond until U Wisconsin folks emailed the entire senior management and others known to be U Wisconsin alum. Netgear says they have no way to recall those routers and no way to insure the products are updated from the web site. The products cost between $20 and $40 depending on rebate.
>
> U Wisconsin have considered several ways to deflect the tide, the most promising may be noting the source port 23457 unique to these products and tossing them at the doorstep. The products do not use DNS and are not configurable. Another way considered is to configure a subnet visible to BGP and convince the ISPs to punch holes in the routing fabric. Send money.
>
> I never thought it could get as bad as that. My reasoned recommendation was to fire up the lawyers and sue the bastards for costs and punitive damages and to injoin the company from selling any products until proved safe. There is apparently some standards group that allegedly reviews and certifies new products for Internet use. The Netgear products were all certified, which surely says nothing about the standards group.
>
> Include me in any replies; I am not on any ntp.org list.
>
> Dave
Re:And then, on friday august 22 2003.. (Score:2, Informative)
Re:So who got fired? (Score:2, Informative)
Not the only offender (Score:2, Informative)
When investigating time (mis)keeping on the D-Link DI614+, I found exactly the same thing there. Walking the strings of the firmware reveals a hardcoded list ntp servers and from observation it looks like they walk down the list, primary ntp servers first, to get the time.
The D-Link firmware is cobbled together from quite a few different libraries. It maybe the code exists in a library both systems use or the systems are re-badged from a common source.
How many others then???
Re:I wonder what NetGear's liability is. (Score:5, Informative)
Re:If they did it to my NTP server... (Score:3, Informative)
An impropperly formatted response, like "2/30/2003", would probably get people's attention.
From RFC 958 [faqs.org]: NTP timestamps are represented as a 64-bit fixed-point number, in seconds relative to 0000 UT on 1 January 1900.
OT: answering side question in parent. (Score:3, Informative)
Alas, not true... (Score:5, Informative)
This is a case of ill-designed, badly written, poorly debugged, wretchedly tested code. The article details the testing of a code fix that still didn't fix things properly. On the bright side, Netgear is trying to Do The Right Thing now, and they deserve credit for that.
Re:Our usage graph...You Jerks! (Score:5, Informative)
The load is fine. It's already subsiding. We can handle slashdottings, heh.
Look at the weekly graph, we had 2 this week already!
Just slows down for a while, but doesn't break anything.
Re:And then, on friday august 22 2003.. (Score:2, Informative)
Strata ain't the issue (Score:4, Informative)
Actually, Netgear was using a stratum 2 time server [gvsu.edu], namely ntp1.cs.wisc.edu [wisc.edu].
As for spending $500 on hardware to service their own customers, as the wisconsin people can tell you, it is costing them a little more than that. It's isn't just the hardware, it's the pipe to which it's attached.
I agree that Netgear should have been the ones to provide a time server if they were going to hard-code one. On the other hand, what if they weren't the ones who wrote the code? Maybe they just bought a "router kit" from some small company, slapped a "Netgear" logo on it, and shipped it out? That small company probably wouldn't know what NTP server NetGear provides. They may also have lots of other customers who each would need their own time server. Obviously though, the answer is not to hard-code the value.
As for the Good Old Days when it was considered polite to ask, the policy [os2site.com] for UWisc's time server was "open access", not "open access; please send a message to notify". So... they didn't ask to be notified. Now I'm sure they're going to change that policy, and I'm also sure they would have wanted to know if their site was being set as the default on tens of thousands of routers.
Routers are standalone devices that are meant to operate without user input, so it doesn't make sense to require the user to manually configure the NTP server. On the other hand, there's currently no good way of providing a default NTP server, unless you provide it yourself. For commercial devices like a router, providing it yourself is reasonable. The bandwidth cost of providing a time server should be offset by the profits they make on the hardware. I suppose the other option is to provide a one-time service that will provide a random NTP server. Each time you hard-reset the router, and out of the box, it would check that service and then know what NTP server it should use.
Re:Netgear has fix (Score:2, Informative)
http://kbserver.netgear.com/kb_web_files/n101176.
This is par for the course for Netgear. (Score:5, Informative)
Story:
I used to work/volunteer for DynDNS.org. The Netgear firmware client for DynDNS tried to update regularly (I believe every 5 minutes) whether or not the IP address had actually changed AND whether or not it got a response. Once enough of these got out into the market, this became quite a problem for DynDNS, especially with users complaining that we "blocked" their hostnames updated with the Netgear client when their router advertised specifically that it worked with our service.
I believe after a year or so of nagging the Netgear people, they finally released a firmware update that actually fixed the problem.
I love statements of this nature (Score:3, Informative)
sPh
Re:SEGA's online game servers (Score:3, Informative)
Well, yeah, with Dreamcast games like Alien Front Online [hotgames.com], or with more or less any game since the birth of the console, the read-only nature of the media is a problem. It's hard to issue a patch for a game cartridge or CD, and recalls would be expensive.
The idea a multiplayer game that only has one server to connect to should stir strong feelings of hatred and scorn in any sensible geek. The sheer idiocy of coding in an IP instead of a domain name should be obvious.
Hewlett Packard did the same thing (Score:3, Informative)
What had happened was the ingenious engineers at HP decided to hardcode some poor soul's URL into their new Internet-enabled keyboards -- you know, the ones with the hotkeys. The point was that every so often (which ended-up being very often) the keyboards would send this ping-esque packet to the URL and if it received a response it would know it's still connected to the Internet.
Unfortunately, there were some lapses in the plan. Number one, HP thought this was a good idea, but I guess not good enough of an idea to have them ping their own site. Secondly, with this keyboard a part of new HP systems, these systems turned into DDoS machines on this poor guy's domain. The tricky part was the domain they were sent to wasn't any other company's site, just some apparently random URL the HP team picked; that guy must of thought he was the luckiest person with all the traffic he received, and all the bandwidth he was charged. We are a small college, and even we saw a hit on our network traffic from these keyboards, imagine what he was seeing at the focal point!
The point is, sometimes lack of common sense can have drastic consequences.
Coda: We tracked the IPs of our computer systems pinging the site and told those who owned them to disable the Internet keyboard.
Download corrected firmware (Score:3, Informative)
Re:Could this happen with GPS? (Score:2, Informative)
Re:So who got fired? (Score:3, Informative)
Re:I wonder what NetGear's liability is. (Score:5, Informative)
Re:Our usage graph...You Jerks! (Score:3, Informative)
Think mrtg
Correct so far.
Its dynamic in the sense that it is generated at regular intervals, but it is static in the sense the webserver is serving pre-generated content.
So, yes, the page is static.... most of the time.
Not necessarily true. I run Cricket on my own network, and the images are generated by grapher.cgi; the HTML doesn't point to static images that get replaced on the server at regular intervals. Although grapher.cgi will return cached copies if one exists, you still have to pay the "CGI penalty" of launching a Perl program every single time you view an image. If they're using mod_perl, the overhead may not be so bad. It's still nonzero, though, and I'd hate to have all of Slashdot joyously reloading a Perl CGI on my already-overloaded server.
To Netgears Credit...Okay maybe not.. (Score:5, Informative)
This specific issues was raised back in may... I can say within that same week they had already started testing firmware to fix the issue. The issue comes with the huge break between Netgear engineers and Netgear support. Umm often times the supports reps do not know of the release of the product until like 2 days or 3 days after its already hit the market. On top of that there is very little communication between the two on firmware and whats the latest version. Its been only in the past couple weeks have they really started to communicate.
Along with that Netgear did not have a device testing program until i would say about 3-4 months ago, before that it was just people there who had the time to test products... woudl test them. I know being one of those who has and still does test there products, that the communication is not very stable and that sometimes issues like these get short-cutted for other major issues such as security and hardware stability.
I am also sure anyone in the hardware market understands the rush that sometimes comes with products; in netgear this is not different. I can this was an issue that was not expected and was fixed as soon as it was reported. It should have never gone out as is and the products should have been tested throughly in the consumer enviorment. But, to Netgear's credit the company does sell pretty good products and there customer support although you may not always be able to get your answer to the issue or may not be able to sometimes understand the reps any and all issues do esclate to people who can fix them. If you issues are not getting fixed at that point the president of the company does read your mail and does forward them to the Head of the customer support. I can say that issues like these will become less of a problem now that Netgear has started a beta program and engineers are required to speak to support engineers on a regualr basis
Re:Windows Time Service (Score:2, Informative)
And if you want to have even more fun with XP's ntp servers, check out the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\DateTime\Servers.
Add a string value, give it a number, and set its data to the ntp server of your choice. Ta-da!
Standard disclaimer applies: back up the registy beforehand. I am not responsible if your computer crashes or blows up or something.
Re:Windows Time Service (Score:2, Informative)
Not sure if you overlooked that, or if you were just pointing out the (useful to know) Registry location for adding default time servers. Probably the second one, but I just wanted to put that out there.