Netgear Routers DoS UWisc Time Server

numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.

and now...

slashdot has hard coded a link to the UWisc CS server, sending a DoS to them too

oh, and fp.

Obligatory Scooby Doo reference

And we would have gotten away too, if it weren't for those meddling kids!

So who got fired?

Simple mistake that should have easily been found and fixed during the testing phase. I hope whoever let this thing be released without following proper testing procedures got canned.

Yah right. Some hapless low level programmer probably got all the blame for putting test data in there in the first place.

Re:So who got fired?

Right, because when you analyse a security product, you don't look at every single packet to and from it when it is on the bench.

QA isn't just for spell checking.

Re:So who got fired?

Simple mistake, sure. Barely a trickle of wasted bandwidth, hard to even believe it matters...

Bah.

This is one 'simple mistake' by one company that namaged to send a constant "250,000 packets-per-second (and over 150 megabits-per-second)".

Now I know Netgear is a pretty big outfit, but there are LOTS of companies like that out there, and these little mistakes can add up. How much network traffic could be avoided with proper programming?

Also, this kind of makes me think about the useless network activity my XP box (bleh) tries to send out. Multiply that by millions and millions, and you get a number a whole lot bigger than the one above.

Who pays for all that wasted bandwidth?

Re:So who got fired?

Not their first simple mistake though. Ask the people behind dyndns.org what they think of the Netgear RT314's (and other products like the RT311????) implementation of the dyndns.org client. Trust me, they have nothing nice to say.

Windows Time Service

Both Windows 2000 and XP have the "Windows Time Service" which once per day query an NTP server to set the system clock. By default, Windows 2000 does not have an NTP server set, and XP looks to time.windows.com -- every blasted installation of Windows XP phones home every day to set its clock and who-knows-what-else.

One would expect millions of XP boxes phoning home daily would overload a time server. For myself, I've changed the NTP server to a different server (which I will not name) and had somewhat more reliable time syncing.

The commands are net time /setsntp:some.ntp.server and net time /querysntp, or in the Time and Date properties in XP there's the Internet Time tab.

Re:Windows Time Service

Sure it's a lot of traffic for some organizations. But for Microsoft to run time.windows.com, it's a drop in the bucket. Lets see... let's say 100M installations (probably high, since it's only XP, and boxes on a domain sync with the domain server) times 1kB per day (again, probably high) is about 100 GB per day and pretty evenly spread out over a 24 hour day. This amounts to less than two T1's. Not a bad deal, considering that in one "simple" move, a big portion of the wrong PC clocks that are out there are fixed. I wouldn't bother switching NTP servers on my XP workstations... why bother if MS is willing to pick up the dime...

Re:So who got fired?

In the full description, you'll notice that they include the "strings" output from the netgear software, which includes hardcoded IP addresses.
Netgear reported that the non-UW addresses were used for debugging by the developers.

Here's the interesting part: at least two of those are 12.* addresses --- cablemodems with attbi.com. So if you want to know who the developer responsible is, it might be a reasonable guess it's whoever lives at those IP addresses! :-)

Re:So who got fired?

Becasue it's not just a use of a public service, it's a complete abuse of a public service. It'd be like you damming up the colorado river for your own personal use and then telling LA to upgrade their water supply.

This was a big screwup - when an NTP query fails, you don't start retrying every second until it comes back. You don't hardcode a single server address for it. And you don't put this in 700,000 pieces of released hardware.

Poor uWisc

First the NTP flood.

Now the /. effect.

Bad form in general

Highlights how not to code embedded devices

Or any other kind of software for that matter.

Indeed

The C comments in the netgear code were a giveaway, they match those in SCOs code.

"/* Huge Bodge */"

"/* Kludge */"

"/* Magic numbers are cool */"

Re:Indeed

You forgot:

/* Too drunk -- debug later */

I did that to myself once

I did that to myself once. It was a piece of software that went to comp.sources.unix (or something similar) and was default-configured to send error mail to an alias that pointed to me. A patch was released very shortly afterwards.

If they did it to my NTP server...

I'd just send the wrong time back to netgear routers. I bet they wouldn't try that again.

Alas, not true...

The problem is, if one reads the article (nudge, nudge), that 1) at least some of the routers do this with NO operator interface or settability, and 2) some older routers would keep hitting the hardcoded server address even when configured to use some other address. Plus 3) there were some fixes that weren't. The routers in question accept ANY response, even if it isn't an NTP packet! Sending the wrong time would have zero impact. (Why does a home-network router need a clock so badly, anyway? It's not like they do useful remote logging or anything...)

This is a case of ill-designed, badly written, poorly debugged, wretchedly tested code. The article details the testing of a code fix that still didn't fix things properly. On the bright side, Netgear is trying to Do The Right Thing now, and they deserve credit for that.

Hasn't /. learned?

It's not nice to kick someone when they're down.

In other news at the University...

"Quick! Block port 80!"

Our usage graph...You Jerks!

want to see what the usage graph for a slashdotting looks like?

http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cgi ?target=%2Fweb-servers%2Fwww;ranges=d%3Aw;view=Acc ess [wisc.edu]

Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.

Re:Our usage graph...You Jerks!

Oh yeah?! Well, we just /.'d that one, too!

Go ahead, give us another, I dare ya! :)

Re:Our usage graph...You Jerks!

Isn't this a tad bit irresponsible?

don't get me wrong, I love the irony, but your network admins are having enough troubles on a Friday already.

Re:Our usage graph...You Jerks!

I am a network admin, heh.

The load is fine. It's already subsiding. We can handle slashdottings, heh.

Look at the weekly graph, we had 2 this week already!

Just slows down for a while, but doesn't break anything.

Re:Our usage graph...You Jerks!

You really just linked to content that

1. is dynamic and has to be generated every time?
2. is graphic?

ShortSpecialBus, eh? ;-)

I wonder what NetGear's liability is.

Were this a Haxor attack, there would be criminal liability. I'm willing to believe that it was a simple mistake, with no criminal intent, but would NetGear be liable civilly?

Re:I wonder what NetGear's liability is.

Of course there is liability - liability means that 'is Party X responsible for the damage'. Netgear quite clearly was responsible for the damage. Even if they allege negligence on the part of their employee, it hardly matters: Netgear had a duty to assure that the software would not cause material harm to others. This is a classic product liability case, far as I can see.

As for the damages, those are somewhat vague. Sure, maybe they could be made to pay for the bandwidth used. The big hit would probably be punitive damages unrelated to the actual loss.

This would be a fun case and I would encourage them to sue. So many frivolous lawsuits floating around - this one would actually have some merit.

Re:I wonder what NetGear's liability is.

Rather than enrichen the lawyers, Netgear should just donate cash and appropriate equipment to the University.

It would probably be deductable, passing some of the cost on to we taxpayers; but would sit alot better with public perceptions of the company.

Set up a few CS scholarships or funding a chair at the University would help.

They could turn it into a publicity coup and end up paying out less in the long run (and screw the lawyers too). Some (not all) insurance companies have finally discovered that it's usually cheaper to negotiate with the plaintiff right away, avoiding all of the sabre rattling and lopping off a third (or more) of the total probable cost.

Litigation is rarely the best answer.

Re:I wonder what NetGear's liability is.

They probably would be liable. What surprised me was that the article made no mention of the financial impact of the flood... are the guys who run the network so far removed from the guys who pay the bills that they have no idea, or do the universities get such sweet deals on bandwidth that it doesn't matter?

I mean, we're talking 150+ Mbps here, for months on end. That's $15K/mo in bandwidth, assuming they have a really good deal and pay only $100/Mbps/mo.

Re:I wonder what NetGear's liability is.

according to a post on an ntp.org mailing list, it's costing $266 per day.

Re:I wonder what NetGear's liability is.

We are discussing several options with NetGear. I can't really go into them at the moment, but NetGear has been VERY cooperative throughout this whole thing.

Now did NetGear get permission

to hardcode an address into thier systems? Do you need permission? There was a law a few years ago about 'deep-linking' and even linking... isn't getting the time somewhat the same thing?

Re:Now did NetGear get permission

Not in this case- it's a public time server. If it wasn't, they'd be able to just block inbound UDP for the ntp port at the firewall.

Check out the NTPd man pages- I believe this server is a second echelon mirror.

Analysis Tools used in this article..

Wow, that list of Analysis Tools used for tracking this down had a bunch that I was not familiar with.

RRGrapher, FlowScan and Cflow being ones I have never messed with..

Cool.. new tools to play with!

Delicious irony

I love the irony of trying to read an article about a DoS from a site that's experiencing one because of the article. Yummy.

Err why ?

why does a router need to sync time anyways ??
especially a home router....sounds like another port open for someone to hack at for no real gain....

Re:Err why ?

Logging. You want your log files to have the right time. I've used my router log files many times.

Re:Err why ?

Routers tend to log activities such as access, configuration changes, firewall violation detection, etc. and it is often handy to know when that event occured.

Home centric routers do not tend to have their clocks set before shipping as there is no assurance that a battery keeping that clock powered will be doing so ver the entire span of time from manufacture to customer plugging it in. Even if it did the drift involved would give some inaccuracy as well.

There are two correct solutions. One is that Netgear should operate their own time server and hard code that server as a secondary or fallback time server. The primary time server should be aquired from the internet service provider when they get their network ip address via dhcp.

-Rusty

NTP should be responsibility of network server

It is foolish to code code dependencies on servers in firmware. There are two problems that result from this. The first is that specified in the article, the denial of service. The second is the high potential for broken network dependencies if, for example the hardcoded site goes offline or the ip address changes. Technically each site should be running their own ntpd to ease the load on the primary servers. ntp syncronization should not be the job of the router, but instead the job of the network administrator.

Ouch!

I'd hate to be working in Netgear's accounts payable dept. when the bandwidth usage bill arrives.

It's not about just embedded devices...

Highlights how not to code embedded devices

I think this highlights a "how not to code" idea, period. In 1986, when I was taking a BASIC (boo, hiss) course in high school, I learned that values should be expressed as variables even if the coder does not expect them to change. So instead of using (32 feet/second^2), one should instead declare g once, using whatever units are appropriate, and thereafter refer to g instead of a hardcoded value. If g changes, the coder need only update one line.

Note: I am not a programmer/coder/developer in any sense of any of the words, so technical nits should remain unpicked; however, if I am completely out in left field, please feel free to point that out.

Re:It's not about just embedded devices...

Good point, but irrelevent. Even if you declare a global variable, you still have to hardcode its value. The fact that the IP address only showed up 1 time in their string search of the binary would indicate that they did exactly what you said.

So you're not in left field, it's just that the developer who wrote the software apparently did exactly what you said, which was not relevent to the mistake at hand, which was more about the faulty implementation of the NTP service, and the fact that it was hardcoded to a single IP address.

SEGA's online game servers

The (official) reason "Alien Front Online" (a game with the word "Online" in the title!) went offline less than a year after its release is that SEGA developers hard coded the server's IP address, and did not provide any means of changing it. When the company hosting the server went under (gameloft?) it couldn't be moved to a different company since it wouldn't have the same address. Hence, buy a game advertised as "online", never be able to play it online.

It's not a new story, but I think it bears repeating as a showcase of stupidity.

Re:It's not about just embedded devices...

Of course if the gravitational constant changes, we've got bigger problems than updating your high school programming assignments! :-)

Re:It's not about just embedded devices...

that is indeed still the case today. This past spring I was a TA for a freshman programming course, and was instructed to deduct points for those who didnt follow such practices -- pi, hours/day, minutes/hour, etc. On exams, the prof would write "-5 - use of magic numbers."

oh, and we laughed long and hard at the guy who put down:

const int SIXTY = 60;
const int TWENTY_FOUR = 24;

Netgear should bear the cost...

IMHO, since this is blatantly a case of Netgear cocking up their appliance they should not only a)refund any monies spent by the university in this problem and b)send out patches, at their own cost, to all users of affected routers. For heavens sake, so many people don't have anti-virus software installed, don't patch, why would they with a router? They just think "I plug this in to my cable modem, plug my computer in and I dun got thar intarnet workun" why would they know that they need to upgrade the products firmware?