Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Security Bug Operating Systems Software Windows

Lessons Learned From Blaster 312

Posted by timothy from the evil-compounds-itself dept.
CowboyRobot writes "It's been nearly a year since Blaster struck, causing hundreds of millions of dollars in fixes and lost revenue. Jim Morrison of Symantec goes step-by-step in looking at how the Blaster worm got out of control so quickly, and what lessons can be learned from that event, by studying how one utility company dealt with it." The story is written as a fun, technothriller narrative; here's an snippet: "The laptops, usually out in the field, were always a hit-and-miss proposition to find on the network and deliver a patch or to have the user take the machine to a field office. That meant that on the 16th they could see a flood of traffic launched against Microsoft. The second phase of Blaster, launching a DoS (denial of service) attack against windowsupdate.com, was imminent."
This discussion has been archived. No new comments can be posted.

Lessons Learned From Blaster More

Lessons Learned From Blaster

Comments Filter:

  • Lesson Learned... (Score:4, Insightful)

    by Terragen ( 727874 ) on Sunday June 20, 2004 @10:16PM (#9481204)
    Don't run windows. :D

    • Re:Lesson Learned... (Score:5, Interesting)

      by Prod_Deity ( 686460 ) <satanic.redneck@ ... com minus distro> on Sunday June 20, 2004 @10:26PM (#9481264)

      First off... I personally agree with that statement.
      Second... I was working a dead end call center job for an ISP when Blaster was running rampant.
      Even though this was a Windows problem (and should have been sent to Microsoft), we trouble shooted it since it did technically stop a customer from getting online.
      I think after that, nearly every Joe 6-pack finally realized that the thing in his "Start Menu" called "Windows Update" was something to use often.

      • Re:Lesson Learned... (Score:5, Insightful)

        by Lshmael ( 603746 ) on Sunday June 20, 2004 @11:24PM (#9481549) Homepage
        I think after that, nearly every Joe 6-pack finally realized that the thing in his "Start Menu" called "Windows Update" was something to use often.
        Which is why the Sasser worm hit so few people? Yes, Blaster caused *some* people to realize it was necessary to run Windows Update, but others only downloaded the Blaster-specific RPC patch in August, causing them to get reinfected again in October and November with newer RPC worms like Gaobot, and again this spring with Sasser.
        • "millions of dollars in fixes and lost revenue"

          But how many millions of dollars saved, when people

          1. stopped playing solitare while their system was hozed.
          2. stopped reading slashdot while their system was hozed.
          3. switched to Linux, saving the company licensing costs for years to come.
          I'd love to see if these millions saved = the millions lost.

      • Re:Lesson Learned... (Score:3, Interesting)

        by j-pimp ( 177072 )
        I think after that, nearly every Joe 6-pack finally realized that the thing in his "Start Menu" called "Windows Update" was something to use often. I have a friend thats an economics major. Very intelligent. In terms of calculas knowledge he probally knows more than anyone here without a master in CS. or a BA in pure math.

        I had to tell him how to hook up his speakers to his computer. He had a simple 3 speaker system. He never owned a non USB keyboard so when he saw the PS/2 looking connector that was s

  • Jim has left the building (Score:5, Funny)

    by Zutroi_Zatatakowsky ( 513851 ) on Sunday June 20, 2004 @10:16PM (#9481207) Homepage Journal
    Eheh, I couldn't help but chuckle when I read "Jim Morrison". Totally destroys the seriousness of the article.

  • VPN's aren't perfect pipes (Score:5, Insightful)

    by LostCluster ( 625375 ) * on Sunday June 20, 2004 @10:23PM (#9481251)
    The main weakness that allowed ingress was that any outside machine with a VPN connection also has a real IP address as well. Those machines, since they were unpatched, were sitting ducks for the virus... and then the trusted nature of the VPN assured that the virus would spread to the inside.

    A basic firewall on the deployed machine to drop any packet not from the VPN could have stopped this before it started...

    • Re:VPN's aren't perfect pipes (Score:5, Interesting)

      by thogard ( 43403 ) on Sunday June 20, 2004 @10:51PM (#9481386) Homepage
      VPNs can be owned too so can "tursted" links to remote controled system. We had a (XP?) box deep inside our network get compromised with a virus that stayed in memory. It got there over a remote control system from another PC that was sometimes hooked to the net. The box deep inside the network then started hunting for other boxes to own, and it found a NT 4 server that could make outbound connections to the net and it set up a nice little email proxy. Lucky for me, my test network isn't as open as it appeared and my freebsd box clampled down on the outbound smtp traffic. A few new rules later (to let the SMTP traffic appear to go out) and the NT box was trying to spam AOL as fast as it could.

      There are some tricky things out there that will take advantage of "internal trust" so my new rule is no PC talks to anything else but its samba, proxy or email server. Windows PC's can't talk to any other Windows PC.

      • Re:VPN's aren't perfect pipes (Score:5, Interesting)

        by HermanAB ( 661181 ) on Sunday June 20, 2004 @11:31PM (#9481588)
        "my new rule is no PC talks to anything else but its samba, proxy or email server"

        Good quality routers, eg. HP2524 can be configured for 'port to port security'. So it is actually very easy to configure a system to prevent PCs from blabbing to each other.

        If the PCs can only see the servers and the servers are all Linux or Mac boxen, then the system is remarkably robust.

    • and then the trusted nature of the VPN assured that the virus would spread to the inside.
      I've never encountered an industry-grade VPN solution that didn't give you the option to specify what ports and IPs the VPN client could connect to. The only trusted nature involved with a VPN is the admin who set it up.

  • I learned from Blaster six months before the fact. (Score:5, Interesting)

    by gfecyk ( 117430 ) on Sunday June 20, 2004 @10:24PM (#9481257) Homepage Journal
    Back when Messenger Service popups happened and started using $80 hardware firewalls that doubled as Internet sharing boxes.

    When Blaster hit I was sitting pretty and so was every client that took my advice.

    *yawn*

    • Re:I learned from Blaster six months before the fa (Score:4, Insightful)

      by LostCluster ( 625375 ) * on Sunday June 20, 2004 @11:00PM (#9481430)
      NAT makes a very good poor man's firewall. Unsolicited packets get dropped... and services you didn't realize you had listening can't be reached.
      • Fascinating. But every person who posts this observation (and at least one person does right off the top of every /. security discussion) forgets that, generally, people who get trashed (apart from network congestion which hits everyone) by this kind of thing barely understand the concept of a "fully patched OS" let alone NAT, firewall, or packets.

        One day, in a galaxy...never mind...One day, internet connections won't even be possible with an exposed PC address. DSL/cable won't even be permitted to conn

        • One day, in a galaxy...never mind...One day, internet connections won't even be possible with an exposed PC address. DSL/cable won't even be permitted to connect directly to a PC without DHCP/NAT interposed between.

          I'm suprised ISPs aren't taking proactive steps and setting up firewalls in front of their DSL/cable/dialup users. Even a Cisco CBAC firewall or simple router access-lists would be better than nothing. I know some of them block NetBIOS ports, but they should really just block anything incomin

  • JIM MORRISON is a senior security consultant with Symantec Security Services, where he manages antivirus security audits and evaluations... etc
    Time to give up his day job and become an author perhaps?

  • Sadly OSX is Next (Score:2, Interesting)

    by artlu ( 265391 )
    I use OSX since I never get virii or worms, but they are coming to the mac soon enough. Although, everyday I am using windows less and less and only for Oracle development (OAF/JDEV) because of my job.

    I guess the only thing to learn from the blaster worm is to switch to OSX. ;)

    GroupShares Inc. [groupshares.com] - A Free Stock Trading Community. Over a 100 active members daily!

    • Re:Sadly OSX is Next (Score:5, Informative)

      by MBCook ( 132727 ) <foobarsoft@foobarsoft.com> on Sunday June 20, 2004 @10:41PM (#9481348) Homepage
      Well, I think that OS X is inherently safer than Windows for various reasons including the Unix core and not being made by Microsoft. That said, if you take the standard precautions, you'll be fine.

      Don't open attachments that you weren't expecting. Get a firewall. A REAL firewall, a HARDWARE firewall. It doesn't have to be expensive, just a little Linksys box or something else designed to act as a router between your PCs and your cable/xDSL modem. Keep your systems patched. Do these things and you'll be just fine.

      But, it's the lowest hanging fruit that get eaten first. As long Windows is popular and there are people running the systems unpatched and doing stupid stuff like executing the newest screensaver they got in an e-mail, Windows will be THE target for viruses. OS X and Linux won't become popular targets for viruses untill they are more common, Microsoft does a better job, and the people who use them are less technical (this applies to Linux more than OS X). This paragraph is my speculation, of course.

      • A hardware firewall? Last I checked all the important work that firewalls do is implemented in software running on the router.

        A HARDWARE firewall? Do such things exist? And no.. don't tell me about the one that NVIDIA makes that claims to be hardware-accelerated, as that's all hype. It's actually just a software firewall. All firewalls are software firewalls. It just may happen that the software might be running on a dedicated system (like a Cisco Router) which does nothing but act like a firewall.
        • When people say "hardware firewall" they don't mean that the entire thing runs on custom-burned chips.

          They mean a device intended to be a firewall first and foremost, where some other bit of software, like the operating system, can't end up with it's ass hanging out because it runs before/beside/around the firewall. That's all.

      • Re:Sadly OSX is Next (Score:4, Insightful)

        by mrchaotica ( 681592 ) on Sunday June 20, 2004 @11:23PM (#9481544)
        Another reason it's safer than Windows is that all the ports are turned off by default.

        I do have a NAT box, but the problem is that it doesn't solve the problem for everyone - I don't use my laptop only at home; I use it on my school's network too... so if you're anything like me, you need firewall software on the individual computers as well.

        Side note - I don't know any good firewall (or antivirus, for that matter) software for OS X; anybody want to suggest some?
    • actually no. the reason that a virus like blaster spread so fast is NO human interaction was required. it was done through an open port exploit in every windows machine (even desktops!). macs come with every service turned off except DHCP and netinfo i believe. i may even be wrong about net info. thus a virus that spreads quickly on the mac platform is going to be a lot more difficult when there aren't really open services running to generate such a propogation.

      trust me, as a mac user (and i am too) you'

  • Contractor Laptop (Score:5, Insightful)

    by eltoyoboyo ( 750015 ) on Sunday June 20, 2004 @10:32PM (#9481296) Journal
    A contractor using the guest offices brought Blaster inside. His laptop infected the security-counter image-storage system, which then found its way to the HR server. That in turn spawned the infections to the HR XP laptops where the patch failed.

    The first thing you learn in ANY security job is that most breaches are from the inside.

    As someone standing right behind the front lines, I will tell you that employees with laptops are the worst. Most end up with administrator access (not that hard to crack if you don't have it). And the fact that they bring their computers home and on the road makes them feel a certain entitlement to install whatever they feel like. Contractors are even worse, since most of the time these laptops ARE their personal PCs. Desktops and servers inside the DMZ are the least likely originators of malware. (Not to say you couldn't surf pr0n on the company mail server as an admin. But then you deserve what you get.)

    Network admins need to lock down MAC addresses and start treating their network like the PBX folks. Nothing gets wired except approved company equipment.
    • Speaking as a contractor, I'm in two minds about this. First off it's enough of a pain having to try and either obtain sufficent access from the sysadmins or else hack it myself just in order to do my job, without having to worry about even more restrictions. My laptop is a hell of a lot more secure than just about any other machine I've seen around my work environment at the moment. I think that the average employee with a laptop is more of a risk than the average contractor.

      On the other hand, I'm probab

  • Trusted Computing is the answer. (Score:5, Interesting)

    by King_of_Prussia ( 741355 ) on Sunday June 20, 2004 @10:32PM (#9481299)
    No, hold back your -1 troll mods, I don't mean that coathanger abortion of an idea that Microsoft has been diddling around with for a while, but a new kind of trust level for computer users. Say everytime a virus has to be removed from a Windows box because a user clicked an attachment a little value increments by one. Once it reaches 10 or so the computer starts throwing up helpful hints like "Don't click on things labelled 'Enlarge your Penii!', they can most likely not deliver on their claims!".

    If it gets really high, eg 50 or so (your average AOL user) automatically turn on the Windows Firewall, and include a flag on every outgoing packet indicating that the user cannot be trusted to operate their computer in a safe fashion. Webmasters can then block traffic from these PC's at their discretion - Problem solved.

    • Re:Trusted Computing is the answer. (Score:5, Interesting)

      by l810c ( 551591 ) * on Sunday June 20, 2004 @11:08PM (#9481466)
      If it gets really high, eg 50 or so (your average AOL user) automatically turn on the Windows Firewall, and include a flag on every outgoing packet indicating that the user cannot be trusted to operate their computer in a safe fashion. Webmasters can then block traffic from these PC's at their discretion - Problem solved.

      How would having Webmasters looking for a 'trusted' flag solve anything? Users don't infect websites. Webmasters from 'bad sites'(porn, warez, etc) would also have a flag telling them that they have a prime target currently browsing their site. Grab the ip and launch other more nefarious processes against the sitting ducks thus furthering the mayhem.

    • Oh, for Chrissakes! A common user should not be able to pick up unsanitary bugs just by using the tools that he was given in the normal way that they were intended to be used.

      If the system allows an ordinary mortal to fsck his computer in the normal line of duty, then the bloody system is broken.

      My users do not pick up viruses and shit - they are behind a Linux firewall and e-mail filter and they use Mozilla for browsing and e-mail. They are running anything from Win98 to WinXP and most of those are tota

    • I don't really think anything like this will be accepted by enough people to become widespread enough to be seriously useful, but for the sake of arguement, let's assume it will, and someone with big money wants to implement it immediately, and solutions can quickly be found to such problems as where to store all the info on users (it can't be on the individual user's machine, obviously, as the worst offenders will never get around to downloading the patch or upgrade needed, and yet the scoring system is go
    • at 75 can we overload the monitor and kill them with glass shrapnel? pretty please.

    • Basic employee education is a good start (Score:4, Insightful)

      by Anonymous Brave Guy ( 457657 ) on Monday June 21, 2004 @07:28AM (#9483048)
      Say everytime a virus has to be removed from a Windows box because a user clicked an attachment a little value increments by one. Once it reaches 10 or so the computer starts throwing up helpful hints ...

      For large corporations, I always quite liked the idea of sending occasional spoofed e-mails with dodgy attachments, similar to your average e-mail virus. If a user opens the attachment, MIS gets notified, and a "three strikes" rule applies.

      The first time, they get a polite warning about their behaviour and how damaging it could be if that had been a real virus, and a friendly reminder to read the corporate IT policy. You're not trying to piss these people off and alienate them, you're trying to educate them.

      The second time, they get another warning, and all non-essential access revoked for a week: no personal mail, no web browsing, nothing. You might mention that this is the sort of thing that viruses try to do to everyone in the company, which is why it's so important not to run attachments carelessly.

      The third time, they get the book thrown at them: automatic formal disciplinary procedures, loss of all personal usage privileges and direct monitoring of their usage by MIS, etc.

      Of course, you need some very senior people on your side to make this work, particularly because managers are often the most incompetent in this respect. However, if your CIO has any clout at all, a quick explanation about the impacts of a real virus on the company and the most likely way to get one should get the CFO and CEO on-side.

      The nice thing about this approach is that it's fair. No-one who's not a liability will be affected. Anyone who's simply naive will be given a friendly reminder of the danger, and how to avoid it. You have to screw up spectacularly several times before really bad stuff happens. And if you really are that stupid, inconsiderate or incompetent, the rest of the organisation doesn't have to suffer the risk you bring to their livelihoods.

  • Getting hit by Blaster is like being.... (Score:2, Funny)

    by Anonymous Coward
    Lost in a Roman
    Wilderness of Pain
    And all the Children
    Are insane!
    • Of our elaborate plans, the end.
      No safety or surprise, the end.

      Was Jim singing about Microsoft or SCO?

      (Visions of slow-mo helicopter fly-bys and napalm exploding.)

  • A little too secure for our own good... (Score:5, Insightful)

    by LostCluster ( 625375 ) * on Sunday June 20, 2004 @10:34PM (#9481310)
    A key paragraph in the story...
    "We had to do some research, but we found out that the way we locked down the users prevented the patch from running properly," lamented one of the policy admins. "What we discovered was that the software restriction policy for the local computer allowed only local computer administrators to select trusted publishers. Because our patch agent ran as a pseudo user, the agent did not have the necessary rights. This was causing the failure. We changed the group policy for the HR systems so that we can patch remotely from now on."

    Sometimes, locking your system too tightly ends up locking the keys in the car. When you really need something to run, it doesn't...
    • Sometimes, locking your system too tightly ends up locking the keys in the car. When you really need something to run, it doesn't...

      However, you have to wonder about the technician, who, after discovering that he has metaphorically locked his keys in his car (cannot patch the system he is being paid to patch), doesn't tell his boss.

      You also have to wonder about the company that would keep said people on the payroll after said incident is discovered.

  • Automatic Updates (Score:3, Interesting)

    by Wedge1212 ( 591767 ) on Sunday June 20, 2004 @10:34PM (#9481311)
    Automatic Updates and Norton...and try to minimize office guests access to the network...

  • Included in TCO? (Score:5, Insightful)

    by Quixote ( 154172 ) on Sunday June 20, 2004 @10:37PM (#9481326) Homepage Journal
    Every time a "Linux -vs- Microsoft" study comes out (for [microsoft-watch.com] example [techtarget.com] , or see this [microsoft.com]), I never see any mention of the costs of these combatting these virii, even though virii have been plaguing MS systems from the DOS days. Why don't these "studies" include the cost of re-installing infected machines, anti-virus software, firewall software, continuous monitoring, etc. ?

    On the one hand, virus writers are aggressively pursued and prosecuted with claimed damages of billions of dollars [com.com]; on the other hand, these losses are not included in the TCO of Windows! What gives?

    • Re:Included in TCO? (Score:3, Interesting)

      by OneSeven ( 680232 )
      you mean something like this ...
      economists and industry analysts believe that the losses in productivity, lost revenue from disabled systems, and the human cost to patch systems and restore those that became nonfunctional are substantial--somewhere between
      $320 million and $500 million or more.
      RTFA
    • The article mentions 320 to 500 Million dollars as the "cost" for the whole episode. It also mentions that Microsoft estimates 16 Million PCs got infected. That would add $20 to $31 to the TCO for everyone that got infected. Those of us that did patch our systems probably spend the same in time: getting the patch, waiting while it is installed, then rebooting,
      all on company time.

  • Today's Tank McNamara says it all (Score:4, Funny)

    by Brian Stretch ( 5304 ) * on Sunday June 20, 2004 @10:37PM (#9481327)
    Heh. [yahoo.com]

  • NCC 1701-D on Blaster (Score:5, Funny)

    by aardwolf204 ( 630780 ) on Sunday June 20, 2004 @10:38PM (#9481332)
    The conference room used for the first discussions had been converted to a war room. The whiteboards were filled with IP addresses gathered by the help desk of systems suspected of being infected and trying to propagate the worm. Another list for all of the nonfunctional pay systems covered the entire portable whiteboard. These systems would have to be patched before they could be used to receive payments again.

    Red Alert! All senior officers to the battle bridge. Prepare for saucer seperation in T minus 3 minutes and counting.

    Picard: Data, can you locate the origin of infection?
    Data: It will take aproximatly 10 minutes to scan each subnet.
    Picard: We don't have that kind of time. Number One, options?
    Riker: Disconnect the OC3 and raise the firewall, leave no ports open.
    Captain: That should buy us some time but we need a better solution than that.
    Diana: I am sensing something captain, it feels as if the SUS server has fallen offline, we may have missed the latest patches
    Data: Her hypothesis could be correct

    We are the Borg, We will assimilate you!

    Captain: Damn, and here I was thinking it was The Boy and his nanites again

    No offense Wil :) [wilwheaton.net]

  • Jim Morrison (Score:5, Funny)

    by Anonymous Coward on Sunday June 20, 2004 @10:42PM (#9481350)
    The Blaster Worm awoke before dawn.
    He put his boots on.

  • Inflexable payment policy comes back to bite... (Score:5, Insightful)

    by LostCluster ( 625375 ) * on Sunday June 20, 2004 @10:48PM (#9481373)
    The utility company lost more than $1 million in revenue that would normally have been generated from the pay systems during the time they were down.

    Wait a second. Blaster didn't directly cut off any customers. How could the virus cost revenue?

    Well, in the case of this story's Mona, it was because her power was cut off despite the fact she had the money to pay her bill through the last-minute pay system. That means a few days that she didn't use power, plus the cost of a needless disconnect that they couldn't charge for.

    If the power company had a brain or heart, they would have not done any disconnects due to non-payment during this time frame. Sure, some deadbeats would get 3 days of free power, but the majority of people who missed their payment deadline would happily pay if just given the chance.

    In short, they could have saved time and money if the bill collectors would have been told to take some time off...
    • That's an important thing that many people don't recognize as necessary. You have to have emergency shutoffs on anything that is automatic!

      In this case, I bet that some automated software shut the power off because the payment wasn't received. If it had been humans, they could easily be told to wait or contact the customer, but a machine will just do what it was programmed to do.

      • Power utilities I am familiar with don't operate like that. The only way you can do a shut off is to go to the customer's house and physically unplug them.

        In this case, I'm betting either (1) it didn't occur to the IT people that customers were being shut off, or (2) the slow-moving bureaucratic nature of the utility took over and by the time anybody decided to do anything, the damage was done.
    • If the power company had a brain or heart, they would have not done any disconnects due to non-payment during this time frame. Sure, some deadbeats would get 3 days of free power, but the majority of people who missed their payment deadline would happily pay if just given the chance.

      Exactly. If they wanted to avoid helping out deadbeats, all they had to do was give the extension only to people who called in to report that they'd been unable to pay. Deadbeats wouldn't bother, because they're probably hop

    • Wait a second. Blaster didn't directly cut off any customers. How could the virus cost revenue?

      Fee's my good man! You think you can just pay your bill at a convience store and not pay an extra fee? You did RTFA right?

  • DCOMbobulator (Score:4, Informative)

    by Kris_J ( 10111 ) * on Sunday June 20, 2004 @10:52PM (#9481390) Homepage Journal
    The first thing I did when Blaster started doing the rounds was put DCOMbobulator [grc.com] in the login script -- bought me more than enough time to get patches in place.
    • All that simple program really does is stop the Windows Service in question and set it to not start again when the system restarts. Gibson's "Shoot the Messenger" works on the same concept, just a different service. So does his "Unplug n' Pray" program.

      In short, these are three rarely-useful-to-home-users services that were turned on by default in XP Home that shouldn't have been. Microsoft has since reversed their policy and now start only essential services on default installs, but Windows XP Home and Pr
  • I thought the lesson was, software monoculture in the global computing industry is opening the door for disaster -- what we need is diversity in platforms and applications.
  • Make sure you have the codes to shut down SkyNet. Oh yeah, lock yourself into a hardend underground base with Claire Daines to reproduce and save the world. Damn worms.

  • 2nd article today, and is just M$ bashing (Score:3, Informative)

    by warren69 ( 187813 ) on Sunday June 20, 2004 @11:15PM (#9481504) Homepage
    OK, so M$ has designed a bad OS. But nobody that I know who has Windows XP and knows how to use it ever got infected with a virus.
    Simple rules:

    1. firewall software (eg. Norton) before connecting
    2. You don't use Outlook/Outlook Express and preferably not MSN
    3. Preferably don't use IE
    4. windowsupdates
    5. update your norton firewall/antivirus

    Don't get me wrong I'm a OS X, and Debian user, but come one, all I can say is if it wasn't for all the dumb people out there who don't get what I call the essentials I would be unemployed.

    Oh crap, I just spilled the beans.

    Warren Peace

  • What about Mona? (Score:3, Interesting)

    by grotgrot ( 451123 ) on Sunday June 20, 2004 @11:15PM (#9481507)
    What I found outrageous is that they disconnected customers. Even though they knew there was a payment issue. Surely the first thing to do would have been to put all disconnections, late fees etc on hold until after you know what the situation is.

    They didn't include the cost of alienating customers or destroying their own brand image in the post mortem. But then again it would be a breath of fresh air to find a utility company that shows compassion or cares about its own image.

  • Blasters effect on Cisco (Score:5, Informative)

    by JRHelgeson ( 576325 ) on Sunday June 20, 2004 @11:23PM (#9481541) Homepage Journal
    Blaster was a worm, and of worms in general I would say that there is little new to be learned from them. I did learn something new with blaster though.

    I was doing some security work for an ISP at the time of blaster. They have a number of Cisco 12000 series GSR routers as well as Foundry Big Iron Switches. For those who are not familiar with the Cisco 12000 series routers, let it be sufficed to say that it is Cisco's biggest, baddest router that stands up to 6 feet tall and comes from the factory with a 4 barrel carburetor, dual testosterone modules and a custom paint job with flames painted on the side (pin stripes are optional). These switches are designed to handle hundreds of gigs of traffic across their backplane and through their interfaces. If the ISP were forewarned that they would be seeing 300 mbps of traffic coming from the MS Blaster worm, they would have said "Bring it on!"

    For those of us that aren't CCIE's, Cisco routers and Layer 3 switches have a function called CEF, or Cisco Express Forwarding. CEF is a technology that by its simplest definition caches routes.

    If a packet from my computer is destined for yahoo.com, it will first hit the DNS server to resolve the host name to its IP address. My computer will then send packets to my ISP with the destination IP of yahoo.com (66.218.71.198). My ISP's router, presuming it's a Cisco router with CEF enabled, will look at its internet BGP tables and determine the optimal route my packet should take on the internet to arrive at that destination. Once the router has processed the route, it caches it so that all future packets coming from my home IP address, destined for yahoo.com will automatically be routed using the cached route. This takes a tremendous load off the router CPU as each packet no longer needs to be processed by the CPU, hence the term "Express Forwarding".

    What the blaster worm did was send out hundreds of thousands of ICMP pings per second. This usually wouldn't be a problem for the router, except for each packet was destined for a unique IP address. What started happening is that each route was looked up, routed, and stored in its cache for future packets - only there weren't any future packets. What happened next was the memory space allocated for caching CEF routes filled up, and once full, the router simply purged its cache so that every packet had to then go to the CPU to be routed. Once this happened, all hell broke loose.

    CPU utilization on the routers jumped to 100%, which should never happen under normal conditions, but this was clearly not a normal condition, and the internet came to a crawl.

    There we were, with a router that should handle hundreds of gigs across the backplane without breaking a sweat being brought to its knees by 100mb of traffic... it was incredible.

    • Re:Blasters effect on Cisco (Score:4, Interesting)

      by Beryllium Sphere(tm) ( 193358 ) on Sunday June 20, 2004 @11:56PM (#9481718) Journal
      Denial of service by cache poisoning!

      I guess it's an example of the kind of attack suggested in http://www.securiteam.com/securityreviews/5AP0V0AA 1W.html

      The general idea is that you attack an application by exploiting differences between its average performance and its worst-case performance.
    • I work on a network where we deployed cisco 3550 layer 3 switches as routers to all our 2000+ sites. Each site only had a 2mb link, and they were all rate limited to ensure the router didn't try to go over that speed.

      Part of the process for implementing each router was to configure, and test each unit before we sent shipped them to site. Bad thing about this was the way that did it left the default route out the WAN interface, and not to the next-hop IP.

      Once blaster hit it took down more routers than I wa

  • Why were they running kiosk systems on XP? (Score:3, Informative)

    by Animats ( 122034 ) on Sunday June 20, 2004 @11:33PM (#9481597) Homepage
    Kiosk systems should be running on something like QNX, not a desktop OS. People who insist on running kiosk systems on Microsoft software should use the Windows XP Embedded toolkit to build a minimal system.

    They're lucky that Blaster was removable by remote control. A more effective virus would lock out any attempt to change system files.

  • 1) On home machines, *all* network accessible services should default off. In most cases, this will mean that remote exploits aren't going to happen - kernel level remote exploits are fairly rare. This means that if I port scan a machine out of the box, I should find 65535 closed TCP ports, and 65535 closed UDP ports.
    2) On buisness workstations, all network accessible services should also default off, but the administrator should be able to provide a configuration to enable services needed for remote management.
    3) Unneeded use of privledged accounts should be actively discouraged. M$ - consider defaulting to popping up "don't do anything stupid" reminders to users running with administrator rights under "end-user" versions of windows. Make it easier to obtain administrator rights when needed without having to log off and log back on. Educate users about the "Run As User" facility.
    4) Operating systems designed for end users should have a facility to lock down the system temporarily while doing emergency maintainance, a "No services" mode if you will, which allows the user to obtain updates without being exposed while doing so.
    5) While it can be argued that automatic updates are themselves a security risk, in practice, lack of updates are a far bigger risk. Anything thats remotely exploitable should be updated frequently and automatically by default.
    6) Reboots are absolutely unacceptable to many users. Microsoft needs to work harder to eliminate unneeded reboots, *including* making changes to the way file locking works so that a reboot isn't needed to replace a file that's in use, or so that the affected subsystems can be stopped and restarted without restarting the entire system.
    7) While blaster didn't use ActiveX, quite a bit of spyware and other ratware does. Fully executable web pages without any kind of sandboxing is a bad idea. Please, Microsoft, *disable* ActiveX out of the box, or require controls to be manually authorized by the administator by adding them to an "Allowed controls" list in the Tools -> Internet Options dialog - NOT as a pop up "Do you want to install and run" box.
    8) Expand user education campaigns. Encourage users to obtain basic computer training, and a basic understanding of computer security.
    9) Provide readily accessable documentation that adresses security concerns. Warning labels get old, but perhaps a big red "STOP: Please review this security information" is appropriate.
    10) Discourage software developers from enabling network-accessible services automatically. (Hopefully the "new" Windows Firewall in SP2 will go a long ways towards making users aware of what they are running, but time will tell.)

  • I can't help but feel sorry for Mona and the uncountable other people in her situation. She had her power disconnected for three days because of this, lost all the food in her refrigerator through no fault of her own, and all she got in return was her reinstatement fees waved. All through the article I kept waiting for somebody to correct their cranial-rectal insertion and put a hold on any disconnects or late fees until the system was back up. Clearly, they know that not everybody can get to their offic
  • It's too bad power companies are monopolies. If I were Mona, I'd want to switch to a different provider for (a) being stupid and (b) cutting me off when it wasn't my fault they were stupid.

    How were they stupid? Lots of ways, including poor security and using Windows for critical systems.

  • What we do (Score:3, Informative)

    by Oriumpor ( 446718 ) on Monday June 21, 2004 @12:26AM (#9481842) Homepage Journal
    Well lets see. Basic measures are necessary for us, since people tend to not follow security policies, and our Tech:PC ratio is so damn high we have had to be pretty ... well creative I guess is the word. Since we haven't the funding, manpower, or infrastructure to deploy anything that would require client reconfiguration 100% we have resorted to the following:
    1. DHCP access listing. (Indexed systems get ips, others don't)
    2. Router Access lists (in non-cisco language port filtering)
    3. Heavily restricted nat firewalls (ipcop+snort)
    4. NAV/Deep Freeze (www.faronics.com... if you can use it, do... no spyware, no viruses, no deliberate destruction of the local system, reboot and it's all fixed.)
    5. Software Update Services (Deepfreeze plays nice if you schedule it right)

    So obviously we use windows... and obviously we have a relatively secure (at least from the current and past virus/worm attacks.)

    About 95-99% of the systems on a campus are frozen. In the case of an outbreak we can shutdown all systems (removing the obviously infected systems from the DHCP access list) and booting the frozen systems back up. This is assuming the virus is 0 day, and it hits us before the SUS updates...

    Still there are horrible gaping holes... for instance, a virus that spreads quickly, before a patch is released, and happens to still be spreading during the SUS thaw could result in a complete infection... but the odds there are pretty slim. And really, it puts us in a better position for 23 hours a day... and on par with most companies for 1 hour a day.
  • ... they seems to keep using Windows.
  • I bought a mac.

  • "Lessons Learned" == SHUDDER (Score:4, Insightful)

    by crucini ( 98210 ) on Monday June 21, 2004 @01:23AM (#9482045)
    Did anyone else read to the end where the employees discuss "lessons learned"? Really encapsulates whats wrong with IT. First, nobody says the obvious, that they shouldn't have used Windows for a dedicated, distriubted application. I guess at least someone must have thought that, and was afraid to speak up. There are hints in the article of an upper manager beating his chest and making the peons shake.

    Second, they vow to not let contractor notebooks on their network without a thorough security vetting. Great, more IT-fascism, and totally impractical. IT needs to support the organization's business objectives, not obstruct them. If you have an attorney who bills $400/hour coming in to meet with the Chief Counsel, and he's got one hour before he has to drive to the airport, who is going to hold him up and scan his notebook? What if you screw it up in the process? There are lots of more practical solutions to this problem, once you accept the basic fact that IT is not an end in itself but just a business enabler.

    Also, did you notice how Windows' overly complicated permission system caused a disaster? The machines were locked down to prevent tampering, which prevented the patch scripts from running. In the end, they had to send people out to each location to fix the machines. I've never had this problem with Unix, because Unix permissions are simple and logical; therefore a sysadmin can easily understand the implications of any permission setting.

    I particularly liked the phrase (quoting from memory) "one of the policy admins". One? Not only do they seem to have a full time employee maintaining these tragic "policies", but they have a team? And still caused a train wreck? Windows is close to being a job-creation program for mediocre technical types.

  • Massive distributed computing (Score:4, Interesting)

    by freeduke ( 786783 ) on Monday June 21, 2004 @01:32AM (#9482084) Journal
    When this worm hit a lot of my friends, at home, I first tried to figure out what it did, beside restarting computers.

    It did nothing to the files, just rebooted the computer, and waited for a precise date to attack Microsoft site. I wanted to participate to this huge distributed computing effort.

    To do this, no patch was required: just open the control panel, clic on ugly icons, and go to the RPC panel. Here, I was surprised to see that the main annoying comportment of this worm was due to a default windows setting!

    The default option on RPC failure is to "restart computer"! So I chose the "restart service" option for every failure and that worked fine! All my friends could now live with this worm and contribute to this distributed computing effort!

    Default options in Windows are users' worst choices: restart the computer on every failure!! The most funny, an stupid, one is the default restart computer on... boot failure!

    To Fix every virus under Windows, put a Knoppix CD in your box and then restart your computer for the last time.

  • Missing the point (Score:3, Interesting)

    by Tom ( 822 ) on Monday June 21, 2004 @01:48AM (#9482144) Homepage Journal
    9.5 for style, 0 for content.

    Patching is dead. In a world where worms can spread faster than patches, patching is by definition a failed paradigm.

    Of course, too much so-called security business depends on the model of adding layer after layer after layer (each layer another product that can be sold) to achieve "security". Whereas security (without quotation marks) is often reached by reducing rather than increasing complexity.

    My bet is 18 months or less before a worm uses some exploit in an anti-virus or anti-worm software to propagate.

    • The worm came months after the patch, how was the worm faster?

      There has already been a worm that disabled a software firewall. It was a 3rd party one, I believe the name was BlackIce.

Slashdot Top Deals

"Hello again, Peabody here..." -- Mister Peabody

Close