FSF FTP Site Cracked, Looking for MD5 Sums

landley writes "The Free Software Foundation's FTP site at ftp.gnu.org has been "compromised", and they don't seem to have full backups. They've yanked a bunch of recent packages (and their whole alpha.gnu.org ftp site), and when I asked about it they responded 'Our FTP server was compromised, yes. We are beginning to find good MD5sums for files which have not yet been restored, and they will be available again Real Soon Now. If you can provide MD5sums for any of the files listed in MISSING-FILES, it would be very much appreciated.' " Update the FSF has a statement on the FTP site explaining the matter.

Have a floppy?

[John Paul Jones]

How hard is it to script a backup of MD5 sums to removeable media? Sheesh.

Oh crap

[Anonymous Coward]

GNU is the definitive location of loads of packages. Virtually everyone who uses Linux is potentially affected. It's as if Windows Update were cracked. I don't see anything on the main GNU page yet though...

Wait? I thought Linux was Secure??

[FortKnox]

I'll wait while the "wind0ze suX0rs!" 1337 Hackors try to make this sound insignificant to linux, but can blow up on MS when a virus is released.

Just a healthy reminder that nothing is 100% secure, so no point in pointing fingers (on MS OR linux).

Re:Correct MD5s

[brechmos]

Yeah, but if enough people send in the same MD5 sums for each file, then it "should be" easy to confirm it is correct or not.

Surely, there aren't that many dishonest people, and if there were, then it would be hard for them all to get together and come up with the same MD5.

Of course, if this was a MS site that was

[Anonymous Coward]

'compromised', the /. crowd would be laughing their heads off. Just goes to show that 'open source' or 'free software' isn't 100%, and the "no backups" just goes to show that poor sysadmin skills is not limited to proprietary platforms.

Where's the snide comments from the /. editors?

[Anonymous Coward]

Oh wait, this wasn't a Microsoft site that was cracked and failed to make full backups, it was the Free Software Foundation. Does this mean I can't look forward to michael writing a one liner in the story header showing that this proves that you can't rely on Free Software.

You're Kidding?

[System Control]

The Free Software Foundation's FTP site at ftp.gnu.org has been "compromised", and they don't seem to have full backups.

Unbelievable. And I'm supposed to trust their methods and products with my enterprise?

Re:the $64,000 question:

[gazbo]

Or maybe, JUST FUCKING MAYBE , Linux isn't some sort of magical bug free OS where every buffer is checked, every race condition averted, and every service that runs on it is guaranteed bug free.

Good God. The fact you can post that comment...no. You're just too much of an unthinking hero-worshipping idiot for me to finish. Yes, it was an inside job or a weak password. Anything except a vulnerability. Yes.

That is awful...

[Badanov]

I run a coupla Linux boxes at work and a couple at home, and I swear I don't even take a dump unless I am certain I have backups.

Having just read the above, let me add: Let a thousand jokes be posted!

Re:Correct MD5s

[Henry V .009]

The man of the million email addresses replies: Are they confirming MD5s in person, or over the phone, or by other electronic means? You have yet to master the art of paranoia, grasshopper.

Re:Wait? I thought Linux was Secure??

[JeffTL]

It IS insignificant as far as security is concerned, because it's almost certainly an inside job or a password theft. It'd be insignificant even if it were on an MS-DOS webserver. The only reason this is on /., or is significant in any way, is that GNU is the victim and evidently they haven't been doing proper backups.

Re:the $64,000 question:

[Trigun]

The compromise was probably a weak password or an inside job.

Which is why syslog should be on another secure computer, and dumped to paper in a locked room for high-security sites.

It won't help the recovery, but helps pinpoint the intrusion

apache?

[DreadSpoon]

What does apache, an http server, have to do with their ftp server being cracked?

But no, Apache isn't 100% secure. There is no such 100% server, except one unplugged from the net, encased in titanium, and buried beneath the Pacific seabed.

Re:You're Kidding?

[Lxy]

While your post is somewhat trollish, I have to agree that this is an interesting prediciment for the FSF. To save face, I hope they post a detailed account of how they were cracked, and own up to their mistakes so they can all teach us what not to do. That's the power of openness :-)

Why no PGP signature?

[molo]

Why does the FSF not use a OpenPGP signature on the files and md5sum lists in their archives? Unless the key is kept on the same (compromised) host, then it becomes easy to figure out what files are valid, and what isn't.

BTW, here is my contribution:

> md5sum sed-4.0.7.tar.gz
005738e7f97bd77d95b6907156c8202a sed-4.0.7.tar.gz

-molo

Re:Oh crap

[Anonymous Coward]

It's as if Windows Update were cracked

Actually, Windows Update has been cracked. During Code Red 1, for a period of a couple hours Windows Update was showing "HACKED BY CHINESE WORM".

But I agree, this is just as horrible as that was. Some kind of inquiry as to how this was allowed to happen, and why the hell weren't there backups, and how this can be absolutely prevented in the future, needs to be publicly demonstrated to have happened within the FSF before I will regain the trust I have lost in them. The software the FSF produces is wonderful but their FTP archive is important enough to people of all OSes and natures all around the world that they should have it secured by whatever means necessary, even if that means running OpenBSD or whatever.

-- Super Ugly Ultraman

Re:So apache no invulnerable then...

[ichimunki]

Hmmm. You mention Apache. This is an FTP server. What kind of tool runs an FTP server using web server software? So far as we know (given that there are no details of how the server compromise was carried out), this says nothing about the security of a particular FTP server software, Apache, GNU/Linux, or any other Free Software package.

As is the case with most installations of MS Windows, other operating systems and pretty much any user level software, the security of the system is only as strong as the weakest link: usually that's the user (and the sysadmin falls into that group). Bad passwords, bad security policies, and lax attention to security patching affect every system because every system has users.

Why might Free Software Zealots be laughing when MS products are demonstrated to be insecure? Because people have paid MS billions of dollars for that software. MS has billions of dollars in the bank. You'd think a company with those kinds of resources could hire a few security experts-- or even a few thousand-- and have them really work out the bugs. Free Software, on the other hand, is largely produced as charity, costs little or nothing to obtain, and at least when the code is demonstrably insecure, you (the user) have both the means and the right to fix it. Not so with the expensive binaries you get from Redmond.

Oh, thanks for trolling. I assume this response is exactly what you were hoping for. :)

Re:Correct MD5s

[javatips]

Anyway, the only purpose of the MD5 checksum should be to make sure that the file was transfered properly. And with TCP/IP it would be quite uncommon to get bit flipped while traveling from the server to you (unless their is a "man" in the middle).

Any use of the checksum to ensure that the file has not beeen altered before the transfer is useless. As a person who crack a server will replace the file and it's checksum.

File checksum should always be signed by someone who can be trusted. If that's not the case, they are worthless.

sheesh! Can you fire a volunteer sysadmin?

[digrieze]

In another thread I post a message criticizing incompetant/lazy sysadmins and now this get noticed (after nearly a week).

Could someone pass on to them that CDR/RW drives get put on sale at CompUSA for around $20 on a fairly regular basis? If you rebate the CDrs you can practically get them for free. DO A BACKUP ONCE IN A WHILE, SOMEBODY WILL BREAK LOOSE FOR THAT MUCH IN POCKETCHANGE!

Re:SCO

[Homology]

Hmm odd...one day they speak of taking sco support out of gcc, the next their ftp server gets comprised, interesting.

There are many bad things one may rightly say about SCO, but to suggest that they have anything to do with the compromise is just plain stupid!

One would think...

[Qbertino]

...that the cream of IT people would do regular revolving backups, securing sessions and have a standalone staging enviroment for all their stuff should the connected setup get compromised. Especially files which are distributed into the entire world to run on bazillions of computers once released. That's all a big fat hairy bad-ass no-brainer.
Sorry, gnu.org team, no icecream tonight.

Re:Wait? I thought Linux was Secure??

[freeweed]

No one's ever claimed Linux is 100% secure.

However, the next time a virus is released that takes down 90% of Linux installs, and toasts most of the internet, let me know. Until then, your point isn't exactly valid.

Re:the $64,000 question:

[iii_rjm]

No. The real $64,000 question is why didn't they have reliable backups and a disaster recovery plan

backups

[chef_raekwon]

maybe im missing something here...but don't most people backup their stuff?

i mean, all the posts here are about how insecure FSF is, or OPensource sucks...or windows sucks more...

what about the bloody principle of backing up your own software? let me guess, stallman and his crew has ONE FTP server, and they never back the bloody thing up? they should all be punished for such foolishness. nobody in a corporation would allow this...what would have happened if the harddrive crashed, or the raid crashed hard on that FTP ser4ver? the same thing!!!
asking the world for MD5 sums...

tsk tsk.

oh, and I use OPen Source just about everywhere, except my workstation (manditory windows). I run a chrooted Wu-FTPD, never had too much trouble either...but, we have a tape backup, just incase...

Putting on my troll hat

[batkins]

Oh, gosh. Look at this. A site running Linux was hacked. Gee, that must mean that Linux is fundamentally insecure and that OSS is just no good. After all, everyone knows that FTP access is provided directly by the kernel. Let's everyone use Windows.

Oh, come on, trolls. Give it a rest.

Re:Well that's good and all, but

[Omnifarious]

They were using wu-ftp? That's a worse security hole magnet than sendmail or bind.

Re:This pisses me off more than it should.

[Anonymous Coward]

I can't agree with you - points get added to the faith in humanity tree every time a church gets burned.

This was just a learning experience, like any other. Now the GNU server maintainers will be more cautious and keep backups and up to date software on the servers, etc.

Don't hack GNU, burn a church instead!!

Re:Well that's good and all, but

[Uruk]

I'll bet that 90% (or more) of all break-ins are the result of problems that could have been patched. Yeah, it sucks that this happened to GNU, but they're only human. Last I heard, they only have one system administrator to handle all of their machines, including Savannah [gnu.org]. I can understand that this happens from time to time. GNU has to be a relatively high profile target (such as for disgruntled BSD h4x0rs and so on) so cut them some slack. If you patch 40 machines 99.9% of the time, nobody remembers that, what they remember is that you got cracked on one tiny detail you missed.

At least they yanked the programs until they could verify that they were correct. That really was the only thing they could do. The lesson to take from this is that with computer security and auditing, nothing less than absolute perfection is necessary. And so long as human beings are doing the admin work, absolute perfection just isn't realistic. :)

Re:BSD Ports trees should have them

[Uruk]

Those archives might be decent as an absolute last resort, but I think GNU is looking for the pure source from the maintainer. Similar to Debian packages [debian.org], don't the ports package contain distro-specific modifications and patches?

They may be verified, but I think in some cases the ports packages will be subtly different than the ones GNU is really looking for.

Re:Wait? I thought Linux was Secure??

[lone_marauder]

Depends on how you define secure. If a major windows site gets broken into like this, you don't hear about it. You only hear about Windows problems when a.) Microsoft decides to release a "security fix", or b.) when large corporations and state governments are brought to their knees.

The real story is (and this groks with your point, by the way), how do you trust someone trying to proselytize you with an alien philosophy of computer use when they still run wu-ftpd and don't do backups?

Re:Full backups

[TheLink]

Uh, if the system was compromised a long time ago, then they can't really use 3rd parties to verify the files are correct - coz the 3rd parties have been getting the stuff from their server.

They have to recompile the stuff from the developers who hopefully have had better success in maintaining the integrity of their systems and data.

How Long

[jpmorgan]

How long was the server compromised and serving out possibly trojan-horse software before it was detected?

Also nicely demonstrates the pointlessness (and stupidity) of serving out your MD5sums from the same machine.

Re:Worse than that

[Feyr]

another piece of software from our big friend d.j.bernstein? tell you what, there is no way in hell that thing gets anywhere near my machine. djbdns sucks enough as it is

Re:Wait? I thought Linux was Secure??

[the_othergy]

the next time a virus is released that takes down 90% of Linux installs, and toasts most of the internet, let me know. Until then, your point isn't exactly valid

The next time a virus takes down 90% of Windows installs and toasts most of the internet, let ME know...

Though don't bother if it only toasts about 50% of Windows installs and bring down only a significant portion of the internet. That's becoming too common place.

Re:Well that's good and all, but

[bmj]

While I agree with the premise of the post, this is sort of thing that would get flamed to hell and back if the thread dealt with a Microsoft security breach (case in point, see yesterday's discussion [slashdot.org] about the RPC worm). According to that thread, being overworked, underpaid, or anything else is not an excuse for having an unpatched machine.

Re:Wait? I thought Linux was Secure??

[crandall]

How about next time that happens to windows, in those numbers, you let me know. In the meantime, why don't you be a little more realistic and a little less biased in your numbers?

Re:Wait? I thought Linux was Secure??

[Slime-dogg]

Last time I checked, it was wu_ftpd that had the vulnerability, not Linux. It doesn't matter if you were running it on Cygwin, *BSD, HURD, or Linux. Geesh. Stop calling everything OS Linux, because it isn't.

Why Configuration Management Is Important

[DoctorMabuse]

This is another illustration of why Configuration Management should be beaten into the head of anyone taking Computer Science or Engineering. Many of the security problems I have to fix at customer sites are caused by systems having different versions, no one knowing what version is correct, not keeping backups, etc. This is not rocket science, folks. Buy a damn DVD-RW drive and back stuff up. Keep the checksums. Know what is the latest version.

End of sermon.

Re:Correct MD5s

[Merk]

I did say "clever" didn't I? The only reason we have any luck catching spammers is that the spam they send is pretty obviously spam: obvious keywords, RFC non-compliant headers, lots of HTML, etc.

No excuse? How about the directional flow of time?

[stewby18]

being overworked, underpaid, or anything else is not an excuse for having an unpatched machine

RFTA before critisizing their admin(s):

For the ptrace bug, an root-shell exploit available on 17 March 2003, and a working fix was not available on linux-kernel until the following week. Evidence found on the machine indicates that were cracked during that week.

Is the lack of a patch an excuse not to be patched?

Re:the $64,000 question:

[vadim_t]

They shouldn't be.

If a bug in IIS causes a remote exploit then that's a bug in IIS, and that's it. Now, if there's a bug in the Windows TCP/IP stack, networking components, some kernel call, etc, which causes an exploit then that *is* a bug in Windows.

A bug in wu-ftpd doesn't just affect Linux. It will also affect the other supported platforms: BSD/OS 1.1, and 3.1, FreeBSD 2.2.6, SCO OpenServer 5.x, SCO UnixWare 2.1, Solaris 2.4, 2.5.1 and 2.6, Sun Sparc Platforms, Solaris 2.6, Solaris 2.5.1, SunOS 4.1.4

The only real security vulnerabilities in Linux are the ones that affect only the kernel and Linux specific tools. Everything else is just a vulnerability in some other program.

Re:Corrupted Backups (a.k.a. Why request MD5s?)

[Valdrax]

Backups don't help if you don't know when you were cracked, and they don't help replace files which only exist after the crack if you can't verify that they weren't cracked. A comprehensive backup is not a magical wand that you can just wave to get back everything that could've been damaged by a crack or other catastrophic event. Backups are there to minimize losses. The FSF is doing what is right in this situation; they're not blindly trusting their backups. It's sad to see the ignorance in this thread where people assume that because they're asking for help that they don't even have any backups.

The FSF's admin is just savvy enough to realize what the limits of backups are. They are hoping that other people who may have downloaded these packages before the crack will have what the valid MD5s for them are. On the other hand, this isn't going to be a reliable answer for them either. People who have cracked binaries will report back the cracked sum. They have to look for files for which they get contradictory responses on. This isn't foolproof either thanks to malicious trolls who post false info and potentially cracked files for which no one responds with the correct MD5 to. I wish them good luck, but they are going to be carrying suspect data for a long time.

Read the link off of the Alpha site for more information on what they're doing and why [gnu.org]. (Yes, Virginia, they did have backups.)

Pointless

[isn't my name]

The whole idea of a mirror is that it actually mirrors what is on another site. If they've been rooted since March 2003, then it is somewhat unlikely that the www.mirror.ac.uk is actually going to have files any different than FSF.

Unless of course, the mirror hasn't been updated since sometime in mid-March.

Easy to point out someone else's mistakes

[ThePyro]

It's very easy to point out other people's "mistakes" like this, but I wonder how many people actually take all these various precautions that they're so quick to accuse others of not implementing?

The fools! They forgot to install a firewall!
The fools! They didn't purge all the old user accounts!
The fools! They didn't install the latest security patch! On all the boxes in the office!
The fools! They didn't require 10 character passwords, to be changed every 15 days!
The fools! They didn't update their virus definition files! Within the last 24 hours!
The fools! They didn't make triple-redundant off site backups!
The fools! They didn't have a plan C!
The fools! They don't know where their towel is!

Now granted, if you're being paid the big bucks to think about nothing but information security all day then all of these things should probably cross your mind... but I would be willing to bet that most people who are so quick and proud to show off their shiny, impenetrable suit of dragon scales have a soft vulnerable spot on their bellies.

Re:Wait? I thought Linux was Secure??

[rokzy]

did you miss the "by default" part?

AFAIK, linux generally doesn't leave unsecure ports open by default. what happens if someone reinstalls XP at some point in the future - could MSBlast come back when all the fuss has died down?

I don't read a single second of usenet security groups, let alone 10 hours a week. SuSE YOU takes care of all that for me automatically.

I let YOU do updates automatically because I trust it, whereas I turn off Windows automatic updating because I don't. since when is Media Player 9 and IE6 a "critical" update? plus windows updates often require a restart, and many need to be applied one at a time.

once I did install IE6 to see what it was like and immediately there were another ~10 critical security updates that I required, so that was hardly a step forward for security imo.

Re:Have a floppy?

[Mark Pitman]

They should have been backing up the sums to removable media every night/week or whatever. It's simple, and makes lots of sense.

Since the server was hacked sometime in March, even the backups have the possibility of being compromised. I doubt they keep 5+ months of nightly or even weekly backups sitting around.

Re:the $64,000 question:

[Zebra_X]

Mirrors as a backup methodolgy have at least one fatal flaw which has been clearly exposed by this incident:

A mirror is a random (whenever the mirror was made) point in time back up. There is no assurance that at any given point in time in the future that a mirror is available in a particular point in time in the past. As a result, the answer to the question "do we have a backup" resolves to "maybe". Generally this sort of answer makes people squirm.

In this particular situation the problem is exacerbate by the fact that every release from march until NOW needs to reaquired from it's source becuase after march 2003 - the source repository and it's mirrors can no longer be considered safe.

Indeed, a very difficult situation to be in.

In order to answer Yes to the point in time question one must invest considerable cash in hardware and software to provide such backups.

Re:You're Kidding?

[Pharmboy]

Actually, its the fact that the server was owned back in March and they just now figured it out that bothers ME. One good thing about FSF is they don't dick around once they do find out, it becomes public fast, which is pretty honest.

What I do on my server, and what you do on your server is our own problem, but you would think the primary FTP site for all FSF would have a little better security. Yea, its like how mechanics don't take great care of their own cars, but this really is a black eye, and potential marketing tool, mainly because the server has been 0wned for MONTHS now. Doesn't shake my faith (been with linux 4 years now), but it MIGHT shake someone considering migrating.

"First Linux steals Unix property from SCO, and now their servers were hacked and it took them months to figure it out."

I'm not trolling, I'm wincing... Right or wrong, some people WILL see it this way.

Go easy on 'em...

[chuckw]

Yeesh guys, go easy on these people. They bust their asses every day for us. Their GPL enforcement queue is usually about 50 cases deep. They're on the phones and on capital hill every day educating and lobbying industry groups and politicians. Say what you will about the GPL, you don't even have to like it or agree with it and perhaps you even think RMS is a narrow minded prick (for the most part RMS isn't even involved in the day to day operations at the fsf). They are making life easier for all of us.

Rather than boast about all of the work they do, they quietly work behind the scenes just so you can play Monday morning quarterback. They have one fulltime systems administrator who is *INCREDIBLY* overworked. They are doing everything they can to keep the boat together. Last year they were over $315,000 in the red. Thanks to the FSF associate program and some skillful fundraising they're back in the black.

Want to help? Go get your FSF associate membership [fsf.org]. It's not that expensive and it goes a long way towards helping to protect your freedoms.

Incidentally, this is also old news. They had MD5 sums verified, and the servers were patched up and back online almost two full weeks ago. None of the software was trojaned.

Who am I? Just another hacker who bothered to pay for an associate membership (#1142)...

Re:RTFA: There *are* backups, and they *did* patch

[Mooncaller]

Maybe because they are a non-profit and have limited funds for doing such things. And don't give me "Well they should have been using automated tools". I'm more of a programmer then an Admin, yet even I know enough to get around any automated tool once I have root. The person who did this exploit knew what they were doing and used the exploit to do something rather subtile. I.e. they were carfull not trigger any alarms, so the intrusion was only detectible by a live person. And please note, this incident involed a very busy server accessed by a large number of people. Taking 4 months to find the intrusion is not suprising at all. If you could do better, I suggest you put your time where your typing finger is, and help out the FSF. Otherwise stop whining.

What's really sad about this...

[Simon Brooke]

Is that it was an inside job. Someone trusted with a shell account on the server. Someone who was seen as part of the team, but betrayed it. A pretty shitty thing to do, in my opinion.

The FSF don't say (and probably shouldn't say) whether they know who did it. I hope they do, because if they don't the mistrust which will be engendered will cause a lot of unhappiness, and will distract maintainers from looking after the packages we all use.

If the FSF don't know, I hope the culprit has the guts to own up, and own up quickly.

Re:This pisses me off more than it should.

[noahm]

The thing is, it was a LOCAL exploit. That means the bad guy had an account.

That's by no means a valid assumption. Consider a remote non-root exploit coupled with a local root exploit. Not that uncommon. Figure that at this point, most network services don't run as root, and you can fairly easily envision a situation in which such a series of compromises might have lead to this situation.

noah

Re:This pisses me off more than it should.

[bmajik]

yeah

this is way worse than when someone writes a worm that intentionally targets home windows+broadband users to destroy the functionality of the internet. see, when someone is doing that, they're making a political/religious/security statement that windows sux0rs.

on the other hand, when someone owns the primary distribution server for the worlds most important, relevant free software and the maintainers really have no clue how badly they've been stung over a period of 6 months, well, nobody questions the bullshit about "many eyeballs", and "i just cant trust microsoft/windows update", etc.

instead, someone has committed a MORAL CRIME that has you feeling sick about humanity.

its time for a readjustment folks. more slashdotter has told me that microsoft is "more evil" than saddam hussein. another suggests that microsoft should be held accountable for when MS machines get hacked, or when non-MS machines running MS software get hacked. Another has said that any system that depends on patches for security fixes is garbage, and linux should be used instead.

Wake up and smell reality.

the people that write and use exploits target what is most likely to give them their kicks, whatever that may be. nothing is secure enough against a suitably motivated attacker. the rablidly pro-linux anti-MS community has been making a lot of unsubstantiated statements for a long time, and the fallacies contained therin are starting to come back to haunt them.

Re:You're Kidding?

[NoOneInParticular]

Maybe they did exactly this? The exploit was the ptrace exploit, a local exploit. Maybe an inside job, maybe not. This could however simply mean that it was this limited connected server that was compromised. Maybe all machines inside were compromised, and the ftp server was just one of them. Once such a crack appears inside the citadel, nothing can be trusted anymore.

Heh, in Canada...

[Anonymous Coward]

LOL -- in Canada, we do all of these things from time to time (well, we don't lock our doors, ever... and most of us have our car key hidden behind the plates... as for stores? well, yes, i've walked into a store and left my money on the counter.)

The sad part is that you think a world where such things are possible is *undesirable.*

I would *love* to trust my fellow man, personally :)