Slashdot Log In
Using Statistics to Cause Spammers Pain
Posted by
michael
on Fri Feb 28, 2003 06:59 PM
from the slow-poison dept.
from the slow-poison dept.
mlamb writes "Statistical mail classifiers like PopFile save time on the part of their users, but don't do anything to actively combat spam. I just published an article that suggests a way to use classifier output against a spammer while they're connected to your SMTP server, and I'm launching a project called TarProxy to implement it."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Anti-Spam software (Score:4, Insightful)
Re:Anti-Spam software (Score:5, Interesting)
I've been using bogofilter for a while now as a pass-through tagging mechanism. I filter on the client side based on the tag information. This sounds a lot like what you are doing.
The only thing close to a false positive I've gotten was having to dumpster dive into my spam folder to retrieve an amazon order confirmation.
Bayesian filtering really works, but you have to train the filter correctly and with as large a corpus as possible.
Parent
Interesting idea (Score:5, Interesting)
Re:Interesting idea (Score:3, Insightful)
Re:Interesting idea (Score:5, Insightful)
Two things here. First, this article wasn't about preventing spammers from using your SMTP server as a relay, but in slowing down the reception of mail at the end-point SMTP server. This will ripple up the chain to hurt the spammers by slowing down the relays they use. Second, it doesn't matter whether I get 10 spam emails or 10,000. One of the goals of TarProxy is to be ubiquitous. I may only receive 10 spammy emails, but my running instance of TarProxy will determine that those are of sufficient spamminess to throttle bandwidth to each of those connections. At the same time, you're doing the same on your SMTP server, and Joe over there is, and so is Susie, and so on. If everybody (defined as "a large number of smtp servers", and not necessarily "everybody") is running such a service, the spammers will be hurt. You're right that a single individual using this won't make much difference, but that didn't seem to be the goal of the article.
Parent
Re:Interesting idea (Score:4, Funny)
"This would transform the server into a sort of dynamic tarpit, in which the spamminess of the incoming message affects the viscosity of the tar"
Its quotes like this why I love Open Source projects. =)
Parent
Re:Interesting idea (Score:4, Interesting)
My current spammer is sending 99-recipient spam, and sometimes he sends as many as 10 in one session. All the spam stays on my system - he is totaly wasting his time.
I've seen a lot of recent single-recipient spam, I've seen single spam messages with recipient counts in the thousands. Much relay spam reaches my relay spam honeypot from open proxies. I think thee was some in January that came direct from the spammer.
This (running a relay spam honeypot) is easy for many Windows users - try it yourself: http://jackpot.uk.net/
Linux users can make Jackpot work (it's in Java) or they could jimmy sendmail (or some other MTA) to be a honeypot - do it on a second Ip with no other email function. The MTA I use is so old it doesn't know EHLO. You don't need sophisticated tools to beat the spammers.
Parent
Re:Interesting idea (Score:3, Insightful)
Nope - you missed what the article was saying. The mailserver being used by the spammer would be slowed down.
"Throttling" is when you send ICMP choke packets to a sender, which in turn tells the connection to stop sending so many packets. It's generally used to tell a sender that you cannot handle the number of messages it is sending.
Now, what this article proposes is that mailservers use software that statistically analyzes messages, and based upon the likelihood of a message being spam, may send choke packets to the sender. You essentially spam the smammer with choke packets until the spammer's SMTP connection slows to a crawl.
At this point, the spammer can either deal with sending *maybe* a small handful of emails at a time, or give up on spamming. For those businesses that make money off spamming, this would destroy their ability to make any decent money.
Quite sad.. (Score:3, Insightful)
Exactly ... (Score:5, Funny)
Exactly [pbs.org] how it should be.
Perhaps public floggings and other corperal punishment as well.
However I have to wonder if all spammers are really sane ... I just got an email about chicks who crave small penis's and those who crave big penis's and then emails about penis enlargement and viagra online purchases, it just seems weird that there is so much concern for my penis. Perhaps we should just imprison them on an island as they might find tar and feathering a bit kinky and enjoy it.
Uh... (Score:4, Insightful)
But, but, but, why would they be connected and sending spam through your server? Unless you run an open relay. And you don't run an open relay, do you? Do you?!
Re:Uh... (Score:5, Insightful)
So, you don't run an open relay. You're not going to slow down the spammer directly, but you will slow down all the connections that come from that open relay to your mail server. For a particularly abused open relay, that could lead to such problems that the admin of that open relay will finally get a clue and look in to configuring their server properly.
Hence, a cascading effect that will eventually harm the spammers. Admins of open relays that get a clue will tighten their servers, thus depriving the spammers of one more relay they can abuse.
Parent
This is too complicated (Score:4, Informative)
Slowing down the mail server (Score:5, Funny)
Well, that's one way to do it.
Re:Slowing down the mail server (Score:4, Funny)
Please don't do that when I'm drinking hot coffee. . . .
Parent
OpenBSD Spam Blocking Engine (Score:5, Interesting)
The hurt-back part of the project is not new. Theo de Raadt is working on just that [deadly.org], in connection with an IP number list (much faster, so suitable for busy servers):
Very simply, this hangs the full list of ~12,000 spam-sending IP/mask entries listed at www.spews.org off a pf(4) rdr-anchor (which is only entered for port 25). When connections from these spammers arrive they are redirected to a daemon which minimally fakes the SMTP protocol with very low overhead -- for multiple connections at the same time -- and then the message is left on the sender's queue by providing a 550 return code.
The theory here is that most spam still comes in via open relays, and the only way we are going to convince them to clean up their act is to waste _their_ disk space, their time, and their network bandwidth more than they waste ours. For those spammers who drop messages when they received a 550, well, we have not wasted any further time or network bandwidth, and even in that situation I think some of the might remove an address if they receive a 550.
Increase prior probabilities of spams if suspectIP (Score:3, Insightful)
Using a list of the spam-sending IP's and Bayesian methods, one could assign a high prior probability of a message being Spam. The affect would be to slow down the connection on less evidence if its from a suspect IP address, and to require more evidence if its from an IP address that you trust. Thus you preferentially slow-down suspect computers, and allow your friends to get away with more spam-like messages before tarring them.
Re:OpenBSD Spam Blocking Engine (Score:4, Insightful)
Another big advantage of going by IP numbers is simply this: I have an IMAP mail account at my university that I use, but I have some external Email addresses as well, which are configured to forward their mail to the university server. Now, if the university's server will add tar based on the message content, I suppose the external mail provider will not be too happy about being slowed down. I would suppose there are quite a number of users simply forwarding mails from one account to another. Maybe (depending on how many people actually use automatic forwarding capabilities) "innocent" servers could be slowed down due to forwarding mail to a "dynamic tarpit", and maybe there are some providers that would not be too happy about such stuff... on the other hand, tarpitting by IP lists seems a little more practical then. But I suppose only practice will show which works best.
Parent
550 is wrong. Use 450 instead! (Score:4, Interesting)
Parent
OpenBSD's spamd (Score:5, Informative)
First post... (Score:4, Funny)
Parallel (Score:5, Insightful)
Re:Parallel (Score:3)
(Yeah, I know, then the spammer can just connect from other IP addresses... but you have to admit it would be a pain for him to have to do that)
Re:Parallel (Score:4, Insightful)
I gave up on this approach. While there was a satisfaction in looking at my message log and seeing all the spam I had hung up on, spammers would often just keep trying to deliver. Some of the worst software would try a second or two after I hung up on them so they literally pounded my system. It didn't cause any problems except for a little bit of bandwidth, but it certainly didn't seem to phase the spammer.
The fact is, there's not much you can technically do to hurt the spammer. Even if everyone implements this there's no reason why spam software can't open up hundreds of tasks running in parallel and simply be patient when necessary. It could even make spam worse because spam software might evolve to where it DOES send spam out in parallel hundreds at a time by default [forgive me if this is already the case, I have no idea what capabilities spam software has].
The fact is, the only way to make spam go away is to make the response rate go down. This approach gives you, as the admin, a certain satisfaction but it really won't reduce spam--it'll just make spam software more advanced. The only way to make the response rate go down is make sure the spam doesn't get to the user, and that's filtering. Feel free to implement this system, but once the thrill of sticking it to some spammers gets old you'll be back to where you were--with the filters doing the real work.
Parent
Re:Parallel (Score:4, Insightful)
The spammer already IS tying up 30-50% of the resources on the mail server; if you throttle the bastards back they'll end up using less. What would you prefer a few hundred megs of spam on your hard disk or a few kilobytes of spam that trickled in over a few days till they eventually kill the run. This way you save both bandwidth AND disk space.
Either they use their own server, in which case they're easy to spot. Or they use someone else's- in which case chances are, it isn't engineered for lots of parallel connections.
This scheme may actually work.
Parent
Misunderstandings (Score:4, Interesting)
Instead, this is meant to be run on the incoming SMTP server, the one that receives the mail. It will only hurt the spammer if he's trying to send a bunch of spam to your domain, but every server running this can help.
Daniel Hartmeier / OpenBSD / pf (Score:3, Informative)
but its usually from an open relay... (Score:5, Insightful)
sure, the open relay deserves some pain. but you're naieve if you think that most spammers send from their OWN systems!
I have qmail running on my mail hub and I reject mail at the time of connect simply based on the receiver they're trying to send to. when they handshake (part of the HELO exchange) I detect the user they're trying to send to, and since I only have a handful of valid users, its easy to know if they're dictionarying me or not. once I know that, I immediately cut them off, AND add an ipfw (I run freebsd) rule to block all traffic from that IP to my port 25. not only do they NOT get to send any DATA to me, but they're for now on (until it ages out, automatically) forbidden from even connecting to my box. I know that's harsh but I can be that selective since its mostly just me on my mailhub.
but I don't think for a second that even tarpitting that source IP is punishing the spammer. they've most likely broken into (or found) an open relay and they're routing thru them. they don't even see the 'address not reachable' error due to my firewalling them.
Re:but its usually from an open relay... (Score:3, Informative)
but the open relay is enabling the spammer. The people operating the open relay should really fix their server.
the tarpits (Score:5, Informative)
TarProxy [martiansoftware.com]
ChuckMail [msg.net]
OpenBSD's spamd [slashdot.org] (tarball [fmonkey.net])
Google Search Results [google.com]
Just like teergrubing... (Score:5, Informative)
This sounds a lot like Marc Merlin's SA-Exim patch [merlins.org] for Exim v4. It runs SpamAssassin at SMTP time. If it scores over a certain threshold, then the connection is stalled.
remove the open relays (Score:5, Interesting)
What we do is have a small app that plugs into eudora, outlook, evolution, kmail etc. Whenever you get a spam, you click a button, it scans the header, finds the smtp server that sent the spam and then sends them 1 email informing them of the fact that they are sending spam (of course you need a way of getting the sysadmin's email address).
If enough people did this then the bad relays would be swamped with emails informing them of the spam they've been relaying, and they might close their relay. And non-open relays that just allow spammers to spam might think about being less friendly to spammers.
What do people think, is it lame?
What the hey (Score:5, Insightful)
Here's what I propose: setup a large number of bogus email accounts. Broadcast them everywhere, and let them be honey-pots for spam. The point is, since you NEVER use this account for anything but dropping in spammable places, anything you receive on it *must* be spam. As soon as you get a connection from a mail server to one of these addresses, you *know* it's an open relay, and you put it in your database -- automatically, with no interaction required.
Step 2: You also do a "fingerprint" on the spam you get in your honeypot (you know the routine - what's the length, average use of the word "dildo", etc) so that you can identify this particular spam "copy" by the message -- NOT the header. This allows you to automatically filter out spam messages. If the spammers want to adapt, they have to rewrite their copy. As long as your signature algorithm is fairly lose -- that is, not a true hash algorithm -- they should have to do a total rewrite if they don't want to be detected. You can then filter these at the relays. Thus, once again, you raise the cost for them to do their spam. Since you are filtering by actual known-spam content -- that is, you're doing this like they do virus signatures -- you should get virtually no false positives.
And, anybody whose friends who are emailing them about penis enlargement doesn't really deserve email anyway.
Anyway, there's step 1 and 2. To summarize:
Just another stage in the arms race (Score:5, Insightful)
Err, I don't think so. This just requires spammers to use more simultaneous connections to overcome the slowdown; it doesn't really increase their network requirements much, only their host CPU requirements. 20,000 simultaneous TCP connections from one process is quite possible with
I also don't understand the rationale behind processing the message incrementally. Why not just do your processing before sending back the final 2xx response to the DATA command? Most spam software does not hang up right after sending the final "\r\n.\r\n" from what I've heard from people who run tarpits.
How about this instead: when you are confident you are receiving spam, you stop reading from the socket entirely, and send perhaps 10MB of data back on the other side of the connection. (If the other endpoint isn't reading, and consequently you can only send one window worth of data, then do something to get your TCP stack to generate a lot of useless ACKs, or send your trash back one octet at a time and push between them, or something.) The intent being that sending spam to a large number of MTAs configured in this manner rapidly just becomes a way to DDOS *yourself*. Probably this is too disruptive for most sites to want to bother implementing, though
I don't know exactly what the profit margin for spammers is like, but I'm not convinced a small multiplier in network costs is going to matter. Anyway, a lot of these "countermeasures" are mostly going to hurt maintainers of open relays, but if that means they actually fix them, I suppose that is almost as good.
Easy to defeat, just use dynamic spamming software (Score:5, Interesting)
I believe linux, for example, allows up to 8192 open sockets, and I think this can be changes with a sysctl command, and most definitely could be with a few changes to kernel headers.
Sure, it would take a machine with decent memory, but that's not too hard to find.
Training (Score:3, Interesting)
If you do an static word frequency list, spammers will pass around it (check in POPfile site for the latest spammers tricks), if is dinamic, then the users of your system must train it for a while (someone must tell that some message is spam or not, reading it). You must have another way to access your server for the training thing, and then another possible point of vulnerability.
And more than this, as it depend on the user, you should not use a common word frequency list, you should have one for each user, and check if the message is spam against destination word base.
At best, it will work for the users that care to train this server, for the other users that don't want to waste their time spam will be coming at the same speed as before. At worst, you'll be using a common list for all, and maybe slow down receptions of mailing lists or things like that, and people in your server could be unsubscribed from some of them.
Is a good idea, but there are some things that should be implemented with care, and should work only for the users that care about it, the others should not be slowed down because you can put obstacles in the reception of normal mail.
"Stations of the Cross" Relays attacking relays. (Score:5, Interesting)
I have several domain names that appear on many of the "million address" CDs and other popular spam lists, but which longer any legitimate recipients/users.
We are also working on obtaining access to true "realtime" RBL lists of currently abused open relay servers. Assistance would be appreciated.
The core of "stations of the cross" is a custom DNS server. This server is authoritative for these oft-spammed domains, and each time a request is made for an MX record, it returns (with a short TTL) a unique randomly generated list of MXes, each address on the list being a known open relay.
So when a spammer or relay first goes to deliver a message, the system will select an open relay off the list of MXes, and hands off the message to that host. Being an open relay, the host accepts the message for my domain, then goes to do a DNS lookup for the MX record. The relay receives a (different) list of other open relays...
Usually, you can get a message to traverse a dozen or more open relays (most sendmail systems default to a maximum "hop count" of 25), after which the message will bounce.
Since the only traffic my server has to deal with is DNS queries and responses, this is very low-overhead for me, but depending on the size of the spammail, very high overhead for the open relay servers.
Re:"Stations of the Cross" Relays attacking relay (Score:4, Interesting)
Write something that listens on port 25. When it receives a connection, connect back to the calling host on port 25. If the connection attempt succeeds, copy characters back and forth. Anything they send to you, you send to their port 25, and vice-versa.
If it's a true open relay, it will gladly accept the mail over and over again. I had a few mail servers looping THOUSANDS of times through me since they didn't check Received: headers. I also realize that it would be trivial to *ahem* "break" the Received: line such that it wouldn't increment the counter.
Granted, that sucks down bandwidth, so back to the point - proving that this is an open relay. What you do is stick a magic header in the message as it heads back to them. If you receive that header back from a host, it's something you've already looped, and they're an open relay.
Now you know they're an open relay, so you can add them to your MX lists. You can also then avoid letting them run through your looper, since it won't provide any more data.
The beauty of this plan is that you're only giving them what they pushed upon you first. If they leave you alone, you leave them alone. It's a nice implementation of a concept I wish more people would honor.
Parent
Predictable failure? (Score:4, Interesting)
- Free: It's no good unless it's everywhere... or at least in lots of places. TarProxy is Open Source Software released under a BSD-style license and available on SourceForge (see project page for details).
- Platform Independent: TarProxy is written in Java, so it runs on Linux, Windows, Solaris, OS X, and any other operating system with a Java Virtual Machine available.
contradict one another, and therefore directly suggest incipient failure. Any program you want widely deployed had better not depend on having some buggy JVM installed.(Arguably that is the reason that Freenet has been a practical failure. Every time I have tried to use it, it has got stuck in an infinite loop, or consumed all my swap space, or crashed. I blame buggy JVMs.)
If you want software to be widely and successfully deployed, it should (must!) resemble the software that already has been. Almost all such code (99%+) has been in C or in C++. Are there any Free Software programs written in Java successfully deployed outside of Java development shops? (Rhetorical question; the answer is "not enough to matter".)
If you want portability to Unixes, to w32, and to Macosix, you already get that with Gcc and autoconf.
If it's in Java, I certainly won't run it as a daemon.
Argh! (Score:4, Interesting)
Personally, the spam solution I like the best is to have procmail+formail or some other tool sitting on your mail server and making unknown senders go through a confirmation step. It doesn't work for everyone (for instance, people expecting email replies to résumés! NAGI...), but if it works for you it tends to work very well. It inconveniences everyone else, but hey, everyone else is not me. I can whitelist all the people I truly care about.
Either that or we should throw out SMTP, email RFCs, sendmail, etc. and build a spam-free system from the ground up. Yeah, right.
Bouncing? (Score:3, Interesting)
The idea is the following: spam gets through whatever filter you might have, but you still want to reject it, and given that some spammers MIGHT be trimming their lists based on bounces, you forge a bounce message from the spam.
Does anyone know if this is possible with, eg, RMail or VM (or something else) running under Emacs?
Why use the statistics? Throttle it all! (Score:4, Insightful)
For example, if your machine only receives a small amount of email per day, why not throttle them to take 10-20 minutes of connect time overall? If you only get two emails per day (one real and one spam), getting them 10 minutes later probably won't bother you too much, but could cost the spammer or his relay-helpers a 5 minute duration on a connection.
I receive about a hundred emails per day from a number of sources, and adding six to sixty seconds of delay per email wouldn't cause me any grief. But if everyone throttled their email, it might cause someone using their '250 million Valid! Tested! Opt-In!' email lists to have to upgrade their machine to half a million connections to process it in an hour.
I don't see that differential throttling has any benefit over a contant throttling rate. For a big site, the differentiation between spam and not-spam would probably cost you any load advantage you earned in slowing the spam, and for a small system, the delay would not be noticable.
Of course, big senders like AOL, prodigy, and yahoo, might have to upgrade...
Not a new idea. (Score:4, Informative)
Read about a method to get SpamAssassin to execute at SMTP time in exim (I'm about to impliment this on my own mailserver) and read [iks-jena.de] about teergrubing which is basically the same idea as a tarpit.
Unlike the original post, Marc seems to have a stable working version of this right now.
That said, this is probably the most realistic method of causing spammers pain that we have right now, short of changing the way mail works in a fundamental manner.
I'll definately be implimenting teergrubing/tarpitting. I might even impliment it on the multi-user hosting system that I helped to build. It probably wouldn't scale too well on a busy site though
I'm going back to splinter cell.
Re:Nice idea (Score:5, Insightful)
If they forward mail from anyone who sends them mail, then they are an open relay, and again, they deserve what they get for leaving an open relay up.
Parent
Re:Nice idea (Score:3, Interesting)
Re:Nice idea (Score:3, Insightful)
Matt's formmail script isn't really intended for use as a mail server, but on a webserver (ok, so I'm arguing semantics here
As for your questions, the idea is *not* to set up false open relays per se, but to set up servers that tie up the 'upstream' mail server. Tarpitting is a pretty cool idea if you ask me - it hurts no one but the spammer, if implemented properly. As for blacklisting/whitelisting servers, sure, let the spammers. Note that if enough people tarpitted, eventually spam wouldn't get *anywhere* - spammers could spam each other all they want, but none of it would ever get delivered.
Unfortunately the critical mass for this to really work is very, very large.
Re:Nice idea (Score:5, Interesting)
For other spam, through open proxies or open relays, you are not hurting the spammer to tarpit. If the spammer is working through open proxies and if you got enough tarpits going then you could hurt them, but until there's enough tarpits there is still zero (0.000) percent pain to the spammer. Some open proxes are slow with one or two tarpits, the others are fast enough to keep the spammer's server fully busy. He only cares if he's running his server flat out. Delays at one or more open proxies mean little.
Right now I'm trapping spam on a relay spam honeypot. It comes to the honeypot from open proxies - theer's nothig I can learn about the spammer by learning about the proxies. It comes (usually) as 99-recipient spam messages. This particular spammer uses imbedded comments in his spam to evade Bayesian filters. Makes no difference to me - I see it is spam. I have no valid email to filter out - everything is spam. That's one of the beauties o a honeypot - the spammer does yor filtering for you.
Somewhere over 20,000 recipients so far, since Wednesday. Here's a tiny sample, showing the URL's he advertises and the random comments he uses to defeat filters:
[a href="http://www.directmailorderbrides.com/?oc=23
[a href="http://www.flati.com/silagra/"]L[!--WPVizB-
(I replaced agle brackets with square brackets - tou'll have to imagine them restored.)
I have no filter, no smarts of any kind. The honeypot is a mail server with the output queue stopped. I got the spammer to start sendng spam by delivering to him three of his relay test messages - he'd sent so many I decided to see who he was, what spam I'd get if I did deliver.
I'm trying various ways to hurt the spammer but I've not yet delivered enough hurt - he's still operating. Other spammers have succumed more readily - this guy is better at hiding himself.
Note, by the way, that he puts no comments in the URL - if you filter on those (or remove comments before filtering - that would be easy) the spam instantly is revealed. One guy simply rejects any email message with three repeated comments in a line (this spam is laced with the comments throughout, not just in the http lines.) The spammer's clever way of obscuring the spam is useful in identifying the spam - no points for Spammy.
Windows users with a permanent connection can step into running a relay spam honeypot very easily: they can run Jackpot: http://jackpot.uk.net/
There is at least one open proxy honeypot out there: Google in news.admin.net-abuse.email for it. These can be very wicked - create your own for even more fun. Or create your own open relay honeypot - see if you can make it even more wicked.
(Oversize reply packets from an open proxy honeypot might have a very interesting efffect.)
Parent
Re:Nice idea (Score:5, Interesting)
Unfortunately, there is a problem. Before TarProxy there was another thing, called a "teergrube" or "tarpit." What it did was slow down the connection (with things like ICMP source-quench and psychotically small TCP window sizes) so that it acted like a spam speed bump. In the meanwhile, it didn't actually forward any of the spam anyhow. Why didn't this technology become more widespread? I'm glad you asked! Because it was trivial for the guys who develop spammer software to recognize these systems, have their software detect such behavior, and cease using them within less than a minute. And that's what will happen with a TarProxy, alas.
Parent
Re:Nice idea (Score:5, Insightful)
what if the spammer sends a message to a (good) SMTP server which haven't got the system, and the SMTP server in turn tries to deliver the "spammail" to the right SMTP server, won't that hurt the good SMTP server, who just tries to do it's job?
The situation you're describing is called relaying.
If you start with the assumption that spammers are evil, then the logical conclusion is that there is no such thing as a "good" SMTP server that would relay mail on a spammer's behalf. Servers that do are either in collusion with the spammer, or are mis-configured to allow anonymous relaying. A server that willingly acts in collusion with evil is, by definition, evil. The level of stupidity necessary to allow your sever to act as an open relay also, by definition, precludes being considered a "good" server.
So the short answer to your query is that it's a non-issue. A truly good server will, by definition, never relay spam!
Parent
Re:If you want to stop spam... (Score:5, Funny)
Great idea! Parse out the URLs, plug 'em into some boilerplate, and automatically submit it as a story to Slashdot! They'll never try THAT again!
Parent