DSPAM v3.2 Released 157
Nuclear Elephant writes "After four months of development DSPAM v3.2 has been released, bringing many new enhancements and filtering technologies. These include distributed computing support, implementation of Bill Yerazunis' Sparse Binary Polynomial Hashing algorithm (from CRM114), and v1.2 of Bayesian Noise Reduction. Other enhancements include SQLite support and many significant performance enhancements for PostgreSQL. DSPAM's official release is next week, but you can download the preview release now. Users of the project have also contributed towards creating a new logo for this release."
second post? (Score:1, Interesting)
I see that most of these bayesian filtering programs mention that they can be used with SpamAssassin. Is it usually best to run both for DoublePlusGood(TM) spam catching?
Re:second post? (Score:5, Interesting)
I would have thought that running 2 bayesian filters would cause more trouble than good. The first filter would be ok as it would be trained like usual.
The second filter would probably have problems because it would only see a small subset of all your mail as the first filter would have removed most of the spam. The second filter's sample would therefore be skewed and it would have far less data to accurately classify spam.
Just my thoughts on the subject anyway...
Re:second post? (Score:2)
The funny part is if the second filter includes headers as part of its bayesian filtering, the second filter could become biased based on spamassassin's results
even spamassain's heuristics are bayesian scoring (Score:1)
Re:second post? (Score:2)
what does surprise me s that sometimes obvious spams seem to get through, ie every now and then a 419 comes through and I'd have thought it would be well trained on those by now. nevertheless, it works a lot better for me than spamassassin did, and it requires less (or easier) mainten
Spamex (Score:2)
Re:second post? (Score:2)
Re:second post? (Score:3, Interesting)
Re:second post? (Score:5, Informative)
It's your server and hopefully you'll never have to suffer the 'collateral damage' of living near a spammer (network neighbourhood wise). It has happened to me a couple of times. The first time I actually spent time sending my reply from my gmail account, and told the guy about it. The second time I didn't even bother.
Netblock blacklisting is a really poor solution. In some cases a single spammer causes a /24 and then a /16 to be blocked. It doesn't make sense to me. OTOH, I discovered some time ago that blocking Windows boxes works wonderfully, and it's extremely easy to do with OpenBSD's pf :-)
Btw, do you understand that changing ISP may not be an option?
Re:second post? (Score:1)
How can the pf detect a windows box ?
Re:second post? (Score:1)
I use CBL and RBLDNS -- neither block entire netblocks, the IPs in them are static IPs which either run open proxies or fail other checks. Very, Very effective.
Complaints come first. (Score:3, Interesting)
It is the only solution when the ISP will do nothing to stop the spammer on their network.
In some cases a single spammer causes a
That is rather difficult without the ISP's assistance (or them repeatedly ignoring the complaints).
Btw, do you understand that changing ISP may not be an option?
Sometimes that is true. In which case, you should get on the phone and make sure that your ISP understands that they have customer
Re:second post? (Score:3, Informative)
Re:second post? (Score:2)
Re:second post? (Score:4, Interesting)
Re:second post? (Score:1)
Care to provide an example of a spam that somebody might want to read ?
Re:second post? (Score:2)
Re:second post? (Score:2)
Re:second post? (Score:2)
Re:second post? (Score:2)
Re:second post? (Score:2)
Re:second post? (Score:1)
DSpam with qmail / vpopmail (Score:4, Interesting)
I am using D-Spam on a qmail/vpopmail server and I find that its great in terms of accuracy. Most of my users have never had a false positive and many havent seen a spam after a couple of weeks of training.
The problem that I have with DSpam is the integration side. Im not sure how it goes with other mail systems but integrating it with vpopmail was a major pain. It seems easy, you just put the command in the dotfiles, but in practice getting it to work was quite a trial. Even now it doesnt integrate properly with the web administration, etc despite some scripting and minor code changes.
Because of this Ive been thinking of switching to Spam Assassin simply because of its integration with qmail-scanner. Has anyone else had similar problems or been in a similar situation and found a good solution?
Re:DSpam with qmail / vpopmail (Score:3, Interesting)
I'll admit I don't really understand your post.
All these new spam removal programs are all very well and good but from an end user's point of view, all I would like to know is:
How long am I going to have to put up with emails like this?
Re:DSpam with qmail / vpopmail (Score:2, Informative)
http://clamav.sourceforge.net/
Re:DSpam with qmail / vpopmail (Score:5, Informative)
This is a legit message from someones mail system. You are receiving this because someone has been infected with a virus. Their computer is sending messages from your email address, and some of these messages are going to non-existant mail addresses. Because they are spoofing your mail address in the From: you are receiving all the bounces.
So technically, this isnt spam or junk mail. Its someones email system doing what its supposed to, returning 'your' email because the sender didnt exist.
Unfortunately, probably not much you can do about this without blocking all such legit system messages.
Re:DSpam with qmail / vpopmail (Score:3, Insightful)
Re:DSpam with qmail / vpopmail (Score:2)
Re:DSpam with qmail / vpopmail (Score:2)
Re:DSpam with qmail / vpopmail (Score:1)
Re:DSpam with qmail / vpopmail (Score:3, Interesting)
Here's a crazy idea: if you crypto-sign all messages you send, it should be possible to check the signature in bounced messages and filter any unsigned bounced messages.
Informative, yes... (Score:3, Insightful)
I have to disagree with you on whether it's spam, however. Just making up statistics here, but I'd guesstimate that the sender address of >99,99% (probably even more) of all virus emails is forged and probably points at an innocent third part. That means that the message from the virus scanner is completely and utterly worthless to the reciptient (i.e. the "sender" of the virus email). That makes it "junk" or "spam" in my book.
You're right that there isn't much you can do
Re:Informative, yes... (Score:2)
Just clearing something up. The message that he was receiving wasnt a reply from a virus scanner, it was a bounce. I totally agree with you that virus scanners that reply to addresses that are 'sending' viruses are a total waste of time as the sender addresses are always forged.
In this case though, the receiving server is not replying to tell him that he has sent a virus, its telling him that hes sent an email to a nonexistant user. Obviously a message like this can be very useful if you have mistyped an a
Re:Informative, yes... (Score:2)
You are not guaranteed to get a delivery error message emailed back to you for each and every delivery error anyways, so you may as well not ever expect one.
--jeff++
Re:Informative, yes... (Score:2)
A good point. However, from what I understand, this message is generated by the MTA and not the virus scrubber. So exactly what are you suggesting?
Maybe MTAs shouldn't alert the sender that the address they used doesn't exist (user no longer has an account, mistyped address, etc.)? That works for this situation.
Re:Informative, yes... (Score:2)
Your client can finger the email address automagicly before sending and have a nice warning if it doesnt think it exists, and then the MTA can finger the sender address to make sure its valid. This way obvious spoofed spam gets dropped. Of course people could still spoof valid addresses, but it would prevent some spam.
Re:DSpam with qmail / vpopmail (Score:1)
Which many of us do routinely. So why bother sending faked "virus warning" messages at all, if the only effect is to worry some people with clean computers, and get the rest of us to block anything with "postmaster" in the header of the email.
Re:DSpam with qmail / vpopmail (Score:2)
Technically, if they are bouncing messages back to me when I didn't send the original message, it is unsolicited email.
Any mail that wasn't delivered because it was a virus shouldn't bounce - everyone *knows* that viruses spoof addresses. If it isn't delivered because a filter decided it was spam, it shouldn't bounce, IMO, as spam usually forges addre
Wrong. You can filter them (Score:2)
Re:DSpam with qmail / vpopmail (Score:2)
Re:DSpam with qmail / vpopmail (Score:2)
Robert
Re:DSpam with qmail / vpopmail (Score:1)
COMPLETE COPY OF NETSKY VIRUS
Make the mail admin install the qmail-send.mimeheaders [orfika.net] patch -- it causes bounces to bounce back only the headers of email with MIME attachments. As google provides, my qmail patchlist is quite long, actually [dasbistro.com]. :-)
I'm moving over to Postfix these days -- it seems to do everything qmail does but without the need to recompile every time I want a change.
Re:DSpam with qmail / vpopmail (Score:2)
Perl script to handle spam bounces...
Re:DSpam with qmail / vpopmail (Score:2)
How long am I going to have to put up with emails like this?
If you install amavisd-new and clamav, you won't have to put up with it at all. amavisd-new is a generic mail proxy that calls both spamassassin for spam filtering, and clamav for virus scanning. If you really want you can get it to call dspam as well. It also can use a huge number of other virus scanners if you prefer them. I now get zero viruses using clamav and zero false positives.
A little Harsh! (Score:2, Insightful)
MOD PARENT UP!!! (Score:2)
Re:DSpam with qmail / vpopmail (Score:1, Informative)
Is DSPAM... (Score:3, Funny)
Both inferior... (Score:1)
OT: It's a Joke!! (Score:2)
Yes, the above counts as "humor" too. :) Have a nice day.
3.2? (Score:1, Redundant)
dave
DSPAM version 3.2 has _NOT_ been released (Score:3, Informative)
Here's what it shows.
ONLY the 3.2 Preview Release 1 is currently out!
Re:DSPAM version 3.2 has _NOT_ been released (Score:2, Informative)
What about false positives. (Score:5, Insightful)
I'm sick of spam filters braging about their overall error rate. All of them do OK at getting rid of the bulk of spams and saving the bulk of time.
The real important differentating factor is how many false positives they mistakenly accuse of being spam.
The consequenses of a spam message getting through are minimal - under a seconds of time, on average, to skip them.
The consequenses of a non-spam getting blocked can be huge - loss of a customer - a mom not knowing her kid is in trouble.
I wish the spam filters focused entirely on reporting how few false positives they produce.
Re:What about false positives. (Score:5, Funny)
Dear Mom,
I hope this email finds you well. All is fine here, out in your garage. As you know, I love working on my cars. I'm currently replacing the engine block in my '76 Trans Am. Well, wouldn't you know it, but just moments ago, this 550 lb engine block fell on my legs and I cannot stand up, and in fact, am probably bleeding to death. Luckily, I have my cell phone handy and so am able to send you this email - the marvels of technology!! Anyway, I know you only check your email about twice weekly, but when you do, please send help.
Your loving son,
Dexter
Re:What about false positives. (Score:2, Interesting)
the big question for me is: how many mails do I need to check for false detection? and here is the dspam issue: it
Re:What about false positives. (Score:3, Insightful)
Re:What about false positives. (Score:2)
(a) Because it would whitelist any emails sent from a virus-infected computer that that person had previously sent an email to.
(b) Because people like that change their address all the time. "Hi! I'm on AOL now -- see my new address?"
(a+b) Because people like that never sign their emails, nor do they use different email-addresses for personal, public, shopping, and mailing-lists.
I think his point was that you need
Re:What about false positives. (Score:3, Insightful)
There's a story about a CEO that used to sweep his pile of memos into the waste bin every morning
Re:What about false positives. (Score:3, Informative)
DSPAM has currently given my 0 false positives.
The clue with dspam is to start with a clean database for each user and let them start to 'sort out their spam'. For imap it's stupidly simple. Everyone has two folders "spam" and "notspam", where you can drag&drop an email to the right folder. A script picks up any emails in each folder every hour and do
Re:What about false positives. (Score:2)
Re:What about false positives. (Score:2)
If you look at how spamassassin works, for example, it's a lot of little things. You can actually send back what each of those little things were, by sending back SA's report.
Re:What about false positives. (Score:2)
People are getting used to there being mail filters in the system and know that email is not perfectly reliable. This can be due to mechanical reasons - a mail filter discarding the message, or due to human reasons - the message got lost in a pile 10,000 spams, since the user doesn't have a spam filter, or it may be an executive with email overload who gets 2000 legitimate messages every day.
Therefore, if someone sends an important messa
Filters? (Score:3, Funny)
Re:How is the weather in India these days? (Score:2)
did they fix the problems? (Score:4, Interesting)
- the database did grow huge. when my single user server with 128 mb had to use a 512 mb spam token database, performance was terrible. even with the tools included I could not do anything to fix the issue.
- dspam knows only yes or now, there is no usable value that gives you some grey information. as a result, I had to check all those spam postings for false positives. Spamassassin on the other hand has that spam result 0
i wont go back to dspam unless someone can offer speciic help for those issues. I believe everyone will face them sooner or later.
Re:did they fix the problems? (Score:2, Informative)
As
Re:did they fix the problems? (Score:3, Informative)
Did you run the nightly and weekly purge scripts, as documented? (purge.sql for your DBI driver)
Did you also change the model to TUM from the default? ( MUCH more accurate results over TOE or TEFT in our case, and we get a lot of spam!)
Re:did they fix the problems? (Score:3, Informative)
Did you try TOE mode? Instead of analyzing everything, it just uses the errors. That means significantly less utilization of your data backend. From the FAQ:
Switch to TOE Mode. DSPAM v2.10 supports TOE (Train-On-Error) mode, which only performs writes to the database in the event that a misclassification has occured (or if a user has fewer than 4000 innocent messages in corpus). Train-on-error mode should make a significant reduction in the numbe
Does DSPAM inform the sender? (Score:3, Funny)
Re:Does DSPAM inform the sender? (Score:5, Interesting)
Re:Does DSPAM inform the sender? (Score:1)
Re:Does DSPAM inform the sender? (Score:2)
No bounces! (Score:2)
What you should do, however, is reject the message in the SMTP session. My mail server issues a 554 during SMTP if you send me a spam or a virus. That way, legitamate senders will still get a notification of the delivery failure (generated by th
Ask /. (Score:2)
Which provider do you think does the best effort to filter/fight spam and uses the most state of the art techniques for that? The german freemailer GMX I use now is good, but I wonder if others do better.
And I wouldn't mind paying for never receiving spam again. Is Apple
Re:Ask /. (Score:2)
Re:Ask /. (Score:2)
I tried marking the messages as 'not spam' based on the sender, but every single message has a different - unique - sender so that failed. To top it all, I could not even remove the 30-odd senders from the list again.
Now it is down to Mozilla's spam blocker again. It has virtually zero false-positives, but misses too many (30%) spam messages.
There are times when I'd love to have a baseball bat and a list
Platforms... (Score:2, Interesting)
HOWTO for idiots? (Score:1)
think- spamassassin, clam, spammassassin howto or something similar but it has to be VERY verbose to bring in the crowds (newbies).
my 2c
AC
Alternative spam solution: Change the culture. (Score:2, Interesting)
Here's another spam solution:
If we had a respected national leader who could often talk to millions of people, that person could change the culture. The leader could tell everyone never to buy anything or even respond to unsolicited email advertising.
It might take years, but eventually it would not be economic for spammers to operate, particularly since spam filters would continue to improve.
The only person who could do this in the U.S. now would be Oprah Winfrey. She has an enormous following,
Call me bitter, but... (Score:5, Informative)
Perhaps it's because I don't tend to make super-wild claims about POPFile's accuracy? Or come up with cool marketing names for the internal technology?
POPFile's the only Bayesian filter that can:
1. Do more than spam vs. anti-spam and
2. Filter POP3, IMAP, SMTP and NNTP (that's right Usenet news)
Do I have an axe to grind with Jonathan and DSPAM? No, it's a cool project. Does it annoy me that
John.
Re:Call me bitter, but... (Score:2)
Re:Call me bitter, but... (Score:3, Funny)
Do I like to ask questions aloud and then answer them myself? You bet.
;)
Re:Call me bitter, but... (Score:2)
Re:Call me bitter, but... (Score:2)
By the way I love PopFile.
Re:Call me bitter, but... (Score:2)
Here's what i want in a spam filter... (Score:1, Interesting)
Once the admins start getting hundreds of thousands of spam complaints in their abuse boxes PER DAY. Then maybe they'll start to think of ways to fix this problem.
Re:Here's what i want in a spam filter... (Score:2)
BTW, the mail admins already get hundreds of thousands of complaints per day, they don't need more.
Re:Here's what i want in a spam filter... (Score:2)
Before filtering (Score:2, Informative)
I got nothing against content-filtering measures, as long as one is aware that this should be just the last layer of defense againts spam. Think about it, if your SMTP has already swallowed the spammer's email content, you have already lost precious bandwith.
Especially if you host your own SMTP, you should put up a layered system of defenses: RBL lists, maybe tarpitting, white/graylisting, and then content filtering.
XMail (Score:2)
GPLware (Score:2)
Why should a business donate money (Score:2)
Shouldn't they pay for the costs when they are caught?
Re:None of the above (Score:2, Insightful)
Re:None of the above (Score:2)
:-)
Re:Bayesian Filter Will Stop Working Soon (Score:1)
The only way spammers could slip under the radar of Bayesian filters is to start sending mail that is completely identical to legit mail you get. Which would be rather pointless, unless you're legitimately getting a lot of ads.
Re:Bayesian Filter Will Stop Working Soon (Score:2)
As for the rest of us, whatever schtuff the spammers add, just makes the spam easier to remove, since it increases the statistical distance between regular mail and spam. Since spammers started to do that, my systems went from 99.6% accuracy to practically 100% accuracy. I get 2000 messages per day and maybe s
Re:Just use RBLs (Score:3, Interesting)
Having users sort their mail and train a statistical filter from scratch is just way too much to ask - you'll get inundated with support calls and executives just don't have time to sort out the crud - they hired YOU to do it - passing the buck back to them ain't gonna fly...
The system should get rid of 99.9% of the crud by default, then let the users wholfeel like doing it, report the remaining 0.1% to a central mailbox where you can sort it and re
Re:Sigh... (Score:2)
See this extract for Postfix:
Re:Sigh... (Score:2)