[Dspam-user] Dspam not working like expected anymore

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Dspam-user] Dspam not working like expected anymore

Christoph Pleger
Hello,

it seems that my dspam does not work like expected any more. It did once, but  
now, for a long time nothing has been automatically marked as spam, even if
the spam emails are very similar to what I have defined as spam by moving
emails to my Spam folder.

About my configuration: I use dovecot as IMAP daemon and there is an antispam
plugin available for dovecot. I configured that plugin so that emails are saved
in ${MAILUSER_HOME}/spam/${UNIQUE_FILENAME} when moving  them to an IMAP
folder called "Spam" and that emails are saved in
${MAILUSER_HOME}/ham/${UNIQUE_FILENAME} when moving them away from the Spam
folder. From time to time, a cron job calls a script /usr/local/sbin/dspam-
retrain which processes the files in the ham and spam subdirectories of every
mail user.

The attached file contains my dovecot configuration files from /etc/dspam, a file
/etc/default/dspam which sets some dspam options and the dspam-retrain script.
As already mentioned, there was a time when my dspam recognized and marked
spam emails. I do not remember if I changed any of my configuration files since
then.

My dspam version is 3.10.1

Regards
  Christoph
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user

dspam.tar.gz (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

ktm@rice.edu
On Mon, Mar 09, 2015 at 08:02:50PM +0100, Christoph Pleger wrote:

> Hello,
>
> it seems that my dspam does not work like expected any more. It did once, but  
> now, for a long time nothing has been automatically marked as spam, even if
> the spam emails are very similar to what I have defined as spam by moving
> emails to my Spam folder.
>
> About my configuration: I use dovecot as IMAP daemon and there is an antispam
> plugin available for dovecot. I configured that plugin so that emails are saved
> in ${MAILUSER_HOME}/spam/${UNIQUE_FILENAME} when moving  them to an IMAP
> folder called "Spam" and that emails are saved in
> ${MAILUSER_HOME}/ham/${UNIQUE_FILENAME} when moving them away from the Spam
> folder. From time to time, a cron job calls a script /usr/local/sbin/dspam-
> retrain which processes the files in the ham and spam subdirectories of every
> mail user.
>
> The attached file contains my dovecot configuration files from /etc/dspam, a file
> /etc/default/dspam which sets some dspam options and the dspam-retrain script.
> As already mentioned, there was a time when my dspam recognized and marked
> spam emails. I do not remember if I changed any of my configuration files since
> then.
>
> My dspam version is 3.10.1
>
> Regards
>   Christoph


Hi Christoph,

Do DSPAM headers get added to your messages? That will indicate that they are
being processed, at least. We see poor tagging most often as a result of using
the TEFT (train on everything) setting. That is good for initial learning but
over time the sensitivity continues to decrease based on your ratio of good to
bad messages. Effectively, since no mail is exactly 50:50 the system gets swamped
with data from the most frequent type. If that is the cause, we usually delete
all the tokens and start from scratch with training using the TOE (train on error)
setting. That gets it going again.

Regards,
Ken

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

Christoph Pleger
Hello,

> Do DSPAM headers get added to your messages?

Yes, they are.

> We see poor tagging most often as a result of
> using
> the TEFT (train on everything) setting. That is good for initial learning
> but
> over time the sensitivity continues to decrease based on your ratio of
> good to
> bad messages. Effectively, since no mail is exactly 50:50 the system gets
> swamped
> with data from the most frequent type. If that is the cause, we usually
> delete
> all the tokens and start from scratch with training using the TOE (train
> on error)
> setting. That gets it going again.

I changed my setting from "teft" to "toe". Additionally, to really start
with a clean database, I deleted all entries from the dspam tables.

Regards
  Christoph


------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

ktm@rice.edu
On Tue, Mar 10, 2015 at 05:17:36PM +0100, Christoph Pleger wrote:

> Hello,
>
> > Do DSPAM headers get added to your messages?
>
> Yes, they are.
>
> > We see poor tagging most often as a result of
> > using
> > the TEFT (train on everything) setting. That is good for initial learning
> > but
> > over time the sensitivity continues to decrease based on your ratio of
> > good to
> > bad messages. Effectively, since no mail is exactly 50:50 the system gets
> > swamped
> > with data from the most frequent type. If that is the cause, we usually
> > delete
> > all the tokens and start from scratch with training using the TOE (train
> > on error)
> > setting. That gets it going again.
>
> I changed my setting from "teft" to "toe". Additionally, to really start
> with a clean database, I deleted all entries from the dspam tables.
>
> Regards
>   Christoph
>

Hi Christoph,

If you were using TEFT, that is almost certainly the cause of your reduced
accuracy.

Regards,
Ken

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

Eric Broch
On 3/10/2015 10:25 AM, [hidden email] wrote:

> On Tue, Mar 10, 2015 at 05:17:36PM +0100, Christoph Pleger wrote:
>> Hello,
>>
>>> Do DSPAM headers get added to your messages?
>> Yes, they are.
>>
>>> We see poor tagging most often as a result of
>>> using
>>> the TEFT (train on everything) setting. That is good for initial learning
>>> but
>>> over time the sensitivity continues to decrease based on your ratio of
>>> good to
>>> bad messages. Effectively, since no mail is exactly 50:50 the system gets
>>> swamped
>>> with data from the most frequent type. If that is the cause, we usually
>>> delete
>>> all the tokens and start from scratch with training using the TOE (train
>>> on error)
>>> setting. That gets it going again.
>> I changed my setting from "teft" to "toe". Additionally, to really start
>> with a clean database, I deleted all entries from the dspam tables.
>>
>> Regards
>>   Christoph
>>
> Hi Christoph,
>
> If you were using TEFT, that is almost certainly the cause of your reduced
> accuracy.
>
> Regards,
> Ken
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Dspam-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/dspam-user
Hi Ken,

I've been running TEFT for the better part of 2 years with great
success. Can I switch mid-stream over to TOE with no ill affects? And,
would I change the setting in the configuration file and in training?

Eric

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

ktm@rice.edu
On Tue, Mar 10, 2015 at 03:11:43PM -0600, Eric Broch wrote:

> On 3/10/2015 10:25 AM, [hidden email] wrote:
> > On Tue, Mar 10, 2015 at 05:17:36PM +0100, Christoph Pleger wrote:
> >> Hello,
> >>
> >>> Do DSPAM headers get added to your messages?
> >> Yes, they are.
> >>
> >>> We see poor tagging most often as a result of
> >>> using
> >>> the TEFT (train on everything) setting. That is good for initial learning
> >>> but
> >>> over time the sensitivity continues to decrease based on your ratio of
> >>> good to
> >>> bad messages. Effectively, since no mail is exactly 50:50 the system gets
> >>> swamped
> >>> with data from the most frequent type. If that is the cause, we usually
> >>> delete
> >>> all the tokens and start from scratch with training using the TOE (train
> >>> on error)
> >>> setting. That gets it going again.
> >> I changed my setting from "teft" to "toe". Additionally, to really start
> >> with a clean database, I deleted all entries from the dspam tables.
> >>
> >> Regards
> >>   Christoph
> >>
> > Hi Christoph,
> >
> > If you were using TEFT, that is almost certainly the cause of your reduced
> > accuracy.
> >
> > Regards,
> > Ken
> >
> Hi Ken,
>
> I've been running TEFT for the better part of 2 years with great
> success. Can I switch mid-stream over to TOE with no ill affects? And,
> would I change the setting in the configuration file and in training?
>
> Eric

Hi Eric,

Changing from TEFT, if it has high accuracy results to TOE should be
fine. If you are not getting good results (high 90's percent accuracy),
I would make a clean start.

Regards,
Ken

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

[Dspam-user] Dspam wiki for postfix

Remo Mattei
Hello all

Not sure if anyone has any suggestions on how to have postfix dspam integration and also if anyone has implemented this on centos 7. I was looking at postfix admin as well any other suggestions will be great.  I am running some older version of Linux with qmail.

Thanks.

Remo
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam wiki for postfix

Phil Stracchino
On 03/10/15 17:47, Remo Mattei wrote:
> Hello all
>
> Not sure if anyone has any suggestions on how to have postfix dspam
> integration and also if anyone has implemented this on centos 7. I
> was looking at postfix admin as well any other suggestions will be
> great.  I am running some older version of Linux with qmail.

I'm doing dspam with postfix by configuring it to use dspam as a content
filter.
In /etc/postfix/master.cf:

smtp      inet  n       -       n       -       -       smtpd
        -o content_filter=dspam:

dspam           unix    -       n       n       -       -       pipe
        flags=Rhq user=dspam argv=/usr/bin/dspamfilter -f ${sender} --
${recipient}


And /usr/bin/dspamfilter can be as little as:


#!/bin/bash
USER=$(echo $4 | cut -d@ -f1)
/usr/bin/dspam --deliver=innocent --stdout --user $USER |
/usr/lib/sendmail -i "$@"


This isn't the only way to do it.  But it's a simple way and it works
well.  You may need to fiddle a little with groups and permissions.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

Christoph Pleger
In reply to this post by ktm@rice.edu
Hello,

> If you were using TEFT, that is almost certainly the cause of your reduced
> accuracy.

It is working again! Thanks for your advice. Some emails have been
mis-identified as spam since yesterday, but I guess that these errors will
become fewer after some time of training.

Regards
  Christoph


------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

Phil Stracchino
On 03/11/15 04:10, Christoph Pleger wrote:
> Hello,
>
>> If you were using TEFT, that is almost certainly the cause of your reduced
>> accuracy.
>
> It is working again! Thanks for your advice. Some emails have been
> mis-identified as spam since yesterday, but I guess that these errors will
> become fewer after some time of training.

I had some misidentification problems a while back, after I reinstalled
DSpam during a mailserver move and it somehow got accidentally set to
TEFT instead of TOE.  (I still don't understand how that happened.)  A
few mis-classifications were so recalcitrant that I had to use
dspam-dump to identify the correct tokens, then go into the database and
manually edit their innocent/spam hits to force them to be auto-whitelisted.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|

Re: [Dspam-user] Dspam not working like expected anymore

Martin Wheldon
Hi Phil,

It may have been due to TEFT being the default configuration.

Best Regards

Martin

On 2015-03-11 11:36, Phil Stracchino wrote:

> On 03/11/15 04:10, Christoph Pleger wrote:
>> Hello,
>>
>>> If you were using TEFT, that is almost certainly the cause of your
>>> reduced
>>> accuracy.
>>
>> It is working again! Thanks for your advice. Some emails have been
>> mis-identified as spam since yesterday, but I guess that these
>> errors will
>> become fewer after some time of training.
>
> I had some misidentification problems a while back, after I
> reinstalled
> DSpam during a mailserver move and it somehow got accidentally set to
> TEFT instead of TOE.  (I still don't understand how that happened.)  
> A
> few mis-classifications were so recalcitrant that I had to use
> dspam-dump to identify the correct tokens, then go into the database
> and
> manually edit their innocent/spam hits to force them to be
> auto-whitelisted.
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   [hidden email]
>   [hidden email]
>   Landline: 603.293.8485
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your
> hub for all
> things parallel software development, from weekly thought leadership
> blogs to
> news, videos, case studies, tutorials and more. Take a look and join
> the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Dspam-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/dspam-user
>
> !DSPAM:41,550028f245172001316752!


------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user