[Dspam-user] Redelivery from quarantine not working

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Dspam-user] Redelivery from quarantine not working

Phil Stracchino
I'm having a problem with redelivery from quarantine not working.  As
far as I can tell, everything SHOULD be working.  I'm not sure what I'm
missing.  It used to work, and to the best of my knowledge nothing in
either my DSpam or Postfix configuration has changed since it was
regularly worked.

Can anyone perhaps tell me where *you* would start troubleshooting this
problem?

My DSpam-web runs as user/group dspam.  The dspam user is a member of
groups mail, apache, and postfix.  Postfix's /usr/sbin/sendmail drop-in
is configured as DSpam's trusted and untrusted delivery agents.  DSpam
is configured to deliver:

DeliveryHost 127.0.0.1
DeliveryPort 10026
DeliveryIdent localhost
DeliveryProto SMTP

which Postfix listens on:

localhost:10026 inet  n -       n       -       -        smtpd
        -o content_filter=
        -o
receive_override_options=no_address_mappings,no_unknown_recipient_checks,no_header_body_checks
        -o smtpd_helo_restrictions=
        -o smtpd_client_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks=127.0.0.0/8
        -o smtpd_authorized_xforward_hosts=127.0.0.0/8


Users apache, dspam, root, and mailman are trusted.  Training mode is
toe.  Nose and whitelist features are enabled, as are graham and burton
algorithms.  Classification rate is doing very well (99.517% overall
accuracy, 91.16% spam identification, 0.311% missed on roughly 200,000
messages delivered).  There's clearly no classification problem.  Only
redelivery of false positives is not working as it should.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

ktm@rice.edu
On Thu, Oct 01, 2015 at 07:39:21AM -0400, Phil Stracchino wrote:

> I'm having a problem with redelivery from quarantine not working.  As
> far as I can tell, everything SHOULD be working.  I'm not sure what I'm
> missing.  It used to work, and to the best of my knowledge nothing in
> either my DSpam or Postfix configuration has changed since it was
> regularly worked.
>
> Can anyone perhaps tell me where *you* would start troubleshooting this
> problem?
>
> My DSpam-web runs as user/group dspam.  The dspam user is a member of
> groups mail, apache, and postfix.  Postfix's /usr/sbin/sendmail drop-in
> is configured as DSpam's trusted and untrusted delivery agents.  DSpam
> is configured to deliver:
>
> DeliveryHost 127.0.0.1
> DeliveryPort 10026
> DeliveryIdent localhost
> DeliveryProto SMTP
>
> which Postfix listens on:
>
> localhost:10026 inet  n -       n       -       -        smtpd
> -o content_filter=
> -o
> receive_override_options=no_address_mappings,no_unknown_recipient_checks,no_header_body_checks
> -o smtpd_helo_restrictions=
> -o smtpd_client_restrictions=
> -o smtpd_sender_restrictions=
> -o smtpd_recipient_restrictions=permit_mynetworks,reject
> -o mynetworks=127.0.0.0/8
> -o smtpd_authorized_xforward_hosts=127.0.0.0/8
>
>
> Users apache, dspam, root, and mailman are trusted.  Training mode is
> toe.  Nose and whitelist features are enabled, as are graham and burton
> algorithms.  Classification rate is doing very well (99.517% overall
> accuracy, 91.16% spam identification, 0.311% missed on roughly 200,000
> messages delivered).  There's clearly no classification problem.  Only
> redelivery of false positives is not working as it should.
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   [hidden email]
>   [hidden email]
>   Landline: 603.293.8485
>

Hi Phil,

What is in your web logs and postfix logs? Postfix is pretty good about
recording delivery problems and their cause. You may also want to check
the MySQL logs for any errors as well.

Regards,
Ken

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/01/15 09:07, [hidden email] wrote:
> Hi Phil,
>
> What is in your web logs and postfix logs? Postfix is pretty good about
> recording delivery problems and their cause. You may also want to check
> the MySQL logs for any errors as well.

Aha! This looks significant:

==> mail.log <==
Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: connect from
localhost[127.0.0.1]
Oct  3 14:48:23 epsilon3 dspam[7638]: Got error 500 in response to RCPT
TO: 500 5.5.2 Error: bad UTF-8 syntax^M
Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: disconnect from
localhost[127.0.0.1] helo=1 mail=1 quit=1 commands=3


Now the question is, what could be causing this?  Which is to say, is it
Postfix or dspam that is doing something wrong here?  Since plenty of
UTF-8 mail seems to come through just fine, I'm assuming a Dspam
configuration problem, but I don't know what.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/03/15 14:52, Phil Stracchino wrote:

> On 10/01/15 09:07, [hidden email] wrote:
>> Hi Phil,
>>
>> What is in your web logs and postfix logs? Postfix is pretty good about
>> recording delivery problems and their cause. You may also want to check
>> the MySQL logs for any errors as well.
>
> Aha! This looks significant:
>
> ==> mail.log <==
> Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: connect from
> localhost[127.0.0.1]
> Oct  3 14:48:23 epsilon3 dspam[7638]: Got error 500 in response to RCPT
> TO: 500 5.5.2 Error: bad UTF-8 syntax^M
> Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: disconnect from
> localhost[127.0.0.1] helo=1 mail=1 quit=1 commands=3
>
>
> Now the question is, what could be causing this?  Which is to say, is it
> Postfix or dspam that is doing something wrong here?  Since plenty of
> UTF-8 mail seems to come through just fine, I'm assuming a Dspam
> configuration problem, but I don't know what.


Well, the first part of the problem seems to be that Dspam may be
sending incorrectly formatted UTF-8, because when I turned off
smtputf8_enable in Postfix, the 'bad UTF-8 syntax' error went away.
However, it seems to have been replaced by this:


Oct  4 14:01:00 epsilon3 postfix/smtpd[11430]: connect from
localhost[127.0.0.1]
Oct  4 14:01:00 epsilon3 postfix/smtpd[11430]: 66BC5157019:
client=localhost[127.0.0.1]
Oct  4 14:01:00 epsilon3 postfix/cleanup[11433]: 66BC5157019:
message-id=<[hidden email]>
Oct  4 14:01:00 epsilon3 opendkim[3414]: 66BC5157019: s=mail d=houzz.com SSL
Oct  4 14:01:00 epsilon3 opendkim[3414]: 66BC5157019: bad signature data
Oct  4 14:01:00 epsilon3 postfix/qmgr[11405]: 66BC5157019: from=<>,
size=132690, nrcpt=1 (queue active)
Oct  4 14:01:00 epsilon3 postfix/smtpd[11430]: disconnect from
localhost[127.0.0.1] helo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Oct  4 14:01:00 epsilon3 postfix/local[11434]: 66BC5157019:
to=<????@caerllewys.net>, orig_to=<????>, relay=local, delay=0.56,
delays=0.46/0.01/0/0.09, dsn=5.1.1, status=bounced (unknown user: "????")
Oct  4 14:01:00 epsilon3 postfix/qmgr[11405]: 66BC5157019: removed


This could possibly be because the quarantined message already contained
UTF8 data.  I'm going to have to wait for another false positive now
before I can test anything else...


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/04/15 14:07, Phil Stracchino wrote:
> Well, the first part of the problem seems to be that Dspam may be
> sending incorrectly formatted UTF-8, because when I turned off
> smtputf8_enable in Postfix, the 'bad UTF-8 syntax' error went away.

Actually, this was pretty clearly a false trail, because if I turn if
smtputf8 in Postfix, I can no longer send misses for retraining.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Sebastian Toepfer
Am 2015-10-05 23:45, schrieb Phil Stracchino:
> On 10/04/15 14:07, Phil Stracchino wrote:
>> Well, the first part of the problem seems to be that Dspam may be
>> sending incorrectly formatted UTF-8, because when I turned off
>> smtputf8_enable in Postfix, the 'bad UTF-8 syntax' error went away.
>
> Actually, this was pretty clearly a false trail, because if I turn if
> smtputf8 in Postfix, I can no longer send misses for retraining.
Hi,

search the mailing list, is a know issue - debian has fixed it.

best regards
sebastian

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
In reply to this post by Phil Stracchino
On 10/03/15 14:52, Phil Stracchino wrote:

> On 10/01/15 09:07, [hidden email] wrote:
>> Hi Phil,
>>
>> What is in your web logs and postfix logs? Postfix is pretty good about
>> recording delivery problems and their cause. You may also want to check
>> the MySQL logs for any errors as well.
>
> Aha! This looks significant:
>
> ==> mail.log <==
> Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: connect from
> localhost[127.0.0.1]
> Oct  3 14:48:23 epsilon3 dspam[7638]: Got error 500 in response to RCPT
> TO: 500 5.5.2 Error: bad UTF-8 syntax^M
> Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: disconnect from
> localhost[127.0.0.1] helo=1 mail=1 quit=1 commands=3


So I haven't made any progress on this except that it seems to be a
DSpam problem, not a Postfix problem.  DSpam apparently *requires*
smtputf8, and I cannot retrain misses without it, but DSpam seems to be
sending postfix incorrectly formatted UTF8 when it attempts redelivery.

I don't SEE any relevant DSpam configuration settings.  Am I missing
anything?



--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

ktm@rice.edu
On Tue, Oct 13, 2015 at 10:01:21AM -0400, Phil Stracchino wrote:

> On 10/03/15 14:52, Phil Stracchino wrote:
> > On 10/01/15 09:07, [hidden email] wrote:
> >> Hi Phil,
> >>
> >> What is in your web logs and postfix logs? Postfix is pretty good about
> >> recording delivery problems and their cause. You may also want to check
> >> the MySQL logs for any errors as well.
> >
> > Aha! This looks significant:
> >
> > ==> mail.log <==
> > Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: connect from
> > localhost[127.0.0.1]
> > Oct  3 14:48:23 epsilon3 dspam[7638]: Got error 500 in response to RCPT
> > TO: 500 5.5.2 Error: bad UTF-8 syntax^M
> > Oct  3 14:48:23 epsilon3 postfix/smtpd[7639]: disconnect from
> > localhost[127.0.0.1] helo=1 mail=1 quit=1 commands=3
>
>
> So I haven't made any progress on this except that it seems to be a
> DSpam problem, not a Postfix problem.  DSpam apparently *requires*
> smtputf8, and I cannot retrain misses without it, but DSpam seems to be
> sending postfix incorrectly formatted UTF8 when it attempts redelivery.
>
> I don't SEE any relevant DSpam configuration settings.  Am I missing
> anything?
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   [hidden email]
>   [hidden email]
>   Landline: 603.293.8485
>
> ------------------------------------------------------------------------------

Hi Phil,

What version of postfix are you running? Did you happen to upgrade to v3.0+?
Maybe there is an incompatability with the way that it handles UTF-8 version
v2.x.

Regards,
Ken

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/13/15 10:52, [hidden email] wrote:
> Hi Phil,
>
> What version of postfix are you running? Did you happen to upgrade to v3.0+?
> Maybe there is an incompatibility with the way that it handles UTF-8 version
> v2.x.
>
> Regards,
> Ken

Indeed, I am running 3.0, and have been for some months now.  (I
actually just updated from 3.0.2-r2 to 3.0.3 this morning.)




--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Marco
> Indeed, I am running 3.0, and have been for some months now.  (I
> actually just updated from 3.0.2-r2 to 3.0.3 this morning.)

Just for info... I run Postfix 3.0.2 (compatibility_level = 3) with  
DSPAM 3.10.2, but I don't find this error in my log. Maybe our clients  
don't use smtputf8 extensions yet...

Regards
Marco





------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/14/15 03:03, Marco wrote:
>> Indeed, I am running 3.0, and have been for some months now.  (I
>> actually just updated from 3.0.2-r2 to 3.0.3 this morning.)
>
> Just for info... I run Postfix 3.0.2 (compatibility_level = 3) with  
> DSPAM 3.10.2, but I don't find this error in my log. Maybe our clients  
> don't use smtputf8 extensions yet...

I don't believe any longer that it is actually a smtputf8 issue.  It
appears I am running into this bug reported by Håkon_Alstadheim, which
appears to have never been fixed:

http://marc.info/?l=dspam-users&m=132382086221854&w=2


Thinking back, I've realized a key detail.  Redelivery worked *FOR ME*
when the master Postfix, and DSpam, ran *on my workstation*, and mail
released from *MY* quarantine was delivered by the *local* delivery
agent.  I think that when redelivery stopped working for me may coincide
with when I moved the master Postfix and DSpam to a separate server.  My
wife had complained before that redelivery didn't work for her, but
every time I tested it, it worked *for me*.


As a workaround for now, I'm testing changing my *own* DSpam preferences
to tag-and-deliver, and then I'm going to have to write a filter to
segregate the spam.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

ktm@rice.edu
On Wed, Oct 14, 2015 at 07:09:46AM -0400, Phil Stracchino wrote:

> On 10/14/15 03:03, Marco wrote:
> >> Indeed, I am running 3.0, and have been for some months now.  (I
> >> actually just updated from 3.0.2-r2 to 3.0.3 this morning.)
> >
> > Just for info... I run Postfix 3.0.2 (compatibility_level = 3) with  
> > DSPAM 3.10.2, but I don't find this error in my log. Maybe our clients  
> > don't use smtputf8 extensions yet...
>
> I don't believe any longer that it is actually a smtputf8 issue.  It
> appears I am running into this bug reported by Håkon_Alstadheim, which
> appears to have never been fixed:
>
> http://marc.info/?l=dspam-users&m=132382086221854&w=2
>
>
> Thinking back, I've realized a key detail.  Redelivery worked *FOR ME*
> when the master Postfix, and DSpam, ran *on my workstation*, and mail
> released from *MY* quarantine was delivered by the *local* delivery
> agent.  I think that when redelivery stopped working for me may coincide
> with when I moved the master Postfix and DSpam to a separate server.  My
> wife had complained before that redelivery didn't work for her, but
> every time I tested it, it worked *for me*.
>
>
> As a workaround for now, I'm testing changing my *own* DSpam preferences
> to tag-and-deliver, and then I'm going to have to write a filter to
> segregate the spam.
>
> --
>   Phil Stracchino
>   Babylon Communications
>   [hidden email]
>   [hidden email]
>   Landline: 603.293.8485
>

Hi Phil,

That would explain why we have not seen an issue because we use LMTP
for delivery from DSPAM, and not SMTP. You might want to give it a try
instead.

Regards,
Ken

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
On 10/14/15 09:03, [hidden email] wrote:
> Hi Phil,
>
> That would explain why we have not seen an issue because we use LMTP
> for delivery from DSPAM, and not SMTP. You might want to give it a try
> instead.

I tried to set it up that way once, and wasn't able to get it to work.
I suppose I could try it again ...

Is there a how-to somewhere for that configuration?


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

ktm@rice.edu
On Wed, Oct 14, 2015 at 09:07:49AM -0400, Phil Stracchino wrote:

> On 10/14/15 09:03, [hidden email] wrote:
> > Hi Phil,
> >
> > That would explain why we have not seen an issue because we use LMTP
> > for delivery from DSPAM, and not SMTP. You might want to give it a try
> > instead.
>
> I tried to set it up that way once, and wasn't able to get it to work.
> I suppose I could try it again ...
>
> Is there a how-to somewhere for that configuration?
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   [hidden email]
>   [hidden email]
>   Landline: 603.293.8485
>

Hi Phil,

I just checked and we actually patched the UI to use a special purpose
command to deliver the Email from quarantine, which uses the postfix
sendmail command directly if the dspam command has a problem. The only
problems that we had seen though were caused by missing signatures
and not the bungled RCPT. We will be delivering with SMTP after the
upgrade. I will let you know what we find.

Regards,
Ken

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Dspam-user] Redelivery from quarantine not working

Phil Stracchino
In reply to this post by ktm@rice.edu
Ken,
Could you perhaps post the details of that redelivery-from-quarantine
patch to the UI?  I'd like to give it a try and see if it solves my problem.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: 603.293.8485

------------------------------------------------------------------------------
_______________________________________________
Dspam-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dspam-user
Loading...