Bug 3533

Summary: DSN (delivery status notification)
Product: Claws Mail (GTK 2) Reporter: Gerard Seibert <gerard.seibert>
Component: SMTPAssignee: users
Status: RESOLVED PATCHESWELCOME    
Severity: enhancement CC: claws-bugzilla
Priority: P3    
Version: 3.12.0   
Hardware: PC   
OS: FreeBSD   
Attachments:
Description Flags
Enables Delivery Status Notification request
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Add Delivery Status Notification (DSN) support
none
Patch patched to apply on 3.17.8
none
New version of the DSN patch none

Description Gerard Seibert 2015-10-05 10:06:42 UTC
Is it possible to add DSN (delivery status notification) to claws-mail? Some MUAs have that ability built in, like MS Outlook; however, I can find no easy way to do it in claws-mail.
Comment 1 Paul 2015-10-05 10:56:09 UTC
In the Compose window, set /Options/Request Return Receipt
Comment 2 Ricardo Mones 2015-10-05 11:21:06 UTC
(In reply to comment #1)
> In the Compose window, set /Options/Request Return Receipt

I think OP could be talking about RFC 3461 and related ones (which is currently not supported), not to RFC 3798 (Message Disposition Notification) which is the one of the return receipt feature. But I may be wrong, so I'll let him reopen if that's the case.
Comment 3 Gerard Seibert 2015-10-05 12:30:09 UTC
Yes, I am referring to RFC 3461. I am sorry, I probably should have mentioned that in my original post. This feature is supported by several MUAs. It should be fairly simple to implement since it only requires a simple addition to the message headers.

If there is no interest in adding this feature, feel free to close this report.
Comment 4 Paul 2015-10-05 12:38:48 UTC
(In reply to comment #3)
> It should be fairly simple to implement since it only requires a
> simple addition to the message headers.

In that case, please submit a patch.
Comment 5 Ricardo Mones 2015-12-28 13:57:24 UTC
(In reply to comment #3)
> Yes, I am referring to RFC 3461. I am sorry, I probably should have
> mentioned that in my original post. This feature is supported by several
> MUAs. It should be fairly simple to implement since it only requires a
> simple addition to the message headers.
> 
> If there is no interest in adding this feature, feel free to close this
> report.

Given the RFC example¹ this is far from a "simple addition to message headers", it needs also a per-address UI, and at least one account property to start with.

But clean patches still welcome :)

¹ https://tools.ietf.org/html/rfc3461#section-10.1
Comment 6 Ricardo Mones 2016-02-16 11:13:39 UTC
Reported also on Debian BTS: https://bugs.debian.org/814884
Comment 7 Alex Fedorov 2017-02-20 23:32:26 UTC
Created attachment 1721 [details]
Enables Delivery Status Notification request

This patch adds the Delivery Status Notification request option to Claws Mail. When enabled, a catch-all (success, failure and delay) notification will be requested from server, along with copy of sent message headers.
Please note, that due to Claws Mail architecture it was found to be very difficult to add such option to compose message dialog, so instead a checkbox controlling the option was added to account properties, under the Send tab. As a result, a "request_dsn" property is added to accountrc config file.
Tested on 3.14.1, should work on git commit bbeb55d, but the latter does not compile due to unrelated bug.
Comment 8 Ricardo Mones 2017-02-21 08:45:44 UTC
(In reply to comment #7)
> Created attachment 1721 [details]
> Enables Delivery Status Notification request
> 
> This patch adds the Delivery Status Notification request option to Claws
> Mail. When enabled, a catch-all (success, failure and delay) notification
> will be requested from server, along with copy of sent message headers.
> Please note, that due to Claws Mail architecture it was found to be very
> difficult to add such option to compose message dialog, so instead a
> checkbox controlling the option was added to account properties, under the
> Send tab. As a result, a "request_dsn" property is added to accountrc config
> file.
> Tested on 3.14.1, should work on git commit bbeb55d, but the latter does not
> compile due to unrelated bug.

Thanks for the patch, haven't tested it, but looks correct. Nevertheless, I think always sending the 3 types of DSN request is a bit inflexible: for some accounts you could be interested only in failures, for example. This can be solved with three additional checkboxes (with their sensitivity linked to main checkbox) and building the right part of NOTIFY= string from them. The UI could be something like this:

+-Request Delivery Status Notification--------------+
| [ ] Enable                                        |
| [ ] Failures [ ] Delays [ ] Success               |
+---------------------------------------------------+

No RET parameter is added to mail command¹, it's not worth to control which is returned on the DSN? 

No ENVID parameter is generated, maybe it's a good idea to generate one, and save it in Claws Mail special headers, so in the future you could track which DSN-requested messages have received the DSN or not.

There's also no special handling of received DSN multipart/report + message/delivery-status messages, though not sure what can be done with those.

Not sure if you're willing to improve the patch, in any case thanks in advance ;)

¹ https://tools.ietf.org/html/rfc3461#section-4.3
Comment 9 Olivier Brunel 2017-07-03 18:11:50 UTC
Created attachment 1764 [details]
Add Delivery Status Notification (DSN) support

Had a look at this recently, so here's a new patch. This is based on Alex Fedorov work - thanks! - and basically just changes UI things.

Specifically, the account option moved to Compose, and was split into 3 checkboxes as suggested: Success, Failure, and Delay.

Those options only set a default, which can itself be overwritten by similar options under folder properties, page Compose.
Either way, a new menu in Compose window will show the current options as well as allow to set them on a per-message basis.
Comment 10 Olivier Brunel 2018-08-28 16:21:34 UTC
Created attachment 1909 [details]
Add Delivery Status Notification (DSN) support

Bumping this with a rebase on 3.17 :)
Comment 11 wwp 2018-08-28 22:32:32 UTC
Good! Why not defaulting to Failure+Delay in folder properties as it's done in account prefs and compose window?
Comment 12 Olivier Brunel 2018-08-30 15:50:20 UTC
(In reply to comment #11)
> Good! Why not defaulting to Failure+Delay in folder properties as it's done
> in account prefs and compose window?

First off, it's only done in account prefs. What you see in the compose window is what will be applied to that mail, and taken from either the folder properties if any, else the account prefs -- IOW this isn't a default that's hard-coded anywhere in the code, it's loaded from user settings.

Now, why not defaulting to Failure+Delay in folder properties as it's done in account prefs? Well, would that be a question or a suggestion?

If the former, I believe my thinking was that the account pref feels more like a "general" setting (trying not to say "user default" here to avoid confusion), where sensible (hard-coded) defaults are good to have, while folder properties felt more like a specific/"restricted" override of said general setting, thus it was less important/better as blank.
I also wonder if there couldn't be "confusion" regarding whether those values were hard-coded defaults or "imported" from an account prefs, and again leaving it "blank" felt both simpler & easier. :)

If the later, sure, I could do that.
Comment 13 wwp 2018-08-30 15:57:46 UTC
Thanks for elaborating. Formerly it was a question (to know if it was a bug or made on purpose, then what purpose?); but it's also a suggestion since I think it would be more consistent to use the same defaults in every place; and I don't think we apply strict options in some place and loose ones in other places for the same feature, elsewhere in CM. Anyway, that is not a big deal at all.
Comment 14 Olivier Brunel 2018-08-30 16:27:55 UTC
Created attachment 1910 [details]
Add Delivery Status Notification (DSN) support

v2: Set proper defaults on folder property
Comment 15 Olivier Brunel 2018-09-05 17:23:52 UTC
Created attachment 1912 [details]
Add Delivery Status Notification (DSN) support

v3: compose: Update current options when changing account

Makes things more consistent with other similar options (e.g. auto-Cc, Privacy System, ...)
Comment 16 wwp 2018-09-14 21:35:46 UTC
Created attachment 1915 [details]
Add Delivery Status Notification (DSN) support

Update of the v3 to build against 3.17.0git53
Comment 17 wwp 2018-11-23 07:53:12 UTC
Created attachment 1935 [details]
Add Delivery Status Notification (DSN) support

Updated to compile against -git176 (2018-11-23).
Comment 18 wwp 2018-11-23 10:30:13 UTC
Created attachment 1936 [details]
Add Delivery Status Notification (DSN) support

Fixes missing diff hunks for compose.h and prefs_folder_item.c, and fix widget layout packing in prefs_account.
Comment 19 Luc Pardon 2020-11-12 11:06:43 UTC
Created attachment 2136 [details]
Patch patched to apply on 3.17.8

Is there any particular reason why this patch has not been accepted yet?

It's not exactly "feature creep", because DSN is quite common and many other MUA's support it.

More importantly <g>, I do need it if I switch from Thunderbird to Claws.

Attached is a version of the patch that applies against the most recent release, 3.17.8. I didn't bother to fix the offsets, because that would be pointless if - as I hope - DSN support will be added in the next release.

I have it applied locally and it seems to work - mostly. There appear to be some issues with the UI, but that may be just me.
Comment 20 Paul 2020-11-12 11:57:29 UTC
In my opinion, the options should be on the "Send" page of the account prefs, and the default should be everything disabled.
Comment 21 Luc Pardon 2020-11-12 16:43:23 UTC
Too many things in one reply, I'm afraid, so I have numbered them for easy reference.

1. Agree with "default = everything disabled". That is how I would set it for my own use anyway.

Also, it preserves current behavior. No surprises for unsuspecting upgraders.

2. I'm not so sure about the "Send" page thing. I say "not sure" because, as long as I have the menu option to request a DSN when I compose a message, I don't really care where I have to change the defaults (it helps that I won't change them :-).

But precisely because the decision about "to DSN or not to DSN" is made by the user when preparing (composing) the message that is about to be sent, the "Compose" page may be the proper home for it - along with the "to/cc/bcc" settings, dictionaries etc. In short, all the defaults for everything that pertains to one specific message.

The "Send" page - as the name suggests (to me) - has more to do with the way the MUA talks to the MTA, and that's probably of a more technical nature (or company/site policy, if you want). 

IMHO the settings on the "Send" page would apply to all messages sent with this account, no need to override them on a message-by-message basis.

In other words: one could argue that these are not defaults, but server settings, and the DSN would be out of place there.


3. In any case, the setting for "Request Return Receipt" in the current UI is on the "Compose" page as well. 

RRR and DSN are similar, so it would be "most illogical" to have them on two different pages.

Meaning: if we send DSN to "Send", RRR has to go there as well.


4. Problem is also that there is no "Send" page when I navigate to the account properties via the account"s context menu (right-click on it in the list), only "General", "Compose" and "Templates" pages - but no "Send". 

Same story for the folder properties (again with RMB). 

That is probably intentional, but if the DSN options should go to "Send", the user would have to remember to use the main menu ("Configuration/Preferences for current account") to access them. 

To put it differently: if I am looking for the DSN settings, and if I have both the "Send" and "Compose" pages on hand, and I don't find DSN in the one, the other is only a mouse-click away. That is what a GUI is all about. 

But that exploration is not possible if I don't have both pages on hand.
Comment 22 Paul 2020-11-13 10:09:53 UTC
There is some confusion here between Folder Properties and Account Preferences.

In comment #20 I said that the options should been on a different page in the Account Preferences, but you seemed to have understood Folder Properties instead.

Account Preferences can never be accessed by right-clicking on a folder.

The Return Receipts option is not in the Account Preferences, it is on the 'Sending' page of the General Preferences.

Yes, in the Folder Properties the Return Receipts option is on the Compose page, as the Folder Properties do not have a Send page. The lack of a Send page here does make that seem slightly illogical but, I imagine, adding a Send page to the Folder Properties for a single option was considered worse at the time it was added. Perhaps, with the DSN options added, it would make a better case for having a Send page in the Folder Properties.

The Folder Properties override all other prefs for that particular folder.

So, yes, as in comment #20, I think these options should be on the Send page rather than the Compose page of the Account Preferences.
Comment 23 Luc Pardon 2020-11-16 15:06:09 UTC
> There is some confusion here between Folder Properties and Account Preferences.

Yes, sorry. I'm still suffering a paradigm-shift hangover <g>. When I right-clicked on an *Account* object, I had expected to get the *Account* properties editor, and I overlooked the fact that I landed on a kind of "super-folder" properties page instead.

Re: "Send" vs. "Compose": As I said before, I don't really care what page the default is on. All I need is the ability to request a "successful delivery" DSN when I compose a mail.

But that brings me to retract my position on the default settings. It seems that the current patch is right after all - kind of. I'll try to clarify.

After reading RFC 3461 (section 4.2, last part) more carefully, it becomes clear that when you send a DSN NOTIFY command, you get what you ask for. No more and no less.

And the "no more" part is the problem here.

By default, every SMTP server MUST send notification on failure (RFC 821 section 3.6) - unless explicitly told that the client does not want that. But that is precisely what you do when you send a DSN NOTIFY without the "FAILURE" keyword.

Same story for delay notification, except that the server admin can decide about whether to send it by default or not. But if your MUA sends NOTIFY without the DELAY keyword, you won't get it, never. RFC 3461 is even very explicit here.

That means that, if the default is "everything disabled", as suggested in comment 20, the user may inadvertently loose failure and delay notifications.

Let's say I start out with everything disabled by default, and I check only the DSN "success" option when composing a mail, and leave the others unchecked. When I click the send button, the patch code duly sends "NOTIFY=SUCCESS" ... and I will *not* be told when the mail bounces or is delayed. It just vanishes.

This was confirmed by testing.

So, upon reflection, I now think that the current patch was right about the default options after all.

The only "problem" is that it now *always* sends "NOTIFY=FAILURE,DELAY", for every single mail, even when that is not really needed nor requested.

So, upon even more reflection, I think that the current solution is a little bit over-engineered. 

The original proposal in comment 7 is much simpler, it fulfills the user's need (i.e. ability to request notification on success, see the Debian bug report mentioned in comment 6), and it matches user expectations (i.e. not unwittingly telling the server that you do NOT want to learn about failures or delays).

It is entirely possible that the user only wants notification on failure, as suggested in comment 8 above. But in that case, no DSN is needed at all, because this is mandatory for any SMTP server anyway.

Conversely, I can see no reason why anybody would *not* want to know when a mail bounces, i.e. why anybody would want to be able to switch these bounces off (in an MUA at least).

It's similar for delay notifications.

If the server admin decided not to send them by default, there will probably be a reason for it, like reducing load, reducing queue size or whatever. The MUA should probably not overrule that decision, and certainly not with its default settings.

Conversely, if I want to be notified of successful delivery, I would probably also want to know about any delays. That might be a valid reason to overrule the admin's decision, but then on a case-by-case basis, i.e. for some "special" mails only, i.e. only in addition to a "SUCCESS" request.

Bottom line:

* I would be happy with a patch that allows me to check a single option when composing a mail, something like "Request notification on successful delivery?" (or just plain "Delivery Status Notification" with a checkbox next to it, as is done for RRR). Default: "no", meaning: do not send any "NOTIFY" command. If set to "yes", always send "NOTIFY=SUCCESS,DELAY,FAILURE".

* if this must be configurable at all, it would only be about whether or not to "Also request delay notification when a success DSN is requested" for a particular mail. Default: "yes", meaning "NOTIFY=SUCCESS,DELAY,FAILURE" as above. If "no", that becomes "NOTIFY=SUCCESS,FAILURE". Personally, I do not think this is needed, because the combination "SUCCESS but no DELAY" does not make sense to me. But if configurable, it should probably be at the Account level, since this is tied to the capabilities of the outgoing SMTP server.

* if we do want to make this "overrideable" (sp?) at the folder level at all, we might want to add another option to "Always request notification on successful delivery" for this folder, at it is done for the RRR. Default: "no". If set to "yes", the option for DSN would already be checked when composing a mail (but it can be unchecked at that time). 

* it should never be possible to send any NOTIFY command without at least the "FAILURE" keyword, to prevent users from shooting themselves in the foot.


As an aside, it would be nice if there was a warning if the user did request some form of DSN but, when CM tries to send the mail, it discovers that the SMTP server does not support DSN at all. The current patch silently ignores that mismatch.
Comment 24 Paul 2020-11-16 16:11:10 UTC
Yes, good point. The option should only be turning the SUCCESS notification on/off.

In addition to a server mot supporting DSN, a server can also just ignore DSN requests.
Comment 25 Luc Pardon 2020-11-24 14:06:33 UTC
Created attachment 2143 [details]
New version of the DSN patch

Attached is a new version of the patch, along the lines that have been discussed above. A description follows, with emphasis on the changes.

The three DSN checkboxes on the Compose/Options menu have been replaced by a single one: “Request Notification of Successful Delivery”.

As agreed before, the CM SMTP client code adds a “NOTIFY=SUCCESS,DELAY,FAILURE” parameter to the RCPT TO command when the option is checked, otherwise no NOTIFY is sent at all.

The handling of the ORCPT parameter is slightly modified: it now strips off the angle brackets if they are present in the mail address, as required by (my understanding of) RFC 822 or its successor, RFC 2822 section 3.4.1. No further attempt is made to enforce conformance to that that format.

When the user requests DSN but the server does not support it, a warning is written to the log and no DSN parameters are sent. The user is not bothered with a pop-up, as there is nothing that can be done about it. If an SMTP server advertises DSN support in its EHLO greeting, it MUST play by the RFC 4631 rules, otherwise it is simply broken. And even a conforming server can not guarantee that a notification will come in all cases. If the message must traverse multiple MTA's, strange things can happen on the way to its destination. In that case, the server admin(s) will probably have to be involved in diagnosing the problem, and the CM log can be a useful starting point.

Back at the GUI side, the “Compose/Options” menu still gets its default from the Folder properties, as with the previous version of the patch.

However, the checkbox for the default DSN is now installed on a brand new “Send” page on the Folder properties, as requested by Paul.

The existing (and related) “Request Return Receipt” setting has moved in there as well. It no longer lives on the “Compose” page, again as discussed above.

These two are being joined there by the “Save copy of outgoing messages” setting. The new “Send” page seemed a more logical home for it than “Compose”.

The new “Send” page is not displayed for an NNTP folder, as it would be empty anyway: none of the three settings apply in that case. 

Unlike the original patch, this version does not add any DSN setting to the "Account" properties (as opposed to "Folder"), for two reasons. 

First, with the simple yes/no model and a single checkbox, there is no obvious  way to configure the Folder to either accept or override an Account setting.

Second, the new DSN property as implemented here is consistent with the existing “Request Return Notification” Folder setting. That can't be inherited from Account either, for the same simple reason: Account has no setting for it. Both properties (RRR and DSN) are similar enough to treat them in a consistent way. Adding both of them to "Account properties" (plus the "inherit/override" mechanism) seemed out of scope for an enhancement request that just asks for DSN support in CM.

That is about all.

If there are no more design issues, and if no bugs are found, I hope that this version of the patch will be accepted.

If it is, credit is due only to the original submitters, Alex Fedorov and Olivier Brunel. I did nothing more than some “copy-coding”, and only the bugs would be mine. Indeed, bugs are not to be excluded, not just because this thing is lightly tested, but mainly because this is my first venture into GTK territory. And somehow I don't think it's going to become my favorite GUI toolkit ... :^)