Bug 3993 - Claws Mail connects to IMAP server when it should not
Summary: Claws Mail connects to IMAP server when it should not
Status: RESOLVED FIXED
Alias: None
Product: Claws Mail
Classification: Unclassified
Component: Other (show other bugs)
Version: 3.16.0
Hardware: PC Linux
: P3 normal
Assignee: users
URL:
: 4048 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-03-20 00:25 CET by George
Modified: 2018-07-13 10:51 CEST (History)
0 users

See Also:


Attachments
network log exceprt (303 bytes, text/plain)
2018-03-20 00:25 CET, George
no flags Details
debug.txt (6.45 KB, text/plain)
2018-03-20 13:06 CET, George
no flags Details
screenshot (137.79 KB, image/png)
2018-03-20 17:46 CET, George
no flags Details
console output of STR (2.12 KB, text/plain)
2018-04-14 13:46 CEST, George
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description George 2018-03-20 00:25:04 CET
Created attachment 1858 [details]
network log exceprt

I noticed that CM starts very slowly during the last few runs. So I tried to find out why. It turns out that the slowdown was due to a connection it attempts.

** IMAP error on imap.mail.yahoo.com: stream error
** IMAP connection broken
[01:00:20] IMAP< Error logging in to imap.mail.yahoo.com
(the error was temporary and server side)

Investigating further:

STR:
1. Add an IMAP account (tested with GMAIL one)
'Get Mail' checks for new messages on this account = enabled.

2. Add another IMAP account (tested with YAHOO one)
'Get Mail' checks for new messages on this account = disabled.

3. 'Check for new mail on start-up' = disabled.

4. Quit CM leaving the selected folder to be some of the folders of the first account

5. Restart CM

EXPECTED:
Connection only to the Gmail account.

ACTUAL:
Network log shows that CM logs in the Yahoo account too (see attachment).
Seems to happen only with IMAP accounts (not with POP3).


Additional note (can be filed in a separate bug if needed):

The program windows does not to show up if there is some slow down in connectivity (as explained in the beginning). This is incorrect behavior: The main window must show regardless of network problems.
Comment 1 Paul 2018-03-20 11:27:58 CET
your attached extracts from the netwok log are too brief. Start with `claws-mail --debug` and attach the debug output.
Comment 2 George 2018-03-20 13:06:00 CET
Created attachment 1859 [details]
debug.txt

Well, it still illustrates the fact that an unrequested connection is made. Anyway I am attaching a more detailed excerpt. It shows that a whole 1 minute is lost in attempting this connection.
Comment 3 Paul 2018-03-20 13:18:13 CET
Even this longer debug output is chopped too much. It shows a connection is made, yes, but it doesn't show that this was an unrequested connection.

What is the output of

grep -n receive_at ~/.claws-mail/accountrc
Comment 4 Paul 2018-03-20 13:21:22 CET
What your debug.txt does suggest is that you are selecting one of the folders within your yahoo imap mailbox.
Comment 5 George 2018-03-20 13:26:26 CET
Of course it will be chopped. You are asking for a debug of a login process, so I am careful not to publish any secure info.

[~]: grep -n receive_at ~/.claws-mail/accountrc
27:receive_at_get_all=1
133:receive_at_get_all=0
239:receive_at_get_all=1
345:receive_at_get_all=0
451:receive_at_get_all=0
557:receive_at_get_all=0
663:receive_at_get_all=0

None of these 2 enabled accounts are Yahoo accounts. The first one (27) is an IMAP account, the other one (239) is a POP account.
Comment 6 George 2018-03-20 13:28:24 CET
> What your debug.txt does suggest is that you are selecting one of the folders within your yahoo imap mailbox.

That is not true. The Yahoo account folders are completely collapsed under their root (account level) folder and the selected folder is one of the folders in the active ('get mail' enabled) IMAP account.
Comment 7 Paul 2018-03-20 14:20:11 CET
If you provided the complete debug output - which you can send privately, if you want - then it may become more apparent what is happening.

What I can say is that the behaviour you describe has never been reported by anyone else and is not repducible. This suggests the root cause is your local set up and configuration choices.

The more information you can give the more likely a cause, be it a bug or not, can be found.
Comment 8 George 2018-03-20 14:47:11 CET
Which part of the STR is not reproducible?

The full debug output was 3.8Mb txt revealing all my IMAP folder structures, particular message counts and all kinds of other private info. There is no way to send this to anyone - even privately. It is not even necessary as it has nothing to do with the bug report. That's why I have extracted just the part which reveals the actual connection and the startup slowdown it results in. I hope you understand, so if you need further info please advise how to provide what you actually need, not just "complete debug log".
Comment 9 Paul 2018-03-20 15:18:44 CET
(In reply to comment #8)
> Which part of the STR is not reproducible?

"Claws Mail connects to IMAP server when it should not"

> The full debug output was 3.8Mb txt revealing all my IMAP folder structures,
> particular message counts and all kinds of other private info. There is no
> way to send this to anyone - even privately.

Nothing very private would be be revealed, including your passwords.

> It is not even necessary as it
> has nothing to do with the bug report.

If you know best then there's nothing further I can do here.
Comment 10 Andrej Kacian 2018-03-20 17:02:36 CET
How does "last_opened_folder" line look in your ~/.claws-mail/clawsrc ?
Comment 11 George 2018-03-20 17:46:56 CET
Created attachment 1860 [details]
screenshot

Paul, the way you reply is telling me that either you are using different software version or you have not repeated the STR. To triple check the validity of my own words I have started from scratch, deleted all my accounts and followed strictly the STR provided here and I am getting the same result. I am attaching a screenshot showing that. Here is also the additional info requested:

[~]: grep -n receive_at ~/.claws-mail/accountrc
27:receive_at_get_all=1
133:receive_at_get_all=0

[~]: grep last_opened_folder ~/.claws-mail/clawsrc
last_opened_folder=#imap/GMAIL/INBOX

I am not going to paste the full output of '--debug' because as explained it reveals private data but you can debug further yourself. The STR are valid and reproducible.
Comment 12 Andrej Kacian 2018-03-21 20:55:33 CET
I tried reproducing this step by step, but I do not see any connection to the second account's server. Claws Mail started, and opened the folder I had opened before. Network log shows no second connection attempt.

The only difference was that I did not use Yahoo and Gmail, but my private IMAP account, and my work IMAP account.

There has to be something else happening at your end.
Comment 13 George 2018-03-22 01:40:28 CET
Andrej, thanks for testing this. I have started from scratch again, this time not only deleting clawsrc but also with an empty ~/.claws-mail as described. I can confirm the same result as you - no connection to YAHOO (the secondary account).

> There has to be something else happening at your end.

Obviously. After spending more time trying to investigate what exactly that may be I found something quite strange.

Using the problematic .claws-mail dir I removed all existing accounts except the GMAIL and YAHOO ones:

[~]: grep Account .claws-mail/accountrc
[Account: 1] // GMAIL
[Account: 5]
[Account: 10]
[Account: 7]
[Account: 8] // YAHOO
[Account: 9]
[Account: 4]

I.e. after the change only 1 and 8 were left. Then I closed CM and deleted everything except:

~/.claws-mail/accountrc
~/.claws-mail/clawsrc
~/.claws-mail/folderlist.xml
~/.claws-mail/passwordstorerc

To make sure I have the same settings as in the "problematic" version I diff-ed them with the files in the "clean" (non-problematic version) and reduced the differences to a point at which only the account IDs of the YAHOO account were different. In particular:

In the problematic .claws-mail YAHOO account had ID=8 (as seen above) and in the "clean" one it had ID=2. Everything else was the same.

So I went on and deleted the YAHOO account from the problematic version and re-added it again, using identical settings. RESULT: no more connection to yahoo. The only change I noticed the config files was that now that account received ID=6.

This made me question: is the number '8' special in some way? So I went on and changed account numbers from '2' to '8' in the clean version (checking each relevant section in the files above) and ran CM again: no connection to yahoo.

I also did the opposite: I changed the account number from '8' to '6' in the initial .claws-mail version. RESULT: there still IS connection to yahoo. I also tried changing '8' to '2' - same, still connection to yahoo.

I remember a few days ago by mistake I did some drag and drop of one or two the accounts in the window which lists all accounts. This resulted in duplicated accounts and I had to delete the duplicates. I am mentioning this because it may in some way be related to the non-sequential account IDs. But still I can't find what the exact mathematical or logical relationship may be.

So this is totally mysterious to me.
Comment 14 Ricardo Mones 2018-03-22 09:44:12 CET
(In reply to comment #13)
[...]
> 
> So this is totally mysterious to me.

Assuming you also had disabled automatic mail checking and enabled the 'Open last opened folder at start-up', which you didn't mention in your initial description, I've followed your steps and no connection was made to the disabled server.

The only explanation I can found is that you may have inadvertently left open a folder of the second (disabled) account instead of first, which obviously makes it to connect.
Comment 15 George 2018-03-22 14:22:47 CET
Thanks for testing Ricardo. The explanation you give may sound logical and I would think about it myself as a side observer but it is not the actuality. The last selected folder has always been in the GMAIL account as previously mentioned.

I actually found that CM makes a connection even to a third account (when using my actual setup which has more than 2 accounts). So I have been digging further into this because I really wanted to see what was causing it. And I think/hope I have nailed it.

The things which result in connection related to the secondary (YAHOO) account:

1. Not having the file:

~/.claws-mail/imapcache/imap.mail.yahoo.com/<yahoo_username>/Queue/.claws_cache

2. Having any rule(s) in the [preglobal] section of

~/.claws-mail/matcherrc

Any or both of this result in a connection to the YAHOO account on program start.

FWIW: this is not information which the debug log would show directly or explicitly. I found the first factor by running diff on 2 different debug logs which simply showed me that .claws_cache was not found but there was no direct feedback explaining that *because* of that a connection is made. I just guessed it through the sequence of things and was able to confirm it only after more meticulous testing. For the second factor I found no mention in the debug log whatsoever.

Please consider reopening this ticket to investigate this further because it seems related to program logic, not to settings per se.

Another thing I found during testing: right after setting up the secondary account (YAHOO) a connection to yahoo is made immediately after completing the configuration without clicking "Get mail" or anything explicitly requiring connectivity. I don't know if that is a bug or deliberate logic but you may want to check it too because ideally in a system in which the user (and not the program) is in control the connection should be initiated not upon account setup but when the user explicitly requests it (e.g. on "Get mail", folder select/refresh etc).
Comment 16 George 2018-03-22 18:15:29 CET
Add to 2:

Having rules in [postglobal] also results in connections.
Comment 17 George 2018-03-30 15:01:12 CEST
Please kindly review this bug report. It was unfairly marked as invalid.
Comment 18 Paul 2018-03-30 15:11:11 CEST
There was no unfairness that I could see.
Comment 19 George 2018-03-30 18:19:20 CEST
Here has been given a factual evidence of a bug, investigated down to the exact cause for it. If you are closing it as invalid you are trying to ignore an actual fact (perhaps without even testing it).

It is a valid issue. The existence of a pre/post-processing rule should not lead to unnecessary connections to mail servers when they were not requested, especially on startup, especially if "Check for new mail on startup" is off.

Anyway I am not looking for an argument. It is your software and it is up to you if you want to improve it and how you (dis)regard bug reports.
Comment 20 Paul 2018-03-30 19:09:36 CEST
(In reply to comment #19)
> Here has been given a factual evidence of a bug, investigated down to the
> exact cause for it. If you are closing it as invalid you are trying to
> ignore an actual fact (perhaps without even testing it).

It was tested, by at least 3 different people.
Comment 21 George 2018-03-30 21:41:24 CEST
> It was tested, by at least 3 different people.

The bug was marked as invalid before Comment #15. There is no indication that anyone has tested what was described in #15 and #16. If anyone has taken 10 minutes to do it I am sure he would be able to confirm the issue because the test starts with an empty ~/.claws-mail
Comment 22 Ronald Smith 2018-04-13 21:35:19 CEST
I have tested and I can confirm this is a bug. No connections should be made without explicit request or simply due to existence of pre/post filters.
Comment 23 Paul 2018-04-13 23:26:41 CEST
(In reply to comment #22)
> I have tested and I can confirm this is a bug. No connections should be made
> without explicit request or simply due to existence of pre/post filters.

I tested and could not reproduce it.

What are your pre- and post-processing rules? Note that processing rules only get run on entering a folder or by manual use of a folder's processing rules via the folder context menu.

If you are able to reproduce it, perhaps you can attach the debug log?
Comment 24 Ronald Smith 2018-04-14 11:15:09 CEST
I see the OP has shared a debug log. Why is that not enough?
Comment 25 Paul 2018-04-14 11:22:23 CEST
(In reply to comment #24)
> I see the OP has shared a debug log. Why is that not enough?

I know, but if you look at it you will see it is quite useless. I'm not asking just for fun.
Comment 26 Michael Schwendt 2018-04-14 12:01:49 CEST
I've been curious enough as to give it a try, too, but couldn't reproduce it so far.

What are the exact steps for reproducing it after a clean "rm -rf ~/.claws-mail"?
Comment 27 George 2018-04-14 12:38:11 CEST
The steps are:

1. rm -rf ~/.claws-mail
2. The STR from the OP - you won't see a connection to the YAHOO account
3. Add some pre or post filtering rule. Example:

[preglobal]
enabled rulename "testrule" unread color 1

4. Restart CM - you will see a connection to the YAHOO account
Comment 28 Andrej Kacian 2018-04-14 13:45:06 CEST
I can reproduce this, but I get connection to both accounts, not just the second one. It is because Claws Mail applies these global rules to all folders it knows about during startup by calling this from main():

folder_func_to_all_folders(initial_processing, (gpointer *)mainwin);

This has been the case for many years, possibly even before Claws Mail was originally forked from Sylpheed, so I'm not sure I would categorize this as bug. Maybe we just need better user control about what happens on startup.

Perhaps an "Apply processing rules on startup" checkbox, either global or per-account (or maybe even per-folder) ?
Comment 29 George 2018-04-14 13:46:53 CEST
Created attachment 1865 [details]
console output of STR

Here is how I test in order to isolate the issue. After creating the 2 accounts (GMAIL and YAHOO) I delete everything except the files shown in the output of the tree command and back this up in a dir named TEST. The clawsrc file is the default one which CM creates. Then I run the commands shown.

Using the same procedure anyone should be able to generate one's own debug logs with full details.
Comment 30 George 2018-04-14 13:57:07 CEST
> I can reproduce this, but I get connection to both accounts, not just the second one.

Yes, same here. Thanks for confirming.

> so I'm not sure I would categorize this as bug.

Perhaps it can be categorized as a potential privacy issue. If one starts the program one doesn't need to inform all the IMAP servers "Hey, I started my mail client". One may simply want to read RSS without connecting to any IMAP accounts etc.

> Perhaps an "Apply processing rules on startup" checkbox, either global or per-account (or maybe even per-folder) ?

Per-account and per-folder sounds like an overkill. Also while a global option may sound as a good idea it is somewhat "fighting" with the other options like "Check for mail at startup" and not checking a particular account. IOW: there already are options what to check and when so it would make more sense the program logic to follow those preferences rather than initiate connections on its own.

Thinking about all this from a privacy perspective: it would be good to have an option to disable the "NOOP" commands sent every now and then without going offline. If mail should be checked for example every 30 minutes, there is hardly any point to tell the server "Well, I am not checking my mail but you should know that I am online". All this background chattering only helps the companies to create better analytics and profile us more and more deeply. Perhaps this needs a separate bug report?
Comment 31 Andrej Kacian 2018-04-14 14:19:34 CEST
(In reply to comment #30)
> Per-account and per-folder sounds like an overkill. Also while a global
> option may sound as a good idea it is somewhat "fighting" with the other
> options like "Check for mail at startup" and not checking a particular
> account. IOW: there already are options what to check and when so it would
> make more sense the program logic to follow those preferences rather than
> initiate connections on its own.

Fair enough, that was just the first idea that popped into my mind. :) Maybe it would be simpler just to skip IMAP (or any other remote) folders for this "initial processing" if appropriate.

> Thinking about all this from a privacy perspective: it would be good to have
> an option to disable the "NOOP" commands sent every now and then without
> going offline. If mail should be checked for example every 30 minutes, there
> is hardly any point to tell the server "Well, I am not checking my mail but
> you should know that I am online". All this background chattering only helps
> the companies to create better analytics and profile us more and more
> deeply. Perhaps this needs a separate bug report?

Yes, this is an entirely separate topic, just like your mention elsewhere about optionally not connecting to a newly added IMAP account.
Comment 32 George 2018-04-14 14:29:00 CEST
> Yes, this is an entirely separate topic...

OK. Added bug reports for both things: 4008 and 4009.
Comment 33 George 2018-04-14 14:33:09 CEST
Now that this is a confirmed and reproduced issue, can someone please consider reopening this bug and hopefully fixing it? After so many hours of testing to find the issue it would be a pity to leave it to sink.
Comment 34 users 2018-04-17 21:23:03 CEST
Changes related to this bug have been committed.
Please check latest Git and update the bug accordingly.
You can also get the patch from:
http://git.claws-mail.org/

++ ChangeLog	2018-04-17 21:23:03.627927074 +0200
http://git.claws-mail.org/?p=claws.git;a=commitdiff;h=93f58c355b267a733a6756972925d904e5c3b166
Merge: 846f511 a29676e
Author: Colin Leroy <colin@colino.net>
Date:   Tue Apr 17 21:23:02 2018 +0200

    Merge branch 'master' of file:///home/git/claws

http://git.claws-mail.org/?p=claws.git;a=commitdiff;h=a29676ed69c4ff8566ae4eb8ca7e650e40cda22b
Author: Andrej Kacian <ticho@claws-mail.org>
Date:   Tue Apr 17 21:17:34 2018 +0200

    Fix unwanted IMAP connections on startup caused by processing rules.
    
    The culprit was a combination of folder_item_prefs_clear()
    setting item->enable_processing to TRUE even though default
    value should be FALSE, and initial_processing() not skipping
    over root folders, which do not have this variable later set
    to correct default value in xml_to_folder_item().
    
    Closes bug #3993: Claws Mail connects to IMAP server when it should not
Comment 35 Andrej Kacian 2018-04-17 21:24:03 CEST
*poof*, bug fixed.
Comment 36 George 2018-04-17 23:04:06 CEST
Great! Thanks.
Comment 37 Ronald Smith 2018-04-21 11:55:14 CEST
Thank you.
Comment 38 Paul 2018-07-13 10:51:59 CEST
*** Bug 4048 has been marked as a duplicate of this bug. ***