Created attachment 1858 [details]
network log exceprt
I noticed that CM starts very slowly during the last few runs. So I tried to find out why. It turns out that the slowdown was due to a connection it attempts.
** IMAP error on imap.mail.yahoo.com: stream error
** IMAP connection broken
[01:00:20] IMAP< Error logging in to imap.mail.yahoo.com
(the error was temporary and server side)
1. Add an IMAP account (tested with GMAIL one)
'Get Mail' checks for new messages on this account = enabled.
2. Add another IMAP account (tested with YAHOO one)
'Get Mail' checks for new messages on this account = disabled.
3. 'Check for new mail on start-up' = disabled.
4. Quit CM leaving the selected folder to be some of the folders of the first account
5. Restart CM
Connection only to the Gmail account.
Network log shows that CM logs in the Yahoo account too (see attachment).
Seems to happen only with IMAP accounts (not with POP3).
Additional note (can be filed in a separate bug if needed):
The program windows does not to show up if there is some slow down in connectivity (as explained in the beginning). This is incorrect behavior: The main window must show regardless of network problems.
your attached extracts from the netwok log are too brief. Start with `claws-mail --debug` and attach the debug output.
Created attachment 1859 [details]
Well, it still illustrates the fact that an unrequested connection is made. Anyway I am attaching a more detailed excerpt. It shows that a whole 1 minute is lost in attempting this connection.
Even this longer debug output is chopped too much. It shows a connection is made, yes, but it doesn't show that this was an unrequested connection.
What is the output of
grep -n receive_at ~/.claws-mail/accountrc
What your debug.txt does suggest is that you are selecting one of the folders within your yahoo imap mailbox.
Of course it will be chopped. You are asking for a debug of a login process, so I am careful not to publish any secure info.
[~]: grep -n receive_at ~/.claws-mail/accountrc
None of these 2 enabled accounts are Yahoo accounts. The first one (27) is an IMAP account, the other one (239) is a POP account.
> What your debug.txt does suggest is that you are selecting one of the folders within your yahoo imap mailbox.
That is not true. The Yahoo account folders are completely collapsed under their root (account level) folder and the selected folder is one of the folders in the active ('get mail' enabled) IMAP account.
If you provided the complete debug output - which you can send privately, if you want - then it may become more apparent what is happening.
What I can say is that the behaviour you describe has never been reported by anyone else and is not repducible. This suggests the root cause is your local set up and configuration choices.
The more information you can give the more likely a cause, be it a bug or not, can be found.
Which part of the STR is not reproducible?
The full debug output was 3.8Mb txt revealing all my IMAP folder structures, particular message counts and all kinds of other private info. There is no way to send this to anyone - even privately. It is not even necessary as it has nothing to do with the bug report. That's why I have extracted just the part which reveals the actual connection and the startup slowdown it results in. I hope you understand, so if you need further info please advise how to provide what you actually need, not just "complete debug log".
(In reply to comment #8)
> Which part of the STR is not reproducible?
"Claws Mail connects to IMAP server when it should not"
> The full debug output was 3.8Mb txt revealing all my IMAP folder structures,
> particular message counts and all kinds of other private info. There is no
> way to send this to anyone - even privately.
Nothing very private would be be revealed, including your passwords.
> It is not even necessary as it
> has nothing to do with the bug report.
If you know best then there's nothing further I can do here.
How does "last_opened_folder" line look in your ~/.claws-mail/clawsrc ?
Created attachment 1860 [details]
Paul, the way you reply is telling me that either you are using different software version or you have not repeated the STR. To triple check the validity of my own words I have started from scratch, deleted all my accounts and followed strictly the STR provided here and I am getting the same result. I am attaching a screenshot showing that. Here is also the additional info requested:
[~]: grep -n receive_at ~/.claws-mail/accountrc
[~]: grep last_opened_folder ~/.claws-mail/clawsrc
I am not going to paste the full output of '--debug' because as explained it reveals private data but you can debug further yourself. The STR are valid and reproducible.
I tried reproducing this step by step, but I do not see any connection to the second account's server. Claws Mail started, and opened the folder I had opened before. Network log shows no second connection attempt.
The only difference was that I did not use Yahoo and Gmail, but my private IMAP account, and my work IMAP account.
There has to be something else happening at your end.
Andrej, thanks for testing this. I have started from scratch again, this time not only deleting clawsrc but also with an empty ~/.claws-mail as described. I can confirm the same result as you - no connection to YAHOO (the secondary account).
> There has to be something else happening at your end.
Obviously. After spending more time trying to investigate what exactly that may be I found something quite strange.
Using the problematic .claws-mail dir I removed all existing accounts except the GMAIL and YAHOO ones:
[~]: grep Account .claws-mail/accountrc
[Account: 1] // GMAIL
[Account: 8] // YAHOO
I.e. after the change only 1 and 8 were left. Then I closed CM and deleted everything except:
To make sure I have the same settings as in the "problematic" version I diff-ed them with the files in the "clean" (non-problematic version) and reduced the differences to a point at which only the account IDs of the YAHOO account were different. In particular:
In the problematic .claws-mail YAHOO account had ID=8 (as seen above) and in the "clean" one it had ID=2. Everything else was the same.
So I went on and deleted the YAHOO account from the problematic version and re-added it again, using identical settings. RESULT: no more connection to yahoo. The only change I noticed the config files was that now that account received ID=6.
This made me question: is the number '8' special in some way? So I went on and changed account numbers from '2' to '8' in the clean version (checking each relevant section in the files above) and ran CM again: no connection to yahoo.
I also did the opposite: I changed the account number from '8' to '6' in the initial .claws-mail version. RESULT: there still IS connection to yahoo. I also tried changing '8' to '2' - same, still connection to yahoo.
I remember a few days ago by mistake I did some drag and drop of one or two the accounts in the window which lists all accounts. This resulted in duplicated accounts and I had to delete the duplicates. I am mentioning this because it may in some way be related to the non-sequential account IDs. But still I can't find what the exact mathematical or logical relationship may be.
So this is totally mysterious to me.
(In reply to comment #13)
> So this is totally mysterious to me.
Assuming you also had disabled automatic mail checking and enabled the 'Open last opened folder at start-up', which you didn't mention in your initial description, I've followed your steps and no connection was made to the disabled server.
The only explanation I can found is that you may have inadvertently left open a folder of the second (disabled) account instead of first, which obviously makes it to connect.
Thanks for testing Ricardo. The explanation you give may sound logical and I would think about it myself as a side observer but it is not the actuality. The last selected folder has always been in the GMAIL account as previously mentioned.
I actually found that CM makes a connection even to a third account (when using my actual setup which has more than 2 accounts). So I have been digging further into this because I really wanted to see what was causing it. And I think/hope I have nailed it.
The things which result in connection related to the secondary (YAHOO) account:
1. Not having the file:
2. Having any rule(s) in the [preglobal] section of
Any or both of this result in a connection to the YAHOO account on program start.
FWIW: this is not information which the debug log would show directly or explicitly. I found the first factor by running diff on 2 different debug logs which simply showed me that .claws_cache was not found but there was no direct feedback explaining that *because* of that a connection is made. I just guessed it through the sequence of things and was able to confirm it only after more meticulous testing. For the second factor I found no mention in the debug log whatsoever.
Please consider reopening this ticket to investigate this further because it seems related to program logic, not to settings per se.
Another thing I found during testing: right after setting up the secondary account (YAHOO) a connection to yahoo is made immediately after completing the configuration without clicking "Get mail" or anything explicitly requiring connectivity. I don't know if that is a bug or deliberate logic but you may want to check it too because ideally in a system in which the user (and not the program) is in control the connection should be initiated not upon account setup but when the user explicitly requests it (e.g. on "Get mail", folder select/refresh etc).
Add to 2:
Having rules in [postglobal] also results in connections.
Please kindly review this bug report. It was unfairly marked as invalid.
There was no unfairness that I could see.
Here has been given a factual evidence of a bug, investigated down to the exact cause for it. If you are closing it as invalid you are trying to ignore an actual fact (perhaps without even testing it).
It is a valid issue. The existence of a pre/post-processing rule should not lead to unnecessary connections to mail servers when they were not requested, especially on startup, especially if "Check for new mail on startup" is off.
Anyway I am not looking for an argument. It is your software and it is up to you if you want to improve it and how you (dis)regard bug reports.
(In reply to comment #19)
> Here has been given a factual evidence of a bug, investigated down to the
> exact cause for it. If you are closing it as invalid you are trying to
> ignore an actual fact (perhaps without even testing it).
It was tested, by at least 3 different people.
> It was tested, by at least 3 different people.
The bug was marked as invalid before Comment #15. There is no indication that anyone has tested what was described in #15 and #16. If anyone has taken 10 minutes to do it I am sure he would be able to confirm the issue because the test starts with an empty ~/.claws-mail
I have tested and I can confirm this is a bug. No connections should be made without explicit request or simply due to existence of pre/post filters.
(In reply to comment #22)
> I have tested and I can confirm this is a bug. No connections should be made
> without explicit request or simply due to existence of pre/post filters.
I tested and could not reproduce it.
What are your pre- and post-processing rules? Note that processing rules only get run on entering a folder or by manual use of a folder's processing rules via the folder context menu.
If you are able to reproduce it, perhaps you can attach the debug log?
I see the OP has shared a debug log. Why is that not enough?
(In reply to comment #24)
> I see the OP has shared a debug log. Why is that not enough?
I know, but if you look at it you will see it is quite useless. I'm not asking just for fun.
I've been curious enough as to give it a try, too, but couldn't reproduce it so far.
What are the exact steps for reproducing it after a clean "rm -rf ~/.claws-mail"?
The steps are:
1. rm -rf ~/.claws-mail
2. The STR from the OP - you won't see a connection to the YAHOO account
3. Add some pre or post filtering rule. Example:
enabled rulename "testrule" unread color 1
4. Restart CM - you will see a connection to the YAHOO account
I can reproduce this, but I get connection to both accounts, not just the second one. It is because Claws Mail applies these global rules to all folders it knows about during startup by calling this from main():
folder_func_to_all_folders(initial_processing, (gpointer *)mainwin);
This has been the case for many years, possibly even before Claws Mail was originally forked from Sylpheed, so I'm not sure I would categorize this as bug. Maybe we just need better user control about what happens on startup.
Perhaps an "Apply processing rules on startup" checkbox, either global or per-account (or maybe even per-folder) ?
Created attachment 1865 [details]
console output of STR
Here is how I test in order to isolate the issue. After creating the 2 accounts (GMAIL and YAHOO) I delete everything except the files shown in the output of the tree command and back this up in a dir named TEST. The clawsrc file is the default one which CM creates. Then I run the commands shown.
Using the same procedure anyone should be able to generate one's own debug logs with full details.
> I can reproduce this, but I get connection to both accounts, not just the second one.
Yes, same here. Thanks for confirming.
> so I'm not sure I would categorize this as bug.
Perhaps it can be categorized as a potential privacy issue. If one starts the program one doesn't need to inform all the IMAP servers "Hey, I started my mail client". One may simply want to read RSS without connecting to any IMAP accounts etc.
> Perhaps an "Apply processing rules on startup" checkbox, either global or per-account (or maybe even per-folder) ?
Per-account and per-folder sounds like an overkill. Also while a global option may sound as a good idea it is somewhat "fighting" with the other options like "Check for mail at startup" and not checking a particular account. IOW: there already are options what to check and when so it would make more sense the program logic to follow those preferences rather than initiate connections on its own.
Thinking about all this from a privacy perspective: it would be good to have an option to disable the "NOOP" commands sent every now and then without going offline. If mail should be checked for example every 30 minutes, there is hardly any point to tell the server "Well, I am not checking my mail but you should know that I am online". All this background chattering only helps the companies to create better analytics and profile us more and more deeply. Perhaps this needs a separate bug report?
(In reply to comment #30)
> Per-account and per-folder sounds like an overkill. Also while a global
> option may sound as a good idea it is somewhat "fighting" with the other
> options like "Check for mail at startup" and not checking a particular
> account. IOW: there already are options what to check and when so it would
> make more sense the program logic to follow those preferences rather than
> initiate connections on its own.
Fair enough, that was just the first idea that popped into my mind. :) Maybe it would be simpler just to skip IMAP (or any other remote) folders for this "initial processing" if appropriate.
> Thinking about all this from a privacy perspective: it would be good to have
> an option to disable the "NOOP" commands sent every now and then without
> going offline. If mail should be checked for example every 30 minutes, there
> is hardly any point to tell the server "Well, I am not checking my mail but
> you should know that I am online". All this background chattering only helps
> the companies to create better analytics and profile us more and more
> deeply. Perhaps this needs a separate bug report?
Yes, this is an entirely separate topic, just like your mention elsewhere about optionally not connecting to a newly added IMAP account.
> Yes, this is an entirely separate topic...
OK. Added bug reports for both things: 4008 and 4009.
Now that this is a confirmed and reproduced issue, can someone please consider reopening this bug and hopefully fixing it? After so many hours of testing to find the issue it would be a pity to leave it to sink.
Changes related to this bug have been committed.
Please check latest Git and update the bug accordingly.
You can also get the patch from:
++ ChangeLog 2018-04-17 21:23:03.627927074 +0200
Merge: 846f511 a29676e
Author: Colin Leroy <email@example.com>
Date: Tue Apr 17 21:23:02 2018 +0200
Merge branch 'master' of file:///home/git/claws
Author: Andrej Kacian <firstname.lastname@example.org>
Date: Tue Apr 17 21:17:34 2018 +0200
Fix unwanted IMAP connections on startup caused by processing rules.
The culprit was a combination of folder_item_prefs_clear()
setting item->enable_processing to TRUE even though default
value should be FALSE, and initial_processing() not skipping
over root folders, which do not have this variable later set
to correct default value in xml_to_folder_item().
Closes bug #3993: Claws Mail connects to IMAP server when it should not
*poof*, bug fixed.
*** Bug 4048 has been marked as a duplicate of this bug. ***