Bug 3049 - Claws fetches all messages instead of only new & doesn't rely on imapcache
Summary: Claws fetches all messages instead of only new & doesn't rely on imapcache
Status: RESOLVED INVALID
Alias: None
Product: Claws Mail (GTK 2)
Classification: Unclassified
Component: Folders/IMAP (show other bugs)
Version: 3.9.3
Hardware: PC Linux
: P3 normal
Assignee: users
URL:
Depends on:
Blocks:
 
Reported: 2014-01-06 01:35 UTC by Pranesh Prakash
Modified: 2015-05-29 10:31 UTC (History)
2 users (show)

See Also:


Attachments
relevant debug log (37.68 KB, text/plain)
2014-01-06 06:26 UTC, Pranesh Prakash
no flags Details

Description Pranesh Prakash 2014-01-06 01:35:17 UTC
## Summary ##
Even though my entire email account has been cached (synchronised offline), each time I open a folder (via IMAP) in Claws, it goes through the process of re-downloading all headers and message bodies from the remote server, meaning it often has to download hundreds of MB before it even shows the message list.

## Background ##
I've recently shifted to using Claws on Ubuntu, having installed it from the PPA (version 3.9.3).

I connect to my organization's mail through IMAP (say, imaps://mail.example.org).  To create a local cache of all my email, I did the following:

1. Created a local user with the same name and password as my organizational email account
2. Downloaded a copy of all mails using OfflineIMAP, pointed Dovecot at it
3. Modified /etc/hosts to redirect mail.example.org to 127.0.0.1
4. Loaded Claws, configured it to connect to mail.example.org.  I tried opening the various folders to see the message list.  It took 5 seconds to 3 minutes to open any folder depending on its size.  (I've folders with anywhere from 10 messages to 21000 messages.)
5. Went to "Properties for folder #imap/[example]/INBOX" and checked "Synchronise for offline use" and checked "apply to subfolders", with "fetch message bodies for 0 days".
6. I clicked on File > Synchronize folders, and waited for a couple of hours.
7. I checked ~/.claws-mail/imapcache/mail.example.org/[username] and that is now 10.4 GB, meaning it has cached all my mails.
8. I tested this by opening Claws and browsing through the folders.  The message list takes just as long to open as was the case in step 4.
9. I clicked on "Offline mode".  Every folder opens up instantly as does every message. (I clicked "No" when asked if I wanted to temporarily go online.)

10. With trepidation, I modified /etc/hosts to remove the redirection from mail.example.org.
11. I opened Claws, and it takes ages to open any folder with more than 10 messages.  It took me around 5 minutes to open a 50 MB folder with around 300 messages.  While this is happening, I can't do anything else, and have to wait.

I went through the various IMAP-related bugs here, but couldn't find one that was appropriate.  I did however find this, which describes my bug:
https://bugs.launchpad.net/ubuntu/+source/claws-mail/+bug/221503

I've gone through http://www.claws-mail.org/faq/index.php/General_Information#Why_does_Claws_Mail_load_big_folders_slowly.3F and ensured that all of the options that slow things down are turned off.

Is the above behaviour usual / expected?  If so, how

Is this the expected behaviour?  Is there an option to change this behaviour?  If not, how do people use IMAP folders with more than 50 mails, and what exactly is the purpose of imapcache?
Comment 1 Pranesh Prakash 2014-01-06 06:26:22 UTC
Created attachment 1315 [details]
relevant debug log
Comment 2 Pranesh Prakash 2014-01-06 06:33:14 UTC
Comment on attachment 1315 [details]
relevant debug log

More debugging:

I added another account (my university's Exchange server, to which I connect through IMAP).  This has ~ 1.3K messages total (compared to ~ 120K in my work account), which the largest folder having 882 mail.

In this account, everything works perfectly, with synchronising creating an offline copy that is instantaneously accessible, being loaded from my local cache.

On the other (work) account, even the folders with 5 mails are loaded from the remote server, and hence take time.

The work account (both the actual remote server and my 'fake remote' localhost server) use Dovecot, while the university uses Exchange.

I ran $ dbg claws-mail, then selected two small folders on my work account (2 messages and 4 messages), then selected two folders on my university account (9 messages and 8 messages), and then quit the program.

I've attached the relevant bit of the debug log, changing the server addresses and user name.

Interestingly, it keeps saying "folder.c:4628:Folder [work] wants sync" over and over again.  Could that be it?
Comment 3 Pranesh Prakash 2014-01-06 06:47:57 UTC
(In reply to comment #2)
> Interestingly, it keeps saying "folder.c:4628:Folder [work] wants sync" over
> and over again.  Could that be it?

More debugging:
1. An incorrect "CHANGED (status) scan_required"
2. Then: 
imap.c:4508:Freeing imap uid cache
imap.c:3581:Deleting all cached messages...

=====

folderview.c:2157:Folder INBOX/Archives/Admin/Tech is selected
folderview.c:2173:Opening folder INBOX/Archives/Admin/Tech...
imap.c:4690:getting session...
imap.c:526:locking session 0x1178020 (0)
folder.c:4628:Folder CIS wants sync
folder.c:4628:Folder CIS wants sync
imap.c:3658:using separator: .
imap-thread.c:1015:imap status - begin
imap-thread.c:388:found imap 0xa58c00
imap-thread.c:388:found imap 0xa58c00
[01:39:03] IMAP4> 36 STATUS "INBOX.Archives.Admin.Tech" (MESSAGES UIDNEXT UIDVALIDITY UNSEEN) 
[01:39:03] IMAP4< * STATUS "INBOX.Archives.Admin.Tech" (MESSAGES 214 UIDNEXT 216 UIDVALIDITY 1313572813 UNSEEN 0) 
[01:39:03] IMAP4< 36 OK Status completed. 
imap-thread.c:1004:imap status run - end 0
imap-thread.c:404:generic_cb
imap-thread.c:1044:imap status - end
imap.c:4738:exists 214, item->item.total_msgs 214
imap.c:4744:CHANGED (status)! scan_required
imap.c:539:unlocking session 0x1178020
folder.c:4628:Folder CIS wants sync
folder.c:4628:Folder CIS wants sync
folder.c:2164:Scanning folder INBOX/Archives/Admin/Tech for cache changes.
imap.c:4463:get_num_list
imap.c:4491:getting session...
imap.c:526:locking session 0x1178020 (0)
folder.c:4628:Folder CIS wants sync
folder.c:4628:Folder CIS wants sync
imap.c:4507:get_num_list: trashing num list
imap.c:4508:Freeing imap uid cache
imap.c:3581:Deleting all cached messages...
imap.c:3588:done.
imap.c:3658:using separator: .
imap-thread.c:1375:imap select - begin
imap-thread.c:388:found imap 0xa58c00
imap-thread.c:388:found imap 0xa58c00
[01:39:03] IMAP4> 37 SELECT "INBOX.Archives.Admin.Tech" 
[01:39:04] IMAP4< * OK [CLOSED] Previous mailbox closed. 
[01:39:04] IMAP4< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded contact) 
[01:39:04] IMAP4< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded contact \*)] Flags permitted. 
[01:39:04] IMAP4< * 214 EXISTS 
[01:39:04] IMAP4< * 0 RECENT 
[01:39:04] IMAP4< * OK [UIDVALIDITY 1313572813] UIDs valid 
[01:39:04] IMAP4< * OK [UIDNEXT 216] Predicted next UID 
[01:39:04] IMAP4< * OK [HIGHESTMODSEQ 78] Highest 
[01:39:04] IMAP4< 37 OK [READ-WRITE] Select completed. 
imap-thread.c:1363:imap select run - end 0
imap-thread.c:404:generic_cb
imap-thread.c:1449:imap select - end
imap.c:3822:select: exists 214 recent 0 expunge 0 uid_validity 1313572813 can_create_flags 1
imap-thread.c:1678:imap search - begin
imap-thread.c:388:found imap 0xa58c00
imap-thread.c:388:found imap 0xa58c00
[01:39:04] IMAP4> 38 UID SEARCH ALL 
[01:39:04] IMAP4< * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21[... - 693 bytes more]
[01:39:04] IMAP4< 38 OK Search completed (0.000 secs). 
imap-thread.c:1667:imap search run - end 0
imap-thread.c:404:generic_cb
imap-thread.c:1692:imap search - end
imap.c:539:unlocking session 0x1178020
folder.c:4628:Folder CIS wants sync
folder.c:4628:Folder CIS wants sync
imap.c:4522:get_num_list: got 214 msgs
imap.c:4532:removing old messages from
Comment 4 users 2014-09-28 09:31:03 UTC
Changes related to this bug have been committed.
Please check latest Git and update the bug accordingly.
You can also get the patch from:
http://git.claws-mail.org/

++ ChangeLog	2014-09-28 11:31:02.850392047 +0200
http://git.claws-mail.org/?p=claws.git;a=commitdiff;h=a8efd865dbf93bdf3e68ec545b78ebef7dd8b093
Merge: 07a9444 72d7c35
Author: Colin Leroy <colin@colino.net>
Date:   Sun Sep 28 11:31:02 2014 +0200

    Merge branch 'master' of file:///home/git/claws

http://git.claws-mail.org/?p=claws.git;a=commitdiff;h=72d7c35f0abb53675f0d02d5094adaf00481f194
Author: Paul <paul@claws-mail.org>
Date:   Sun Sep 28 10:29:56 2014 +0100

    add more debug_prints in relation to bug 3049
Comment 5 Colin Leroy 2014-09-29 08:14:39 UTC
Hi,

Message bodies are apparently not re-downloaded according to the first debug log.

Flags are being re-fetched, which is expected because they need to be checked to make sure they in sync locally. 
You may have more performance unchecking "Bandwidth-efficient mode" in the account's Receive preferences.

In the other log from comment #3, I believe the UIDVALIDITY of the folder changed, which means we can't trust the local cache even if all counts are the same. Paul's extra debug patch will probably make it clear.

"Folder X wants sync" is the debug line for "folder is marked for offline use and must be synchronized". It does trigger a download of every uncached email and in the case of an UIDVALIDITY change, it does indeed trigger a full re-download.

Can you test with "Bandwidth-efficient mode" unchecked?
Comment 6 Ricardo Mones 2015-05-29 10:31:29 UTC
Eight months without feedback about unchecked bandwidth-efficient mode, closing.

Note You need to log in before you can comment on or make changes to this bug.