Did you try using the 'Update all local folders' option in the 'After receiving new mail...' frame of the Mail Handling/Receiving page of the Prefs, and the 'Scan for new mail' option on the General page of the Folder Properties?

(Forget the cachesaver plugin, it's more-or-less a relic)

Dear Paul,

Thank you for responding so quickly.

I did a little testing on this issue and yesterday I was confident that
enabling 'Update all local folders' and enabling 'Scan for new mail' (which
was on previously anyway) had done the trick. However I left CM running
overnight and it lost sync again. It managed to display a completely
different message when I clicked on the subject line of a new message so
it's back to square one. The cachesaver plugin remains unloaded.

This set me thinking about why this might happen, and after a little
bit of investigation I think I have found something quite interesting. It all
seems to come down to the order in which changes to CM and the mailbox happen.
Let me explain.

If I have a box called Deposits into which procmail drops new messages
and another box called Irrelevant and I perform the following operations
in the order below I can show what is going on. I've used two columns to
show this more clearly.

This sequence in this column works        This sequence in this column doesn't
------------------------------------------------------------------------------

Press Inbox                               Press Inbox

Deposit a message via procmail            Deposit a message via procmail

Press Check for new messages,             Press Check for new messages,
which shows (1)                           which shows (1)

Press Irrelevant                          Press Irrelevant

Press Inbox(1)                            Press Inbox(1)

Deposit a message via procmail            Delete the new message in CM

Delete the new message in CM              Deposit a message via procmail            

Press Check for new messages,             Press Check for new messages,
which shows (1)                           which shows (2)


What matters is the relative timing of deletions and deposits. Unlike
those CM users who collect their mail via IMAP (or some such) I have no
control over when deposits are made, which is perhaps why this bug has
remained unreported until now.

It makes no difference whether 'Update all local folders' is selected or not.

It seems to me that the mailbox merging code works just fine but there is
a deficiency in a check before merging which cannot cope with the notion that
a message might be added to a mailbox after a change has been made in CM's
cache.

I wonder whether this is just true of mailmbox or whether it applies to
MH as well.

The bug also affects CM 3.3.0 with CM plugins 3.3.0 and libetpan 0.54.

I have finally diagnosed the problem, or so I believe. It goes like this:

CM keeps a 'cache' of the messages in the mailbox file. It keeps two sorts of caches: a set of flags for each message indicating such things as whether a message is unread, and; a duplicate of the content of each message.

When another programme alters the mailbox file CM must update its cache to reflect the changes to the mailbox. This is where the problems set in. The problem is not that CM keeps a cache of the messages per-se, but that it keeps a cache of the /content/ of the messages.

Each message, no matter where it originated from, or how easily accessible it is, is stored in its own separate numbered cache file. For a single locally-accessible mbox file containing 10,000 messages, the filesystem must cope with 10,001 files, 10,000 of which are verbatim copies of the original data. Apart from the obvious inefficiencies of this approach (think I/O time, the time spent syncing directory operations and the time taken to dump the filesystem) having 10,000 cached copies of slices of the mailbox is asking for things to slip out of sync.

The basic problem is that once a message is deleted from a folder, the 'flag cache' (the .claws_* files) is updated (first in memory, and later on disc), and (with my expunge patch) so is the mailbox itself, but the cache file is not deleted. Seen in isolation this is not a problem, but at a later date when a new message is plonked into the mailbox by a proper, old-school programme, CM makes a mess of updating its caches.

If CM concludes that a new message deposited into the mailbox by a proper programme is to be given the same message number as a message which used to be in the mailbox file, the flags cache will be updated correctly, but s_claws_mailmbox_fetch_msg() will conclude that the old, undeleted cache file from the earlier message is a perfectly satisfactory cached copy of the new message. It is this which results in CM displaying the wrong content for a message.

How likely is this scenario? In my experience with over a million messages stored in mbox files and a turn-over of perhaps 1000-2000 messages per day across several folders, it happens several times every day. It happens particularly often when deleting or moving messages which have been delivered only recently. My solution up until now has been to stop CM and blow away the entire cache (apart from the .claws_mark files) every time this happens, but some respite can be gained by blowing away the requisite files containing the cached content whilst CM is running.

So, how to fix it. At first I was tempted to delete the cached content files when the real messages are deleted, but this is complicated somewhat by the message moving code. Someone more familiar with this code should look into it before deleting these files.

The easiest route to take at this stage is simply to delete the cached files in s_claws_mailmbox_fetch_msg() before re-creating them lower down in the function. Although this is not efficient, it is at least reliable. Simply replacing

        if (is_file_exist(file)) {
             ....
        }

with

        /* Bug 1434
         */
        unlink(file);

does the trick.

Then we need to convince CM to actually delete the emails from the mailbox when it is told to do so. We do this by calling claws_mailmbox_expunge() when removing a message, like so:

static gint claws_mailmbox_remove_msg(Folder *folder, FolderItem *item, gintnum)
{
        struct claws_mailmbox_folder * mbox;
        int r;

        g_return_val_if_fail(item != NULL, -1);
        mbox = get_mbox(item, 0);
        if (mbox == NULL)
                return -1;
        r = claws_mailmbox_delete_msg(mbox, num);

        /* Bug 1434
         */
        claws_mailmbox_expunge(mbox);

        if (r != MAILMBOX_NO_ERROR)
                return -1;
        return 0;
}

This keeps the mailbox file in sync with the GUI. It is vital to do this or CM will mis-recognise the messages in the mailbox themselves and then corrupt its flags cache as well. This can cause CM to delete different messages from a mailbox to the ones you intend. This same call must be added to the end of claws_mailmbox_remove_all_msg() as well.

One further bug in the same vein is the fact that CM can't delete a message from an mbox file containing just one message. This in itself causes messages to be resurected. The fix is inside claws_mailmbox_expunge_to_file_no_lock() as follows....

  r = ftruncate(dest_fd, size);
  if (r < 0) {
    res = MAILMBOX_ERROR_FILE;
    goto err;
  }

    /* Bug 1434.
     */
    if( size==0 )
    {
        goto good;
    }

  dest = (char *) mmap(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, dest_fd, 0);
  if (dest == MAP_FAILED) {
    res = MAILMBOX_ERROR_FILE;
    goto err;
  }

....followed by....

  fflush(stdout);

  msync(dest, size, MS_SYNC);
  munmap(dest, size);

    /* Bug 1434.
     */
good:
    fflush(stdout);
  * result_size = size;
  return MAILMBOX_NO_ERROR;

 err:
  return res;
}

It comes about because it is not possible to mmap() a zero-length block of memory without error.

These fixes are mature and are ported to 3.7.4.