maildir-sync.c revision 538303a216166f3526c0ae9658c9978275cfa100
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen/* Copyright (C) 2004 Timo Sirainen */
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Here's a description of how we handle Maildir synchronization and
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen it's problems:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen We want to be as efficient as we can. The most efficient way to
137ea7ca34005345aa2304a940149b7f3774d727Timo Sirainen check if changes have occurred is to stat() the new/ and cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen directories and uidlist file - if their mtimes haven't changed,
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen there's no changes and we don't need to do anything.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Problem 1: Multiple changes can happen within a single second -
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen nothing guarantees that once we synced it, someone else didn't just
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen then make a modification. Such modifications wouldn't get noticed
137ea7ca34005345aa2304a940149b7f3774d727Timo Sirainen until a new modification occurred later.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Problem 2: Syncing cur/ directory is much more costly than syncing
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen new/. Moving mails from new/ to cur/ will always change mtime of
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen cur/ causing us to sync it as well.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Problem 3: We may not be able to move mail from new/ to cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen because we're out of quota, or simply because we're accessing a
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen read-only mailbox.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen MAILDIR_SYNC_SECS
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen -----------------
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Several checks below use MAILDIR_SYNC_SECS, which should be maximum
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen clock drift between all computers accessing the maildir (eg. via
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen NFS), rounded up to next second. Our default is 1 second, since
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen everyone should be using NTP.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Note that setting it to 0 works only if there's only one computer
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen accessing the maildir. It's practically impossible to make two
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen clocks _exactly_ synchronized.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen It might be possible to only use file server's clock by looking at
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen the atime field, but I don't know how well that would actually work.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen cur directory
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen -------------
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen We have dirty_cur_time variable which is set to cur/ directory's
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen synchronized the directory.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen When dirty_cur_time is non-zero, we don't synchronize the cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen directory until
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen a) cur/'s mtime changes
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen b) opening a mail fails with ENOENT
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen c) time() > dirty_cur_time + MAILDIR_SYNC_SECS
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen This allows us to modify the maildir multiple times without having
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen to sync it at every change. The sync will eventually be done to
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen make sure we didn't miss any external changes.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen The dirty_cur_time is set when:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - we change message flags
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - we expunge messages
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - we move mail from new/ to cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen It's unset when we do the final syncing, ie. when mtime is
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen older than time() - MAILDIR_SYNC_SECS.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen new directory
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen -------------
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen it. dirty_cur_time-like feature might save us a few syncs, but
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen that might break a client which saves a mail in one connection and
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen tries to fetch it in another one. new/ directory is almost always
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen empty, so syncing it should be very fast anyway. Actually this can
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen still happen if we sync only new/ dir while another client is also
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen moving mails from it to cur/ - it takes us a while to see them.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen That's pretty unlikely to happen however, and only way to fix it
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen would be to always synchronize cur/ after new/.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Normally we move all mails from new/ to cur/ whenever we sync it. If
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen it's not possible for some reason, we mark the mail with "probably
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen exists in new/ directory" flag.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen If rename() still fails because of ENOSPC or EDQUOT, we still save
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen the flag changes in index with dirty-flag on. When moving the mail
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen to cur/ directory, or when we notice it's already moved there, we
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen apply the flag changes to the filename, rename it and remove the
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen dirty flag. If there's dirty flags, this should be tried every time
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen after expunge or when closing the mailbox.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen This file contains UID <-> filename mappings. It's updated only when
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen new mail arrives, so it may contain filenames that have already been
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen deleted. Updating is done by getting uidlist.lock file, writing the
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen whole uidlist into it and rename()ing it over the old uidlist. This
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen means there's no need to lock the file for reading.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Whenever uidlist is rewritten, it's mtime must be larger than the old
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen one's. Use utime() before rename() if needed. Note that inode checking
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen wouldn't have been sufficient as inode numbers can be reused.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen This file is usually read the first time you need to know filename for
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen given UID. After that it's not re-read unless new mails come that we
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen don't know about.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen broken clients
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen --------------
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Originally the middle identifier in Maildir filename was specified
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen only as <process id>_<delivery counter>. That however created a
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen problem with randomized PIDs which made it possible that the same
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen PID was reused within one second.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen So if within one second a mail was delivered, MUA moved it to cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen and another mail was delivered by a new process using same PID as
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen the first one, we likely ended up overwriting the first mail when
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen the second mail was moved over it.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Nowadays everyone should be giving a bit more specific identifier,
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen for example include microseconds in it which Dovecot does.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen There's a simple way to prevent this from happening in some cases:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Don't move the mail from new/ to cur/ if it's mtime is >= time() -
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen MAILDIR_SYNC_SECS. The second delivery's link() call then fails
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen because the file is already in new/, and it will then use a
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen different filename. There's a few problems with this however:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - it requires extra stat() call which is unneeded extra I/O
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - another MUA might still move the mail to cur/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen - if first file's flags are modified by either Dovecot or another
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen but that'd be ugly)
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Because this is useful only for very few people and it requires
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen extra I/O, I decided not to implement this. It should be however
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen quite easy to do since we need to be able to deal with files in new/
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen It's also possible to never accidentally overwrite a mail by using
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen link() + unlink() rather than rename(). This however isn't very
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen good idea as it introduces potential race conditions when multiple
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen clients are accessing the mailbox:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen Trying to move the same mail from new/ to cur/ at the same time:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen a) Client 1 uses slightly different filename than client 2,
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen for example one sets read-flag on but the other doesn't.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen You have the same mail duplicated now.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen b) Client 3 sees the mail between Client 1's and 2's link() calls
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen and changes it's flag. You have the same mail duplicated now.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen And it gets worse when they're unlink()ing in cur/ directory:
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen c) Client 1 changes mails's flag and client 2 changes it back
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen between 1's link() and unlink(). The mail is now expunged.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen d) If you try to deal with the duplicates by unlink()ing another
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen one of them, you might end up unlinking both of them.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen So, what should we do then if we notice a duplicate? First of all,
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen it might not be a duplicate at all, readdir() might have just
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen returned it twice because it was just renamed. What we should do is
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen create a completely new base name for it and rename() it to that.
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen If the call fails with ENOENT, it only means that it wasn't a
0c17af9d3f9323136a94e66605776ed4462a172dTimo Sirainen duplicate after all.
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen/* When rename()ing many files from new/ to cur/, it's possible that next
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen readdir() skips some files. we don't of course wish to lose them, so we
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen go and rescan the new/ directory again from beginning until no files are
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen left. This value is just an optimization to avoid checking the directory
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen twice unneededly. usually only NFS is the problem case. 1 is the safest
b66a7b7ab0db2c9ad425912d3f21a36fcf76d876Timo Sirainen bet here, but I guess 5 will do just fine too. */
d3eff05aaa4c2bc0a7580ee87a54f6693f4a8241Timo Sirainen/* After moving 100 mails from new/ to cur/, check if we need to touch the
d3eff05aaa4c2bc0a7580ee87a54f6693f4a8241Timo Sirainen uidlist lock. */
d3eff05aaa4c2bc0a7580ee87a54f6693f4a8241Timo Sirainen/* readdir() should be pretty fast to do, but check anyway every 10000 mails
d3eff05aaa4c2bc0a7580ee87a54f6693f4a8241Timo Sirainen to see if we need to touch the uidlist lock. */
cbf7138b49d32fce0645dc6523fbb42cc07cb2faTimo Sirainen/* This is mostly to avoid infinite looping when rename() destination already
cbf7138b49d32fce0645dc6523fbb42cc07cb2faTimo Sirainen exists as the hard link of the file itself. */
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen struct maildir_uidlist_sync_ctx *uidlist_sync_ctx;
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen struct maildir_index_sync_context *index_sync_ctx;
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen struct maildir_sync_context *maildir_sync_ctx;
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen struct maildir_keywords_sync_ctx *keywords_sync_ctx;
8d80659e504ffb34bb0c6a633184fece35751b18Timo Sirainen ARRAY_DEFINE(sync_recs, struct mail_index_sync_rec);
e1ca7af110ea6eeb6303bdd8f07c172b11dff2faTimo Sirainenmaildir_sync_get_keywords_sync_ctx(struct maildir_index_sync_context *ctx)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainenstatic int maildir_expunge(struct maildir_mailbox *mbox, const char *path,
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen mail_storage_set_critical(&mbox->storage->storage,
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainenstatic int maildir_sync_flags(struct maildir_mailbox *mbox, const char *path,
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen unsigned int i, count;
8fd00f8716a931e84f33e29c8d3c579ebd718411Timo Sirainen maildir_filename_get_flags(ctx->keywords_sync_ctx,
8d80659e504ffb34bb0c6a633184fece35751b18Timo Sirainen recs = array_get_modifiable(&ctx->sync_recs, &count);
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen for (i = 0; i < count; i++) {
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen mail_index_sync_flags_apply(&recs[i], &flags8);
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen mail_index_sync_keywords_apply(&recs[i], &keywords);
b142deb9a831c89b1bb9129ada655f3e56b9d4ccTimo Sirainen newfname = maildir_filename_set_flags(ctx->keywords_sync_ctx,
b92813e2f96d4b28f989528ed5dd6115da7d9bdbTimo Sirainen if ((flags8 & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0)
b35f7104715edee0cfac6d46ab0b342033867eb7Timo Sirainen mail_index_update_flags(ctx->trans, ctx->seq, MODIFY_ADD,
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen mail_storage_set_critical(&mbox->storage->storage,
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainenmaildir_sync_check_timeouts(struct maildir_sync_context *ctx, bool move)
6bf1543bb7af03324c04e8f9ac8e430f395989aeTimo Sirainen /* we got here from maildir-save.c. it has no
6bf1543bb7af03324c04e8f9ac8e430f395989aeTimo Sirainen maildir_sync_context, */
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen if ((ctx->move_count % MAILDIR_SLOW_MOVE_COUNT) != 0)
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen if ((ctx->check_count % MAILDIR_SLOW_CHECK_COUNT) != 0)
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen if (now - ctx->last_touch > MAILDIR_LOCK_TOUCH_SECS) {
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen (void)maildir_uidlist_lock_touch(ctx->mbox->uidlist);
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen if (now - ctx->last_notify > MAIL_STORAGE_STAYALIVE_SECS) {
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen if (box->storage->callbacks->notify_ok != NULL) {
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainenmaildir_sync_record_commit_until(struct maildir_index_sync_context *ctx,
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen unsigned int i, count;
8d80659e504ffb34bb0c6a633184fece35751b18Timo Sirainen recs = array_get_modifiable(&ctx->sync_recs, &count);
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen for (seq = recs[0].uid1; seq <= last_seq; seq++) {
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen for (i = 0; i < count; i++) {
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen if (mail_index_lookup_uid(ctx->view, seq, &uid) < 0) {
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen mail_storage_set_index_error(&ctx->mbox->ibox);
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen maildir_sync_check_timeouts(ctx->maildir_sync_ctx,
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen maildir_sync_check_timeouts(ctx->maildir_sync_ctx,
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen for (i = count; i > 0; i--) {
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen /* all sync_recs committed */
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainenstatic int maildir_sync_record(struct maildir_index_sync_context *ctx,
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen if (sync_rec->type == MAIL_INDEX_SYNC_TYPE_APPEND)
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen return 0; /* ignore */
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen /* convert to sequences to avoid looping through huge holes in
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen if (mail_index_lookup_uid_range(view, sync_rec->uid1,
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen mail_storage_set_index_error(&ctx->mbox->ibox);
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen /* UIDs were expunged */
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen if (maildir_sync_record_commit_until(ctx, sync_copy.uid1-1) < 0)
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainenstatic int maildir_sync_index_records(struct maildir_index_sync_context *ctx)
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen ret = mail_index_sync_next(ctx->sync_ctx, &sync_rec);
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen mail_storage_set_index_error(&ctx->mbox->ibox);
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen ret = mail_index_sync_next(ctx->sync_ctx, &sync_rec);
7fb7365a8fad104a17538a73c338ee3d3420e7b0Timo Sirainen } while (ret > 0);
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen mail_storage_set_index_error(&ctx->mbox->ibox);
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainenmaildir_sync_context_new(struct maildir_mailbox *mbox)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen ctx->new_dir = t_strconcat(mbox->path, "/new", NULL);
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen ctx->cur_dir = t_strconcat(mbox->path, "/cur", NULL);
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainenstatic void maildir_sync_deinit(struct maildir_sync_context *ctx)
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen (void)maildir_uidlist_sync_deinit(&ctx->uidlist_sync_ctx);
0b3662995e9fa0d0d857ec5350ce2b1ee6d3b94fTimo Sirainen (void)maildir_sync_index_finish(&ctx->index_sync_ctx,
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainenstatic int maildir_fix_duplicate(struct maildir_sync_context *ctx,
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainen maildir_uidlist_sync_get_full_filename(ctx->uidlist_sync_ctx,
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainen existing_path = t_strconcat(dir, "/", existing_fname, NULL);
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen old_path = t_strconcat(dir, "/", old_fname, NULL);
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainen /* most likely the files just don't exist anymore.
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainen don't really care about other errors much. */
cbf7138b49d32fce0645dc6523fbb42cc07cb2faTimo Sirainen /* Files are the same. this means either a race condition
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen between stat() calls, or that the files were link()ed. */
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen if (ex_st.st_nlink > 1 && old_st.st_nlink == ex_st.st_nlink &&
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen ex_st.st_ctime < ioloop_time - DUPE_LINKS_DELETE_SECS) {
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen /* The file has hard links and it hasn't had any
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen changes (such as renames) for a while, so this
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen isn't a race condition.
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen rename()ing one file on top of the other would fix
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen this safely, except POSIX decided that rename()
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen doesn't work that way. So we'll have unlink() one
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen and hope that another process didn't just decide to
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen unlink() the other (uidlist lock prevents this from
992a9e2d6c6ee45d87089ac54267e0198a7802c3Timo Sirainen happening) */
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen new_path = t_strconcat(mbox->path, "/new/", new_fname, NULL);
a87e5f15283e057c7dc26dd9db7b616268c95ca7Timo Sirainen i_warning("Fixed a duplicate: %s -> %s", old_fname, new_fname);
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen mail_storage_set_critical(&mbox->storage->storage,
c93cd163f9c1d4b0ca29f49cbfdbf474caeef5bfTimo Sirainen "Couldn't fix a duplicate: rename(%s, %s) failed: %m",
6ef7e31619edfaa17ed044b45861d106a86191efTimo Sirainenstatic int maildir_scan_dir(struct maildir_sync_context *ctx, bool new_dir)
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen struct mail_storage *storage = &ctx->mbox->storage->storage;
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen const char *dir;
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen move_new = new_dir && !mailbox_is_readonly(&ctx->mbox->ibox.box) &&
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen ret = maildir_uidlist_sync_next_pre(ctx->uidlist_sync_ctx,
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen /* new file and we couldn't lock uidlist, check this
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen later in next sync. */
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen str_printfa(src, "%s/%s", ctx->new_dir, dp->d_name);
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen str_printfa(dest, "%s/%s", ctx->cur_dir, dp->d_name);
8754bb7a1f24705ffa5434f9e10d57e0b3b88d6eTimo Sirainen if (strchr(dp->d_name, MAILDIR_INFO_SEP) == NULL) {
d67f54632110cfb6aafe2d7cd1f99b031c0b208aTimo Sirainen /* we moved it - it's \Recent for us */
641f0c0900ee6e7cf9667f4b40ed95cec7d0cdcaTimo Sirainen /* someone else moved it already */
9a099a65160987349f441c82ab0e38f32b747adbTimo Sirainen } else if (ENOSPACE(errno) || errno == EACCES) {
9a099a65160987349f441c82ab0e38f32b747adbTimo Sirainen /* not enough disk space / read-only maildir,
9a099a65160987349f441c82ab0e38f32b747adbTimo Sirainen leave here */
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen "rename(%s, %s) failed: %m",
641f0c0900ee6e7cf9667f4b40ed95cec7d0cdcaTimo Sirainen } else if (new_dir) {
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen ret = maildir_uidlist_sync_next(ctx->uidlist_sync_ctx,
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen /* possibly duplicate - try fixing it */
4b43f50117630aa12b3cfd0cbd05ae22ba27fec1Timo Sirainen if (maildir_fix_duplicate(ctx, dir, dp->d_name) < 0) {
44dc970b18c4e2d06f34cb908924152156e4a45bTimo Sirainen (ctx->move_count <= MAILDIR_RENAME_RESCAN_COUNT ? 0 : 1);
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainenmaildir_sync_update_from_header(struct maildir_mailbox *mbox,
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen /* open a new view so we get the latest header */
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen view = mail_index_view_open(mbox->ibox.index);
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen /* FIXME: ugly, replace with extension header */
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen mbox->last_new_mtime = hdr->sync_size & 0xffffffff;
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen mbox->last_dirty_flags = (hdr->sync_size >> 32) &
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen if ((mbox->last_dirty_flags & MAILDIR_DIRTY_CUR) != 0 &&
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainenmaildir_sync_quick_check(struct maildir_mailbox *mbox,
463531a5ca96d33072416807442a114c080e3726Timo Sirainen /* mailbox was deleted under us. this isn't the only
463531a5ca96d33072416807442a114c080e3726Timo Sirainen way it can break, but the most common one. */
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen mail_storage_set_critical(&mbox->storage->storage,
ccc895c0358108d2304239063e940b7d75f364abTimo Sirainen mail_storage_set_critical(&mbox->storage->storage,
abbe0657a4d6271740d41cf1de55e8686f5769fcTimo Sirainen /* cur stamp is kept in index, we don't have to sync if
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen someone else has done it and updated the index.
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen FIXME: For now we're using sync_size field as the new/ dir's stamp.
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen Pretty ugly.. */
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen if ((mbox->dirty_cur_time == 0 && cur_mtime != mbox->last_cur_mtime) ||
abbe0657a4d6271740d41cf1de55e8686f5769fcTimo Sirainen /* check if the index has been updated.. */
ad8b3389ab62c5facb3d49037963076e3d32c147Timo Sirainen /* If we're removing recent flags, always sync new/ directory if
ad8b3389ab62c5facb3d49037963076e3d32c147Timo Sirainen it has mails. */
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen ((mbox->last_dirty_flags & MAILDIR_DIRTY_NEW) != 0 &&
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen new_mtime < ioloop_time - MAILDIR_SYNC_SECS) ||
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen (!ibox->keep_recent && hdr.recent_messages_count > 0)) {
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen if (new_mtime < ioloop_time - MAILDIR_SYNC_SECS)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen ioloop_time - mbox->dirty_cur_time > MAILDIR_SYNC_SECS)) {
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen /* cur/ changed, or delayed cur/ check */
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen if (cur_mtime < ioloop_time - MAILDIR_SYNC_SECS) {
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainenint maildir_sync_index_begin(struct maildir_mailbox *mbox,
aa0647f2debf0d48d504a321186f66c85596aaf4Timo Sirainen if (mail_index_sync_begin(mbox->ibox.index, &sync_ctx, &view, &trans,
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen ctx = i_new(struct maildir_index_sync_context, 1);
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen maildir_keywords_sync_init(mbox->keywords, mbox->ibox.index);
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainenint maildir_sync_index_finish(struct maildir_index_sync_context **_sync_ctx,
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen struct maildir_index_sync_context *sync_ctx = *_sync_ctx;
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen struct maildir_mailbox *mbox = sync_ctx->mbox;
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen mail_index_sync_rollback(&sync_ctx->sync_ctx);
947a2df7c6a2c48711b051736e08402f6d91795cTimo Sirainen /* Set syncing_commit=TRUE so that if any sync callbacks try
947a2df7c6a2c48711b051736e08402f6d91795cTimo Sirainen to access mails which got lost (eg. expunge callback trying
947a2df7c6a2c48711b051736e08402f6d91795cTimo Sirainen to open the file which was just unlinked) we don't try to
947a2df7c6a2c48711b051736e08402f6d91795cTimo Sirainen start a second index sync and crash. */
6f66e585998aa88a4b0ccad531d329a103325d57Timo Sirainen if (mail_index_sync_commit(&sync_ctx->sync_ctx) < 0) {
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen maildir_keywords_sync_deinit(sync_ctx->keywords_sync_ctx);
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainenint maildir_sync_index(struct maildir_index_sync_context *sync_ctx,
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen struct maildir_mailbox *mbox = sync_ctx->mbox;
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen struct mail_index_view *view = sync_ctx->view;
aa0647f2debf0d48d504a321186f66c85596aaf4Timo Sirainen struct mail_index_transaction *trans = sync_ctx->trans;
7ba90b7ec9e0a97c01e89e2714ff54e35c5ddfbfTimo Sirainen i_assert(maildir_uidlist_is_locked(sync_ctx->mbox->uidlist));
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen uid_validity = maildir_uidlist_get_uid_validity(mbox->uidlist);
92888ef30960c30ccc9e030fe7eab5d4d04a7d1cTimo Sirainen uid_validity != 0 && hdr->uid_validity != 0) {
e06c0b65c16ccce69bbee009ead14d7d3d17a256Timo Sirainen /* uidvalidity changed and mailbox isn't being initialized,
aa38d1a0945f0bc13a225d043f53fad2eec666b1Timo Sirainen reset mailbox so we can add all messages as new */
f6699a08521aacc4c2bb5b6175691dad5f715f8cTimo Sirainen i_warning("Maildir %s: UIDVALIDITY changed (%u -> %u)",
91dca97b367c54a139c268b56a0c67f564bd9197Timo Sirainen t_array_init(&keywords, MAILDIR_MAX_KEYWORDS);
91dca97b367c54a139c268b56a0c67f564bd9197Timo Sirainen t_array_init(&idx_keywords, MAILDIR_MAX_KEYWORDS);
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen iter = maildir_uidlist_iter_init(mbox->uidlist);
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen while (maildir_uidlist_iter_next(iter, &uid, &uflags, &filename)) {
e1ca7af110ea6eeb6303bdd8f07c172b11dff2faTimo Sirainen maildir_filename_get_flags(sync_ctx->keywords_sync_ctx,
24ddec4ee3571be32f615138b56cf0ae6f922af0Timo Sirainen /* the private flags are kept only in indexes. don't use them
24ddec4ee3571be32f615138b56cf0ae6f922af0Timo Sirainen at all even for newly seen mails */
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RECENT) != 0 &&
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen (uflags & MAILDIR_UIDLIST_REC_FLAG_NEW_DIR) != 0 &&
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen (uflags & MAILDIR_UIDLIST_REC_FLAG_MOVED) == 0) {
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen /* mail is recent for next session as well */
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen /* most likely a race condition: we read the
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen maildir, then someone else expunged messages
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen and committed changes to index. so, this
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen message shouldn't actually exist. mark it
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen racy and check in next sync.
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen the difference between this and the later
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen check is that this one happens when messages
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen are expunged from the end */
a33fde36c57839342dafdf3c8d4b5c009e3ef4bcTimo Sirainen /* partial syncing */
6969f13267ad8495a132ece39d34be36b9883f37Timo Sirainen "Maildir %s sync: "
6969f13267ad8495a132ece39d34be36b9883f37Timo Sirainen "UID < next_uid "
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen "(%u < %u, file = %s)",
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen mail_index_update_flags(trans, seq, MODIFY_REPLACE,
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen /* expunged */
a038139a470d2942759b9b86a9852aee7b460996Timo Sirainen /* most likely a race condition: we read the
a038139a470d2942759b9b86a9852aee7b460996Timo Sirainen maildir, then someone else expunged messages and
a038139a470d2942759b9b86a9852aee7b460996Timo Sirainen committed changes to index. so, this message
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen shouldn't actually exist. mark it racy and check
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen in next sync. */
a33fde36c57839342dafdf3c8d4b5c009e3ef4bcTimo Sirainen /* partial syncing */
316689d0ef55e2fa4e2fb4ac5b1ea35ce65688d3Timo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
0e2686dfe29a18772fa4026bad53e2c7c560403fTimo Sirainen "Maildir %s sync: "
0e2686dfe29a18772fa4026bad53e2c7c560403fTimo Sirainen "UID inserted in the middle of mailbox "
a038139a470d2942759b9b86a9852aee7b460996Timo Sirainen "(%u > %u, file = %s)",
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen maildir_uidlist_add_flags(mbox->uidlist, filename,
24ddec4ee3571be32f615138b56cf0ae6f922af0Timo Sirainen /* the private flags are stored only in indexes, keep them */
24ddec4ee3571be32f615138b56cf0ae6f922af0Timo Sirainen flags |= rec->flags & mbox->private_flags_mask;
a33fde36c57839342dafdf3c8d4b5c009e3ef4bcTimo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
a33fde36c57839342dafdf3c8d4b5c009e3ef4bcTimo Sirainen /* partial syncing */
4d938f46f4f956ecb802c30ca771922f5539a660Timo Sirainen /* we last saw this mail in new/, but it's
4d938f46f4f956ecb802c30ca771922f5539a660Timo Sirainen not there anymore. possibly expunged,
4d938f46f4f956ecb802c30ca771922f5539a660Timo Sirainen make sure. */
fd3d711f219fd6813492acbe051e04327f0ca0f0Timo Sirainen if ((rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0) {
fd3d711f219fd6813492acbe051e04327f0ca0f0Timo Sirainen /* we haven't been able to update maildir with this
fd3d711f219fd6813492acbe051e04327f0ca0f0Timo Sirainen record's flag changes. don't sync them. */
16c89b1260c9d07c01c83a9219424d3727069b2eTimo Sirainen (rec->flags & (MAIL_FLAGS_MASK^MAIL_RECENT))) {
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen /* FIXME: this is wrong if there's pending changes in
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen transaction log already. it gets fixed in next sync
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen mail_index_update_flags(trans, seq, MODIFY_REPLACE,
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen /* just remove recent flag */
d482b35af87f5fd872bad007da0475813a401a49Timo Sirainen mail_index_update_flags(trans, seq, MODIFY_REMOVE,
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen /* update keywords if they have changed */
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen if (mail_index_lookup_keywords(view, seq, &idx_keywords) < 0) {
bb26f09873c18f342cd1ab2d0ee0b9018e6546d9Timo Sirainen if (!index_keyword_array_cmp(&keywords, &idx_keywords)) {
b7b81543899e306c71e6152516d8698416162bcbTimo Sirainen /* expunge the rest */
b7b81543899e306c71e6152516d8698416162bcbTimo Sirainen for (seq++; seq <= hdr->messages_count; seq++)
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen /* next_uid must be updated only in non-partial syncs since
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen partial syncs don't add the new mails to index. also we'll
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen have to do it here before syncing index records, since after
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen that the uidlist's next_uid value may have changed. */
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen next_uid = maildir_uidlist_get_next_uid(mbox->uidlist);
f6699a08521aacc4c2bb5b6175691dad5f715f8cTimo Sirainen if (!mbox->syncing_commit && hdr != &empty_hdr) {
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen /* now, sync the index. NOTE: may recurse back to here with
bd606153e12c04799be8f27933b5c983db0eaa15Timo Sirainen partial syncs */
4b41116563110d00330896a568eff1078c382827Timo Sirainen mbox->ibox.box.v.sync_notify(&mbox->ibox.box, 0, 0);
5fb3bff645380804c9db2510940c41db6b8fdb01Timo Sirainen if (mbox->last_cur_mtime != (time_t)hdr->sync_stamp) {
e06c0b65c16ccce69bbee009ead14d7d3d17a256Timo Sirainen offsetof(struct mail_index_header, sync_stamp),
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen /* FIXME: use a header extension instead of sync_size.. */
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen offsetof(struct mail_index_header, sync_size),
92888ef30960c30ccc9e030fe7eab5d4d04a7d1cTimo Sirainen /* get the initial uidvalidity */
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen if (maildir_uidlist_update(mbox->uidlist) < 0)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen uid_validity = maildir_uidlist_get_uid_validity(mbox->uidlist);
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen maildir_uidlist_set_uid_validity(mbox->uidlist,
92888ef30960c30ccc9e030fe7eab5d4d04a7d1cTimo Sirainen } else if (uid_validity == 0) {
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen maildir_uidlist_set_uid_validity(mbox->uidlist,
92888ef30960c30ccc9e030fe7eab5d4d04a7d1cTimo Sirainen if (uid_validity != hdr->uid_validity && uid_validity != 0) {
e06c0b65c16ccce69bbee009ead14d7d3d17a256Timo Sirainen offsetof(struct mail_index_header, uid_validity),
6ef7e31619edfaa17ed044b45861d106a86191efTimo Sirainenstatic int maildir_sync_context(struct maildir_sync_context *ctx, bool forced,
6ef7e31619edfaa17ed044b45861d106a86191efTimo Sirainen bool new_changed, cur_changed, full_rescan = FALSE;
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen } else if (!forced) {
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen Locking, locking, locking.. Wasn't maildir supposed to be lockless?
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen We can get here either as beginning a real maildir sync, or when
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen committing changes to maildir but a file was lost (maybe renamed).
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen So, we're going to need two locks. One for index and one for
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen uidlist. To avoid deadlocking do the uidlist lock always first.
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen uidlist is needed only for figuring out UIDs for newly seen files,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen so theoretically we wouldn't need to lock it unless there are new
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen files. It has a few problems though, assuming the index lock didn't
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen already protect it (eg. in-memory indexes):
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen 1. Just because you see a new file which doesn't exist in uidlist
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen file, doesn't mean that the file really exists anymore, or that
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen your readdir() lists all new files. Meaning that this is possible:
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen A: opendir(), readdir() -> new file ...
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen -- new files are written to the maildir --
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen B: opendir(), readdir() -> new file, lock uidlist,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen readdir() -> another new file, rewrite uidlist, unlock
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen A: ... lock uidlist, readdir() -> nothing left, rewrite uidlist,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen The second time running A didn't see the two new files. To handle
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen this correctly, it must not remove the new unseen files from
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen uidlist. This is possible to do, but adds extra complexity.
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen 2. If another process is rename()ing files while we are
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen readdir()ing, it's possible that readdir() never lists some files,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen causing Dovecot to assume they were expunged. In next sync they
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen would show up again, but client could have already been notified of
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen that and they would show up under new UIDs, so the damage is
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen already done.
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen Both of the problems can be avoided if we simply lock the uidlist
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen before syncing and keep it until sync is finished. Typically this
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen would happen in any case, as there is the index lock..
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen The second case is still a problem with external changes though,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen because maildir doesn't require any kind of locking. Luckily this
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen problem rarely happens except under high amount of modifications.
57a91f930a12d2cd1220da4f3f7cb2c47557cd37Timo Sirainen ret = maildir_uidlist_sync_init(ctx->mbox->uidlist, ctx->partial,
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen /* failure / timeout. if forced is TRUE, we could still go
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen forward and check only for renamed files, but is it worth
d4dcb9c30dba354cff7af6d303ecef7698194c55Timo Sirainen the trouble? .. */
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen /* if we're going to check cur/ dir our current logic requires
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen that new/ dir is checked as well. it's a good idea anyway. */
b9b48aaaebf6f72dfab567cda073cde8a7b26598Timo Sirainen unsigned int count = 0;
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen while ((ret = maildir_scan_dir(ctx, TRUE)) > 0) {
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen /* rename()d at least some files, which might have
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen caused some other files to be missed. check again
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen (see MAILDIR_RENAME_RESCAN_COUNT). */
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen /* finish uidlist syncing, but keep it still locked */
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen maildir_uidlist_sync_finish(ctx->uidlist_sync_ctx);
57a91f930a12d2cd1220da4f3f7cb2c47557cd37Timo Sirainen /* NOTE: index syncing here might cause a re-sync due to
57a91f930a12d2cd1220da4f3f7cb2c47557cd37Timo Sirainen files getting lost, so this function might be called
57a91f930a12d2cd1220da4f3f7cb2c47557cd37Timo Sirainen re-entrantly. FIXME: and that breaks in
57a91f930a12d2cd1220da4f3f7cb2c47557cd37Timo Sirainen maildir_uidlist_sync_deinit() */
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen ret = maildir_sync_index(ctx->index_sync_ctx, ctx->partial);
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen if (maildir_sync_index_finish(&ctx->index_sync_ctx,
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen i_assert(maildir_uidlist_is_locked(ctx->mbox->uidlist));
e05181d973025627ba08b631c12c07c3bbc99528Timo Sirainen ret = maildir_uidlist_sync_deinit(&ctx->uidlist_sync_ctx);
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainenint maildir_storage_sync_force(struct maildir_mailbox *mbox)
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainenint maildir_sync_last_commit(struct maildir_mailbox *mbox)
d756ebcfa96bd7cff02097c8f26df9df368b81b1Timo Sirainenmaildir_storage_sync_init(struct mailbox *box, enum mailbox_sync_flags flags)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen struct maildir_mailbox *mbox = (struct maildir_mailbox *)box;
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainen mbox->ibox.sync_last_check + MAILBOX_FULL_SYNC_INTERVAL <=
cce169a321c9c629e4f2db1a69dae3b75bbcb27aTimo Sirainen ret = maildir_sync_context(ctx, FALSE, FALSE);
d67ac5f76cc02c227f4997878bb4aef48ee298faTimo Sirainen i_assert(!maildir_uidlist_is_locked(mbox->uidlist) ||
4d938f46f4f956ecb802c30ca771922f5539a660Timo Sirainen /* lost some files from new/, see if thery're in cur/ */
d756ebcfa96bd7cff02097c8f26df9df368b81b1Timo Sirainen return index_mailbox_sync_init(box, flags, ret < 0);
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainenint maildir_sync_is_synced(struct maildir_mailbox *mbox)
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen new_dir = t_strconcat(mbox->path, "/new", NULL);
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen cur_dir = t_strconcat(mbox->path, "/cur", NULL);
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen ret = maildir_sync_quick_check(mbox, new_dir, cur_dir,
a3d22d3cb0e5436128ca7287cedc921f1789b2c8Timo Sirainen return ret < 0 ? -1 : (!new_changed && !cur_changed);