maildir-sync.c revision d77c309fccbc6a7594f8cb08fb01009fa613c568
e59faf65ce864fe95dc00f5d52b8323cdbd0608aTimo Sirainen/* Copyright (C) 2004 Timo Sirainen */
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen Here's a description of how we handle Maildir synchronization and
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen it's problems:
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen We want to be as efficient as we can. The most efficient way to
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen check if changes have occured is to stat() the new/ and cur/
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen directories and uidlist file - if their mtimes haven't changed,
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen there's no changes and we don't need to do anything.
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen Problem 1: Multiple changes can happen within a single second -
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen nothing guarantees that once we synced it, someone else didn't just
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen then make a modification. Such modifications wouldn't get noticed
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen until a new modification occured later.
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen Problem 2: Syncing cur/ directory is much more costly than syncing
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen new/. Moving mails from new/ to cur/ will always change mtime of
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen cur/ causing us to sync it as well.
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen Problem 3: We may not be able to move mail from new/ to cur/
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen because we're out of quota, or simply because we're accessing a
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen read-only mailbox.
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen MAILDIR_SYNC_SECS
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen -----------------
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen Several checks below use MAILDIR_SYNC_SECS, which should be maximum
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen clock drift between all computers accessing the maildir (eg. via
f16c114c20bbd7d292d93415d1e56c8dd6abd3e7Timo Sirainen NFS), rounded up to next second. Our default is 1 second, since
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen everyone should be using NTP.
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen Note that setting it to 0 works only if there's only one computer
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen accessing the maildir. It's practically impossible to make two
16f816d3f3c32ae3351834253f52ddd0212bcbf3Timo Sirainen clocks _exactly_ synchronized.
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen It might be possible to only use file server's clock by looking at
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen the atime field, but I don't know how well that would actually work.
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen cur directory
84e1634acc701d14e358e27f1beff5ad74f5004aTimo Sirainen -------------
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen We have dirty_cur_time variable which is set to cur/ directory's
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen synchronized the directory.
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen When dirty_cur_time is non-zero, we don't synchronize the cur/
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen directory until
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen a) cur/'s mtime changes
0cb2e8eb55e70f8ebe1e8349bdf49e4cbe5d8834Timo Sirainen b) opening a mail fails with ENOENT
e06c0b65c16ccce69bbee009ead14d7d3d17a256Timo Sirainen c) time() > dirty_cur_time + MAILDIR_SYNC_SECS
e05ea8311ae16687295048e88ca205dfe29fbcbfTimo Sirainen This allows us to modify the maildir multiple times without having
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen to sync it at every change. The sync will eventually be done to
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen make sure we didn't miss any external changes.
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen The dirty_cur_time is set when:
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen - we change message flags
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen - we expunge messages
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen - we move mail from new/ to cur/
98c1cf256927e254f0c092acd2ddcd7ea50bd009Timo Sirainen - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen It's unset when we do the final syncing, ie. when mtime is
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen older than time() - MAILDIR_SYNC_SECS.
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen new directory
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen -------------
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen it. dirty_cur_time-like feature might save us a few syncs, but
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen that might break a client which saves a mail in one connection and
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen tries to fetch it in another one. new/ directory is almost always
ca98892a6b8a30ffc1fe26fcf02c7d59e3204e7eTimo Sirainen empty, so syncing it should be very fast anyway. Actually this can
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen still happen if we sync only new/ dir while another client is also
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen moving mails from it to cur/ - it takes us a while to see them.
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen That's pretty unlikely to happen however, and only way to fix it
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen would be to always synchronize cur/ after new/.
ca98892a6b8a30ffc1fe26fcf02c7d59e3204e7eTimo Sirainen Normally we move all mails from new/ to cur/ whenever we sync it. If
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen it's not possible for some reason, we mark the mail with "probably
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen exists in new/ directory" flag.
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen If rename() still fails because of ENOSPC or EDQUOT, we still save
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen the flag changes in index with dirty-flag on. When moving the mail
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen to cur/ directory, or when we notice it's already moved there, we
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen apply the flag changes to the filename, rename it and remove the
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen dirty flag. If there's dirty flags, this should be tried every time
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen after expunge or when closing the mailbox.
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen This file contains UID <-> filename mappings. It's updated only when
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen new mail arrives, so it may contain filenames that have already been
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen deleted. Updating is done by getting uidlist.lock file, writing the
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen whole uidlist into it and rename()ing it over the old uidlist. This
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen means there's no need to lock the file for reading.
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen Whenever uidlist is rewritten, it's mtime must be larger than the old
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen one's. Use utime() before rename() if needed. Note that inode checking
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen wouldn't have been sufficient as inode numbers can be reused.
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen This file is usually read the first time you need to know filename for
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen given UID. After that it's not re-read unless new mails come that we
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen don't know about.
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen broken clients
6288d3611eda14a017dae9927b73f46afb646c96Timo Sirainen --------------
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen Originally the middle identifier in Maildir filename was specified
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen only as <process id>_<delivery counter>. That however created a
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen problem with randomized PIDs which made it possible that the same
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen PID was reused within one second.
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen So if within one second a mail was delivered, MUA moved it to cur/
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen and another mail was delivered by a new process using same PID as
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen the first one, we likely ended up overwriting the first mail when
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen the second mail was moved over it.
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen Nowadays everyone should be giving a bit more specific identifier,
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen for example include microseconds in it which Dovecot does.
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen There's a simple way to prevent this from happening in some cases:
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen Don't move the mail from new/ to cur/ if it's mtime is >= time() -
299183fbb6ec5d0828a0880da372540421ac4665Timo Sirainen MAILDIR_SYNC_SECS. The second delivery's link() call then fails
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen because the file is already in new/, and it will then use a
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen different filename. There's a few problems with this however:
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen - it requires extra stat() call which is unneeded extra I/O
9e59a1f3f095b3099478562cf3f3970a24736970Timo Sirainen - another MUA might still move the mail to cur/
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen - if first file's flags are modified by either Dovecot or another
178511b57faa7c3f8203dd8b7e4059d00cbfc23aTimo Sirainen MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen but that'd be ugly)
2e937ed8585299b2e879a28314902a5f644813d2Timo Sirainen Because this is useful only for very few people and it requires
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen extra I/O, I decided not to implement this. It should be however
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen quite easy to do since we need to be able to deal with files in new/
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen It's also possible to never accidentally overwrite a mail by using
4d25408732be27e91f0430f71e87242760c2517cTimo Sirainen link() + unlink() rather than rename(). This however isn't very
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen good idea as it introduces potential race conditions when multiple
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen clients are accessing the mailbox:
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen Trying to move the same mail from new/ to cur/ at the same time:
4d25408732be27e91f0430f71e87242760c2517cTimo Sirainen a) Client 1 uses slightly different filename than client 2,
c3412ddeb9abc13f99d3caf50faf76cd99f7e9d2Timo Sirainen for example one sets read-flag on but the other doesn't.
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen You have the same mail duplicated now.
6f25019b337e27600159b596824da08732965576Timo Sirainen b) Client 3 sees the mail between Client 1's and 2's link() calls
6f25019b337e27600159b596824da08732965576Timo Sirainen and changes it's flag. You have the same mail duplicated now.
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen And it gets worse when they're unlink()ing in cur/ directory:
02b79f9c2636da1829eee5b92753602bba8b67edTimo Sirainen c) Client 1 changes mails's flag and client 2 changes it back
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen between 1's link() and unlink(). The mail is now expunged.
1e76a5b92f9d82d557f81f080f3dfad1c9d8f200Timo Sirainen d) If you try to deal with the duplicates by unlink()ing another
ced118ac5caf6fe83d34339c2c65c63b2aa768acTimo Sirainen one of them, you might end up unlinking both of them.
9a06cabdfdf4d5e2f19a07e506c3c7d08a7e7038Timo Sirainen So, what should we do then if we notice a duplicate? First of all,
9a06cabdfdf4d5e2f19a07e506c3c7d08a7e7038Timo Sirainen it might not be a duplicate at all, readdir() might have just
e8a35266a5ceacdfafeeffd6bddae77931ff97ebTimo Sirainen returned it twice because it was just renamed. What we should do is
9aa52288a4b53186d81b0ec9afa7d9e0a8ee8753Timo Sirainen create a completely new base name for it and rename() it to that.
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen If the call fails with ENOENT, it only means that it wasn't a
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen duplicate after all.
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen struct maildir_uidlist_sync_ctx *uidlist_sync_ctx;
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainenstatic int maildir_expunge(struct index_mailbox *ibox, const char *path,
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainenstatic int maildir_sync_flags(struct index_mailbox *ibox, const char *path,
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen struct maildir_index_sync_context *ctx = context;
6843896c40bee4f9b6680ca7ced598c446e9f999Timo Sirainen (void)maildir_filename_get_flags(path, &flags, keywords);
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen mail_index_sync_flags_apply(&ctx->sync_rec, &flags8, keywords);
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen newpath = maildir_filename_set_flags(path, flags8, keywords);
f6699a08521aacc4c2bb5b6175691dad5f715f8cTimo Sirainen mail_index_update_flags(ctx->trans, ctx->seq, MODIFY_ADD,
7f3be7d885c75cdd77f536929a45bc9764595960Timo Sirainenstatic int maildir_sync_record(struct index_mailbox *ibox,
7f3be7d885c75cdd77f536929a45bc9764595960Timo Sirainen struct mail_index_sync_rec *sync_rec = &ctx->sync_rec;
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* make it go through sequences to avoid looping through huge
6ef7e31619edfaa17ed044b45861d106a86191efTimo Sirainen holes in UID range */
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen if (mail_index_lookup_uid_range(view, sync_rec->uid1,
4d25408732be27e91f0430f71e87242760c2517cTimo Sirainen if (mail_index_lookup_uid(view, seq, &uid) < 0)
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen if (maildir_file_do(ibox, uid, maildir_expunge,
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen if (mail_index_lookup_uid_range(view, sync_rec->uid1,
bb10ebcf076c959c752f583746d83805d7686df8Timo Sirainen for (ctx->seq = seq1; ctx->seq <= seq2; ctx->seq++) {
bb10ebcf076c959c752f583746d83805d7686df8Timo Sirainen if (mail_index_lookup_uid(view, ctx->seq, &uid) < 0)
84ed9f8f3d0e5ed47607ef417618e49e4f865557Timo Sirainen /* if this flag was dirty, drop it */
bb10ebcf076c959c752f583746d83805d7686df8Timo Sirainen if (mail_index_lookup(view, ctx->seq, &rec) < 0)
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen if (rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) {
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainenint maildir_sync_last_commit(struct index_mailbox *ibox)
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen ret = mail_index_sync_begin(ibox->index, &ctx.sync_ctx, &ctx.view,
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen if (mail_index_get_header(ctx.view, &hdr) == 0 &&
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen (hdr->flags & MAIL_INDEX_HDR_FLAG_HAVE_DIRTY) != 0)
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen ctx.trans = mail_index_transaction_begin(ctx.view, FALSE);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen while ((ret = mail_index_sync_next(ctx.sync_ctx,
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen if (mail_index_transaction_commit(ctx.trans, &seq, &offset) < 0)
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainenmaildir_sync_context_new(struct index_mailbox *ibox)
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen ctx->new_dir = t_strconcat(ibox->path, "/new", NULL);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen ctx->cur_dir = t_strconcat(ibox->path, "/cur", NULL);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainenstatic void maildir_sync_deinit(struct maildir_sync_context *ctx)
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen (void)maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainenstatic int maildir_fix_duplicate(struct index_mailbox *ibox, const char *dir,
7e6749d69b9d3f87e598b8dc3346b82f3cb3b8e7Timo Sirainen old_path = t_strconcat(dir, "/", old_fname, NULL);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen new_fname = maildir_generate_tmp_filename(&ioloop_timeval);
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen new_path = t_strconcat(ibox->path, "/new/", new_fname, NULL);
33d63688ed8b26dc333e3c2edbfb2fe6e412604dTimo Sirainen "rename(%s, %s) failed: %m", old_path, new_path);
5d98a279f2726a8c38ef1e8e3c54d6c62d523a65Timo Sirainenstatic int maildir_scan_dir(struct maildir_sync_context *ctx, int new_dir)
5d98a279f2726a8c38ef1e8e3c54d6c62d523a65Timo Sirainen struct mail_storage *storage = ctx->ibox->box.storage;
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen const char *dir;
3f26c5aced2e71efc783f26bb8a7ac53f7504622Timo Sirainen move_new = new_dir && !mailbox_is_readonly(&ctx->ibox->box) &&
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen ret = maildir_uidlist_sync_next_pre(ctx->uidlist_sync_ctx,
1cd97699af9c77d8f5920832ec3374884544fd68Timo Sirainen /* new file and we couldn't lock uidlist, check this
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen later in next sync. */
279cc94ab086f6a3cb764b1b98ff6b936efa3eaeTimo Sirainen str_printfa(src, "%s/%s", ctx->new_dir, dp->d_name);
279cc94ab086f6a3cb764b1b98ff6b936efa3eaeTimo Sirainen str_printfa(dest, "%s/%s", ctx->cur_dir, dp->d_name);
e8a35266a5ceacdfafeeffd6bddae77931ff97ebTimo Sirainen /* we moved it - it's \Recent for us */
73b50eecfc31750a312e2f940023f522eb07178cTimo Sirainen /* someone else moved it already */
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* not enough disk space, leave here */
4d25408732be27e91f0430f71e87242760c2517cTimo Sirainen "rename(%s, %s) failed: %m",
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen } else if (new_dir) {
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen ret = maildir_uidlist_sync_next(ctx->uidlist_sync_ctx,
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen /* possibly duplicate - try fixing it */
84e1634acc701d14e358e27f1beff5ad74f5004aTimo Sirainenstatic int maildir_sync_quick_check(struct maildir_sync_context *ctx,
a2637488c8d514ec1ac3914811deee814f9761b3Timo Sirainen /* cur stamp is kept in index, we don't have to sync if
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen someone else has done it and updated the index. make sure
84e1634acc701d14e358e27f1beff5ad74f5004aTimo Sirainen we have a fresh index with latest sync_stamp. */
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen new_mtime >= ibox->last_new_sync_time - MAILDIR_SYNC_SECS) {
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen ioloop_time - ibox->dirty_cur_time > MAILDIR_SYNC_SECS)) {
e68309fcfa2eaa88217fd51e7b4900fc9c20ef5dTimo Sirainen /* cur/ changed, or delayed cur/ check */
39775ad03c459efe64cce924658da5094ba417e1Timo Sirainen cur_mtime >= ioloop_time - MAILDIR_SYNC_SECS ?
39775ad03c459efe64cce924658da5094ba417e1Timo Sirainenstatic int maildir_sync_index(struct maildir_sync_context *ctx)
e063aca6bc2f08bec516d4b631052ea9191f011dTimo Sirainen if (mail_index_sync_begin(ibox->index, &sync_ctx.sync_ctx, &view,
5137d2d80255938a0f5fb8f3c1a21b34cf11ada3Timo Sirainen i_assert(ret == 0); /* view is locked, can't happen */
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen uid_validity != 0 && hdr->uid_validity != 0) {
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* uidvalidity changed and mailbox isn't being initialized,
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen index must be rebuilt */
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen "Maildir %s sync: UIDVALIDITY changed (%u -> %u)",
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen trans = mail_index_transaction_begin(view, FALSE);
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen iter = maildir_uidlist_iter_init(ibox->uidlist);
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen while (maildir_uidlist_iter_next(iter, &uid, &uflags, &filename)) {
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen maildir_filename_get_flags(filename, &flags, keywords);
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RECENT) != 0 &&
8fcff4c5b52f24d9c681805fdf06b486f1d0fcbeTimo Sirainen (uflags & MAILDIR_UIDLIST_REC_FLAG_NEW_DIR) != 0 &&
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen (uflags & MAILDIR_UIDLIST_REC_FLAG_MOVED) == 0) {
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* mail is recent for next session as well */
7fd72a47d7ddfbd38c8697e228b6951f495dfb61Timo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
7fd72a47d7ddfbd38c8697e228b6951f495dfb61Timo Sirainen /* partial syncing */
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* most likely a race condition: we read the
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen maildir, then someone else expunged messages
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen and committed changes to index. so, this
313fe89df4d91cd0cd7f3558dc6d7fd21ad39eeeTimo Sirainen message shouldn't actually exist. mark it
bb10ebcf076c959c752f583746d83805d7686df8Timo Sirainen racy and check in next sync.
60d3fa9883237e896a8704275b6116fa46f7ffdaTimo Sirainen the difference between this and the later
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen check is that this one happens when messages
659fe5d24825b160cae512538088020d97a60239Timo Sirainen are expunged from the end */
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen "Maildir %s sync: "
c35d0c3eb4ba89432896e1f9770ab31cc63c232bTimo Sirainen "UID < next_uid "
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen "(%u < %u, file = %s)",
7a6b45405fb1544ac476e6eb1402a70cc1ddcdcfTimo Sirainen mail_index_update_flags(trans, seq, MODIFY_REPLACE,
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* expunged */
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen /* most likely a race condition: we read the
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen maildir, then someone else expunged messages and
1e76a5b92f9d82d557f81f080f3dfad1c9d8f200Timo Sirainen committed changes to index. so, this message
36816b5af1472ae76a1909ae3cf29fd614b2ebfcTimo Sirainen shouldn't actually exist. mark it racy and check
829c036d4ddfbd9ea49bd8a7c54e3057177d346eTimo Sirainen in next sync. */
36816b5af1472ae76a1909ae3cf29fd614b2ebfcTimo Sirainen if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen "Maildir %s sync: UID inserted in the "
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen "middle of mailbox "
dc8552739fa29f011ab71ec383ec6d580a5a9661Timo Sirainen "(%u > %u, file = %s)",
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen maildir_uidlist_add_flags(ibox->uidlist, filename,
4214b59ac7f3899f8d887d055ef519f5a622d249Timo Sirainen if ((rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0) {
7a6b45405fb1544ac476e6eb1402a70cc1ddcdcfTimo Sirainen /* we haven't been able to update maildir with this
f5b4979e2780c4df112a300967d647e2fdd73511Timo Sirainen record's flag changes. don't sync them. */
5486e1e123484f3a8d7d895e8ac41156f2a96305Timo Sirainen (rec->flags & (MAIL_FLAGS_MASK^MAIL_RECENT)) ||
5486e1e123484f3a8d7d895e8ac41156f2a96305Timo Sirainen /* FIXME: this is wrong if there's pending changes in
f5b4979e2780c4df112a300967d647e2fdd73511Timo Sirainen transaction log already. it gets fixed in next sync
5486e1e123484f3a8d7d895e8ac41156f2a96305Timo Sirainen mail_index_update_flags(trans, seq, MODIFY_REPLACE,
b2ecd50bb98c44816cb07c17aa17fae2b425f941Timo Sirainen /* just remove recent flag */
dc8552739fa29f011ab71ec383ec6d580a5a9661Timo Sirainen mail_index_update_flags(trans, seq, MODIFY_REMOVE,
dc8552739fa29f011ab71ec383ec6d580a5a9661Timo Sirainen /* expunge the rest */
b2ecd50bb98c44816cb07c17aa17fae2b425f941Timo Sirainen for (seq++; seq <= hdr->messages_count; seq++)
4214b59ac7f3899f8d887d055ef519f5a622d249Timo Sirainen /* now, sync the index */
4214b59ac7f3899f8d887d055ef519f5a622d249Timo Sirainen while ((ret = mail_index_sync_next(sync_ctx.sync_ctx,
8d5991f5c4a8840bf1ea754093dbec505564ab78Timo Sirainen if (maildir_sync_record(ibox, &sync_ctx) < 0) {
930dcf1576f99057ad572420d9c75f3212e46a2eTimo Sirainen ibox->last_cur_mtime != (time_t)hdr->sync_stamp) {
c9a03c8a03a782488981f81bd7c6b5e01ff55f06Timo Sirainen offsetof(struct mail_index_header, sync_stamp),
39775ad03c459efe64cce924658da5094ba417e1Timo Sirainen /* get the initial uidvalidity */
6a1e4eb2c6a267bec1e8704ce9137bebb7792702Timo Sirainen if (maildir_uidlist_update(ibox->uidlist) < 0)
6a1e4eb2c6a267bec1e8704ce9137bebb7792702Timo Sirainen uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
7a6b45405fb1544ac476e6eb1402a70cc1ddcdcfTimo Sirainen maildir_uidlist_set_uid_validity(ibox->uidlist,
6a1e4eb2c6a267bec1e8704ce9137bebb7792702Timo Sirainen } else if (uid_validity == 0) {
6a1e4eb2c6a267bec1e8704ce9137bebb7792702Timo Sirainen maildir_uidlist_set_uid_validity(ibox->uidlist,
a614397cf1a4dde152eb1a38493a6ec3d817da16Timo Sirainen if (uid_validity != hdr->uid_validity && uid_validity != 0) {
a614397cf1a4dde152eb1a38493a6ec3d817da16Timo Sirainen offsetof(struct mail_index_header, uid_validity),
8907d617ce7c4f390c0f42f6f694db2fecdd5775Timo Sirainen next_uid = maildir_uidlist_get_next_uid(ibox->uidlist);
dda2c506c8fc8ac2f88272de4523ded42baa0aa0Timo Sirainen if (next_uid != 0 && hdr->next_uid != next_uid) {
abe8230dd1dd37d7ccf0163100e934bb5e658c20Timo Sirainen if (mail_index_transaction_commit(trans, &seq, &offset) < 0)
abe8230dd1dd37d7ccf0163100e934bb5e658c20Timo Sirainen else if (seq != 0) {
abe8230dd1dd37d7ccf0163100e934bb5e658c20Timo Sirainen if (mail_index_sync_end(sync_ctx.sync_ctx) < 0)
b5e6f6f27c1461f0f9f202615eeb738a645188c3Timo Sirainenstatic int maildir_sync_context(struct maildir_sync_context *ctx)
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen if (maildir_sync_quick_check(ctx, &new_changed, &cur_changed) < 0)
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen maildir_uidlist_sync_init(ctx->ibox->uidlist, ctx->partial);
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen /* we have to lock uidlist immediately, otherwise there's race
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen conditions with other processes who might write older maildir
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen file list into uidlist.
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen alternative would be to lock it when new files are found, but
abe8230dd1dd37d7ccf0163100e934bb5e658c20Timo Sirainen the directory scans _must_ be restarted then */
319944c0f35b311c998854e96d6463a084fd90aeTimo Sirainen if (maildir_uidlist_try_lock(ctx->ibox->uidlist) < 0)
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen /* finish uidlist syncing, but keep it still locked */
cece2b9cd692c06025cc0a7a0ff54d996a8c90efTimo Sirainen if (maildir_uidlist_sync_finish(ctx->uidlist_sync_ctx)) {
f0569d9fbb25c8437760be69f194595a841ad711Timo Sirainen ret = maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainenstatic int maildir_sync_context_readonly(struct maildir_sync_context *ctx)
4b9f99761df5014c659cd87fddaf6854af428cfcTimo Sirainen maildir_uidlist_sync_init(ctx->ibox->uidlist, FALSE);
4b9f99761df5014c659cd87fddaf6854af428cfcTimo Sirainen ret = maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
4b9f99761df5014c659cd87fddaf6854af428cfcTimo Sirainenint maildir_storage_sync_readonly(struct index_mailbox *ibox)
0a6f8311541ae59381171620b77f82be58be562eTimo Sirainenint maildir_storage_sync(struct mailbox *box, enum mailbox_sync_flags flags)
f1901fd21906911f7be075c965ac882f6a87b4c3Timo Sirainen struct index_mailbox *ibox = (struct index_mailbox *)box;