mbox-sync.c revision 022412398e56a8f31ef111cfd7271498d64af9a9
0N/A/* Copyright (C) 2004 Timo Sirainen */ 0N/A Modifying mbox can be slow, so we try to do it all at once minimizing the 0N/A required disk I/O. We may need to: 0N/A - Update message flags in Status, X-Status and X-Keywords headers 0N/A - Write missing X-UID and X-IMAPbase headers 0N/A - Write missing or broken Content-Length header if there's space 0N/A - Expunge specified messages 0N/A Here's how we do it: 0N/A - Start reading the mails from the beginning 0N/A - X-Keywords, X-UID and X-IMAPbase headers may contain padding at the end 0N/A of them, remember how much each message has and offset to beginning of the 0N/A - If header needs to be rewritten and there's enough space, do it 1472N/A - If we didn't have enough space, remember how much was missing 1472N/A - Continue reading and counting the padding in each message. If available 1472N/A padding is enough to rewrite all the previous messages needing it, do it 0N/A - When we encounter expunged message, treat all of it as padding and 0N/A rewrite previous messages if needed (and there's enough space). 0N/A Afterwards keep moving messages backwards to fill the expunged space. 0N/A Moving is done by rewriting each message's headers, with possibly adding 0N/A missing Content-Length header and padding. Message bodies are moved 0N/A without modifications. 0N/A - If we encounter end of file, grow the file and rewrite needed messages 0N/A - Rewriting is done by moving message body forward, rewriting message's 0N/A header and doing the same for previous message, until all of them are 3041N/A/* The text below was taken exactly as c-client wrote it to my mailbox, 3041N/A so it's probably copyrighted by University of Washington. */ 3041N/A"This text is part of the internal format of your mail folder, and is not\n" \
3041N/A"a real message. It is created automatically by the mail system software.\n" \
3041N/A"If deleted, important folder data will be lost, and it will be re-created\n" \
3041N/A"with the data reset to initial values.\n" 0N/A "mbox file %s was modified while we were syncing, " 0N/A /* Do this even if ext_modified is already set. Expunging code relies 3178N/A on last_stat being updated. */ 1879N/A /* just mark the stat as dirty. */ /* need to add 'O' flag to Status-header */ /* nothing for this or the future ones */ /* we can't expunge anything from read-only mboxes */ i_assert(
ret != 0);
/* we should be looking at head index */ /* externally expunged message, remove from index */ /* this UID was already in index and it was expunged */ "mbox sync: Expunged message reappeared in mailbox %s " "(UID %u < %u, seq=%u, idx_msgs=%u)",
/* new UID in the middle of the mailbox - shouldn't happen */ "mbox sync: UID inserted in the middle of mailbox %s " /* externally expunged message, remove from index */ /* see if from_offset needs updating */ /* see if we need to update flags in index file. the flags in sync records are automatically applied to rec->flags at the end of index syncing, so calculate those new flags first */ /* flags are dirty. ignore whatever was in the mbox, /* keep index's internal flags */ /* drop recent flag (it can only be dropped) */ /* dirty flag state changed */ /* see if we need to update md5 sum. */ /* update from_offsets, but not if we're going to rewrite this message. rewriting would just move it anyway. */ const unsigned char *
data;
/* first check that the 10 bytes are there and they're exactly as expected. just an extra safety check to make sure we never write to wrong location in the mbox file. */ "X-IMAPbase uid-last unexpectedly points outside " if (
buf[i] <
'0' ||
buf[i] >
'9') {
"X-IMAPbase uid-last unexpectedly lost in mbox file %s",
for (i = 0; i <
count; i++) {
/* expunging first message, fix space to contain next message's \n header too since it will be removed. */ /* uid-last offset is invalid now */ /* move the header backwards to fill expunged space */ /* we're moving this mail to beginning of file. skip the initial \n (it's already counted in /* read the From-line before rewriting overwrites it */ /* rewrite successful, write From-line to /* didn't have enough space, move the offset back so seeking into it doesn't fail */ /* mark it dirty and do it later */ /* first mail with no space to write it */ /* create dummy message to describe the expunged data */ /* mail's keywords are allocated from a pool that's cleared for each mail. we'll need to copy it to something more /* we have enough space now */ /* this message was expunged. fill more or less of the space. space_diff now consists of a negative "bytes needed" sum, plus the expunged space of this message. so it contains how many bytes of _extra_ space we have. */ /* don't waste too much on padding */ /* this message gave enough space from headers. rewriting stops at the end of this message's headers. */ /* mail_ctx may contain wrong data after rewrite, so make sure we don't try to access it */ "Mailbox isn't a valid mbox file");
"Message was expunged unexpectedly " "Error seeking back to original " "offset %s in mbox file %s",
/* set to -1, since it's always increased later */ /* this mbox has pseudo mail which contains the X-IMAP header */ /* doesn't exist anymore, seek to end of file */ "Error seeking to end of mbox file %s",
/* delete sync records up to next message. so if there's still something left in array, it means the next message needs modifying */ /* we can skip forward to next record which needs updating. */ /* if there's no sync records left, we can stop. except if this is a dirty sync, check if there are new messages. */ /* seek failed because the offset is dirty. just ignore and continue from where we are now. */ i_warning(
"UIDVALIDITY changed (%u -> %u) in mbox file %s",
/* always start from first message so we can read X-IMAP or /* UID ordering problems, resync everything to make sure we get everything right */ "UIDs broken with partial sync in mbox file %s",
/* UID found but it's broken */ /* If we can't use/store X-UID header, use MD5 sum. Also check for existing MD5 sums when we're actually /* get all sync records related to this message. with pseudo message just get the first sync record so we can jump to it with partial seeking. */ /* if it was set, it was for the next message */ /* message wasn't found from index. we have to read everything from now on, no skipping */ /* oh no, we're out of UIDs. this shouldn't happen normally, so just try to get it fixed "Out of UIDs, renumbering them in mbox " /* rest of the messages in index don't exist -> expunge them */ /* once we get around to writing the changes, we'll need to do a full sync to avoid the "UIDs broken in partial sync" "From: Mail System Internal Data <MAILER-DAEMON@%s>\n" "Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA" "\nMessage-ID: <%s@%s>\n" /* out of disk space, truncate to empty */ "file size unexpectedly shrinked in mbox file %s " /* copy trailer, then truncate the file */ /* everything deleted, the trailer_size still contains /* We moved messages inside the mbox file without changing the file's size. If mtime doesn't change, another process not using the same index file as us can't know that the file was changed. So make sure the mtime changes. This should happen rarely enough that the sleeping doesn't become a Note that to do this perfectly safe we should do this wait whenever mails are moved or expunged, regardless of whether the file's size changed. That however could become a performance problem and the consequences of being wrong are quite minimal (an extra logged error message). */ /* only reason not to have UID validity at this point is if the file is entirely empty. In that case just make up a new one if needed. */ /* forcing a full sync. assume file has changed. */ /* file is fully synced */ /* we want to do full syncing. always do this if file size hasn't changed but timestamp has. it most likely means that someone had modified some header and we probably want to know about it */ /* see if we can delay syncing the whole file. normally we only notice expunges and appends for (i = 0; i <
3; i++) {
/* partial syncing didn't work, do it again. we get here also if we ran out of UIDs. */ /* only syncs left should be just appends (and their updates) which weren't synced yet for some reason (crash). we'll just ignore them, as we've overwritten them above. */ /* we just want to lock it for reading. if mbox hasn't been modified don't do any syncing. */ /* have to sync to make sure offsets have stayed the same */ /* reopen input stream to make sure it has nothing buffered */ /* we're most likely modifying the mbox while syncing, just lock it for writing immediately. the mbox must be locked before index syncing is started to avoid deadlocks, so we don't have much choice either (well, easy ones anyway). */ /* index may need to do internal syncing though, so commit instead of rollbacking. */ /* make sure we've read the latest keywords in index */ /* if we have only flag changes, we don't need to open the /* ok, we have something to do but no locks. we'll have to restart syncing to avoid deadlocking. */ /* Rewrite uid_last in X-IMAPbase header if we've seen it (ie. the file isn't empty) */ /* try to set atime back to its original value */