mail-index-strmap.c revision fde0b1793a2842da00eaa105d5e13fec465f0443
1281N/A/* Copyright (c) 2008-2009 Dovecot authors, see the included COPYING file */ 1388N/A/* number of bytes required to store one string idx */ 1388N/A/* renumber the string indexes when highest string idx becomes larger than 1388N/A <number of indexes>*STRMAP_FILE_MAX_STRIDX_MULTIPLIER */ 1186N/A "%s failed with strmap index file %s: %m",
1388N/A "Corrupted strmap index file: %s",
1388N/A /* need to rebuild. if we already had something in the strmap, 1388N/A /* we'll read the entire file from the beginning */ 1388N/A /* last read failed because view had a message 1388N/A that didn't exist in the strmap (because it 1388N/A was expunged by another session). if the 1388N/A message still isn't expunged in this view, 1388N/A just continue using the current strmap. */ 1388N/A /* our view isn't synced with the disk, we 1388N/A can't read strmap without first resetting 1388N/A /* thread index has larger UIDs than what we've seen 1388N/A in our view. we'll have to read them again later 1388N/A /* record that exists in index is missing from strmap. 1388N/A see if it's because the strmap is corrupted or because 1388N/A our current view is a bit stale and the message has already 1388N/A /* <uid> <n> <crc32>*count <str_idx>*count 1388N/A n = 0 -> count=1 (only Message-ID:) 1388N/A n = 1 -> count=2 (Message-ID: + In-Reply-To:) 1388N/A n = 2+ -> count=n (Message-ID: + References:) /* this message has already been expunged, ignore it. update highest string indexes anyway. */ for (i = 0; i <
count; i++) {
/* everything exists. save it. FIXME: these ref_index values are thread index specific, perhaps something more generic should be used some day */ /* FIXME: str_idx could be stored as packed relative values (first relative to highest_idx, the rest relative to the /* read the record contents */ /* get to the next record */ const unsigned char *
data;
/* come back later when we know about the new UIDs */ /* the rest of the file is either not written, or the previous /* block size too large */ /* FIXME: when reading multiple blocks we shouldn't have to calculate /* if all string indexes are unique, highest_str_index equals total_ref_count. otherwise it's always lower. */ "Corrupted strmap index file %s: " "String indexes too high " /* our view contained a message that had since been expunged. */ /* hopefully it's a message that has since been expunged */ /* message is no longer in our view. remove it completely. */ /* it's quite likely a conflict. we may not be able to verify it, so just assume it is. nothing breaks even if we guess wrong, the performance just suffers a bit. */ /* 0 means "doesn't match", which is the only acceptable case */ /* unique string - there are no conflicts */ /* check for conflicting string indexes. they may happen if 1) msgid exists only for a message X that has been expunged 2) another process doesn't see X, but sees msgid for another message and writes it using a new string index 3) if we still see X, we now see the same msgid with two if we detect such a conflict, we can't continue using the strmap index until X has been expunged. */ /* CRC32 matches, but string index doesn't */ /* we've already added this */ /* add the record to records array */ /* add a separate copy of the record to hash */ /* reading the strmap failed - just ignore and do this in-memory based on whatever we knew last */ /* something failed - we can still use the strmap as far as we managed to read it, but our view is now out /* The string already exists, use the same unique idx */ /* Newly seen string, assign a new unique idx to it */ /* zero-terminate the records array */ /* create a map of old -> new index and remove records of /* see if this record should be removed */ /* notify caller of the renumbering */ /* renumber the indexes in-place and recreate the hash */ for (i = 0; i <
count; i++) {
/* update the new next_str_idx only after remapping */ /* skip over the block size for now, we don't know it yet */ /* @UNSAFE: <uid diff> */ /* find how many records belong to this UID */ for (j = i +
1; j <
count; j++) {
/* <n> <crc32>*count <str_idx>*count - FIXME: thread index specific code */ /* Only Message-ID: header */ /* In-Reply-To: header */ /* we know the block size now - write it */ /* everything expunged - just unlink the existing index */ "safe_mkstemp_hostpid(%s) failed: %m",
"rename(%s, %s) failed: %m",
"file_dotlock_create()");
/* Check first if another process had written new records to the file. If there are any, hopefully they're the same as what we would be writing. There are two problematic cases when messages have been 1) The file contains UIDs that we don't have. This means the string indexes won't be compatible anymore, so we'll have to renumber ours to match the ones in the strmap file. Currently we don't bother handling 1) case. If indexes don't match what we have, we just don't write anything. 2) We have UIDs that don't exist in the file. We can't simply skip those records, because other records may have pointers to them using different string indexes than we have. Even if we renumbered those, future appends by other processes might cause the same problem (they see the string for the first time and assign it a new index, but we already have internally given it another index). So the only sensible choice is to write nothing and hope that the message goes /* string index mismatch, /* nothing new to write */ /* write the new records */ /* FIXME: this renumbering doesn't work well when running for a long time since records aren't removed from hash often enough */ /* initial file creation */ /* append the new records to the strmap file */ /* the file was already recreated - leave the syncing as it is for now and let the next sync re-read the file. */ /* zero-terminate the records array */