lib-index: Fix assert-crash with lock_method=dotlock The dotlock wasn't deleted in all code paths. Fix this by simplifying the unlocking to be done the same way with and without dotlock. Fixes: Panic: file mail-cache.c: line 624 (mail_cache_lock_file): assertion failed: (cache->dotlock == NULL)

global: start relying on file_lock_free(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - file_lock_free(&E); - } + file_lock_free(&E);

Updated copyright notices to include the year 2018.

lib-index: Add mail_cache_open_or_create_path() This allows moving cache file to another directory.

lib-index: Add mail_index_cache_optimization_settings

global: start relying on buffer_free(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - buffer_free(&E); - } + buffer_free(&E);

lib-index: Add mail_cache_set_seq_corrupted_reason Marks a seq in cache corrupted, and removes that instead of the whole cache.

spelling fixes

lib-index: zero mmap_length after logging error Logging that mmap(size=0) failed looks suspicious.

lib-index: Don't log error on dovecot.index.cache delete if it doesn't exist For example if a cache is marked as corrupted and there is no cache, it's not really an error that it doesn't exist. Fixes: Error: unlink(.../dovecot.index.cache) failed: No such file or directory (in mail-cache.c:29)

index: don't try to unlink in-memory cache file Avoid these errors: Error: unlink((in-memory index).cache) failed: No such file or directory (in mail-cache.c:29)

Updated copyright notices to include the year 2017.

lib-index: Fix checking if cache file becomes >4GB

lib-index: Use file_cache_new_path() for dovecot.index.cache

lib-index: Limit mmap errors in index cache to 1/s

lib-index: If mmap() fails, include the attempted size in the error.

lib-index: If opening a cache file fails, try again later. The previous code would never retry opening the cache file within the same session.

lib-index: Fixed calling mail_cache_open_and_verify() on an already open cache. This was done at least by index_index_rebuild_init(). Either the currently open cache->fd was leaked, or if the cache file open() failed we left the cache in an inconsistent state where cache->fd == -1, but cache->hdr != NULL, so it caused MAIL_CACHE_IS_UNUSABLE() to also be TRUE. This could have ended up in an assert: Panic: file mail-index-lock.c: line 31 (mail_index_lock_fd): assertion failed: (MAIL_INDEX_IS_IN_MEMORY(index))

lib-index: After MAIL_INDEX_OPEN_FLAG_SAVEONLY access cache's read_buf wasn't always reset. This could have caused at least errors like: Corrupted index cache file .../dovecot.index.cache: File too small I'm not sure if it might have also returned stale data sometimes. Easy way to reproduce this bug was: imaptest copy=100 copybox=Trash & imaptest box=Trash append=0

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

Replaced unlink() calls with i_unlink*() wherever possible.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-index: Fixed assert-crash in some cache locking race conditions. If mail_index_map() is being called, we can't refresh the index or it'll crash. It wouldn't help anyway, since the index was just refreshed.

lib-index: mail_cache_lock() partial rewrite. require_same_reset_id is no longer needed, if it ever was. If we're locking the cache file, we always want the latest one. The logic of locking in general was somewhat confusing and it probably didn't always successfully lock when it should have, because the reset_id happened to match an old file.

lib-index: Code cleanup - avoid code duplication.

lib-index: Fixed cache file creation race condition. If two processes are creating the index files at the same time, don't have one of them delete the dovecot.index.cache that the other one just created. This means we never should be calling mail_cache_create(), so it was removed entirely.

Updated copyright notices to include year 2014.

lib-index: Avoid assert-crashing later if mmap(dovecot.index.cache) fails.

lib-index: Bug in cache file size verification caused the whole cache file to be mapped.

Oops :) Update copyrights to 2013 without breaking all .c files.

Updated copyright notices to include year 2013.

lib-index: Fixed invalidating buffered cache file with map_with_read.

lib-index: Fixes to handling broken cache records that point outside file. Especially try to avoid failing by trying to allocate gigabytes of memory.

lib-index: If map_with_read gets disabled, reset the cache buffers properly.

lib-index: MAIL_INDEX_OPEN_FLAG_SAVEONLY was buggy when reading data near end of dovecot.index.cache. We assumed that we read as much as we requested, even if the file was smaller.

lib-index: Fixed assert-crash caused by recent cache file changes.

lib-index: Fixed memory leak with MAIL_INDEX_OPEN_FLAG_SAVEONLY

lib-index: MAIL_INDEX_OPEN_FLAG_SAVEONLY may have caused stale data to be read from cache.

lib-index: Replaced cache's deleted_space with deleted_record_count. This way when expunging messages we don't need to actually read through the cache file and calculate how many bytes will be freed, we can simply increase the deleted_record_count by 1. We'll still compress the cache file approximately after 20% of the space (records) has been deleted. This also converts the old "hole offset" header field to tracking the number of records in the cache file. Otherwise we'd have to assume that all of the messages in mailbox have a cache record, but that isn't true for users who have only temporary cache fields. This "hole offset" cannot be trusted completely, because nothing prevents old Dovecot version from simply writing a hole offset there and messing up the record counts. Because of this there are a couple of extra sanity checks for it. (Unfortunately there wasn't any easy way to simply add a new field to header without breaking backwards compatibility.)

lib-index: Added minor version to dovecot.index.cache file header.

lib-index: Cache record linking is now while writing new records, not later with pwrite(). This should make performance somewhat better. It also means that now the only overwrites to cache file are to the file header and the fields header. This means that we no longer have to invalidate file_cache (with mmap_disable=yes) except when reading the headers, which is always done anyway. It would be nice to move the overwritten fields in the headers to a cache header in dovecot.index file, but this is a bit tricky currently. When writing these headers the cache file needs to be locked to avoid race conditions. To avoid deadlocks the transaction log would have to be locked before the cache. The code to do all this would still be a bit complex, and to keep backwards compatibility with old Dovecot versions it would still have to do the overwriting until the backwards compatibility code is disabled.

lib-index: Recent cache file changes broke writing.

lib-index: Recent cache file changes broke writing.

lib-index: Optimize cache file reads with MAIL_INDEX_OPEN_FLAG_SAVEONLY

lib-index: After recent cache changes, cache was often wrongly being thought of as unusable