lib-index: Fix assert-crash with lock_method=dotlock The dotlock wasn't deleted in all code paths. Fix this by simplifying the unlocking to be done the same way with and without dotlock. Fixes: Panic: file mail-cache.c: line 624 (mail_cache_lock_file): assertion failed: (cache->dotlock == NULL)

lib-index: Update comments for mail_cache_decision_*()

lib-index: Add mail_index_cache_optimization_settings

lib-index: Fix reset_id handling in mail_cache_get_missing_reason() cache Broken by bd897f82a5d9b00ba256b462d2056c2dc7df257c

lib-index: Add cache to mail_cache_get_missing_reason() This avoids excessive CPU usage when it's called in a loop for many mails.

lib-index: Add mail_cache_expunge_count It can be called to correctly handle cache counters when records are removed.

lib-index: Limit mmap errors in index cache to 1/s

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib-index: mail_cache_lock() partial rewrite. require_same_reset_id is no longer needed, if it ever was. If we're locking the cache file, we always want the latest one. The logic of locking in general was somewhat confusing and it probably didn't always successfully lock when it should have, because the reset_id happened to match an old file.

lib-index: Don't keep cache file locked for as long while syncing index. The earlier code was required for updating the cache offsets, but this code no longer exists. Now we just need to update the record counts in the header, which can be done quickly at the end of the sync.

lib-index: mail_cache_lookup*() can now return fields recently added with mail_cache_add() Previously it was returning them if they had already been written to dovecot.index.cache, but not if they were still in the in-memory buffer. This avoids caching/parsing the same field multiple times when messages aren't accessed in ascending order (e.g. when sorting messages).

lib-index: Added v2.2+ forwards compatibility to dovecot.index.cache

lib-index: Replaced cache's deleted_space with deleted_record_count. This way when expunging messages we don't need to actually read through the cache file and calculate how many bytes will be freed, we can simply increase the deleted_record_count by 1. We'll still compress the cache file approximately after 20% of the space (records) has been deleted. This also converts the old "hole offset" header field to tracking the number of records in the cache file. Otherwise we'd have to assume that all of the messages in mailbox have a cache record, but that isn't true for users who have only temporary cache fields. This "hole offset" cannot be trusted completely, because nothing prevents old Dovecot version from simply writing a hole offset there and messing up the record counts. Because of this there are a couple of extra sanity checks for it. (Unfortunately there wasn't any easy way to simply add a new field to header without breaking backwards compatibility.)

lib-index: Added minor version to dovecot.index.cache file header.

lib-index: If a cache record is larger than 64 kB, don't add it to cache file. This shouldn't affect anything except mails that probably shouldn't exist in the first place.

lib-index: Cache record linking is now while writing new records, not later with pwrite(). This should make performance somewhat better. It also means that now the only overwrites to cache file are to the file header and the fields header. This means that we no longer have to invalidate file_cache (with mmap_disable=yes) except when reading the headers, which is always done anyway. It would be nice to move the overwritten fields in the headers to a cache header in dovecot.index file, but this is a bit tricky currently. When writing these headers the cache file needs to be locked to avoid race conditions. To avoid deadlocks the transaction log would have to be locked before the cache. The code to do all this would still be a bit complex, and to keep backwards compatibility with old Dovecot versions it would still have to do the overwriting until the backwards compatibility code is disabled.

lib-index: Added MAIL_INDEX_OPEN_FLAG_SAVEONLY to do only minimal reads from cache file.

lib-index: Code cleanup: Removed mail_cache.data

lib-index: mail_cache_map() API cleanup

lib-index: Simplified writing to dovecot.index.cache file. The old method was basically: - write max. 32 kB to internal buffer - flush it by writing to reserved areas (with no locks) The reserved areas were acquired by doing (whenever needed): - lock dovecot.index.cache - reserve data from dovecot.index.cache for writing, potentially increasing the file size by writing 0 bytes. the reserved area size varies. - unlock dovecot.index.cache This worked, but if multiple processes were writing to the cache file it could have left incomplete reserved areas as holes. The holes were attempted to be filled if they were large enough. The new method is: - write max. 256 kB to internal buffer - lock dovecot.index.cache - append the buffer to dovecot.index.cache - unlock dovecot.index.cache No reserved areas, holes or anything else weird going on. Ideally no data would be overwritten in the dovecot.index.cache file, only appended. Unfortunately currently some data is still overwritten: - mail_cache_header.{deleted_space,continued_record_count} - mail_cache_header_fields.next_offset when writing a new one - mail_cache_header_fields.{last_used,decision} - mail_cache_record.prev_offset The changing headers could eventually be moved to dovecot.index. This however is a backwards-incompatible change. The record's prev_offset could maybe simply just not be written in those (somewhat rare) problematic situations.

Reverted "support for non-pointers" part of the hash table API changes. Originally I wrote it using clang, which didn't give as many warnings as gcc did. I guess this way is safer anyway..

Hash table API is now (mostly) type safe.

lib-index: mail_cache_open_and_verify() is now public.

lib-index: Allow updating cache's last_used field with mail_cache_register_fields()

lib-index: Added mail_cache_view_update_cache_decisions()

lib-index: Added mail_cache_view_update_cache_decisions()

Mail cache: If locking failed, the next time try to lock it only once. Also increased the first timeout to 10 seconds. --HG-- branch : HEAD

Mail cache compression: If we can't get lock immediately, do it later. --HG-- branch : HEAD

Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. --HG-- branch : HEAD

Mail cache file: Changed lock timeout to just 2 seconds. It's better to not update cache than waste a lot of time waiting for a lock. Also increased staleness timeout to 5 minutes - it's safer that way and with the reduced lock timeout doesn't matter as much. --HG-- branch : HEAD

mail_cache_lock(): Return -1 if locking times out. This fixes cache compression starting even when locking timed out. --HG-- branch : HEAD

Optimized cache record loop tracking. --HG-- branch : HEAD

Optimized cache record loop tracking. --HG-- branch : HEAD

Link cache records together directly when writing the new records, instead of delaying them until later and causing lots of small writes. We still do this delayed check and do the writes when it's required, but it shouldn't happen normally. --HG-- branch : HEAD

Fixes to handling when fields are dropped from cache file. --HG-- branch : HEAD

Update caching decision for fields even if it doesn't currently exist in the cache file. If the field is added later within the session, the decision and last_used field will be correct. --HG-- branch : HEAD

mail_cache_field_want_add(): Return TRUE for temp fields only if we're adding the field to a new enough message. --HG-- branch : HEAD

NFS cache flushing APIs changed and backend implements it a bit differently. Hopefully this works now more correctly. --HG-- branch : HEAD

Fixed cache file's NFS attribute cache flushing. --HG-- branch : HEAD

When reading the latest fields, begin finding the fields from the last known header instead of from the beginning. --HG-- branch : HEAD

Keep a separate copy of mail_cache_header when mmap_disable=yes, otherwise it could get cleared unexpectedly. --HG-- branch : HEAD

If we need to do more than 4 reads to get to the cached fields header, compress the cache file. --HG-- branch : HEAD

mail_cache_transaction_commit()/rollback() API is now public and takes a ** pointer. Don't break if transaction is committed/rollbacked multiple times for a cache view. --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

Drop fields that haven't been used for 30 days when compressing. --HG-- branch : HEAD

mail_cache_lookup_iter_next() returned -1 if entry didn't exist, causing the first field for the record not to be added to cache. --HG-- branch : HEAD

Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. --HG-- branch : HEAD

Don't create cache file until something is actually being added to it. --HG-- branch : HEAD

Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_*() --HG-- branch : HEAD

Cache code cleanups --HG-- branch : HEAD

Removed file offset caching from mail_cache_foreach(). It didn't work correctly when cache file was just compressed, and caused "cache file corrupted" errors. --HG-- branch : HEAD

Prefix COMPRESS_* defines with MAIL_CACHE_. --HG-- branch : HEAD

Type safe callbacks weren't as easy as I thought. Only callback(void *context) can be handled generically. Others can be handled specially, but only if all the parameters are pointers, otherwise eg. int parameter can be replaced with long without compiler giving any warnings. --HG-- branch : HEAD

Added context parameter type safety checks for most callback APIs. --HG-- branch : HEAD

Lock handling changes. Everything goes through file-lock API now and there's only a single enum listing the different lock methods. This change exposed some unneeded (or possibly even wrong?) unlock calls in index file handling which were fixed. --HG-- branch : HEAD

Don't use time_t anymore in cache file. uint32_t is large enough and makes cache files compatible between 32bit and 64bit systems. --HG-- branch : HEAD

Delay opening the cache file until it's actually needed. --HG-- branch : HEAD

Added support for calling expunge handler for an extension even if the extension isn't used in the opened index file. Added context-parameter to the callback function. If the function returns -1, the sync is failed. --HG-- branch : HEAD

Array API redesigned to work using unions. It now provides type safety without having to enable DEBUG, as long as the compiler supports typeof(). Its API changed a bit. It now allows directly accessing the array contents, although that's not necessarily recommended. Changed existing array usage to be type safe in a bit more places. Removed array_t completely. Also did s/modifyable/modifiable/. --HG-- branch : HEAD

Don't compress cache file if it was just compressed by another process. --HG-- branch : HEAD

Removed immediate_stale_timeout and changed the stale_timeout behavior to check both dotlock and the file it protects, and overwrite the lock file whenever neither of them have been modified for stale_timeout seconds (ie. also immediately if their timestamps are old enough). --HG-- branch : HEAD

Moved uoff_t and time_t compatibility checks to cache file, since they're not needed for index/log. Only index file incompatibility check is now endianess flag. --HG-- branch : HEAD

Unlocking cache file can also corrupt it since it modifies it. Added a return value for it and checks to handle the failure. Fixes a crash in some error situations. --HG-- branch : HEAD

Whenever writing to cache file, also updated file_cache. Fixes problems with mmap_disable=yes. --HG-- branch : HEAD

Changed many buffers to arrays. Cleans up the code a lot. --HG-- branch : HEAD

Changed dotlocking API. --HG-- branch : HEAD

mail_cache_decision_*() prototypes were wrong. --HG-- branch : HEAD

lock_method=dotlock doesn't crash anymore while trying to modify cache file. We didn't previously deal with any fatal errors that fcntl/flock locking could have given but assumed that the locking succeeded. --HG-- branch : HEAD

Added "sync lost" handlers which are called with mmap_disable=yes when index file is re-read without going through all of the transaction log. --HG-- branch : HEAD

Crashfixes for reading corrupted cache files. --HG-- branch : HEAD

Cache file works now with mmap_disable=yes. Still needs a few optimizations. --HG-- branch : HEAD

Removed cache_offset from mail_index_record and changed it to use extension instead. Added possibility to register sync and expunge handlers for extensions. Changed the way extension resets work: all extension updates which were committed without having seen the reset are ignored. --HG-- branch : HEAD

Fix for circular record list detection, we can't use data stack for buffer. --HG-- branch : HEAD

ID mapping fixes. --HG-- branch : HEAD

Transaction log file is now read-lockless. --HG-- branch : HEAD

Fixes for big endian systems. --HG-- branch : HEAD

Limit how much a single transaction can reserve space --HG-- branch : HEAD

Cache fixes. Decisions are saved again. --HG-- branch : HEAD

Header caching redesigned. New design allows caching decisions per field, so they can be divided to temporary/permanent. Cached headers are now always returned in original order, old code didn't guarantee it. Some other caching changes. (still missing code to store changes in caching decisions) --HG-- branch : HEAD

Cache fixes. Lookups now look into transactions too. --HG-- branch : HEAD

Cache file fixes, API changes, etc. It's still in somewhat ugly state, but getting better.. --HG-- branch : HEAD

Cache updating is done now by first reserving space where to write, and then writing to it whenever buffer gets full. There is no persistent cache file locks anymore because of this, but it also means that the same cached field may be written multiple times to the file by different processes. Also since we reserve more space than we actually need at first, it some space can be wasted if multiple processes are updating the cache. --HG-- branch : HEAD

Caching fixes and optimizations. Removed all network byte ordering code - it's not worth the trouble and would require massive changes in indexing code as well to be useful. Changed next_offset to prev_offset which is updated while syncing index. --HG-- branch : HEAD

Cache file compression works now and compressed cache file is reopened. Several other cleanups related to opening - cache file isn't created immediately anymore. --HG-- branch : HEAD

Added some smartness for deciding what to cache. Cache compression code compiles, but untested. --HG-- branch : HEAD

Several fixes and cleanups to cache file code, still badly broken --HG-- branch : HEAD

Beginnings of getting cache file working again. Easy to break currently, but basics seem to work. --HG-- branch : HEAD

importing new index code. mbox still broken. --HG-- branch : HEAD