lib-index: Improve error messages when transaction log is unexpectedly lost

lib-index: Fixes to handling resized records.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Easy way to update your existing code: perl -i -pe 's:ARRAY_DEFINE\(([^,]+), *([^)]+)\);:ARRAY($2) $1;:' **/*.[ch]

lib-index: Updated dovecot.index file only by recreating it, never write to it directly. This is safer, and nowadays there shouldn't be much of a performance loss with it either, since dovecot.index isn't updated very often. This also allows removing all locking from dovecot.index file, although for now we'll keep it in case old Dovecot versions are simultaneously writing to the index.

lib-index: mail_index_sync_ext_atomic_inc() does now better error checking. --HG-- branch : HEAD

mail_index_update_header_ext(): Couldn't handle >=64k headers. --HG-- branch : HEAD

Added mail_index_atomic_inc_ext() for atomically incrementing numbers in extensions. --HG-- branch : HEAD

Initial CONDSTORE support. --HG-- branch : HEAD

Reverted fix for dropping expunged messages from flag updates. It's more common that the UID range was actually meant to include the expunged message. --HG-- branch : HEAD

If a message was expunged but is still visible in a view, its flags/keywords might have been updated if an uid range included the expunged message. --HG-- branch : HEAD

Renamed __attr_*__ to ATTR_*. Renamed __attrs_used__ to ATTRS_DEFINED. --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

Make sure extensions updates don't modify keywords. --HG-- branch : HEAD

Only index extension indexes are now called "ext_id". For map extension indexes use "map_ext_idx". --HG-- branch : HEAD

Complain about unknown extension introduction IDs in log only once per sync. --HG-- branch : HEAD

Prevent copying record data when appending new messages. --HG-- branch : HEAD

mail_index_map now contains pointer to index, so all functions taking map parameter don't need index anymore. --HG-- branch : HEAD

Handle losing index file better. Handle fsck better. --HG-- branch : HEAD

DEBUG: Verify header counters/lowwaters more often. --HG-- branch : HEAD

Update log_file_head_offset when replacing current syncing map. --HG-- branch : HEAD

View's counters can no longer be unreliable or broken. --HG-- branch : HEAD

Removed unused/unneeded code. --HG-- branch : HEAD

Initial commit for major index file code cleanup. - dovecot.index file isn't anymore required to be updated when syncing. - Getting the latest index file mapping is now done always by reading dovecot.index and then reading the latest changes from dovecot.index.log. - mmap()ing dovecot.index file is slower than reading it, so it's not currently done unless the file is at 256kB. This may change though. - Some things are still broken. --HG-- branch : HEAD

mail_index_sync_begin() returns now transaction directly so the syncing code doesn't need to create it. It's also automatically committed/rollbacked. --HG-- branch : HEAD

Don't set log sync position right after extension introduction, unless it's at the end of the file. This should fix some "Extension reset without intro prefix" errors. --HG-- branch : HEAD

Some mmap_disable=no fixes. Some debug additions. Some generic index fixes. --HG-- branch : HEAD

Keywords were sometimes being re-added to the index even while they already existed, causing the record size to bloat. --HG-- branch : HEAD

Added support for calling expunge handler for an extension even if the extension isn't used in the opened index file. Added context-parameter to the callback function. If the function returns -1, the sync is failed. --HG-- branch : HEAD

Array API redesigned to work using unions. It now provides type safety without having to enable DEBUG, as long as the compiler supports typeof(). Its API changed a bit. It now allows directly accessing the array contents, although that's not necessarily recommended. Changed existing array usage to be type safe in a bit more places. Removed array_t completely. Also did s/modifyable/modifiable/. --HG-- branch : HEAD

If we're synchronizing a view (and not index itself) from transaction log, give a different error message and don't mark the file corrupted. --HG-- branch : HEAD

Index header's seen/recent/deleted counters weren't always correct after syncing a view. This caused some problems with recent SEARCH optimizations, eg. EXPUNGE not always working. --HG-- branch : HEAD

Added "bool" type and changed all ints that were used as booleans to bool. --HG-- branch : HEAD

Changed mail_index_view_sync_next() to return struct mail_index_view_sync_rec records which only contain the type and UID range. Makes it clearer that the caller won't get anything else, and flag updates now don't need to update add_flags/remove_flags fields. --HG-- branch : HEAD

Keywords are now stored in X-Keywords headers in mbox. Did several related API changes to get better performance. --HG-- branch : HEAD

Added ARRAY_DEFINE_PTR() which needs to be used instead of ARRAY_DEFINE() for pointers. --HG-- branch : HEAD

mail_index_sync_next() now returns keyword updates (now only thing left to do is to fix maildir and mbox syncing to use them). --HG-- branch : HEAD

Changed many buffers to arrays. Cleans up the code a lot. --HG-- branch : HEAD

Internal changes in how keywords are handled. struct mail_keywords isn't automatically freed anymore, added *_keywords_free() for that. --HG-- branch : HEAD

Recent flags weren't being removed from index causing needless syncing. --HG-- branch : HEAD

Instead of using separate transaction sorting code for syncing, just put the data from transactions into a temporary transaction and read it from there. --HG-- branch : HEAD

Keyword fixes. --HG-- branch : HEAD

Initial support for keywords. Syncing to mbox/maildir doesn't work yet. --HG-- branch : HEAD

Moved extension syncing code to separate file. --HG-- branch : HEAD

Cache invalidating needed to hook into view syncing as well. --HG-- branch : HEAD

Removed cache_offset from mail_index_record and changed it to use extension instead. Added possibility to register sync and expunge handlers for extensions. Changed the way extension resets work: all extension updates which were committed without having seen the reset are ignored. --HG-- branch : HEAD

Use separate sync offsets for internal/external transactions. Pending external transactions are committed into index at the beginning of syncing, internal ones aren't. --HG-- branch : HEAD

New way of handling extension introductions in transaction log. --HG-- branch : HEAD

When we had dirty messages, we kept resyncing the whole mailbox constantly. --HG-- branch : HEAD

fixing crashes from last changes --HG-- branch : HEAD

Caching fixes and optimizations. Removed all network byte ordering code - it's not worth the trouble and would require massive changes in indexing code as well to be useful. Changed next_offset to prev_offset which is updated while syncing index. --HG-- branch : HEAD

Record size is allowed to change between index files. This will allow adding extensions dynamically for existing indexes. --HG-- branch : HEAD

fixes --HG-- branch : HEAD

Index header changes now go through transaction log. Removed the kludgy parameters for mail_index_sync_end(). Removed code duplication of syncing index root mapping and view mapping. Some fixes to handling uidvalidity and nextuid in syncing. --HG-- branch : HEAD

Set dirty flags through transaction log, not directly. Some other flag fixes etc. --HG-- branch : HEAD

Transaction log contains only UIDs now, no more sequences which just mess up everything. --HG-- branch : HEAD

syncing fixes --HG-- branch : HEAD

Added support for setting dirty flags for messages (TODO: undirty..) s/mail_index_record_flag/mail_cache_record_flag/ --HG-- branch : HEAD

Syncing optimizations. --HG-- branch : HEAD

importing new index code. mbox still broken. --HG-- branch : HEAD