global: Fix spelling mistakes in comments Original work by @andreasschulze and @jsoref

lib-index: Fix mail_index_get_modification_time() to work when index isn't open. index->filepath may be NULL after a failed index open, and it's a bit unsafe to trust that index->log->filepath isn't NULL either. So just build the full path from elements that are definitely non-NULL. Also stat() only dovecot.index.log, because it's always supposed to exist. If it doesn't, something's broken and stat()ing dovecot.index doesn't make much sense. This commit removes mail_transaction_log_get_mtime(), which is no longer needed. Fixes: Panic: file mail-index.c: line 931 (mail_index_file_set_syscall_error): assertion failed: (filepath != NULL)

lib-index: Don't ignore mail_transaction_log_move_to_memory() failure

lib-index: Don't increase modseq for backend/dirty flag changes These flags are used only for internal changes and they shouldn't be triggering any modseq changes. To avoid modseqs from unexpectedly shrinking, the new modseq counting behavior is enabled only for newly rotated transaction log files that have a new minor_version.

lib-index: Add macros to check if transaction log file version is new enough. The version numbers are 8bit, so we can generate a single 16bit number out of them.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib-index: Small improvements to logging the reason for long transaction log lock waits

lib-index: Added reason_r parameter to mail_transaction_log_view_set() This is used to improve some of the error messages about index corruption.

lib-index: Include reason string in warnings about keeping transaction log locked for too long.

lib-index: Don't update log_file_tail_offset unnecessarily. Update it only if we're already writing to transaction log anyway or if we're required to update the offset because mail_index_sync_commit() has increased it past non-external transactions (this is especially important with mdbox map index).

lib-index: Backported MAIL_TRANSACTION_ATTRIBUTE_UPDATE from v2.2

lib-index: mail_index_attribute_[un]set() adds changed attributes' keys to transaction log. This provides them both a modseq (so their changes become visible) as well as an efficient way to see what attributes have changed by reading the transaction log. The values themselves aren't written to the log, because they could be large.

lib-index: Write to transaction log using O_APPEND flag. Most OSes should do the writes atomically so that reads won't see partially written data. We don't currently rely on this, but it would be possible in future to remove locking entirely from writing to transaction log.

lib-index: Support clearing the whole log view with mail_transaction_log_view_clear(0)

shared mailboxes: Per-user flags can now be stored in private index files. This can be enabled by adding e.g.: mail_location = mdbox:/var/shared/mdbox:INDEXPVT=~/mdbox/shared

Added MAIL_INDEX_TRANSACTION_FLAG_SYNC

lib-index: Added mail_transaction_log_view_set_all()

lib-index: Handle transaction log read errors separately from log corruption.

lib-index: When marking index corrupted, delete also transaction log file.

Moved GUID code to liblib. Use guid_128_t type consistently everywhere.

lib-index: Give better assert-crashes if view or log view isn't closed. Earlier it should have also assert-crashed with "log->files == NULL", which didn't make it clear enough why the files aren't NULL.

lib-index: Added "transaction boundary" record for future compatibility.

Removed mail_update_uid() / mail_index_update_uid(). It was working properly only with mdbox and there wasn't really a need for it. --HG-- branch : HEAD

lib-index: Added mail_index_get_modification_time(). --HG-- branch : HEAD

lib-index: Added support for undeleting a deleted index. --HG-- branch : HEAD

Added support for marking mailbox index deleted. Don't allow any changes after that. This is going to help with race conditions when deleting mailboxes. --HG-- branch : HEAD

lib-index: MAIL_TRANSACTION_TYPE_MASK didn't contain the new _EXT_HDR_UPDATE32. --HG-- branch : HEAD

lib-index: Updated MAIL_TRANSACTION_EXT_MASK to include thew new _EXT_HDR_UPDATE32. --HG-- branch : HEAD

mail_index_update_header_ext(): Couldn't handle >=64k headers. --HG-- branch : HEAD

Added ability to specify message's minimum modseq value. --HG-- branch : HEAD

Added ability to change existing messages' UIDs with mail_[index_]update_uid(). --HG-- branch : HEAD

Added mail_transaction_log_get_tail(). --HG-- branch : HEAD

Keep track of expunged messages' GUIDs and expose them via mailbox_get_expunges(). The message GUIDs are stored in expunge records to transaction log. Before doing the final expunge, Maildir and dbox verify that the GUID in expunge request matches the current actual GUID. The GUID is stored in 128 bit field. If the real GUID isn't 128 bit, the bits are taken from SHA1 of the GUID. --HG-- branch : HEAD

Added mail_index_unlink(). --HG-- branch : HEAD

Tests are now run on "make check". Added initial tests for lib-index. Moved old tests away from tests/ to libraries' individual directories. --HG-- branch : HEAD

More cleanups to transaction log writing code. --HG-- branch : HEAD

Moved transaction commiting code to mail-index-transaction-*.c --HG-- branch : HEAD

mail_transaction_log_view_clear() should keep oldest_file_seq and newer referenced. --HG-- branch : HEAD

Added mail_index_atomic_inc_ext() for atomically incrementing numbers in extensions. --HG-- branch : HEAD

Comment update. --HG-- branch : HEAD

Added padding to struct mail_transaction_log_header so it's 64bit aligned. Most compilers did that automatically already. --HG-- branch : HEAD

Replaced "no extension resizing" flag with "no extension shrinking". It fixes some bugs while still handling the original reason it was added. --HG-- branch : HEAD

Write CPU endianess to transaction log header and check it's correct when reading. --HG-- branch : HEAD

Mail index extension introductions now add a "no resizes" flag to the intro records. If it's found, the header and record sizes are used only if the extension doesn't exist yet, otherwise they're ignored. --HG-- branch : HEAD

Modseqs are no longer calculated from transaction log sequence + offset. Now they begin from 1 and each "visible" transaction increases it by one. --HG-- branch : HEAD

Fixes to handling races in initial index creation. --HG-- branch : HEAD

Added mail_index_ext_reset_inc() to atomically increase extension's reset_id. Added clear_data parameter to mail_index_ext_reset*(). --HG-- branch : HEAD

Keep transaction logs referenced while we know we need them. This fixes desync errors with in-memory indexes. --HG-- branch : HEAD

Renamed __attr_*__ to ATTR_*. Renamed __attrs_used__ to ATTRS_DEFINED. --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

Added support for resetting index. --HG-- branch : HEAD

Handle indexid changes better. --HG-- branch : HEAD

int/ext/mailbox sync offset changes: Combined mailbox and int offsets to "tail" offset and renamed ext offset to "head". This makes it clearer how they're supposed to be used. --HG-- branch : HEAD

Removed syncs_done array. Instead track appends and expunges separately. Also fixed handling no-appends in some situations. --HG-- branch : HEAD

Removed sync_mask parameter from mail_transaction_log_view_set(). The user will now have to check the types itself. --HG-- branch : HEAD

mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to -log-file.c. --HG-- branch : HEAD

Initial commit for major index file code cleanup. - dovecot.index file isn't anymore required to be updated when syncing. - Getting the latest index file mapping is now done always by reading dovecot.index and then reading the latest changes from dovecot.index.log. - mmap()ing dovecot.index file is slower than reading it, so it's not currently done unless the file is at 256kB. This may change though. - Some things are still broken. --HG-- branch : HEAD

Added mail_transaction_log_view_is_last() --HG-- branch : HEAD

If we run out of disk space, move to in-memory indexes. --HG-- branch : HEAD

If index file is corrupted, recreate transaction log file instead of trying to reuse it. --HG-- branch : HEAD

deinit, unref, destroy, close, free, etc. functions now take a pointer to their data pointer, and set it to NULL. This makes double-frees less likely to cause security holes. --HG-- branch : HEAD

Added "bool" type and changed all ints that were used as booleans to bool. --HG-- branch : HEAD

Beginnings of fallbacking to in-memory indexes when write fails with "out of disk space" error. --HG-- branch : HEAD

Handle missing/broken transaction logs better. Handle broken sync position in index header better. --HG-- branch : HEAD

Several transaction log cleanups and fixes. --HG-- branch : HEAD

We could have gone past the transaction log view's boundaries if log was just rotated. Plus other transaction log cleanups, fixes and new asserts. --HG-- branch : HEAD

Added versioning to transaction log header. Added create_stamp to its header and it's not used to determine if the transaction log should be rotated, not the last time the log was updated. So now if the log is over 128kB, it's rotated as soon as it's at least 5 minutes old. --HG-- branch : HEAD

Internal changes in how keywords are handled. struct mail_keywords isn't automatically freed anymore, added *_keywords_free() for that. --HG-- branch : HEAD

Added padding field explicitly for struct mail_transaction_flag_update. It was done before anyway by compiler. --HG-- branch : HEAD

Initial support for keywords. Syncing to mbox/maildir doesn't work yet. --HG-- branch : HEAD

Removed cache_offset from mail_index_record and changed it to use extension instead. Added possibility to register sync and expunge handlers for extensions. Changed the way extension resets work: all extension updates which were committed without having seen the reset are ignored. --HG-- branch : HEAD

New way of handling extension introductions in transaction log. --HG-- branch : HEAD

Index extensions can now specify record alignment, and it's possible to resize header and records. Fixes mbox crashes with some 64bit systems. --HG-- branch : HEAD

Renamed "extra record info" and variations of it to "extension" or "ext" in short. --HG-- branch : HEAD

Save extra record/header infos into index file permanently. --HG-- branch : HEAD

Small cleanups. --HG-- branch : HEAD

Transaction log file is now read-lockless. --HG-- branch : HEAD

Record size is allowed to change between index files. This will allow adding extensions dynamically for existing indexes. --HG-- branch : HEAD

Several fixes and cleanups to cache file code, still badly broken --HG-- branch : HEAD

Modifying extra_records should work now. --HG-- branch : HEAD

Keep the transaction log view open all the time for index views (not just while syncing) to avoid losing the log files. --HG-- branch : HEAD

Index header changes now go through transaction log. Removed the kludgy parameters for mail_index_sync_end(). Removed code duplication of syncing index root mapping and view mapping. Some fixes to handling uidvalidity and nextuid in syncing. --HG-- branch : HEAD

Transaction log contains only UIDs now, no more sequences which just mess up everything. --HG-- branch : HEAD

When committing with no changes, don't try to sync them later. --HG-- branch : HEAD

Don't rotate transaction log too often --HG-- branch : HEAD

s/custom flags/keywords/ --HG-- branch : HEAD

handle losing index file --HG-- branch : HEAD

automatically fix broken/missing transaction log files on the fly --HG-- branch : HEAD

fixes --HG-- branch : HEAD

Don't modify index file when creating new transaction log. --HG-- branch : HEAD

importing new index code. mbox still broken. --HG-- branch : HEAD