lib-index: Add mail_index_transaction_commit_result.changes_mask This can be used to determine what type of changes were committed in a transaction.

lib-index: Add mail_index.event

lib-index: Add mail_index_set_cache_dir() to change .cache's directory

global: Fix spelling mistakes in comments Original work by @andreasschulze and @jsoref

lib-index: Add mail_index_cache_optimization_settings

lib-index: Add mail_index_base_optimization_settings

lib-index: Replace mail_index_set_log_rotation() with mail_index_set_optimization_settings() This allows more easily adding optimization-related settings.

lib-index: Remove size from struct mail_keywords.idx[] This was confusing static analyzers, which thought that using [1] meant that its size really was 1.

lib-index: If mail_index_view is leaked, include in Panic the file:line where it was opened

lib-index: Add mail_index_use_existing_permissions() This has annoyingly lot of copy&pasting from mailbox_list_get_permissions_stat(), but there didn't seem to be any nice place where to share the code.

lib-index: Fix mail_index_get_modification_time() to work when index isn't open. index->filepath may be NULL after a failed index open, and it's a bit unsafe to trust that index->log->filepath isn't NULL either. So just build the full path from elements that are definitely non-NULL. Also stat() only dovecot.index.log, because it's always supposed to exist. If it doesn't, something's broken and stat()ing dovecot.index doesn't make much sense. This commit removes mail_transaction_log_get_mtime(), which is no longer needed. Fixes: Panic: file mail-index.c: line 931 (mail_index_file_set_syscall_error): assertion failed: (filepath != NULL)

lib-index: Track .log.2 rotation time in index header This avoids unnecessarily stat()ing the file. Also it's a bit better since it's tracking the actual rotation time, not the mtime of what the .log file happened to have at the time of rotation. The initial rotation timestamp is written only to the dovecot.index header without going through dovecot.index.log. This works, because the dovecot.index is written practically always after a log rotation. For the rare cases when it doesn't happen, the dovecot.index.log.2 just gets deleted later after the next log rotation.

lib-index, lib-storage: Add mail_index_header.last_temp_file_scan Also add index_mailbox_update_last_temp_file_scan() for easily updating it. This is reusing an old "sync timestamp" field. Because it was a timestamp, it doesn't matter if the old data still exists in it. This field could have been added as an extension, but that's more work and this feature is generic enough that it should be useful for many of the mail storage backends.

lib-index: Add MAIL_INDEX_OPEN_FLAG_NO_DIRTY This way mailbox format backends that don't need dirty flags can use them for other purposes.

lib-index: Add mail_index_revert_changes() This can be used to revert changes done in a transaction to the specified existing mail.

lib-index: Add mail_index_unset_fscked() This can be used to easily remove MAIL_INDEX_HDR_FLAG_FSCKD. It takes a transaction parameter instead of sync_ctx because some index rebuilds are done with a separate transaction while the sync_ctx is rolled back.

lib-index: fsck now adds MAIL_INDEX_HDR_FLAG_FSCKD to header. It can only be removed by an explicit header update.

lib-index: Add mail_index_set_log_rotation()

lib-index: Added mail_index_transaction_get_highest_modseq()

lib-index: Don't waste 3 bytes of space in struct mail_index_record. It was always supposed to be used mainly for keywords, but it got broken many years ago and nobody noticed.

lib-index: If mail_debug=yes, log cache compressions.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib-index: Added mail_index_sync_set_reason() to improve lock wait log warnings

lib-index: Added mail_index_sync_have_any_expunges() This can be used to quickly check before mail_index_sync_begin() if there are likely to be any expunges that will be synced.

lib-index: Added mail_index_ext_register_resize_defaults()

lib-index: Avoid writing tail offset update to transaction log if it's not necessary. This should avoid extra writes that do nothing but update the tail offset. mdbox's map requires this behavior, so disable it for it. For others it might cause the next sync to read more data unnecessarily, but it should be worth the extra cost of write most of the times.

lib-index: Added mail_index_sync_no_warning() to prevent "long transaction lock" warnings. Use it with Maildir to prevent double-warning.

lib-index: Moved MAIL_TRANSACTION_LOG_LOCK_WARN_SECS as part of public API.

lib-index, lib-storage: Fixed race conditions with deleting mailbox. Now only one process can successfully finish mailbox_mark_index_deleted().

lib-index: Added mail_index_ext_resize_hdr()

lib-index: Add timestamps and value lengths to attribute change records in transaction log. The timestamps will be useful for dsync, and value lengths will be useful for metadata quota.

lib-index: mail_index_attribute_[un]set() adds changed attributes' keys to transaction log. This provides them both a modseq (so their changes become visible) as well as an efficient way to see what attributes have changed by reading the transaction log. The values themselves aren't written to the log, because they could be large.

Replaced all -1U and (unsigned int)-1 with UINT_MAX. It's somewhat clearer this way. Also clang's -fsanitize=integer gives runtime errors about -1U (but not about explicit casts, so no need to change (type)-1 casts).

lib-index: Make sure unused_old_sync_* fields are cleared in header in old inedx files.

lib-index: Comment update

Treat modseq updates explicitly instead of as if they were flag changes.

lib-index: Added MAIL_INDEX_OPEN_FLAG_SAVEONLY to do only minimal reads from cache file.

lib-index: Removed MAIL_INDEX_SYNC_TYPE_APPEND. No backend cares about it.

lib-index: Changed mail_index_set_fsync_mode() to use a separate enum for the mask.

Avoid using (void)s by adding ATTR_NOWARN_UNUSED_RESULT attributes and other ways.

Marked functions parameters that are allowed to be NULL. Some APIs were also changed. The non-obvious APIs where NULL parameter was changed to "" are master_service_init() and auth_master_user_list_init(). These checks can currently be enabled only on a patched clang: http://llvm.org/bugs/show_bug.cgi?id=6786

shared mailboxes: Per-user flags can now be stored in private index files. This can be enabled by adding e.g.: mail_location = mdbox:/var/shared/mdbox:INDEXPVT=~/mdbox/shared

Added MAIL_INDEX_TRANSACTION_FLAG_SYNC

Don't write "keyword reset" records to transaction log anymore. These are a bit problematic for dsync's keyword merging to handle.

lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. This can be used to avoid race conditions on mailbox creation for mailbox formats that require this (sdbox).

lib-index: Added mail_index_reset_fscked()

Moved GUID code to liblib. Use guid_128_t type consistently everywhere.

lib-index: MAIL_INDEX_SYNC_FLAG_DELETING_INDEX comment updated

lib-index: sync_stamp and sync_size fields are unused, mark them as such.

Removed mail_update_uid() / mail_index_update_uid(). It was working properly only with mdbox and there wasn't really a need for it. --HG-- branch : HEAD

lib-index: Added mail_index_sync_has_expunges() --HG-- branch : HEAD

Renamed fsync_disable to mail_fsync=optimized|always|never. --HG-- branch : HEAD

Added mail_max_lock_timeout setting. This could be useful inside lda/lmtp protocol sections to avoid them from spending too much time waiting for locks, since they can easily just return "try again later". --HG-- branch : HEAD

lib-index: Added mail_index_view_get_transaction_count(). --HG-- branch : HEAD

lib-index: Added mail_index_get_modification_time(). --HG-- branch : HEAD

Mailbox deletion: Fixed race condition where a mailbox couldn't get deleted. --HG-- branch : HEAD

lib-index: Added support for undeleting a deleted index. --HG-- branch : HEAD

Added support for marking mailbox index deleted. Don't allow any changes after that. This is going to help with race conditions when deleting mailboxes. --HG-- branch : HEAD

Transaction commits can now track how many uid/modseq updates were ignored. --HG-- branch : HEAD

lib-index: Added mail_index_update_highest_modseq(). --HG-- branch : HEAD

Added ability to specify message's minimum modseq value. --HG-- branch : HEAD

Changed mail_index_append_assign_uids() to mail_index_append_finish_uids() with API changes. It's now possible to call mail_index_append() with UIDs before locking mailbox and have the _finish_uids() change them if another session had already used those UIDs. --HG-- branch : HEAD

Added ability to change existing messages' UIDs with mail_[index_]update_uid(). --HG-- branch : HEAD

Keep track of expunged messages' GUIDs and expose them via mailbox_get_expunges(). The message GUIDs are stored in expunge records to transaction log. Before doing the final expunge, Maildir and dbox verify that the GUID in expunge request matches the current actual GUID. The GUID is stored in 128 bit field. If the real GUID isn't 128 bit, the bits are taken from SHA1 of the GUID. --HG-- branch : HEAD

Whenever file's group changing fails, show the group origin in the error message. --HG-- branch : HEAD

Added reference counting to struct mail_keywords and related APIs. --HG-- branch : HEAD

Added mail_index_unlink(). --HG-- branch : HEAD

Removed duplicate mail_index_is_in_memory(). --HG-- branch : HEAD

maildir: When saving messages, in some race conditions we might have written a duplicate UID. --HG-- branch : HEAD

Added mail_index_lookup_view_flags() --HG-- branch : HEAD

Comment update. --HG-- branch : HEAD

mail_index_atomic_inc_ext() now returns the current diff sum. --HG-- branch : HEAD

dbox: Make sure indexes are never moved to memory (in case of out-of-disk space). --HG-- branch : HEAD

Added FSYNC flag for transactions. --HG-- branch : HEAD

Added mail_index_sync_get_offsets(). --HG-- branch : HEAD

dbox: Create dovecot.index.backup files. --HG-- branch : HEAD

Removed rarely used parameters from mail_index_transaction_commit(). mail_index_transaction_commit_get_pos() can be used where they're required. --HG-- branch : HEAD

Added mail_index_open_or_create(). --HG-- branch : HEAD

Added mail_index_atomic_inc_ext() for atomically incrementing numbers in extensions. --HG-- branch : HEAD

CONDSTORE: STORE UNCHANGEDSINCE conflicts are now checked atomically. --HG-- branch : HEAD

Added mail_index_map_lookup_keywords(). --HG-- branch : HEAD

Changed mail index view syncing API to return only flag changes. Currently it has only a single caller and it's not interested in anything else. --HG-- branch : HEAD

mail_index_view_sync_begin() now delays its failures to _commit(). --HG-- branch : HEAD

QRESYNC: If MODSEQs were returned in FETCH replies but there are pending expunges, send a low enough HIGHESTMODSEQ reply to make sure the client will later see the expunges. --HG-- branch : HEAD

Added mail_index_ext_reset_inc() to atomically increase extension's reset_id. Added clear_data parameter to mail_index_ext_reset*(). --HG-- branch : HEAD

Make sure MAIL_RECENT flag is cleared from index file's flags so it can be used in future for other purposes. Increased minor version number because of this. --HG-- branch : HEAD

View sync returns now also hidden records, but they're marked as such. Mailbox syncing returns hidden records as modseq changes. --HG-- branch : HEAD

Initial CONDSTORE support. --HG-- branch : HEAD

Added MAIL_INDEX_MAIL_FLAG_BACKEND flag that can be used for backend-specific uses. --HG-- branch : HEAD

Moved fdatasync_path() to a global function. --HG-- branch : HEAD

Moved keyword_indexes array type to lib-mail. --HG-- branch : HEAD

Added MAIL_INDEX_SYNC_FLAG_REQUIRE_CHANGES flag and changed mail_index_sync_begin() to return 1 at success and 0 if there's nothing to do. --HG-- branch : HEAD

fsync transaction commits only if the transaction contains change types specified by mail_index_set_fsync_types(). --HG-- branch : HEAD

Added MAIL_INDEX_OPEN_FLAG_READONLY. --HG-- branch : HEAD

Added mail_index_ext_lookup() --HG-- branch : HEAD

Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made it return a bool and added mail_index_lookup_seq(). Cleaned up the code to use these functions. --HG-- branch : HEAD

Added MAIL_INDEX_VIEW_SYNC_FLAG_FIX_INCONSISTENT flag. --HG-- branch : HEAD

Don't forward declare enums. It doesn't seem to work with all compilers. --HG-- branch : HEAD

Added mail_index_sync_have_any(). --HG-- branch : HEAD

Changed .h ifdef/defines to use <NAME>_H format. --HG-- branch : HEAD

Removed MAIL_INDEX_HDR_FLAG_FSCK. It's not used anymore. --HG-- branch : HEAD

fsck won't fail anymore with "corrupted index", all problems are fixed. Added mail_index_fsck_locked(). --HG-- branch : HEAD

Changed mail_keywords creation APIs to take mailbox/index instead of transaction. --HG-- branch : HEAD

Removed autocreate parameter from mail_index_keyword_lookup(). Added a new mail_index_keyword_lookup_or_create(). --HG-- branch : HEAD

Renamed mail_index_sync_begin() to mail_index_sync_begin_to() and added a new mail_index_sync_begin() without seq/offset parameters. --HG-- branch : HEAD

mail_index_view_sync_next() and mailbox_sync_next() returns now bool. Renamed void mailbox_index_view_sync_end() to int mailbox_index_view_sync_commit(). --HG-- branch : HEAD

mail_index_sync_next() can't fail anymore. --HG-- branch : HEAD

mail_index_lookup*() can't fail anymore. Changed several APIs not to return failure anymore. --HG-- branch : HEAD

Removed explicit locking from views and maps. They were already locked all the time when they were used. Because of this change several functions can no longer fail, so they were changed to return void, and a lot of pointless error handling was removed. --HG-- branch : HEAD

Updated mail_index_sync_begin() comments. --HG-- branch : HEAD

mail_index_ext_get_reset_id() takes now map parameters. Fixed it to work properly with transaction views. --HG-- branch : HEAD

Added mail_index_ext_get_reset_id() which returns the latest reset_id when using transaction views. --HG-- branch : HEAD

Added MAIL_INDEX_TRANSACTION_FLAG_AVOID_FLAG_UPDATES and MAIL_INDEX_SYNC_FLAG_AVOID_FLAG_UPDATES flags to make mail_index_update_flags() not do anything if the flags in view already are those. --HG-- branch : HEAD

Changed mail_transaction_begin() API to take flags parameter instead of two booleans. --HG-- branch : HEAD

Recent flag handling rewrite. Still not perfect with maildir. --HG-- branch : HEAD

Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. --HG-- branch : HEAD

Added mail_index_ext_set_reset_id(). --HG-- branch : HEAD

Added support for resetting index. --HG-- branch : HEAD

mail_index_sync_begin() takes now flags parameter instead of two booleans. Cleanups to recent handling. --HG-- branch : HEAD

Give MAIL_INDEX_VIEW_SYNC_FLAG_NOEXPUNGES a non-zero value so it actually works --HG-- branch : HEAD

mail_index_view_sync_begin() can now sync only all or all but expunges. Changed sync_type to be flags. --HG-- branch : HEAD

int/ext/mailbox sync offset changes: Combined mailbox and int offsets to "tail" offset and renamed ext offset to "head". This makes it clearer how they're supposed to be used. --HG-- branch : HEAD

Changed mail_index_view_sync_begin() to take enum mail_index_view_sync_type. This limits how the view can be synced, but we couldn't handle before more than the 3 useful cases anyway. --HG-- branch : HEAD

Added padding to struct mail_index_header so it's 64bit aligned with 32bit systems also. --HG-- branch : HEAD

Initial commit for major index file code cleanup. - dovecot.index file isn't anymore required to be updated when syncing. - Getting the latest index file mapping is now done always by reading dovecot.index and then reading the latest changes from dovecot.index.log. - mmap()ing dovecot.index file is slower than reading it, so it's not currently done unless the file is at 256kB. This may change though. - Some things are still broken. --HG-- branch : HEAD

Added mail_index_transaction_is_expunged() --HG-- branch : HEAD

Typofix: transction -> transaction --HG-- branch : HEAD

Added mail_index_transaction_get_view() --HG-- branch : HEAD

mail_index_sync_begin() returns now transaction directly so the syncing code doesn't need to create it. It's also automatically committed/rollbacked. --HG-- branch : HEAD

Added mail_index_transction_reset(). --HG-- branch : HEAD

Removed mmap_no_write setting. The only OS requiring it is OpenBSD, so we're now forcing mmap_disable=yes with it instead. dovecot.index.cache file is the most important file to mmap(), but since this didn't work with mmap_no_write, there's not much point in keeping special code paths for minimal gains. --HG-- branch : HEAD

Removed mail_index_get_last_error() and MAIL_INDEX_ERROR_*. All the index errors are internal errors. --HG-- branch : HEAD

Better type safety to module_contexts arrays. Already fixed some bugs. --HG-- branch : HEAD

Added dotlock_use_excl setting. --HG-- branch : HEAD

Added fsync_disable setting. Also added missing fsync()ing to dbox when saving mails. --HG-- branch : HEAD

Lock handling changes. Everything goes through file-lock API now and there's only a single enum listing the different lock methods. This change exposed some unneeded (or possibly even wrong?) unlock calls in index file handling which were fixed. --HG-- branch : HEAD

Try to avoid sending duplicate/useless flag updates. --HG-- branch : HEAD

Removed MAIL_INDEX_OPEN_FLAG_FAST. It wasn't being used, and we probably want to have that as a default anyway. --HG-- branch : HEAD

Beginnings of joining mail-storage API more closely to mail-index, so that mail-index could be directly used if needed. Currently only transactions are joined. --HG-- branch : HEAD

Array API redesigned to work using unions. It now provides type safety without having to enable DEBUG, as long as the compiler supports typeof(). Its API changed a bit. It now allows directly accessing the array contents, although that's not necessarily recommended. Changed existing array usage to be type safe in a bit more places. Removed array_t completely. Also did s/modifyable/modifiable/. --HG-- branch : HEAD

Fixed mail_index_move_to_memory() to work better and made it public --HG-- branch : HEAD

deinit, unref, destroy, close, free, etc. functions now take a pointer to their data pointer, and set it to NULL. This makes double-frees less likely to cause security holes. --HG-- branch : HEAD

Added "bool" type and changed all ints that were used as booleans to bool. --HG-- branch : HEAD

mail_index_ext_get_size() hadn't been implemented even though it was in mail-index.h. Implemented it and changed mail_index_lookup_keywords() to use it. --HG-- branch : HEAD

Changed mail_index_view_sync_next() to return struct mail_index_view_sync_rec records which only contain the type and UID range. Makes it clearer that the caller won't get anything else, and flag updates now don't need to update add_flags/remove_flags fields. --HG-- branch : HEAD

Comment updates, small cleanups and optimizations. --HG-- branch : HEAD

We assumed that keyword index arrays were always sorted. This isn't always the case. Caused unneeded keyword updates. --HG-- branch : HEAD

Moved uoff_t and time_t compatibility checks to cache file, since they're not needed for index/log. Only index file incompatibility check is now endianess flag. --HG-- branch : HEAD

Check broken flag counter values every time when updating them. Added fsck-flag to index header, if it's set do fsck when opening index and when syncing. --HG-- branch : HEAD

If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all existing messages and update uidvalidity/nextuid fields. Now we don't have to re-login when this happens. --HG-- branch : HEAD

Several size_t -> unsigned int changes. Try to keep "unsigned int" for counters, size_t for actual memory range sizes. --HG-- branch : HEAD

Added mail_index_sync_reset() to restart syncing from beginning. --HG-- branch : HEAD

Keywords are now stored in X-Keywords headers in mbox. Did several related API changes to get better performance. --HG-- branch : HEAD

Changed keywords_buf to array. Added mail_index_sync_get_keywords(). --HG-- branch : HEAD

mail_index_sync_next() now returns keyword updates (now only thing left to do is to fix maildir and mbox syncing to use them). --HG-- branch : HEAD

Internal changes in how keywords are handled. struct mail_keywords isn't automatically freed anymore, added *_keywords_free() for that. --HG-- branch : HEAD

Added mail_index_update_flags_range() and optimized the non-range version as well. --HG-- branch : HEAD

Keyword fixes. --HG-- branch : HEAD

Initial support for keywords. Syncing to mbox/maildir doesn't work yet. --HG-- branch : HEAD

Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public mail_index_refresh() which makes sure index is fully refreshed at the time. Added mbox code to call it after mbox is locked to avoid using old mbox offsets. --HG-- branch : HEAD

Removed view->messages_count, view->hdr.messages_count is enough. Also fixes assert crash in mail_index_bsearch_uid(). Renamed .._get_message_count() to get_messages_count() to be more consistent. --HG-- branch : HEAD

mail_index_refresh() isn't public anymore, mail_index_view_open_locked() works better for the purpose. --HG-- branch : HEAD

Changed mail_index_get_header() to return the header as return value because it can't fail anymore. --HG-- branch : HEAD

Allow specifying to transaction if it's external. Make mail saving transactions external to avoid problems with uids. --HG-- branch : HEAD

Removed cache_offset from mail_index_record and changed it to use extension instead. Added possibility to register sync and expunge handlers for extensions. Changed the way extension resets work: all extension updates which were committed without having seen the reset are ignored. --HG-- branch : HEAD

Use separate sync offsets for internal/external transactions. Pending external transactions are committed into index at the beginning of syncing, internal ones aren't. --HG-- branch : HEAD

Replaced fcntl_locks_disable with lock_method, so it's now possible to use flock() to lock indexes. --HG-- branch : HEAD

When we had dirty messages, we kept resyncing the whole mailbox constantly. --HG-- branch : HEAD

Index extensions can now specify record alignment, and it's possible to resize header and records. Fixes mbox crashes with some 64bit systems. --HG-- branch : HEAD

Renamed "extra record info" and variations of it to "extension" or "ext" in short. --HG-- branch : HEAD

Get index file mode and GID from dovecot-shared file. --HG-- branch : HEAD

Save extra record/header infos into index file permanently. --HG-- branch : HEAD

Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox deadlocking issue. --HG-- branch : HEAD

Fixed recent flags with mbox. --HG-- branch : HEAD

Removed mail_index_is_in_memory(). It wasn't implemented and it's useless anyway. User doesn't need to see "couldn't use index files" error message. --HG-- branch : HEAD

mail_index_sync_sort_flags() now merges flag changes so mail storage backends don't need to do it (and maildir didn't before). Dirty flags will be synced now too. --HG-- branch : HEAD

UIDs for appended messages can now be assigned all at once. Maildir now updates indexes immediately while saving messages. --HG-- branch : HEAD

Added some smartness for deciding what to cache. Cache compression code compiles, but untested. --HG-- branch : HEAD

Record size is allowed to change between index files. This will allow adding extensions dynamically for existing indexes. --HG-- branch : HEAD

Added mail_index_transaction_get_updated_view() which can be used to access uncommitted messages. --HG-- branch : HEAD

Cache doesn't crash anymore if we're asking it about messages that exist only in uncommitted transactions. --HG-- branch : HEAD

mail_index_lookup() and mail_index_lookup_extra() now returns 0 if message has been expunged, 1 if not. --HG-- branch : HEAD

mailbox_save() and mailbox_copy() functions can now return the saved mail so it can be immediately queried. Implemented UIDPLUS extension using it. Maildir implementation missing, so it crashes with it for now.. APPEND with mbox now doesn't require resyncing the mailbox since it updates indexes directly. --HG-- branch : HEAD

Fixes for extra_records --HG-- branch : HEAD

Modifying extra_records should work now. --HG-- branch : HEAD

Added support for per-index sized mail_index_record. --HG-- branch : HEAD

mail_index_refresh() - allows forcing a refresh check --HG-- branch : HEAD

Index header changes now go through transaction log. Removed the kludgy parameters for mail_index_sync_end(). Removed code duplication of syncing index root mapping and view mapping. Some fixes to handling uidvalidity and nextuid in syncing. --HG-- branch : HEAD

Recent flag fixes. Should work perfectly now with maildir. --HG-- branch : HEAD

Set dirty flags through transaction log, not directly. Some other flag fixes etc. --HG-- branch : HEAD

Transaction log contains only UIDs now, no more sequences which just mess up everything. --HG-- branch : HEAD

comment update --HG-- branch : HEAD

INDEX_KEYWORDS_BYTE_COUNT was counted wrong so index files were larger than they needed to be. Added sizeof(keywords_mask_t) to compat_data. Added limit to growing index file exponentially. --HG-- branch : HEAD

s/custom flags/keywords/ --HG-- branch : HEAD

Added support for setting dirty flags for messages (TODO: undirty..) s/mail_index_record_flag/mail_cache_record_flag/ --HG-- branch : HEAD

Syncing optimizations. --HG-- branch : HEAD

cleanup --HG-- branch : HEAD

remove mail_index_reset() completely --HG-- branch : HEAD

Added fcntl_lock_disable setting to allow indexes to work with NFS. Some other locking fixes. --HG-- branch : HEAD

Forced locking to be right with mprotect()ing index file. Support for disabling mmap for indexes, and disabling just mmap+write(). --HG-- branch : HEAD

importing new index code. mbox still broken. --HG-- branch : HEAD

removed out of the way of new index code --HG-- branch : HEAD

we didn't sync mbox again after rewriting which left us broken mail offsets. --HG-- branch : HEAD

fixed dovecot-uidlist permissions for shared mailboxes --HG-- branch : HEAD

If maildir contains dovecot-shared file Dovecot does two things differently: it allows some flags to be private and stored only in index file (currently hardcoded to \Seen flag only) and new mails are created with dovecot-shared file's mode & 0666. So if you set filesystem permissions correctly, you should have fully functioning shared mailboxes. --HG-- branch : HEAD

We didn't notice cache compression immediately in other processes. Also some other bugfixes. --HG-- branch : HEAD

mbox reading is kind of working again. Just don't try rewriting or expunging :) Changing headers are also hidden from clients so mbox messages are finally seen immutable as required by IMAP. --HG-- branch : HEAD

Added back compat_data[]. It wasn't so simple to access data portably in index files after all.. --HG-- branch : HEAD

Removed index_align stuff. It's not needed anymore. --HG-- branch : HEAD

Removed NBO32_BIT* stuff --HG-- branch : HEAD

Save sync stamp in index header rather than in file's mtime. --HG-- branch : HEAD

Fixes for opening broken cache file. --HG-- branch : HEAD

Maildir syncing works now without requiring base filenames to be in index cache file. Also message flag updates with +FLAGS and -FLAGS works correctly now if another client had just changed it's flags. --HG-- branch : HEAD

Index cache file rewrite. It's not finished yet and mbox support is completely broken. But it's getting difficult to maintain outside cvs :) --HG-- branch : HEAD

API change for expunging messages. Not exactly what I wanted, but good enough. --HG-- branch : HEAD

Removed .imap.index.tree file. For now we'll just rewrite .imap.index file whenever there's expunges. "Expunge binary tree" might be added later, but it's not really useful until you have thousands of messages in mailbox and you're deleting old messages from it. --HG-- branch : HEAD

index_mmap_invalidate = yes now invalidate memory maps before accessing them. Setting this on should fix some problems with OpenBSD. It should also make it possible to use index files over NFS as long as lock daemon is used. It might be such a good idea however. --HG-- branch : HEAD

Only NOOP and CHECK will now always do a real mailbox sync. Other commands will also do it but no often than once in 5 seconds. Also with maildir we don't anymore try to sync it before running commands since syncing is now done automatically whenever we try to access a file that doesn't exist. --HG-- branch : HEAD

Dirty maildir flags support works now. --HG-- branch : HEAD

IMAP protocol doesn't allow server to send NULs to client. Send ascii #128 instead. --HG-- branch : HEAD

Fixes for syncing mails that are kept in new/ dir. --HG-- branch : HEAD

uidlist file is read/updated much less often now. did some other improvements for syncing as well. --HG-- branch : HEAD

Maildir syncing updates: it's now faster, it handles read-only folders, it handles many out of disk space conditions (uidlist doesn't) and all commands finally handle the condition when maildir filename was renamed after our syncing but before we had the chance to open it. --HG-- branch : HEAD

Maildir: CONTROL=<dir> in MAIL environment now specifies where to save .customflags and dovecot-uidlist files. This can be useful if the main mail directory is under hard quota. --HG-- branch : HEAD

Use separate index alignment from memory alignment. We don't use pointers in index file so required alignmentation comes pretty much from sizeof(off_t). --HG-- branch : HEAD

Use mtime to check changes in dovecot-uidlist file rather than inode changes which aren't reliable. --HG-- branch : HEAD

We don't need these fields --HG-- branch : HEAD

Rewritten maildir syncing. Uses dovecot-uidlist file to store UIDs permanently. --HG-- branch : HEAD

UIDs are now saved into mbox file. added a few rewriting optimizations so that we don't always have to rewrite the whole file when updating messages at the beginning of file. --HG-- branch : HEAD

Added index->append_abort() to abort append cleanly. --HG-- branch : HEAD

Index opening rewrites. We don't try to support .imap.index-<hostname> style indexes anymore. They just made things more difficult, and it's really not a good idea to use index files via NFS anyway. Added support for :INDEX=MEMORY in MAIL env to disable using index files. Also if we can't open or create index file for any reason, we now fallback to in-memory indexes. Before we fallbacked only with out of disk space errors. If .customflags can't be created, we still work now by not allowing to create new custom flags. --HG-- branch : HEAD

Locking code cleanups and small fixes --HG-- branch : HEAD

Function typedefs now define them as functions, not function pointers. --HG-- branch : HEAD

Modify log used to test if there were other users for every changed flag or expunged message, causing two fcntl() syscalls every time -> SLOW. Now we check it only once after exclusively locking the index. --HG-- branch : HEAD

Naming change for function typedefs. --HG-- branch : HEAD

We don't actually need to reserve the highest UID anymore, don't do it :) --HG-- branch : HEAD

Reserve UID 2^31-1 --HG-- branch : HEAD

Naming style changes, finally got tired of most of the typedefs. Also the previous enum -> macro change reverted so that we don't use the highest bit anymore, that's incompatible with old indexes so they will be rebuilt. --HG-- branch : HEAD

Replaced MailDataField from being enum to being unsigned int with values defined with macros. Not that pretty, but enums are signed integers and we want to use the highest bit without ugly kludges. --HG-- branch : HEAD

Make sure UIDs won't wrap around unexpectedly. --HG-- branch : HEAD

Added :INDEX=<dir> for both Maildir and mbox to specify different location where to store index files. This would allow keeping mailboxes accessible through NFS but still keep the indexes in fast local disk. Did also some other related cleanups and minor fixes. --HG-- branch : HEAD

Renamed IBuffer and OBuffer to IStream and OStream which describes their functionality better. I tried to keep the variable names and comments also sensible. --HG-- branch : HEAD

We compile now if dev_t is a structure (eg. glibc+tcc). --HG-- branch : HEAD

Locking changes triggered a bit larger cleanup :) If we have to wait for a lock longer, the client is now notified about it every 30 seconds. Also if mailbox opening fails because of lock timeout, we won't overwrite the index anymore. Finally user gets a clear error message about lock timeout instead of "internal error". --HG-- branch : HEAD

MAIL_READ_MMAPED setting was set too late, plus a few cleanups. --HG-- branch : HEAD

mail_read_mmaped: Use mmap() instead of read() to read mail files. read() seems to be a bit faster with my Linux/x86 and it's better with NFS, so that's the default. --HG-- branch : HEAD

Added mbox lock settings to config file. Support timeouting fcntl() and flock() locks. Plus before the fcntl/flocks weren't even set. --HG-- branch : HEAD

mbox_unlock() actually set exclusive lock --HG-- branch : HEAD

And more locking/syncing fixes. Now it's finally beginning to look sane again. --HG-- branch : HEAD

Mostly mbox locking/syncing fixes. Still some problems though. --HG-- branch : HEAD

mbox locking wasn't working. --HG-- branch : HEAD

Moved several fields from .imap.index file to .imap.index.data file. Fixed code so that most of the fields do not need to be set when building index, allowing the index building to be fast (just readdir()s with maildir). This still needs some configuration and ability to update the fields whenever it can grab exclusive lock. Also fixed SEARCH LARGER, SMALLER and KEYWORD. --HG-- branch : HEAD

mbox locking fixes / changes. we now read-lock the mbox file before syncing so we can be sure no-one corrupts our reads by expunging mailbox. mbox file is kept open all the time, but re-opened when device/inode for file has changed (ie. someone overwrote it). File is also kept mmap()ed until lock is dropped. we currently create .lock file even when reading the file (ie. being effectively write-lock), but that'll be made configurable later. --HG-- branch : HEAD

Use fdatasync() instead of fsync() where possible. msync() all files first, then fsync them instead of msync+fsync+msync+fsync+.. data file now keeps track of when it's been changed, tree file didn't do mmap updates, plus other related cleanups. --HG-- branch : HEAD

Added mail_cache_fields and mail_never_cache_fields settings, plus settings cleanup. --HG-- branch : HEAD

cache_fields_later() doesn't need MailIndexRecord parameter. --HG-- branch : HEAD

Added cache_fields_later() to MailIndex interface. That can be used to specify when some field is wanted to be cached, instead of doing it at every lookup_field*() call. --HG-- branch : HEAD

If maildir files are unexpectedly deleted, don't treat it as error, and don't return partially fetched data. Instead try to handle it just as if sync() had noticed it earlier. --HG-- branch : HEAD

Modify log now stores the changes in ranges, so store 1:100 doesn't generate 100 entries but just one. For single message changes this takes 8 bytes unneededly though, that could be fixed but for extra complexity so I don't think it's worth it. Also a change in sync-interface, expunge event no longer gets UID. It's not currently needed and I don't see any rfcs/drafts wanting it either. It's removed mostly because the log ranges makes it impossible to get it. --HG-- branch : HEAD

MailStorage interface change. We now let storage call several sync-functions whenever it wants instead of forcing it through sync() function. This now allows us to send client FLAGS and PERMANENTFLAGS messages whenever new custom flags are added. Also EXPUNGE command now goes nicely through expunge() function (which CLOSE used) instead of through flag in sync(). "Out of disk space" alert is also sent now whenever it happens, before it was sent only when opening mailbox. Message set parser now complains when referencing mail sequences outside the possible range. Modify log before sent flags/expunges changes to client even if it didn't know about those messages yet. Also some other smaller cleanups. Beginnings of fixing APPEND and COPY when trying to store mails into selected mailbox. Before this didn't work too well or simply broke things. Still broken, needs a bit more changes.. --HG-- branch : HEAD

If BODYSTRUCTURE is cached, we can use it to generate BODY reply. It takes some CPU, but probably still faster than needing to open the file. Also removed requirement of FIELD_TYPE_LOCATION to exist in .data file. This will be useful when we don't need the location (eg. filename == uid). --HG-- branch : HEAD

Delete unused temp files. --HG-- branch : HEAD

Split IOBuffer into mmaped IBuffer, file IBuffer, memory data IBuffer and file OBuffer. --HG-- branch : HEAD