lib-storage: Add and use INDEX_LIST_CONTEXT_REQUIRE Satisfies static analyzers

lib-storage: mailbox_list_index - List prefix/INBOX if it necessary

LAYOUT=index: Fix updating STATUS changes in mailbox list index Mailbox list index backend code was overriding sync_init and sync_deinit methods, which STATUS handling had already already overridden. They both used the same super struct, so STATUS's sync_* were never called.

lib-storage: Rebuild list index when doing doveadm force-resync

LAYOUT=index: Rebuild list index if INBOX is missing on inbox=yes namespace This avoids having to manually rebuild the list index afterwards if it becomes lost.

lib-storage: Fix MAILBOX_LIST_INDEX_HIERARHCY_SEP spelling Original work by @jsoref

lib-storage: Fix mailbox_list_index=yes + ACLs + auto=subscribe mailboxes If the auto=subscribe mailbox didn't have a lookup ACL, it shouldn't have been listed. This didn't work properly, because ACL plugin initialized the autocreate_ctx only for the index iterator, while the autoboxes were listed using the backend iterator. Fixed by not creating index iterator at all when doing a passthrough iteration.

lib-storage: Implement mailbox_list_notify_flush() This flushes any pending notifications. For now it only checks if there's a pending notification waiting for a timeout. It could also stat() the files to make sure, but that's probably unnecessary.

lib-storage: Fix mailbox list index records without guid having invalid flags With LAYOUT=index the mailbox is found with its GUID. If it doesn't exist, the mailbox has to be either \NonExistent or \NoSelect. It probably doesn't make much difference which one, so we'll fix them by adding \NoSelect. With other layouts the GUID is allowed to be empty, because it can be looked up from the actual mailbox.

lib-storage: Fix mailbox list index hook usage The previous method appeared to work, but not if the internal hooks were used more than once.

LAYOUT=index: Rename mailbox_list_index_node.corrupted_parent to corrupted_ext The code will be useful for other purposes also besides updating corrupted parents.

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-storage: Make dovecot.list.index's filename configurable. This is useful when there are multiple namespaces pointing to the same mail root directory. For example mdbox with lazy-expunge: namespace { prefix = Expunged/ location = mdbox:~/mdbox:MAILBOXDIR=expunged:LISTINDEX=expunged.list.index .. }

lib-storge: Call mail_storage.list_index_corrupted() when needed The callback is called whenever mailbox list index appears to be corrupted with LAYOUT=index. The storage is responsible for adding to the index any mailboxes that are missing.

lib-storage: Rename mailbox_list_index.corrupted to corrupted_names_or_parents Makes it clearer what exactly the flag means.

LAYOUT=index: Don't write corrupted mailbox names to box-name header. This way when opening a mailbox the box-name header isn't overwritten by a corrupted name. Keep track of the corrupted names with MAILBOX_LIST_INDEX_FLAG_CORRUPTED_NAME flag in list index records. The flag isn't removed until the mailbox is renamed.

lib-storage: Fix invalid parents in LAYOUT=index Previously it just kept complaining without fixing the errors.

lib-storage: If LAYOUT=index loses mailbox names, write the generated ones to index. Previously it just kept complaining and regenerating new names, without actually fixing up the situation.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

Added mailbox { autoexpunge } setting. This can be used to automatically expunge mails from specified mailboxes after they're old enough. The expunges are done when the user is being deinitialized. mailbox_list_index=yes should be enabled to have the best performance with this setting. Example: namespace inbox { mailbox Spam { auto = create special_use = \Junk autoexpunge = 30d } }

lib-storage: If mailboxes' vsizes are used, keep them updated also in mailbox list index. This allows looking them up quickly without opening the actual mailbox indexes.

Compiling fix for some systems.

lib-storage: Fixed some race condition crashes with LAYOUT=index

lib-storage: Avoid refreshing mailbox list index too often.

lib-storage: Avoid rebuilding the whole mailbox list index when one mailbox changes.

lib-storage: Fixed assert-crash in mailbox rename with layout=index

LAYOUT=index: Fixed race condition during mailbox creation.

mailbox_list_index=yes: Fixed list iteration when namespace and backend separators differed.

layout=index: Allow mailbox create/delete/rename during mailbox list iteration.

mailbox_list_index: Delay allocating the index so mbox can override the index path.

lib-storage: Fixed mailbox_update() when using mailbox_list_index=yes

lib-storage: Fixes to LAYOUT=index backend.

lib-storage: Avoid assert-crashing when updating mailbox list index.

lib-storage: Added initial implementation for layout=index. The idea is that all of the mailbox names only exist in the mailbox list index. Storage directories use mailbox GUIDs. This avoids all kinds of race conditions with mailbox renames. mailbox_list_index=yes is required for this layout to work. There are probably still some race conditions within the indexing code itself. Also error recovery is missing.

Removed all the "enum foo;" declarations. They didn't seem to be very standard. Old gcc versions warn about them and C++ code doesn't like them.

NOTIFY: Implemented SubscriptionChange

Reverted "support for non-pointers" part of the hash table API changes. Originally I wrote it using clang, which didn't give as many warnings as gcc did. I guess this way is safer anyway..

Hash table API is now (mostly) type safe.

lib-storage: Added mailbox-list-notify API for tracking changes in all mailboxes. Requires mailbox_list_index=yes to work.

mailbox_list_index=yes: Delay opening/creating the index until it's needed. This also fixes an error when trying to create the index before the parent director is created.

mailbox_list_index=yes iteration: Don't allocate returned mailbox names from data stack. This breaks dsync.

mailbox list indexes: Avoid unnecessary refresh flag writes.

mailbox list indexes: Added comment.

mailbox list indexes: Code cleanups.

mailbox list indexes: Error handling fixes.

mailbox list indexes: Moved syncing code to separate file.

mailbox list indexes: Moved iteration code to separate file.

Renamed index-mailbox-list* to mailbox-list-index*