lib-storage: Add tests for mailbox_verify_name

lib-storage: mailbox_verify_name() - Deduplicate error handling

lib-storage: mailbox_verify_name() - Move prefix check to its own function No functional changes.

lib-storage: mailbox_verify_name() - Add comments for separator checks

lib-storage: Assert unique_root_dir is set with MAIL_STORAGE_CLASS_FLAG_UNIQUE_ROOT

lib-storage: Lock mailbox_list for mailbox create/delete/rename This is only required for mailbox creation to fix a race condition with LAYOUT=index: If INBOX doesn't exist it will rescan the mailboxes to find out if there are any missing ones. If INBOX creation isn't locked, it's possible that the first process hasn't finished creating INBOX before the second process find it and attempts to open it. The delete and rename locking are probably useful to guard against race conditions when clients intentionally issues create/delete/rename commands concurrently.

lib-storage: mailbox_delete() - Fix cleanup in error handling If removing index deletion mark failed, box->deleting wasn't set to FALSE and the mailbox was left opened.

lib-storage: mailbox_rename() - Use source storage for errors It was documented to use source storage for errors, but some of the errors were set to destination storage.

lib-storage: Fix mailbox rename checking child mailbox name lengths It was supposed to prevent allowing renames that would cause any child mailbox name to be too long. However, the check wasn't working.

lib-storage: Set mailbox_transaction_context.flags earlier Set it in index_transaction_init() so plugins' transaction_begin() methods see it after calling super.transaction_begin().

lib-storage: mail_storage_lock_create() - add support for dotlocks

lib-storage: Add mail_storage_lock_create() This is split off of mailbox_lock_file_create().

lib-storage: Set keyword based on attachment presence when saving If attachment is detected, use $HasAttachment, if not use $HasNoAttachment

Updated copyright notices to include the year 2018.

lib-storage: mail_storage_set_index_error() - handle NULL index error This avoids assert-crashing later on in mail*_get_last_internal_error(). This could potentially be an assert instead of setting it as "BUG", but it looks like there are various code paths in lib-index that return -1 without setting an error. (That's to avoid duplicate error logging, although it could now be fixed with mail_index_set_error_nolog().)

lib-storage: Delay creating mail root directory until it's necessary

lib-index: Add mail_index.event

lib-storage: mail_storage_set_critical() - log error using user's event Fix test-storage unit test to also initialize mail_user.event so it won't crash.

lib-storage: Add mail.event

lib-storage: Add mailbox.event

lib-storage: Add mail_user.event and give parent event in mail_user_alloc*()

global: Use mail_set_critical() and mailbox_set_critical() if possible Replace calls to mail_storage_set_critical() with mail_set_critical() or mailbox_set_critical() in places where mailbox or mail are easily available.

lib-storage: Add mailbox_set_critical() and mail_set_critical() These add the mailbox vname and mail UID to the log prefix and then call mail_storage_set_critical().

lib-storage: Fix mailbox_create_missing_dir() with storages that have no mail root dir The mailbox still exists even if the mail root directory doesn't, so there's no point in even trying to stat() it.

lib-storage: Make sure mailbox autocreation doesn't get into infinite loop If mailbox_create() -> mailbox_open() fails with MAIL_ERROR_NOTFOUND, it would have looped back to mailbox_create() and continued this until running out of stack.

lib-storage: Add INDEXCACHE=<path> to mail_location This allows configuring a different directory for dovecot.index.cache files than the rest of the index files.

lib-storage: mailbox_rename() - Improve debug logging when namespaces are incompatible

lib-storage: Created mailbox_alloc_delivery(), which initializes a mailbox for message delivery.

lib-storage: Implemented new mailbox flag MAILBOX_FLAG_AUTO_SUBSCRIBE. When this flag is enabled, the mailbox is implicitly subscribed to when it is created automatically.

lib-storage: Implemented new mailbox flag MAILBOX_FLAG_AUTO_CREATE. When this flag is enabled, the mailbox is automatically created when it is opened (if possible and appropriate).

lib-storage: Implemented mailbox_is_autosubscribed(). Indicates whether the mailbox is automatically subscribed to when it is automatically created.

lib-storage: Remove obsolete mailbox_save_set_dest_mail()

lib-storage: Add reason to mailbox.transaction_begin()

lib-storage: Add reason to mailbox_transaction_begin() Remove mailbox_transaction_set_reason().

global: start relying on pool_unref(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - pool_unref(&E); - } + pool_unref(&E);

lib-storage: mail_storage_set_critical() - don't free old error_string too early It may be used in one of the parameters.

lib-storage: Add mail_storage_set_index_error()

lib-storage: mail_set_mail_cache_corrupted() - set also internal error string

lib-storage: mailbox_set_index_error() - Set internal error string to index's error

lib-storage: Fix error logging after mail_storage_set_internal_error() The function doesn't actually set the last_internal_error and it also doesn't update last_error_is_internal, so calling mail_storage_get_last_internal_error() afterwards could be returning an error for some different older error. Since there's no parameter to set the internal error string, just reset last_error_is_internal=FALSE.

lib-storage: Panic if at mailbox_free() there are open attribute iterators.

lib-storage: If ITERINDEX is set, delay checking whether mail root exists In future this could be done even without ITERINDEX. It's not done yet, since it might break backwards compatibility with some systems.

lib-storage: mailbox_create_missing_dir() - Fix handling NULL mailbox directory It would have caused stat(NULL), which would have failed. Broken by 7071a02a8413fb409ade1625f6a1763c7493b66b

lib-storage: mailbox_create_missing_dir() - Handle mailbox delete race condition Make sure the directory isn't created if the mail root directory no longer exists. This might happen during mailbox deletion when another process is opening the mailbox at the same time.

lib-storage: Improve MAILBOX_LIST_PROP_AUTOCREATE_DIRS related comments

lib-storage: mailbox_sync_init() - open mailbox immediately if it's not open yet This simplifies the work for plugins that want to hook into mailbox.sync_init() so they no longer have to handle the "mailbox isn't opened" case.

lib-storage: Clarify keywords handling in mail_save_finish

lib-storage: Check for storage existence from index dir with ITERINDEX The root path creation doesn't really even seem to be necessary, because any mailbox access will automatically mkdir the missing directories anyway. Although writing other files might not work so well, such as mail_attribute_dict that points inside the mail directory. This change simply changes the mailboxes/ directory to be looked up from index directory instead of the mail root directory. It also mkdirs the index/mailboxes/ directory afterwards if it didn't exist. So practically this change shouldn't break anything, since the mailboxes/ directory should always exist for both root and the indexes.

lib-storage: Cleanup - Use mailbox_permissions_copy() instead of duplicating it

lib-storage: Add VOLATILEDIR setting to mail_location This is useful for creating temporary locks that could exist in tmpfs. Currently this is used for .vsize.lock and dovecot.autoexpunge.lock.

lib-storage: Move .vsize.lock creation to a generic mailbox_lock_file_create()

lib-storage: Fail at startup if root mail directory isn't a directory

lib-storage: Don't autocreate mailbox during deletion. Trying to delete a nonexistent autocreated mailbox first created it and then immediately deleted it.

lib-storage: Make mailbox_is_autocreated() public This also removes duplicated code from acl plugin.

lib-storage: Fix setting the correct cache record corrupted It was mixing UIDs and sequences, so a wrong mail could have been set corrupted or it could have crashed with: Panic: file mail-index-transaction-update.c: line 1018 (mail_index_update_ext): assertion failed: (seq > 0 && (seq <= mail_index_view_get_messages_count(t->view) || seq <= t->last_new_seq))

lib-storage: simplify mailbox_get_path_to() Instead of special casing MAILBOX_LIST_PATH_TYPE_{MAILBOX,INDEX}, we can use a common helper to remove code duplication.

lib-storage: Fix mail_storage_set_critical() when input parameter is an old internal error This could have caused garbage in the error string.

lib-storage: Fix mail_storage_last_error_push/pop() to work with internal errors

lib-storage: Use new mail_cache_set_seq_corrupted_reason

lib-storage: Add mailbox_transaction_set_reason()

global: Add mailbox_set_reason() calls Added to the most important places.

lib-storage: Add mailbox_set_reason() Use it to log a debug message when mailbox is being opened.

global: Log internal storage error on failure

lib-storage: Add mail_storage_get_last_internal_error() This returns the error given to mail_storage_set_critical().

lib-storage: Allow mail_storage_set_error() to use existing error_string Fixes some code that may be attempting to copy an error from one storage to another, but actually ends up trying to copy within the same storage.

Partially implemented IMAP SEARCH=X-MIMEPART capability. This capability is currently Dovecot-specific.

lib-storage: mail_storage_init/deinit() is now reference counted. This allows calling them multiple times and only the final deinit frees the data.

lib-storage: Re-open mailbox after it's undeleted. This is needed with mailbox formats that didn't fully open the mailbox while it was undeleted.

lib-storage: Don't sync mailbox after undeleting it. This is useful only when deleting it. With undeletion the syncing isn't useful and might actually be harmful with mailbox formats that didn't fully open the mailbox while it was undeleted.

lib-storage: Make sure mailbox undeletion won't go to infinite loop

lib-storage: Add mailbox_save_get_dest_mail() This marks mailbox_save_set_dest_mail deprecated. It's not efficient to use since it frees the already-created dest_mail.

lib-storage: Always create mail_save_context.dest_mail This allows removing similar mail_alloc() from storage backends and plugins that need it. As a side effect, this changes mbox code to always assign UIDs to saved mails. This shouldn't be much of a problem, since it happened practically always already.

lib-storage: Copy cache decisions from inbox on create Otherwise the decisions are definitely wrong, since they are nonexistent. Copying from INBOX would make sense.

lib-storage: Fix assert-crash on mailbox_copy/move() failure If mailbox_save_cancel() was being called outside mail_storage_copy() it assert-crashed: Panic: file mail-storage.c: line 2168 (mailbox_save_context_reset): assertion failed: (ctx->copying_via_save)

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-storage: Deduplicate code into mailbox_list_settings_init_defaults()

lib-storage: Add struct mail_save_context.copy_src_mail

lib-storage: If mailbox_create() fails, don't leave box in partially opened state. For example with sdbox it may have opened the indexes, but not set mailbox's GUID. A following MAILBOX_METADATA_GUID would then assert-crash because GUID is empty.

lib-storage: require MAIL_STORAGE_CLASS_FLAG_STUBS when caching If we are going to be using the storage for caching, we should check that the storage actually supports mail stubs.

lib-storage: Fix checking stub_seq failure in mailbox_save_set_uid()

lib-storage: Add support for storing "mail stubs" These allow treating a storage backend as a cache where the mail bodies don't necessarily exist.

lib-storage: Add asserts to make sure mailbox_header_lookup_ctx is used correctly.

lib-storage: Fix doveadm search query parsing to use timezones correctly. When using a date string, e.g. "senton 25-Mar-2007" it should work the same way as IMAP does and show all mails whose sent date is on 25th, ignoring whatever the timezone is in the Date header (or in case of received/saved-date it would be using the timestamps in server's local timezone, again just like IMAP). When search with an exact UNIX timestamp or a time interval (e.g. "3 days"), it should be using UTC times for doing all comparisons.

lib-storage: Don't enable modseqs on STATUS_HIGHESTMODSEQ. This is requested always by IMAP's SELECT command even when the IMAP client hasn't requested it. We don't want to unnecessarily enable modseqs that use up memory and disk space when they're not really needed. Other callers may also be interested in asking for HIGHESTMODSEQ (which is nowadays actually always available) without enabling full modseq tracking.

lib-storage: Fixed a boolean expression. Problem found with clang -Wstrict-bool.

lib-storage: treat empty root_dir as an empty string

lib-storage: Fixed assert-crash in mailbox_save_cancel() Broken by 05150df8f. Fixes: Panic: file mail-storage.c: line 2137 (mailbox_save_context_reset): assertion failed: (ctx->saving)

lib-storage: Don't reset mail_save_context.saving too early. If mailbox_save_using_mail() ended up in mail_storage_copy(), saving was set to FALSE before the copy() method was finished running. This caused e.g. notify plugin to think that this was a copy event instead of a save event. Added comments and asserts to clarify how the logic should work between all the different copying/moving/saving flags.

lib-storage: If mailbox_move() fails, reset mail_save_context.moving==FALSE

Removed dead code to make static analyzer happier.

lib-storage: Limit folder full name only Before we had limit of 16 levels and 255 bytes per name which is same as 4096 bytes. Now we limit only the total length of the name to MAILBOX_LIST_NAME_MAX_LENGTH. For compability reasons, we are restricting individual component names to 255 characters.

lib-storage: mailbox_set_subscribed() ignores duplicate changes now.

lib-storage: Added MAIL_STORAGE_CLASS_FLAG_NO_LIST_DELETES and MAILBOX_LIST_FLAG_NO_DELETES

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-storage: mailbox_exists() shouldn't return INBOX as nonexistent if it's not created yet.

dsync: Don't restrict what mailbox names can be created (except basic sanity checks) The mailbox names already exist on the other side, so we don't want to add arbitrary limits to them.

lib-storage: dovecot.index.pvt* ignored several settings. mmap_disable=yes, dotlock_use_excl=yes and nfs_flush_index=yes weren't used.

Remove now-unnecessary direct stdlib.h #includes.

mail-storage: Added registration API for builtin mailbox attributes.

lib-storage: Moved most of the \Recent flag handling code to mailbox-recent-flags.c There are also some API changes, because functions were renamed and the recent_* fields were moved to struct mailbox. I'm not aware of any plugins using these though, except for index_mailbox_set_recent_seq() which for now is kept as a backwards compatibility macro. No changes were made to the actual code logic.

lib-storage: Make sure tryagain_r is never randomly set in mailbox_search_next_nonblock() This infinite looping in some fts plugin failure situations.

mail-storage: Moved mailbox attributes API to separate module.

lib-storage: Avoid assert-crashing on mailbox_verify_*_name() with invalid namespace prefix.

lib-storage: Added mail_parse_human_timestamp() to parse human-writable timestamps. This isn't really the ideal location for the function, but since it uses both lib-settings and lib-imap, there's not any good location for it that I can think of..

imapc: Added imapc_sasl_mechanisms setting The first supported SASL mechanism is used, otherwise the login fails entirely.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-storage: When mailbox is deleted, reset its cached paths. If mailbox is still later on recreated, the paths may have changed.

Use i_stream_get_error() wherever possible instead of %m

lib-storage: Added data stack frames for most calls. So neither the callers nor the implementations need to worry about those so much.

lib-storage: Added mailbox_get_index_path() and caching of the index path. This solves some potential assert-crashes on race conditions when opening a mailbox while it's being deleted.

lib-index, lib-storage: Fixed race conditions with deleting mailbox. Now only one process can successfully finish mailbox_mark_index_deleted().

Compiler warning fix

lib-storage: Added mail_storage_last_error_push/pop()

lib-storage: mailbox_get_*status(STATUS_HIGHESTMODSEQ) now enables CONDSTORE feature. This makes sure that the highestmodseq is added to the mailbox list index if not already there.

lib-storage: Shrink "mailbox is being deleted" timeout from 5 mins to 30 secs. Even 30s may be too much since normally a few seconds would be enough, but keep it high enough just in case.

lib-storage: mailbox_delete() now handled "mailbox is being deleted" explicitly. Earlier we just treated it as \NoSelect mailbox and failed later.

lib-storage: Don't leak memory if mailbox_save_set_flags() is used multiple times for keywords. This affected only some external plugins.

lib-storage: mailbox_settings_find() now takes namespace parameter instead of user. This fixes an infinite (until stack gets full) loop with shared mailboxes: mail_namespace_find() -> mailbox_list_get_storage() -> mailbox_settings_find() -> mail_namespace_find() -> ...

lib-storage: Fixed crash if mailbox_save_cancel() was called in save_finish()/copy() method.

lib-storage: Further changes to mailbox_alloc_guid() caching. We'll now cache the GUIDs forever and force invalidation after mailbox rename or when mailbox GUIDs have been updated. This is what dsync needs, especially for good performance, and currently there aren't any other users for this API.

lib-storage: mailbox_alloc_guid() shouldn't refresh GUID cache on every nonexistent GUID. Refresh it only if we know it has had some changes (made by our process) or timestamp has changed. This should handle the cases that are currently important (=single dsync run).

lib-storage: Replaced mail_get_real_mail() with mail_get_backend_mail() that can fail. For now the mail_get_real_mail() can still be used for backwards compatibility.

Updated copyright notices to include year 2014.