lib-storage: Fix memory leak when search accessed headers but found no mails For example fetching headers with UID FETCH for a nonexistent UID.

lib-storage: Code cleanup - remove unnecessary temporary variable

lib-storage: Make index_mail_want_cache() global

lib-storage: move snippet generation to mail-save-finish This is necessary because some storage backends (most notably sdbox) do not allow getting the mail stream before the mail is fully written out. (See written_to_disk in sdbox-file.h.) If we could avoid getting the stream to generate a snippet we could leave this where it is.

global: start relying on mailbox_header_lookup_unref(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - mailbox_header_lookup_unref(&E); - } + mailbox_header_lookup_unref(&E);

lib-storage: Fix adding body.snippet to cache Unless body.snippet was in mail_always_cache_fields, it didn't get added to the cache file.

lib-storage: Add comment to how mail_cache_field_can/want_add() is used

global: don't cast NULL to void * NULL is guaranteed to be a void * thanks to dd6043c05e32a8e8db1233ed711a2c74d1477a89.

Updated copyright notices to include the year 2018.

lib-storage: Add mail.event

global: Use mail_set_critical() and mailbox_set_critical() if possible Replace calls to mail_storage_set_critical() with mail_set_critical() or mailbox_set_critical() in places where mailbox or mail are easily available.

global: Use t_buffer_create sed -i -e 's/buffer_create_dynamic(pool_datastack_create(), */t_buffer_create(/g'

global: Remove dead code Found with clang static analyzer.

global: start relying on buffer_free(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - buffer_free(&E); - } + buffer_free(&E);

global: start relying on [io]_stream_unref(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - i_stream_unref(&E); - } + i_stream_unref(&E); @@ expression E; @@ - if (E != NULL) { - o_stream_unref(&E); - } + o_stream_unref(&E);

lib-storage: convert index to use container_of

lib-storage: index_mail_set_cache_corrupted() - Don't reset internal error string It's already set correctly by the earlier functions.

lib-storage: Call mail_cache_close_mail() when mail is closed Fixes caching problems with INDEX=MEMORY

lib-storage: Don't allow removing \Recent flags with mail_update_flags() Reverts an ancient commit 7deb24e7453249d09741641bff0f269f68165033. I don't know why it was added in the first place. Normally \Recent flags are removed during sync with MAIL_INDEX_SYNC_FLAG_DROP_RECENT. This should be enough, especially since it's not even possible to remove a single \Recent flag - only update the first_recent_uid. The code was also wrong: It was dropping \Recent flags even when modify_type/flags combination didn't ask for it. Even if this was fixed, there would still be race conditions with multiple processes since this update is done without locking. Fixes: Error: Recent flags state corrupted for mailbox

lib-storage: mail_add_temp_wanted_fields() - delay opening stream Allows plugins to call it in mail.get_stream() without infinite loop.

lib-storage: Don't crash if cache is thought broken for a newly saved mail It's a bug that it happens, but the resulting crash is confusing. Although arguably this new error could be i_panic() instead. Fixes: Panic: file mail-index-transaction-update.c: line 1018 (mail_index_update_ext): assertion failed: (seq > 0 && (seq <= mail_index_view_get_messages_count(t->view) || seq <= t->last_new_seq))

lib-storage: Use new mail_cache_set_seq_corrupted_reason

lib-storage: Fix crash in mail_get_parts() with a special plugin If plugin hooks into mail_get_stream() which causes data->parts to be set, this code crashed with: Panic: file index-mail.c: line 1163 (index_mail_parse_body): assertion failed: (data->parser_ctx != NULL)

lib-storage: Add mail_storage.nonbody_access_fields This avoids index_mail_update_access_parts_pre() from opening the mail stream unnecessarily for fields that can be looked up via other methods by the storage.

lib-storage: Add index_mail_get_cached_body[structure]() This just moves the code to these functions without changing any of the logic.

lib-storage: Allow mail_add_temp_wanted_fields() to be called before mail_set_seq*() Quota plugin was already doing this, but it didn't actually work. It was also crashing with imapc: Panic: file mail-cache-lookup.c: line 341 (mail_cache_field_exists): assertion failed: (seq > 0)

lib-storage: Add mail.mail_stream_opened and .mail_metadata_accessed These allow determining after mail_*() calls how efficiently they were performed.

lib-storage: Remove unnecessary mail_save_context.dest_mail==NULL checks It can never be NULL after the previous change: "lib-storage: Always create mail_save_context.dest_mail". The code removal in maildir_transaction_save_commit_pre() seemed potentially dangerous, but I don't think such code path is possible anymore. Also even if it is, it's probably fine since the mail_free() is called even earlier than before (although that itself might have been a problem). This also removes last traces of code that made it possible to save mails to mbox without assigning UID to the mail. The previous commit already caused this, so this is just removing dead code.

lib-storage: Expunging now uses GUID only if it's in cache. Otherwise the GUID lookup may be expensive, which is especially bad when deleting a large number of mails.

lib-storage: Put vsize in index only if it's not there

lib-imap: imap-bodystructure: Moved message_part_data header parsing and query functions to their own module in lib-mail. Moved code from imap-bodystructure and imap-envelope.

lib-imap: imap-bodystructure: Renamed imap_bodystructure_is_plain_7bit() to message_part_data_is_plain_7bit().

lib-imap: imap-bodystructure: Renamed imap_bodystructure_parse_header() to message_part_data_parse_from_header().

lib-imap: imap-bodystructure: Changed struct message_part_body to contain fully decoded data. Contained unparsed IMAP string data before.

lib-mail: Gave message_part_data its own field in struct message_part.

lib-imap: imap-bodystructure: Renamed struct message_part_body_data to message_part_data.

lib-storage: Add index_mail_get_vsize_extension()

lib-storage: Move index_mail.vsize_ext_id to mailbox.mail_vsize_ext_id It's mailbox-specific, so it doesn't have to be looked up every time. Also this is needed for the following fix.

lib-storage: Fix comparison to use virtual size instead of body virtual size Was forgotten on 7f1ecb14

lib-storage: Filling vsize to index on read used wrong size

lib-storage: Clear virtual size from index if size is corrupted

lib-storage: Add vsize extension to index Keep virtual size in index, instead of cache when it's less than 2^32-1. This helps when cache becomes corrupted, and goes away, we still have virtual sizes for quota calculations.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

lib-storage: rename get_real_mail to get_backend_mail Also change the function signature so that it returns an int to indicate success or failure.

lib-storage: remove set_cache_corrupted_reason from mail_vfuncs Change set_cache_corrupted signature to what set_cache_corrupted_reason used to be.

lib-storage: remove mailbox-recent-flags.h from index-storage.h

Use mail_get_*stream_because() wherever possible.

lib-storage: Added mail_get_stream_because() and mail_get_hdr_stream_because() With mail_debug=yes each mail access is now logged with a reason. This can be helpful when figuring out why something isn't in dovecot.index.cache.

lib-storage: Don't crash fetching body.snippet when multipart/alternative part has no Content-Type

lib-storage: mail_set_cache_corrupted*() now sets internal error to storage. At least index_mail_set_message_parts_corrupted() assumed that this was being done.

global: Fixed mismatched bool vs. int/pointer handling I don't think these fix any actual bugs.

lib-storage: Make it easier to debug mail's unclosed istream

lib-storage: When cached mime.parts is detected to be inconsistent, log it as hex-encoded.

lib-storage: If cached mime.parts is corrupted, log it hex-encoded.

Use mail_set_cache_corrupted_reason() wherever possible.

lib-storage: mail_set_cache_corrupted*() now logs also the mailbox name that was corrupted.

lib-storage: Added mail_set_cache_corrupted_reason()

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-storage: mail_add_temp_wanted_fields() was reading already-freed memory. This mainly happened with mail_log plugin enabled.

lib-storage: Support latest cache fields in struct mailbox_metadata.precache_fields

lib-storage: Renamed MAIL_FETCH_UIDL_FILE_NAME to MAIL_FETCH_STORAGE_ID. Keep MAIL_FETCH_UIDL_FILE_NAME for backwards compatibility for now.

lib-storage: Moved most of the \Recent flag handling code to mailbox-recent-flags.c There are also some API changes, because functions were renamed and the recent_* fields were moved to struct mailbox. I'm not aware of any plugins using these though, except for index_mailbox_set_recent_seq() which for now is kept as a backwards compatibility macro. No changes were made to the actual code logic.

Make Coverity happier.

lib-storage: MAIL_FETCH_BODY_SNIPPET should seek the mail stream back to original position.

lib-storage: Added MAIL_FETCH_BODY_SNIPPET.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-storage: Mail prefetching was still not working right. I wonder if it ever actually worked (except for SEARCH TEXT).

lib-storage: Don't fetch wanted_fields for messages that don't match the search query. For example "doveadm fetch text subject foo" was opening all the mails, even though only a few matched the subject (that was hopefully already in cache file). The behavior still isn't perfect though. The wanted_fields should probably be split into search_wanted_fields and fetch_wanted_fields, but the current behavior is likely good enough for now.

lib-storage: Fixed assert-crash when BODYSTRUCTURE parsing fails due to broken mail size.

lib-storage: Don't assume that we must read the whole message to get its virtual size. This is true only with mbox format. All the other formats have various metadata available where it's stored.

lib-storage: Bodystructure parsing flags weren't updated correctly on error conditions. This fixes an assert-crash sometimes when mail parsing failed.

lib-storage: istream-mail updates mail->expunged if it notices ENOENT from parent stream.

lib-storage: Log mail istream read failures in one place. Also handle ENOENT errors by checking if the mail has already been expunged, and if so don't log an error, just return "mail is already expunged" error to client.

lib-storage: If mail body reading failed, the error message may have contained only minimal errno string. Even though the istream could have had a much better internal error message. So show it.

lib-storage: BODYSTRUCTURE parsing failures weren't treated correctly. We still assumed that the parsing succeeded and assert-crashed later or maybe returned invalid results. (This could have happened only if there was a problem reading the mail stream.)

lib-storage: "Message has no NUL characters" flag was being set wrong to cache file. Only the first MIME part was checked for its existence. When this flag was wrong, IMAP FETCH may have returned NUL characters instead of converting them to 0x80 character. This apparently caused Outlook to hang.

lib-storage: Fixed assert-crash when looking up private flags for a mail being saved.

lib-storage: mail_never_cache_fields=* means now to disable all caching.

lib-storage: Partially reverted the mail.get_real_mail() API change for backwards compatibility. I hadn't realized Pigeonhole was also using it. (I thought it was using only the mail_get_real_mail() public API.) This fixes Pigeonhole v0.4.2 to work again.

lib-storage: Replaced mail_get_real_mail() with mail_get_backend_mail() that can fail. For now the mail_get_real_mail() can still be used for backwards compatibility.

lib-storage: When trying to get body size quickly, make sure we don't end up reading the mail body.

Updated copyright notices to include year 2014.

lib-storage: Fixed memory leak in mail_add_temp_wanted_fields()

dbox: Added support for POP3 message order.

lib-storage: mail_update_pvt_modseq() shouldn't crash if there is no private index.

istream API change: Added support for multiple destroy callbacks.

lib-storage: Added MAIL_FETCH_REFCOUNT for getting mail's reference count. This is useful only for backends where mailbox_copy() increases the refcount. With maildir&sdbox it can be looked up from the file's link count. With mdbox the refcount is stored in the map index. Other formats don't currently implement this.

Compiler warning fix.

lib-storage: If mail saving is cancelled, don't check that all the input was read.

Replaced all -1U and (unsigned int)-1 with UINT_MAX. It's somewhat clearer this way. Also clang's -fsanitize=integer gives runtime errors about -1U (but not about explicit casts, so no need to change (type)-1 casts).

lib-storage: If message parser's input stream is closed early, don't treat it as error. This seems to happen sometimes during message saving(?), although I'm not exactly sure why. In any case it shouldn't log an error about it. This behavior is probably better than failing. The caller should be the one to figure out if saving has failed or not.

Oops :) Update copyrights to 2013 without breaking all .c files.