lib-storage: Generate snippet while saving new mails

Updated copyright notices to include the year 2018.

global: Use t_buffer_create sed -i -e 's/buffer_create_dynamic(pool_datastack_create(), */t_buffer_create(/g'

lib-storage: convert index to use container_of

lib-storage: Fix crash in mail_get_header_stream() when its previous stream wasn't at EOF At least this could have happened when indexes were disabled and running: FETCH 1 (envelope body.peek[header.fields (foo)] bodystructure) Fixes: Panic: file index-mail-headers.c: line 198 (index_mail_parse_header_init): assertion failed: (!mail->data.header_parser_initialized)

lib-storage: Use new mail_cache_set_seq_corrupted_reason

lib-storage: Add assert to make sure header parsing isn't recursed

lib-mail: message-part-data: Renamed struct message_part_envelope_data to message_part_envelope.

lib-imap: imap-envelope: Moved imap_envelope_headers to lib-mail/message-part-data as message_part_envelope_headers.

lib-imap: imap-bodystructure: Moved message_part_data header parsing and query functions to their own module in lib-mail. Moved code from imap-bodystructure and imap-envelope.

lib-imap: imap-envelope: Renamed imap_envelope_parse_header() to message_part_envelope_parse_from_header().

lib-imap: imap-bodystructure: Renamed imap_bodystructure_parse_header() to message_part_data_parse_from_header().

lib-storage: Compile & indent fix for previous change.

lib-storage: Prevent recursion in header parsing If header parsing error occurs and error handling tries to get fields, such as Message-ID, it will cause crash. This fixes problem by preventing reading from non-cached headers while they are being parsed. Fixes lmtp: Panic: file ../../../src/lib/array.h: line 219 (array_idx_i): assertion failed: (idx * array->element_size < array->buffer->used)

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-storage: fix index_mail_get_headers return value And make it more obvious that we're not setting the actual return value in index_mail_get_first_header.

lib-storage: Add asserts to make sure mailbox_header_lookup_ctx is used correctly.

lib-storage: Make sure data->stream isn't NULL Makes static analysers happy

Use mail_get_*stream_because() wherever possible.

global: Use only explicit int -> bool conversions These were checked with a patched clang.

global: Avoid using buffer_get_data() Especially buffer_get_data(buf, NULL) generates worse code than just using buf->data directly.

lib-storage: When cached mime.parts is detected to be inconsistent, log it as hex-encoded.

Use mail_set_cache_corrupted_reason() wherever possible.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-storage: Removed caching body snippet while message is being saved. This broke saving mails with dbox, because it attempts to open the mails too early. Also the generation was a bit costly operation to do if it wasn't actually wanted to be cached. So in future we'd first want to check if body snippet is actually wanted. If it is, then generate the potential snippets while the message is being parsed (always generate the snippet for the best found MIME part, replacing existing snippets when needed). This requires the message_snippet_generate() API to support sending input as partial message_blocks.

lib-storage: Added MAIL_FETCH_BODY_SNIPPET.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-storage: If uncached header unfolding fails, panic instead of returning error. This really shouldn't be happening. This also makes it clear that return value -1 means some kind of I/O error instead of corruption.

lib-storage: Fixed header parsing when there were multiple same header names. For example if a mail had: Name1: a Name1: b Name2: c If the Name1: was initially added to cache and Name2: not, but later on both were attempted to be added to cache, the Name2: lookup would have been added with "b" instead of "c" value.

lib-storage: Headers were sometimes added to dovecot.index.cache with wrong line number value. This happened if the header parsing was done multiple times for the same mail. The end result being that when fetching multiple headers as a stream, the headers could have been returned in the wrong order. This messed up at least dsync's header hash comparisons.

lib-storage: Bodystructure parsing flags weren't updated correctly on error conditions. This fixes an assert-crash sometimes when mail parsing failed.

lib-storage: Log mail istream read failures in one place. Also handle ENOENT errors by checking if the mail has already been expunged, and if so don't log an error, just return "mail is already expunged" error to client.

lib-storage: mail_get_headers_utf8() now replaces NULs with spaces. Perhaps there should also be another API function which actually allows returning strings with NULs in them.

Updated copyright notices to include year 2014.

lib-storage: mail_get_headers*() returned only the first header from cache.

lib-storage: mail_get_*header*() still didn't handle cached values correctly. If header ends with ":", the ":" shouldn't be returned as part of the value.

lib-storage: mail_get_*header*() were unnecessarily looking up the headers twice.

lib-storage: mail_get_*header*() didn't remove leading whitespace for headers from cache. But it was removed when the headers were found by parsing the full header.

Replaced all -1U and (unsigned int)-1 with UINT_MAX. It's somewhat clearer this way. Also clang's -fsanitize=integer gives runtime errors about -1U (but not about explicit casts, so no need to change (type)-1 casts).

Oops :) Update copyrights to 2013 without breaking all .c files.

Updated copyright notices to include year 2013.