pop3-migration-plugin: Fix context requirements Fixes Panic: Module context pop3_migration_storage_module missing Broken by a8703ce24540b7efaa51a8c7d3c72e72727f9789

pop3-migration: Contexts are now required or checked Satisfies static analyzers

Updated copyright notices to include the year 2018.

global: Use t_buffer_create sed -i -e 's/buffer_create_dynamic(pool_datastack_create(), */t_buffer_create(/g'

lib-storage: Add reason to mailbox_transaction_begin() Remove mailbox_transaction_set_reason().

lib-storage: convert index to use container_of

pop3-migration: Strip away invalid header lines. If there's no ":" in the header, remove it. Fixes matching mails with Zimbra.

pop3-migration: Fail if all IMAP mails were matched by size, but POP3 had extra mails This makes the handling same as what happens when the same situation happens and the last mails were matched by headers. That's an error, unless pop3_migration_ignore_*_uidls=yes.

pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting

pop3-migration: Try to assign UIDLs based on dovecot.index.cache Add the UIDLs to (imapc) mailbox cache after they've been assigned. Try to read them from the cache on the next sync to avoid reading mails' headers. This can be disabled with pop3_migration_skip_uidl_cache=yes, just in case there's some kind of a problem with it.

pop3-migration: Delete unnecessary POP3 order checking code Nothing is actually using the result.

pop3-migration: Drop lines with only spaces or tabs from comparison Zimbra drops out those lines from IMAP BODY[HEADER] replies.

pop3-migration: Replace trailing whitespace removal with new header hashing version This is now done by message_header_hash(), which makes it work correctly also for dsync+imapc. Reverts 0cf3b30b86e6c39f43b8e13a718cd078187ca86d, except for the unit tests.

pop3-migration: Strip trailing spaces from headers when calculating hash Fixes matching mails with Zimbra.

global: Add mailbox_set_reason() calls Added to the most important places.

global: Log internal storage error on failure

lib-storage, pop3-migration: Reset lookup_abort before continuing to search This doesn't really fix anything right now, but it'll allow adding the asserts in the following commit.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

lib-mail: message_header_hash_more() now allows input in any slices. There wasn't necessarily any guarantees that the input would be sliced in such a way that the repeating '?' would be dropped the same way every time.

pop3-migration: "first POP3 msg" warning didn't actually show the first one. It was showing the first index in an array, which had nothing to do with being the first POP3 message number.

pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting.

pop3-migration: Avoid unnecessarily using stream's hdr_size. Shouldn't change any behavior, except reduce CPU usage a little bit.

pop3-migration-plugin: Add to index after successful retry Also change return value to indicate that the hash has been assigned to header.

Use mail_get_*stream_because() wherever possible.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

[LEN] to [static LEN] on some function parameters Also add STATIC_ARRAY macro to hide it in c++ compilation.

lib-mail: Fixed istream-header-filter callback's matched with eoh Reversed the matched-parameter handling for hdr->eoh, so it's consistent with non-eoh headers.

lib: istream - migrate to i_stream_read_more() Scripted to find all the low-hanging-fruit (single line calls), but hand-checked: git grep i_stream_read_data.*,\ 0\) | sed s/:.*// | \ xargs sed -i -e 's/i_stream_read_data(\(.*\), \(.*\), \(.*\), 0)/i_stream_read_more(\1, \2, \3)/' Signed-off-by: Phil Carmody <phil@dovecot.fi>

global: Use mail_user_plugin_getenv_bool() wherever possible New API is used to check boolean setting OR exsitence of an env variable.

pop3-migration: Ignore mails returned as expunged by pop3c. Whether the -ERR actually meant they were expunged or not is another issue, but the answer to that belongs to pop3c code. MAIL_ERROR_EXPUNGED in any case can be safely handled this way here.

pop3-migration: Use message_header_hash_more() instead of duplicating the code

pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to

pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug.

pop3-migration: If reading message header for hashing fails, fail immediately. This avoids flooding the log with a ton of errors in case the POP3 connection dies.

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

pop3-migration: Don't change TAB to '?' when calculating header hash.

pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes This skipped the header check for mailboxes that had only a single mail.

pop3-migration: Cached header hashes weren't actually being used for imapc. We'll need to do the search twice: Once to find out the actual cached header hashes and then second time do a search for the message headers excluding the emails whose hashes we already know. This allows prefetching to work for imapc without prefetching all the emails as it was doing.

pop3-migration: Add calculated header hashes to local cache. This allows faster resuming on failures if local pop3c indexes are used.

pop3-migration: Fix to 2f19f8 - Don't convert LF also to '?' in header hashes.

pop3-migration: Filter out headers with invalid names.

pop3-migration: Convert all non-ASCII in headers to '?'

pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it.

pop3-migration: Ignore X-Yahoo-Newman-Property: header This exists only for Yahoo IMAP mails, but not for Yahoo POP3 mails.

pop3-migration: When logging the first missing POP3 mail, it logged a wrong mail

pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL This was already being done for the warning message when pop3_migration_ignore_missing_uidls was set, but not for the error message when it wasn't set.

pop3-migration: Added pop3_migration_skip_size_check setting If we know that we're not going to be able to match any messages by sizes, we might as well not ask for the messages' sizes. This is useful at least with Yahoo.

pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes This fixes matching between Yahoo IMAP and Yahoo POP3

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

pop3-migration: Fetch physical sizes instead of virtual sizes so pop3c uses LIST 4bebfbb32410 caused the fetching to break entirely.

pop3-migration: Use LIST instead of RETRs to get the messages' sizes.

pop3-migration: Truncate header if there's line containing only CR(s). This fixes matching IMAP <-> POP3 messages when the servers behave differently.

pop3-migration: Show the first message's number and UIDL which wasn't found from IMAP.

pop3-migration: Added more debug and error logging.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

Updated copyright notices to include year 2014.

pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes.

pop3-migration: Convert NULs to 0x80 chars in header when hashing. This should help at least with Dovecot and I think also with UW-IMAP/POP3.

pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early.

pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Fixes a memory leak where the user wasn't destroyed at all because the mailbox still caused the user to be referenced.

Oops :) Update copyrights to 2013 without breaking all .c files.