dsync: Log a warning if copying a mail fails unexpectedly Don't log a warning if it happens because the source message was expunged. That's an expected failure.

Updated copyright notices to include the year 2018.

global: Use PRI* macros and %zu instead of casting

dsync: Add hashed_headers setting This makes it possible to configure them

lib-storage: Add reason to mailbox_transaction_begin() Remove mailbox_transaction_set_reason().

dsync: Add missing transaction flags when performing UID renumbering The transaction didn't have MAILBOX_TRANSACTION_FLAG_SYNC or MAILBOX_TRANSACTION_FLAG_NO_NOTIFY (with DSYNC_BRAIN_FLAG_NO_NOTIFY). This caused replication-dsync to unnecessarily trigger another dsync replication notification.

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

global: Log internal storage error on failure

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

dsync: Fixed boolean expression in dsync_mailbox_import_deinit(). Found with Clang -Wstrict-bool.

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

doveadm-sync: Add end-date support

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

global: Update mailbox_attribute_get API usage

dsync: Fixed empty-header-workaround

dsync: Add support for features Add empty_header_workaround as first feature

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

dsync: Debug logging improvement.

doveadm backup: Fixed unnecessary mailbox deletion. Broken by 79490ec1a. The idea is to delete the mailbox early to avoid unnecessary work, but we deleted it too early. If the last mail(s) were deleted instead of just expunging them we deleted the whole mailbox.

global: Code cleanup - remove unnecessary errno changes. These were left behind from changing %m to [io]_stream_get_error()

lib: istream - migrate to i_stream_read_more() Scripted to find all the low-hanging-fruit (single line calls), but hand-checked: git grep i_stream_read_data.*,\ 0\) | sed s/:.*// | \ xargs sed -i -e 's/i_stream_read_data(\(.*\), \(.*\), \(.*\), 0)/i_stream_read_more(\1, \2, \3)/' Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Fixed potential crash This seems to be possible with dsync_mailbox_find_common_expunged_uid() -> dsync_mailbox_common_uid_found() -> dsync_mailbox_rewind_search() -> following search doesn't find anything.

dsync: Fixed assert-crash if mailbox_attribute_set() fails We shouldn't set importer->mail_error, because we're not going to fail the import. Fixes assert-crash: dsync-mailbox-import.c: line 2812 (dsync_mailbox_import_deinit): assertion failed: (importer->failed == (importer->mail_error != 0))

dsync: If state is invalid, exit with code 2 instead of tempfail. We'll have dsync_mailbox_import_*() just return success to everything until _deinit() is called. The _deinit() will return a failure and set "resync needed"-flag, which caller will know how to handle.

dsync: Improved detecting state state string.

dsync: If mailbox importing fails, stop immediately without waiting for exporter to finish.

doveadm backup: If local mailbox has more mails than remote, detect and delete it earlier.

doveadm backup: Detect and handle conflicts earlier. This avoids doing a lot of work, only for the next doveadm backup to just delete the entire mailbox and start from the beginning.

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

dsync: Removed unnecessary code - cur_mail is never NULL at this point. Found by Coverity.

dsync: Assert-crashfix Fixes: Panic: file mail-storage.c: line 1897 (mailbox_save_alloc): assertion failed: (!ctx->unfinished)

dsync: Crashfix in certain situation.

dsync: Compiler warning fix to previous change.

dsync: Fixed memory leaks when importing attributes whose values were in istreams.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Crashfix to -F parameter handling.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added more debug logging.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Added debug logging for attribute importing

dsync: Fixed assert-crash / memory leak on error handling path.

dsync: Debug logging improvements and comment clarifications.

doveadm backup: Don't unnecessarily delete mailbox when handling expunges

doveadm backup: Local mailbox was deleted also when there were just new messages.

doveadm backup: When deleting a mailbox, log a warning, not just a debug message. Because we're also returning temporary failure in any case, so there needs to be some kind of a warning/error logged.

dsync: Debug message fix

dsync: Improved debug logging.

dsync: Added more debug logging.

doveadm backup: Fix to earlier commit - don't try to delete an already empty mailbox.

doveadm backup: Locally expunged mails weren't always added back.

dsync: Added DSYNC_BRAIN_FLAG_NO_MAIL_PREFETCH to avoid opening mails unnecessarily.

dsync: Fixed infinite looping on error condition.

Removed pointless NULL checks. Found by Coverity

dsync: Make static analyzer happier (hopefully)

dsync: saved-date doesn't need to be looked up until mail body is being read. This should improve the performance when the saved-date isn't already cached or otherwise quickly accessible. This change also makes dsync slightly incompatible with earlier versions. When using dsync with an earlier version the saved-dates aren't synced. It would be too much trouble to try to preserve full backwards compatibility, especially because saved-date doesn't matter so much and isn't even visible to IMAP clients.

dsync: Include messages_count in the mailbox states. This allows detecting that stateful dsync can't be done when message count is suddenly wrong.

Compiler warning fixes

dsync: Fixed high data stack memory usage

dsync: Optimization for a large number of mails with the same GUID. Still not ideal, but better than before.

Updated copyright notices to include year 2014.

dsync: Fix to 128bit GUID syncing

dsync: Support syncing storages with 128bit GUIDs <-> string GUIDs.

dsync: Use i_stream_get_error() instead of just errno in stream error messages.

dsync: Removed periodic commits while importing. The problem is that the current algorithm doesn't handle it correctly. If there already are some mails that exist locally, or remote has duplicate mails, the messages may not be saved in the UID order, and if there's a commit between such wrong ordering, the UIDs get renumbered.

dsync: Added more consistency when debug logging about changes during sync.

dsync: Added some more debug logging.

dsync: Improved debug/error logging for changes during sync.

dsync: Avoid assert-crash after an error.

dsync: Fixed syncing when one of the backends supported GUIDs.

dsync: Fixed handling expunges when GUIDs aren't supported by the backend(s).

dsync: If verbose_proctitle=yes, show the current state in it.

dsync: Don't fail the sync if attribute couldn't be set. It's probably a system configuration mismatch where some/all attributes don't work in one system but do on another. This might or might not be a problem, so it deserves an error message, but probably doesn't deserve failing entirely.

dsync: Fixed assert-crash caused by previous change

dsync: Commit large transactions every 100 new messages. This way if the dsync crashes or transaction fails in the middle, the next run can finish faster. Also the rollbacking finishes faster.

dsync: If saving mails fail, stop trying to save more of them and flooding logs.

Make static analyzer happier.

dsync: If remote doesn't send some mails, don't exit with code 0.

dsync: Crashfix when importing from storage without GUIDs.

Compiler warning fix

dsync: Fixed attribute value comparisons.

lib-storage: Mailbox attributes can now be accessed via istreams. The idea is to use istreams for larger values.

dsync: Set last_change timestamp for unset attributes.

lib-storage: mailbox_attribute_set() now uses struct mail_attribute_value. This allows settig the last_change value, as well as using streams and setting other flags in future.

dsync: Added support for syncing mailbox attributes.

dsync: Assert-crashfix for handling conflicting private flags.

dsync: Fixes to handling local changes during dsync.

dsync: Fixed infinite loop in importer.

dsync: Improved unexpected error logging.

dsync: If unexpected changes happen during sync, return a safe last-common-uid for state.

dsync: Fixed last-common-uid lookup when local UID was expunged but remote wasn't.

dsync: Assert-crashfix for previous importer changes.

dsync: Various importer fixes.

dsync: Avoid possibly long function recursion.

dsync: Various importer fixes. Handle unexpectedly wrong mailbox state better.

dsync: Minor fix to deciding when to use GUIDs vs. header hashes.

dsync: Fixes for syncing without GUIDs.

dsync: Removed unused DSYNC_MAILBOX_IMPORT_FLAG_MAILS_HAVE_GUIDS flag.

Oops :) Update copyrights to 2013 without breaking all .c files.