dsync: Add hashed_headers setting This makes it possible to configure them

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

doveadm-sync: Add end-date support

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

dsync: If state is invalid, exit with code 2 instead of tempfail. We'll have dsync_mailbox_import_*() just return success to everything until _deinit() is called. The _deinit() will return a failure and set "resync needed"-flag, which caller will know how to handle.

dsync: If mailbox importing fails, stop immediately without waiting for exporter to finish.

doveadm backup: Detect and handle conflicts earlier. This avoids doing a lot of work, only for the next doveadm backup to just delete the entire mailbox and start from the beginning.

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

dsync: Include messages_count in the mailbox states. This allows detecting that stateful dsync can't be done when message count is suddenly wrong.

dsync: Support syncing storages with 128bit GUIDs <-> string GUIDs.

dsync: If verbose_proctitle=yes, show the current state in it.

dsync: Added support for syncing mailbox attributes.

dsync: Various importer fixes.

dsync: Various importer fixes. Handle unexpectedly wrong mailbox state better.

dsync: Fixes for syncing without GUIDs.

dsync: Removed unused DSYNC_MAILBOX_IMPORT_FLAG_MAILS_HAVE_GUIDS flag.

dsync: Don't wrongly log errors about missing message bodies. Also added some debug logging related to it.

dsync: Use private modseqs to support syncing private flags in shared mailboxes.

doveadm backup: Revert all local changes.

Initial version of dsync rewrite. * doveadm backup not implemented at all yet * syncing mailbox renames is somewhat broken (at least renaming \noselect mailboxes) * saving/restoring "state" is implemented by dsync brain, but not by doveadm. this should be easy to do, just need to figure out how the replication code wants it.