dsync: Add per-mailbox sync lock that is always used. Both importing and exporting gets the lock before they even sync the mailbox. The lock is kept until the import/export finishes. This guarantees that no matter how dsync is run, two dsyncs can't be working on the same mailbox at the same time. This lock is in addition to the optional per-user lock enabled by the -l parameter. If the -l parameter is used, the same lock timeout is used for the per-mailbox lock. Otherwise 30s timeout is used. This should help to avoid email duplication when replication is enabled for public namespaces, and maybe in some other rare situations as well.

dsync: Add hashed_headers setting This makes it possible to configure them

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

doveadm-sync: Add end-date support

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

dsync: Added -D parameter to disable mailbox renaming. The renaming logic is annoyingly complex and there are some bugs left in it. With this parameter renames are never even attempted, but instead a rename would be done (slowly) with mailbox delete + create + fill. Although with imapc protocol mailbox renames are rarely detected anyway.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Added an extra "finish" state to allow slave-dsync to report error to master-dsync. I'm not sure if that's actually necessary, but just trying to follow the different possibilities on how dsync run can finish made me unsure about it. This should make it at least clear that if a slave-dsync has a failure flag set at deinit master-dsync will know about it before it returns success.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

dsync: Added DSYNC_BRAIN_FLAG_NO_MAIL_PREFETCH to avoid opening mails unnecessarily.

dsync: Moved all doveadm-specific code to doveadm-dsync.c

dsync: mailbox_metadata.cache_fields must be copied to permanent memory. Any other call to mailbox_get_metadata() would have cleared the memory.

dsync: If incremental dsync finds that its state is stale, retry (later) with full sync. This works only when the master itself notices such changes locally. Remote changes aren't noticed, because the master doesn't send the state to remote dsync and the master is also the one that decides which mailboxes are synced. The fix to this probably means sending the state string to remote dsync and have it check if the state is still valid (= protocol change).

dsync: Finished previous change to actually compile

dsync: Support multiple -n parameters.

dsync: Added -P parameter to do a purge for the remote storage after syncing.

dsync: Improved debug/error logging for changes during sync.

dsync: Added -1 parameter to do a "one way sync" without reverting changes. This can be useful during migration when you don't want to delete any mails, but you also don't want to send changes to the old server either.

dsync: If verbose_proctitle=yes, show the current state in it.

dsync: Added -x parameter to exclude mailboxes from sync. Multiple -x parameters can be added. Giving \flag as parameter means that the mailbox with the given SPECIAL-USE \flag is skipped. For example: doveadm sync -x '\All' -x '\Flagged' -x '\Important' mdbox:~/mdbox

doveadm sync/backup: Added -g <guid> to sync only the specified mailbox (by GUID) Similar to -m <mailbox name>.

dsync: -m '' parameter now syncs mailbox list, but no actual mails.

dsync: Added support for syncing mailbox attributes.

dsync: If I/O gets stalled, log the state in which it happened.

dsync: Renamed -a parameter to -N. It also now skips invisible namespaces.

dsync: -l parameter locking is now done on the server with "lower" hostname. This allows running multi-master replication on two servers without two dsyncs mixing up changes by running at the same time.

dsync: Automatically figure out which mailboxes can sync with message GUIDs.

dsync: Renamed "guid_requests" to "mail_requests" The mails aren't necessarily requested by their GUID.

dsync: Don't start reading a mailbox until we know we actually need to sync it.

dsync: Added back support for syncing only one mailbox (-m parameter)

dsync: Added -a parameter to sync all namespaces, not just the default one.

dsync: Added debugging and rawlogging support.

doveadm sync/backup: Added -s <state> parameter to do a fast stateful sync. Initially use an empty string for the input state. The output state is written to stdout.

doveadm: "backup" command is working again.

dsync: Renamed "slave" to "ibc" (= inter-brain communicator)

Hash table API is now (mostly) type safe.

Initial version of dsync rewrite. * doveadm backup not implemented at all yet * syncing mailbox renames is somewhat broken (at least renaming \noselect mailboxes) * saving/restoring "state" is implemented by dsync brain, but not by doveadm. this should be easy to do, just need to figure out how the replication code wants it.

dsync: If message with same GUID is saved multiple times in session, copy it instead of re-saving. This only works if the messages are in different mailboxes. It would be possible to fix it also for copying within same mailbox, but that's probably rare enough that it's not worth the extra code.

Merged dsync into "doveadm dsync". dsync symlink is installed for backwards compatibility.