Updated copyright notices to include the year 2018.

dsync: Add per-mailbox sync lock that is always used. Both importing and exporting gets the lock before they even sync the mailbox. The lock is kept until the import/export finishes. This guarantees that no matter how dsync is run, two dsyncs can't be working on the same mailbox at the same time. This lock is in addition to the optional per-user lock enabled by the -l parameter. If the -l parameter is used, the same lock timeout is used for the per-mailbox lock. Otherwise 30s timeout is used. This should help to avoid email duplication when replication is enabled for public namespaces, and maybe in some other rare situations as well.

dsync: Add debug logging for .dovecot-sync.lock locking/unlocking

global: Use PRI* macros and %zu instead of casting

dsync: Add hashed_headers setting This makes it possible to configure them

global: start relying on pool_unref(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - pool_unref(&E); - } + pool_unref(&E);

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

global: Log internal storage error on failure

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

dsync: Fix .dovecot-sync.lock timeout checking Whenever the lock file was recreated, the lock timeout was reset. Switched to using file_create_locked(), which already solves this problem and has compatible locking.

dsync: Improve process title during initialization If something is hanging, this should make it clear what exactly it is.

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

doveadm-sync: Add end-date support

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

dsync: When full resync is wanted in a stateful sync, output empty state. This continues 3d49dc64d, which didn't actually work because brain->require_full_resync was either cleared earlier or it was never even set in this brain.

global: Use only explicit int -> bool conversions These were checked with a patched clang.

dsync: If full resync is requested, return empty state string. If the state is wrong, it's better to fully resync all the mailboxes rather than just the one where a problem was noticed.

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

Replaced unlink() calls with i_unlink*() wherever possible.

dsync: Added -D parameter to disable mailbox renaming. The renaming logic is annoyingly complex and there are some bugs left in it. With this parameter renames are never even attempted, but instead a rename would be done (slowly) with mailbox delete + create + fill. Although with imapc protocol mailbox renames are rarely detected anyway.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Added an extra "finish" state to allow slave-dsync to report error to master-dsync. I'm not sure if that's actually necessary, but just trying to follow the different possibilities on how dsync run can finish made me unsure about it. This should make it at least clear that if a slave-dsync has a failure flag set at deinit master-dsync will know about it before it returns success.

dsync: Open mailboxes with readonly-flag whenever possible. There shouldn't be any actual functional difference though.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Removed in/out state from debugging. This was initially added for debugging hangs, but those haven't existed for a long time now. Maybe we should eventually make debug categories configurable, but for now nobody wants to see these messages.

dsync: Added DSYNC_BRAIN_FLAG_NO_MAIL_PREFETCH to avoid opening mails unnecessarily.

dsync: Moved all doveadm-specific code to doveadm-dsync.c

dsync: mailbox_metadata.cache_fields must be copied to permanent memory. Any other call to mailbox_get_metadata() would have cleared the memory.

dsync: Include messages_count in the mailbox states. This allows detecting that stateful dsync can't be done when message count is suddenly wrong.

dsync: If dsync was started via doveadm-server, show the remote client's IP address in ps output.

dsync: Added more debug output

dsync: Fix to previous commit to actually compile..

dsync: Fixed giving -n "" followed by other -n parameters.

dsync: Fixed using -n "" parameter

dsync: Support multiple -n parameters.

Updated copyright notices to include year 2014.

dsync: Added -P parameter to do a purge for the remote storage after syncing.

dsync: Added -1 parameter to do a "one way sync" without reverting changes. This can be useful during migration when you don't want to delete any mails, but you also don't want to send changes to the old server either.

dsync: Don't crash at deinit when dsync fails early.

dsync: Don't try to find mailboxes from unwanted namespaces.

dsync: If verbose_proctitle=yes, show the current state in it.

dsync: State names were wrong in debug/error messages.

Memory leak fixes.

dsync: If unexpected changes happened during sync, log a warning and exit with code 2. This was done by v2.1 dsync, but the code got temporarily lost in v2.2.

dsync: Added -x parameter to exclude mailboxes from sync. Multiple -x parameters can be added. Giving \flag as parameter means that the mailbox with the given SPECIAL-USE \flag is skipped. For example: doveadm sync -x '\All' -x '\Flagged' -x '\Important' mdbox:~/mdbox

dsync: Small code cleanup.

dsync: -U parameter never updated replicator's full_sync state.

dsync: If dsync fails due to lock timeout, give a better error message.

dsync: Fixed talking to earlier dsync without mailbox attribute support. Most importantly it can now be used as an example how to add more features to dsync.

doveadm sync/backup: Added -g <guid> to sync only the specified mailbox (by GUID) Similar to -m <mailbox name>.

dsync: Use fcntl() locking instead of flock().

dsync: -m '' parameter now syncs mailbox list, but no actual mails.

dsync: If I/O gets stalled, log the state in which it happened.

dsync: Renamed -a parameter to -N. It also now skips invisible namespaces.

dsync: Fix to checking which side should do the locking.

dsync: If locking fails, fail instead of continuing.

dsync: Use full hostname+domain when comparing if hosts are different in locking.

dsync: -l parameter locking is now done on the server with "lower" hostname. This allows running multi-master replication on two servers without two dsyncs mixing up changes by running at the same time.

dsync: Automatically figure out which mailboxes can sync with message GUIDs.

dsync: Renamed "guid_requests" to "mail_requests" The mails aren't necessarily requested by their GUID.

Oops :) Update copyrights to 2013 without breaking all .c files.