dsync: Set brain->failed to TRUE when UIDVALIDITY changes This happens if UIDVALIDITY changes during stateful sync. Fixes Panic: doveadm: file dsync-brain-mailbox.c: line 377 (dsync_brain_sync_mailbox_deinit): assertion failed: (brain->failed)

Updated copyright notices to include the year 2018.

dsync: Add per-mailbox sync lock that is always used. Both importing and exporting gets the lock before they even sync the mailbox. The lock is kept until the import/export finishes. This guarantees that no matter how dsync is run, two dsyncs can't be working on the same mailbox at the same time. This lock is in addition to the optional per-user lock enabled by the -l parameter. If the -l parameter is used, the same lock timeout is used for the per-mailbox lock. Otherwise 30s timeout is used. This should help to avoid email duplication when replication is enabled for public namespaces, and maybe in some other rare situations as well.

lib-index: Add mail_index_cache_optimization_settings

global: Use PRI* macros and %zu instead of casting

dsync: Add hashed_headers setting This makes it possible to configure them

dsync: Improve reason when logging "modseq .. no longer in transaction log" Show the current highest, as well as the last common UID and the current next UID. These can help to try to figure out problems.

dsync: Ignore missing remote mailbox when doing unidirectional sync If there are some folders on remote system that are being ignored by remote brain, do not error out.

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

global: Log internal storage error on failure

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

dsync: Fix potential assert on mailbox deinit. Should have happened only in somewhat rare error conditions that require resyncing. Fixes: Panic: file dsync-brain-mailbox.c: line 358 (dsync_brain_sync_mailbox_deinit): assertion failed: (brain->failed || brain->sync_type == DSYNC_BRAIN_SYNC_TYPE_CHANGED)

doveadm-sync: Add end-date support

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

dsync: When full resync is wanted in a stateful sync, output empty state. This continues 3d49dc64d, which didn't actually work because brain->require_full_resync was either cleared earlier or it was never even set in this brain.

dsync: If state is invalid, exit with code 2 instead of tempfail. We'll have dsync_mailbox_import_*() just return success to everything until _deinit() is called. The _deinit() will return a failure and set "resync needed"-flag, which caller will know how to handle.

dsync: Don't export header hashes if remote mailbox is empty. This avoids doing a lot of work of reading through all mails' headers and hashing them, when the hashes aren't actually going to be used for anything.

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Added an extra "finish" state to allow slave-dsync to report error to master-dsync. I'm not sure if that's actually necessary, but just trying to follow the different possibilities on how dsync run can finish made me unsure about it. This should make it at least clear that if a slave-dsync has a failure flag set at deinit master-dsync will know about it before it returns success.

doveadm backup -R: Open the mailbox as readonly, so with imapc EXAMINE is used.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Added DSYNC_BRAIN_FLAG_NO_MAIL_PREFETCH to avoid opening mails unnecessarily.

Avoid logging warnings about increasing memory pool/data stack with DEBUG on. These could have a minor effect on memory usage, but shouldn't be much.

dsync: If incremental sync fails because of desync, log the reason why

dsync: mailbox_metadata.cache_fields must be copied to permanent memory. Any other call to mailbox_get_metadata() would have cleared the memory.

dsync: If we notice that incremental sync won't work, don't continue doing it.

dsync: If incremental dsync finds that its state is stale, retry (later) with full sync. This works only when the master itself notices such changes locally. Remote changes aren't noticed, because the master doesn't send the state to remote dsync and the master is also the one that decides which mailboxes are synced. The fix to this probably means sending the state string to remote dsync and have it check if the state is still valid (= protocol change).

dsync: Include messages_count in the mailbox states. This allows detecting that stateful dsync can't be done when message count is suddenly wrong.

dsync: Added more debug output

dsync: Support multiple -n parameters.

Updated copyright notices to include year 2014.

dsync: have_only_guid128 was never actually being set.

dsync: Support syncing storages with 128bit GUIDs <-> string GUIDs.

doveadm sync: When -1 parameter is used, ignore missing mailboxes in source. Previously the syncs finished with "Mailbox changes caused a desync." and exit code 2. This was especially common when the destination server had new autocreated mailboxes and user logged in after the first dsync.

dsync: Added more consistency when debug logging about changes during sync.

dsync: Improved debug/error logging for changes during sync.

dsync: Added -1 parameter to do a "one way sync" without reverting changes. This can be useful during migration when you don't want to delete any mails, but you also don't want to send changes to the old server either.

dsync: Don't try to find mailboxes from unwanted namespaces.

dsync: Don't try to access mailboxes with no_mail_sync flag enabled.

Memory leak fixes.

dsync: Don't try to sync with GUIDs if we can't set them on the needed side. With two-way syncing both sides need to have writable GUIDs. With one-way syncing only the writing side needs to have writable GUIDs.

dsync: Fixed talking to earlier dsync without mailbox attribute support. Most importantly it can now be used as an example how to add more features to dsync.

dsync: -m '' parameter now syncs mailbox list, but no actual mails.

dsync: Added support for syncing mailbox attributes.

dsync: Improved debug output.

dsync: Various importer fixes. Handle unexpectedly wrong mailbox state better.

dsync: Fixes for syncing without GUIDs.

dsync: Automatically figure out which mailboxes can sync with message GUIDs.

dsync: Renamed "guid_requests" to "mail_requests" The mails aren't necessarily requested by their GUID.

dsync: Removed unused DSYNC_MAILBOX_IMPORT_FLAG_MAILS_HAVE_GUIDS flag.

Oops :) Update copyrights to 2013 without breaking all .c files.

Updated copyright notices to include year 2013.