Updated copyright notices to include the year 2018.

global: Replace o_stream_nfinish() with o_stream_finish() whenever possible

global: Use PRIdTIME_T and PRIxTIME_T format specifiers

global: Use PRI* macros and %zu instead of casting

global: stop including fd-set-nonblock.h & fd-close-on-exec.h directly

dsync: Add hashed_headers setting This makes it possible to configure them

dsync: Add missing space to handshake optional keys

global: start relying on io_remove{,_closed}(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - io_remove(&E); - } + io_remove(&E); @@ expression E; @@ - if (E != NULL) { - io_remove_closed(&E); - } + io_remove_closed(&E);

dsync: Ignore missing remote mailbox when doing unidirectional sync If there are some folders on remote system that are being ignored by remote brain, do not error out.

dsync: Use header hashing version 3

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

dsync-server: Fix support for dsync_features=empty-header-workaround Fixes: Panic: Unknown key: empty_hdr_workaround

dsync: Fix syncing attributes with large values. This mainly meant that large Sieve scripts weren't synced properly, because their last_change field was never deserialized, so it was set to 0.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

global: Replaced t_strsplit_tab() calls with t_strsplit_tabescaped() This is useful especially in auth code to support LFs in extra fields. Other pieces of code were also tab-escaping strings, but never unescaping them. Usually it didn't matter, because nobody would use the escaped characters. Still, the code wasn't exactly behaving correctly. One downside to this change is that it's now possible to pass through TABs, CRs and LFs through the various protocols. In theory this shouldn't cause any problems, but combined with other bugs this could trigger some security problems.

doveadm-sync: Add end-date support

dsync: Further fixes to received_timestamp and virtual_size parsing (-t and -S parameters)

dsync: Added missing fields to ibc-stream.

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

global: Removed redundant corking in ostream flush callbacks.

dsync: When full resync is wanted in a stateful sync, output empty state. This continues 3d49dc64d, which didn't actually work because brain->require_full_resync was either cleared earlier or it was never even set in this brain.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

dsync: race condition where done/finish is received after one side has closed We do not tell the remote we are closing if they have already told us because they close the connection after sending ITEM_FINISH or ITEM_DONE and will not be ever receive anything else from us unless it just happens to get combined into the same packet as a previous response and is already in the buffer.

lib: istream - migrate more to i_stream_read_more() More coccinelle semantic patching, again hand-checked. git grep 'i_stream_read_data' | sed s/:.*// | \ while read f; do spatch --sp-file istream0.cocci --in-place "$f" ; done -- 8< --- istream0.cocci --- @@ expression e1, e2, e3; @@ - i_stream_read_data(e1, e2, e3, 0) + i_stream_read_more(e1, e2, e3) -- 8< --- end Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: When comparing headers' hashes to match messages, try to normalize the input. This is especially useful because some IMAP servers return different data depending on whether we're fetching only specific header fields, all headers or entire body. For now we're assuming that any non-ASCII is going to be replaced with '?', which helps at least with Zimbra and Yahoo. The header hashing algorithm is now versionable, so it can be modified more easily in future. This change should make imapc_features=zimbra-workarounds setting obsolete.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

dsync: Added DSYNC_BRAIN_FLAG_NO_NOTIFY to enable MAILBOX_TRANSACTION_FLAG_NO_NOTIFY It's arguable that this should be enabled by default, but people might like to keep mail_log notifications for dsync.

Remove now-unnecessary direct stdlib.h #includes.

Replaced unlink() calls with i_unlink*() wherever possible.

dsync: If remote disconnects, log the last sent/recv state.

dsync: If we disconnect with I/O timeout, log the last sent/recv state.

Removed all invocations of strtoll() and friends.

dsync: Added -D parameter to disable mailbox renaming. The renaming logic is annoyingly complex and there are some bugs left in it. With this parameter renames are never even attempted, but instead a rename would be done (slowly) with mailbox delete + create + fill. Although with imapc protocol mailbox renames are rarely detected anyway.

dsync: Added -T parameter to specify the I/O stall timeout.

dsync: Don't try to use the new "finish" state with old dsync versions.

dsync: Fixed returning wrong mail_error in some remote dsync.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: Added an extra "finish" state to allow slave-dsync to report error to master-dsync. I'm not sure if that's actually necessary, but just trying to follow the different possibilities on how dsync run can finish made me unsure about it. This should make it at least clear that if a slave-dsync has a failure flag set at deinit master-dsync will know about it before it returns success.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Added DSYNC_BRAIN_FLAG_NO_MAIL_PREFETCH to avoid opening mails unnecessarily.

dsync: Reset I/O timeout every time when receiving input. This could have caused unwanted I/O timeouts when receiving large mails.

Added (void) prefix for ignoring return values we don't care about. Hopefully this quiets down Coverity warnings also of them?

dsync: saved-date doesn't need to be looked up until mail body is being read. This should improve the performance when the saved-date isn't already cached or otherwise quickly accessible. This change also makes dsync slightly incompatible with earlier versions. When using dsync with an earlier version the saved-dates aren't synced. It would be too much trouble to try to preserve full backwards compatibility, especially because saved-date doesn't matter so much and isn't even visible to IMAP clients.

dsync: Yet another fix to working with old versions.

dsync: Fixed new dsync versions to work again with older versions.

dsync: Include messages_count in the mailbox states. This allows detecting that stateful dsync can't be done when message count is suddenly wrong.

dsync: Added assert

Use io_add_istream() wherever possible. This shouldn't fix anything, but might make some functionality easier to implement in future.

dsync: Support multiple -n parameters.

Updated copyright notices to include year 2014.

dsync: Support syncing storages with 128bit GUIDs <-> string GUIDs.

dsync: Use i_stream_get_error() instead of just errno in stream error messages.

dsync: Added -P parameter to do a purge for the remote storage after syncing.

dsync: We didn't send the new protocol version in handshake. This caused problems when syncing mailbox formats that didn't support saving GUIDs.

dsync: Stream disconnection error message improvement

dsync: Memory leak fix at deinit.

dsync: After reading remote hostname, use it for the log messages instead of IP.

dsync: Recent changes broke remote dsync

dsync: Added -1 parameter to do a "one way sync" without reverting changes. This can be useful during migration when you don't want to delete any mails, but you also don't want to send changes to the old server either.

dsync: Send DSYNC_BRAIN_FLAG_NO_MAIL_SYNC via ibc-stream to slave brain.

dsync: Fixed dsync handshaking since recent change.

dsync: Added -x parameter to exclude mailboxes from sync. Multiple -x parameters can be added. Giving \flag as parameter means that the mailbox with the given SPECIAL-USE \flag is skipped. For example: doveadm sync -x '\All' -x '\Flagged' -x '\Important' mdbox:~/mdbox

dsync: Previous have_save_guids change somewhat broke compatibility with earlier dsync versions.

dsync: Fixed unsubscribing from an already deleted mailbox.

dsync: Don't try to sync with GUIDs if we can't set them on the needed side. With two-way syncing both sides need to have writable GUIDs. With one-way syncing only the writing side needs to have writable GUIDs.

dsync: Fixed talking to earlier dsync without mailbox attribute support. Most importantly it can now be used as an example how to add more features to dsync.

doveadm sync/backup: Added -g <guid> to sync only the specified mailbox (by GUID) Similar to -m <mailbox name>.

dsync: Fixed syncing attribute streams.

lib-storage: Mailbox attributes can now be accessed via istreams. The idea is to use istreams for larger values.

dsync: Fixed sending mailbox attributes with stream ibc.

dsync: Added support for syncing mailbox attributes.

dsync: If I/O gets stalled, log the state in which it happened.

dsync: Renamed -a parameter to -N. It also now skips invisible namespaces.

dsync: Merge last-common-* values from both local and remote when they differ.