doveadm sync/backup: Don't override BROKENCHAR if it's already set This allows migrating invalid mailbox names by specifying BROKENCHAR. Previously it would always try to use \003 control character, which isn't valid character in mailbox names so the mailbox creation would fail.

global: start relying on ssl_iostream_destroy(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - ssl_iostream_destroy(&E); - } + ssl_iostream_destroy(&E);

doveadm: dsync: Switch ioloop for input/output streams while making TCP connection. This task is performed in a sub-ioloop, and when returning from that ioloop, the output stream would sometimes still have an object on the sub-ioloop that was just destroyed.

doveadm: Add NOREPLICATE error when "noreplicate" user flag is used It will be only understood if -U flag has been used

Updated copyright notices to include the year 2018.

doveadm-server: Fix potential dsync-server panic at deinit If the ostream is still used (e.g. for logging) after dsync is finished, it crashed with: Panic: file ostream.c: line 276 (o_stream_sendv_int): assertion failed: (!_stream->finished)

global: Replace ssl_iostream_context_init_client() with ssl_iostream_client_context_cache_get()

global: Rename ssl_iostream_context_deinit() to ssl_iostream_context_unref()

doveadm: Send hostname without ":port" as TLS SNI name for outgoing SSL connections.

global: Rename client_connection_type to doveadm_connection_type

doveadm: mail: Removed redundant cur_username field from struct doveadm_mail_cmd_context.

doveadm: mail: Removed redundant cur_client_ip field from struct doveadm_mail_cmd_context.

doveadm: Changed command contexts to contain the input, output, and connection type values directly. Before, it used a direct pointer to the connection. It used also flags to indicate the connection type, which is now consolidated in the connection type enum.

global: stop including fd-set-nonblock.h & fd-close-on-exec.h directly

dsync: Add hashed_headers setting This makes it possible to configure them

dsync: Fix typo, it's ibc not icb

global: start relying on io_remove{,_closed}(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - io_remove(&E); - } + io_remove(&E); @@ expression E; @@ - if (E != NULL) { - io_remove_closed(&E); - } + io_remove_closed(&E);

global: start relying on i_close_fd(-1) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != -1) - i_close_fd(&E); + i_close_fd(&E);

dsync: fix splitting login from host Using strchr() was splitting login and host at the first occurrence of '@' which leads to troublesome behaviour. When calling strace one would notice the misbehaviour: execve("/usr/sbin/ssh", ["ssh", "-lthomas", "domain.org@10.8.13.2" using strrch() however splits login and host at the last occurrence of '@'. Signed-off-by: Thomas Reifferscheid <thomas@reifferscheid.org>

dsync: Try to commit transactions every dsync_commit_msgs_interval messages This was first attempted to be implemented by ec0cc8fa647794e44a1afaa448f495a713048dc4, but it was later partially reverted by 5973d496b16721af6d2c1fa90b016aacddf13554. This current commit should fix its problems.

global: Fix ssl_set usage Remove verify_remote_cert, as it's always TRUE now. Set allow_invalid_cert to TRUE if verification is not required.

doveadm: Outgoing dsync TCP connections leaked socket If doveadm-server was running with service_count>1, each dsync run leaked a socket.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

dsync: Improve process title during initialization If something is hanging, this should make it clear what exactly it is.

dsync: When logging "Mailbox changed caused a desync", log also the reason. The reason is usually somewhere in the debug logs, but it's difficult to find from there.

global: Replace some settings_parse_line() calls with settings_parse_keyvalue()

lib-storage: Fix doveadm search query parsing to use timezones correctly. When using a date string, e.g. "senton 25-Mar-2007" it should work the same way as IMAP does and show all mails whose sent date is on 25th, ignoring whatever the timezone is in the Date header (or in case of received/saved-date it would be using the timestamps in server's local timezone, again just like IMAP). When search with an exact UNIX timestamp or a time interval (e.g. "3 days"), it should be using UTC times for doing all comparisons.

lib: API change - var_expand*() now returns error string. This allows callers to fail properly if the format string is invalid.

doveadm-sync: Add end-date support

doveadm-sync: Document start date

dsync: Renamed "max sync size" parameter from -S to -I. -S can't be used because it was already used for the generic socket path, so it didn't actually work.

doveadm sync: Fixed -S parameter parsing to actually work.

doveadm sync/backup: Added -S <max size> parameter to skip too large mails.

dsync: Add support for features Add empty_header_workaround as first feature

dsync: Add session ID to log line prefix.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib: remove autoclose parameter from [io]_stream_create_fd Use [io]_stream_create_fd_autoclose() for autoclose.

Remove t_malloc in favour of t_malloc_no0 Using either t_malloc_no0 or t_malloc0 makes it clear whether the allocated memory is zeroed or not.

lib-storage: mail_storage_service_next() now returns error string.

doveadm sync: -R parameter shouldn't switch to "doveadm backup -R" behavior. Just ignore the -R parameter for doveadm sync, unless -1 parameter is also used. Alternatively we could also fail the command, but maybe that's unnecessary extra work.

doveadm: Reverted 7a3b52b8f - doveadm sync -1 -R is useful.

doveadm sync: Don't allow -R parameter. -R implicitly enabled doveadm backup mode, which wasn't supposed to happen.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

doveadm: Fixed reseting getopt() with glibc when processing multiple commands in a single process. This means commands run with doveadm batch or multiple commands in a single doveadm-server connection. glibc's getopt() man page says that optind=1 should reset it for scanning a new argument vector, but this doesn't actually seem to work. Setting optind=0 does work though, so use it everywhere.

Remove now-unnecessary direct stdlib.h #includes.

dsync: Fixed running with tcp/tcps destination.

dsync: Added -D parameter to disable mailbox renaming. The renaming logic is annoyingly complex and there are some bugs left in it. With this parameter renames are never even attempted, but instead a rename would be done (slowly) with mailbox delete + create + fill. Although with imapc protocol mailbox renames are rarely detected anyway.

dsync: Fixed again waiting for remote process wait to die. We can't rely on stderr getting closed. It doesn't happen if the remote process crashes. Now waiting for SIGCHLD in ioloop should solve this and still log all the error messages at exit.

dsync: Make sure we print all the stderr output from remote processes at deinit. Earlier the final messages may have been lost, especially if debug logging was enabled.

dsync: Added -T parameter to specify the I/O stall timeout.

doveadm backup: Fixed assert-crash due to duplicate 'R' in getopt args.

dsync: Renamed -F parameter to -O to avoid a conflict doveadm mail commands already had a generic -F parameter.

dsync: If we stop because of a signal, log a warning about it.

dsync: Stop running if SIGINT/SIGTERM is received.

dsync: Improved error message when remote dsync-server couldn't be started.

dsync: Use storage's mail_error to choose the doveadm exit code. Instead of always assuming that all errors are EX_TEMPFAIL.

dsync: If we get disconnected from remoset server, exit with EX_TEMPFAIL Instead of 1000, which gets truncated to 232.

doveadm: Added support for mail commands to read an input stream (from stdin) This is done by calling doveadm_mail_get_input() from the command's init() function. Currently it reads the entire input into a seekable istream with hardcoded 5 minute timeout. The input stream sending works also through doveadm proxying. This could probably be used by dsync at some point to support proxying over doveadm proxies, but that would require some more work. Especially a flag for commands to specify that they allow non-blocking input streams.

dsync: Added -F parameter to sync only mails with[out] specific flag.

dsync: If same GUID already exists in storage, try to copy it instead of recreating the mail. This way most mailbox backends can reduce disk space by only doing a reference count update. This feature isn't enabled by default. A virtual "All Mails" mailbox needs to be configured using the virtual plugin. Then you need to give this mailbox as -a parameter, e.g.: doveadm sync -a "Virtual/All Mails" ... Currently this is implemented by reading through all the GUIDs in the virtual mailbox. This of course isn't very efficient for things like incremental replication. An upcoming conversation plugin will keep track of all the mails' GUIDs, so in future replication should be able to have this functionality efficiently as well.

dsync: Added -t <timestamp> parameter to save only mails newer than <timestamp> If one side has old mails that don't exist on the other side, they are ignored (not synced and not deleted).

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

dsync: Fixed notifying replicator (-U parameter) if it didn't answer instantly.

doveadm sync/backup: Updated usage string

dsync: Moved doveadm-specific code to doveadm directory.