Updated copyright notices to include the year 2018.

director: avoid comparison between signed and unsigned ints

director: Fix logging disconnection error reasons The previous commit set errno=0, which weren't logged. If error string is provided, it doesn't matter what the errno is set (as long as it's not 0), so we'll just use EINVAL.

director: Log connection stats for all types of disconnections

director: Log whether connection is synced when it's being disconnected.

director: Fix crash when handling expired USER timestamps. The fix in 154f91726624265fce15097eb4bbbf6e55f8c477 wasn't complete.

director: Fix logging of uninitialized PONG buffer size PONG always has 0 or 2 parameters. The easiest fix here is to just not even try to support 1 parameter.

director: Improve PING/PONG timeout errors and add new warnings Log a warning if PING-PONG takes over 5 secs in total, or also if the recipient notices that the PING took >= 5 seconds to receive.

director: Add director_ping_idle/max_timeout setting. director_ping_idle_timeout is used when there's otherwise no input coming from the connection. Changed its default from 10 secs to 30 secs. director_ping_max_timeout is used when the other director keeps sending input, but among it is no PONG reply.

director: Log exactly how long PING was waited on before it timed out.

director: DIRECTOR-LIST - Show number of USERs sent/received in "handshaking" state

director: Include number of users sent in handshake in disconnection log lines

director: Show number of incoming USERs/sec in ps title

director: Include number of USERs received in disconnection log lines Separate handshake and refresh USERs.

director: Don't send USERs in handshake that were already sent between handshake If the user was refreshed since the handshake was started, it means that the same user was already sent to the other side (added to the stream immediately after it was received/handled). There's no need to send it again. This fixes a potentally infinite handshake when users are rapidly changing and the handshake iterator never sees the end of the list. (Each refreshed user is moved to the end of the list, so handshaking can keep sending the same user over and over again.)

director: Avoid USER loops with >1s ring latency also with old directors Do this by ignoring USER refreshes that were already updated recently. The "recently" is calculated by director_user_expire/4 seconds ago, but with an upper limit of 15 secs. This means that the USER loops can now only exist if the director ring latency is above 15 seconds. (Once all directors in the ring are running the new version, there's no looping at any latency.)

director: Avoid USER loops when ring latency is over 1 second Do this by adding a timestamp parameter to USER events. This way if it takes over 1 second for the USER event to traverse the ring, it won't get into an infinite loop getting the user updated over and over again.

director: Replace USER command during handshake with "U" This clearly differentiates the two commands and allows extending the USER command with new parameters without mixing it up with the handshake-USER.

director: Ignore refresh requests for already expired user timestamps

director: Keep users unsorted during handshake and sort them at the end This is simpler and sometimes more efficient than the current way of immediately inserting the users to the correct place in the linked list. This is especially useful if handshaking is mixed with regular USER updates, because each switch between them required walking the linked list backwards to find the proper insert position. It's not a big problem if the users list is temporarily unordered. It mainly means that some of the users won't be expired as early as they should have.

director: Make sure users are sorted after unfinished handshake The users were sorted after the handshake was finished, but if the connection was closed before that hapepned, the users were left unsorted. This could have caused the users to not expire early enough.

director: Make sure user's timestamp isn't set to future The sending director could have the system clock slightly in the future.

director: Reconnect after detecting a write failure to director If disconnection is detected during write failure, or "Output buffer full" occurs, the connection is disconnected. However, if this was the right side connection, it wasn't automatically reconnected to. This left the ring nonworking.

director: Include used CPU secs in director connection log messages It's counting the process's full CPU seconds used since the handshake started, so it's not specific to the connection itself. Still, this is likely to be very useful in debugging whether a slow handshake was due to CPU usage or something else.

director: Deduplicate code into director_connection_set_connected()

director: Include peak output buffer size in director connection log messages

director: Don't block too long when sending users during director handshake All the other work is blocked while the users are being sent.

director: Avoid str_printfa() in director_connection_send_users() Optimizes the CPU usage.

director: Use *_host.ip_str to avoid net_ip2addr() calls

director: Use t_strsplit_tabescaped_inplace() for director connection input

director: Fix spelling mistakes Original work by @jsoref

global: Fix spelling mistakes in comments Original work by @andreasschulze and @jsoref

director: Track show long the last ring sync took.

director: Track connections' last ping time

director: Add director_connection_get_status() Can be used to access connection-specific status information.

director: Fix updating director connection's last_output timestamp It was previously updated only in ostream's flush callback, which was called only when there were a lot of output. This only caused the "last output" timestamp in disconnection log lines to be wrong.

director: Close director connection immediately when output buffer is full Only the ostream was closed, which didn't actually cause the disconnection until the other side closed the connection.

director: Add director_output_buffer_size setting This allows configuring the max buffer size for outgoing connections. Previously it was hardcoded to 10 MB, which wasn't necessarily enough for very busy directors.

director: Don't use undefined DIRECTOR_CONNECTION_PING_TIMEOUT_MSECS

director: Don't crash if DIRECTOR-REMOVE is received for itself This triggers the director removal from the ring, which causes the connection to be destroyed. But since we're still in the middle of handling the connection it needs refcounting.

global: start relying on timeout_remove(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - timeout_remove(&E); - } + timeout_remove(&E);

global: start relying on io_remove{,_closed}(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - io_remove(&E); - } + io_remove(&E); @@ expression E; @@ - if (E != NULL) { - io_remove_closed(&E); - } + io_remove_closed(&E);

director: Ignore CONNECT requests to hosts that have been removed already

director: After CONNECT was received, make sure we disconnect The remote side won't be reading anything after the CONNECT, so we have to disconnect anyway. If we decide that the CONNECT request is wrong, reconnect after a short delay and hope that the remote agrees with us the next time.

director: Don't reset directors' last_network_failure while handshaking The reset is done mainly to make it faster for a director that has been down to connect back to the ring, without other directors still thinking that it's down. But DIRECTOR that is sent during handshake says nothing about whether the director is up at the moment or not.

director: Log info line for every incoming/outgoing connection This can help with debugging problems.

director: Log info line whenever a director is added/removed from ring This can help with debugging problems.

director: Fix rapid reconnection on failed outgoing connections last_network_failure wasn't set, which caused a failed outgoing connection to immediately reconnect to it. This resulted in rapid logging of connect() errors.

director: Fix logging last input/output time when there was no input/output

director: Fix whitespace position in some log messages

director: Use longer timeout for receiving user list in handshake The sender also has a send timeout, so this allows the sender to abort early and give a nicer error message.

director: Log more connection state information on handshaking errors

director: Use more accurate timestamps for handshake timeout logging There could be some time between the create, connect and when ME was received.

director: Fix "sending handshake timed out" error to be actually logged io was never NULL here, so it was never logged.

director: Fix handshake timeout lengths They weren't too bad, but also they weren't what the #defines described. Also added a separate connect() timeout.

director: Make director_consistent_hashing=yes mandatory

director: Log vhost count changes and HOST-UP/DOWN

Updated copyright notices to include the year 2017.

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

director: Keep per-tag directory

director: Code cleanup - Reorder includes This is required to enable typedef of user_free_hook

director: Code cleanup - added user director_iterate_users_*() wrappers. This will shrink the diff for the following changes.

director: Code cleanup - use temporary users variables This will shrink the diff for the following changes.

director: Moved all user killing state to struct director_kill_context This should make it a bit easier to understand the life time of user killing. It also simplifies code by removing struct director_user_kill_finish_ctx. Finally, this already reduces memory usage with 32bit systems, and would make it possible to reduce also on 64bit systems if timestamp is shrank to 31 bits and weak bit moved after it. I'm not sure if that would be better for performance though. In any case it would provide free space for 4 extra bytes if that were needed in future.

director: Code cleanup - added USER_IS_BEING_KILLED() macro

director: Improve debug logging output.

director: Fix user move hangs when another move is triggered early. Running "doveadm director flush" before the previous flush hadn't fully finished could have caused users to hang until their move timeout: director: Error: Finishing user 3249070169 move timed out, its state may now be inconsistent (state=waiting-for-everyone)

director: If user host conflict is detected, make sure new host is sent back. USER-KICK-HASH was sent, but the sender didn't get back a USER reply with the new host. This could have increased how long user's host differred in directors. Avoids repeating this error: Error: User hash 2957018085 is being redirected to two hosts: 10.0.0.30 and 10.0.0.201 (old_ts=1477338836)

director: Log user's kill_state with a human-readable string.

director: Fix sending up/down state in handshakes. They were never sent, because HOSTs were sent before director had waited for the remote to send its version number. So sender thought that the remote's minor_version was too old and it didn't send the up/down state at all. This caused errors like: Warning: director(10.0.0.30:9090/left): Host 10.0.0.30 is being updated before previous update had finished (up -> down) - setting to state=down vhosts=100 Error: director(10.0.0.30:9090/left): Director 10.0.0.30 SYNC request hosts don't match us - resending hosts (seq=6, remote hosts_hash=262126213, my hosts_hash=2458934259)

director: If SYNCs are received during handshake, send them later. This fixes delays during handshake: Error: Ring SYNC appears to have got lost, resending

global: Replaced t_strsplit_tab() calls with t_strsplit_tabescaped() This is useful especially in auth code to support LFs in extra fields. Other pieces of code were also tab-escaping strings, but never unescaping them. Usually it didn't matter, because nobody would use the escaped characters. Still, the code wasn't exactly behaving correctly. One downside to this change is that it's now possible to pass through TABs, CRs and LFs through the various protocols. In theory this shouldn't cause any problems, but combined with other bugs this could trigger some security problems.

doveadm director kick: Added -f <passdb field> parameter. This works for all the user_* passdb fields.

director: Added more debug information to "Ping timed out" error

global: Removed redundant corking in ostream flush callbacks.

global: Use only explicit int -> bool conversions These were checked with a patched clang.

global: Require comparisons to be strict boolean expressions * No implicit integer -> boolean or pointer -> boolean conversions * !expr can be used only if expr is boolean type These were checked with a patched clang. It found various actual bugs, which were fixed by the previous commits.

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib: remove autoclose parameter from [io]_stream_create_fd Use [io]_stream_create_fd_autoclose() for autoclose.

director: Fixed error handling when directors support incompatible tags Connection should have been disconnected immediately, not after the next command that would have produced "Incompatible protocol".

director: Fixed ignoring an obsolete up/down change while host is desynced.

global: Use [io]_stream_get_error() insted of %m

director: Avoid a potential assert-crash after removing a director from ring. This should fix the crash: Panic: director: file director-connection.c: line 1926 (director_connection_init_out): assertion failed: (!host->removed) Also moved the last_network_failure timestamp reset a bit later, since there's no need to reset the timestamp if we're not actually connecting to the server.

director: Don't re-send SYNC unnecessarily often.

director: Fixed user weakness getting stuck if multiple directors set user weak simultaneously

director: "Host is being updated before previous update had finished" logs now details.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

director: Don't trigger a ring resync if only last_updown_change has changed.

director: Fixed backend selection when multiple tags were used. The previous algorithm was causing an uneven load for backends. This change breaks compatibility with older director servers that were using tags because of the different selection algorithm. The new director code refuses to run within a cluster with old directors if tags are used.

director: Code cleanup - access host->tag via mail_host_get_tag() In preparation for the following changes.

director: Code cleanup - make most mail_host_*() list parameters unnecessary.

director: Remember backends' hostnames and send them in login reply. This allows login processes to verify the remote server's hostname in SSL certificate.

director: Small code cleanup - make it easier to add parameters to HOST

director: Log a warning if directors' clocks are too much out of sync.

director: Detect if directors' hosts have become desynced by sending hosts_hash in SYNC parameter. Also fix up such a situation by resending all HOSTs.

director: Use mail_hosts_hash() to improve debug log messages.

director: Small code cleanup - make it easier to add parameters to SYNC

director: Don't become desynced if two directors change the same backend in incompatible ways. This would have caused "User hash .. is being redirected to two hosts" errors, which wouldn't easily go away as the directors have a different view of what hosts currently exist.

Removed all invocations of atoi().

Changed type of internet port values to in_port_t everywhere. Created special SET_IN_PORT setting type for internet port values. Created net_str2port() for parsing internet port values. Removed several atoi() invocations in the process.

director: explicitly mark _disconnect() helper static to match proto No functional change, compiler would not have exported the symbol, this just shuts up sparse. Signed-off-by: Phil Carmody <phil@dovecot.fi>

director: Added "up" vs "down" states and doveadm director up/down commands. These commands are intended to be used by automated watchdogs that detect if backends are up or down. This way the vhost count doesn't get forgotten after server goes down. It also means that admin can manually take down a server by setting its vhost count to 0 without the watchdog automatically bringing it back up.

director: Moving a user to another host sometimes caused the move to fail. It could have given "User hash .. is being redirected to two hosts" error and afterwards moved the user back to its original host.

director: Don't send DIRECTOR command infinitely in loop if that director got disconnected.

director: Include useful statistics in process title.

director: Fixed crash if director sent invalid data too early.

director: If we disconnect a director, pass the reason all the way to deinit's debug logging.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

director: Added support for backend cluster "tags". This allows using a single director ring for multiple backend clusters. By default everything has an empty tag. A passdb lookup can return "director_tag" field containing the wanted tag name. If there aren't any backend servers with the wanted tag, it's treated the same as if there aren't any backend servers available (= wait for 30 secs for a backend and then return temporary failure). Tags can be added to configuration by adding @tag suffix to IPs/hosts. For example: director_mail_servers = 10.0.0.100-10.0.0.110@name1 10.0.0.120@name2 "doveadm director add" can also add tags either with @tag suffix or with -t parameter. "doveadm director status user@domain" requires giving the user's correct tag with -t parameter or the results won't be correct (empty tag's results are shown). Tags can't currently be changed for an existing host without removing it first.

director: Added director_consistent_hashing setting. This should have been the default since the beginning. I didn't thik it was worth the trouble originally because in theory it shouldn't matter much if users jump between multiple backends. But this makes caching behavior worse, especially for systems which are using local cache files.

director: Always log a message when director handshake finishes.

doveadm director ring status shows now more information

director: If we detect that a user is being proxied to multiple backends, disconnect wrong connections. Especially IMAP connections can otherwise stay alive for a long time and cause problems.

doveadm director kick command added. The kick gets sent to all the proxies within the director ring.

director: Fixed handling weak users when there were exactly 2 director servers in the ring.

Updated copyright notices to include year 2014.