lib-http: client: Fix request statistics text to properly report send attempts. If the request was first sent in the same ioloop cycle in which the text is generated, the text would claim it was not sent at all yet. With this commit the text now explicitly makes the distinction between request attempts and actual send attempts. The number of attempts is increased at each retry, while the send attempts are increased each time the request is actually being sent to a server.

lib-http: client: Add event fields for the number of bytes sent and received for each request in the last attempt.

lib-http: client: Record last status code in the request object.

lib-http: Create http-common.h, which contains global definitions shared by client and server. Currently contains only the default HTTP port definitions that used to be private to the client.

lib-http: client: Perform output stream error handling in one place.

lib-http: client: Make http_client_connection_lost() available outside http-client-connection.c.

lib-http: client: Handle output stream errors in a separate function.

lib-http: client: Make sure all ioloop objects are created on the ioloop that the client/context is switched to.

lib-http: client: Use merged DNS settings from all clients connected to a shared context for DNS lookups.

lib-http: client: Register client objects in the associated client shared context.

lib-http: client: Made http_client record the current ioloop it is switched to. This prevents http_client_wait() from switching the client to an ioloop it was never explicitly switched to.

lib-http: client: Add support for event API

lib-http: client: Added identifier to client log messages.

lib-http: client: shared context: Allow sharing peer state between clients.

lib-http: client: shared context: Allow sharing host lookups between clients.

lib-http: client: Avoid direct access to host object members from queue.

lib-http: client: Moved connection pool handling from peer to separate object.

lib-http: client: Moved connection list from client to shared context.

lib-http: client: Created basic shared context between clients. In this basic form it only shares default settings.

lib-http: client: Cosmetic changes to http-client-private.h.

lib-http: client: Created separate http_client_queue_get() function. Before, this was combined in http_client_queue_create() which caused a large code block to be indentet. Separating creation and obtaining the current instance makes the code cleaner.

lib-http: client: Send empty payload (Content-Length: 0) for requests that normally expect a payload. This includes the standard POST and PUT methods. Others need to use the new http_client_request_set_payload_empty() function to force sending an empty payload.

lib-http: client: Only drop queued requests when a DNS lookup fails; not also the ones that are already in progress.

lib-http: client: Explicitly destroy the response payload timeout stream when the request is destroyed while receiving payload. This way, the timeout is stopped. This caused ioloop panics.

lib-http: Track request's first and last send attempt times separately.

lib-http: Remove unnecessary enum declaration. Most of these were removed by a8fe899601735459641edae975c0fa08be8482e2

lib-http: client: Fixed i_unreached() failure occurring when a host's list of IPs changes while a connection is still pending. In that case, the IP of the pending connection may no longer be associated with that host. If the IP was not found anymore, the i_unreached() error occurred.

lib-http: client: Treat connections that get disconnected prematurely as connection failures. This means that the backoff time is increased when this happens. A premature disconnection happens when the connection is disconnected before any data is received from the server.

lib-http: client: Consolidated connection loss handling into a single function.

lib-http: Use io_wait_timer for tracking wait times. The previous code was completely incorrect, since it wasn't even tracking the HTTP request's start usecs.

lib-http: client: Destroy host and associated queue objects after no more requests remain and an idle timeout expires.

lib-http: client: Implemented host name lookup TTL. Host name lookups will now be performed again when the results have expired. Without access to TTL information from DNS lookups, all lookups will use the same default TTL for now.

lib-http: client: Added http_client_queue_requests_active(). It counts how many active requests a queue has.

lib-http: client: Link peers to queue earlier: during connection attempts. This makes sure that queues keep track of which peers are doing stuff on its behalf. This is important to be able to manage the active peers when a new host name lookup was performed; if a peer is no longer listed in the returned IPs it should be dropped.

lib-http: client: Unlink all queues from peer when it is disconnected. Before, queues were only destroyed when the whole client was destroyed. This change and subsequent changes prepare for being able to destroy a queue when it becomes unused.

lib-http: client: Added function that counts how many connections are active within a peer.

lib-http: client: Added function that checks whether a connection is currently active.

lib-http: client: Improved handling of hosts with explicit IP address.

lib-http: Add ioloop and lock wait information to timeout messages. It'll now log for example: 9007 Request timed out (Request sent 7.087 secs ago, 0.076 in other ioloops, 7.012 in locks, connected 7.087 secs ago) Which points out that the problem wasn't really with the HTTP, but with locking. This likely should be fixed in some way also in lib-http so that it gives a bit of extra time for reading the request, but that's a separate fix.

lib-http: Support per-request timeout and max_attempts

global: unsigned int:1 -> bool:1 perl -i -pe 's/unsigned int ([^,:;]+):1;/bool $1:1;/' **/*.[ch]

lib-http: client: Fixed bug in handling of lost connections while returning from another ioloop. At one instance the http_client_connection_is_ready() function could have destroyed the connection while the caller still depended on it. Renamed the http_client_connection_is_ready() function to http_client_connection_check_ready(). This now returns -1 when the connection got destroyed. Before it returned a bool that just indicated whether the connection was ready or not. So, there is no need anymore to preserve a connection reference while calling this function.

lib-http: client: Reworked connection close handling. Now, the peer is immediately notified of the lost connection. Before, this step was only taken when the connection was fully dereferenced. To prevent recursive notifications between peer and connection, handling the loss of a connection is deferred to the request handler. When a peer is freed, any associated lingering connections have conn->peer set to NULL.

lib-http: client: Improved labeling of debug messages. Request label is corrected. Labels are now pre-composed and stored.

lib-http: client: Prevent useless and unexpected request callbacks during http_client_deinit(). Requests are now destroyed before queues, hosts, peers and connections. As a side-effect, requests are now removed from the client request list at http_client_request_destroy(), so that requests with lingering references will no longer make http_client_wait() hang.

lib-http: client: Made peer object reference-counted to prevent invalid memory access in request handling routine. Resetting the peer->handling_requests flag risked triggering a segfault, since the peer object could be deleted from within the request handler loop.

lib-http: Changed http_client_request_error to set request to NULL It's going to internally unreference it, so the caller should be aware of it also. I also changed request state check to be an assert, since I don't think there's any safe way this could work otherwise.

lib-http: If connect fails, include attempt count and total time in error. This will produce errors such as: 9002 connect(1.2.3.4:801) failed: Connection timed out in 1.001 secs (4 attempts in 4.706 secs)

lib-http: Include information about number of request attempts and its timing in response reason. Because the reason is usually logged as part of the error string, this causes all of the error messages to include the attempts count and how long the requests took in total. This should make it easier to understand problems in error logs. http_client_request_set_preserve_exact_reason() can be used to disable modifying the reason string. This may also apply to other reason modifications that may be done in the future.

lib-http: client: Cleaned up headers and added some more documentation there.

lib-http: client: Fixed request timeout handling during pipelining. The timeout was not managed correctly. If an earlier request finished, it would not restart the timeout for the next pending request. Also, filling the pipeline caused the timout to be reset inappropriately, postponing its expiry.

lib-http: client: Gave request, connect and dns timeouts defaults that make more sense.

lib-http: client: Fixed crash happening when connection hits an error while handling a new request. At this point the connection dies within the peer request handler loop. Before, the dying connection could free the peer when it ended up being obsolete. But because it was still in the loop, a segfault occurred as the loop continued. Fixed by deferring the the deallocation of the peer from the connection_lost() function to the request handler loop itself if it is active.

lib-http: http_client_request_unref() now always sets *req=NULL This makes its behavior consistent with other APIs in Dovecot. Also http_client_request_finish() no longer sets req=NULL, because all of its callers already keep a reference. Instead added an assert to make sure the reference is there.

lib-http: http_client_connection_unref() now always sets *conn=NULL This makes its behavior consistent with other APIs in Dovecot.

lib-http: client: Fixed pipelining bug: client sometimes sent new request while still waiting for 100-continue.

lib-http: client: Make sure that any pending request is aborted and destroyed before connection FDs are closed. This way, any payload io struct created from the response callback can be freed before the associated FD becomes invalid. This would cause an assert failure otherwise.

lib-http: Assign an ID for each HTTP request and log it in debug lines. The ID stays the same when request is retried. Added a "Req" prefix so it's easier to search for the IDs. Based on patch by Stephan Bosch.

http-client: Added support for using an HTTP proxy running on a unix socket.

lib-http: client: Fixed handling of request timeout. It was inappropriately active when the client needed to take action. This showed particularly with large payloads sent using http_client_request_send_payload().

lib-http client: Added inline function to check whether request is directed at a proxy.

lib-http client: Implemented simple authentication using Basic scheme.

lib-http client: Minor whitespace cleanups in struct http_client_request.

lib-http: client: Fixed handling of requests aborted while still sending payload to server.

lib-http: client: Fixed conflict between request abort and request delayed error.

lib-http: client: Fixed proxied client connect timeout. Timeout didn't cause requests to be aborted with error.

lib-http: client: Implemented maximum for connection backoff time. Without a maximum, the backoff time grows exponentially to enormous values, because it isn't reset until the connection succeeds. This causes recovery from connection failures to become very slow. Current maximum is one minute.

lib-http: client: Created list of all submitted requests. Currently only needed to improve debugging of hanging requests.

lib-http: client: Fixed behavior for max_connect_attempts with fewer IPs than attempts. This was broken by earlier 'fix'.

lib-http: client: Fixed handling of connection timeouts. A timeout caused an assert failure, because the connection status was altered before the peer_connection_failure handler was called.

lib-http: client: Moved delayed failed requests from host to client object.

lib-http: Fixed detecting disconnection when ioloop is running only intermittently. This fix only applies to ioloops created and run by lib-http itself.

lib-http: client: Added support for absolute request timeout. Requests cannot survive beyond this deadline.

lib-http: client: Fixed problem occuring when a nested ioloop was run inside a request callback using the same client. If requests in the nested ioloop would use the same connection as the one that called the callback, the requests would (in the best scenario) all be doomed to time out.

lib-http: client: Fixed recovery after connection failure. If a parallel already connected connection was active, the queue wasn't notified of the failure. Only pending connections should be considered in this case and not established ones.

lib-http: client: Implemented support for connection failure backoff.

lib-http: client: Added support for attempting a single IP several times. Also limits the number of attempts when there are many IPs.

lib-http: Debug log now includes where HTTP requests spent their time on.

http: Implemented delayed requests scheduling. Requests can now be (re)submitted with a delay. The request is not sent until the delay time expires. This facilitates handling the Retry-After header in responses. This can either be performed automatically if the indicated delay is not too long or explicitly by the code using lib-http.

http-client: Improved handing of delayed request errors. Originally each request would have its own zero timeout, but now only one timeout is put in the host object.

http-client: Changed struct http_client_host_port into a struct http_client_queue object. Peer and request objects now reference the queue object directly rather than the host object. This way, there is no need to find the matching host:port in the host anymore. This makes the queueing structure more intuitive and more efficient. This is a first step towards support for connecting to HTTP services through unix sockets or directing requests at specific hosts (so not from the URL). This patch also fixes a potential timeout leak (to_connect) in http_client_host_port (now http_client_queue) and makes sure it is moved during switch_ioloop(). Finally it updates the structure comment at the top of http-client.c.

lib-http: http-client: Added support for tunneling SSL conntections through proxy.

lib-http: Added support for creating CONNECT tunnels through HTTP.

lib-http: http-client: Implemented proxy support.

lib-http: http-client: Added request error code for broken payload input stream. This error is triggered when reading from the provided payload input stream fails while sending the request. Previously this would yield the same error code as for a failure to write to the connection output.

lib-http: http-client: When client request is retried based on a response received from server, return full response to caller if retry is not possible. Before, it would construct a dummy-response that basically retains only the status and reason elements. This is currently only relevant for the way a 417 Expectation Failed response from the server is handled.

lib-http: Added setting for User-Agent header.

lib-http: http-client: Allow overriding all implicitly generated special headers. Which are: Connection, Content-Length, Date, Expect, Host, and Transfer-Encoding.

lib-http: Added support for asynchronous payload for requests. This means that the payload stream passed to the request can be a non-blocking socket stream from some other connection (e.g. proxy client connection).

lib-http: http-client: Requests now automatically generate a Date header. The used date value is normally the submission time of the request, but it can be set explicitly.

lib-http: Fixed client connection and peer log labels to show proper IPv6 peers. The labels did not put IPv6 addresses in square brackets.

lib-http: Removed useless prev,next fields from struct http_client_request.

lib-http: http-client: Fixed request scheduling and connection management.

lib-http: Minor comment update.

lib-http: Fix/cleanup to handling connect failures. There was some kind of double-error handling which messed up things.

lib-http: Fixed tracking number of pending connections to host-port.

lib-http: Always try to connect to host's all IPs when connections fail. Previously this was done only when the new connections started from the first IP.

lib-http: Minor code cleanup.

lib-http: Added soft_connect_timeout_msecs setting to connect to multiple IPs in parallel. Based on patch by Stephan Bosch.

lib-http: After peer has received 100 response, don't add ambiguity timeout anymore. The server should be sending the 100 responses then, and long delays shouldn't be confused with them being missing.

lib-http: Added connect and request timeout settings.

lib-http: Prepare for TLS SNI support.

lib-http: Pass connect failures all the way to request callback's error string.

lib-http: Simplify error handling by delaying connect() failures.

lib-http: If http_client_request_submit() fails, don't immediately call the callback. This simplifies the caller's error handling since there is now only one error code path instead of two.

lib-http: Each peer doesn't need a separate ssl context, enough to have one for http_client. Also removed #ifdefs for building with SSL support. lib-ssl-iostream nowadays dynamically loads the SSL library when needed, and also handles failures if Dovecot was built without SSL support.

lib-http: Callback can now request a retry with http_client_request_try_retry() This can be useful for handling "try again" errors from HTTP servers.

lib-http: If host can't be connected to, don't keep retrying with more than one connection. This also fixes infinitely trying to reconnect to host and retry.

lib-http: Allow caller to find out what state a request is in.

lib-http: Avoid hanging on urgent requests. Patch by Timo & Stephan. There are still some problems though, all urgent requests don't seem to get a new connection.

lib-http: If we get disconnected, say exactly what the error was.

lib-http: Track peers using a linked list also. Mainly because this makes it much easier to debug using gdb. The overhead isn't that much extra either.

lib-http: Track list of hosts using a linked list also. http_client_switch_ioloop() is done very often, and scanning a few entries in a linked list is much faster than going through a hash table.

lib-http: Added http_client_request_set_destroy_callback() This is useful for io_remove()ing the payload stream's fd at the right time.

lib-http: Various bugfixes

lib-http: Various fixes Patch by Stephan Bosch.

lib-http: Added support for chunked input/output streams and some bugfixes.

lib-http: Added initial HTTP client implementation.