lib-http: client: Add event fields for the number of bytes sent and received for each request in the last attempt.

lib-http: client: Uncork and flush the output stream explicitly after continuing request payload. This allows detecting any output stream errors.

lib-http: client: Hold reference to the connection while continuing request payload. Subsequent changes may cause the connection to be destroyed while sending.

lib-http: client: Continue sending request payload in a separate function.

lib-http: client: Hold reference to the connection while sending the next request. Subsequent changes may cause the connection to be destroyed while sending.

lib-http: client: Perform output stream error handling in one place.

lib-http: client: Make http_client_connection_lost() available outside http-client-connection.c.

lib-http: client: Handle output stream errors in a separate function.

lib-http: client: Recreate connection IO after streams change.

lib-http: client: Use the new connection_input_halt/resume() functions.

lib-http: client: Use conn->conn.ioloop rather than cctx->ioloop in connection. This leads to somewhat simpler code, and makes having connections on different ioloops easier in the future.

lib-http: client: Move connection to shared context's ioloop, rather than the current_ioloop. This is an omission.

lib-http: client: Fix using non-context SSL settings The SSL settings were used for the SSL context, but they weren't used for individual SSL streams. This broke stream-only settings, like allow_invalid_cert=TRUE.

lib-http: client: Make sure all ioloop objects are created on the ioloop that the client/context is switched to.

Updated copyright notices to include the year 2018.

lib-http: client: Check whether client is running in a wait loop with the correct condition. This used to check client->ioloop != NULL, but now it should check client->waiting. Previous commit (e8a1b62fe4a81b211dcccd1a58b44f254074eab6) omitted a few instances where this should have been changed.

lib-http: client: Replace http_*_debug() with e_debug()

lib-http: client: Remove redundant debug logging checks

lib-http: client: Add support for event API

lib-http: client: Added identifier to client log messages.

lib-http: client: shared context: Allow sharing peer state between clients.

lib-http: client: Moved connection pool handling from peer to separate object.

lib-http: client: Moved connection list from client to shared context.

global: start relying on timeout_remove(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - timeout_remove(&E); - } + timeout_remove(&E);

global: start relying on io_remove{,_closed}(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - io_remove(&E); - } + io_remove(&E); @@ expression E; @@ - if (E != NULL) { - io_remove_closed(&E); - } + io_remove_closed(&E);

lib-http: Restructured the header, message, request, and response parsers to have an extensible and consistent API using flags. Extended the test suites with a few cases that test parsing with and without the STRICT flag.

lib-http: client: When a request is destroyed prematurely during payload input, consider the payload stream destroyed and act accordingly. The application may hold a reference to the payload stream still, and it may be difficult to prevent that. This causes lib-http to keep waiting for the payload to be destroyed. When nothing else is going on, the current ioloop may then become empty, which caused the familiar assert failure.

lib-http: client: Explicitly destroy the response payload timeout stream when the request is destroyed while receiving payload. This way, the timeout is stopped. This caused ioloop panics.

lib-http: client: Implemented means to obtain request statistics.

lib-http: client: Fixed request-specific attempt timeout. This is the timeout applied to a single request attempt. Using http_client_request_set_attempt_timeout_msecs() this can be set for a specific request. However, this was mostly ignored for requests that weren't in the process of handling response payload. Instead, the global request_timeout_msecs client setting was used. Also amended the (currently manual) test suite with tests that demonstated the problem and now verify the fix.

global: Use http_response_get_message() to log HTTP responses.

global: Fix ssl_set usage Remove verify_remote_cert, as it's always TRUE now. Set allow_invalid_cert to TRUE if verification is not required.

lib-http: client: Make sure req->conn is only not NULL when that connection holds a reference to that request. This consolidates the management of req->conn to one place, thereby preventing mishaps. It makes sure req->conn is always properly assigned, making it more reliable. This fixes a problem that emerged in the http-proxy.

lib-http: client: Keep connection reference throughout http_client_connection_return_response(). Before, it was held in two intervals. Next commit requires this change, but it also simplifies this function significantly.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

lib-http: client: Treat connections that get disconnected prematurely as connection failures. This means that the backoff time is increased when this happens. A premature disconnection happens when the connection is disconnected before any data is received from the server.

lib-http: client: Consolidated connection loss handling into a single function.

lib-http: Use io_wait_timer for tracking wait times. The previous code was completely incorrect, since it wasn't even tracking the HTTP request's start usecs.

lib-http: client: Added function that checks whether a connection is currently active.

lib-http: client: Created http_client_connection_failure() handler.

lib-http: client: Set conn->connect_request to NULL immediately in http_client_connection_tunnel_response(). Prevents referring to this completed request in other parts of the code.

lib-http: Add ioloop and lock wait information to timeout messages. It'll now log for example: 9007 Request timed out (Request sent 7.087 secs ago, 0.076 in other ioloops, 7.012 in locks, connected 7.087 secs ago) Which points out that the problem wasn't really with the HTTP, but with locking. This likely should be fixed in some way also in lib-http so that it gives a bit of extra time for reading the request, but that's a separate fix.

global: Make sure i_stream_read() calls handle 0 and -2 return values correctly.

lib-http: client: Fixed hang occurring when nested ioloops are used in response callbacks. To prevent missing disconnect events, i_stream_read() is called once a change in ioloop is detected. However, if something was actually read into the stream, the input handler was never called. So, a response could linger in the stream buffer, without being handled, thereby causing the connection to hang indefinitely. An additional input event could end the hang, but sometimes this doesn't happen before the request times out. This problem was seen in test-http-payload once in about 10 invocations. Obox uses nested ioloops, to this applies there.

lib-http: Support per-request timeout and max_attempts

lib-http: client: Fixed potential segfault problem in http_client_connection_server_close(). Used wrong variable: it is set to NULL by http_client_request_unref() before it is used. Problem found by Coverity.

lib-http: client: Implemented no_auto_retry setting that disables all automatic request retries. This currently only applies to requests sent over a connection that is subsequently lost before a response is received. Before, such requests were always implicitly resumbitted for a new connection, without the application knowing about it. By enabling the no_auto_retry client setting, the application is always notified of connection loss through the request's response callback. As a consequence, requests need to be retried explicitly using the http_client_request_try_retry().

lib-http: client/server: Enable the TCP_NODELAY option for all connections. This disables the TCP Nagle algorithm. With the Nagle algorithm enabled, TCP waits a little to accumulate more data in a small segment before it is sent. For transfer of large continuous payloads, this is not useful and even harmful. If the final remaining bit of the payload is small, the TCP layer will wait for a significant amount of time at the end of the payload. For many sequential transfers, this amounts to much waiting time. This is particularly evident in the test-http-payload test suite tool. Setting TCP_NODELAY decreases its run time from up to 20 minutes to about half a minute my system.

lib-ssl-iostream: Changed require_valid_cert -> allow_invalid_cert We should default to being safe.

lib-http: Improved error logging for net_set_*_buffer_size() failures.

lib-http: client: Fixed bug in handling of lost connections while returning from another ioloop. At one instance the http_client_connection_is_ready() function could have destroyed the connection while the caller still depended on it. Renamed the http_client_connection_is_ready() function to http_client_connection_check_ready(). This now returns -1 when the connection got destroyed. Before it returned a bool that just indicated whether the connection was ready or not. So, there is no need anymore to preserve a connection reference while calling this function.

lib-http: client: Reworked connection close handling. Now, the peer is immediately notified of the lost connection. Before, this step was only taken when the connection was fully dereferenced. To prevent recursive notifications between peer and connection, handling the loss of a connection is deferred to the request handler. When a peer is freed, any associated lingering connections have conn->peer set to NULL.

lib-http: client: Improved labeling of debug messages. Request label is corrected. Labels are now pre-composed and stored.

lib-http: client: Added settings to configure the connection's socket kernel buffer sizes. This is mainly useful for use in the lib-http test suite.

lib-http: client: Improved request reference counting in connection code. It should now always be clear when the connection object holds a reference to a request and when it is released. Only while the reference is held, req->conn points to a connection. This also makes the assertion in http_client_request_unref() more robust and clear.

lib-http: client: Fixed reference counting for requests that are aborted due to an early server response. An additional http_client_request_unref() is (now) unneccessary.

lib-http: Changed http_client_request_error to set request to NULL It's going to internally unreference it, so the caller should be aware of it also. I also changed request state check to be an assert, since I don't think there's any safe way this could work otherwise.

lib-http: use ssl_iostream_settings in http_client_settings

lib-http: client: Fixed request timeout handling during pipelining. The timeout was not managed correctly. If an earlier request finished, it would not restart the timeout for the next pending request. Also, filling the pipeline caused the timout to be reset inappropriately, postponing its expiry.

lib-http: client: Improved test-http-payload to test partial reading of response payload.

lib-http: Fix to earlier http_client_connection_unref() change 1dead6 Patch by Stephan Bosch

lib-http: http_client_request_unref() now always sets *req=NULL This makes its behavior consistent with other APIs in Dovecot. Also http_client_request_finish() no longer sets req=NULL, because all of its callers already keep a reference. Instead added an assert to make sure the reference is there.

lib-http: http_client_connection_unref() now always sets *conn=NULL This makes its behavior consistent with other APIs in Dovecot.

lib-http: client: Fixed pipelining bug: client sometimes sent new request while still waiting for 100-continue.

lib-http: client: Make sure that any pending request is aborted and destroyed before connection FDs are closed. This way, any payload io struct created from the response callback can be freed before the associated FD becomes invalid. This would cause an assert failure otherwise.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-http: Add more timing information to debug logs when HTTP connections get closed.

lib-http: Give a better error message if request times out. Instead of just giving the configured timeout, log how long the timeout actually took for the oldest request in wait list. Also if the request was retried, log how many times the request was sent and how long the request took in total.

lib-http: Added asserts to all refcount increments. These could catch situations where an already freed object's refcount is attempted to be increased.

lib-http: Avoid crashes when server is already disconnected. I'm not sure if this is the nicest fix, but seems to work for now at least.

lib-http: Disconnection from HTTP server could have caused accessing already freed memory. Freeing a request could have freed the connection.

http-client: Added support for using an HTTP proxy running on a unix socket.

Changed type of internet port values to in_port_t everywhere. Created special SET_IN_PORT setting type for internet port values. Created net_str2port() for parsing internet port values. Removed several atoi() invocations in the process.

lib-http: client: Added proper handling of 408 response status. This is treated as a special server connection close event, rather than a response to the last issued request.

lib-http: client: Fixed handling of request timeout. It was inappropriately active when the client needed to take action. This showed particularly with large payloads sent using http_client_request_send_payload().

lib-http: client: Fixed memory leak in CONNECT tunnel support. Forgot to dereference streams once tunnel was completed and passed to the new connection object.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-http: client: Removed connection code that is already in src/lib/connection.c.

lib-http: Make sure we don't access already freed client connection in payload-destroyed callback.

lib-http: client: Removed asssert in http_client_connection_is_ready(). There is no reason why the input stream couldn't be full.

lib-http: client: The asserts recently added to http_connection_check_idle() are bogus.

lib-http: client: Handle situation in which server sends response before request payload is fully sent.

lib-http: client: Fixed handling of requests aborted while still sending payload to server.

lib-http: client: Fixed cleanup of remaining waiting requests at connection destroy. Forgot to dereference the requests for the reference that the connection itself holds for each waiting request.

lib-http: client: Aborted requests were not counted as finished in the connection's response receive loop. This caused a hang.

lib-http: client: Fixed proxied client connect timeout. Timeout didn't cause requests to be aborted with error.

lib-http: client: Clear req->conn when request is finished. This prevents a spurious assert failure.

lib-http: client: Connection erroneously indicated that it was ready while SSL handshake was not yet completed.

lib-http: client: Fixed handling of connection timeouts. A timeout caused an assert failure, because the connection status was altered before the peer_connection_failure handler was called.

lib-http: client: Fixed segfault caused by earlier change.

lib-http: Fixed detecting disconnection when ioloop is running only intermittently. This fix only applies to ioloops created and run by lib-http itself.

lib-http: client: Added support for absolute request timeout. Requests cannot survive beyond this deadline.

lib-http: client: Fixed problem occuring when a nested ioloop was run inside a request callback using the same client. If requests in the nested ioloop would use the same connection as the one that called the callback, the requests would (in the best scenario) all be doomed to time out.

lib-http: client: Fixed aborting request in the middle of sending payload. If the request payload is so big that it cannot be sent all at once, the caller may at some point abort the request when it is still being sent. The bug occurred when the request finally finished sending. It erroneously advanced the state to WAITING rather than remaining ABORTED, thus 'reviving' the request unexpectedly.

lib-http: Fixed assert-crash when http_client_connection_disconnect() is called multiple times.

lib-http: client: Tunnel connection failure would cause segfault. That happened because it tried to retry waiting requests, which makes no sense for a CONNECT tunnel.

lib-http: Debug log now includes where HTTP requests spent their time on.

lib-http: Made sure that connections that are still connecting to the server aren't marked as idle.

lib-http: Updated comments to RFC7230/RFC7231.

lib: Moved rawlog dir stat()s to iostream_rawlog_create() internally. This is required to allow TCP rawlogs to actually work.

lib-http: Fixed problem with connections idling indefinitely. Forgot to check for idle connections when request finished after waiting for payload to be read by application.

Use io_add_istream() wherever possible. This shouldn't fix anything, but might make some functionality easier to implement in future.

lib-http: Wrap response payload istream into a istream-timeout.

Updated copyright notices to include year 2014.

lib-http: Don't try to automatically retry requests whose payload was already lost. Patch by Stephan Bosch.

lib-http: Fixed request hang on connect() timeout.

http: Implemented delayed requests scheduling. Requests can now be (re)submitted with a delay. The request is not sent until the delay time expires. This facilitates handling the Retry-After header in responses. This can either be performed automatically if the indicated delay is not too long or explicitly by the code using lib-http.

lib-http: Use [io]_stream_get_error() instead of just errno strings.

lib-http: http-client: Added support for tunneling SSL conntections through proxy.

lib-http: Added support for creating CONNECT tunnels through HTTP.

lib-http: http-client: Added request error code for broken payload input stream. This error is triggered when reading from the provided payload input stream fails while sending the request. Previously this would yield the same error code as for a failure to write to the connection output.

lib-http: Added support for disabling automatic redirects.

lib-http: http-client: When client request is retried based on a response received from server, return full response to caller if retry is not possible. Before, it would construct a dummy-response that basically retains only the status and reason elements. This is currently only relevant for the way a 417 Expectation Failed response from the server is handled.

lib-http: http-client: Fixed segfault caused by earlier improvement of connection output locking. Segfault was triggered when an aborted request received a response.

lib-http: Implemented limits on overall HTTP header size, size of individual header fields and the number of fields in the header.

lib-http: Adjusted response and request parsers to accept a request/response object to fill with data, rather than have it return one.

lib-http: Added support for asynchronous payload for requests. This means that the payload stream passed to the request can be a non-blocking socket stream from some other connection (e.g. proxy client connection).

lib-http: http-client: Fixed request scheduling and connection management.

lib-http: http-client: Connection was using wrong request index in request_wait_list to continue sending outgoing payload.

lib-http: http-client: Fixed pipelining when payload synchronizatio (100-continue) is used. Forgot to lock the connection output, which meant that new requests would enqueued for the connection while waiting for 100 Continue. This would cause an assert failure.

lib-http: Don't crash when receiving 304 or a redirect without Location: header. Patch by Stephan Bosch

lib-http: Http requests weren't freed when connection was aborted. Patch by Stephan Bosch.

lib-http: Fix/cleanup to handling connect failures. There was some kind of double-error handling which messed up things.