lib-http: client: Fix request statistics text to properly report send attempts. If the request was first sent in the same ioloop cycle in which the text is generated, the text would claim it was not sent at all yet. With this commit the text now explicitly makes the distinction between request attempts and actual send attempts. The number of attempts is increased at each retry, while the send attempts are increased each time the request is actually being sent to a server.

lib-http: client: Add event fields for the number of bytes sent and received for each request in the last attempt.

lib-http: client: Trigger special events when a request is finished, retried or redirected.

lib-http: client: Record last status code in the request object.

lib-http: client: Reset redirect counter when the request is retried. The absence of this reset caused problems when there was a redirect limit and the request was retried, in which case the limit is reached too soon.

lib-http: client: Add "host" and "port" fields to request event.

lib-http: client: Use the new http_url_get_port*() functions.

lib-http: client: Simplified return code handling in http_client_request_send_real().

lib-http: client: Make sure output is used only when valid in http_client_request_send_real().

lib-http: client: Perform output stream error handling in one place.

lib-http: client: Flush and check output for errors after uncorking. Otherwise, a disconnection event may be missed.

global: start relying on str_free(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - str_free(&E); - } + str_free(&E);

lib-http: client: Make sure all ioloop objects are created on the ioloop that the client/context is switched to.

lib-http: client: Assert that req->client != NULL in http_client_request_send_error(). Applies when blocking payload output API is being used. Addresses a report by scan-build.

lib-http: client: Set req->client to NULL once the request is destroyed. The http_client_request_destroy() function does not free the request immediately, as long as it is still referenced. It can still be referenced by a connection that has sent it and is waiting for a reply (payload). In the mean time the actual client can be gone, so we want to make sure nothing is pointing to that anymore. This change adds a few extra assertions to make sure nothing tries to use a NULL client later on. Some direct references to req->client are replaced with a local client variable if there is one.

Updated copyright notices to include the year 2018.

lib-http: client: Check whether client is running in a wait loop with the correct condition. This used to check client->ioloop != NULL, but now it should check client->waiting. Previous commit (e8a1b62fe4a81b211dcccd1a58b44f254074eab6) omitted a few instances where this should have been changed.

lib-http: client: Made http_client record the current ioloop it is switched to. This prevents http_client_wait() from switching the client to an ioloop it was never explicitly switched to.

lib-http: Add named event: http_request_finished

lib-http: client: Replace http_*_debug() with e_debug()

lib-http: client: Remove redundant debug logging checks

lib-http: client: Add support for event API

lib-http: client: Added identifier to client log messages.

lib-http: client: shared context: Allow sharing peer state between clients.

lib-http: client: Send empty payload (Content-Length: 0) for requests that normally expect a payload. This includes the standard POST and PUT methods. Others need to use the new http_client_request_set_payload_empty() function to force sending an empty payload.

global: start relying on [io]_stream_unref(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - i_stream_unref(&E); - } + i_stream_unref(&E); @@ expression E; @@ - if (E != NULL) { - o_stream_unref(&E); - } + o_stream_unref(&E);

global: start relying on io_remove{,_closed}(NULL) being a no-op Cleanup performed with the following semantic patch: @@ expression E; @@ - if (E != NULL) { - io_remove(&E); - } + io_remove(&E); @@ expression E; @@ - if (E != NULL) { - io_remove_closed(&E); - } + io_remove_closed(&E);

lib-http: client: Explicitly destroy the response payload timeout stream when the request is destroyed while receiving payload. This way, the timeout is stopped. This caused ioloop panics.

lib-http: Improve request stats text. It's important to know how long the request was in queue before it was sent. Also the "n attempts in m secs" makes more sense if it was counting only the time after the initial request was sent, not including the queuing time. If there is more than 1 attempt, log separately how long all the attempts were waited on vs. how long the last attempt took.

lib-http: Add http_client_request_stats.first_sent_msecs Also rename sent_msecs to last_sent_msecs.

lib-http: Track request's first and last send attempt times separately.

lib-http: client: Implemented means to obtain request statistics.

lib-http: http_client_request_delay_from_response() should use per-request timeout as max It was using the global request_timeout_msecs for the maximum timeout, which could be different from the per-request timeout set by http_client_request_set_attempt_timeout_msecs().

lib-http: client: Fix aborting a request that has a delayed error. Caused by recent changes in delayed error handling.

lib-http: client: Implemented http_client_request_url_str() function that accepts an URL string rather than a pre-parsed URL object. If the provided HTTP URL is invalid, the callback with the error is called some time later from the ioloop. This change also amends the test-http-client-errors test suite with a new test for this new feature.

lib-http: client: Make sure a request's log label never gets a duplicate request target part. Make sure the URL part of the label always only consists of a http://host:port part and nothing more.

lib-http: client: Restructured the handling of delayed errors. The req->delayed_error* fields are cleared before the callback is called and before the request is dereferenced. This way, freeing the request never removes it from the delayed request list when the list is being processed. It also makes it possible to resubmit the request from the callback (mainly for future use).

lib-http: client: Clear request pointer immediately in http_client_request_error().

lib-http: http_client_request_add_header() - Add key/value asserts These don't check that they're entirely correct as required by HTTP specifications. They're mainly there as a quick check that if the caller didn't validate the key/value in any way, we'll crash instead of creating a potential security hole. (Because with line feeds the attacker could add extra headers or even entirely new HTTP requests.)

lib-http: Improve http_client_request_set_destroy_callback It now allows using non-void* context

lib-http: client: Fixed handling of errors occurring for unsubmitted requests during http_client_request_send_payload(). When http_client_request_send_payload() is executed for the first time, the request is submitted. Errors occurring during submission don't trigger a callback immediately. Instead, these are queued in the client and will trigger a callback when an ioloop is run with the client. However, in http_client_request_send_payload() the ioloop is never executed when the request fails that way, meaning that the callback was never called. Since for example SOLR assumes the callback is always called for an error in http_client_request_send_payload(), this causes all kinds of problems. Fixed by manually handling the delayed request errors in http_client_request_send_payload() explicitly.

lib-http: client: Make sure req->conn is only not NULL when that connection holds a reference to that request. This consolidates the management of req->conn to one place, thereby preventing mishaps. It makes sure req->conn is always properly assigned, making it more reliable. This fixes a problem that emerged in the http-proxy.

Updated copyright notices to include the year 2017.

global: Replaced all instances of memset(p, 0, sizeof(*p)) with the new i_zero() macro. Used the following script: C_FILES=`git ls-files *.c` H_FILES=`git ls-files *.h` for F in "$C_FILES $H_FILES"; do echo "$F" perl -p -i -e 's/safe_memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero_safe(&$1)/g' $F perl -p -i -e 's/safe_memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero_safe($1)/g' $F perl -p -i -e 's/memset\(&\(?([^,]*)\)?,\s*0,\s*sizeof\(\g1\)\)/i_zero(&$1)/g' $F perl -p -i -e 's/memset\(([^,]*),\s*0,\s*sizeof\(\*\g1\)\)/i_zero($1)/g' $F done

global: Change string position/length from unsigned int to size_t Mainly to avoid truncating >4GB strings, which might potentially cause some security holes. Normally there are other limits, which prevent such excessive strings from being created in the first place. I'm sure this didn't find everything. Maybe everything could be found with compiler warnings. -Wconversion kind of does it, but it gives way too many unnecessary warnings. These were mainly found with: grep " = strlen" egrep "unsigned int.*(size|len)"

lib-http: Use io_wait_timer for tracking wait times. The previous code was completely incorrect, since it wasn't even tracking the HTTP request's start usecs.

lib-http: client: Fixed assert failure occurring when server returns error status early while client is still sending blocking payload.

lib-http: Add ioloop and lock wait information to timeout messages. It'll now log for example: 9007 Request timed out (Request sent 7.087 secs ago, 0.076 in other ioloops, 7.012 in locks, connected 7.087 secs ago) Which points out that the problem wasn't really with the HTTP, but with locking. This likely should be fixed in some way also in lib-http so that it gives a bit of extra time for reading the request, but that's a separate fix.

lib-http: Support per-request timeout and max_attempts

lib-http: Clarify response message when retries have happened. "1 attempts" sounds like 1 attempt in total, while this was actually the second attempt.

lib-http: client: Allow retrying requests that failed internally.

o_stream_send_istream() API changed again The new API enforces the caller to correctly handle all the possible situations. It also makes in unambiguous whether to wait for input or output stream.

lib-http: Default port for http should be 80, not 443.

lib-http: client: Improved labeling of debug messages. Request label is corrected. Labels are now pre-composed and stored.

lib-http: client: Prevent useless and unexpected request callbacks during http_client_deinit(). Requests are now destroyed before queues, hosts, peers and connections. As a side-effect, requests are now removed from the client request list at http_client_request_destroy(), so that requests with lingering references will no longer make http_client_wait() hang.

lib-http: client: Removed curiously duplicated code. Probably a patch got applied with sufficient fuzz that it duplicated this code fragment, while it actually already existed.

lib-http: client: Improved request reference counting in connection code. It should now always be clear when the connection object holds a reference to a request and when it is released. Only while the reference is held, req->conn points to a connection. This also makes the assertion in http_client_request_unref() more robust and clear.

lib-http: client: Fixed reference counting for requests that are aborted due to having a broken outgoing payload stream.

lib-http: Changed http_client_request_error to set request to NULL It's going to internally unreference it, so the caller should be aware of it also. I also changed request state check to be an assert, since I don't think there's any safe way this could work otherwise.

lib-http: Fixed checking if istream or ostream needs waiting.

global: Simplify code with the new o_stream_send_istream() API

global: Code cleanup - remove unnecessary errno changes. These were left behind from changing %m to [io]_stream_get_error()

lib-http: Started using struct uri_host in struct http_url.

uri-util: Removed have_port and have_host_ip flags. Presence of port can be tested with port != 0. Presence of IP can be tested with ip->family != 0.

lib-http: client: Added ability to configure a specific proxy for individual requests. This way, a request can be routed to a specific proxy (or origin server). The destination can also be a unix socket.

lib-http: Include information about number of request attempts and its timing in response reason. Because the reason is usually logged as part of the error string, this causes all of the error messages to include the attempts count and how long the requests took in total. This should make it easier to understand problems in error logs. http_client_request_set_preserve_exact_reason() can be used to disable modifying the reason string. This may also apply to other reason modifications that may be done in the future.

lib-http: client: Added pure attribute to some of the request property functions.

lib-http: client: Implemented means to set request payload buffer rather than an input stream. This is not purely a convenience function: there have been bugs caused by allocating a data input stream from a datastack buffer. With this function, the buffer is copied to the request pool, so that it is durably allocated while the request exists. This prevents futher mishaps. The server already has an equivalent function for its response object.

lib-http: client: Fixed request timeout handling during pipelining. The timeout was not managed correctly. If an earlier request finished, it would not restart the timeout for the next pending request. Also, filling the pipeline caused the timout to be reset inappropriately, postponing its expiry.

lib-http: Clarify http_client_request_*_payload() API and minor change to it The earlier behavior was pretty confusing, and potentially could have caused double-freeing memory in some situations. Now it's clear that req is set to NULL always when the request is finished, regardless of whether it has any references left. Changed http_client_request_finish_payload() to return 0 on success instead of 1. This could have been left alone, but it's unlikely that there is any code outside Dovecot core that calls it and this way is cleaner.

lib-http: http_client_request_unref() now always sets *req=NULL This makes its behavior consistent with other APIs in Dovecot. Also http_client_request_finish() no longer sets req=NULL, because all of its callers already keep a reference. Instead added an assert to make sure the reference is there.

lib-http: client: Fixed pipelining bug: client sometimes sent new request while still waiting for 100-continue.

lib-http: client: Make sure that any pending request is aborted and destroyed before connection FDs are closed. This way, any payload io struct created from the response callback can be freed before the associated FD becomes invalid. This would cause an assert failure otherwise.

lib-http: Assign an ID for each HTTP request and log it in debug lines. The ID stays the same when request is retried. Added a "Req" prefix so it's easier to search for the IDs. Based on patch by Stephan Bosch.

global: freshen copyright git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-5]/$1-2016/g;s/ (201[0-5]) Dovecot/ $1-2016 Dovecot/'

lib-http: client: Fixed handling of response timout if there is no payload going to the server.

lib-http: Added asserts to all refcount increments. These could catch situations where an already freed object's refcount is attempted to be increased.

lib-http: client: Fixed bug that caused queues to be duplicated over time when host has multiple IPs. The peer address comparison did not account for the fact that the initial queue address has no IP assigned.

lib-http: Optimized http_client_request_send_payload() fix Badly copy&pasted my original fix, instead of noticing that Stephan's fix was slightly different. Probably shouldn't have made actual difference other than for performance.

lib-http: Avoid crashes on failing http_client_request_send_payload() calls It HTTP server connection died, ioloop might not have anything to do anymore: Panic: file ioloop-epoll.c: line 187 (io_loop_handler_run_internal): assertion failed: (msecs >= 0)

http-client: Added support for using an HTTP proxy running on a unix socket.

lib-http: client: Fixed handling of request timeout. It was inappropriately active when the client needed to take action. This showed particularly with large payloads sent using http_client_request_send_payload().

lib-http client: Implemented proxy authentication using Basic scheme.

lib-http client: Added inline function to check whether request is directed at a proxy.

lib-http client: Implemented simple authentication using Basic scheme.

lib-http client: Removed useless assignment of request->authority in http_client_request_redirect(). It is assigned again in http_client_request_do_submit().

lib-http: http_client_request_send() failure returned already-freed error string.

lib-http: client: Fixed double unref of request when error occurs during http_client_request_try_retry(). Much like when the request is first submitted, any errors that occur while attempting a retry from within the callback are now delayed in a zero timer.

global: freshen copyright Robomatically: git ls-files | xargs perl -p -i -e 's/(\d+)-201[0-4]/$1-2015/g;s/ (201[0-4]) Dovecot/ $1-2015 Dovecot/' Happy 2015 everyone! Signed-off-by: Phil Carmody <phil@dovecot.fi>

lib-http: client: Fixed conflict between request abort and request delayed error.

lib-http: client: Created list of all submitted requests. Currently only needed to improve debugging of hanging requests.

lib-http: client: Moved delayed failed requests from host to client object.

lib-http: client: Fixed assert crash occurring when DNS lookup fails immediately during request submission. In that situation, the request was not dropped from the queue immediately, triggering the assert crash.

lib-http: Fixed detecting disconnection when ioloop is running only intermittently. This fix only applies to ioloops created and run by lib-http itself.

lib-http: client: Added support for absolute request timeout. Requests cannot survive beyond this deadline.

lib-http: client: Added support for delaying requests in milliseconds.

lib-http: client: Fixed aborting request in the middle of sending payload. If the request payload is so big that it cannot be sent all at once, the caller may at some point abort the request when it is still being sent. The bug occurred when the request finally finished sending. It erroneously advanced the state to WAITING rather than remaining ABORTED, thus 'reviving' the request unexpectedly.

lib-http: client: Fixed crash occurring sometimes when delay timeout expired. Forgot to drop destroyed requests from delay queue.

lib-http: Added http_client_request_get_method()

lib-http: Debug log now includes where HTTP requests spent their time on.

lib-http: Updated comments to RFC7230/RFC7231.

lib-http: Added http_client_request_get_target() This can be useful in error message logging without having to duplicate the string.

lib-http: Fixed assertion failure in http_client_request_send_payload() caused by inappropriate retry attempt.

lib-http: http-client: Prevented http_client_request_continue_payload() from segfaulting when the callback sets the request pointer to NULL.

lib-http: http-client: Forgot to switch dns-client io_loop in http_client_request_continue_payload() in ealier change (6c3bd941e153).

lib-http: Better error handling fix for payload read/write failures

lib-http: Failed payload read/write error was handled wrongly as "stream input size changed".

lib-http: Added more information to "stream input size changed" error

lib-http: http_client_request_finish_payload() no longer waits for the entire result payload. The caller can now read it instead, or just discard it by not reading it.

Use io_add_istream() wherever possible. This shouldn't fix anything, but might make some functionality easier to implement in future.

Updated copyright notices to include year 2014.

lib-http: Added http_client_request_remove_header()

lib-http: Allow calling http_client_request_add_header() when retrying requests.

lib-http: Don't try to automatically retry requests whose payload was already lost. Patch by Stephan Bosch.

lib-http: Compiler warning fix

http: Implemented delayed requests scheduling. Requests can now be (re)submitted with a delay. The request is not sent until the delay time expires. This facilitates handling the Retry-After header in responses. This can either be performed automatically if the indicated delay is not too long or explicitly by the code using lib-http.

http-client: Improved handing of delayed request errors. Originally each request would have its own zero timeout, but now only one timeout is put in the host object.

http-client: Changed struct http_client_host_port into a struct http_client_queue object. Peer and request objects now reference the queue object directly rather than the host object. This way, there is no need to find the matching host:port in the host anymore. This makes the queueing structure more intuitive and more efficient. This is a first step towards support for connecting to HTTP services through unix sockets or directing requests at specific hosts (so not from the URL). This patch also fixes a potential timeout leak (to_connect) in http_client_host_port (now http_client_queue) and makes sure it is moved during switch_ioloop(). Finally it updates the structure comment at the top of http-client.c.

Use io_loop_set_current() instead of directly setting it.

lib-http: Use [io]_stream_get_error() instead of just errno strings.

lib-http: http-client: Added support for tunneling SSL conntections through proxy.

lib-http: Added support for creating CONNECT tunnels through HTTP.