mod_deflate: - fix signed/unsigned (int/size_t) comparisons, - add consume_buffer() to factorize code used multiple times, - cleanup passed brigade (don't rely on next output filters to do it).

mod_deflate: follow up to r1619444. Fix counting of inflated bytes in deflate_in_filter() when asked to flush, since we now count bytes per inflate() call everywhere, we can't count all the produced bytes there. We still need to include all the produced bytes in the brigade.

mod_deflate: follow up to r1619444. Add missing APLOGNO.

leave a hint while scrolling through inflate() calls

mod_deflate: follow up to r1619383. deflate_in_filter(): - use inflated bytes per inflate() call to compute the total output bytes, - check zlib errors before limits, - add missing check_ratio() when asked to flush. deflate_out_filter(): - check ratio after each inflate() call.

A misplaced check for inflation limits prevented limiting relatively small inputs. PR56872 Submitted By: Edward Lu Committed By: covener

mod_deflate: don't require <limits.h> by using APR_INT32_MAX instead of INT_MAX.

Add missing APLOGNO + fix a typo in a comment

*) SECURITY: CVE-2014-0118 (cve.mitre.org) mod_deflate: The DEFLATE input filter (inflates request bodies) now limits the length and compression ratio of inflated request bodies to avoid denial of sevice via highly compressed bodies. See directives DeflateInflateLimitRequestBody, DeflateInflateRatioLimit, and DeflateInflateRatioBurst. Thanks to Giancarlo Pellegrino and Davide Balzarotti for reporting the issue. Submitted By: ylavic, covener Reviewed By: jorton, covener, jim

fix c++/c99 comment

Strip useless apr_brigade_cleanup() calls.

Rollback r1591302, wrong file commited.

mod_cache: Don't add cached/revalidated entity headers to a 304 response. PR 55547. When the conditional request meets the conditions of the stale then revalidated entry, the forwarded 304 response includes the entity headers merged from the cached headers (before updating the entry). Strip them before returning a 304. Since the entity headers are stripped elsewhere, factorize the code using a new table (MOD_CACHE_ENTITY_HEADERS[]) containing these headers's names.

mod_deflate: follow up to r1587639. Don't break the looped brigade when moving the FLUSH bucket to the returned bb and continue reading.

mod_deflate: Don't return gzip-ed data when reading FLUSH bucket in INFLATE input filter, forward the FLUSH but keep reading should EOS/more follow (should not happen, but mod_deflate won't fix it).

mod_deflate: Delay INFLATE input filter's self removal until all the buffered buckets are out (including EOS). PR 46146.

mod_deflate: update empty log tags.

be explicit, for now the 0 val is the default, but it's a likely candidate for someone to flip later.

Attempt to make progress on PR39727/PR45023 blocking migration to 2.4. Provide DeflateAlterETag directive to choose between 2.2 behavior, 2.4 behavior, or dropping ETag from the compressed representation. Preserves 2.4 default which breas 304 responses for compressed content.

mod_deflate: when consuming zlibs flags, APR_INCOMPLETE implies no more bytes available.

mod_deflate: follows up r1572896. Be safe from successive or post end-of-stream flush buckets.

mod_deflate: Don't fail when asked to flush inflated data to the user-agent and that coincides with the end of stream ("Zlib error flushing inflate buffer"). PR 56196. Submitted By: [Christoph Fausak <christoph.fausak glueckkanja com>] Committed By: ylavic

Commit 6 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Ignore empty buckets and split buckets longer than INT_MAX (since zlib uses 32-bit ints only) in all filters.

Commit 5 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Handle Zlib flags in the inflate input filter as in the output filter, using consume_zlib_flags().

Commit 4 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Handle non blocking reads which would block in the inflate input filter (not an error).

Commit 3 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Handle Zlib validation bytes buffering (CRC + length) in the inflate input filter : - use validation_buffer and validation_length as state, - loop until all the bytes are received.

Commit 2 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Handle Zlib header buffering in the inflate input filter : - loop until all the header is received, - handle non blocking reads returning empty brigade, - fix a double ap_get_brigade() when an EOS brigade is encountered while reading the header, - in that case and no data was received so far, don't return an error but SUCCESS with the EOS, otherwise fail, - don't remove the Content-Length and Content-MD5 headers until some data is read. Still does not handle Zlib flags for now, next commits.

Commit 1 on 6 to fix reentrance (incomplete Zlib header or validation bytes) in mod_deflate's output and input filters. PR 46146 (patches from duplicated PR 55666) Handle Zlib header buffering in the inflate output filter : - add the new deflate_ctx_t fields needed to re-enter the Zlib header parsing, - introduce the new consume_zlib_flags() function to parse/consume the ZLib flags (will be used by the other filters too), - use it to handle incomplete header in the output filter (deflate). This alone fixes PR 55666, but the issue remains for PR 46146 (inflate/deflate input filters), hence the following patches.

mod_deflate: fix decompression of files larger than 4GB. According to RFC1952, Input SIZE (compLen) contains the size of the original input data modulo 2^32. PR: 56062 Submitted by: Lukas Bezdicka

Improve error detection when decompressing request bodies with trailing garbage: - handle case where trailing bytes are in the same bucket.

style fix (no code change)

protect 'AP_DECLARE_MODULE(deflate)' from expansion to 'AP_DECLARE_MODULE(z_deflate)' if zlib has been built with -DZ_PREFIX.

mod_deflate: Make sure we process any EOS bucket in our brigade and save going the long way round.

mod_deflate: Remove assumptions as to when an EOS bucket might arrive. Gracefully step aside if the body size is zero.

Make mod_deflate skip compression if compression is enabled at SSL level.

Add lots of unique tags to error log messages

Log reason for not deflating at level trace1

Cleanup effort in prep for GA push: Trim trailing whitespace... no func change

Add comment that the check for empty brigade at the start of the function is essential.

Fix 'Content-Encoding: gzip' missing if the first brigade passed to deflate_out_buffer contained zero data bytes but no EOS bucket. Don't compress if the added headers and checksums are larger than the data to compress (and we know the size of the data in advance).

Fix endless loop if first bucket is metadata PR: 51590 Submitted by: Torsten Foertsch <torsten foertsch gmx net>

Don't try to compress requests with a zero sized body. PR: 51350

Fix memory leak in mod_deflate.c when SetOutputFilter is enabled and DEFLATE is the first filter in the chain - this is seen with mod_dav + mod_deflate when the client can not do gzip and we constantly do the filter checks. * modules/filters/mod_deflate.c (deflate_ctx): Add a flag as to whether we are successfully initialized. (deflate_out_filter): Always create filter ctx and set filter_init flag only after we are successfully initialized.

Log messages at the various points where mod_deflate returns APR_EGENERAL. Try to use level warning if the client sent invalid data and level error if something went wrong on our side.

fix some dead assignments found by the clang analyzer

zRC is never used before its overwritten below the while loop Reported by: clang static analyzer

Stop compressing if we are doing a HEAD request and the content-length filter can't determine the content-length anyway. Submitted by: Ruediger Pluem

Use the new APLOG_USE_MODULE/AP_DECLARE_MODULE macros everywhere to take advantage of per-module loglevels

Kill off edge case in mod_deflate where data may be sent before headers are determined. PR 49369 - Matthew Steele

Correctly handle the case where apr_brigade_partition() returns APR_INCOMPLETE and bkt points to the sentinel of ctx->proc_bb and not the sentinel of bb. Submitted by: Joe Schaefer <joe_schaefer yahoo com>

In case zlib initialization fails, make sure we do not modify the Content-* headers before sending the uncompressed content down the filter chain.

core, mod_deflate, mod_sed: Reduce memory usage by reusing bucket brigades in several places

Adjust content metadata on deflate/inflate response before bailing out on a 304 response so that the metadata does not differ from 200 response.

Reimplement deflate_check_etag() so that it isn't such a pig and correctly works with weak etags. Related to PR 39727. Note that there remains an error in ap_meets_conditions because that function does not currently check for a transformed etag. I am working on that.

* Only drop the last char (the '"') and not the last one of the etag itself.

mod_deflate: Fix creation of invalid Etag headers. We now make sure that the Etag value is properly quoted when adding the gzip marker. PR 39727.

* Fix r->content_encoding for inflate_in and inflate_out filters.

* Fix r->content_encoding in deflate_out_filter if it was set before.

PR 39727: Fixup ETag handling in mod deflate (updated following extensive discussion on-list). This is not a full-and-final fix, because we don't ourselves do anything useful with these ETags. But at least we're no longer screwing up clients.

mod_deflate: Don't leave a strong ETag in place while transforming the entity. PR 39727 Comment: Another user just subscribed to this bug. We need a fix more than we need an inconclusive discussion!

mod_deflate: initialise inflate-out filter correctly when the first brigade contains no data buckets. PR 43512

Correcting defects in r563317: fix to PR 42993

Check all sources of Content-Encoding in inflate_out filter PR 42993 Reasoning: http://marc.info/?l=apache-httpd-dev&m=118643107831358&w=2

* Also unset Content-MD5 in the deflate_out_filter

* We already unset Content-Length in the inflate_out_filter. So move the comment and unsetting of Content-MD5 in the right place.

Don't try to compress/decompress where there's a Content-Range. According to RFC2616, the range would have to apply *after* applying content-encoding, so anything that's been set before running the deflate filter is going to be broken.

mod_deflate - discussion on dev@ Postpone unsetting of headers until after first get_brigade from upstream

Generalise the content encoding detection and protocol: so it aslo works in inflate out filter, as suggested by rpluem. NOTE: this fails with some generators (cgi, asis) due to a deeper bug: content-encoding is set later than mod_deflate sees it. This has always been the case, and could use a separate fix if anyone wants inflate_out other than in a proxy situation. And it's not mod_deflate's problem.

Be liberal in what we accept. Where it matters.

Be liberal and correct in what we accept

Update r560689 following rpluem's analysis thereof.

Fix protocol handling in mod_deflate input filter PR 23287

don't need the definitions for error buckets any more

* generalize from error buckets to metadata buckets * also applicable to the deflate input filter * pass along metadata buckets rather than giving up mid-stream vastly improved by: Rüdiger Plüm

with LimitRequestBody xxx and the deflate output filter configured, mod_deflate eats the 413 error bucket, a 500 error is logged, and a blank screen appears at the browser.

Housekeeping: keep track of size even in the edge-case where validation bytes trickle in over multiple buckets.

* Fix some comments. No functional change.

* Also log the presence of garbage data at the end of the stream if all validation data is available in the first round.

* Actually append new data to the validation buffer and do not overwrite old data already there by setting the correct offset for the target buffer.

* Ensure that we do not perform our final operations twice if a second EOS falls down the chain by accident. Otherwise we are likely to run in a SEGFAULT. So remove ourselves from the chain.

* Ensure that we do not try to inflate validation data or garbage data. None of this is zlib's business.

* Rework inflate out filter and give it a similar workflow as the deflate out filter. This fixes the following bugs in the inflate out filter: - Incorrect handling of flush buckets. - Excessive memory usage for large compressed content (because we now already sent parts down the chain and do not process the whole brigade first before sending something down the chain). - Handle the case correctly where the validation bytes at the end of the compressed data stream are distributed across different buckets / brigades. - Fix a memory leak due to not cleaning up the internal structures of libz in some error cases. PR: 39854

* some optimizations taken from the inflate out filter

* We can ignore Z_BUF_ERROR in flush_libz_buffer because: When we call libz_func we can assume that - avail_in is zero (due to the surrounding code that calls flush_libz_buffer) - avail_out is non zero due to the fact that we just emptied the output buffer and stored it into a brigade So the only reason for Z_BUF_ERROR is that the internal libz buffers are now empty and thus we called libz_func one time too often. This does not hurt. It simply says that we are done.

* Use a define for the number of validation bytes (CRC and length) after the compressed data

* Add parameter crc to flush_libz_buffer in order to call the libz's crc32 function on the output buffer if needed. This is actually needed by the later rework of the inflate out filter.

* Rather use a pool cleanup function than calling deflateEnd before every return from function to ensure that libz's internals get cleaned up.

* Initialize zRC to avoid a compiler warning.

* This shortcut is too short. It is not up to the filters to decide whether filters down the chain can do something useful with this empty brigade.

* In preparation of the changes of the inflate out filter: - rename flush_zlib_buffer to flush_libz_buffer (this name seems better) - add a parameter to tell flush_libz_buffer whether it should deflate or inflate as this function should be also used for the inflate out filter.

* Remove ourselves from the filter chain if we failed to init libz, as we pass data down the filter chain uncompressed afterwards.

* Fix potential memory leaks in deflate_out_filter if bailing out due to an error (either in the lower filter chain or during a libz operation). We need to call deflateEnd as it is very likely that this filter is never called again to ensures that libz's internal structures get cleaned properly.

* Fix wrong FLUSH bucket handling in deflate_out_filter: Actually the internal structures of libz never got flushed as ctx->stream.avail_in is always zero here. As the EOS and the FLUSH bucket case use the same code for flushing libz's internal buffers move this code to the new function flush_zlib_buffer.

update license header text

PR#39854 Remove bogus code that chokes on flush buckets

Update the copyright year in all .c, .h and .xml files

Correctly handle internal redirects, by testing only for r-main == NULL. ap_is_initial_req() returns false for internal redirects.

No functional Change: Removing trailing whitespace. This also means that "blank" lines consisting of just spaces or tabs are now really blank lines

- mod_deflate currently only does C-E, and not T-E. Fix the comment at the top saying it does T-E.

mod_deflate should be merging the Vary header, not Setting it, and ignoring what other modules put there. Spotted By: Rici Lake

Update copyright year to 2005 and standardize on current copyright owner line.

* modules/filters/mod_deflate.c (deflate_out_filter): Check explicitly for a 304 or 204 response rather than a brigade which begins with EOS, to correctly handle such responses when generated by a CGI script. Don't alter the headers for this case either.

* modules/filters/mod_deflate.c: Remove use of zutil.h (not installed by zlib 1.2.1) and OS_CODE. (deflate_out_filter): Simplify to use an immortal brigade for the gzip header. PR: 28673

Use the ap_is_initial_req() function to test for a subrequest instead of using r->main.

Tidy up handling of empty brigades (Joe Orton); Ensure we still set headers on HEAD and 204/304; and fix inflate_out_filter handling of HEAD/etc.

Tidy up: get rid of dubious cast (nd) and initialise encoding.

Fix previous patch to deal correctly with multiple empty brigades before we know if there's any content, and not re-process the headers.

Fix headers in deflate_out_filter for the case of an empty first brigade

style

that's not really an error condition, so make the message APLOG_DEBUG.

Update inflate_out_filter to support gzip compression flags.

Don't deflate responses with zero length e.g. proxied 304's

* modules/filters/mod_deflate.c (deflate_out_filter): Destroy buckets immediately after are used so that memory consumption is not proportional to the size of the response. PR: 29318

Fix bug in mod_deflate that unconditionally sent deflate'd output even when Accept-Encoding is not present. Reported by Roy Fielding for http://cvs.apache.org/viewcvs/ with Safari.

content-encoding needs to be unset. Submitted by: Nick Kew <nick webthing.com>

remove unused variables

mod_deflate: - New option for DEFLATE output file (force-gzip), which skips checking the accept-encoding header. - New output filter 'INFLATE' for uncompressing responses. Submitted by: Nick Kew <Nick at WebThing dot com> Reviewed by: Ian Holsman

fix name of The Apache Software Foundation

fix copyright dates according to the first checkin

apply Apache License, Version 2.0

remove email address in the hopes ppl stop emailing me directly ;(

update license to 2004.

get rid of _FOREACH

here applies the same. Don't skip the \0 delimiter when searching for already applied encodings. Additionally don't compress if *any* non-identity encoding was applied before. (deflate, pkzip, whatever).

Doh. If Accept-Encoding contains no gzip token, we skip the 0 delimiter and search for "gzip" somehwere in the memory. This was originally discovered by Joe Orton. But there's more. We must skip any parameters, since these do not contain what we're looking for. PR: 21523

Fix bug in mod_deflate, which was passing calling deflate() without checkinig first whether it has something to deflate. (currently this causes deflate to generate a fatal error according to the zlib spec). PR 22259. PR: Obtained from: Submitted by: Reviewed by:

Don't attempt to hold all of the response until we're done. We'll pass data on when the zlib buffer becomes full and we need to reset the buffer anyway. Also, tidy up a similar semantic when we see the EOS by returning an error if ap_pass_brigade gets an error (we'd lose it otherwise).

Check also for r->content_encoding (now we should have them all...) for already compressed content in mod_deflate. PR: 19913 Submitted by: Tsuyoshi SASAMOTO <nazonazo@super.win.ne.jp>

reflect backport of mod_deflate.c r1.31 change directive from compressionlevel to deflatecompressionlevel PR: Obtained from: Submitted by: Ian Holsman, Andre Malo Reviewed by:

Check also err_headers_out for an already set Content-Encoding: gzip header. This prevents gzip compressed content from a CGI script from being compressed once more. PR: 17797

new directive 'compressionlevel' Default compression level now changed to Zlib's default (was 'best_speed') Obtained from: Stephen Pierzchala <stephen@pierzchala.com> Michael Schroepl <Michael.Schroepl@telekurs.de> Reviewed by: Ian Holsman

Fix potential memory leaks in mod_deflate on malformed input data. PR: 16046

finished that boring job: update license to 2003. Happy New Year! ;-))

break down macro code and avoid mixed case. Thanks to Justin for his hints.

The patch allows the user to log the accurate filter input and output byte count, instead of only the rounded compression ratio. The DeflateFilterNote directive will be extended as follows: DeflateFilterNote [type] name type can be one of "input", "output" or "ratio". "ratio" is assumed if the type is omitted (backwards compatible).

Fix a memory leak in mod_deflate with dynamic content. Because the brigade was being destroyed (ie, cleanup unregistered) rather than just emptied out and then reused anyway, the last call down the filter stack would leak buckets. PR: 14321 Submitted by: Ken Franken <kfranken@decisionmark.com> Reviewed by: Jeff Trawick, Cliff Woolley

Always emit Vary header if mod_deflate is involved in the request. Submitted by: Andr��Malo <nd@perlig.de> Reviewed by: Justin Erenkrantz

actuall fix the bug

deflate's filter removal was reversed for non-html files, resulting in it never being applied for non-html files Submitted by: Kris Verbeeck <Kris.Verbeeck@ubizen.com>

mod-deflate now checks to make sure gzip-only-text-html is set to 1, not anything. This will allow things like... SetEnv gzip-only-text/html 1 BrowserMatch "GECKO" gzip-only-text/html=0 Obtained from: Andr� Schild <A.Schild@aarboard.ch> Reviewed by: Ian Holsman

Add a filter_init function to the filters so that a filter can execute arbitrary code before the handlers are invoked. This resolves an issue with incorrect 304s on If-Modified-Since mod_include requests since ap_meets_conditions() is not aware that this is a dynamic request and it is not possible to satisfy 304 for these requests (unless xbithack full is on, of course). When mod_include runs as a filter, it is too late to set any flag since the handler is responsible for calling ap_meets_conditions(), which it should do before generating any data. If a module doesn't need to run such arbitrary code, it can just pass NULL as the argument and all is well. PR: 9673 Reviewed by: Ryan Bloom and others

Clean up a signedness emit

Modify the deflate input filter so that it should always return data on a blocking read.

An important observations before some x86'er attacks this idea.

get the prototype for strcasecmp() on AIX

Oops. When deflating: only reset ctx->stream buffers when the ctx was NULL. When inflating: remember to clear the ctx->bb after we're done with it.

Why the stack buffer and the copy? This seems much more straightforward.

keep the signed-ness of char buffers consistent with what is expected by functions they are passed to (putLong() this time)

Unroll the putLong and getLong loops and correct putLong's input param type to be unsigned char* instead of char*. Submitted by: Sander Striker Reviewed by: Justin Erenkrantz

Make sure that z_RC is not Z_STREAM_END in case we get a 0-length bucket and we don't enter the while loop. Submitted by: Jeff Trawick's horde of picky compilers

get mod_deflate to compile with compilers that care about the signed-ness of char * it looks like it should work fine, but it is untested

Input filter support for mod_deflate. Highly experimental, but known to work for some tests.

Add a note indicating what it would take to get MTIME field populated. (I did this locally, but didn't figure it was worth committing, but it's still worth noting.)

Implement flushing support for mod_deflate. (Review of RFC 1952 indicates that this is indeed easily allowable.)

Allow mod_deflate to be more tolerant of multiple content-encodings. - Tighten scope of a few affected variables (accepts, token). - Look for gzip in r->headers_out (!) - if there, we've already compressed. (Previously, we'd look at r->headers_in which was bogus - we're output.) - Switch the Accept-Encoding check to be case-insensitive per RFC 2616. - Remove commented-out code. - If C-E wasn't present or it was just "identity," set C-E to "gzip", otherwise, merge "gzip" onto the C-E list. This patch was originally submitted by Ian and cleaned up and tested by Justin. Submitted by: Ian Holsman Reviewed by: Justin Erenkrantz

content with "Content-Encoding" header, content is encoded. But mod_deflate does not check it. It cause to encode content twice. This problem is reproducable by getting encoded content via mod_proxy. Patch Contributed by kaz@asada.sytes.net (ASADA Kazuhisa) Bug #9222 Thanks Kazuhisa

stop using APLOG_NOERRNO in calls to ap_log_[pr]error()

Fix mod_deflate corruption when using multiple input buckets. We should only clear *_out when we are the beginning or avail_out == 0. Justin was tracking this down before he went to WWDC and didn't finish before he left. A (virtual) beer goes to Asada who filed this PR with the right patch! PR: 9014 Submitted by: kaz@asada.sytes.net (ASADA Kazuhisa) Reviewed by: Justin Erenkrantz

Add DeflateBufferSize directive to replace hardcoded FILTER_BUFSIZE definition. Also speeds up ctx init since it takes the buffer and does a malloc rather than a calloc on it.

If we determine that we shouldn't be involved in this request, remove ourselves from the filter chain (so we don't duplicate this path each time a brigade comes through).

Move mod_deflate from experimental to filters to represent our belief in its stability.