fin_wait_2.html revision 2eaf662cbc81e823e8d9aeb8d54e69e63032493e
e59faf65ce864fe95dc00f5d52b8323cdbd0608aTimo Sirainen<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <meta name="generator" content="HTML Tidy, see www.w3.org" />
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <title>Connections in FIN_WAIT_2 and Apache</title>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <link rev="made" href="mailto:marc@apache.org" />
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <!-- Background white, links blue (unvisited), navy (visited), red (active) -->
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <body bgcolor="#FFFFFF" text="#000000" link="#0000FF"
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <!--#include virtual="header.html" -->
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <h1 align="CENTER">Connections in the FIN_WAIT_2 state and
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen Starting with the Apache 1.2 betas, people are reporting
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen many more connections in the FIN_WAIT_2 state (as reported
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen by <code>netstat</code>) than they saw using older
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen versions. When the server closes a TCP connection, it sends
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen a packet with the FIN bit sent to the client, which then
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen responds with a packet with the ACK bit set. The client
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen then sends a packet with the FIN bit set to the server,
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen which responds with an ACK and the connection is closed.
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen The state that the connection is in during the period
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen between when the server gets the ACK from the client and
f5a7396b31762a1f876517e13ce9065820139f7cTimo Sirainen the server gets the FIN from the client is known as
f5a7396b31762a1f876517e13ce9065820139f7cTimo Sirainen FIN_WAIT_2. See the <a
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen href="ftp://ds.internic.net/rfc/rfc793.txt">TCP RFC</a> for
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen the technical details of the state transitions.
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen <p>The FIN_WAIT_2 state is somewhat unusual in that there
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen is no timeout defined in the standard for it. This means
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen that on many operating systems, a connection in the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen FIN_WAIT_2 state will stay around until the system is
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen rebooted. If the system does not have a timeout and too
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen many FIN_WAIT_2 connections build up, it can fill up the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen space allocated for storing information about the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen connections and crash the kernel. The connections in
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen FIN_WAIT_2 do not tie up an httpd process.</p>
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen There are numerous reasons for it happening, some of them
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen may not yet be fully clear. What is known follows.
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen <h3>Buggy clients and persistent connections</h3>
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen Several clients have a bug which pops up when dealing with
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen <a href="/keepalive.html">persistent connections</a> (aka
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen keepalives). When the connection is idle and the server
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen closes the connection (based on the <a
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen href="/mod/core.html#keepalivetimeout">KeepAliveTimeout</a>),
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen the client is programmed so that the client does not send
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen back a FIN and ACK to the server. This means that the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen connection stays in the FIN_WAIT_2 state until one of the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen following happens:
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen <li>The client opens a new connection to the same or a
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen different site, which causes it to fully close the older
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen connection on that socket.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li>The user exits the client, which on some (most?)
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen clients causes the OS to fully shutdown the
cd8507179823de33d6e8242e10dbc15d136245b5Timo Sirainen connection.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li>The FIN_WAIT_2 times out, on servers that have a
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen timeout for this state.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>If you are lucky, this means that the buggy client will
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen fully close the connection and release the resources on
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen your server. However, there are some cases where the socket
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen is never fully closed, such as a dialup client
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen disconnecting from their provider before closing the
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen client. In addition, a client might sit idle for days
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen without making another connection, and thus may hold its
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen end of the socket open for days even though it has no
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen further use for it. <strong>This is a bug in the browser or
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen in its operating system's TCP implementation.</strong></p>
2615df45a8027948a474abe5e817b34b0499c171Timo Sirainen <p>The clients on which this problem has been verified to
4ece61edd7c266a4b8f3b290a7f0a3cb3d13ca0fTimo Sirainen <li>Mozilla/3.01 (X11; I; FreeBSD 2.1.5-RELEASE
4ece61edd7c266a4b8f3b290a7f0a3cb3d13ca0fTimo Sirainen <li>Mozilla/2.02 (X11; I; FreeBSD 2.1.5-RELEASE
4ece61edd7c266a4b8f3b290a7f0a3cb3d13ca0fTimo Sirainen <li>Mozilla/3.01Gold (X11; I; SunOS 5.5 sun4m)</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>This does not appear to be a problem on:</p>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>It is expected that many other clients have the same
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen problem. What a client <strong>should do</strong> is
4ece61edd7c266a4b8f3b290a7f0a3cb3d13ca0fTimo Sirainen periodically check its open socket(s) to see if they have
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen been closed by the server, and close their side of the
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen connection if the server has closed. This check need only
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen occur once every few seconds, and may even be detected by a
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen OS signal on some systems (<em>e.g.</em>, Win95 and NT
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen clients have this capability, but they seem to be ignoring
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen <p>Apache <strong>cannot</strong> avoid these FIN_WAIT_2
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen states unless it disables persistent connections for the
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen buggy clients, just like we recommend doing for Navigator
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen 2.x clients due to other bugs. However, non-persistent
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen connections increase the total number of connections needed
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen per client and slow retrieval of an image-laden web page.
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen Since non-persistent connections have their own resource
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen consumptions and a short waiting period after each closure,
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen a busy server may need persistence in order to best serve
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen its clients.</p>
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen <p>As far as we know, the client-caused FIN_WAIT_2 problem
c93ff0433cc3d348116f75a64f9988fedb86fd18Timo Sirainen is present for all servers that support persistent
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen connections, including Apache 1.1.x and 1.2.</p>
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen <h3>A necessary bit of code introduced in 1.2</h3>
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen While the above bug is a problem, it is not the whole
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen problem. Some users have observed no FIN_WAIT_2 problems
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen with Apache 1.1.x, but with 1.2b enough connections build
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen up in the FIN_WAIT_2 state to crash their server. The most
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen likely source for additional FIN_WAIT_2 states is a
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen function called <code>lingering_close()</code> which was
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen added between 1.1 and 1.2. This function is necessary for
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen the proper handling of persistent connections and any
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen request which includes content in the message body
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen (<em>e.g.</em>, PUTs and POSTs). What it does is read any
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen data sent by the client for a certain time after the server
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen closes the connection. The exact reasons for doing this are
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen somewhat complicated, but involve what happens if the
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen client is making a request at the same time the server
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen sends a response and closes the connection. Without
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen lingering, the client might be forced to reset its TCP
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen input buffer before it has a chance to read the server's
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen response, and thus understand why the connection has
755aea84bbe2b15ed7fe991f6462a93333ff571fTimo Sirainen closed. See the <a href="#appendix">appendix</a> for more
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen <p>The code in <code>lingering_close()</code> appears to
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen cause problems for a number of factors, including the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen change in traffic patterns that it causes. The code has
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen been thoroughly reviewed and we are not aware of any bugs
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen in it. It is possible that there is some problem in the BSD
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen TCP stack, aside from the lack of a timeout for the
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen FIN_WAIT_2 state, exposed by the
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <code>lingering_close</code> code that causes the observed
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen problems.</p>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen What can I do about it? There are several possible
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen workarounds to the problem, some of which work better than
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen The obvious workaround is to simply have a timeout for the
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen FIN_WAIT_2 state. This is not specified by the RFC, and
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen could be claimed to be a violation of the RFC, but it is
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen widely recognized as being necessary. The following systems
e5fd6dfd0a492e4708d4dbb7971d7fc5d7b8fd85Timo Sirainen are known to have a timeout:
132487b9a47c2eb6fc80cfa2b0aaf82c6dc3af56Timo Sirainen <li><a href="http://www.freebsd.org/">FreeBSD</a>
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen versions starting at 2.0 or possibly earlier.</li>
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <li><a href="http://www.netbsd.org/">NetBSD</a> version
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <li><a href="http://www.openbsd.org/">OpenBSD</a> all
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen versions(?)</li>
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <li><a href="http://www.bsdi.com/">BSD/OS</a> 2.1, with
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen href="ftp://ftp.bsdi.com/bsdi/patches/patches-2.1/K210-027">
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li><a href="http://www.sun.com/">Solaris</a> as of
7f3b826a89bcb7a72759912e99f574b28309fe1bTimo Sirainen around version 2.2. The timeout can be tuned by using
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <code>tcp_fin_wait_2_flush_interval</code>, but the
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen default should be appropriate for most servers and
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen improper tuning can have negative impacts.</li>
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <li><a href="http://www.linux.org/">Linux</a> 2.0.x and
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen earlier(?)</li>
6b399f555c9c5c722d4cd5eab8faa02b2a4731d3Timo Sirainen <li><a href="http://www.hp.com/">HP-UX</a> 10.x defaults
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen to terminating connections in the FIN_WAIT_2 state after
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen the normal keepalive timeouts. This does not refer to the
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen persistent connection or HTTP keepalive timeouts, but the
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <code>SO_LINGER</code> socket option which is enabled by
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen Apache. This parameter can be adjusted by using
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <code>nettune</code> to modify parameters such as
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <code>tcp_keepstart</code> and <code>tcp_keepstop</code>.
e10d8b1291090c26b9ef499637e6e632485ca5beTimo Sirainen In later revisions, there is an explicit timer for
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen connections in FIN_WAIT_2 that can be modified; contact
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen HP support for details.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li><a href="http://www.sgi.com/">SGI IRIX</a> can be
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen patched to support a timeout. For IRIX 5.3, 6.2, and 6.3,
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen use patches 1654, 1703 and 1778 respectively. If you have
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen trouble locating these patches, please contact your SGI
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen support channel for help.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li><a href="http://www.ncr.com/">NCR's MP RAS Unix</a>
18ccd19c244f49665fe03cda785efa066d2c38dfTimo Sirainen 2.xx and 3.xx both have FIN_WAIT_2 timeouts. In 2.xx it
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen is non-tunable at 600 seconds, while in 3.xx it defaults
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen to 600 seconds and is calculated based on the tunable
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen "max keep alive probes" (default of 8) multiplied by the
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen "keep alive interval" (default 75 seconds).</li>
d22301419109ed4a38351715e6760011421dadecTimo Sirainen <li><a href="http://www.sequent.com">Sequent's ptx/TCP/IP
d22301419109ed4a38351715e6760011421dadecTimo Sirainen for DYNIX/ptx</a> has had a FIN_WAIT_2 timeout since
d22301419109ed4a38351715e6760011421dadecTimo Sirainen around release 4.1 in mid-1994.</li>
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen <p>The following systems are known to not have a
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li><a href="http://www.sun.com/">SunOS 4.x</a> does not
ecdce39e5ef4b62eefa9f5818f17d153fd5d710aTimo Sirainen and almost certainly never will have one because it as at
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen the very end of its development cycle for Sun. If you
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen have kernel source should be easy to patch.</li>
d22301419109ed4a38351715e6760011421dadecTimo Sirainen href="http://www.apache.org/dist/httpd/contrib/patches/1.2/fin_wait_2.patch">
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen patch available</a> for adding a timeout to the FIN_WAIT_2
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen state; it was originally intended for BSD/OS, but should be
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen adaptable to most systems using BSD networking code. You
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen need kernel source code to be able to use it. If you do
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen adapt it to work for any other systems, please drop me a
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen href="mailto:marc@apache.org">marc@apache.org</a>.</p>
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <h3>Compile without using
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen It is possible to compile Apache 1.2 without using the
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <code>lingering_close()</code> function. This will result
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen in that section of code being similar to that which was in
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen 1.1. If you do this, be aware that it can cause problems
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen with PUTs, POSTs and persistent connections, especially if
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen the client uses pipelining. That said, it is no worse than
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen on 1.1, and we understand that keeping your server running
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen is quite important.
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <p>To compile without the <code>lingering_close()</code>
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen function, add <code>-DNO_LINGCLOSE</code> to the end of the
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <code>Configure</code> and rebuild the server.</p>
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen <h3>Use <code>SO_LINGER</code> as an alternative to
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen On most systems, there is an option called
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <code>setsockopt(2)</code>. It does something very similar
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen to <code>lingering_close()</code>, except that it is broken
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen on many systems so that it causes far more problems than
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen <code>lingering_close</code>. On some systems, it could
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen possibly work better so it may be worth a try if you have
1d2b188f0eedc3cab6e27ceac5425a037f38042eTimo Sirainen no other alternatives.
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen -DNO_LINGCLOSE</code> to the end of the
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <code>Configure</code> and rebuild the server.</p>
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <code>SO_LINGER</code> and <code>lingering_close()</code>
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen at the same time is very likely to do very bad things, so
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <h3>Increase the amount of memory used for storing
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen connection state</h3>
942302b0247403645394d848b3c620ead262a2a5Timo Sirainen BSD stores network data, such as connection states, in
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen something called an mbuf. When you get so many
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen connections that the kernel does not have enough mbufs
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen to put them all in, your kernel will likely crash. You
18ccd19c244f49665fe03cda785efa066d2c38dfTimo Sirainen can reduce the effects of the problem by increasing the
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen number of mbufs that are available; this will not
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen prevent the problem, it will just make the server go
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen longer before crashing.
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen <p>The exact way to increase them may depend on your
eca30f1fe8556c46abc75c94d03f59b2e89d4162Timo Sirainen OS; look for some reference to the number of "mbufs" or
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen "mbuf clusters". On many systems, this can be done by
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen adding the line <code>NMBCLUSTERS="n"</code>, where
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen <code>n</code> is the number of mbuf clusters you want
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen to your kernel config file and rebuilding your
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>If you are unable to do any of the above then you
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen should, as a last resort, disable KeepAlive. Edit your
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen httpd.conf and change "KeepAlive On" to "KeepAlive
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen Feedback If you have any information to add to this page,
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen please contact me at <a
17ad2164c747cedbf81dae1893063e71a3df0356Timo Sirainen href="mailto:marc@apache.org">marc@apache.org</a>.
d859478e8b106de6cea54f26861bd4232c92f62cTimo Sirainen <h2><a id="appendix" name="appendix"></a></h2>
d859478e8b106de6cea54f26861bd4232c92f62cTimo Sirainen <p>Below is a message from Roy Fielding, one of the authors
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen <h3>Why the lingering close functionality is necessary with
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen The need for a server to linger on a socket after a close
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen is noted a couple times in the HTTP specs, but not
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen explained. This explanation is based on discussions between
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen myself, Henrik Frystyk, Robert S. Thau, Dave Raggett, and
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen John C. Mallery in the hallways of MIT while I was at W3C.
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen <p>If a server closes the input side of the connection
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen while the client is sending data (or is planning to send
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen data), then the server's TCP stack will signal an RST
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen (reset) back to the client. Upon receipt of the RST, the
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen client will flush its own incoming TCP buffer back to the
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen un-ACKed packet indicated by the RST packet argument. If
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen the server has sent a message, usually an error response,
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen to the client just before the close, and the client
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen receives the RST packet before its application code has
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen read the error message from its incoming TCP buffer and
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen before the server has received the ACK sent by the client
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen upon receipt of that buffer, then the RST will flush the
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen error message before the client application has a chance to
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen see it. The result is that the client is left thinking that
77d8223da3da23b731257596abefa77e4485b77dTimo Sirainen the connection failed for no apparent reason.</p>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>There are two conditions under which this is likely to
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen <li>sending POST or PUT data without proper
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen authorization</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <li>sending multiple requests before each response
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen (pipelining) and one of the middle requests resulting in
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen an error or other break-the-connection result.</li>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>The solution in all cases is to send the response, close
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen only the write half of the connection (what shutdown is
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen supposed to do), and continue reading on the socket until
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen it is either closed by the client (signifying it has
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen finally read the response) or a timeout occurs. That is
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen what the kernel is supposed to do if SO_LINGER is set.
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen Unfortunately, SO_LINGER has no effect on some systems; on
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen some other systems, it does not have its own timeout and
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen thus the TCP memory segments just pile-up until the next
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen reboot (planned or not).</p>
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen <p>Please note that simply removing the linger code will
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen not solve the problem -- it only moves it to a different
24e5e4526d8f5cbc056ab97fd0d154d0936d7a5eTimo Sirainen and much harder one to detect.</p>
dca6d617a23e3f93af3b8df59acb46478179fe55Timo Sirainen <!--#include virtual="footer.html" -->