fin_wait_2.html revision 199a8ee5984e3708982fab1ba6ebb0a5feaea90c
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <meta name="generator" content="HTML Tidy, see www.w3.org" />
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <title>Connections in FIN_WAIT_2 and Apache</title>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <!-- Background white, links blue (unvisited), navy (visited), red (active) -->
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <body bgcolor="#FFFFFF" text="#000000" link="#0000FF"
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <!--#include virtual="header.html" -->
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h1 align="center">Connections in the FIN_WAIT_2 state and
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek Starting with the Apache 1.2 betas, people are reporting
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek many more connections in the FIN_WAIT_2 state (as reported
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek by <code>netstat</code>) than they saw using older
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek versions. When the server closes a TCP connection, it sends
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek a packet with the FIN bit set to the client, which then
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek responds with a packet with the ACK bit set. The client
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek then sends a packet with the FIN bit set to the server,
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek which responds with an ACK and the connection is closed.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek The state that the connection is in during the period
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek between when the server gets the ACK from the client and
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek the server gets the FIN from the client is known as
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek FIN_WAIT_2. See the <a
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek href="ftp://ds.internic.net/rfc/rfc793.txt">TCP RFC</a> for
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek the technical details of the state transitions.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>The FIN_WAIT_2 state is somewhat unusual in that there
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek is no timeout defined in the standard for it. This means
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek that on many operating systems, a connection in the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek FIN_WAIT_2 state will stay around until the system is
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek rebooted. If the system does not have a timeout and too
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek many FIN_WAIT_2 connections build up, it can fill up the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek space allocated for storing information about the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connections and crash the kernel. The connections in
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek FIN_WAIT_2 do not tie up an httpd process.</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek There are numerous reasons for it happening, some of them
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek may not yet be fully clear. What is known follows.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h3>Buggy clients and persistent connections</h3>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek Several clients have a bug which pops up when dealing with
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek persistent connections (aka
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek keepalives). When the connection is idle and the server
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek closes the connection (based on the <a
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek href="/mod/core.html#keepalivetimeout">KeepAliveTimeout</a>),
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek the client is programmed so that the client does not send
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek back a FIN and ACK to the server. This means that the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connection stays in the FIN_WAIT_2 state until one of the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek following happens:
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li>The client opens a new connection to the same or a
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek different site, which causes it to fully close the older
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connection on that socket.</li>
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li>The user exits the client, which on some (most?)
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek clients causes the OS to fully shutdown the
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek connection.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li>The FIN_WAIT_2 times out, on servers that have a
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek timeout for this state.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>If you are lucky, this means that the buggy client will
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek fully close the connection and release the resources on
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek your server. However, there are some cases where the socket
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek is never fully closed, such as a dialup client
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek disconnecting from their provider before closing the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek client. In addition, a client might sit idle for days
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek without making another connection, and thus may hold its
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek end of the socket open for days even though it has no
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek further use for it. <strong>This is a bug in the browser or
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek in its operating system's TCP implementation.</strong></p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>The clients on which this problem has been verified to
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li>Mozilla/3.01 (X11; I; FreeBSD 2.1.5-RELEASE
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li>Mozilla/2.02 (X11; I; FreeBSD 2.1.5-RELEASE
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li>Mozilla/3.01Gold (X11; I; SunOS 5.5 sun4m)</li>
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <p>This does not appear to be a problem on:</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>It is expected that many other clients have the same
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek problem. What a client <strong>should do</strong> is
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek periodically check its open socket(s) to see if they have
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek been closed by the server, and close their side of the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connection if the server has closed. This check need only
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek occur once every few seconds, and may even be detected by a
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek OS signal on some systems (<em>e.g.</em>, Win95 and NT
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek clients have this capability, but they seem to be ignoring
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>Apache <strong>cannot</strong> avoid these FIN_WAIT_2
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek states unless it disables persistent connections for the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek buggy clients, just like we recommend doing for Navigator
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek 2.x clients due to other bugs. However, non-persistent
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek connections increase the total number of connections needed
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek per client and slow retrieval of an image-laden web page.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek Since non-persistent connections have their own resource
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek consumptions and a short waiting period after each closure,
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek a busy server may need persistence in order to best serve
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek its clients.</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>As far as we know, the client-caused FIN_WAIT_2 problem
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek is present for all servers that support persistent
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connections, including Apache 1.1.x and 1.2.</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h3>A necessary bit of code introduced in 1.2</h3>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek While the above bug is a problem, it is not the whole
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek problem. Some users have observed no FIN_WAIT_2 problems
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek with Apache 1.1.x, but with 1.2b enough connections build
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek up in the FIN_WAIT_2 state to crash their server. The most
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek likely source for additional FIN_WAIT_2 states is a
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek function called <code>lingering_close()</code> which was
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek added between 1.1 and 1.2. This function is necessary for
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek the proper handling of persistent connections and any
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek request which includes content in the message body
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek (<em>e.g.</em>, PUTs and POSTs). What it does is read any
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek data sent by the client for a certain time after the server
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek closes the connection. The exact reasons for doing this are
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek somewhat complicated, but involve what happens if the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek client is making a request at the same time the server
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek sends a response and closes the connection. Without
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek lingering, the client might be forced to reset its TCP
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek input buffer before it has a chance to read the server's
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek response, and thus understand why the connection has
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek closed. See the <a href="#appendix">appendix</a> for more
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>The code in <code>lingering_close()</code> appears to
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek cause problems for a number of factors, including the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek change in traffic patterns that it causes. The code has
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek been thoroughly reviewed and we are not aware of any bugs
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek in it. It is possible that there is some problem in the BSD
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek TCP stack, aside from the lack of a timeout for the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek FIN_WAIT_2 state, exposed by the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>lingering_close</code> code that causes the observed
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek problems.</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek What can I do about it? There are several possible
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek workarounds to the problem, some of which work better than
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek The obvious workaround is to simply have a timeout for the
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek FIN_WAIT_2 state. This is not specified by the RFC, and
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek could be claimed to be a violation of the RFC, but it is
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek widely recognized as being necessary. The following systems
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek are known to have a timeout:
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li><a href="http://www.freebsd.org/">FreeBSD</a>
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek versions starting at 2.0 or possibly earlier.</li>
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li><a href="http://www.netbsd.org/">NetBSD</a> version
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li><a href="http://www.openbsd.org/">OpenBSD</a> all
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek versions(?)</li>
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li><a href="http://www.bsdi.com/">BSD/OS</a> 2.1, with
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek href="ftp://ftp.bsdi.com/bsdi/patches/patches-2.1/K210-027">
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek <li><a href="http://www.sun.com/">Solaris</a> as of
f17d26a8db285622a5cd5f21c7488b62eedc2cf8Jakub Hrozek around version 2.2. The timeout can be tuned by using
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>tcp_fin_wait_2_flush_interval</code>, but the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek default should be appropriate for most servers and
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek improper tuning can have negative impacts.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li><a href="http://www.linux.org/">Linux</a> 2.0.x and
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek earlier(?)</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li><a href="http://www.hp.com/">HP-UX</a> 10.x defaults
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek to terminating connections in the FIN_WAIT_2 state after
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek the normal keepalive timeouts. This does not refer to the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek persistent connection or HTTP keepalive timeouts, but the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>SO_LINGER</code> socket option which is enabled by
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek Apache. This parameter can be adjusted by using
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>nettune</code> to modify parameters such as
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>tcp_keepstart</code> and <code>tcp_keepstop</code>.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek In later revisions, there is an explicit timer for
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connections in FIN_WAIT_2 that can be modified; contact
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek HP support for details.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li><a href="http://www.sgi.com/">SGI IRIX</a> can be
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek patched to support a timeout. For IRIX 5.3, 6.2, and 6.3,
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek use patches 1654, 1703 and 1778 respectively. If you have
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek trouble locating these patches, please contact your SGI
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek support channel for help.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li><a href="http://www.ncr.com/">NCR's MP RAS Unix</a>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek 2.xx and 3.xx both have FIN_WAIT_2 timeouts. In 2.xx it
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek is non-tunable at 600 seconds, while in 3.xx it defaults
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek to 600 seconds and is calculated based on the tunable
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek "max keep alive probes" (default of 8) multiplied by the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek "keep alive interval" (default 75 seconds).</li>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <li><a href="http://www.sequent.com">Sequent's ptx/TCP/IP
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek for DYNIX/ptx</a> has had a FIN_WAIT_2 timeout since
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek around release 4.1 in mid-1994.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>The following systems are known to not have a
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek timeout:</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li><a href="http://www.sun.com/">SunOS 4.x</a> does not
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek and almost certainly never will have one because it as at
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek the very end of its development cycle for Sun. If you
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek have kernel source should be easy to patch.</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek href="http://www.apache.org/dist/httpd/contrib/patches/1.2/fin_wait_2.patch">
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek patch available</a> for adding a timeout to the FIN_WAIT_2
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek state; it was originally intended for BSD/OS, but should be
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek adaptable to most systems using BSD networking code. You
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek need kernel source code to be able to use it. </p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h3>Compile without using
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek It is possible to compile Apache 1.2 without using the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>lingering_close()</code> function. This will result
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek in that section of code being similar to that which was in
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek 1.1. If you do this, be aware that it can cause problems
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek with PUTs, POSTs and persistent connections, especially if
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek the client uses pipelining. That said, it is no worse than
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek on 1.1, and we understand that keeping your server running
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek is quite important.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>To compile without the <code>lingering_close()</code>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek function, add <code>-DNO_LINGCLOSE</code> to the end of the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>Configure</code> and rebuild the server.</p>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h3>Use <code>SO_LINGER</code> as an alternative to
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek On most systems, there is an option called
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>setsockopt(2)</code>. It does something very similar
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek to <code>lingering_close()</code>, except that it is broken
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek on many systems so that it causes far more problems than
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <code>lingering_close</code>. On some systems, it could
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek possibly work better so it may be worth a try if you have
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek no other alternatives.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek -DNO_LINGCLOSE</code> to the end of the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>Configure</code> and rebuild the server.</p>
64ddff90c7fcc02ccb06824ac93af7d5f361a88fJan Zeleny <code>SO_LINGER</code> and <code>lingering_close()</code>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek at the same time is very likely to do very bad things, so
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <h3>Increase the amount of memory used for storing
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connection state</h3>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek BSD stores network data, such as connection states, in
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek something called an mbuf. When you get so many
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek connections that the kernel does not have enough mbufs
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek to put them all in, your kernel will likely crash. You
f3dc9ac52a808fcbedc7fb1ae83964b523c72df2Stephen Gallagher can reduce the effects of the problem by increasing the
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek number of mbufs that are available; this will not
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek prevent the problem, it will just make the server go
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek longer before crashing.
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>The exact way to increase them may depend on your
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek OS; look for some reference to the number of "mbufs" or
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek "mbuf clusters". On many systems, this can be done by
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek adding the line <code>NMBCLUSTERS="n"</code>, where
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <code>n</code> is the number of mbuf clusters you want
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek to your kernel config file and rebuilding your
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <p>If you are unable to do any of the above then you
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek should, as a last resort, disable KeepAlive. Edit your
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek httpd.conf and change "KeepAlive On" to "KeepAlive
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <h2><a id="appendix" name="appendix">Appendix</a></h2>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>Below is a message from Roy Fielding, one of the authors
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <h3>Why the lingering close functionality is necessary with
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek The need for a server to linger on a socket after a close
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek is noted a couple times in the HTTP specs, but not
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek explained. This explanation is based on discussions between
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek myself, Henrik Frystyk, Robert S. Thau, Dave Raggett, and
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek John C. Mallery in the hallways of MIT while I was at W3C.
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>If a server closes the input side of the connection
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek while the client is sending data (or is planning to send
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek data), then the server's TCP stack will signal an RST
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek (reset) back to the client. Upon receipt of the RST, the
f3dc9ac52a808fcbedc7fb1ae83964b523c72df2Stephen Gallagher client will flush its own incoming TCP buffer back to the
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek un-ACKed packet indicated by the RST packet argument. If
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek the server has sent a message, usually an error response,
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek to the client just before the close, and the client
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek receives the RST packet before its application code has
2f3ee3f49019f5b60adbe073070f31e6e2d7c7abStephen Gallagher read the error message from its incoming TCP buffer and
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek before the server has received the ACK sent by the client
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek upon receipt of that buffer, then the RST will flush the
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek error message before the client application has a chance to
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek see it. The result is that the client is left thinking that
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek the connection failed for no apparent reason.</p>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>There are two conditions under which this is likely to
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li>sending POST or PUT data without proper
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek authorization</li>
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek <li>sending multiple requests before each response
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek (pipelining) and one of the middle requests resulting in
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek an error or other break-the-connection result.</li>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>The solution in all cases is to send the response, close
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek only the write half of the connection (what shutdown is
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek supposed to do), and continue reading on the socket until
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek it is either closed by the client (signifying it has
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek finally read the response) or a timeout occurs. That is
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek what the kernel is supposed to do if SO_LINGER is set.
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek Unfortunately, SO_LINGER has no effect on some systems; on
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek some other systems, it does not have its own timeout and
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek thus the TCP memory segments just pile-up until the next
cc84fd46f356c4a36a721ab135a33ec77c93e34dJakub Hrozek reboot (planned or not).</p>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <p>Please note that simply removing the linger code will
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek not solve the problem -- it only moves it to a different
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek and much harder one to detect.</p>
7f5de490e24f1389501b939b742a9e5675f1c41dJakub Hrozek <!--#include virtual="footer.html" -->