perf-tuning.html revision e383bf77be4a4398d407bbb6c7656984a9cdfacc
91d632c867159b669d90fc7e172295433d0519efgwr<!-- Background white, links blue (unvisited), navy (visited), red (active) -->
91d632c867159b669d90fc7e172295433d0519efgwr BGCOLOR="#FFFFFF"
91d632c867159b669d90fc7e172295433d0519efgwr TEXT="#000000"
91d632c867159b669d90fc7e172295433d0519efgwr LINK="#0000FF"
91d632c867159b669d90fc7e172295433d0519efgwr VLINK="#000080"
91d632c867159b669d90fc7e172295433d0519efgwr ALINK="#FF0000"
91d632c867159b669d90fc7e172295433d0519efgwr<P>Author: Dean Gaudet
91d632c867159b669d90fc7e172295433d0519efgwr<P>Apache is a general webserver, which is designed to be correct first, and
91d632c867159b669d90fc7e172295433d0519efgwrfast second. Even so, it's performance is quite satisfactory. Most
91d632c867159b669d90fc7e172295433d0519efgwrsites have less than 10Mbits of outgoing bandwidth, which Apache can
91d632c867159b669d90fc7e172295433d0519efgwrfill using only a low end Pentium-based webserver. In practice sites
148c5f43199ca0b43fc8e3b643aab11cd66ea327Alan Wrightwith more bandwidth require more than one machine to fill the bandwidth
91d632c867159b669d90fc7e172295433d0519efgwrdue to other constraints (such as CGI or database transaction overhead).
91d632c867159b669d90fc7e172295433d0519efgwrFor these reasons the development focus has been mostly on correctness
91d632c867159b669d90fc7e172295433d0519efgwrand configurability.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Unfortunately many folks overlook these facts and cite raw performance
91d632c867159b669d90fc7e172295433d0519efgwrnumbers as if they are some indication of the quality of a web server
91d632c867159b669d90fc7e172295433d0519efgwrproduct. There is a bare minimum performance that is acceptable, beyond
91d632c867159b669d90fc7e172295433d0519efgwrthat extra speed only caters to a much smaller segment of the market.
91d632c867159b669d90fc7e172295433d0519efgwrBut in order to avoid this hurdle to the acceptance of Apache in some
91d632c867159b669d90fc7e172295433d0519efgwrmarkets, effort was put into Apache 1.3 to bring performance up to a
91d632c867159b669d90fc7e172295433d0519efgwrpoint where the difference with other high-end webservers is minimal.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Finally there are the folks who just plain want to see how fast something
91d632c867159b669d90fc7e172295433d0519efgwrcan go. The author falls into this category. The rest of this document
91d632c867159b669d90fc7e172295433d0519efgwris dedicated to these folks who want to squeeze every last bit of
91d632c867159b669d90fc7e172295433d0519efgwrperformance out of Apache's current model, and want to understand why
91d632c867159b669d90fc7e172295433d0519efgwrit does some things which slow it down.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Note that this is tailored towards Apache 1.3 on Unix. Some of it applies
91d632c867159b669d90fc7e172295433d0519efgwrto Apache on NT. Apache on NT has not been tuned for performance yet,
91d632c867159b669d90fc7e172295433d0519efgwrin fact it probably performs very poorly because NT performance requires
91d632c867159b669d90fc7e172295433d0519efgwra different programming model.
91d632c867159b669d90fc7e172295433d0519efgwr<P>The single biggest hardware issue affecting webserver performance
91d632c867159b669d90fc7e172295433d0519efgwris RAM. A webserver should never ever have to swap, swapping increases
91d632c867159b669d90fc7e172295433d0519efgwrthe latency of each request beyond a point that users consider "fast
91d632c867159b669d90fc7e172295433d0519efgwrenough". This causes users to hit stop and reload, further increasing
91d632c867159b669d90fc7e172295433d0519efgwrthe load. You can, and should, control the <CODE>MaxClients</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrsetting so that your server does not spawn so many children it starts
91d632c867159b669d90fc7e172295433d0519efgwr<P>Beyond that the rest is mundane: get a fast enough CPU, a fast enough
91d632c867159b669d90fc7e172295433d0519efgwrnetwork card, and fast enough disks, where "fast enough" is something
91d632c867159b669d90fc7e172295433d0519efgwrthat needs to be determined by experimentation.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Operating system choice is largely a matter of local concerns. But
91d632c867159b669d90fc7e172295433d0519efgwra general guideline is to always apply the latest vendor TCP/IP patches.
91d632c867159b669d90fc7e172295433d0519efgwrHTTP serving completely breaks many of the assumptions built into Unix
91d632c867159b669d90fc7e172295433d0519efgwrkernels up through 1994 and even 1995. Good choices include
91d632c867159b669d90fc7e172295433d0519efgwrrecent FreeBSD, and Linux.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Prior to Apache 1.3, <CODE>HostnameLookups</CODE> defaulted to On.
91d632c867159b669d90fc7e172295433d0519efgwrThis adds latency
91d632c867159b669d90fc7e172295433d0519efgwrto every request because it requires a DNS lookup to complete before
91d632c867159b669d90fc7e172295433d0519efgwrthe request is finished. In Apache 1.3 this setting defaults to Off.
91d632c867159b669d90fc7e172295433d0519efgwrHowever (1.3 or later), if you use any <CODE>allow from domain</CODE> or
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>deny from domain</CODE> directives then you will pay for a
91d632c867159b669d90fc7e172295433d0519efgwrdouble reverse DNS lookup (a reverse, followed by a forward to make sure
91d632c867159b669d90fc7e172295433d0519efgwrthat the reverse is not being spoofed). So for the highest performance
91d632c867159b669d90fc7e172295433d0519efgwravoid using these directives (it's fine to use IP addresses rather than
91d632c867159b669d90fc7e172295433d0519efgwrdomain names).
91d632c867159b669d90fc7e172295433d0519efgwr<P>Note that it's possible to scope the directives, such as within
91d632c867159b669d90fc7e172295433d0519efgwra <CODE><Location /server-status></CODE> section. In this
91d632c867159b669d90fc7e172295433d0519efgwrcase the DNS lookups are only performed on requests matching the
91d632c867159b669d90fc7e172295433d0519efgwrcriteria. Here's an example which disables
91d632c867159b669d90fc7e172295433d0519efgwrlookups except for .html and .cgi files:
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon RossHostnameLookups off
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<Files ~ "\.(html|cgi)$>
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross HostnameLookups on
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross</Files>
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon RossBut even still, if you just need DNS names
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Rossin some CGIs you could consider doing the
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<CODE>gethostbyname</CODE> call in the specific CGIs that need it.
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<P>Wherever in your URL-space you do not have an
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>Options SymLinksIfOwnerMatch</CODE> Apache will have to
91d632c867159b669d90fc7e172295433d0519efgwrissue extra system calls to check up on symlinks. One extra call per
91d632c867159b669d90fc7e172295433d0519efgwrfilename component. For example, if you had:
91d632c867159b669d90fc7e172295433d0519efgwr<Directory />
91d632c867159b669d90fc7e172295433d0519efgwr Options SymLinksIfOwnerMatch
91d632c867159b669d90fc7e172295433d0519efgwr</Directory>
91d632c867159b669d90fc7e172295433d0519efgwrand a request is made for the URI <CODE>/index.html</CODE>.
91d632c867159b669d90fc7e172295433d0519efgwrThen Apache will perform <CODE>lstat(2)</CODE> on <CODE>/www</CODE>,
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>/www/htdocs</CODE>, and <CODE>/www/htdocs/index.html</CODE>. The
91d632c867159b669d90fc7e172295433d0519efgwrso they will occur on every single request. If you really desire the
91d632c867159b669d90fc7e172295433d0519efgwrsymlinks security checking you can do something like this:
91d632c867159b669d90fc7e172295433d0519efgwr<Directory />
91d632c867159b669d90fc7e172295433d0519efgwr Options FollowSymLinks
91d632c867159b669d90fc7e172295433d0519efgwr</Directory>
91d632c867159b669d90fc7e172295433d0519efgwr Options -FollowSymLinks +SymLinksIfOwnerMatch
91d632c867159b669d90fc7e172295433d0519efgwr</Directory>
91d632c867159b669d90fc7e172295433d0519efgwrThis at least avoids the extra checks for the <CODE>DocumentRoot</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrpath. Note that you'll need to add similar sections if you have any
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<CODE>Alias</CODE> or <CODE>RewriteRule</CODE> paths outside of your
91d632c867159b669d90fc7e172295433d0519efgwrdocument root. For highest performance, and no symlink protection,
91d632c867159b669d90fc7e172295433d0519efgwrset <CODE>FollowSymLinks</CODE> everywhere, and never set
91d632c867159b669d90fc7e172295433d0519efgwr<P>Wherever in your URL-space you allow overrides (typically
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>.htaccess</CODE> files) Apache will attempt to open
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>.htaccess</CODE> for each filename component. For example,
91d632c867159b669d90fc7e172295433d0519efgwr<Directory />
91d632c867159b669d90fc7e172295433d0519efgwr AllowOverride all
91d632c867159b669d90fc7e172295433d0519efgwr</Directory>
91d632c867159b669d90fc7e172295433d0519efgwrand a request is made for the URI <CODE>/index.html</CODE>. Then
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>/www/.htaccess</CODE>, and <CODE>/www/htdocs/.htaccess</CODE>.
91d632c867159b669d90fc7e172295433d0519efgwrThe solutions are similar to the previous case of <CODE>Options
91d632c867159b669d90fc7e172295433d0519efgwrFollowSymLinks</CODE>. For highest performance use
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>AllowOverride None</CODE> everywhere in your filesystem.
91d632c867159b669d90fc7e172295433d0519efgwr<P>If at all possible, avoid content-negotiation if you're really
91d632c867159b669d90fc7e172295433d0519efgwrinterested in every last ounce of performance. In practice the
91d632c867159b669d90fc7e172295433d0519efgwrbenefits of negotiation outweigh the performance penalties. There's
91d632c867159b669d90fc7e172295433d0519efgwrone case where you can speed up the server. Instead of using
91d632c867159b669d90fc7e172295433d0519efgwra wildcard such as:
91d632c867159b669d90fc7e172295433d0519efgwrDirectoryIndex index
91d632c867159b669d90fc7e172295433d0519efgwrUse a complete list of options:
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Rosswhere you list the most common choice first.
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<P>Prior to Apache 1.3 the <CODE>MinSpareServers</CODE>,
02d09e03eb27f3a2dc299de704e45dae5173f43fGordon Ross<CODE>MaxSpareServers</CODE>, and <CODE>StartServers</CODE> settings
91d632c867159b669d90fc7e172295433d0519efgwrall had drastic effects on benchmark results. In particular, Apache
91d632c867159b669d90fc7e172295433d0519efgwrrequired a "ramp-up" period in order to reach a number of children
91d632c867159b669d90fc7e172295433d0519efgwrsufficient to serve the load being applied. After the initial
91d632c867159b669d90fc7e172295433d0519efgwrspawning of <CODE>StartServers</CODE> children, only one child per
91d632c867159b669d90fc7e172295433d0519efgwrsecond would be created to satisfy the <CODE>MinSpareServers</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrsetting. So a server being accessed by 100 simultaneous clients,
91d632c867159b669d90fc7e172295433d0519efgwrusing the default <CODE>StartServers</CODE> of 5 would take on
91d632c867159b669d90fc7e172295433d0519efgwrthe order 95 seconds to spawn enough children to handle the load. This
91d632c867159b669d90fc7e172295433d0519efgwrworks fine in practice on real-life servers, because they aren't restarted
91d632c867159b669d90fc7e172295433d0519efgwrfrequently. But does really poorly on benchmarks which might only run
91d632c867159b669d90fc7e172295433d0519efgwrfor ten minutes.
91d632c867159b669d90fc7e172295433d0519efgwr<P>The one-per-second rule was implemented in an effort to avoid
91d632c867159b669d90fc7e172295433d0519efgwrswamping the machine with the startup of new children. If the machine
91d632c867159b669d90fc7e172295433d0519efgwris busy spawning children it can't service requests. But it has such
91d632c867159b669d90fc7e172295433d0519efgwra drastic effect on the perceived performance of Apache that it had
91d632c867159b669d90fc7e172295433d0519efgwrto be replaced. As of Apache 1.3,
91d632c867159b669d90fc7e172295433d0519efgwrthe code will relax the one-per-second rule. It
91d632c867159b669d90fc7e172295433d0519efgwrwill spawn one, wait a second, then spawn two, wait a second, then spawn
91d632c867159b669d90fc7e172295433d0519efgwrfour, and it will continue exponentially until it is spawning 32 children
91d632c867159b669d90fc7e172295433d0519efgwrper second. It will stop whenever it satisfies the
91d632c867159b669d90fc7e172295433d0519efgwr<P>This appears to be responsive enough that it's
91d632c867159b669d90fc7e172295433d0519efgwralmost unnecessary to twiddle the <CODE>MinSpareServers</CODE>,
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>MaxSpareServers</CODE> and <CODE>StartServers</CODE> knobs. When
91d632c867159b669d90fc7e172295433d0519efgwrmore than 4 children are spawned per second, a message will be emitted
91d632c867159b669d90fc7e172295433d0519efgwrto the <CODE>ErrorLog</CODE>. If you see a lot of these errors then
613a2f6ba31e891e3d947a356daf5e563d43c1ceGordon Rossconsider tuning these settings. Use the <CODE>mod_status</CODE> output
91d632c867159b669d90fc7e172295433d0519efgwras a guide.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Related to process creation is process death induced by the
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>MaxRequestsPerChild</CODE> setting. By default this is 30, which
91d632c867159b669d90fc7e172295433d0519efgwris probably far too low unless your server is using a module such as
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>mod_perl</CODE> which causes children to have bloated memory
91d632c867159b669d90fc7e172295433d0519efgwrimages. If your server is serving mostly static pages then consider
91d632c867159b669d90fc7e172295433d0519efgwrraising this value to something like 10000. The code is robust enough
91d632c867159b669d90fc7e172295433d0519efgwrthat this shouldn't be a problem.
91d632c867159b669d90fc7e172295433d0519efgwr<P>When keep-alives are in use, children will be kept busy
91d632c867159b669d90fc7e172295433d0519efgwrdoing nothing waiting for more requests on the already open
91d632c867159b669d90fc7e172295433d0519efgwr15 seconds attempts to minimize this effect. The tradeoff
91d632c867159b669d90fc7e172295433d0519efgwrhere is between network bandwidth and server resources.
91d632c867159b669d90fc7e172295433d0519efgwrIn no event should you raise this above about 60 seconds, as
91d632c867159b669d90fc7e172295433d0519efgwr<A HREF="http://www.research.digital.com/wrl/techreports/abstracts/95.4.html"
91d632c867159b669d90fc7e172295433d0519efgwr>most of the benefits are lost</A>.
91d632c867159b669d90fc7e172295433d0519efgwrand you also set <CODE>ExtendedStatus On</CODE> when building and running
91d632c867159b669d90fc7e172295433d0519efgwrApache, then on every request Apache will perform two calls to
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>gettimeofday(2)</CODE> (or <CODE>times(2)</CODE> depending
91d632c867159b669d90fc7e172295433d0519efgwron your operating system), and (pre-1.3) several extra calls to
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>time(2)</CODE>. This is all done so that the status report
91d632c867159b669d90fc7e172295433d0519efgwrcontains timing indications. For highest performance, set
91d632c867159b669d90fc7e172295433d0519efgwr<P>This discusses a shortcoming in the Unix socket API.
91d632c867159b669d90fc7e172295433d0519efgwrSuppose your
91d632c867159b669d90fc7e172295433d0519efgwrweb server uses multiple <CODE>Listen</CODE> statements to listen on
91d632c867159b669d90fc7e172295433d0519efgwreither multiple ports or multiple addresses. In order to test each
91d632c867159b669d90fc7e172295433d0519efgwrsocket to see if a connection is ready Apache uses <CODE>select(2)</CODE>.
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>select(2)</CODE> indicates that a socket has <EM>zero</EM> or
91d632c867159b669d90fc7e172295433d0519efgwr<EM>at least one</EM> connection waiting on it. Apache's model includes
91d632c867159b669d90fc7e172295433d0519efgwrmultiple children, and all the idle ones test for new connections at the
91d632c867159b669d90fc7e172295433d0519efgwrsame time. A naive implementation looks something like this
91d632c867159b669d90fc7e172295433d0519efgwr(these examples do not match the code, they're contrived for
91d632c867159b669d90fc7e172295433d0519efgwrpedagogical purposes):
91d632c867159b669d90fc7e172295433d0519efgwr for (;;) {
91d632c867159b669d90fc7e172295433d0519efgwr for (;;) {
91d632c867159b669d90fc7e172295433d0519efgwr fd_set accept_fds;
91d632c867159b669d90fc7e172295433d0519efgwr FD_ZERO (&accept_fds);
91d632c867159b669d90fc7e172295433d0519efgwr for (i = first_socket; i <= last_socket; ++i) {
91d632c867159b669d90fc7e172295433d0519efgwr FD_SET (i, &accept_fds);
91d632c867159b669d90fc7e172295433d0519efgwr rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);
91d632c867159b669d90fc7e172295433d0519efgwr if (rc < 1) continue;
91d632c867159b669d90fc7e172295433d0519efgwr new_connection = -1;
91d632c867159b669d90fc7e172295433d0519efgwr for (i = first_socket; i <= last_socket; ++i) {
91d632c867159b669d90fc7e172295433d0519efgwr if (FD_ISSET (i, &accept_fds)) {
91d632c867159b669d90fc7e172295433d0519efgwr new_connection = accept (i, NULL, NULL);
91d632c867159b669d90fc7e172295433d0519efgwr if (new_connection != -1) break;
91d632c867159b669d90fc7e172295433d0519efgwr if (new_connection != -1) break;
91d632c867159b669d90fc7e172295433d0519efgwr process the new_connection;
91d632c867159b669d90fc7e172295433d0519efgwrBut this naive implementation has a serious starvation problem. Recall
91d632c867159b669d90fc7e172295433d0519efgwrthat multiple children execute this loop at the same time, and so multiple
91d632c867159b669d90fc7e172295433d0519efgwrchildren will block at <CODE>select</CODE> when they are in between
91d632c867159b669d90fc7e172295433d0519efgwrrequests. All those blocked children will awaken and return from
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>select</CODE> when a single request appears on any socket
91d632c867159b669d90fc7e172295433d0519efgwr(the number of children which awaken varies depending on the operating
91d632c867159b669d90fc7e172295433d0519efgwrsystem and timing issues).
91d632c867159b669d90fc7e172295433d0519efgwrThey will all then fall down into the loop and try to <CODE>accept</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrthe connection. But only one will succeed (assuming there's still only
91d632c867159b669d90fc7e172295433d0519efgwrone connection ready), the rest will be <EM>blocked</EM> in
91d632c867159b669d90fc7e172295433d0519efgwrThis effectively locks those children into serving requests from that
91d632c867159b669d90fc7e172295433d0519efgwrone socket and no other sockets, and they'll be stuck there until enough
91d632c867159b669d90fc7e172295433d0519efgwrnew requests appear on that socket to wake them all up.
91d632c867159b669d90fc7e172295433d0519efgwrThis starvation problem was first documented in
91d632c867159b669d90fc7e172295433d0519efgwr<A HREF="http://bugs.apache.org/index/full/467">PR#467</A>. There
91d632c867159b669d90fc7e172295433d0519efgwrare at least two solutions.
91d632c867159b669d90fc7e172295433d0519efgwr<P>One solution is to make the sockets non-blocking. In this case the
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>accept</CODE> won't block the children, and they will be allowed
91d632c867159b669d90fc7e172295433d0519efgwrto continue immediately. But this wastes CPU time. Suppose you have
91d632c867159b669d90fc7e172295433d0519efgwrten idle children in <CODE>select</CODE>, and one connection arrives.
91d632c867159b669d90fc7e172295433d0519efgwrThen nine of those children will wake up, try to <CODE>accept</CODE> the
91d632c867159b669d90fc7e172295433d0519efgwrconnection, fail, and loop back into <CODE>select</CODE>, accomplishing
91d632c867159b669d90fc7e172295433d0519efgwrnothing. Meanwhile none of those children are servicing requests that
91d632c867159b669d90fc7e172295433d0519efgwroccurred on other sockets until they get back up to the <CODE>select</CODE>
91d632c867159b669d90fc7e172295433d0519efgwragain. Overall this solution does not seem very fruitful unless you
91d632c867159b669d90fc7e172295433d0519efgwrhave as many idle CPUs (in a multiprocessor box) as you have idle children,
91d632c867159b669d90fc7e172295433d0519efgwrnot a very likely situation.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Another solution, the one used by Apache, is to serialize entry into
91d632c867159b669d90fc7e172295433d0519efgwrthe inner loop. The loop looks like this (differences highlighted):
613a2f6ba31e891e3d947a356daf5e563d43c1ceGordon Ross fd_set accept_fds;
91d632c867159b669d90fc7e172295433d0519efgwr FD_ZERO (&accept_fds);
91d632c867159b669d90fc7e172295433d0519efgwr for (i = first_socket; i <= last_socket; ++i) {
91d632c867159b669d90fc7e172295433d0519efgwr FD_SET (i, &accept_fds);
91d632c867159b669d90fc7e172295433d0519efgwr rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);
91d632c867159b669d90fc7e172295433d0519efgwr if (rc < 1) continue;
91d632c867159b669d90fc7e172295433d0519efgwr new_connection = -1;
91d632c867159b669d90fc7e172295433d0519efgwr for (i = first_socket; i <= last_socket; ++i) {
91d632c867159b669d90fc7e172295433d0519efgwr if (FD_ISSET (i, &accept_fds)) {
91d632c867159b669d90fc7e172295433d0519efgwr new_connection = accept (i, NULL, NULL);
91d632c867159b669d90fc7e172295433d0519efgwr if (new_connection != -1) break;
91d632c867159b669d90fc7e172295433d0519efgwr if (new_connection != -1) break;
91d632c867159b669d90fc7e172295433d0519efgwr process the new_connection;
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakki<CODE>accept_mutex_on</CODE> and <CODE>accept_mutex_off</CODE>
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakkiimplement a mutual exclusion semaphore. Only one child can have the
91d632c867159b669d90fc7e172295433d0519efgwrmutex at any time. There are several choices for implementing these
91d632c867159b669d90fc7e172295433d0519efgwrmutexes. The choice is defined in <CODE>src/conf.h</CODE> (pre-1.3) or
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>src/main/conf.h</CODE> (1.3 or later). Some architectures
91d632c867159b669d90fc7e172295433d0519efgwrdo not have any locking choice made, on these architectures it is unsafe
91d632c867159b669d90fc7e172295433d0519efgwr<DD>This method uses the <CODE>flock(2)</CODE> system call to lock a
91d632c867159b669d90fc7e172295433d0519efgwrlock file (located by the <CODE>LockFile</CODE> directive).
91d632c867159b669d90fc7e172295433d0519efgwr<DD>This method uses the <CODE>fcntl(2)</CODE> system call to lock a
91d632c867159b669d90fc7e172295433d0519efgwrlock file (located by the <CODE>LockFile</CODE> directive).
91d632c867159b669d90fc7e172295433d0519efgwr<DD>(1.3 or later) This method uses SysV-style semaphores to implement the
91d632c867159b669d90fc7e172295433d0519efgwrmutex. Unfortunately SysV-style semaphores have some bad side-effects.
91d632c867159b669d90fc7e172295433d0519efgwrOne is that it's possible Apache will die without cleaning up the semaphore
91d632c867159b669d90fc7e172295433d0519efgwr(see the <CODE>ipcs(8)</CODE> man page). The other is that the semaphore
91d632c867159b669d90fc7e172295433d0519efgwrAPI allows for a denial of service attack by any CGIs running under the
91d632c867159b669d90fc7e172295433d0519efgwrsame uid as the webserver (i.e. all CGIs unless you use something
91d632c867159b669d90fc7e172295433d0519efgwrlike suexec or cgiwrapper). For these reasons this method is not used
91d632c867159b669d90fc7e172295433d0519efgwron any architecture except IRIX (where the previous two are prohibitively
91d632c867159b669d90fc7e172295433d0519efgwrexpensive on most IRIX boxes).
91d632c867159b669d90fc7e172295433d0519efgwr<DD>(1.3 or later) This method is only available on IRIX, and uses
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>usconfig(2)</CODE> to create a mutex. While this method avoids
91d632c867159b669d90fc7e172295433d0519efgwrthe hassles of SysV-style semaphores, it is not the default for IRIX.
91d632c867159b669d90fc7e172295433d0519efgwrThis is because on single processor IRIX boxes (5.3 or 6.2) the
91d632c867159b669d90fc7e172295433d0519efgwruslock code is two orders of magnitude slower than the SysV-semaphore
91d632c867159b669d90fc7e172295433d0519efgwrcode. On multi-processor IRIX boxes the uslock code is an order of magnitude
91d632c867159b669d90fc7e172295433d0519efgwrfaster than the SysV-semaphore code. Kind of a messed up situation.
91d632c867159b669d90fc7e172295433d0519efgwrSo if you're using a multiprocessor IRIX box then you should rebuild your
91d632c867159b669d90fc7e172295433d0519efgwrwebserver with <CODE>-DUSE_USLOCK_SERIALIZED_ACCEPT</CODE> on the
91d632c867159b669d90fc7e172295433d0519efgwr<DD>(1.3 or later) This method uses POSIX mutexes and should work on
91d632c867159b669d90fc7e172295433d0519efgwrany architecture implementing the full POSIX threads specification,
91d632c867159b669d90fc7e172295433d0519efgwrhowever appears to only work on Solaris (2.5 or later), and even then
91d632c867159b669d90fc7e172295433d0519efgwronly in certain configurations. If you experiment with this you should
91d632c867159b669d90fc7e172295433d0519efgwrwatch out for your server hanging and not responding. Static content
91d632c867159b669d90fc7e172295433d0519efgwronly servers may work just fine.
91d632c867159b669d90fc7e172295433d0519efgwr<P>If your system has another method of serialization which isn't in the
91d632c867159b669d90fc7e172295433d0519efgwrabove list then it may be worthwhile adding code for it (and submitting
91d632c867159b669d90fc7e172295433d0519efgwra patch back to Apache).
91d632c867159b669d90fc7e172295433d0519efgwr<P>Another solution that has been considered but never implemented is
91d632c867159b669d90fc7e172295433d0519efgwrto partially serialize the loop -- that is, let in a certain number
91d632c867159b669d90fc7e172295433d0519efgwrof processes. This would only be of interest on multiprocessor boxes
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakkiwhere it's possible multiple children could run simultaneously, and the
91d632c867159b669d90fc7e172295433d0519efgwrserialization actually doesn't take advantage of the full bandwidth.
c1374a13e412c4ec42cba867e57347a0e049a822Surya PrakkiThis is a possible area of future investigation, but priority remains
91d632c867159b669d90fc7e172295433d0519efgwrlow because highly parallel web servers are not the norm.
91d632c867159b669d90fc7e172295433d0519efgwr<P>Ideally you should run servers without multiple <CODE>Listen</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrstatements if you want the highest performance. But read on.
91d632c867159b669d90fc7e172295433d0519efgwr<P>The above is fine and dandy for multiple socket servers, but what
91d632c867159b669d90fc7e172295433d0519efgwrabout single socket servers? In theory they shouldn't experience
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakkiany of these same problems because all children can just block in
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>accept(2)</CODE> until a connection arrives, and no starvation
91d632c867159b669d90fc7e172295433d0519efgwrresults. In practice this hides almost the same "spinning" behaviour
91d632c867159b669d90fc7e172295433d0519efgwrdiscussed above in the non-blocking solution. The way that most TCP
91d632c867159b669d90fc7e172295433d0519efgwrstacks are implemented, the kernel actually wakes up all processes blocked
91d632c867159b669d90fc7e172295433d0519efgwrin <CODE>accept</CODE> when a single connection arrives. One of those
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakkiprocesses gets the connection and returns to user-space, the rest spin in
91d632c867159b669d90fc7e172295433d0519efgwrthe kernel and go back to sleep when they discover there's no connection
91d632c867159b669d90fc7e172295433d0519efgwrfor them. This spinning is hidden from the user-land code, but it's
91d632c867159b669d90fc7e172295433d0519efgwrthere nonetheless. This can result in the same load-spiking wasteful
91d632c867159b669d90fc7e172295433d0519efgwrbehaviour that a non-blocking solution to the multiple sockets case can.
91d632c867159b669d90fc7e172295433d0519efgwr<P>For this reason we have found that many architectures behave more
91d632c867159b669d90fc7e172295433d0519efgwr"nicely" if we serialize even the single socket case. So this is
91d632c867159b669d90fc7e172295433d0519efgwractually the default in almost all cases. Crude experiments under
91d632c867159b669d90fc7e172295433d0519efgwrLinux (2.0.30 on a dual Pentium pro 166 w/128Mb RAM) have shown that
91d632c867159b669d90fc7e172295433d0519efgwrthe serialization of the single socket case causes less than a 3%
91d632c867159b669d90fc7e172295433d0519efgwrdecrease in requests per second over unserialized single-socket.
91d632c867159b669d90fc7e172295433d0519efgwrBut unserialized single-socket showed an extra 100ms latency on
91d632c867159b669d90fc7e172295433d0519efgwreach request. This latency is probably a wash on long haul lines,
91d632c867159b669d90fc7e172295433d0519efgwrand only an issue on LANs. If you want to override the single socket
91d632c867159b669d90fc7e172295433d0519efgwrserialization you can define <CODE>SINGLE_LISTEN_UNSERIALIZED_ACCEPT</CODE>
91d632c867159b669d90fc7e172295433d0519efgwrand then single-socket servers will not serialize at all.
91d632c867159b669d90fc7e172295433d0519efgwr<P>As discussed in
91d632c867159b669d90fc7e172295433d0519efgwr HREF="ftp://ds.internic.net/internet-drafts/draft-ietf-http-connection-00.txt"
91d632c867159b669d90fc7e172295433d0519efgwrin order for an HTTP server to <STRONG>reliably</STRONG> implement the protocol
91d632c867159b669d90fc7e172295433d0519efgwrit needs to shutdown each direction of the communication independently
c1374a13e412c4ec42cba867e57347a0e049a822Surya Prakki(recall that a TCP connection is bi-directional, each half is independent
91d632c867159b669d90fc7e172295433d0519efgwrof the other). This fact is often overlooked by other servers, but
91d632c867159b669d90fc7e172295433d0519efgwris correctly implemented in Apache as of 1.2.
91d632c867159b669d90fc7e172295433d0519efgwr<P>When this feature was added to Apache it caused a flurry of
91d632c867159b669d90fc7e172295433d0519efgwrproblems on various versions of Unix because of a shortsightedness.
91d632c867159b669d90fc7e172295433d0519efgwrThe TCP specification does not state that the FIN_WAIT_2 state has a
91d632c867159b669d90fc7e172295433d0519efgwrtimeout, but it doesn't prohibit it. On systems without the timeout,
91d632c867159b669d90fc7e172295433d0519efgwrApache 1.2 induces many sockets stuck forever in the FIN_WAIT_2 state.
91d632c867159b669d90fc7e172295433d0519efgwrIn many cases this can be avoided by simply upgrading to the latest
91d632c867159b669d90fc7e172295433d0519efgwrTCP/IP patches supplied by the vendor, in cases where the vendor has
91d632c867159b669d90fc7e172295433d0519efgwrnever released patches (i.e. SunOS4 -- although folks with a source
91d632c867159b669d90fc7e172295433d0519efgwrlicense can patch it themselves) we have decided to disable this feature.
91d632c867159b669d90fc7e172295433d0519efgwr<P>There are two ways of accomplishing this. One is the
91d632c867159b669d90fc7e172295433d0519efgwrsocket option <CODE>SO_LINGER</CODE>. But as fate would have it,
91d632c867159b669d90fc7e172295433d0519efgwrthis has never been implemented properly in most TCP/IP stacks. Even
91d632c867159b669d90fc7e172295433d0519efgwron those stacks with a proper implementation (i.e. Linux 2.0.31) this
91d632c867159b669d90fc7e172295433d0519efgwrmethod proves to be more expensive (cputime) than the next solution.
91d632c867159b669d90fc7e172295433d0519efgwr<P>For the most part, Apache implements this in a function called
91d632c867159b669d90fc7e172295433d0519efgwr<CODE>lingering_close</CODE> (in <CODE>http_main.c</CODE>). The
91d632c867159b669d90fc7e172295433d0519efgwrfunction looks roughly like this:
91d632c867159b669d90fc7e172295433d0519efgwr void lingering_close (int s)
91d632c867159b669d90fc7e172295433d0519efgwr char junk_buffer[2048];
91d632c867159b669d90fc7e172295433d0519efgwr /* shutdown the sending side */
91d632c867159b669d90fc7e172295433d0519efgwr shutdown (s, 1);
91d632c867159b669d90fc7e172295433d0519efgwr signal (SIGALRM, lingering_death);
91d632c867159b669d90fc7e172295433d0519efgwr alarm (30);
91d632c867159b669d90fc7e172295433d0519efgwr for (;;) {
91d632c867159b669d90fc7e172295433d0519efgwr select (s for reading, 2 second timeout);
91d632c867159b669d90fc7e172295433d0519efgwr if (error) break;
91d632c867159b669d90fc7e172295433d0519efgwr if (s is ready for reading) {
91d632c867159b669d90fc7e172295433d0519efgwr read (s, junk_buffer, sizeof (junk_buffer));
91d632c867159b669d90fc7e172295433d0519efgwr /* just toss away whatever is here */
91d632c867159b669d90fc7e172295433d0519efgwr close (s);
91d632c867159b669d90fc7e172295433d0519efgwrThis naturally adds some expense at the end of a connection, but it
91d632c867159b669d90fc7e172295433d0519efgwris required for a reliable implementation. As HTTP/1.1 becomes more
91d632c867159b669d90fc7e172295433d0519efgwrprevalent, and all connections are persistent, this expense will be
91d632c867159b669d90fc7e172295433d0519efgwramortized over more requests. If you want to play with fire and
91d632c867159b669d90fc7e172295433d0519efgwrdisable this feature you can define <CODE>NO_LINGCLOSE</CODE>, but
91d632c867159b669d90fc7e172295433d0519efgwrthis is not recommended at all. In particular, as HTTP/1.1 pipelined
91d632c867159b669d90fc7e172295433d0519efgwrpersistent connections come into use <CODE>lingering_close</CODE>
91d632c867159b669d90fc7e172295433d0519efgwris an absolute necessity (and
91d632c867159b669d90fc7e172295433d0519efgwr<A HREF="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
getsockname(3, {sin_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
writev(3, [{"HTTP/1.1 200 OK\r\nDate: Thu, 11"..., 245}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 6144}], 2) = 6389
(i.e. due to a timeout or because of a maximum number of requests).
getsockname(3, {sin_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
writev(3, [{"HTTP/1.1 200 OK\r\nDate: Thu, 11"..., 245}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 6144}], 2) = 6389
network packets). On testing, various Unixes (BSDI 2.x, Solaris 2.5,
be atomic. (i.e. entries from multiple children could become mixed together).
getsockname(3, {sin_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0