perf-bsd44.html revision b71ea83f2a0b307ea3877b972d20e9150e406676
12b42c76672a66c2d4ea7212c14f8f1b5a62b78dTom Gundersen<title>Running a High-Performance Web Server for BSD</title>
0b3b020a178cf3b957fed627de13c895773995ecLennart Poettering<!--#include virtual="header.html" -->
0b3b020a178cf3b957fed627de13c895773995ecLennart Poettering<H2>Running a High-Performance Web Server for BSD</H2>
5430f7f2bc7330f3088b894166bf3524a067e3d8Lennart PoetteringLike other OS's, the listen queue is often the <b>first limit hit</b>. The
0b3b020a178cf3b957fed627de13c895773995ecLennart Poetteringfollowing are comments from "Aaron Gifford <agifford@InfoWest.COM>"
0b3b020a178cf3b957fed627de13c895773995ecLennart Poetteringon how to fix this on BSDI 1.x, 2.x, and FreeBSD 2.0 (and earlier):
5430f7f2bc7330f3088b894166bf3524a067e3d8Lennart PoetteringEdit the following two files:
0b3b020a178cf3b957fed627de13c895773995ecLennart Poettering<blockquote><code> /usr/include/sys/socket.h <br>
5430f7f2bc7330f3088b894166bf3524a067e3d8Lennart Poettering /usr/src/sys/sys/socket.h </code></blockquote>
0b3b020a178cf3b957fed627de13c895773995ecLennart PoetteringIn each file, look for the following:
56ba3c78ae35065064c4289a0c8e22a81256af20Zbigniew Jędrzejewski-Szmek * Maximum queue length specifiable by listen.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek #define SOMAXCONN 5
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekJust change the "5" to whatever appears to work. I bumped the two
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekmachines I was having problems with up to 32 and haven't noticed the
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekAfter the edit, recompile the kernel and recompile the Apache server
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekFreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekset to 32 already.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<b>Addendum for <i>very</i> heavily loaded BSD servers</b><br>
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringfrom Chuck Murcko <chuck@telebase.com>
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekIf you're running a really busy BSD Apache server, the following are useful
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekthings to do if the system is acting sluggish:<p>
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Run vmstat to check memory usage, page/swap rates, etc.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Run netstat -m to check mbuf usage
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Run fstat to check file descriptor usage
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThese utilities give you an idea what you'll need to tune in your kernel,
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekand whether it'll help to buy more RAM.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekHere are some BSD kernel config parameters (actually BSDI, but pertinent to
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekFreeBSD and other 4.4-lite derivatives) from a system getting heavy usage.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThe tools mentioned above were used, and the system memory was increased to
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek48 MB before these tuneups. Other system parameters remained unchanged.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekMaxusers drives a <i>lot</i> of other kernel parameters:
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Maximum # of processes
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Maximum # of processes per user
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> System wide open files limit
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Per-process open files limit
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<li> Maximum # of mbuf clusters
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThe actual formulae for these derived parameters are in
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<i>/usr/src/sys/conf/param.c</i>.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThese calculated parameters can also be overridden (in part) by specifying
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekyour own values in the kernel configuration file:
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# Network options. NMBCLUSTERS defines the number of mbuf clusters and
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# defaults to 256. This machine is a server that handles lots of traffic,
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# so we crank that value.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekoptions SOMAXCONN=256 # max pending connects
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekoptions NMBCLUSTERS=4096 # mbuf clusters at 4096
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekoptions CHILD_MAX=512 # maximum number of child processes
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekoptions OPEN_MAX=512 # maximum fds (breaks RPC svcs)
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekSOMAXCONN is not derived from maxusers, so you'll always need to increase
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekthat yourself. We used a value guaranteed to be larger than Apache's
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekdefault for the listen() of 128, currently.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekIn many cases, NMBCLUSTERS must be set much larger than would appear
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmeknecessary at first glance. The reason for this is that if the browser
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekdisconnects in mid-transfer, the socket fd associated with that particular
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekconnection ends up in the TIME_WAIT state for several minutes, during
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekwhich time its mbufs are not yet freed.
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart PoetteringSome more info on mbuf clusters (from sys/mbuf.h):
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poettering * Mbufs are of a single size, MSIZE (machine/machparam.h), which
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poettering * includes overhead. An mbuf may add a single "mbuf cluster" of size
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poettering * MCLBYTES (also in machine/machparam.h), which has no additional overhead
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poettering * and is used instead of the internal data area; this is done when
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poettering * at least MINCLSIZE of data must be stored.
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart PoetteringCHILD_MAX and OPEN_MAX are set to allow up to 512 child processes (different
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringthan the maximum value for processes per user ID) and file descriptors.
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart PoetteringThese values may change for your particular configuration (a higher OPEN_MAX
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringvalue if you've got modules or CGI scripts opening lots of connections or
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringfiles). If you've got a lot of other activity besides httpd on the same
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringmachine, you'll have to set NPROC higher still. In this example, the NPROC
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringvalue derived from maxusers proved sufficient for our load.
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart PoetteringBe aware that your system may not boot with a kernel that is configured
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringto use more resources than you have available system RAM. <b>ALWAYS</b>
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringhave a known bootable kernel available when tuning your system this way,
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringand use the system tools beforehand to learn if you need to buy more
4afd3348c7506dd1d36305b7bcb9feb8952b9d6bLennart Poetteringmemory before tuning.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekRPC services will fail when the value of OPEN_MAX is larger than 256.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThis is a function of the original implementations of the RPC library,
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekwhich used a byte value for holding file descriptors. BSDI has partially
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekaddressed this limit in its 2.1 release, but a real fix may well await
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekthe redesign of RPC itself.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekFinally, there's the hard limit of child processes configured in Apache.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekFor versions of Apache later than 1.0.5 you'll need to change the
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekdefinition for <b>HARD_SERVER_LIMIT</b> in <i>httpd.h</i> and recompile
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekif you need to run more than the default 150 instances of httpd.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# Limit on total number of servers running, i.e., limit on the number
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# of clients who can simultaneously connect --- if this limit is ever
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# It is intended mainly as a brake to keep a runaway server from taking
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek# Unix with it as it spirals down...
707b66c66381c899d7ef640e158ffdd5bcff4debLennart PoetteringKnow what you're doing if you bump this value up, and make sure you've
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmekdone your system monitoring, RAM expansion, and kernel tuning beforehand.
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-SzmekThen you're ready to service some serious hits!
667c24a6a86a5a26a906b7477ae81dcf4c73e64eLennart PoetteringThanks to <i>Tony Sanders</i> and <i>Chris Torek</i> at BSDI for their
707b66c66381c899d7ef640e158ffdd5bcff4debLennart Poetteringhelpful suggestions and information.
707b66c66381c899d7ef640e158ffdd5bcff4debLennart PoetteringIf you have tips to contribute, send mail to <a
667c24a6a86a5a26a906b7477ae81dcf4c73e64eLennart Poetteringhref="mailto:brian@organic.com">brian@organic.com</a>
798d3a524ea57aaf40cb53858aaa45ec702f012dZbigniew Jędrzejewski-Szmek<!--#include virtual="footer.html" -->