1708N/A<
title>Running a High-Performance Web Server for BSD</
title>
1708N/A<
H2>Running a High-Performance Web Server for BSD</
H2>
1708N/ALike other OS's, the listen queue is often the <
b>first limit hit</
b>. The
1708N/Afollowing are comments from "Aaron Gifford <agifford@InfoWest.COM>"
1708N/Aon how to fix this on BSDI
1.x,
2.x, and FreeBSD 2.0 (and earlier):
1708N/AEdit the following two files:
1708N/AIn each file, look for the following:
1708N/A * Maximum queue length specifiable by listen.
1708N/AJust change the "5" to whatever appears to work. I bumped the two
1708N/Amachines I was having problems with up to 32 and haven't noticed the
1708N/AAfter the edit, recompile the kernel and recompile the Apache server
1708N/AFreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN
1708N/A<
b>Addendum for <
i>very</
i> heavily loaded BSD servers</
b><
br>
1708N/Afrom Chuck Murcko <chuck@telebase.com>
1708N/AIf you're running a really busy BSD Apache server, the following are useful
2630N/Athings to do if the system is acting sluggish:<
p>
1708N/A<
li> Run netstat -m to check mbuf usage
1708N/A<
li> Run fstat to check file descriptor usage
2630N/AThese utilities give you an idea what you'll need to tune in your kernel,
1708N/Aand whether it'll help to buy more RAM.
1708N/AHere are some BSD kernel config parameters (actually BSDI, but pertinent to
1708N/AFreeBSD and other 4.4-lite derivatives) from a system getting heavy usage.
2630N/AThe tools mentioned above were used, and the system memory was increased to
1708N/A48 MB before these tuneups. Other system parameters remained unchanged.
1708N/AMaxusers drives a <
i>lot</
i> of other kernel parameters:
1708N/A<
li> Maximum # of processes per user
1708N/A<
li> System wide open files limit
1708N/A<
li> Per-process open files limit
1708N/A<
li> Maximum # of mbuf clusters
1708N/AThe actual formulae for these derived parameters are in
1708N/AThese calculated parameters can also be overridden (in part) by specifying
1708N/Ayour own values in the kernel configuration file:
1708N/A# Network options. NMBCLUSTERS defines the number of mbuf clusters and
1708N/A# defaults to 256. This machine is a server that handles lots of traffic,
1708N/Aoptions SOMAXCONN=256 # max pending connects
1708N/Aoptions NMBCLUSTERS=4096 # mbuf clusters at 4096
2630N/Aoptions CHILD_MAX=512 # maximum number of child processes
2630N/Aoptions OPEN_MAX=512 # maximum fds (breaks RPC svcs)
2630N/ASOMAXCONN is not derived from maxusers, so you'll always need to increase
2630N/Athat yourself. We used a value guaranteed to be larger than Apache's
2630N/Adefault for the listen() of 128, currently.
2630N/AIn many cases, NMBCLUSTERS must be set much larger than would appear
1708N/Anecessary at first glance. The reason for this is that if the browser
1708N/Adisconnects in mid-transfer, the socket fd associated with that particular
1708N/Aconnection ends up in the TIME_WAIT state for several minutes, during
1708N/Awhich time its mbufs are not yet freed.
1708N/A * includes overhead. An mbuf may add a single "mbuf cluster" of size
1708N/A * and is used instead of the internal data area; this is done when
1708N/A * at least MINCLSIZE of data must be stored.
1708N/ACHILD_MAX and OPEN_MAX are set to allow up to 512 child processes (different
than the maximum value for processes per user ID) and file descriptors.
These values may change for your particular configuration (a higher OPEN_MAX
value if you've got modules or CGI scripts opening lots of connections or
files). If you've got a lot of other activity besides httpd on the same
machine, you'll have to set NPROC higher still. In this example, the NPROC
value derived from maxusers proved sufficient for our load.
Be aware that your system may not boot with a kernel that is configured
to use more resources than you have available system RAM. <
b>ALWAYS</
b>
have a known bootable kernel available when tuning your system this way,
and use the system tools beforehand to learn if you need to buy more
RPC services will fail when the value of OPEN_MAX is larger than 256.
This is a function of the original implementations of the RPC library,
which used a byte value for holding file descriptors. BSDI has partially
addressed this limit in its 2.1 release, but a real fix may well await
the redesign of RPC itself.
Finally, there's the hard limit of child processes configured in Apache.
For versions of Apache later than 1.0.5 you'll need to change the
definition for <
b>HARD_SERVER_LIMIT</
b> in <
i>
httpd.h</
i> and recompile
if you need to run more than the default 150 instances of httpd.
# Limit on total number of servers running,
i.e., limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# Unix with it as it spirals down...
Know what you're doing if you bump this value up, and make sure you've
done your system monitoring, RAM expansion, and kernel tuning beforehand.
Then you're ready to service some serious hits!
Thanks to <
i>Tony Sanders</
i> and <
i>Chris Torek</
i> at BSDI for their
helpful suggestions and information.
If you have tips to contribute, send mail to <
ahref="mailto:brian@organic.com">brian@organic.com</
a>
