842ae4bd224140319ae7feec1872b93dfd491143fielding<?xml version="1.0" encoding="utf-8"?>

842ae4bd224140319ae7feec1872b93dfd491143fielding<!DOCTYPE manualpage SYSTEM "/style/manualpage.dtd">

842ae4bd224140319ae7feec1872b93dfd491143fielding<?xml-stylesheet type="text/xsl" href="/style/manual.en.xsl"?>

842ae4bd224140319ae7feec1872b93dfd491143fielding

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh<manualpage metafile="perf-scaling.xml.meta">

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <title>Performance Scaling

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </title>

2d2eda71267231c2526be701fe655db125852c1ffielding

2d2eda71267231c2526be701fe655db125852c1ffielding <summary>

2d2eda71267231c2526be701fe655db125852c1ffielding

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <p>The Performance Tuning page in the Apache 1.3 documentation says:

ab5581cc78e9d865b0a6ab1404c53347b3276968rbb </p>

92f3af936ce61f25358a3ee4f28df2f6d62040dfdreid <blockquote><p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb “Apache is a general webserver, which is designed to be

c9a95767fbf0f5fb0976a06b97a256033925e433rbb correct first, and fast

fd0edaa8e3d4dd67d0604ccef2e96b071db96643fielding second. Even so, its performance is quite satisfactory. Most

2d2eda71267231c2526be701fe655db125852c1ffielding sites have less than 10Mbits of outgoing bandwidth, which

2d2eda71267231c2526be701fe655db125852c1ffielding Apache can fill using only a low end Pentium-based

2d2eda71267231c2526be701fe655db125852c1ffielding webserver.”</p>

2d2eda71267231c2526be701fe655db125852c1ffielding </blockquote>

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj <p>However, this sentence was written a few years ago, and in the

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj meantime several things have happened. On one hand, web server

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj hardware has become much faster. On the other hand, many sites now

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj are allowed much more than ten megabits per second of outgoing

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj bandwidth. In addition, web applications have become more complex.

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj The classic brochureware site is alive and well, but the web has

52de7a47876ce1748910cf3a0ee97f78842fab54rederpj grown up substantially as a computing application platform and

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh webmasters may find themselves running dynamic content in Perl, PHP

fd492f9543f14fb5bae78e04b135c3448eb9cc56rbb or Java, all of which take a toll on performance.

fd492f9543f14fb5bae78e04b135c3448eb9cc56rbb </p>

fd492f9543f14fb5bae78e04b135c3448eb9cc56rbb <p>Therefore, in spite of strides forward in machine speed and

fd492f9543f14fb5bae78e04b135c3448eb9cc56rbb bandwidth allowances, web server performance and web application

2d2eda71267231c2526be701fe655db125852c1ffielding performance remain areas of concern. In this documentation several

2d2eda71267231c2526be701fe655db125852c1ffielding aspects of web server performance will be discussed.

2d2eda71267231c2526be701fe655db125852c1ffielding </p>

2d2eda71267231c2526be701fe655db125852c1ffielding

2d2eda71267231c2526be701fe655db125852c1ffielding </summary>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="what-will-and-will-not-be-discussed">

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <title>What Will and Will Not Be Discussed

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </title>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>The session will focus on easily accessible configuration and tuning

2d2eda71267231c2526be701fe655db125852c1ffielding options for Apache httpd 2.2 and 2.4 as well as monitoring tools.

2d2eda71267231c2526be701fe655db125852c1ffielding Monitoring tools will allow you to observe your web server to

8af88bd6958b80c224e964892b8237720b13ab1ajerenkrantz gather information about its performance, or lack thereof.

8af88bd6958b80c224e964892b8237720b13ab1ajerenkrantz We'll assume that you don't have an unlimited budget for

8af88bd6958b80c224e964892b8237720b13ab1ajerenkrantz server hardware, so the existing infrastructure will have to do the

8af88bd6958b80c224e964892b8237720b13ab1ajerenkrantz job. You have no desire to compile your own Apache, or to recompile

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp the operating system kernel. We do assume, though, that you have

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp some familiarity with the Apache httpd configuration file.

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </p>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </section>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <section id="monitoring-your-server">

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <title>Monitoring Your Server

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </title>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <p>The first task when sizing or performance-tuning your server is to

8af88bd6958b80c224e964892b8237720b13ab1ajerenkrantz find out how your system is currently performing. By monitoring

bfb62a96023822c56c9120e4ee627d4091cc59c2rbb your server under real-world load, or artificially generated load,

bfb62a96023822c56c9120e4ee627d4091cc59c2rbb you can extrapolate its behavior under stress, such as when your

61fd0cab072a05b855cbef9c585702401ac5ae29rbb site is mentioned on Slashdot.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="monitoring-tools">

3d96ee83babeec32482c9082c9426340cee8c44dwrowe <title>Monitoring Tools

2d2eda71267231c2526be701fe655db125852c1ffielding </title>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="top">

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <title>top

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </title>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>The top tool ships with Linux and FreeBSD. Solaris offers

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <code>prstat(1)</code>. It collects a number of statistics for the

61fd0cab072a05b855cbef9c585702401ac5ae29rbb system and for each running process, then displays them

2d2eda71267231c2526be701fe655db125852c1ffielding interactively on your terminal. The data displayed is

3d96ee83babeec32482c9082c9426340cee8c44dwrowe refreshed every second and varies by platform, but

2d2eda71267231c2526be701fe655db125852c1ffielding typically includes system load average, number of processes

2d2eda71267231c2526be701fe655db125852c1ffielding and their current states, the percent CPU(s) time spent

2d2eda71267231c2526be701fe655db125852c1ffielding executing user and system code, and the state of the

2d2eda71267231c2526be701fe655db125852c1ffielding virtual memory system. The data displayed for each process

000b67449410515eac43e76ef6667915bfd4d2abgstein is typically configurable and includes its process name and

2d2eda71267231c2526be701fe655db125852c1ffielding ID, priority and nice values, memory footprint, and

2d2eda71267231c2526be701fe655db125852c1ffielding percentage CPU usage. The following example shows multiple

2d2eda71267231c2526be701fe655db125852c1ffielding httpd processes (with MPM worker and event) running on an

61fd0cab072a05b855cbef9c585702401ac5ae29rbb Linux (Xen) system:

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <example><pre>

61fd0cab072a05b855cbef9c585702401ac5ae29rbbtop - 23:10:58 up 71 days, 6:14, 4 users, load average: 0.25, 0.53, 0.47

7bdef86e15d47d16dcbe7a5611683191774bd5fbgsteinTasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie

7bdef86e15d47d16dcbe7a5611683191774bd5fbgsteinCpu(s): 11.6%us, 0.7%sy, 0.0%ni, 87.3%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st

61fd0cab072a05b855cbef9c585702401ac5ae29rbbMem: 2621656k total, 2178684k used, 442972k free, 100500k buffers

61fd0cab072a05b855cbef9c585702401ac5ae29rbbSwap: 4194296k total, 860584k used, 3333712k free, 1157552k cached

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

61fd0cab072a05b855cbef9c585702401ac5ae29rbb16687 example_ 20 0 1200m 547m 179m S 45 21.4 1:09.59 httpd-worker

3d96ee83babeec32482c9082c9426340cee8c44dwrowe15195 www 20 0 441m 33m 2468 S 0 1.3 0:41.41 httpd-worker

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein 1 root 20 0 10312 328 308 S 0 0.0 0:33.17 init

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 3 root RT -5 0 0 0 S 0 0.0 0:00.14 migration/0

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 4 root 15 -5 0 0 0 S 0 0.0 0:04.58 ksoftirqd/0

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 5 root RT -5 0 0 0 S 0 0.0 4:45.89 watchdog/0

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 6 root 15 -5 0 0 0 S 0 0.0 1:42.52 events/0

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 7 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 19 root 15 -5 0 0 0 S 0 0.0 0:00.00 xenwatch

3d96ee83babeec32482c9082c9426340cee8c44dwrowe 20 root 15 -5 0 0 0 S 0 0.0 0:00.00 xenbus

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein 28 root RT -5 0 0 0 S 0 0.0 0:00.14 migration/1

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 29 root 15 -5 0 0 0 S 0 0.0 0:00.20 ksoftirqd/1

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 30 root RT -5 0 0 0 S 0 0.0 0:05.96 watchdog/1

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 31 root 15 -5 0 0 0 S 0 0.0 1:18.35 events/1

cf8d02ea0c91653917b044529f3133c5a1bb9200fielding 32 root RT -5 0 0 0 S 0 0.0 0:00.08 migration/2

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 33 root 15 -5 0 0 0 S 0 0.0 0:00.18 ksoftirqd/2

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 34 root RT -5 0 0 0 S 0 0.0 0:06.00 watchdog/2

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 35 root 15 -5 0 0 0 S 0 0.0 1:08.39 events/2

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 36 root RT -5 0 0 0 S 0 0.0 0:00.10 migration/3

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh 37 root 15 -5 0 0 0 S 0 0.0 0:00.16 ksoftirqd/3

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 38 root RT -5 0 0 0 S 0 0.0 0:06.08 watchdog/3

c9a95767fbf0f5fb0976a06b97a256033925e433rbb 39 root 15 -5 0 0 0 S 0 0.0 1:22.81 events/3

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp 68 root 15 -5 0 0 0 S 0 0.0 0:06.28 kblockd/0

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp 69 root 15 -5 0 0 0 S 0 0.0 0:00.04 kblockd/1

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp 70 root 15 -5 0 0 0 S 0 0.0 0:00.04 kblockd/2</pre></example>

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp <p>Top is a wonderful tool even though it’s slightly resource

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh intensive (when running, its own process is usually in the

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp top ten CPU gluttons). It is indispensable in determining

d82d78a97558238d16c52ec5278fe921bb7d7ec3brianp the size of a running process, which comes in handy when

c9a95767fbf0f5fb0976a06b97a256033925e433rbb determining how many server processes you can run on your

61fd0cab072a05b855cbef9c585702401ac5ae29rbb machine. How to do this is described in <a href="#sizing-maxClients">sizing MaxClients</a>.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb Top is, however, an interactive tool and running it

61fd0cab072a05b855cbef9c585702401ac5ae29rbb continuously has few if any advantages.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </section>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="free">

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <title>free

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </title>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe <p>This command is only available on Linux. It shows how much

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein memory and swap space is in use. Linux allocates unused

61fd0cab072a05b855cbef9c585702401ac5ae29rbb memory as file system cache. The free command shows usage

61fd0cab072a05b855cbef9c585702401ac5ae29rbb both with and without this cache. The free command can be

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh used to find out how much memory the operating system is

61fd0cab072a05b855cbef9c585702401ac5ae29rbb using, as described in the paragraph <a href="#sizing-maxClients">sizing MaxClients</a>.

3d96ee83babeec32482c9082c9426340cee8c44dwrowe The output of free looks like this:

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <example><pre>

61fd0cab072a05b855cbef9c585702401ac5ae29rbbsctemme@brutus:~$ free

61fd0cab072a05b855cbef9c585702401ac5ae29rbb total used free shared buffers cached

3d96ee83babeec32482c9082c9426340cee8c44dwroweMem: 4026028 3901892 124136 0 253144 841044

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein-/+ buffers/cache: 2807704 1218324

61fd0cab072a05b855cbef9c585702401ac5ae29rbbSwap: 3903784 12540 3891244

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </pre></example>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </section>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="vmstat">

78ae1cbba5a848a24ca551629bbf0a90d2b931f0stas <title>vmstat

78ae1cbba5a848a24ca551629bbf0a90d2b931f0stas </title>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>This command is available on many unix platforms. It

3d96ee83babeec32482c9082c9426340cee8c44dwrowe displays a large number of operating system metrics. Run

2d2eda71267231c2526be701fe655db125852c1ffielding without argument, it displays a status line for that

2d2eda71267231c2526be701fe655db125852c1ffielding moment. When a numeric argument is added, the status is

2d2eda71267231c2526be701fe655db125852c1ffielding redisplayed at designated intervals. For example,

2d2eda71267231c2526be701fe655db125852c1ffielding <code>vmstat 5</code>

2d2eda71267231c2526be701fe655db125852c1ffielding causes the information to reappear every five seconds.

2d2eda71267231c2526be701fe655db125852c1ffielding Vmstat displays the amount of virtual memory in use, how

2d2eda71267231c2526be701fe655db125852c1ffielding much memory is swapped in and out each second, the number

2d2eda71267231c2526be701fe655db125852c1ffielding of processes currently running and sleeping, the number of

2d2eda71267231c2526be701fe655db125852c1ffielding interrupts and context switches per second and the usage

2d2eda71267231c2526be701fe655db125852c1ffielding percentages of the CPU.

2d2eda71267231c2526be701fe655db125852c1ffielding </p>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb The following is <code>vmstat</code> output of an idle server:

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <example><pre>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb[sctemme@GayDeceiver sctemme]$ vmstat 5 3

61fd0cab072a05b855cbef9c585702401ac5ae29rbb procs memory swap io system cpu

61fd0cab072a05b855cbef9c585702401ac5ae29rbb r b w swpd free buff cache si so bi bo in cs us sy id

61fd0cab072a05b855cbef9c585702401ac5ae29rbb 0 0 0 0 186252 6688 37516 0 0 12 5 47 311 0 1 99

3d96ee83babeec32482c9082c9426340cee8c44dwrowe 0 0 0 0 186244 6696 37516 0 0 0 16 41 314 0 0 100

1ccd992d37d62c8cb2056126f2234f64ec189bfddougm 0 0 0 0 186236 6704 37516 0 0 0 9 44 314 0 0 100

7bdef86e15d47d16dcbe7a5611683191774bd5fbgstein </pre></example>

23ff73a56371e21f16744cb94d06399b877829f1rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>And this is output of a server that is under a load of one

61fd0cab072a05b855cbef9c585702401ac5ae29rbb hundred simultaneous connections fetching static content:

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <example><pre>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb[sctemme@GayDeceiver sctemme]$ vmstat 5 3

61fd0cab072a05b855cbef9c585702401ac5ae29rbb procs memory swap io system cpu

61fd0cab072a05b855cbef9c585702401ac5ae29rbb r b w swpd free buff cache si so bi bo in cs us sy id

3d96ee83babeec32482c9082c9426340cee8c44dwrowe 1 0 1 0 162580 6848 40056 0 0 11 5 150 324 1 1 98

2d2eda71267231c2526be701fe655db125852c1ffielding 6 0 1 0 163280 6856 40248 0 0 0 66 6384 1117 42 25 32

23ff73a56371e21f16744cb94d06399b877829f1rbb11 0 0 0 162780 6864 40436 0 0 0 61 6309 1165 33 28 40

2d2eda71267231c2526be701fe655db125852c1ffielding </pre></example>

62db15de4c1f335a64d45821796ae197cff94ef8rbb

62db15de4c1f335a64d45821796ae197cff94ef8rbb <p>The first line gives averages since the last reboot. The

62db15de4c1f335a64d45821796ae197cff94ef8rbb subsequent lines give information for five second

62db15de4c1f335a64d45821796ae197cff94ef8rbb intervals. The second argument tells vmstat to generate

62db15de4c1f335a64d45821796ae197cff94ef8rbb three reports and then exit.

62db15de4c1f335a64d45821796ae197cff94ef8rbb </p>

62db15de4c1f335a64d45821796ae197cff94ef8rbb

62db15de4c1f335a64d45821796ae197cff94ef8rbb

62db15de4c1f335a64d45821796ae197cff94ef8rbb </section>

48d2edbfb84e5559b5da0f8d614ccab805cc67a8rbb <section id="se-toolkit">

62db15de4c1f335a64d45821796ae197cff94ef8rbb <title>SE Toolkit

62db15de4c1f335a64d45821796ae197cff94ef8rbb </title>

62db15de4c1f335a64d45821796ae197cff94ef8rbb <p>The SE Toolkit is a system monitoring toolkit for Solaris.

62db15de4c1f335a64d45821796ae197cff94ef8rbb Its programming language is based on the C preprocessor and

62db15de4c1f335a64d45821796ae197cff94ef8rbb comes with a number of sample scripts. It can use both the

62db15de4c1f335a64d45821796ae197cff94ef8rbb command line and the GUI to display information. It can

62db15de4c1f335a64d45821796ae197cff94ef8rbb also be programmed to apply rules to the system data. The

62db15de4c1f335a64d45821796ae197cff94ef8rbb example script shown in Figure 2, Zoom.se, shows green,

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh orange or red indicators when utilization of various parts

62db15de4c1f335a64d45821796ae197cff94ef8rbb of the system rises above certain thresholds. Another

62db15de4c1f335a64d45821796ae197cff94ef8rbb included script, Virtual Adrian, applies performance tuning

62db15de4c1f335a64d45821796ae197cff94ef8rbb metrics according to.

48d2edbfb84e5559b5da0f8d614ccab805cc67a8rbb </p>

48d2edbfb84e5559b5da0f8d614ccab805cc67a8rbb <p>The SE Toolkit has drifted around for a while and has had

62db15de4c1f335a64d45821796ae197cff94ef8rbb several owners since its inception. It seems that it has

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar now found a final home at Sunfreeware.com, where it can be

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar downloaded at no charge. There is a single package for

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar Solaris 8, 9 and 10 on SPARC and x86, and includes source

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar code. SE Toolkit author Richard Pettit has started a new

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar company, Captive Metrics4 that plans to bring to market a

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar multiplatform monitoring tool built on the same principles

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar as SE Toolkit, written in Java.

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar </p>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

3d96ee83babeec32482c9082c9426340cee8c44dwrowe

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </section>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <section id="dtrace">

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <title>DTrace

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </title>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <p>Given that DTrace is available for Solaris, FreeBSD and OS

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh X, it might be worth exploring it. There's also

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh mod_dtrace available for httpd.

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </p>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

599c5abcc7fec60611956f56b732eca033c287d2pquerna </section>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <section id="mod_status">

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <title>mod_status

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar </title>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <p>The mod_status module gives an overview of the server

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar performance at a given moment. It generates an HTML page

41287245b1632a4722b55f6dfe5fcf81f361f043wrowe with, among others, the number of Apache processes running

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar and how many bytes each has served, and the CPU load caused

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar by httpd and the rest of the system. The Apache Software

3d96ee83babeec32482c9082c9426340cee8c44dwrowe Foundation uses <module>mod_status</module> on its own

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <a href="http://apache.org/server-status">web site</a>.

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar If you put the <code>ExtendedStatus On</code>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar directive in your <code>httpd.conf</code>,

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar the <module>mod_status</module>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar page will give you more information at the cost of a little

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar extra work per request.

41287245b1632a4722b55f6dfe5fcf81f361f043wrowe </p>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </section>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar </section>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <section id="web-server-log-files">

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <title>Web Server Log Files

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar </title>

41287245b1632a4722b55f6dfe5fcf81f361f043wrowe <p>Monitoring and analyzing the log files httpd writes is one of

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar the most effective ways to keep track of your server health and

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar performance. Monitoring the error log allows you to detect

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar error conditions, discover attacks and find performance issues.

3d96ee83babeec32482c9082c9426340cee8c44dwrowe Analyzing the access logs tells you how busy your server is,

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar which resources are the most popular and where your users come

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar from. Historical log file data can give you invaluable insight

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar into trends in access to your server, which allows you to

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar predict when your performance needs will overtake your server

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar capacity.

41287245b1632a4722b55f6dfe5fcf81f361f043wrowe </p>

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar

3d96ee83babeec32482c9082c9426340cee8c44dwrowe <section id="ErrorLog">

91f0d8da77152d24e4bbb31ce199282b3fd6e3b2coar <title>Error Log

2864362ca8266097928e84f101010bdf814ffa08stoddard </title>

db08da9ddcd65c31f9ea44b823898b72a1b24fbestoddard <p>The error log will contain messages if the server has

db08da9ddcd65c31f9ea44b823898b72a1b24fbestoddard reached the maximum number of active processes or the

bf648cbfa4a723c9c02fe8444103cd41ed8642d0wrowe maximum number of concurrently open files. The error log

bf648cbfa4a723c9c02fe8444103cd41ed8642d0wrowe also reflects when processes are being spawned at a

bf648cbfa4a723c9c02fe8444103cd41ed8642d0wrowe higher-than-usual rate in response to a sudden increase in

2864362ca8266097928e84f101010bdf814ffa08stoddard load. When the server starts, the stderr file descriptor is

db08da9ddcd65c31f9ea44b823898b72a1b24fbestoddard redirected to the error logfile, so any error encountered

2864362ca8266097928e84f101010bdf814ffa08stoddard by httpd after it opens its logfiles will appear in this

2d2eda71267231c2526be701fe655db125852c1ffielding log. This makes it good practice to review the error log

2d2eda71267231c2526be701fe655db125852c1ffielding frequently.

2d2eda71267231c2526be701fe655db125852c1ffielding </p>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>Before Apache httpd opens its logfiles, any errors will be

61fd0cab072a05b855cbef9c585702401ac5ae29rbb written to the stderr stream. If you start httpd manually,

61fd0cab072a05b855cbef9c585702401ac5ae29rbb this error information will appear on your terminal and you

61fd0cab072a05b855cbef9c585702401ac5ae29rbb can use it directly to troubleshoot your server. If your

61fd0cab072a05b855cbef9c585702401ac5ae29rbb httpd is started by a startup script, the destination of

61fd0cab072a05b855cbef9c585702401ac5ae29rbb early error messages depends on their design. The

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <code>/var/log/messages</code>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe file is usually a good bet. On Windows, early error

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein messages are written to the Applications Event Log, which

61fd0cab072a05b855cbef9c585702401ac5ae29rbb can be viewed through the Event Viewer in Administrative

b5c963a560a55d08763229330dc661b16b726da0sf Tools.

b5c963a560a55d08763229330dc661b16b726da0sf </p>

b5c963a560a55d08763229330dc661b16b726da0sf <p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb The Error Log is configured through the <directive module="core">ErrorLog</directive>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb and <directive module="core">LogLevel</directive>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb configuration directives. The error log of httpd’s main

b5c963a560a55d08763229330dc661b16b726da0sf server configuration receives the log messages that pertain

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein to the entire server: startup, shutdown, crashes, excessive

61fd0cab072a05b855cbef9c585702401ac5ae29rbb process spawns, etc. The <directive module="core">ErrorLog</directive>

b5c963a560a55d08763229330dc661b16b726da0sf directive can also be used in virtual host containers. The

b5c963a560a55d08763229330dc661b16b726da0sf error log of a virtual host receives only log messages

61fd0cab072a05b855cbef9c585702401ac5ae29rbb specific to that virtual host, such as authentication

61fd0cab072a05b855cbef9c585702401ac5ae29rbb failures and 'File not Found' errors.

b5c963a560a55d08763229330dc661b16b726da0sf </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>On a server that is visible to the Internet, expect to see a

b5c963a560a55d08763229330dc661b16b726da0sf lot of exploit attempt and worm attacks in the error log. A

b5c963a560a55d08763229330dc661b16b726da0sf lot of these will be targeted at other server platforms

b5c963a560a55d08763229330dc661b16b726da0sf instead of Apache, but the current state of affairs is that

b5c963a560a55d08763229330dc661b16b726da0sf attack scripts just throw everything they have at any open

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein port, regardless of which server is actually running or

61fd0cab072a05b855cbef9c585702401ac5ae29rbb what applications might be installed. You could block these

61fd0cab072a05b855cbef9c585702401ac5ae29rbb attempts using a firewall or <a href="http://www.modsecurity.org/">mod_security</a>,

61fd0cab072a05b855cbef9c585702401ac5ae29rbb but this falls outside the scope of this discussion.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb The <directive module="core">LogLevel</directive>

64945a581b504264270920efbad4f6af0d5a1978sf directive determines the level of detail included in the

64945a581b504264270920efbad4f6af0d5a1978sf logs. There are eight log levels as described here:

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <table>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p><strong>Level</strong></p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p><strong>Description</strong></p>

aecb17a45c6d3ee4729ed5f68dc4270f211ee7a8fielding </td>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein </tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>emerg</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Emergencies - system is unusable.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

aecb17a45c6d3ee4729ed5f68dc4270f211ee7a8fielding </tr>

2d2eda71267231c2526be701fe655db125852c1ffielding <tr>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>alert</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Action must be taken immediately.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </tr>

2d2eda71267231c2526be701fe655db125852c1ffielding <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>crit</p>

2d2eda71267231c2526be701fe655db125852c1ffielding </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Critical Conditions.</p>

2d2eda71267231c2526be701fe655db125852c1ffielding </td>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </tr>

2d2eda71267231c2526be701fe655db125852c1ffielding <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

d839a9822ee53ce00da24c15f2d9fe054233d342gstein <p>error</p>

d839a9822ee53ce00da24c15f2d9fe054233d342gstein </td>

d839a9822ee53ce00da24c15f2d9fe054233d342gstein <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Error conditions.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

d839a9822ee53ce00da24c15f2d9fe054233d342gstein </tr>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe <tr>

d839a9822ee53ce00da24c15f2d9fe054233d342gstein <td>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>warn</p>

2d2eda71267231c2526be701fe655db125852c1ffielding </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Warning conditions.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </tr>

346029f34d03eb20d84fc35664426d3874b00f9ewrowe <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>notice</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

346029f34d03eb20d84fc35664426d3874b00f9ewrowe <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Normal but significant condition.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </tr>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>info</p>

face099b681c052cb1f52176a5499661c44b2982trawick </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Informational.</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe <td>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein <p>debug</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Debug-level messages</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </table>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>The default log level is warn. A production server should

61fd0cab072a05b855cbef9c585702401ac5ae29rbb not be run on debug, but increasing the level of detail in

61fd0cab072a05b855cbef9c585702401ac5ae29rbb the error log can be useful during troubleshooting.

aecb17a45c6d3ee4729ed5f68dc4270f211ee7a8fielding Starting with 2.3.8 <directive module="core">LogLevel</directive>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein can be specified on a per module basis:

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <highlight language="config">

61fd0cab072a05b855cbef9c585702401ac5ae29rbb LogLevel debug mod_ssl:warn

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </highlight>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb This puts all of the server in debug mode, except for

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <module>mod_ssl</module>, which tends to be very noisy.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe

2d2eda71267231c2526be701fe655db125852c1ffielding

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </section>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <section id="AccessLog">

a44d29a3794110c558c940bd903a1930d717a7d7sf <title>Access Log

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </title>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Apache httpd keeps track of every request it services in its

3d96ee83babeec32482c9082c9426340cee8c44dwrowe access log file. In addition to the time and nature of a

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein request, httpd can log the client IP address, date and time

61fd0cab072a05b855cbef9c585702401ac5ae29rbb of the request, the result and a host of other information.

a44d29a3794110c558c940bd903a1930d717a7d7sf The various logging format features are documented in the

61fd0cab072a05b855cbef9c585702401ac5ae29rbb manual. This file exists by default for the main server and can be

3d96ee83babeec32482c9082c9426340cee8c44dwrowe configured per virtual host by using the <directive module="mod_log_config">TransferLog</directive>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein or <directive module="mod_log_config">CustomLog</directive>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb configuration directive.

a44d29a3794110c558c940bd903a1930d717a7d7sf </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>The access logs can be analyzed with any of several free and

3d96ee83babeec32482c9082c9426340cee8c44dwrowe commercially available programs. Popular free analysis

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein packages include Analog and Webalizer. Log analysis should

a44d29a3794110c558c940bd903a1930d717a7d7sf be done offline so the web server machine is not burdened

a44d29a3794110c558c940bd903a1930d717a7d7sf by processing the log files. Most log analysis packages

a44d29a3794110c558c940bd903a1930d717a7d7sf understand the Common Log Format. The fields in the log

a44d29a3794110c558c940bd903a1930d717a7d7sf lines are explained in in the following:

a44d29a3794110c558c940bd903a1930d717a7d7sf </p>

a44d29a3794110c558c940bd903a1930d717a7d7sf

a44d29a3794110c558c940bd903a1930d717a7d7sf

a44d29a3794110c558c940bd903a1930d717a7d7sf <example><pre>

a44d29a3794110c558c940bd903a1930d717a7d7sf195.54.228.42 - - [24/Mar/2007:23:05:11 -0400] "GET /sander/feed/ HTTP/1.1" 200 9747

61fd0cab072a05b855cbef9c585702401ac5ae29rbb64.34.165.214 - - [24/Mar/2007:23:10:11 -0400] "GET /sander/feed/atom HTTP/1.1" 200 9068

61fd0cab072a05b855cbef9c585702401ac5ae29rbb60.28.164.72 - - [24/Mar/2007:23:11:41 -0400] "GET / HTTP/1.0" 200 618

61fd0cab072a05b855cbef9c585702401ac5ae29rbb85.140.155.56 - - [24/Mar/2007:23:14:12 -0400] "GET /sander/2006/09/27/44/ HTTP/1.1" 200 14172

61fd0cab072a05b855cbef9c585702401ac5ae29rbb85.140.155.56 - - [24/Mar/2007:23:14:15 -0400] "GET /sander/2006/09/21/gore-tax-pollution/ HTTP/1.1" 200 15147

61fd0cab072a05b855cbef9c585702401ac5ae29rbb74.6.72.187 - - [24/Mar/2007:23:18:11 -0400] "GET /sander/2006/09/27/44/ HTTP/1.0" 200 14172

94d240fc213d4a96160d7014633374cc22a204d8trawick74.6.72.229 - - [24/Mar/2007:23:24:22 -0400] "GET /sander/2006/11/21/os-java/ HTTP/1.0" 200 13457

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </pre></example>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <table>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p><strong>Field</strong></p>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </td>

2d2eda71267231c2526be701fe655db125852c1ffielding <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p><strong>Content</strong></p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <td>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <p><strong>Explanation</strong></p>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </td>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>Client IP</p>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </td>

2d2eda71267231c2526be701fe655db125852c1ffielding <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>195.54.228.42</p>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>IP address where the request originated</p>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </tr>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <tr>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>RFC 1413 ident</p>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </td>

aa4af3da5c68fe5dbd9d2c3fd7b2fe7103daa2b7dougm <td>

e351a4349a3dcc2e8d9c27bcdf72414bdde0942frbb <p>-</p>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <p>Remote user identity as reported by their identd</p>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </tr>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <tr>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <p>username</p>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>-</p>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>Remote username as authenticated by Apache</p>

45613d36b9466a48def0498cffa07f48980720f8jerenkrantz </td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp </tr>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <tr>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <td>

45613d36b9466a48def0498cffa07f48980720f8jerenkrantz <p>timestamp</p>

45613d36b9466a48def0498cffa07f48980720f8jerenkrantz </td>

45613d36b9466a48def0498cffa07f48980720f8jerenkrantz <td>

fd8b91502bc200ed4cca3810560a2a570522b3debrianp <p>[24/Mar/2007:23:05:11 -0400]</p>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <p>Date and time of request</p>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </td>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </tr>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <tr>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <td>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <p>Request</p>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </td>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <td>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <p>&quot;GET /sander/feed/ HTTP/1.1&quot;</p>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames </td>

4111de96e9f75c58e77c2bdda23be83b8ebf81ccgregames <td>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp <p>Request line</p>

50e60f30bdc074fbc887f0b98f4d570457ac97c9brianp </td>

5a9667916c79d8c699b069068e5570aa1c331c80gstein </tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <tr>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>Status Code</p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </td>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <td>

2d2eda71267231c2526be701fe655db125852c1ffielding <p>200</p>

3d96ee83babeec32482c9082c9426340cee8c44dwrowe </td>

2d2eda71267231c2526be701fe655db125852c1ffielding <td>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar <p>Response code</p>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar </td>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar </tr>

5a9667916c79d8c699b069068e5570aa1c331c80gstein <tr>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar <td>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar <p>Content Bytes</p>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar </td>

5a9667916c79d8c699b069068e5570aa1c331c80gstein <td>

344f3bc38dfccf6261d5bb8d689794cde113b3d6coar <p>9747</p>

d5df46b7972c0c4a5ca0ba5068e238f6053c2e6ftrawick </td>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim <td>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim <p>Bytes transferred w/o headers</p>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim </td>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim </tr>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim </table>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim </section>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim <section id="rotating-log-files">

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim <title>Rotating Log Files

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim </title>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim <p>There are several reasons to rotate logfiles. Even though

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim almost no operating systems out there have a hard file size

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim limit of two Gigabytes anymore, log files simply become too

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim large to handle over time. Additionally, any periodic log

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim file analysis should not be performed on files to which the

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim server is actively writing. Periodic logfile rotation helps

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim keep the analysis job manageable, and allows you to keep a

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim closer eye on usage trends.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb </p>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <p>On unix systems, you can simply rotate logfiles by giving

61fd0cab072a05b855cbef9c585702401ac5ae29rbb the old file a new name using mv. The server will keep

61fd0cab072a05b855cbef9c585702401ac5ae29rbb writing to the open file even though it has a new name.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb When you send a graceful restart signal to the server, it

61fd0cab072a05b855cbef9c585702401ac5ae29rbb will open a new logfile with the configured name. For

61fd0cab072a05b855cbef9c585702401ac5ae29rbb example, you could run a script from cron like this:

42ce672c516baf6e4eaed18ccc1647de2d456d8edougm </p>

a2930b0b01bfdbaa9e5c3324266bce9ef74158c2jim

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

61fd0cab072a05b855cbef9c585702401ac5ae29rbb <example>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb APACHE=/usr/local/apache2<br />

61fd0cab072a05b855cbef9c585702401ac5ae29rbb HTTPD=$APACHE/bin/httpd<br />

61fd0cab072a05b855cbef9c585702401ac5ae29rbb mv $APACHE/logs/access_log

61fd0cab072a05b855cbef9c585702401ac5ae29rbb $APACHE/logarchive/access_log-‘date +%F‘<br />

42ce672c516baf6e4eaed18ccc1647de2d456d8edougm $HTTPD -k graceful

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein </example>

61fd0cab072a05b855cbef9c585702401ac5ae29rbb

7b6ba9c468f26bdb3492d5e8cb79628a3b04e8c8wrowe <p>This approach also works on Windows, just not as smoothly.

7b6ba9c468f26bdb3492d5e8cb79628a3b04e8c8wrowe While the httpd process on your Windows server will keep

61fd0cab072a05b855cbef9c585702401ac5ae29rbb writing to the log file after it has been renamed, the

7b6ba9c468f26bdb3492d5e8cb79628a3b04e8c8wrowe Windows Service that runs Apache can not do a graceful

61fd0cab072a05b855cbef9c585702401ac5ae29rbb restart. Restarting a Service on Windows means stopping it

7b6ba9c468f26bdb3492d5e8cb79628a3b04e8c8wrowe and then starting it again. The advantage of a graceful

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein restart is that the httpd child processes get to complete

61fd0cab072a05b855cbef9c585702401ac5ae29rbb responding to their current requests before they exit.

61fd0cab072a05b855cbef9c585702401ac5ae29rbb Meanwhile, the httpd server becomes immediately available

61fd0cab072a05b855cbef9c585702401ac5ae29rbb again to serve new requests. The stop-start that the

61fd0cab072a05b855cbef9c585702401ac5ae29rbb Windows Service has to perform will interrupt any requests

61fd0cab072a05b855cbef9c585702401ac5ae29rbb currently in progress, and the server is unavailable until

42ce672c516baf6e4eaed18ccc1647de2d456d8edougm it is started again. Plan for this when you decide the

fd0edaa8e3d4dd67d0604ccef2e96b071db96643fielding timing of your restarts.

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <p>

a6b9ed64fdf548c61de9714e2cfb999ec59d149cgstein A second approach is to use piped logs. From the

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <directive module="mod_log_config">CustomLog</directive>,

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <directive module="mod_log_config">TransferLog</directive>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh or <directive module="core">ErrorLog

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </directive>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb directives you can send the log data into any program using

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb a pipe character (<code>|</code>). For instance:

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb </p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <example>CustomLog "|/usr/local/apache2/bin/rotatelogs

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb /var/log/access_log 86400" common

0de8f88b22d24dd57ed10b96fb594e69b6aa0072jwoolley </example>

0de8f88b22d24dd57ed10b96fb594e69b6aa0072jwoolley

8b7047e519340545e6807c9749576a40a76b6d3frbb <p>The program on the other end of the pipe will receive the

8b7047e519340545e6807c9749576a40a76b6d3frbb Apache log data on its stdin stream, and can do with this

8b7047e519340545e6807c9749576a40a76b6d3frbb data whatever it wants. The rotatelogs program that comes

6d00a43be9ab145c89e8cc22bad59e3aa746f761jwoolley with Apache seamlessly turns over the log file based on

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb time elapsed or the amount of data written, and leaves the

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb old log files with a timestamp suffix to its name. This

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh method for rotating logfiles works well on unix platforms,

9484ae61c7cc5fa8d8d9a836efdbdb1e88d3036dwrowe but is currently broken on Windows.

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb </p>

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz </section>

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz <section id="logging-and-performance">

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz <title>Logging and Performance

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz </title>

2e75499d05677dd589f7ce6b5139ae15ba049419jerenkrantz <p>Writing entries to the Apache log files obviously takes some

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb effort, but the information gathered from the logs is so

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb valuable that under normal circumstances logging should not

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb be turned off. For optimal performance, you should put your

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb disk-based site content on a different physical disk than

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb the server log files: the access patterns are very

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb different. Retrieving content from disk is a read operation

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb in a fairly random pattern, and log files are written to

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb disk sequentially.

561e5f16a2f9fb397aac4c283aaa87a752520a4ddougm </p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb Do not run a production server with your error <directive module="core">LogLevel</directive>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb set to debug. This log level causes a vast amount of

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb information to be written to the error log, including, in

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb the case of SSL access, complete dumps of BIO read and

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb write operations. The performance implications are

2fc50921b88defeb7127985dfe4b4130175e069ejwoolley significant: use the default warn level instead.

2fc50921b88defeb7127985dfe4b4130175e069ejwoolley </p>

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb <p>If your server has more than one virtual host, you may give

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb each virtual host a separate access logfile. This makes it

2fc50921b88defeb7127985dfe4b4130175e069ejwoolley easier to analyze the logfile later. However, if your

2fc50921b88defeb7127985dfe4b4130175e069ejwoolley server has many virtual hosts, all the open logfiles put a

2fc50921b88defeb7127985dfe4b4130175e069ejwoolley resource burden on your system, and it may be preferable to

fcc25eda7b150e226d3c1cdaea66a943d3fdee4erbb log to a single file. Use the <code>%v</code>

c9a95767fbf0f5fb0976a06b97a256033925e433rbb format character at the start of your <directive module="mod_log_config">LogFormat</directive>

c9a95767fbf0f5fb0976a06b97a256033925e433rbb and starting 2.3.8 of your <directive module="core">ErrorLog</directive>

c9a95767fbf0f5fb0976a06b97a256033925e433rbb to make httpd print the hostname of the virtual host that

c9a95767fbf0f5fb0976a06b97a256033925e433rbb received the request or the error at the beginning of each

c9a95767fbf0f5fb0976a06b97a256033925e433rbb log line. A simple Perl script can split out the log file

c9a95767fbf0f5fb0976a06b97a256033925e433rbb after it rotates: one is included with the Apache source

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh under <code>support/split-logfile</code>.

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </p>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh <p>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh You can use the <directive module="mod_log_config">BufferedLogs</directive>

c9a95767fbf0f5fb0976a06b97a256033925e433rbb directive to have Apache collect several log lines in

3311183b8638d2ba2ff8496cad5e01c27d329a05nd memory before writing them to disk. This might yield better

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh performance, but could affect the order in which the

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh server's log is written.

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </p>

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh </section>

3311183b8638d2ba2ff8496cad5e01c27d329a05nd </section>

f5abdb59b6579717b5de18de9de340f4b67d6f26niq <section id="generating-a-test-load">

f5abdb59b6579717b5de18de9de340f4b67d6f26niq <title>Generating A Test Load

f5abdb59b6579717b5de18de9de340f4b67d6f26niq </title>

f5abdb59b6579717b5de18de9de340f4b67d6f26niq <p>It is useful to generate a test load to monitor system

3347ef7496eff25aaca86b2852a14bab667d0e34niq performance under realistic operating circumstances. Besides

f5abdb59b6579717b5de18de9de340f4b67d6f26niq commercial packages such as <a href="http://learnloadrunner.com/">LoadRunner</a>

3347ef7496eff25aaca86b2852a14bab667d0e34niq ,there are a number of freely available tools to generate a

c2213b3a46a2666e2e7606ceec509cc4978f187fminfrin test load against your web server.

2d2eda71267231c2526be701fe655db125852c1ffielding </p>

2d2eda71267231c2526be701fe655db125852c1ffielding <ul>

2d2eda71267231c2526be701fe655db125852c1ffielding <li>Apache ships with a test program called ab, short for

2d2eda71267231c2526be701fe655db125852c1ffielding Apache Bench. It can generate a web server load by

2d2eda71267231c2526be701fe655db125852c1ffielding repeatedly asking for the same file in rapid succession.

9d129b55f5a43abf43865c6b0eb6dd19bc22aba8ianh You can specify a number of concurrent connections and have

the program run for either a given amount of time or a

specified number of requests.

</li>

<li>Another freely available load generator is http load11 .

This program works with a URL file and can be compiled with

SSL support.

</li>

<li>The Apache Software Foundation offers a tool named flood12

. Flood is a fairly sophisticated program that is

configured through an XML file.

</li>

<li>Finally, JMeter13 , a Jakarta subproject, is an all-Java

load-testing tool. While early versions of this application

were slow and difficult to use, the current version 2.1.1

seems to be versatile and useful.

</li>

<li>

<p>ASF external projects, that have proven to be quite

good: grinder, httperf, tsung, <a href="http://funkload.nuxeo.org/">FunkLoad</a>

</p>

</li>

</ul>

<p>When you load-test your web server, please keep in mind that if

that server is in production, the test load may negatively

affect the server’s response. Also, any data traffic you

generate may be charged against your monthly traffic allowance.

</p>

</section>

</section>

<section id="configuring-for-performance">

<title>Configuring for Performance

</title>

<section id="apache-configuration">

<title>Httpd Configuration

</title>

<p>The Apache 2.2 httpd is by default a pre-forking web server.

When the server starts, the parent process spawns a number of

child processes that do the actual work of servicing requests.

But Apache httpd 2.0 introduced the concept of the

Multi-Processing Module (MPM). Developers can write MPMs to

suit the process- or threadingarchitecture of their specific

operating system. Apache 2 comes with special MPMs for Windows,

OS/2, Netware and BeOS. On unix-like platforms, the two most

popular MPMs are Prefork and Worker. The Prefork MPM offers the

same pre-forking process model that Apache 1.3 uses. The Worker

MPM runs a smaller number of child processes, and spawns

multiple request handling threads within each child process. In

2.4 MPMs are no longer hard-wired. They too can be exchanged

via <directive module="core">LoadModule</directive>.

The default MPM in 2.4 is the event MPM.

</p>

<p>The maximum number of workers, be they pre-forked child

processes or threads within a process, is an indication of how

many requests your server can manage concurrently. It is merely

a rough estimate because the kernel can queue connection

attempts for your web server. When your site becomes busy and

the maximum number of workers is running, the machine

doesn't hit a hard limit beyond which clients will be

denied access. However, once requests start backing up, system

performance is likely to degrade.

</p>

<p>Finally, if the httpd server in question is not executing any third-party

code, via <code>mod_php</code>, <code>mod_perl</code> or similar,

we recommend the use of <module>mpm_event</module>. This MPM is ideal

for situations where httpd serves as a thin layer between clients and

backend servers doing the real job, such as a proxy or cache.

</p>

<section id="MaxClients">

<title>MaxClients

</title>

<p>

The <code>MaxClients

</code>

directive in your Apache httpd configuration file specifies

the maximum number of workers your server can create. It

has two related directives, <code>MinSpareServers

</code>

and <code>MaxSpareServers

</code>

,which specify the number of workers Apache keeps waiting

in the wings ready to serve requests. The absolute maximum

number of processes is configurable through the <code>

ServerLimit

</code>

directive.

</p>

</section>

<section id="spinning-threads">

<title>Spinning Threads

</title>

<p>For the prefork MPM of the above directives are all there is

to determining the process limit. However, if you are

running a threaded MPM the situation is a little more

complicated. Threaded MPMs support the <code>

ThreadsPerChild

</code>

directive1 . Apache requires that <code>MaxClients

</code>

is evenly divisible by <code>ThreadsPerChild

</code>

.If you set either directive to a number that doesn’t

meet this requirement, Apache will send a message of

complaint to the error log and adjust the <code>

ThreadsPerChild

</code>

value downwards until it is an even factor of <code>

MaxClients

</code>

.

</p>

</section>

<section id="sizing-maxClients">

<title>Sizing MaxClients

</title>

<p>Optimally, the maximum number of processes should be set so

that all the memory on your system is used, but no more. If

your system gets so overloaded that it needs to heavily

swap core memory out to disk, performance will degrade

quickly. The formula for determining <directive module="mpm_common">MaxClients</directive>

is fairly simple:

</p>

<example>

total RAM − RAM for OS − RAM for external programs<br />

MaxClients =

-------------------------------------------------------<br />

RAM per httpd process

</example>

<p>The various amounts of memory allocated for the OS, external

programs and the httpd processes is best determined by

observation: use the top and free commands described above

to determine the memory footprint of the OS without the web

server running. You can also determine the footprint of a

typical web server process from top: most top

implementations have a Resident Size (RSS) column and a

Shared Memory column.

</p>

<p>The difference between these two is the amount of memory

per-process. The shared segment really exists only once and

is used for the code and libraries loaded and the dynamic

inter-process tally, or 'scoreboard,' that Apache

keeps. How much memory each process takes for itself

depends heavily on the number and kind of modules you use.

The best approach to use in determining this need is to

generate a typical test load against your web site and see

how large the httpd processes become.

</p>

<p>The RAM for external programs parameter is intended mostly

for CGI programs and scripts that run outside the web

server process. However, if you have a Java virtual machine

running Tomcat on the same box it will need a significant

amount of memory as well. The above assessment should give

you an idea how far you can push <code>MaxClients

</code>

,but it is not an exact science. When in doubt, be

conservative and use a low <code>MaxClients

</code>

value. The Linux kernel will put extra memory to good use

for caching disk access. On Solaris you need enough

available real RAM memory to create any process. If no real

memory is available, httpd will start writing ‘No space

left on device’ messages to the error log and be unable

to fork additional child processes, so a higher <code>

MaxClients

</code>

value may actually be a disadvantage.

</p>

</section>

<section id="selecting-your-mpm">

<title>Selecting your MPM

</title>

<p>The prime reason for selecting a threaded MPM is that

threads consume fewer system resources than processes, and

it takes less effort for the system to switch between

threads. This is more true for some operating systems than

for others. On systems like Solaris and AIX, manipulating

processes is relatively expensive in terms of system

resources. On these systems, running a threaded MPM makes

sense. On Linux, the threading implementation actually uses

one process for each thread. Linux processes are relatively

lightweight, but it means that a threaded MPM offers less

of a performance advantage than in other environments.

</p>

<p>Running a threaded MPM can cause stability problems in some

situations For instance, should a child process of a

preforked MPM crash, at most one client connection is

affected. However, if a threaded child crashes, all the

threads in that process disappear, which means all the

clients currently being served by that process will see

their connection aborted. Additionally, there may be

so-called &quot;thread-safety&quot; issues, especially with

third-party libraries. In threaded applications, threads

may access the same variables indiscriminately, not knowing

whether a variable may have been changed by another thread.

</p>

<p>This has been a sore point within the PHP community. The PHP

processor heavily relies on third-party libraries and

cannot guarantee that all of these are thread-safe. The

good news is that if you are running Apache on Linux, you

can run PHP in the preforked MPM without fear of losing too

much performance relative to the threaded option.

</p>

</section>

<section id="spinning-locks">

<title>Spinning Locks

</title>

<p>Apache httpd maintains an inter-process lock around its

network listener. For all practical purposes, this means

that only one httpd child process can receive a request at

any given time. The other processes are either servicing

requests already received or are 'camping out' on

the lock, waiting for the network listener to become

available. This process is best visualized as a revolving

door, with only one process allowed in the door at any

time. On a heavily loaded web server with requests arriving

constantly, the door spins quickly and requests are

accepted at a steady rate. On a lightly loaded web server,

the process that currently &quot;holds&quot; the lock may

have to stay in the door for a while, during which all the

other processes sit idle, waiting to acquire the lock. At

this time, the parent process may decide to terminate some

children based on its <code>MaxSpareServers

</code>

directive.

</p>

</section>

<section id="the-thundering-herd">

<title>The Thundering Herd

</title>

<p>The function of the 'accept mutex' (as this

inter-process lock is called) is to keep request reception

moving along in an orderly fashion. If the lock is absent,

the server may exhibit the Thundering Herd syndrome.

</p>

<p>Consider an American Football team poised on the line of

scrimmage. If the football players were Apache processes

all team members would go for the ball simultaneously at

the snap. One process would get it, and all the others

would have to lumber back to the line for the next snap. In

this metaphor, the accept mutex acts as the quarterback,

delivering the connection &quot;ball&quot; to the

appropriate player process.

</p>

<p>Moving this much information around is obviously a lot of

work, and, like a smart person, a smart web server tries to

avoid it whenever possible. Hence the revolving door

construction. In recent years, many operating systems,

including Linux and Solaris, have put code in place to

prevent the Thundering Herd syndrome. Apache recognizes

this and if you run with just one network listener, meaning

one virtual host or just the main server, Apache will

refrain from using an accept mutex. If you run with

multiple listeners (for instance because you have a virtual

host serving SSL requests), it will activate the accept

mutex to avoid internal conflicts.

</p>

<p>

You can manipulate the accept mutex with the <code>

AcceptMutex

</code>

directive. Besides turning the accept mutex off, you can

select the locking mechanism. Common locking mechanisms

include fcntl, System V Semaphores and pthread locking. Not

all are available on every platform, and their availability

also depends on compile-time settings. The various locking

mechanisms may place specific demands on system resources:

manipulate them with care.

</p>

<p>There is no compelling reason to disable the accept mutex.

Apache automatically recognizes the single listener

situation described above and knows if it is safe to run

without mutex on your platform.

</p>

</section>

</section>

<section id="tuning-the-operating-system">

<title>Tuning the Operating System

</title>

<p>People often look for the 'magic tune-up' that will

make their system perform four times as fast by tweaking just

one little setting. The truth is, present-day UNIX derivatives

are pretty well adjusted straight out of the box and there is

not a lot that needs to be done to make them perform optimally.

However, there are a few things that an administrator can do to

improve performance.

</p>

<section id="ram-and-swap-space">

<title>RAM and Swap Space

</title>

<p>The usual mantra regarding RAM is &quot;more is

better&quot;. As discussed above, unused RAM is put to good

use as file system cache. The Apache processes get bigger

if you load more modules, especially if you use modules

that generate dynamic page content within the processes,

like PHP and mod_perl. A large configuration file-with many

virtual hosts-also tends to inflate the process footprint.

Having ample RAM allows you to run Apache with more child

processes, which allows the server to process more

concurrent requests.

</p>

<p>While the various platforms treat their virtual memory in

different ways, it is never a good idea to run with less

disk-based swap space than RAM. The virtual memory system

is designed to provide a fallback for RAM, but when you

don't have disk space available and run out of

swappable memory, your machine grinds to a halt. This can

crash your box, requiring a physical reboot for which your

hosting facility may charge you.

</p>

<p>Also, such an outage naturally occurs when you least want

it: when the world has found your website and is beating a

path to your door. If you have enough disk-based swap space

available and the machine gets overloaded, it may get very,

very slow as the system needs to swap memory pages to disk

and back, but when the load decreases the system should

recover. Remember, you still have <code>MaxClients

</code>

to keep things in hand.

</p>

<p>Most unix-like operating systems use designated disk

partitions for swap space. When a system starts up it finds

all swap partitions on the disk(s), by partition type or

because they are listed in the file <code>/etc/fstab

</code>

,and automatically enables them. When adding a disk or

installing the operating system, be sure to allocate enough

swap space to accommodate eventual RAM upgrades.

Reassigning disk space on a running system is a cumbersome

process.

</p>

<p>Plan for available hard drive swap space of at least twice

your amount of RAM, perhaps up to four times in situations

with frequent peaking loads. Remember to adjust this

configuration whenever you upgrade RAM on your system. In a

pinch, you can use a regular file as swap space. For

instructions on how to do this, see the manual pages for

the <code>mkswap

</code>

and <code>swapon

</code>

or <code>swap

</code>

programs.

</p>

</section>

<section id="ulimit-files-and-processes">

<title>ulimit: Files and Processes

</title>

<p>Given a machine with plenty of RAM and processor capacity,

you can run hundreds of Apache processes if necessary. . .

and if your kernel allows it.

</p>

<p>Consider a situation in which several hundred web servers

are running; if some of these need to spawn CGI processes,

the maximum number of processes would occur quickly.

</p>

<p>However, you can change this limit with the command

</p>

<example>

ulimit [-H|-S] -u [newvalue]

</example>

<p>This must be changed before starting the server, since the

new value will only be available to the current shell and

programs started from it. In newer Linux kernels the

default has been raised to 2048. On FreeBSD, the number

seems to be the rather unusual 513. In the default user

shell on this system, <code>csh

</code>

the equivalent is <code>limit

</code>

and works analogous the the Bourne-like <code>ulimit

</code>

:

</p>

<example>

limit [-h] maxproc [newvalue]

</example>

<p>Similarly, the kernel may limit the number of open files per

process. This is generally not a problem for pre-forked

servers, which just handle one request at a time per

process. Threaded servers, however, serve many requests per

process and much more easily run out of available file

descriptors. You can increase the maximum number of open

files per process by running the

</p>

<example>ulimit -n [newvalue]

</example>

<p>command. Once again, this must be done prior to starting

Apache.

</p>

</section>

<section id="setting-user-limits-on-system-startup">

<title>Setting User Limits on System Startup

</title>

<p>Under Linux, you can set the ulimit parameters on bootup by

editing the <code>/etc/security/limits.conf

</code>

file. This file allows you to set soft and hard limits on a

per-user or per-group basis; the file contains commentary

explaining the options. To enable this, make sure that the

file <code>/etc/pam.d/login

</code>

contains the line

</p>

<example>session required /lib/security/pam_limits.so

</example>

<p>All items can have a 'soft' and a 'hard'

limit: the first is the default setting and the second the

maximum value for that item.

</p>

<p>

In FreeBSD's <code>/etc/login.conf

</code>

these resources can be limited or extended system wide,

analogously to <code>limits.conf</code>.

'Soft' limits can be specified with <code>-cur</code>

and 'hard' limits with <code>-max</code>.

</p>

<p>Solaris has a similar mechanism for manipulating limit

values at boot time: In <code>/etc/system</code>

you can set kernel tunables valid for the entire system at

boot time. These are the same tunables that can be set with

the <code>mdb</code>

kernel debugger during run time. The soft and hard limit

corresponding to ulimit -u can be set via:

</p>

<example>

set rlim_fd_max=65536<br />

set rlim_fd_cur=2048

</example>

<p>Solaris calculates the maximum number of allowed processes

per user (<code>maxuprc</code>) based on the total amount

available memory on the system (<code>maxusers</code>).

You can review the numbers with

</p>

<example>sysdef -i | grep maximum

</example>

<p>but it is not recommended to change them.

</p>

</section>

<section id="turn-off-unused-services-and-modules">

<title>Turn Off Unused Services and Modules

</title>

<p>Many UNIX and Linux distributions come with a slew of

services turned on by default. You probably need few of

them. For example, your web server does not need to be

running sendmail, nor is it likely to be an NFS server,

etc. Turn them off.

</p>

<p>On Red Hat Linux, the chkconfig tool will help you do this

from the command line. On Solaris systems <code>svcs</code>

and <code>svcadm</code>

will show which services are enabled and disable them

respectively.

</p>

<p>In a similar fashion, cast a critical eye on the Apache

modules you load. Most binary distributions of Apache

httpd, and pre-installed versions that come with Linux

distributions, have their modules enabled through the

<directive>LoadModule</directive> directive.

</p>

<p>Unused modules may be culled: if you don't rely on

their functionality and configuration directives, you can

turn them off by commenting out the corresponding

<directive>LoadModule</directive>

lines. Read the documentation on each module’s

functionality before deciding whether to keep it enabled.

While the performance overhead of an unused module is

small, it's also unnecessary.

</p>

</section>

</section>

</section>

<section id="caching-content">

<title>Caching Content

</title>

<p>Requests for dynamically generated content usually take

significantly more resources than requests for static content.

Static content consists of simple filespages, images, etc.-on disk

that are very efficiently served. Many operating systems also

automatically cache the contents of frequently accessed files in

memory.

</p>

<p>Processing dynamic requests, on the contrary, can be much more

involved. Running CGI scripts, handing off requests to an external

application server and accessing database content can introduce

significant latency and processing load to a busy web server. Under

many circumstances, performance can be improved by turning popular

dynamic requests into static requests. In this section, two

approaches to this will be discussed.

</p>

<section id="making-popular-pages-static">

<title>Making Popular Pages Static

</title>

<p>By pre-rendering the response pages for the most popular queries

in your application, you can gain a significant performance

improvement without giving up the flexibility of dynamically

generated content. For instance, if your application is a

flower delivery service, you would probably want to pre-render

your catalog pages for red roses during the weeks leading up to

Valentine's Day. When the user searches for red roses,

they are served the pre-rendered page. Queries for, say, yellow

roses will be generated directly from the database. The

mod_rewrite module included with Apache is a great tool to

implement these substitutions.

</p>

<section id="example-a-statically-rendered-blog">

<title>Example: A Statically Rendered Blog

</title>

<!--we should provide a more useful example here.

<p>Blosxom is a lightweight web log package that runs as a CGI.

It is written in Perl and uses plain text files for entry

input. Besides running as CGI, Blosxom can be run from the

command line to pre-render blog pages. Pre-rendering pages

to static HTML can yield a significant performance boost in

the event that large numbers of people actually start

reading your blog.

</p>

<p>To run blosxom for static page generation, edit the CGI

script according to the documentation. Set the $static dir

variable to the <directive>DocumentRoot</directive>

of the web server, and run the script from the command line

as follows:

</p>

<example>$ perl blosxom.cgi -password='whateveryourpassword'

</example>

<p>This can be run periodically from Cron, after you upload

content, etc. To make Apache substitute the statically

rendered pages for the dynamic content, we’ll use

mod_rewrite. This module is included with the Apache source

code, but is not compiled by default. It can be built with

the server by passing the option <code>--enable-rewrite[=shared]</code>

to the configure command. Many binary distributions of

Apache come with <module>mod_rewrite </module> included. The following is an

example of an Apache virtual host that takes advantage of

pre-rendered blog pages:

</p>

<highlight language="config">

Listen *:8001

&lt;VirtualHost *:8001&gt;

ServerName blog.sandla.org:8001

ServerAdmin sander@temme.net

DocumentRoot "/home/sctemme/inst/blog/httpd/htdocs"

&lt;Directory

"/home/sctemme/inst/blog/httpd/htdocs"&gt;

Options +Indexes

Order allow,deny

Allow from all

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} !-f

RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule ^(.*)$ /cgi-bin/blosxom.cgi/$1 [L,QSA]

&lt;/Directory&gt;

RewriteLog /home/sctemme/inst/blog/httpd/logs/rewrite_log

RewriteLogLevel 9

ErrorLog /home/sctemme/inst/blog/httpd/logs/error_log

LogLevel debug

CustomLog /home/sctemme/inst/blog/httpd/logs/access_log common

ScriptAlias /cgi-bin/ /home/sctemme/inst/blog/bin/

&lt;Directory "/home/sctemme/inst/blog/bin"&gt;

Options +ExecCGI

Order allow,deny

Allow from all

&lt;/Directory&gt;

&lt;/VirtualHost&gt;

</highlight>

<p>

The <directive>RewriteCond</directive>

and <directive>RewriteRule</directive>

directives say that, if the requested resource does not

exist as a file or a directory, its path is passed to the

Blosxom CGI for rendering. Blosxom uses Path Info to

specify blog entries and index pages, so this means that if

a particular path under Blosxom exists as a static file in

the file system, the file is served instead. Any request

that isn't pre- rendered is served by the CGI. This

means that individual entries, which show the comments, are

always served by the CGI which in turn means that your

comment spam is always visible. This configuration also

hides the Blosxom CGI from the user-visible URL in their

Location bar. mod_rewrite is a fantastically powerful and

versatile module: investigate it to arrive at a

configuration that is best for your situation.

</p>

</section>

</section>

<section id="caching-content-with-mod_cache">

<title>Caching Content With mod_cache

</title>

<p>The mod_cache module provides intelligent caching of HTTP

responses: it is aware of the expiration timing and content

requirements that are part of the HTTP specification. The

mod_cache module caches URL response content. If content sent

to the client is considered cacheable, it is saved to disk.

Subsequent requests for that URL will be served directly from

the cache. The provider module for mod_cache, mod_disk_cache,

determines how the cached content is stored on disk. Most

server systems will have more disk available than memory, and

it's good to note that some operating system kernels cache

frequently accessed disk content transparently in memory, so

replicating this in the server is not very useful.

</p>

<p>To enable efficient content caching and avoid presenting the

user with stale or invalid content, the application that

generates the actual content has to send the correct response

headers. Without headers like <code>Etag:</code>,

<code>Last-Modified:</code> or <code>Expires:</code>,

<module>mod_cache</module> can not make the right decision on whether to cache

the content, serve it from cache or leave it alone. When

testing content caching, you may find that you need to modify

your application or, if this is impossible, selectively disable

caching for URLs that cause problems. The mod_cache modules are

not compiled by default, but can be enabled by passing the

option <code>--enable-cache[=shared]</code>

to the configure script. If you use a binary distribution of

Apache httpd, or it came with your port or package collection,

it may have <module>mod_cache</module> already included.

</p>

<section id="example-wiki">

<title>Example: wiki.apache.org

</title>

<!-- Is this still the case? Maybe we should give

<p>

The Apache Software Foundation Wiki is served by MoinMoin.

MoinMoin is written in Python and runs as a CGI. To date, any

attempts to run it under mod_python has been unsuccessful.

The CGI proved to place an untenably high load on the

server machine, especially when the Wiki was being indexed

by search engines like Google. To lighten the load on the

server machine, the Apache Infrastructure team turned to

mod_cache. It turned out <a href="/httpd/MoinMoin">MoinMoin

</a>

needed a small patch to ensure proper behavior behind the

caching server: certain requests can never be cached and

the corresponding Python modules were patched to send the

proper HTTP response headers. After this modification, the

cache in front of the Wiki was enabled with the following

configuration snippet in <code>httpd.conf</code>:

</p>

<highlight language="config">

CacheRoot /raid1/cacheroot

CacheEnable disk /

# A page modified 100 minutes ago will expire in 10 minutes

CacheLastModifiedFactor .1

# Always check again after 6 hours

CacheMaxExpire 21600

</highlight>

<p>This configuration will try to cache any and all content

within its virtual host. It will never cache content for

more than six hours (the <directive module="mod_cace">CacheMaxExpire</directive>

directive). If no <code>Expires:</code>

header is present in the response, <module>mod_cache</module> will compute

an expiration period from the <code>Last-Modified:</code>

header. The computation using <directive module="mod_cache">CacheLastModifiedFactor</directive>

is based on the assumption that if a page was recently

modified, it is likely to change again in the near future

and will have to be re-cached.

</p>

<p>

Do note that it can pay off to <em>disable</em>

the <code>ETag:</code>

header: For files smaller than 1k the server has to

calculate the checksum (usually MD5) and then send out a

<code>304 Not Modified</code>

response, which will use up some CPU and still saturate

the same amount of network resources for the transfer (one

TCP packet). For resources larger than 1k it might prove

CPU expensive to calculate the header for each request.

Unfortunately there does currently not exist a way to cache

these headers.

</p>

<highlight language="config">

&lt;FilesMatch \.(jpe?g|png|gif|js|css|x?html|xml)&gt;

FileETag None

&lt;/FilesMatch&gt;

</highlight>

<p>

This will disable the generation of the <code>ETag:</code>

header for most static resources. The server does not

calculate these headers for dynamic resources.

</p>

</section>

</section>

</section>

<section id="further-considerations">

<title>Further Considerations

</title>

<p>Armed with the knowledge of how to tune a sytem to deliver the

desired the performance, we will soon discover that <em>one</em>

system might prove a bottleneck. How to make a system fit for

growth, or how to put a number of systems into tune will be

discussed in <a href="http://wiki.apache.org/httpd/PerformanceScalingOut">PerformanceScalingOut</a>.

</p>

</section>

</manualpage>