<?xml version="1.0" encoding="UTF-8"?>

<chapter xml:id='chap-tuning'

xmlns='http://docbook.org/ns/docbook' version='5.0' xml:lang='en'

xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'

xsi:schemaLocation='http://docbook.org/ns/docbook

xmlns:xlink='http://www.w3.org/1999/xlink'

xmlns:xinclude='http://www.w3.org/2001/XInclude'>

<title>Tuning Servers For Performance</title>

<indexterm><primary>Performance tuning</primary></indexterm>

<para>Server tuning refers to the art of adjusting server, JVM, and system

configuration to meet the service level performance requirements of directory

clients. In the optimal case you achieve service level performance

requirements without much tuning at all, perhaps only setting JVM runtime

options when installing OpenDJ.</para>

<para>If you are reading this chapter, however, you are probably not

facing an optimal situation. Instead you are looking for trade offs that

maximize performance for clients given the constraints of your deployment.

This chapter therefore aims to provide suggestions on how to measure and

to improve directory service performance for better trade offs.</para>

<section xml:id="perf-define-starting-points">

<title>Defining Performance Requirements &amp; Constraints</title>

<para>Your key performance requirement is most likely to satisfy your

users or customers with the resources available to you. Before you can

solve potential performance problems, define what those users or customers

expect, and determine what resources you will have to satisfy their

expectations.</para>

<section xml:id="perf-sla">

<title>Service-Level Agreements</title>

<para>Service-level agreement (SLA) is a formal name for what directory

client applications and the people who run them expect from your service in

terms of performance.</para>

<para>SLAs might cover many aspects of the directory service. Whether or not

your SLA is formally defined, you ought to know what is expected, or at least

what you provide, in the following four areas.</para>

<itemizedlist>

<listitem>

<para>Directory service <firstterm>response times</firstterm></para>

<para>Directory service response times range from less than a

millisecond on average across a low latency connection on the same

network to however long it takes your network to deliver the response.

More important than average or best response times is the response time

distribution, because applications set timeouts based on worst case

scenarios. For example, a response time performance requirement might

be defined as, "Directory response times must average less than 10

milliseconds for all operations except searches returning more than 10

entries, with 99.9% of response times under 40 milliseconds."</para>

</listitem>

<listitem>

<para>Directory service <firstterm>throughput</firstterm></para>

<indexterm>

<primary>Replication</primary>

<secondary>Write throughput</secondary>

</indexterm>

<para>Directory service throughput can range up to many thousands of

operations per second. In fact there is no upper limit for read operations

such as searches, because only write operations must be replicated. To

increase read throughput, simply add additional replicas. More important

than average throughput is peak throughput. You might have peak write

throughput in the middle of the night when batch jobs update entries in

bulk, and peak binds for a special event or first thing Monday morning.

For example, a throughput performance requirement might be expressed as,

"The directory service must sustain a mix of 5,000 operations per second

made up of 70% reads, 25% modifies, 3% adds, and 2% deletes."</para>

<para>Even better is to mimic the behavior of key operations for

performance testing, so that you understand the patterns of operations

in the throughput you need to provide.</para>

</listitem>

<listitem>

<para>Directory service <firstterm>availability</firstterm></para>

<para>OpenDJ is designed to let you build directory services that are

basically available, including during maintenance and even upgrade of

individual servers. Yet, in order to reach very high levels of

availability, you must make sure not only that the software is

designed for availability, but also that your operations execute in

such a way as to preserve availability. Availability requirements

can be as lax as best effort, or as stringent as 99.999% or more

uptime.</para>

<para>Replication is the OpenDJ feature that allows you to build a

highly available directory service.</para>

</listitem>

<listitem>

<para>Directory service administrative support</para>

<para>Do not forget to make sure you understand and set expectations

about how you support your users when they run into trouble. Directory

services can perhaps help you turn password management into a self-service

visit to a web site, but some users no doubt still need to know what they

can expect if they need your help.</para>

</listitem>

</itemizedlist>

<para>Writing down the SLA, even if your first version consists of

guesses, helps you reduce performance tuning from an open-ended project

to a clear set of measurable goals for a manageable project with a definite

outcome.</para>

</section>

<section xml:id="perf-constraints">

<title>Available Resources</title>

<para>With your SLA in hand, take inventory of the server, networks,

storage, people, and other resources at your disposal. Now is the time to

estimate whether it is possible to meet the requirements at all.</para>

<para>If for example you are expected to serve more throughput than the

network can transfer, maintain high availability with only one physical

machine, store 100 GB of backups on a 50 GB partition, or provide 24/7

support all alone, no amount of tweaking available resources is likely to

fix the problem.</para>

<para>When checking that the resources you have at least theoretically

suffice to meet your requirements, do not forget that high availability in

particular requires at least two of everything to avoid single points

of failure. Be sure to list the resources you expect to have, when and how

long you expect to have them, and why you need them. Also make note of

what is missing and why.</para>

<para>

In addition to the suggestions in this section, also read the section on

<link xlink:href="release-notes#prerequisites-hardware" xlink:show="new"

xlink:role="http://docbook.org/xlink/role/olink"><citetitle

>Hardware</citetitle></link> in the <citetitle>Release Notes</citetitle>.

</para>

<section xml:id="perf-hardware">

<title>Server Hardware Recommendations</title>

<para>Concerning server hardware, OpenDJ runs on systems with Java support,

and is therefore quite portable. That said, OpenDJ tends to perform best on

single-board, x86 systems due to low memory latency.</para>

</section>

<section xml:id="perf-storage">

<title>Storage Recommendations</title>

<para>OpenDJ is designed to work with local storage for the database,

not for network file systems such as NFS.</para>

<para>High performance storage is essential if you need to handle high

write throughput.</para>

<para>The Berkeley Java Edition DB works well with traditional disks as

long as the database cache size allows the DB to stay fully cached in

memory. This is the case because the database transaction log is append

only. When the DB is too big to stay cached in memory, however, then

cache misses lead to random disk access, slowing OpenDJ performance.</para>

<para>You might mitigate this effect by using solid-state disks for

persistent storage, or for file system cache.</para>

<para>Regarding database size on disk, if you have sustained write traffic

then the database grows to about twice its initial size on disk. This is

normal, and due to the way the database manages its logs. The size on disk

does not impact the DB cache size requirements.</para>

<xinclude:include href="/shared/para-write-barriers.xml" />

</section>

</section>

</section>

<section xml:id="perf-testing">

<title>Testing Performance</title>

<para>Even if you do not need high availability, you still need two of

everything, because your test environment needs to mimic your production

environment as closely as possible if you want to avoid nasty

surprises.</para>

<para>In your test environment, you set up OpenDJ as you will later in

production, and then conduct experiments to determine how best to meet

the requirements defined in the SLA.</para>

<para>Use <link xlink:show="new" xlink:href="reference#make-ldif-1"

xlink:role="http://docbook.org/xlink/role/olink">make-ldif</link> to generate

sample data that match what you expect to find in production.</para>

<para>

The OpenDJ LDAP Toolkit provides command-line tools

to help with basic performance testing.

</para>

<itemizedlist>

<listitem>

<para>

The

<link

xlink:show="new"

xlink:href="${coreDocBase}dev-guide#addrate-1"

>addrate</link> command measures add and delete throughput and response time.

</para>

</listitem>

<listitem>

<para>The <link xlink:show="new"

xlink:href="${coreDocBase}dev-guide#authrate-1">authrate</link> command

measures bind throughput and response time.</para>

</listitem>

<listitem>

<para>The <link xlink:show="new"

xlink:href="${coreDocBase}dev-guide#modrate-1">modrate</link> command

measures modification throughput and response time.</para>

</listitem>

<listitem>

<para>The <link xlink:show="new"

xlink:href="${coreDocBase}dev-guide#searchrate-1">searchrate</link> command

measures search throughput and response time.</para>

</listitem>

</itemizedlist>

<para>

All these commands show you information about the response time distributions,

and allow you to perform tests at specific levels of throughput.

</para>

<para>

If you need additional precision when evaluating response times,

use the global configuration setting,

<link

xlink:show="new"

xlink:href="${configRefBase}global.html#etime-resolution"

><literal>etime-resolution</literal></link>,

to change elapsed processing time resolution

from milliseconds (default) to nanoseconds.

</para>

<screen>

$ <userinput>dsconfig \

set-global-configuration-prop \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--set etime-resolution:nanoseconds \

--no-prompt</userinput>

</screen>

</section>

<section xml:id="perf-tweaking">

<title>Tweaking OpenDJ Performance</title>

<para>When your tests show that OpenDJ performance is lacking even though

you have the right underlying network, hardware, storage, and system

resources in place, you can tweak OpenDJ performance in a number of ways.

This section mentions the most common tweaks.</para>

<xinclude:include href="/shared/sec-prerequisites-file-descriptors.xml" />

<section xml:id="perf-java">

<title>Java Settings</title>

<para>Default Java settings let you evaluate OpenDJ using limited system

resources. If you need high performance for production system, test with

the following JVM options. These apply to the Sun/Oracle JVM.</para>

<tip>

<para>

To apply JVM settings for your server,

edit <filename>config/java.properties</filename>,

and apply the changes with the

<link

xlink:show="new"

xlink:href="reference#dsjavaproperties-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>dsjavaproperties</command></link> command.

</para>

</tip>

<variablelist>

<varlistentry>

<term><option>-server</option></term>

<listitem>

<para>Use the C2 compiler and optimizer.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-d64</option></term>

<listitem>

<para>To use a heap larger than about 3.5 GB on a 64-bit system, use

this option.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-Xms, -Xmx</option></term>

<listitem>

<para>Set both minimum and maximum heap size to the same value to avoid

resizing. Leave space for the entire DB cache and more.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-Xmn</option></term>

<listitem>

<para>Set the new generation size between 1-4 GB for high throughput

deployments, but leave enough overall JVM heap to avoid overlaps with

the space used for DB cache.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-XX:MaxTenuringThreshold=1</option></term>

<listitem>

<para>Force OpenDJ to create only objects that have either a short

lifetime, or a long lifetime.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-XX:+UseConcMarkSweepGC</option></term>

<listitem>

<para>The CMS garbage collector tends to give the best performance

characteristics. You might also consider the G1 garbage collector.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-XX:+PrintGCDetails</option></term>

<term><option>-XX:+PrintGCTimeStamps</option></term>

<listitem>

<para>Use these when diagnosing JVM tuning problems. You can turn them

off when everything is running smoothly.</para>

</listitem>

</varlistentry>

<varlistentry>

<term><option>-XX:+UseCompressedOops</option></term>

<listitem>

<para>Java object pointers normally have the same size as native machine

pointers. If you run a small, but 64-bit JVM, then compressed object

pointers can save space. Set this option when you have a 64-bit JVM,

<option>-Xmx</option> less than 32 GB, and Java SE 6u23 or later.</para>

</listitem>

</varlistentry>

</variablelist>

</section>

<section xml:id="perf-data-storage">

<title>Data Storage Settings</title>

<para>By default, OpenDJ compresses attribute descriptions and object class

sets to reduce data size. This is called compact encoding.</para>

<para>

By default, OpenDJ does not however compress entries

stored in its backend database.

If your entries hold values that compress well &#8212;

such as text, and not JPEG photos or MP3 audio &#8212;

you can gain space by setting the local DB backend property,

<link

xlink:show="new"

xlink:href="${configRefBase}local-db-backend.html#entries-compressed"

><literal>entries-compressed</literal></link>,

to <literal>true</literal> before you (re-)import data from LDIF.

With <literal>entries-compressed: true</literal>

OpenDJ compresses entries before writing them to the database.<footnote>

<para>

OpenDJ does not proactively rewrite all entries in the database

after you change the settings.

Instead, to force OpenDJ to compress all entries,

import the data from LDIF.

</para>

</footnote>

</para>

<screen>

$ <userinput>dsconfig \

set-backend-prop \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--set entries-compressed:true \

--trustAll \

--no-prompt</userinput>

$ <userinput>import-ldif \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--ldifFile /path/to/Example.ldif \

--backendID userRoot \

--includeBranch dc=example,dc=com \

--start 0</userinput>

<computeroutput>Import task 20120917100628767 scheduled to start Sep 17, 2012 10:06:28 AM CEST</computeroutput>

</screen>

</section>

<section xml:id="perf-import">

<title>LDIF Import Settings</title>

<indexterm>

<primary>Importing data</primary>

<secondary>Performance</secondary>

</indexterm>

<para>You can tweak OpenDJ to speed up import of large LDIF files.</para>

<para>

By default, the temporary directory used for scratch files is

<filename>import-tmp</filename> under the directory

where you installed OpenDJ.

Use the

<link

xlink:show="new"

xlink:href="reference#import-ldif-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>import-ldif</command></link> command

with the <option>--tmpdirectory</option> option

to set this directory to a <literal>tmpfs</literal> file system,

such as <filename>/tmp</filename>.

</para>

<para>In some cases, you can improve performance by using the

<option>--threadCount</option> option with the

<command>import-ldif</command> command to set the thread count larger than

the default, which is twice the number of CPUs.</para>

<para>If you are certain your LDIF contains only valid entries with

correct syntax, because the LDIF was exported from OpenDJ with all checks

active for example, you can skip schema and DN validation. Use the

<option>--skipSchemaValidation</option> and

<option>--skipDNValidation</option> options with the

<command>import-ldif</command> command to skip validation.</para>

</section>

<section xml:id="perf-db-cache">

<title>Database Cache Settings</title>

<para>

Database cache size is, by default, set as a percentage of the JVM heap

by using the backend property,

<link

xlink:show="new"

xlink:href="${configRefBase}local-db-backend.html#db-cache-percent"

><literal>db-cache-percent</literal></link>.

Alternatively, you use the backend property,

<link

xlink:show="new"

xlink:href="${configRefBase}local-db-backend.html#db-cache-size"

><literal>db-cache-size</literal></link>,

to set the size.

If you set up multiple database backends,

the total percent of JVM heap used must remain less than 100,

and must leave space for other uses.

Default settings work for servers

with one user data backend JVM heaps up to 2 GB.

For heaps larger than 2 GB,

you can allocate a larger percentage of heap space to DB cache.

</para>

<para>Depending on the size of your database, you have a choice to make

about database cache settings.</para>

<para>

By caching the entire database in the JVM heap,

you can get more deterministic response times and limit disk I/O.

Yet, caching the whole DB can require a very large JVM,

which you must pre-load on startup,

and which can result in long garbage collections

and a difficult-to-manage JVM.

Test database pre-load on startup by setting the

<link

xlink:show="new"

xlink:href="${configRefBase}local-db-backend.html#preload-time-limit"

><literal>preload-time-limit</literal></link>

for the backend.</para>

<screen>

$ <userinput>dsconfig \

set-backend-prop \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--set preload-time-limit:30m \

--trustAll \

--no-prompt</userinput>

</screen>

<para>Database pre-load is single-threaded, and loads each database one

at a time.</para>

<para>By allowing file system cache to hold the portion of database that

does not fit in DB cache, you trade less deterministic and slightly slower

response times for not having to pre-load the DB and not having garbage

collection pauses with large JVMs. How you configure the file system cache

depends on your operating system.</para>

</section>

<section xml:id="perf-entry-cache">

<title>Entry Cache Settings</title>

<para>OpenDJ implements an entry cache. The entry cache is not designed to

cache every entry in your database, but is instead useful in cases where you

have a few, typically large entries that are regularly used. For example, if

you have a few large static groups and applications that regularly check

group membership, you could cache your group entries.</para>

<screen>

$ <userinput>dsconfig \

create-entry-cache \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--cache-name "Large Group Entry Cache" \

--type fifo \

--set cache-level:1 \

--set include-filter:"(ou=Large Static Groups)" \

--set max-entries:10 \

--set enabled:true \

--trustAll \

--no-prompt</userinput>

</screen>

<para>

You can use the global setting,

<link

xlink:show="new"

xlink:href="${configRefBase}global.html#entry-cache-preload"

><literal>entry-cache-preload</literal></link>,

to force OpenDJ to load the entry cache as part of server startup.

</para>

<screen>

$ <userinput>dsconfig \

set-global-configuration-prop \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--set entry-cache-preload:true \

--no-prompt</userinput>

</screen>

<para>By default, OpenDJ does not pre-load the entry cache.</para>

</section>

<section xml:id="perf-logging">

<title>Logging Settings</title>

<para>Debug logs trace the internal workings of OpenDJ, and therefore

generally should be used sparingly, especially in high performance

deployments.</para>

<para>In general leave other logs active for production environments to

help troubleshoot any issues that arise.</para>

<para>For OpenDJ servers handling very high throughput, however, such as

100,000 operations per second or more, the access log constitute a

performance bottleneck, as each client request results in multiple access log

messages. Consider disabling the access log in such cases.</para>

<screen>

$ <userinput>dsconfig \

set-log-publisher-prop \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--publisher-name "File-Based Access Logger" \

--set enabled:false \

--trustAll \

--no-prompt</userinput>

</screen>

</section>

</section>

</chapter>