6443N/A<?
xml version="1.0" encoding="UTF-8"?>
6443N/A ! This work is licensed under the Creative Commons 6443N/A ! Attribution-NonCommercial-NoDerivs 3.0 Unported License. 6443N/A ! To view a copy of this license, visit 6443N/A ! or send a letter to Creative Commons, 444 Castro Street, 6443N/A ! Suite 900, Mountain View, California, 94041, USA. 6443N/A ! You can also obtain a copy of the license at 6443N/A ! See the License for the specific language governing permissions 6443N/A ! and limitations under the License. 6443N/A ! If applicable, add the following below this CCPL HEADER, with the fields 6443N/A ! enclosed by brackets "[]" replaced with your own identifying information: 6443N/A ! Portions Copyright [yyyy] [name of copyright owner] 7097N/A ! Copyright 2011-2014 ForgeRock AS 6443N/A<
chapter xml:
id='chap-troubleshooting' 6443N/A version='5.0' xml:
lang='en' 6443N/A <
title>Troubleshooting Server Problems</
title>
6443N/A <
indexterm><
primary>Troubleshooting</
primary></
indexterm>
6443N/A <
para>This chapter describes how to troubleshoot common server problems,
6443N/A and how to collect information necessary when seeking support help.</
para>
6443N/A <
section xml:
id="troubleshoot-identify-problem">
6443N/A <
title>Identifying the Problem</
title>
6443N/A <
para>In order to solve your problem methodically, save time by defining the
6443N/A problem clearly up front. In a replicated environment with multiple directory
6443N/A servers and many client applications, it can be particularly important to
6443N/A pin down not only the problem (difference in observed behavior compared to
6443N/A expected behavior), but also the circumstances and steps that lead to the
6443N/A <
para>Answer the following questions.</
para>
6443N/A <
para>How do you reproduce the problem?</
para>
6443N/A <
para>What exactly is the problem? In other words, what is the behavior
6443N/A you expected? What is the behavior you observed?</
para>
6443N/A <
para>When did the problem start occurring? Under similar circumstances,
6443N/A when does the problem not occur?</
para>
6443N/A <
para>Is the problem permanent? Intermittent? Is it getting worse?
6443N/A Getting better? Staying the same?</
para>
6443N/A <
para>Pinpointing the problem can sometimes indicate where you should
6443N/A start looking for solutions.</
para>
6443N/A <
section xml:
id="troubleshoot-installation">
6443N/A <
title>Troubleshooting Installation & Upgrade</
title>
6443N/A <
para>Installation and upgrade procedures result in a log file tracing
6443N/A the operation. The log location differs by operating system, but look for
6443N/A lines in the command output of the following form.</
para>
6443N/A <
literallayout class="monospaced">See /
var/
....log for a detailed log of this operation.</
literallayout>
6443N/A <
section xml:
id="troubleshoot-reset-admin-passwords">
6443N/A <
title>Resetting Administrator Passwords</
title>
6443N/A <
para>This section describes what to do if you forgot the password for
6443N/A Directory Manager or for the global (replication) administrator.</
para>
6443N/A <
procedure xml:
id="reset-directory-manager-password">
6443N/A <
title>Resetting the Directory Manager's Password</
title>
6443N/A <
primary>Resetting passwords</
primary>
6443N/A <
secondary>cn=Directory Manager</
secondary>
6443N/A <
para>OpenDJ directory server stores the entry for Directory Manager in
6443N/A the LDIF representation of its configuration. You must be able to edit
6443N/A directory server files in order to reset Directory Manager's password.</
para>
6443N/A <
para>Generate the encoded version of the new password using the OpenDJ
6443N/A <
command>encode-password</
command> command.</
para>
7097N/A$ <
userinput>encode-password --storageScheme SSHA512 --clearPassword password</
userinput>
7097N/A<
computeroutput>Encoded Password: "{SSHA512}yWqHnYV4a5llPvE7WHLe5jzK27oZQWLIlVcs9gySu4TyZJMg
6443N/A <
para>Stop OpenDJ directory server while you edit the configuration.</
para>
7097N/A$ <
userinput>stop-ds</
userinput>
6443N/A <
para>Find Directory Manager's entry, which has DN <
literal>cn=Directory
6443N/A Manager,cn=Root DNs,cn=config</
literal>, in
6443N/A replace the <
literal>userpassword</
literal> attribute value with the
6443N/A encoded version of the new password, taking care not to leave any
6443N/A whitespace at the end of the line.</
para>
7097N/A <
programlisting language="ldif">
7097N/Adn: cn=Directory Manager,cn=Root DNs,cn=config
6443N/AobjectClass: organizationalPerson
6443N/AobjectClass: ds-cfg-root-dn-user
6443N/Auserpassword: {SSHA512}yWqHnYV4a5llPvE7WHLe5jzK27oZQWLIlVcs9gySu4TyZJMg
6443N/Ads-cfg-alternate-bind-dn: cn=Directory Manager
6443N/Ads-pwp-password-policy-dn: cn=Root Password Policy,cn=Password Policies
6443N/Ads-rlim-lookthrough-limit: 0
6443N/A <
para>Start OpenDJ directory server again.</
para>
7097N/A$ <
userinput>start-ds</
userinput>
6443N/A <
para>Verify that you can administer the server as Directory Manager using
7097N/A>>>> OpenDJ configuration console main menu
6443N/AWhat do you want to configure?
7097N/AEnter choice:</
computeroutput> <
userinput>q</
userinput>
6443N/A <
procedure xml:
id="reset-repl-admin-password">
6443N/A <
title>To Reset the Global Administrator's Password</
title>
6443N/A <
primary>Resetting passwords</
primary>
6443N/A <
secondary>Global (replication) administrator</
secondary>
6443N/A <
para>When you enable replication, part of the process involves creating a
6443N/A global administrator and setting that user's password. This user is present
6443N/A on all replicas. If you chose default values, this user has DN
6443N/A <
literal>cn=admin,cn=Administrators,cn=admin data</
literal>. You reset the
6443N/A password as you would for any other user, though you do so as Directory
6443N/A <
para>Use the <
command>ldappasswordmodify</
command> command to reset the
6443N/A global administrator's password</
para>
7097N/A$ <
userinput>ldappasswordmodify \
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --authzID "cn=admin,cn=Administrators,cn=admin data" \
7097N/A --newPassword password</
userinput>
7097N/A<
computeroutput>The LDAP password modify operation was successful</
computeroutput>
6443N/A <
para>Let replication copy the password change to other replicas.</
para>
6443N/A <
section xml:
id="troubleshoot-enable-debug-logging">
6443N/A <
title>Enabling Debug Logging</
title>
6443N/A <
indexterm><
primary>Debug log</
primary></
indexterm>
6443N/A <
secondary>Debug</
secondary>
6443N/A <
para>OpenDJ can write debug information and stack traces to the server
6443N/A debug log. What is logged depends both on debug targets that you create,
6443N/A and also on the debug level that you choose.</
para>
6443N/A <
procedure xml:
id="configure-debug-logging">
6443N/A <
title>To Configure Debug Logging</
title>
6443N/A is not enabled by default.</
para>
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --publisher-name "File-Based Debug Logger" \
7097N/A --set default-debug-level:all \
6443N/A <
para>You can set <
literal>default-debug-level</
literal> to a less verbose
6443N/A <
para>Create a debug target or targets.</
para>
6443N/A <
para>No debug targets are enabled by default.</
para>
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --publisher-name "File-Based Debug Logger" \
6443N/ADebug Target : debug-level : debug-category
6443N/A-------------:-------------:---------------
6443N/A <
para>A debug target specifies a fully-qualified OpenDJ Java package,
6443N/A class, or method for which to log debug messages at the level you
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --publisher-name "File-Based Debug Logger" \
6443N/A <
para>Restart OpenDJ to see debug messages in the log.</
para>
6443N/A <
para>If you have set <
literal>debug-level:all</
literal>, OpenDJ generates
6443N/A a great deal of output in the debug log file. Use debug logging very
6443N/A sparingly on production systems.</
para>
6443N/A <
section xml:
id="troubleshoot-use-lockdown-mode">
6443N/A <
title>Preventing Access While You Fix Issues</
title>
6443N/A <
indexterm><
primary>Lockdown mode</
primary></
indexterm>
6443N/A <
para>Misconfiguration can potentially put OpenDJ in a state where you must
6443N/A intervene, and where you need to prevent users and applications
6443N/A from accessing the directory until you are done fixing the problem.</
para>
6443N/A <
para>OpenDJ provides a <
firstterm>lockdown mode</
firstterm> that allows
6443N/A connections only on the loopback address, and allows only operations
6443N/A requested by root users, such as <
literal>cn=Directory
6443N/A Manager</
literal>. You can use lockdown mode to prevent all but
6443N/A administrative access to OpenDJ in order to repair the server.</
para>
6443N/A <
para>To put OpenDJ into lockdown mode, the server must be running. You
6443N/A cause the server to enter lockdown mode by using a task. Notice that
6443N/A the modify operation is performed over the loopback address (accessing
6443N/A OpenDJ on the local host).</
para>
7097N/A --bindDN "cn=Directory Manager" \
6443N/Adn: ds-task-id=Enter Lockdown Mode,cn=Scheduled Tasks,cn=tasks
6443N/Ads-task-id: Enter Lockdown Mode
7097N/A<
computeroutput>Processing ADD request for
6443N/A ds-task-id=Enter Lockdown Mode,cn=Scheduled Tasks,cn=tasks
6443N/AADD operation successful for DN
7097N/A ds-task-id=Enter Lockdown Mode,cn=Scheduled Tasks,cn=tasks</
computeroutput>
6443N/A when lockdown mode takes effect.</
para>
6443N/A <
literallayout class="monospaced">
6443N/A[
30/
Jan/
2012:17:04:32 +0100] category=BACKEND severity=NOTICE msgID=9896350
6443N/A msg=Lockdown task Enter Lockdown Mode finished execution</
literallayout>
6443N/A <
para>Client applications that request operations get a message concerning
7097N/A$ <
userinput>ldapsearch --port 1389 --baseDN "" --searchScope base "(objectclass=*)" +</
userinput>
7097N/A<
computeroutput>SEARCH operation failed
6443N/AResult Code: 53 (Unwilling to Perform)
6443N/AAdditional Information: Rejecting the requested operation because the server
6443N/A is in lockdown mode and will only accept requests from root users over
7097N/A loopback connections</
computeroutput>
6443N/A <
para>You also leave lockdown mode by using a task.</
para>
7097N/A --bindDN "cn=Directory Manager" \
6443N/Adn: ds-task-id=Leave Lockdown Mode,cn=Scheduled Tasks,cn=tasks
6443N/Ads-task-id: Leave Lockdown Mode
7097N/A<
computeroutput>Processing ADD request for
6443N/A ds-task-id=Leave Lockdown Mode,cn=Scheduled Tasks,cn=tasks
6443N/AADD operation successful for DN
7097N/A ds-task-id=Leave Lockdown Mode,cn=Scheduled Tasks,cn=tasks</
computeroutput>
6443N/A <
para>OpenDJ also logs a notice message when leaving lockdown.</
para>
6443N/A <
literallayout class="monospaced">
6443N/A[
30/
Jan/
2012:17:13:05 +0100] category=BACKEND severity=NOTICE msgID=9896350
6443N/A msg=Leave Lockdown task Leave Lockdown Mode finished execution</
literallayout>
6443N/A <
section xml:
id="troubleshoot-import">
6443N/A <
title>Troubleshooting LDIF Import</
title>
6443N/A <
para>By default OpenDJ requires that LDIF data you import respect standards.
6443N/A In particular, OpenDJ is set to check that entries to import match the
6443N/A schema defined for the server. You can temporarily bypass this check by using
6443N/A the <
option>--skipSchemaValidation</
option> with the
6443N/A <
command>import-ldif</
command> command.</
para>
6443N/A <
para>OpenDJ also ensures by default that entries have only one structural
6443N/A object class. You can relax this behavior by using the advanced global
6443N/A <
literal>single-structural-objectclass-behavior</
literal>. This can be useful
6443N/A when importing data exported from Sun Directory Server. For example, to
6443N/A warn when entries have more than one structural object class instead of
6443N/A reject such entries being added, set
6443N/A <
literal>single-structural-objectclass-behavior:warn</
literal> as
7097N/A set-global-configuration-prop \
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --set single-structural-objectclass-behavior:warn \
6443N/A <
para>By default, OpenDJ also checks syntax for a number of attribute types.
6443N/A You can relax this behavior as well by using the <
command>dsconfig
6443N/A set-attribute-syntax-prop</
command> command. See the list of attribute
6443N/A syntaxes and use the <
option>--help</
option> option for further
6443N/A <
para>When running <
command>import-ldif</
command>, you can use the <
option>-R
6443N/A <
replaceable>rejectFile</
replaceable></
option> option to capture entries that
6443N/A could not be imported, and the <
option>--countRejects</
option> option to
6443N/A return the number of rejected entries as the <
command>import-ldif</
command>
6443N/A <
para>Once you work through the issues with your LDIF data, reinstate the
6443N/A default behavior to ensure automated checking.</
para>
6443N/A <
section xml:
id="troubleshoot-secure-connections">
6443N/A <
para>In order to trust the server certificate, client applications usually
6443N/A compare the signature on certificates with those of the Certificate
6443N/A Authorities (CAs) whose certificates are distributed with the client
6443N/A software. For example, the Java environment is distributed with a key store
6443N/A holding many CA certificates.</
para>
6443N/A <
para>The self-signed server certificates that can be configured during
6443N/A OpenDJ setup are not recognized as being signed by any CAs. Your software
6443N/A therefore is configured not to trust the self-signed certificates by
6443N/A default. You must either configure the client applications to accept the
6443N/A self-signed certificates, or else use certificates signed by recognized
6443N/A <
para>You can further debug the network traffic by collecting debug traces.
6443N/A To see the traffic going over
TLS/
SSL in debug mode, configure OpenDJ to dump
6443N/A <
section xml:
id="troubleshoot-certificate-authentication">
6443N/A <
title>Troubleshooting Certificates & SSL Authentication</
title>
6443N/A <
para>Replication uses SSL to protect directory data on the network.
6443N/A In some configurations, replica can fail to connect to each other due
6443N/A to SSL handshake errors. This leads to error log messages such as the
7097N/A <
programlisting language="none">
6443N/A msgID=15138921 msg=SSL connection attempt from myserver (123.456.789.012)
7097N/A failed: Remote host closed connection during handshake
6443N/A <
para>Notice these problem characteristics in the message above.</
para>
6443N/A <
para>The host name, <
literal>myserver</
literal>, is not fully
6443N/A <
para>You should not see non fully qualified host names in the error logs.
6443N/A Non fully qualified host names are a sign that an OpenDJ server has not
6443N/A been configured properly.</
para>
6443N/A <
para>Always install and configure OpenDJ using fully-qualified host names.
6443N/A The OpenDJ administration connector, which is used by the
6443N/A <
command>dsconfig</
command> command, and also replication depend upon SSL
6443N/A and, more specifically, self-signed certificates for establishing SSL
6443N/A connections. If the host name used for connection establishment does not
6443N/A correspond to the host name stored in the SSL certificate then the SSL
6443N/A handshake can fail. For the purposes of establishing the SSL connection,
6443N/A a host name like <
literal>myserver</
literal> does not match
6443N/A <
para>The connection succeeded, but the SSL handshake failed, suggesting
6443N/A a problem with authentication or with the cipher or protocol negotiation.
6443N/A As most deployments use the same Java Virtual Machine, and the same JVM
6443N/A configuration for each replica, the problem is likely not related to SSL
6443N/A cipher or protocol negotiation, but instead lies with authentication.</
para>
6443N/A <
para>Follow these steps on each OpenDJ server to check whether the problem
6443N/A lies with the host name configuration.</
para>
6443N/A <
para>Make sure each OpenDJ server uses only fully qualified host names in
6443N/A the replication configuration. You can obtain a quick summary by running
6443N/A the following command against each server's configuration.</
para>
6443N/A <
para>Make sure that the host names in OpenDJ certificates also contain
6443N/A fully qualified host names, and correspond to the host names found in the
7097N/A# Examine the certificates used for the administration connector.
6443N/A# Examine the certificates used for replication.
6443N/A <
para>Unfortunately there is no easy solution to badly configured host
6443N/A names. It is often easier and quicker simply to reinstall your OpenDJ
6443N/A servers remembering to use fully qualified host names everywhere.</
para>
6443N/A <
para>When using the <
command>setup</
command> tool to install and
6443N/A configure a server ensure that the <
option>-h</
option> option is
6443N/A included, and that it specifies the fully qualified host name. Make sure
6443N/A >OPENDJ-363</
link>).</
para>
6443N/A <
para>If you are using the GUI installer, then make sure you specify the
6443N/A fully qualified host name on the first page of the wizard.</
para>
6443N/A <
para>When using the <
command>dsreplication</
command> tool to enable
6443N/A replication make sure that any <
option>--host</
option> options include the
6443N/A fully qualified host name.</
para>
6443N/A <
para>If you cannot reinstall the server, follow these steps.</
para>
6443N/A <
para>Disable replication in each replica.</
para>
7097N/A$ <
userinput>dsreplication \
7097N/A --port <
replaceable>adminPort</
replaceable> \
7097N/A --hostname <
replaceable>hostName</
replaceable> \
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --adminPassword <
replaceable>password</
replaceable> \
6443N/A <
para>Stop and restart each server in order to clear the in-memory ADS
6443N/A trust store backend.</
para>
6443N/A <
para>Enable replication making certain that fully qualified host names
7097N/A$ <
userinput>dsreplication \
7097N/A --adminPassword <
replaceable>password</
replaceable> \
7097N/A --baseDN dc=example,dc=com \
7097N/A --host1 <
replaceable>hostName1</
replaceable> \
7097N/A --port1 <
replaceable>adminPort1</
replaceable> \
7097N/A --bindDN1 "cn=Directory Manager" \
7097N/A --bindPassword1 <
replaceable>password</
replaceable> \
7097N/A --replicationPort1 <
replaceable>replPort1</
replaceable> \
7097N/A --host2 <
replaceable>hostName2</
replaceable> \
7097N/A --port2 <
replaceable>adminPort2</
replaceable> \
7097N/A --bindDN2 "cn=Directory Manager" \
7097N/A --bindPassword2 <
replaceable>password</
replaceable> \
7097N/A --replicationPort2 <
replaceable>replPort2</
replaceable> \
6443N/A <
para>Repeat the previous step for each remaining replica. In other words,
6443N/A host1 with host2, host1 with host3, host1 with host4, ..., host1 with
6443N/A <
para>Initialize all remaining replica with the data from host1.</
para>
7097N/A$ <
userinput>dsreplication \
7097N/A --baseDN dc=example,dc=com \
7097N/A --hostname <
replaceable>hostName1</
replaceable> \
6443N/A <
para>Check that the host names are correct in the configuration and in
6443N/A the key stores by following the steps you used to check for host name
6443N/A problems. The only broken host name remaining should be in the key and
6443N/A trust stores for the administration connector.</
para>
6443N/A <
para>Stop each server, and then fix the remaining admin connector
7097N/A certificate as described here in the procedure <
link xlink:
show="new" 6443N/A xlink:
href="admin-guide#replace-key-pair" 6443N/A Server Key Pair</
citetitle></
link>.</
para>
6443N/A <
section xml:
id="troubleshoot-compromised-key">
6443N/A <
title>Handling Compromised Keys</
title>
6443N/A <
indexterm><
primary>Certificates</
primary></
indexterm>
6443N/A <
indexterm><
primary>SSL</
primary></
indexterm>
6443N/A <
para>As explained in <
link xlink:
href="admin-guide#chap-change-certs" 6443N/A >Changing Server Certificates</
citetitle></
link>, OpenDJ directory server
6443N/A has different keys and key stores for different purposes. The public keys
6443N/A used for replication are also used to encrypt shared secret symmetric keys
6443N/A for example to encrypt and to sign back ups. This section looks at what to
6443N/A do if either a key pair or secret key is compromised.</
para>
6443N/A <
para>How you deal with the problem depends on which key was
6443N/A <
para>For a key pair used for a client connection handler and with a
6443N/A certificate signed by a certificate authority (CA), contact the CA for
6443N/A help. The CA might choose to publish a certificate revocation list (CRL)
6443N/A that identifies the certificate of the compromised key pair.</
para>
6443N/A <
para>Also make sure you replace the key pair. See <
link 6443N/A xlink:
href="admin-guide#replace-key-pair" xlink:
show="new" 6443N/A Server Key Pair</
citetitle></
link> for specific steps.</
para>
6443N/A <
para>For a key pair used for a client connection handler and that has
6443N/A a self-signed certificate, follow the steps in <
link 6443N/A xlink:
href="admin-guide#replace-key-pair" xlink:
show="new" 6443N/A Server Key Pair</
citetitle></
link>, and make sure the clients remove the
6443N/A compromised certificate from their trust stores, updating those trust
6443N/A stores with the new certificate.</
para>
6443N/A <
para>For a key pair that is used for replication, mark the key as
6443N/A compromised as described below, and replace the key pair. See <
link 6443N/A xlink:
href="admin-guide#replace-ads-cert" xlink:
show="new" 6443N/A Server Key Pair</
citetitle></
link> for specific steps.</
para>
6443N/A <
para>To mark the key pair as compromised, follow these steps.</
para>
6443N/A <
para>Identity the key entry by searching administrative data on the
6443N/A server whose key was compromised.</
para>
6443N/A <
para>The server in this example is installed on
6443N/A <
literal>4444</
literal>.</
para>
7097N/AÂ --baseDN "cn=admin data" \
7097N/Ads-cfg-key-id: 4F2F97979A7C05162CF64C9F73AF66ED</
computeroutput>
6443N/A <
para>The key ID, <
literal>4F2F97979A7C05162CF64C9F73AF66ED</
literal>, is
6443N/A the RDN of the key entry.</
para>
6443N/A <
para>Mark the key as compromised by adding the attribute,
6443N/A <
literal>ds-cfg-key-compromised-time</
literal>, to the key entry.</
para>
6443N/A <
para>The attribute has generalized time syntax, and so takes as its
6443N/A value the time at which the key was compromised expressed in generalized
6443N/A time. In the following example, the key pair was compromised at 8:34 AM
6443N/A UTC on March 21, 2013.</
para>
6443N/A --bindDN "cn=Directory Manager"
6443N/Adn: ds-cfg-key-id=4F2F97979A7C05162CF64C9F73AF66ED,cn=instance keys,cn=admin data
6443N/Aadd: ds-cfg-key-compromised-time
7097N/Ads-cfg-key-compromised-time: 201303210834Z</
userinput>
7097N/A<
computeroutput>Processing MODIFY request for ds-cfg-key-id=4F2F97979A7C05162CF64C9F73AF66ED,
6443N/A cn=instance keys,cn=admin data
6443N/AMODIFY operation successful for DN ds-cfg-key-id=4F2F97979A7C05162CF64C9F73AF66ED
7097N/A ,cn=instance keys,cn=admin data</
computeroutput>
6443N/A <
para>If the server uses encrypted or signed data, then the shared secret
6443N/A keys used for encryption or signing and associated with the compromised
6443N/A key pair should also be considered compromised. Therefore, mark all
6443N/A shared secret keys encrypted with the instance key as compromised.</
para>
6443N/A <
para>To identify the shared secret keys, find the list of secret keys
6443N/A in the administrative data whose <
literal>ds-cfg-symmetric-key</
literal>
6443N/A starts with the key ID of the compromised key.</
para>
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --baseDN "cn=secret keys,cn=admin data" \
7097N/A "(ds-cfg-symmetric-key=4F2F97979A7C05162CF64C9F73AF66ED*)" dn</
userinput>
7097N/A<
computeroutput>dn: ds-cfg-key-id=fba16e59-2ce1-4619-96e7-8caf33f916c8,cn=secret keys,cn=admin d
6443N/Adn: ds-cfg-key-id=57bd8b8b-9cc6-4a29-b42f-fb7a9e48d713,cn=secret keys,cn=admin d
6443N/Adn: ds-cfg-key-id=f05e2e6a-5c4b-44d0-b2e8-67a36d304f3a,cn=secret keys,cn=admin d
6443N/A <
para>For each such key, mark the entry with
6443N/A <
literal>ds-cfg-key-compromised-time</
literal> as shown above for the
6443N/A <
para>Changes to administration data are replicated to other OpenDJ
6443N/A servers in the replication topology.</
para>
6443N/A <
para>For a shared secret key used for data encryption that has been
6443N/A compromised, mark the key entry with
6443N/A <
literal>ds-cfg-key-compromised-time</
literal> as shown in the example
6443N/A above that demonstrates marking the instance key as compromised.</
para>
6443N/A <
para>Again, changes to administration data are replicated to other OpenDJ
6443N/A servers in the replication topology.</
para>
6443N/A <
section xml:
id="troubleshoot-connections">
6443N/A <
title>Troubleshooting Client Operations</
title>
6443N/A <
para>By default OpenDJ logs information about all LDAP client operations in
6443N/A readability, showing a search for the entry with
6443N/A <
literal>uid=bjensen</
literal> as traced in the LDAP access log. In the access
6443N/A log itself, each line starts with a time stamp.</
para>
7097N/A <
programlisting language="none">
6443N/A to=127.0.0.1:1389 protocol=LDAP
6443N/A base="dc=example,dc=com" scope=wholeSubtree filter="(uid=bjensen)" attrs="ALL"
6443N/A result=0 nentries=1 etime=3
6443N/A <
para>As you see, each client connection and set of LDAP operations are
6443N/A traced, starting with a time stamp and information about the operation
6443N/A performed, then including information about the connection, the operation
6443N/A number for the sequence of operations performed by the client, a message
6443N/A identification number, and additional information about the operation.</
para>
6443N/A <
para>To match HTTP client operations with related internal server operations,
6443N/A first prevent OpenDJ from suppressing internal operations from the LDAP access
6443N/A log by using the <
command>dsconfig</
command> command to set the LDAP access
6443N/A log publisher <
literal>suppress-internal-operations</
literal> advanced
6443N/A property to <
literal>false</
literal>. Then match the values of the
6443N/A <
literal>x-connection-id</
literal> field in the HTTP access log with
6443N/A <
literal>conn=<
replaceable>id</
replaceable></
literal> values in the LDAP
6443N/A <
para>For example, consider an HTTP GET request for the <
literal>_id</
literal>
6443N/A field of the user <
literal>newuser</
literal>, which is handled by connection 4
7097N/A <
programlisting language="none">
6443N/A log lines for the related operations have <
literal>conn=4</
literal>.</
para>
7097N/A <
programlisting language="none">
6443N/A from=192.168.0.12:63593 to=192.168.0.12:8080 protocol=
HTTP/
1.1 6443N/A op=0 msgID=0 base="ou=people,dc=example,dc=com" scope=wholeSubtree
6443N/A filter="(&(objectClass=inetOrgPerson)(uid=bjensen))" attrs="1.1"
6443N/A op=0 msgID=0 result=0 nentries=1 etime=5
6443N/A op=1 msgID=1 version=3 type=SIMPLE
6443N/A dn="uid=bjensen,ou=People,dc=example,dc=com"
6443N/A op=1 msgID=1 result=0 authDN="uid=bjensen,ou=People,dc=example,dc=com"
6443N/A op=2 msgID=2 base="uid=newuser,ou=people,dc=example,dc=com" scope=baseObject
6443N/A filter="(objectClass=*)" attrs="uid,etag"
6443N/A op=2 msgID=2 result=0 nentries=1 etime=4
6443N/A <
para>To help diagnose errors due to access permissions, OpenDJ supports the
6443N/A get effective rights control. The control OID,
6443N/A <
literal>1.3.6.1.4.1.42.2.27.9.5.2</
literal>, is not allowed by the default
6443N/A global ACIs. You must therefore add access to use the get effective rights
6443N/A control when not using it as Directory Manager.</
para>
6443N/A <
section xml:
id="troubleshoot-simple-paged-results">
6443N/A <
title>Clients Need Simple Paged Results Control</
title>
6443N/A <
para>For Solaris and some versions of Linux you might see a message in
6443N/A the OpenDJ access logs such as the following.</
para>
6443N/A <
literallayout class="monospaced">
6443N/AThe request control with Object Identifier (OID) "1.2.840.113556.1.4.319"
6443N/Acannot be used due to insufficient access rights</
literallayout>
6443N/A <
para>This message means clients are trying to use the <
link xlink:
show="new" 6443N/A control</
link> without authenticating. By default, OpenDJ includes a global
6443N/A ACI to allow only authenticated users to use the control.</
para>
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --bindPassword "password" \
7097N/A get-access-control-handler-prop</
userinput>
6443N/A-----------:-------------------------------------------------------------------
6443N/Aglobal-aci : (extop="1.3.6.1.4.1.26027.1.6.1 || 1.3.6.1.4.1.26027.1.6.3 ||
6443N/A : (targetcontrol="1.3.6.1.1.12 || 1.3.6.1.1.13.1 || 1.3.6.1.1.13.2
7097N/A : || 1.2.840.113556.1.4.319 || 1.2.826.0.1.3344810.2.3 ||
6443N/A : 2.16.840.1.113730.3.4.18 || 2.16.840.1.113730.3.4.9 ||
6443N/A : 1.2.840.113556.1.4.473 || 1.3.6.1.4.1.42.2.27.9.5.9") (version
6443N/A : 3.0; acl "Authenticated users control access"; allow(read)
6443N/A : userdn="ldap:///all";), (targetcontrol="2.16.840.1.113730.3.4.2 ||
6443N/A : 2.16.840.1.113730.3.4.17 || 2.16.840.1.113730.3.4.19 ||
6443N/A : 1.3.6.1.4.1.4203.1.10.2 || 1.3.6.1.4.1.42.2.27.8.5.1 ||
6443N/A : 2.16.840.1.113730.3.4.16") (version 3.0; acl "Anonymous control
7097N/A : access"; allow(read) userdn="ldap:///anyone";)</
computeroutput>
6443N/A <
para>To grant anonymous (unauthenticated) user access to the control,
6443N/A add the OID for the simple paged results control to the list of those in
6443N/A the <
literal>Anonymous control access</
literal> global ACI.</
para>
7097N/A --bindDN "cn=Directory Manager" \
7097N/A --bindPassword "password" \
7097N/A set-access-control-handler-prop \
7097N/A --remove global-aci:"(targetcontrol=\"2.16.840.1.113730.3.4.2 || \
7097N/A 2.16.840.1.113730.3.4.17 || 2.16.840.1.113730.3.4.19 || \
7097N/A 1.3.6.1.4.1.4203.1.10.2 || 1.3.6.1.4.1.42.2.27.8.5.1 || \
7097N/A 2.16.840.1.113730.3.4.16\") (version 3.0; acl \"Anonymous control access\"; \
7097N/A allow(read) userdn=\"ldap:///anyone\";)" \
7097N/A --add global-aci:"(targetcontrol=\"2.16.840.1.113730.3.4.2 || \
7097N/A 2.16.840.1.113730.3.4.17 || 2.16.840.1.113730.3.4.19 || \
7097N/A 1.3.6.1.4.1.4203.1.10.2 || 1.3.6.1.4.1.42.2.27.8.5.1 || \
7097N/A 2.16.840.1.113730.3.4.16 || 1.2.840.113556.1.4.319\") \
7097N/A (version 3.0; acl \"Anonymous control access\"; allow(read) \
7097N/A userdn=\"ldap:///anyone\";)" \
6443N/A <
para>Alternatively, stop OpenDJ, edit the corresponding ACI carefully in
6443N/A <
footnote><
para>Unlike the <
command>dsconfig</
command> command, the
6443N/A alternative should not be used in production.</
para></
footnote></
para>
6443N/A <
section xml:
id="troubleshoot-repl">
6443N/A <
title>Troubleshooting Replication</
title>
6443N/A <
primary>Replication</
primary>
6443N/A <
secondary>Troubleshooting</
secondary>
6443N/A <
para>Replication can generally recover from conflicts and transient issues.
6443N/A Replication does, however, require that update operations be copied
6443N/A from server to server. It is therefore possible to experience temporary
6443N/A delays while replicas converge, especially when the write operation load is
6443N/A heavy. OpenDJ's tolerance for temporary divergence between replicas is what
6443N/A allows OpenDJ to remain available to serve client applications even when
6443N/A networks linking the replicas go down.</
para>
6443N/A <
para>In other words, the fact that directory services are loosely convergent
6443N/A rather than transactional is a feature, not a bug.</
para>
6443N/A <
para>That said, you may encounter errors. Replication uses its own error log
6443N/A have <
literal>category=SYNC</
literal>. The messages have the following form.
6443N/A Here the line is folded for readability.</
para>
7097N/A <
programlisting language="none">
7097N/A[
27/
Jun/
2011:14:37:48 +0200] category=SYNC severity=INFORMATION msgID=14680169
6443N/A probably benign, but may indicate a transient network outage or a
6443N/A misconfigured client application connecting to this replication server.
7097N/A The error was: Remote host closed connection during handshake
6443N/A <
para>OpenDJ maintains historical information about changes in order to
6443N/A bring replicas up to date, and to resolve replication conflicts. To prevent
6443N/A historical information from growing without limit, OpenDJ purges historical
6443N/A information after a configurable delay
6443N/A (<
literal>replication-purge-delay</
literal>, default: 3 days). A replica
6443N/A can become irrevocably out of sync if you restore it from a backup archive
6443N/A older than the purge delay, or if you stop it for longer than the purge
6443N/A delay. If this happens to you, disable the replica, and then reinitialize it
6443N/A from a recent backup or from a server that is up to date.</
para>
6443N/A <
section xml:
id="troubleshoot-get-help">
6443N/A <
title>Asking For Help</
title>
6443N/A <
para>When you cannot resolve a problem yourself, and want to ask for help,
6443N/A clearly identify the problem and how you reproduce it, and also the version
6443N/A of OpenDJ you use to reproduce the problem. The version includes both a
6443N/A version number and also a build time stamp.</
para>
7097N/A$ <
userinput>dsconfig --version</
userinput>
7097N/A<
computeroutput>OpenDJ ${docTargetVersion}
7097N/ABuild <
replaceable>yyyymmddhhmmss</
replaceable>Z</
computeroutput>
6443N/A <
para>Be ready to provide additional information, too.</
para>
6443N/A <
para>The output from the <
command>java -version</
command> command.</
para>
6443N/A <
para><
filename>access</
filename> and <
filename>errors</
filename> logs
6443N/A showing what the server was doing when the problem started occurring</
para>
6443N/A <
para>A copy of the server configuration file,
6443N/A <
para>Other relevant logs or output, such as those from client applications
6443N/A experiencing the problem</
para>
6443N/A <
para>A description of the environment where OpenDJ is running, including
6443N/A system characteristics, host names, IP addresses, Java versions, storage
6443N/A characteristics, and network characteristics. This helps to understand
6443N/A the logs, and other information.</
para>