<?xml version="1.0" encoding="UTF-8"?>

<chapter xml:id='chap-import-export'

xmlns='http://docbook.org/ns/docbook' version='5.0' xml:lang='en'

xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'

xsi:schemaLocation='http://docbook.org/ns/docbook

xmlns:xlink='http://www.w3.org/1999/xlink'>

<title>Importing &amp; Exporting LDIF Data</title>

<indexterm><primary>Provisioning</primary></indexterm>

<indexterm><primary>Importing data</primary></indexterm>

<indexterm>

<primary>Restoring</primary>

<secondary>From LDIF</secondary>

</indexterm>

<indexterm><primary>Exporting data</primary></indexterm>

<indexterm><primary>Backup</primary></indexterm>

<indexterm>

<primary>LDIF</primary>

<secondary>Import</secondary>

</indexterm>

<indexterm>

<primary>LDIF</primary>

<secondary>Export</secondary>

</indexterm>

<para>LDAP Data Interchange Format provides a mechanism for representing

directory data in text format. LDIF data is typically used to initialize

directory databases, but also may be used to move data between different

directories that cannot replicate directly, or even as an alternative

backup format.</para>

<para>This chapter shows you how to import and export LDIF.

This chapter also covers creating test data in LDIF format, and manipulating

LDIF data with command-line tools.</para>

<section xml:id="generating-ldif">

<title>Generating Test Data</title>

<para>

When you install OpenDJ,

you have the option of importing sample data

that is generated during the installation.

This procedure demonstrates how to generate LDIF by using the

<link

xlink:show="new"

xlink:href="reference#make-ldif-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>make-ldif</command></link> command.

</para>

<procedure xml:id="generate-ldif">

<title>To Generate Test LDIF Data</title>

<indexterm>

<primary>Importing data</primary>

<secondary>Test data</secondary>

</indexterm>

<para>The <command>make-ldif</command> command uses templates to provide

sample data. Default templates are located in the

<filename>OpenDJ/config/MakeLDIF/</filename> directory. The

<filename>example.template</filename> file can be used to create

a suffix with entries of the type <literal>inetOrgPerson</literal>. You can

do the equivalent in OpenDJ Control Panel (Directory Data &gt; New Base

DN... &gt; Import Automatically Generated Example Data).</para>

<step>

<para>Write a file to act as the template for your generated LDIF.</para>

<para>The resulting test data template depends on what data you expect to

encounter in production. Base your work on your knowledge of the production

data, and on the sample template,

<filename>OpenDJ/config/MakeLDIF/example.template</filename>, and

associated data.</para>

<para>See <link xlink:href="reference#make-ldif-template-5"

xlink:role="http://docbook.org/xlink/role/olink" xlink:show="new"><citetitle

>make-ldif.template</citetitle></link> for reference information about

template files.</para>

</step>

<step>

<para>Create additional data files for the content in your template to be

selected randomly from a file, rather than generated by an expression.</para>

<para>Additional data files are located in the same directory as your

template file.</para>

</step>

<step>

<para>Decide whether you want to generate the same test data each time

you run the <command>make-ldif</command> command with your template.</para>

<para>If so, provide the same <literal>randomSeed</literal> integer each

time you run the command.</para>

</step>

<step>

<para>Before generating a very large LDIF file, make sure you have enough

space on disk.</para>

</step>

<step>

<para>Run the <command>make-ldif</command> command to generate your

LDIF file.</para>

<screen>

$ <userinput>make-ldif \

--randomSeed 0 \

--templateFile /path/to/my.template \

--ldifFile /path/to/generated.ldif</userinput>

<computeroutput>Processed 1000 entries

Processed 2000 entries

...

Processed 10000 entries

LDIF processing complete. 10003 entries written</computeroutput>

</screen>

</step>

</procedure>

</section>

<section xml:id="importing-exporting-ldif">

<title>Importing &amp; Exporting Data</title>

<para>

You can use OpenDJ Control Panel

to import data (Directory Data > Import LDIF)

and to export data (Directory Data &gt; Export LDIF).

The following procedures demonstrate how to use the

<link

xlink:show="new"

xlink:href="reference#import-ldif-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>import-ldif</command></link> and

<link

xlink:show="new"

xlink:href="reference#export-ldif-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>export-ldif</command></link> commands.

</para>

<procedure xml:id="import-ldif">

<title>To Import LDIF Data</title>

<para>The most efficient method of importing LDIF data is to take the

OpenDJ server offline. Alternatively, you can schedule a task to import

the data while the server is online.</para>

<step performance="optional">

<para>If you do not want to use the default <literal>userRoot</literal>

backend, create a new JE backend for your data.</para>

<para>See <xref linkend="create-database-backend" /> for details.</para>

</step>

<step>

<para>The following example imports <literal>dc=example,dc=org</literal>

data into the <literal>userRoot</literal> backend, overwriting existing

data.</para>

<stepalternatives>

<step>

<para>If you want to speed up the process&#8212;for example because you

have millions of directory entries to import&#8212;first shut down the

server, and then run the <command>import-ldif</command> command.</para>

<screen>

$ <userinput>stop-ds</userinput>

$ <userinput>import-ldif \

--includeBranch dc=example,dc=org \

--backendID userRoot \

--ldifFile /path/to/generated.ldif</userinput>

</screen>

</step>

<step>

<para>If not, schedule a task to import the data while online.</para>

<screen>

$ <userinput>import-ldif \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--includeBranch dc=example,dc=org \

--backendID userRoot \

--ldifFile /path/to/generated.ldif \

--trustAll</userinput>

</screen>

<para>Notice that the task is scheduled through communication over SSL on

the administration port, by default <literal>4444</literal>. You can

schedule the import task to start at a particular time using the

<option>--start</option> option.</para>

<para>The <option>--trustAll</option> option trusts all SSL certificates,

such as a default self-signed certificate used for testing.</para>

</step>

</stepalternatives>

</step>

</procedure>

<procedure xml:id="export-ldif">

<title>To Export LDIF Data</title>

<step>

<para>The following example exports <literal>dc=example,dc=org</literal>

data from the <literal>userRoot</literal> backend.</para>

<stepalternatives>

<step>

<para>If you want to speed up export, first shut down the server, and then

export data using the <command>export-ldif</command> command.</para>

<screen>

$ <userinput>stop-ds</userinput>

$ <userinput>export-ldif \

--includeBranch dc=example,dc=org \

--backendID userRoot \

--ldifFile /path/to/backup.ldif</userinput>

</screen>

</step>

<step>

<para>If not, schedule a task to export the data while online.</para>

<screen>

$ <userinput>export-ldif \

--port 4444 \

--hostname opendj.example.com \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--includeBranch dc=example,dc=org \

--backendID userRoot \

--ldifFile /path/to/backup.ldif \

--start 20111221230000 \

--trustAll</userinput>

</screen>

<para>The <option>--start 20111221230000</option> option tells OpenDJ to

start the export at 11 PM on December 21, 2012.</para>

<para>If OpenDJ is stopped at this time, then when you start OpenDJ again,

the server attempts to perform the task after starting up.</para>

</step>

</stepalternatives>

</step>

</procedure>

</section>

<section xml:id="ldif-tools">

<title>Other Tools For Working With LDIF Data</title>

<indexterm>

<primary>LDIF</primary>

<secondary>Tools</secondary>

</indexterm>

<para>

This section demonstrates the

<link

xlink:show="new"

xlink:href="reference#ldifsearch-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>ldifsearch</command></link>,

<link

xlink:show="new"

xlink:href="reference#ldifmodify-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>ldifmodify</command></link> and

<link

xlink:show="new"

xlink:href="reference#ldif-diff-1"

xlink:role="http://docbook.org/xlink/role/olink"

><command>ldif-diff</command></link> commands.

</para>

<section xml:id="ldifsearch-example">

<title>Searching in LDIF With <command>ldifsearch</command></title>

<para>The <command>ldifsearch</command> command lets you search LDIF files

in a similar way to how you search LDAP directories with the

<command>ldapsearch</command> command.</para>

<screen>

$ <userinput>ldifsearch \

--baseDN dc=example,dc=org \

--ldifFile generated.ldif \

"(sn=Grenier)" \

mobile</userinput>

<computeroutput>dn: uid=user.4630,ou=People,dc=example,dc=org

mobile: +1 728 983 6669</computeroutput>

</screen>

<para>The <option>--ldifFile <replaceable>ldif-file</replaceable></option>

option replaces the <option>--hostname</option> and <option>--port</option>

options used to connect to an LDAP directory. Otherwise the command syntax

and LDIF output is familiar to <command>ldapsearch</command> users.</para>

</section>

<section xml:id="ldifmodify-example">

<title>Updating LDIF With <command>ldifmodify</command></title>

<para>The <command>ldifmodify</command> command lets you apply changes to

LDIF files, generating a new, changed version of the original file.</para>

<screen>

$ <userinput>cat changes.ldif</userinput>

<computeroutput>dn: uid=user.0,ou=People,dc=example,dc=org

changetype: modify

replace: description

description: This is the new description for Aaccf Amar.

-

replace: initials

initials: AAA

</computeroutput>

$ <userinput>ldifmodify \

--sourceLDIF generated.ldif \

--changesLDIF changes.ldif \

--targetLDIF new.ldif</userinput>

</screen>

<para>Notice that the resulting new LDIF file is likely to be about the

same size as the source LDIF file.</para>

</section>

<section xml:id="ldif-diff-example">

<title>Comparing LDIF With <command>ldif-diff</command></title>

<para>The <command>ldif-diff</command> command reports differences between

two LDIF files in LDIF format.</para>

<screen>

$ <userinput>ldif-diff --sourceLDIF old.ldif --targetLDIF new.ldif</userinput>

<computeroutput>dn: uid=user.0,ou=People,dc=example,dc=org

changetype: modify

add: initials

initials: AAA

-

delete: initials

initials: ASA

-

add: description

description: This is the new description for Aaccf Amar.

-

delete: description

description: This is the description for Aaccf Amar.

</computeroutput>

</screen>

<para>As the <command>ldif-diff</command> command reads both files into

memory, constructing tree maps to perform the comparison, the command

is designed to work with small files and fragments. The command can quickly

run out of memory when calculating differences between large files.</para>

</section>

</section>

<section xml:id="create-database-backend">

<title>Creating a New Database Backend</title>

<indexterm>

<primary>Database backend</primary>

<secondary>Creating</secondary>

</indexterm>

<para>

OpenDJ stores your directory data in a <firstterm>backend</firstterm>.

Backends are what you backup and restore.

By default, OpenDJ stores your data in a backend named <literal>userRoot</literal>.

</para>

<para>

You can create new backends using the

<link

xlink:show="new"

xlink:href="reference#dsconfig-create-backend"

xlink:role="http://docbook.org/xlink/role/olink"

><command>dsconfig create-backend</command></link> command.

The following example creates a local backend named <literal>testData</literal>.

</para>

<screen>

$ <userinput>dsconfig create-backend --backend-name testData --type local-db</userinput>

<computeroutput>

>>>> Configuring the "base-dn" property

Specifies the base DN(s) for the data that the backend handles.

A single backend may be responsible for one or more base DNs. Note that no

two backends may have the same base DN although one backend may have a

base DN that is below a base DN provided by another backend (similar to

the use of sub-suffixes in the Sun Java System Directory Server). If any

of the base DNs is subordinate to a base DN for another backend, then all

base DNs for that backend must be subordinate to that same base DN.

Syntax: DN

Enter a value for the "base-dn" property:</computeroutput> <userinput>dc=example,dc=org</userinput>

<computeroutput>

Enter another value for the "base-dn" property [continue]:

>>>> Configuring the "enabled" property

Indicates whether the backend is enabled in the server.

If a backend is not enabled, then its contents are not accessible when

processing operations.

Select a value for the "enabled" property:

1) true

2) false

?) help

q) quit

Enter choice:</computeroutput> <userinput>1</userinput>

<computeroutput>

>>>> Configure the properties of the Local DB Backend

Property Value(s)

--------------------------------------

1) backend-id testData

2) base-dn "dc=example,dc=org"

3) compact-encoding true

4) db-cache-percent 10

5) db-cache-size 0 b

6) db-directory db

7) enabled true

8) index-entry-limit 4000

9) writability-mode enabled

?) help

f) finish - create the new Local DB Backend

q) quit

Enter choice [f]:

The Local DB Backend was created successfully</computeroutput>

</screen>

<para>Alternatively, you can create a new backend in OpenDJ Control Panel

(Directory Data > New Base DN > Backend > New Backend:

<replaceable>backend-name</replaceable>).</para>

</section>

<section xml:id="set-database-backend-disk-thresholds">

<title>Setting Disk Space Thresholds For Database Backends</title>

<indexterm>

<primary>Database backend</primary>

<secondary>Setting disk space thresholds</secondary>

</indexterm>

<para>

Directory data growth depends on applications that use the directory.

As a result, when directory applications add more data than they delete,

the local database backend grows until it fills the available disk space.

The system can end up in an unrecoverable state if no disk space is available.

</para>

<para>

Local database backends therefore have advanced properties,

<link

xlink:href="${configRefBase}/local-db-backend.html#disk-low-threshold"

xlink:show="new"

><literal>disk-low-threshold</literal></link> and

<link

xlink:href="${configRefBase}/local-db-backend.html#disk-full-threshold"

xlink:show="new"

><literal>disk-full-threshold</literal></link>.

When available disk space falls below <literal>disk-low-threshold</literal>,

OpenDJ server only allows updates from users and applications

that have the privilege to

<link

xlink:show="new"

xlink:href="admin-guide#about-privileges"

xlink:role="http://docbook.org/xlink/role/olink"

><literal>bypass-lockdown</literal></link>.

When available space falls below <literal>disk-full-threshold</literal>,

OpenDJ server stops allowing updates,

instead returning an <literal>UNWILLING_TO_PERFORM</literal> error

to each update request.

</para>

<para>

<emphasis>

OpenDJ server continues to apply replication updates

without regard to the thresholds.

</emphasis>

OpenDJ server can therefore fill available disk space despite the thresholds,

by accepting replication updates made on other servers.

You can give yourself more time to react to the situation

both by monitoring directory data growth

and also by increasing the thresholds.

</para>

<para>

If growth across the directory service tends to happen quickly,

set the thresholds higher than the defaults

to allow more time to react when growth threatens to fill the disk.

The following example sets <literal>disk-low-threshold</literal> to 2 GB

<literal>disk-full-threshold</literal> to 1 GB

for the <literal>userRoot</literal> local backend.

</para>

<screen>

$ <userinput>dsconfig \

set-backend-prop \

--hostname opendj.example.com \

--port 4444 \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--set "disk-low-threshold:2 GB" \

--set "disk-full-threshold:1 GB" \

--trustAll \

--no-prompt</userinput>

</screen>

<para>

The properties

<literal>disk-low-threshold</literal> and <literal>disk-full-threshold</literal>

are listed as "advanced" properties.

To examine their values with the <command>dsconfig</command> command,

use the <option>--advanced</option> option

as shown in the following example.

</para>

<screen>

$ <userinput>dsconfig \

get-backend-prop \

--advanced \

--hostname opendj.example.com \

--port 4444 \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--property disk-low-threshold \

--property disk-full-threshold \

--trustAll \

--no-prompt</userinput>

<computeroutput>Property : Value(s)

--------------------:---------

disk-full-threshold : 1 gb

disk-low-threshold : 2 gb</computeroutput>

</screen>

</section>

<section xml:id="update-database-backend">

<title>Updating an Existing Backend to Add a New Base DN</title>

<indexterm>

<primary>Database backend</primary>

<secondary>Updating</secondary>

</indexterm>

<para>

In addition to letting you create new backends as described in

<xref linkend="create-database-backend" />,

OpenDJ lets you add a new base DN to an existing backend.

</para>

<para>

The following example adds the suffix <literal>o=example</literal>

to the existing backend <literal>userRoot</literal>.

</para>

<screen>

$ <userinput>dsconfig \

set-backend-prop \

--hostname opendj.example.com \

--port 4444 \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--add base-dn:o=example \

--trustAll \

--no-prompt</userinput>

$ <userinput>dsconfig \

get-backend-prop \

--hostname opendj.example.com \

--port 4444 \

--bindDN "cn=Directory Manager" \

--bindPassword password \

--backend-name userRoot \

--property base-dn \

--trustAll \

--no-prompt</userinput>

<computeroutput>Property : Value(s)

---------:-------------------------------

base-dn : "dc=example,dc=com", o=example</computeroutput>

</screen>

<para>

Alternatively, you can update an existing backend in OpenDJ Control Panel

(Directory Data > New Base DN,

then select the existing backend from the dropdown Backend list,

and enter the new Base DN name).

</para>

</section>

<section xml:id="delete-database-backend">

<title>Deleting a Database Backend</title>

<indexterm>

<primary>Database backend</primary>

<secondary>Deleting</secondary>

</indexterm>

<para>

You delete a database backend by using the

<link

xlink:show="new"

xlink:href="reference#dsconfig-delete-backend"

xlink:role="http://docbook.org/xlink/role/olink"

><command>dsconfig delete-backend</command></link> command.

</para>

<para>When you delete a database backend by using the <command>dsconfig

delete-backend</command> command, OpenDJ does not actually remove the

database files for two reasons. First, a mistake could potentially cause

lots of data to be lost. Second, deleting a large database backend could

cause severe service degradation due to a sudden increase in I/O load.</para>

<para>Instead, after you run the <command>dsconfig delete-backend</command>

command you must also manually remove the database backend files.</para>

<para>If you do run the <command>dsconfig delete-backend</command> command by

mistake and have not yet deleted the actual files, then you can recover from

the mistake by creating the backend again, reconfiguring the indexes that

were removed, and rebuilding the indexes as described in the section on <link

xlink:href="admin-guide#configure-indexes"

xlink:role="http://docbook.org/xlink/role/olink"><citetitle>Configuring &amp;

Rebuilding Indexes</citetitle></link>.</para>

</section>

</chapter>