3339N/A * The contents of this file are subject to the terms of the 3339N/A * Common Development and Distribution License, Version 1.0 only 3339N/A * (the "License"). You may not use this file except in compliance 3339N/A * You can obtain a copy of the license at 3339N/A * See the License for the specific language governing permissions 3339N/A * and limitations under the License. 3339N/A * When distributing Covered Code, include this CDDL HEADER in each 3339N/A * file and include the License file at 3339N/A * add the following below this CDDL HEADER, with the fields enclosed 3339N/A * by brackets "[]" replaced with your own identifying information: 3339N/A * Portions Copyright [yyyy] [name of copyright owner] 5007N/A * Copyright 2008-2010 Sun Microsystems, Inc. 6029N/A * Portions Copyright 2011-2013 ForgeRock AS 4963N/A * This class provides the engine that performs both importing of LDIF files and 4963N/A * the rebuilding of indexes. 4963N/A //Defaults for LDIF reader buffers, min memory required to import and default 4963N/A //Min and MAX sizes of phase one buffer. 4963N/A //Min size of phase two read-ahead cache. 4963N/A //Small heap threshold used to give more memory to JVM to attempt OOM errors. 4963N/A //Phase one buffer and imported entries counts. 6032N/A //Phase one buffer size in bytes. 4963N/A //Set to true when validation is skipped. 4963N/A //Temporary environment used when DN validation is done in first phase. 5195N/A // Size in bytes of temporary env. 5275N/A // Available memory at the start of the import. 5195N/A // Size in bytes of DB cache. 4963N/A //The executor service used for the buffer sort tasks. 4963N/A //The executor service used for the scratch file processing tasks. 4591N/A //Queue of free index buffers -- used to re-cycle index buffers; 4643N/A //Map of index keys to index buffers. Used to allocate sorted 4591N/A //index buffers to a index writer thread. 4591N/A //Map of DB containers to index managers. Used to start phase 2. 4963N/A //Map of DB containers to DN-based index managers. Used to start phase 2. 4591N/A //Futures used to indicate when the index file writers are done flushing 4591N/A //their work queues and have exited. End of phase one. 4963N/A //List of index file writer tasks. Used to signal stopScratchFileWriters to 4963N/A //the index file writer tasks when the LDIF file has been done. 4643N/A //Map of DNs to Suffix objects. 4963N/A //Map of container ids to database containers. 4963N/A //Map of container ids to entry containers 4963N/A //Used to synchronize when a scratch file index writer is first setup. 5207N/A //Rebuild index manager used when rebuilding indexes. 4963N/A //Set to true if the backend was cleared. 4963N/A //Used to shutdown import if an error occurs in phase one. 4963N/A //Number of phase one buffers 5195N/A * Create a new import job with the specified rebuild index config. 5195N/A * The rebuild index configuration. 5195N/A * The local DB back-end configuration. 5195N/A * The JEB environment config. 5195N/A * @throws InitializationException 5195N/A * If a problem occurs during initialization. 5195N/A * If an error occurred when opening the DB. 5195N/A * If a problem occurs during initialization. 5195N/A * Create a new import job with the specified ldif import config. 5195N/A * @param importConfiguration 5195N/A * The LDIF import configuration. 5195N/A * The local DB back-end configuration. 5195N/A * The JEB environment config. 5195N/A * @throws InitializationException 5195N/A * If a problem occurs during initialization. 5195N/A * If a problem occurs reading the configuration. 5195N/A * @throws DatabaseException 5195N/A * If an error occurred when opening the DB. 5195N/A // Determine the number of indexes. 5195N/A // Set up temporary environment. 4591N/A * Return the suffix instance in the specified map that matches the specified 4591N/A * @return The suffix instance that matches the DN, or null if no match is 5195N/A * Calculate buffer sizes and initialize JEB properties based on memory. 5195N/A * The environment config to use in the calculations. 5195N/A * @throws InitializationException 5195N/A * If a problem occurs during calculation. 5195N/A // Calculate amount of usable memory. This will need to take into account 5195N/A // various fudge factors, including the number of IO buffers used by the 5195N/A // scratch writers (1 per index). 5665N/A // We need caching when doing DN validation or rebuilding indexes. 5195N/A // No DN validation: calculate memory for DB cache, DN2ID temporary cache, 5195N/A // Appending to existing data so reserve extra memory for the DB cache 5195N/A // since it will be needed for dn2id queries. 5195N/A // No DN validation: calculate memory for DB cache and buffers. 5195N/A // No need for DN2ID cache. 6114N/A // Scratch writers allocate 4 buffers per index as well. 6114N/A // The buffers are big enough: the memory is best used for the DN2ID 6114N/A // Retry using less threads. 5275N/A * Calculates the amount of available memory which can be used by this import, 5195N/A * taking into account whether or not the import is running offline or online 5198N/A // Round up to minimum of 16MB (e.g. unit tests only use 2% cache). 5195N/A // Now take into account various fudge factors. 5195N/A // Be pessimistic when memory is low. 5195N/A // Rebuild seems to require more overhead. 4963N/A //Mainly used to support multiple suffixes. Each index in each suffix gets 4963N/A //an unique ID to identify which DB it needs to go to in phase two processing. 6032N/A // This entire base DN was explicitly excluded. Skip. 6032N/A * There are no branches in the explicitly defined include list under 6032N/A * this base DN. Skip this base DN all together. 6032N/A // Remove any overlapping include branches. 6032N/A // Remove any exclude branches that are not are not under a include 6032N/A // branch since they will be migrated as part of the existing entries 6032N/A // outside of the include branches anyways. 6032N/A // This entire base DN is explicitly included in the import with 6032N/A // no exclude branches that we need to migrate. Just clear the entry 6032N/A // Create a temp entry container 4765N/A * Rebuild the indexes using the specified rootcontainer. 6032N/A * The rootcontainer to rebuild indexes in. 6032N/A * If a configuration error occurred. 6032N/A * @throws InitializationException 6032N/A * If an initialization error occurred. 6032N/A * If the JEB database had an error. 6032N/A * @throws InterruptedException 6032N/A * If an interrupted error occurred. 6032N/A * @throws ExecutionException 6032N/A * If an execution error occurred. 4649N/A * Import a LDIF using the specified root container. 6032N/A * The root container to use during the import. 6032N/A * If the import failed because of an configuration error. 6032N/A * @throws InitializationException 6032N/A * If the import failed because of an initialization error. 6032N/A * If the import failed due to a database error. 6032N/A * @throws InterruptedException 6032N/A * If the import failed due to an interrupted error. 6032N/A * @throws ExecutionException 6032N/A * If the import failed due to an execution error. 5118N/A // Shutdown the executor services 5195N/A // Try to clear as much memory as possible. 5195N/A // Calculate memory / buffer counts. 5195N/A // Cache size is never larger than the buffer size. 5201N/A // Not enough memory - will need to do batching for the biggest indexes. 5201N/A // Ensure that there are always two threads available for parallel 5201N/A // processing of smaller indexes. 5195N/A // Start DN processing first. 4643N/A * Task used to migrate excluded branch. 4643N/A // This is the base entry for a branch that was excluded in the 4643N/A // import so we must migrate all entries in this branch over to 4643N/A // the new entry container. 4643N/A * Task to migrate existing entries. 4660N/A // This is the base entry for a branch that will be included 4660N/A // in the import so we don't want to copy the branch to the 6032N/A * Advance the cursor to next entry at the same level in the DIT 6032N/A * skipping all the entries in this branch. Set the next 6032N/A * starting value to a value of equal length but slightly 6032N/A * greater than the previous DN. Since keys are compared in 6032N/A * reverse order we must set the first byte (the comma). No 6032N/A * possibility of overflow here. 4963N/A * This task performs phase reading and processing of the entries read from 4963N/A * the LDIF file(s). This task is used if the append flag wasn't specified. 4963N/A //Examine the DN for duplicates and missing parents. 4963N/A //If the backend was not cleared, then the dn2id needs to checked first 4963N/A //for DNs that might not exist in the DN cache. If the DN is not in 4963N/A //the suffixes dn2id DB, then the dn cache is used. 6344N/A "Index buffer processing error.");
6032N/A "Cancel processing received.");
4963N/A * This task reads sorted records from the temporary index scratch files, 6032N/A * processes the records and writes the results to the index database. The DN 6032N/A * index is treated differently then non-DN indexes. 5201N/A * Creates a new index DB writer. 5201N/A * The semaphore used for restricting the number of buffer 5201N/A * The maximum number of buffers which can be allocated. 5201N/A * Returns the next batch of buffers to be processed, blocking until enough 5201N/A * buffer permits are available. 5201N/A * @return The next batch of buffers, or {@code null} if there are no more 5201N/A * If an exception occurred. 5201N/A // First release any previously acquired permits. 5201N/A // Block until we can either get enough permits for all buffers, or the 5201N/A // maximum number of permits. 5201N/A // Create all the index buffers for the next batch. 5201N/A // First release any previously acquired permits. 5201N/A * Print out progress stats. 5201N/A * The time since the last update. 5201N/A // Kilo and milli approximately cancel out. 4643N/A * This class is used to by a index DB merge thread performing DN processing 4643N/A * to keep track of the state of individual DN2ID index processing. 5176N/A // This is the root or base DN 5176N/A // Why do we still need this if we are checking parents in the first 4835N/A //Bypass the cache for append data, lookup the parent in DN2ID and 5017N/A //If null is returned than this is a suffix DN. 5176N/A // We have a missing parent. Maybe parent checking was turned off? 5176N/A // We have a missing parent. Maybe parent checking was turned off? 4835N/A //Bypass the cache for append data, lookup the parent DN in the DN2ID 6319N/A // we can just walk to parent cache if available 6319N/A // We have a missing parent. Maybe parent checking was turned off? 6032N/A * This task writes the temporary scratch index files using the sorted buffers 6032N/A * read from a blocking queue private to each index. 5837N/A // Write buffer index information. 6032N/A * This task main function is to sort the index buffers given to it from the 6032N/A * import tasks reading the LDIF file. It will also create a index file writer 6032N/A * task and corresponding queue if needed. The sorted index buffers are put on 6032N/A * the index file writer queues for writing to a temporary file. 6032N/A * The index manager class has several functions: 1. It used to carry 6032N/A * information about index processing created in phase one to phase two. 2. It 6032N/A * collects statistics about phase two processing for each index. 3. It 6032N/A * manages opening and closing the scratch index files. 5195N/A * Updates the bytes read counter. 5195N/A * The number of bytes read. 5195N/A * Returns the file name associated with this index manager. 5195N/A * @return The file name associated with this index manager. 4963N/A * The rebuild index manager handles all rebuild index related processing. 6032N/A //Rebuild index configuration. 6032N/A //Local DB backend configuration. 6032N/A //Map of index keys to indexes. 6032N/A //Map of index keys to extensible indexes. 6032N/A //Total entries to be processed. 4765N/A * Create an instance of the rebuild index manager using the specified 6032N/A * The rebuild configuration to use. 6032N/A * The local DB configuration to use. 4963N/A * Initialize a rebuild index manager. 6032N/A * If an configuration error occurred. 6032N/A * @throws InitializationException 6032N/A * If an initialization error occurred. 6032N/A * @throws DatabaseException 6032N/A * If an database error occurred. 6032N/A * The time the rebuild started. 4963N/A * Perform rebuild index processing. 5195N/A * @throws DatabaseException 5195N/A * If an database error occurred. 5195N/A * @throws InterruptedException 5195N/A * If an interrupted error occurred. 5195N/A * @throws ExecutionException 6032N/A * If an Execution error occurred. 5195N/A * If an JEB error occurred. 6032N/A // Sets only the needed indexes. 6032N/A // If not in a 'clear degraded state' operation, 6032N/A // need to rebuild the indexes. 6032N/A // Depends on rebuild mode, (re)building indexes' lists. 6032N/A // false may be required if the user wants to rebuild specific index. 6032N/A // rebuildList contains the user-selected index(in USER_DEFINED mode). 6032N/A // Get all existing indexes for all && degraded mode. 6032N/A // Get indexes for user defined index. 6032N/A // This index is not a candidate for rebuilding. 6032N/A // This index is not a candidate for rebuilding. 6032N/A // Clears all the entry's container databases 6032N/A // which are containing the indexes. 6032N/A // dn2uri does not have a trusted status. 5198N/A // Try to clear as much memory as possible. 5720N/A // FIXME: since the environment is not started we cannot determine which 5720N/A // indexes are degraded. As a workaround, be conservative and assume all 5720N/A // indexes need rebuilding. 4963N/A * Return the number of entries processed by the rebuild manager. 4963N/A * @return The number of entries processed. 4963N/A * Return the total number of entries to process by the rebuild manager. 4963N/A * @return The total number for entries to process. 6032N/A * This class reports progress of rebuild index processing at fixed intervals. 6032N/A * The number of records that had been processed at the time of the previous 4963N/A * The time in milliseconds of the previous progress report. 4963N/A * The environment statistics at the time of the previous report. 4963N/A * Create a new rebuild index progress task. 6032N/A * @throws DatabaseException 6032N/A * If an error occurred while accessing the JE database. 4963N/A * The action to be performed by this timer task. 6032N/A * This class reports progress of first phase of import processing at fixed 6032N/A * The number of entries that had been read at the time of the previous 3339N/A * The time in milliseconds of the previous progress report. 3339N/A * The environment statistics at the time of the previous report. 4591N/A // Determines if eviction has been detected. 4591N/A // Entry count when eviction was detected. 3339N/A * Create a new import progress task. 4963N/A * The action to be performed by this timer task. 4963N/A //If first phase skip DN validation is specified use the root container 4963N/A //stats, else use the temporary environment stats. 4963N/A // Unlikely to happen and not critical. 4963N/A * This class reports progress of the second phase of import processing at 6032N/A * The number of entries that had been read at the time of the previous 4591N/A * The time in milliseconds of the previous progress report. 4591N/A * The environment statistics at the time of the previous report. 4591N/A // Determines if eviction has been detected. 4591N/A * Create a new import progress task. 5201N/A * The latest count of entries processed in phase one. 4591N/A * The action to be performed by this timer task. 4591N/A // Unlikely to happen and not critical. 4963N/A //Do DN index managers first. 4963N/A //Do non-DN index managers. 4643N/A * A class to hold information about the entry determined by the LDIF reader. 4963N/A * Mainly the suffix the entry belongs under and the ID assigned to it by the 4643N/A * Return the suffix associated with the entry. 4643N/A * @return Entry's suffix instance; 4643N/A * Set the suffix instance associated with the entry. 6032N/A * The suffix associated with the entry. 6032N/A * The entry ID to set the entry ID to. 4643N/A * Return the entry ID associated with the entry. 4643N/A * @return The entry ID associated with the entry. 4643N/A * This class defines the individual index type available. 4649N/A * The sub-string index type. 4643N/A * The approximate index type. 6032N/A * The extensible sub-string index type. 4643N/A * The extensible shared index type. 6032N/A * This class is used as an index key for hash maps that need to process 6032N/A * multiple suffix index elements into a single queue and/or maps based on 6032N/A * both attribute type and index type (ie., cn.equality, sn.equality,...). 4963N/A * Create index key instance using the specified attribute type, index type 6032N/A * The entry limit for the index. 4765N/A * An equals method that uses both the attribute type and the index type. 6032N/A * Only returns {@code true} if the attribute type and index type are equal. 4963N/A * @return {@code true} if the objects are equal, or {@code false} if they 4649N/A * A hash code method that adds the hash codes of the attribute type and 4643N/A * index type and returns that value. 4963N/A * @return The combined hash values of attribute type hash code and the 4643N/A * Return the attribute type. 4643N/A * @return The attribute type. 6032N/A * Return the index key name, which is the attribute type primary name, a 6032N/A * period, and the index type name. Used for building file names and 6032N/A * @return The index key name. 4963N/A * Return the entry limit associated with the index. 6032N/A * The temporary environment will be shared when multiple suffixes are being 4963N/A * processed. This interface is used by those suffix instance to do parental 4963N/A * checking of the DN cache. 6032N/A * Returns {@code true} if the specified DN is contained in the DN cache, or 6032N/A * The DN to check the presence of. 4963N/A * @return {@code true} if the cache contains the DN, or {@code false} if it 6032N/A * @throws DatabaseException 6032N/A * If an error occurs reading the database. 4963N/A * Temporary environment used to check DN's when DN validation is performed 4963N/A * during phase one processing. It is deleted after phase one processing. 4963N/A * Create a temporary DB environment and database to be used as a cache of 4963N/A * DNs when DN validation is performed in phase one processing. 6032N/A * The file path to create the environment under. 6032N/A * @throws DatabaseException 6032N/A * If an error occurs either creating the environment or the DN 4963N/A //Hash the DN bytes. Uses the FNV-1a hash. 4963N/A * Shutdown the temporary environment. 4963N/A * Insert the specified DN into the DN cache. It will return {@code true} if 4963N/A * the DN does not already exist in the cache and was inserted, or 4963N/A * {@code false} if the DN exists already in the cache. 6032N/A * The DN to insert in the cache. 6032N/A * A database entry to use in the insert. 6032N/A * A database entry to use in the insert. 6032N/A * @return {@code true} if the DN was inserted in the cache, or 6032N/A * {@code false} if the DN exists in the cache already and could not 6032N/A * If an error occurs accessing the database. 4963N/A "Search DN cache failed.");
4963N/A //Add the DN to the DNs as because of a hash collision. 4963N/A "Add of DN to DN cache failed.");
4963N/A //Return true if the specified DN is in the DNs saved as a result of hash 4963N/A * Check if the specified DN is contained in the temporary DN cache. 6032N/A * @return {@code true} if the specified DN is in the temporary DN cache, or 6032N/A * {@code false} if it is not. 4963N/A * Return temporary environment stats. 6032N/A * A stats configuration instance. 4963N/A * @return Environment stats. 6032N/A * @throws DatabaseException 6032N/A * If an error occurs retrieving the stats.