search_storage.py revision 959
429N/A# The contents of this file are subject to the terms of the 429N/A# Common Development and Distribution License (the "License"). 429N/A# You may not use this file except in compliance with the License. 429N/A# See the License for the specific language governing permissions 429N/A# and limitations under the License. 429N/A# When distributing Covered Code, include this CDDL HEADER in each 429N/A# If applicable, add the following below this CDDL HEADER, with the 429N/A# fields enclosed by brackets "[]" replaced with your own identifying 429N/A# information: Portions Copyright [yyyy] [name of copyright owner] 941N/A# Copyright 2009 Sun Microsystems, Inc. All rights reserved. 429N/A# Use is subject to license terms. 429N/A """Opens all data holders in data_list and ensures that the 429N/A versions are consistent among all of them. 429N/A It retries several times in case a race condition between file 429N/A migration and open is encountered. 429N/A Note: Do not set timeout to be 0. It will cause an exception to be 441N/A # The assignments to cur_version and missing cannot be 441N/A # placed here. They must be reset prior to breaking out of the 441N/A # for loop so that the while loop condition will be true. They 441N/A # cannot be placed after the for loop since that path is taken 441N/A # when all files are missing or opened successfully. 429N/A # All indexes must have the same version and all must 429N/A # either be present or absent for a successful return. 429N/A # If one of these conditions is not met, the function 429N/A # tries again until it succeeds or the time spent in 429N/A # in the function is greater than timeout. 429N/A # If we get here, then the current index file 429N/A # Read the version. If this is the first file, 429N/A # set the expected version otherwise check that 429N/A # the version matches the expected version. 429N/A # Got inconsistent versions, so close 429N/A # all files and try again. 429N/A # If the index file is missing, ensure 429N/A # that previous files were missing as 429N/A # well. If not, try again. 429N/A # The index is missing (ie, no files were present). 429N/A """Base class for all data storage used by the indexer and 429N/A queryEngine. All members must have a file name and maintain 429N/A an internal file handle to that file as instructed by external 429N/A """Closes the file handle and clears it so that it cannot 429N/A """Writes the dictionary in the expected format. 429N/A Note: Only child classes should call this method. 429N/A """This method uses the modification time and the file size 429N/A to (heuristically) determine whether the file backing this 429N/A storage has changed since it was last read. 429N/A """This uses consistent open to ensure that the version line 429N/A processing is done consistently and that only a single function 429N/A actually opens files stored using this class. 429N/A """Class for representing the main dictionary file 429N/A # Here is an example of a line from the main dictionary, it is 429N/A # (5,3,65691 => 249,202) (5,3,65692 => 249,202) 429N/A # The main dictionary has a more complicated format. Each line begins 429N/A # with a search token (%gconf.xml) followed by a list of mappings. Each 429N/A # mapping takes a token_type, action, and keyvalue tuple ((5,3,65689), 429N/A # (5,3,65690), (5,3,65691), (5,3,65692)) to a list of pkg-stem, version 429N/A # pairs (249,202) in which the token is found in an action with 429N/A # token_type, action, and keyvalues matching the tuple. Further 429N/A # compaction is gained by storing everything but the token as an id 429N/A # which the other dictionaries can turn into human-readable content. 429N/A # In short, the definition of a main dictionary entry is: 429N/A # Note: "(", ")", and "=>" actually appear in the file 429N/A # "[", "]", and "+" are used to specify pattern 429N/A # token [(token_type_id, action_id, keyval_id => [pkg_stem_id,version_id ]+)]+ 429N/A """This class relies on external methods to write the file. 429N/A Making this empty call to protected_write_dict_file allows the 429N/A file to be set up correctly with the version number stored 429N/A """Return the file handle. Note that doing 429N/A anything other than sequential reads or writes 429N/A to or from this file_handle may result in unexpected 429N/A behavior. In short, don't use seek. 429N/A """Parses one line of a main dictionary file. 429N/A Changes to this function must be paired with changes to 429N/A write_main_dict_line below. 941N/A [
" ",
"!",
"@",
"#",
","],
429N/A """Paired with parse_main_dict_line above. Writes 429N/A a line in a main dictionary file in the appropriate format. 941N/A [
"",
" ",
"!",
"@",
"#",
","],
941N/A """Paired with parse_main_dict_line above. Writes 941N/A a line in a main dictionary file in the appropriate format. 941N/A [
"",
" ",
"!",
"@",
"#",
","],
429N/A """Returns the number of entries removed during a second phase 429N/A # This returns 0 because this class is not responsible for 429N/A # storing anything in memory. 429N/A """Moves the existing file with self._name in directory 429N/A use_dir to a new file named self._name + suffix in directory 429N/A use_dir. If it has done this previously, it removes the old 429N/A file it moved. It also opens the newly moved file and uses 429N/A that as the file for its file handle. 429N/A """Used when both a list and a dictionary are needed to 429N/A store the information. Used for bidirectional lookup when 429N/A one item is an int (an id) and the other is not (an entity). It 429N/A maintains a list of empty spots in the list so that adding entities 429N/A can take advantage of unused space. It encodes empty space as a blank 429N/A line in the file format and '' in the internal list. 429N/A """Adds an entity consistently to the list and dictionary 429N/A allowing bidirectional lookup. 429N/A """deletes in_id from the list and the dictionary """ 429N/A """deletes the entity from the list and the dictionary """ 429N/A """returns the id of entity """ 429N/A """Adds entity if it's not previously stored and returns the 429N/A # This code purposefully reimplements add_entity 429N/A # code. Replacing the function calls to has_entity, add_entity, 429N/A # and get_id with direct access to the data structure gave a 429N/A # speed up of a factor of 4. Because this is a very hot path, 429N/A # the tradeoff seemed appropriate. 429N/A """return the entity in_id maps to """ 429N/A """check if entity is in storage """ 429N/A """Check if the structure has any empty elements which 429N/A can be filled with data. 429N/A """returns the next id which maps to no element """ 429N/A """Passes self._list to the parent class to write to a file. 429N/A """Reads in a dictionary previously stored using the above 429N/A # A blank line means that id can be reused. 429N/A """Returns the number of entries removed during a second phase 429N/A """Class used when only entity -> id lookup is needed 429N/A """Reads in a dictionary stored in line number -> entity 429N/A """If it's necessary to reread the file, it rereads the 429N/A file. It matches the line it reads against the contents of 429N/A in_set. If a match is found, the entry on the line is stored 429N/A for later use, otherwise the line is skipped. When all items 429N/A in in_set have been matched, the method is done and returns. 941N/A # skip the version line 429N/A """Returns the number of entries removed during a second phase 429N/A """Dictionary which allows dynamic update of its storage 429N/A """Reads in a dictionary stored in with an entity 429N/A and its number on each line. 429N/A """Opens the output file for this class and prepares it 429N/A to be written via write_entity. 429N/A """Writes the entity out to the file with my_id """ 429N/A """ Generates an iterable list of string representations of 429N/A the dictionary that the parent's protected_write_dict_file 429N/A """Returns the number of entries removed during a second phase 516N/A """Set the has value.""" 516N/A """Calculate the hash value of the sorted members of vals.""" 516N/A """Write self.hash_val out to a line in a file """ 516N/A """Process a dictionary file written using the above method 516N/A """Check the hash value of vals against the value stored 516N/A in the file for this object.""" 516N/A """Returns the number of entries removed during a second phase 429N/A """Used when only set membership is desired. 429N/A This is currently designed for exclusive use 429N/A with storage of fmri.PkgFmris. However, that impact 429N/A is only seen in the read_and_discard_matching_from_argument 429N/A """Remove entity purposfully assumes that entity is 429N/A already in the set to be removed. This is useful for 429N/A error checking and debugging. 429N/A """Write each member of the set out to a line in a file """ 429N/A """Process a dictionary file written using the above method 429N/A """Reads the file and removes all frmis in the file 429N/A """Returns the number of entries removed during a second phase