catalog.py revision 264
23N/A# The contents of this file are subject to the terms of the 23N/A# Common Development and Distribution License (the "License"). 23N/A# You may not use this file except in compliance with the License. 23N/A# See the License for the specific language governing permissions 23N/A# and limitations under the License. 23N/A# When distributing Covered Code, include this CDDL HEADER in each 23N/A# If applicable, add the following below this CDDL HEADER, with the 23N/A# fields enclosed by brackets "[]" replaced with your own identifying 23N/A# information: Portions Copyright [yyyy] [name of copyright owner] 221N/A# Copyright 2008 Sun Microsystems, Inc. All rights reserved. 23N/A# Use is subject to license terms. 221N/A"""Interfaces and implementation for the Catalog object, as well as functions 221N/Athat operate on lists of package FMRIs.""" 34N/A """A Catalog is the representation of the package FMRIs available to 34N/A this client or repository. Both purposes utilize the same storage 24N/A The serialized structure of the repository is an unordered list of 26N/A available package versions, followed by an unordered list of 26N/A incorporation relationships between packages. This latter section 26N/A allows the graph to be topologically sorted by the client. 26N/A S Last-Modified: [timespec] 50N/A XXX A authority mirror-uri ... 270N/A # XXX Mirroring records also need to be allowed from client 270N/A # configuration, and not just catalogs. 270N/A # XXX It would be nice to include available tags and package sizes, 270N/A # although this could also be calculated from the set of manifests. 37N/A # XXX Current code is O(N_packages) O(M_versions), should be 59N/A # O(1) O(M_versions), and possibly O(1) O(1). 59N/A # XXX Initial estimates suggest that the Catalog could be composed of 59N/A # 1e5 - 1e7 lines. Catalogs across these magnitudes will need to be 59N/A # spread out into chunks, and may require a delta-oriented update 59N/A """Create a catalog. If the path supplied does not exist, 59N/A this will create the required directory structure. 59N/A Otherwise, if the directories are already in place, the 59N/A existing catalog is opened. If pkg_root is specified 59N/A and no catalog is found at cat_root, the catalog will be 23N/A rebuilt. authority names the authority that 270N/A is represented by this catalog.""" 157N/A # We need to lock the search database against multiple 157N/A # simultaneous updates from separate threads closing 195N/A # publication transactions. 157N/A """Add a package, named by the fmri, to the catalog. 270N/A Throws an exception if an identical package is already 204N/A present. Throws an exception if package has no version.""" 195N/A "Unversioned FMRI not supported: %s" %
fmri 50N/A # Callers should verify that the FMRI they're going to add is 270N/A # valid; however, this check is here in case they're 270N/A "Existing renames make adding FMRI %s invalid." \
270N/A "Package %s is already in the catalog" % \
258N/A """Perform any catalog transformations necessary if 258N/A prefix p is found in the catalog. Previously, we didn't 258N/A know how to handle this prefix and now we do. If we 50N/A need to transform the entry from server to client form, 157N/A make sure that happens here.""" 157N/A """Takes the list of in-memory attributes and returns 157N/A a list of strings, each string naming an attribute.""" 157N/A s =
"S %s: %s\n" % (k, v)
157N/A """Helper method that takes the full path to the package 30N/A directory and the name of the manifest file, and returns an FMRI 157N/A constructed from the information in those components.""" 270N/A """If this version of the catalog knows about new prefixes, 215N/A check the on disk catalog to see if we can perform any 157N/A transformations based upon previously unknown catalog formats. 220N/A This routine will add a catalog attribute if it doesn't exist, 220N/A otherwise it checks this attribute against a hard-coded 220N/A version-specific tuple to see if new methods were added. 220N/A If new methods were added, it will call an additional routine 220N/A that updates the on-disk catalog, if necessary.""" 220N/A # If a prefixes attribute doesn't exist, write one and get on 157N/A # Prefixes attribute does exist. Check if it has changed. 157N/A # Nothing to do if prefixes haven't changed 204N/A # If known_prefixes contains a prefix not in pfx_set, 204N/A # add the prefix and perform a catalog transform. 220N/A # Write out updated prefixes list 220N/A """Walk the on-disk package data and build (or rebuild) the 220N/A package catalog and search database.""" 220N/A # XXX eschew os.walk in favor of another os.listdir here? 220N/A # XXX force a rebuild despite mtimes? 220N/A # XXX queue this and fork later? 204N/A # XXX force a rebuild despite mtimes? 204N/A # If the database doesn't exist, don't bother 204N/A # building the list; we'll just build it all. 204N/A # If we have no updates to make to the search database but it 204N/A # already exists, just make it available. If we do have updates 204N/A # to make (including possibly building it from scratch), fork it 204N/A # off into another process; when that's done, we'll mark it 204N/A "Failed to open search database", \
157N/A "for writing: %s (errno=%s)" % \
204N/A "Failed to open search " + \
204N/A "database: %s (errno=%s)" % \
204N/A """Handler method for the SIGCLD signal. Checks to see if the 204N/A search database update child has finished, and enables searching 204N/A if it finished successfully, or logs an error if it didn't.""" 264N/A "Failed to open search database", \
264N/A "for writing: %s (errno=%s)" % \
264N/A "Failed to open search " + \
264N/A "database: %s (errno=%s)" % \
204N/A # XXX This should be logged instead 204N/A print "ERROR building search database:" 204N/A """Update the search database with the FMRIs passed in via 204N/A 'fmri_list'. If 'fmri_list' is empty or None, then rebuild the 204N/A database from scratch. 'fmri_list' should be a list of tuples 264N/A where the first element is the full path to the package name in 264N/A pkg_root and the second element is the version string.""" 264N/A # If we're in the process of updating the database in our 264N/A # separate process, and this particular update until that's 264N/A # Since we're here explicitly to update 264N/A # the database, if we fail, there's 264N/A "Failed to open search database", \
264N/A "for writing: %s (errno=%s)" % \
264N/A "Failed to open search database", \
264N/A "for writing: %s (errno=%s)" % \
264N/A # XXX We should probably iterate over the catalog, for 264N/A # cases where manifests have stuck around, but have been 264N/A # moved to historical and removed from the catalog. 204N/A # If we rebuilt the database from scratch ... XXX why would we 204N/A # self.searchdb.close() 204N/A # Five digits of a base-62 number represents a little over 900 million. 204N/A # Assuming 1 million tokens used in a WOS build (current imports use 204N/A # just short of 500k, but we don't have all the l10n packages, and may 204N/A # not have all the search tokens we want) and keeping every nightly 204N/A # build gives us 2.5 years before we run out of token space. We're 204N/A # likely to garbage collect manifests and rebuild the db before then. 204N/A # XXX We're eventually going to run into conflicts with real tokens 204N/A # here. This is unlikely until we hit, say "alias", which is a ways 204N/A # off, but we should still look at solving this. 265N/A # XXX Do we want to log warnings as we approach index capacity? 265N/A """Update the search database with the data from the manifest 265N/A for 'fmri', which has been collected into 'search_dict'""" 204N/A # self.searchdb: token -> (type, fmri, action) 204N/A # XXX search_dict doesn't have action info, but should 204N/A # Don't update the database if it already has this FMRI's 204N/A # XXX The database files are so damned huge (if 204N/A # holey) because we have zillions of copies of 204N/A # the full fmri strings. We might want to 204N/A # indirect these as well. 204N/A """Because of the size limitations of the underlying database 204N/A records, not only do we have to store pointers to the actual 204N/A search data, but once the pointer records fill up, we have to 204N/A chain those records up to spillover records. This method adds 204N/A the pointer to the data to the end of the last link in the 204N/A chain, overflowing as necessary. The search token is passed in 204N/A as 'token', and the pointer to the actual data which should be 204N/A returned is passed in as 'data_token'.""" 204N/A # According to the ndbm man page, the total length of 204N/A # key and value must be less than 1024. Seems like the 204N/A # actual value is 1018, probably due to some padding or 204N/A # accounting bytes or something. The 2 is for the space 204N/A # separator and the plus-sign for the extension token. 204N/A # XXX The comparison should be against 1017, but that 204N/A # crahes in the if clause below trying to append the 204N/A # extension token. Dunno why. 204N/A # If we're adding the first element in the next 204N/A # link of the chain, add the extension token to 204N/A # the end of this link, and put the token 204N/A # pointing to the data at the beginning of the 204N/A break # from while True; we're done 204N/A # If we find an extension token, start looking 204N/A # at the next chain link. 204N/A # If we get here, it's safe to append the data token to 204N/A # the current link, and get out. 204N/A """Search through the search database for 'token'. Return a 204N/A list of token type / fmri pairs.""" 204N/A # For each indirect token in the search token's value, 204N/A # add its value to the return list. If we see a chain 204N/A # token, switch to its value and continue. If we fall 204N/A # out of the loop without seeing a chain token, we can 204N/A """Iterate through the catalog, looking for packages matching 204N/A 'pattern', based on the function in 'matcher' and the versioning 204N/A constraint described by 'constraint'. If 'matcher' is None, 204N/A uses fmri subset matching as the default. Returns a sorted list 204N/A of PkgFmri objects, newest versions first. If 'counthash' is a 204N/A dictionary, instead store the number of matched fmris for each 204N/A package name which was matched.""" 204N/A # 'patterns' may be partially or fully decorated fmris; we want 204N/A # to extract their names and versions to match separately 204N/A # XXX "5.11" here needs to be saner 270N/A # Handle old two-column catalog file, mostly in 161N/A """A generator function that produces FMRIs as it 161N/A iterates over the contents of the catalog.""" 195N/A # Handle old two-column catalog file, mostly in 161N/A """Returns a list of RenameRecords where fmri is listed as the 258N/A destination package.""" 258N/A # Don't bother doing this if no FMRI is present 195N/A # Load renamed packages, if needed 59N/A """Returns a list of RenameRecords where fmri is listed as 258N/A # Don't bother doing this if no FMRI is present 258N/A # Load renamed packages, if needed 258N/A """Return the time at which the catalog was last modified.""" 258N/A """Load attributes from the catalog file into the in-memory 258N/A attributes dictionary""" 270N/A # convert npkgs to integer value 270N/A """Load the catalog's rename records into self.renamed""" 270N/A """Returns the number of packages in the catalog.""" 270N/A """A static method that takes a file-like object and 270N/A a path. This is the other half of catalog.send(). It 270N/A reads a stream as an incoming catalog and lays it down 59N/A # XXX Need to be able to handle old and new 157N/A """Record that the name of package oldname has been changed 215N/A to newname as of version vers. Returns a timestamp 195N/A of when the catalog was modified and a RenamedPackage 195N/A object that describes the rename.""" 270N/A # Check that the destination (new) package is already in the 270N/A # catalog. Also check that the old package does not exist at 270N/A # the version that is being renamed. 270N/A "Destination FMRI %s must be in catalog" % \
270N/A "Src FMRI %s must not be in catalog" % \
270N/A # Load renamed packages, if needed 270N/A # Check that rename record isn't already in catalog 270N/A "Rename %s is already in the catalog" %
rr 270N/A # Keep renames acyclic. Check that the destination of this 270N/A # rename isn't the source of another rename. 270N/A "Can't rename %s. Causes cycle in rename graph." \
270N/A """Returns true if fmri and pfmri are the same package because 270N/A of a rename operation.""" 270N/A """Returns true if fmri is a successor to pfmri by way 270N/A of a rename operation.""" 258N/A """Returns true if fmri is a predecessor to pfmri by 258N/A """Returns a list of packages that are newer than fmri.""" 157N/A """Returns a list of packages that are older than fmri.""" 220N/A """Save attributes from the in-memory catalog to a file 220N/A specified by filenm.""" 161N/A """Send the contents of this catalog out to the filep 157N/A specified as an argument.""" 157N/A # Send attributes first. 258N/A # Missing catalog is fine; other errors need to be 258N/A """Set time to timestamp if supplied by caller. Otherwise 258N/A use the system time.""" 258N/A """Check that the fmri supplied as an argument would be 258N/A valid to add to the catalog. This checks to make sure that 258N/A from adding this FMRI.""" 258N/A# In order to avoid a fine from the Department of Redundancy Department, 258N/A# allow these methods to be invoked without explictly naming the Catalog class. 258N/A# Prefixes that this catalog knows how to handle 258N/A# Method used by Catalog and UpdateLog. Since UpdateLog needs to know 258N/A# about Catalog, keep it in Catalog to avoid circular dependency problems. 258N/A """Return an integer timestamp that can be used for comparisons.""" 258N/A """Take timestamp ts in string isoformat, and convert it to a datetime 258N/A """Iterate through the given list of PkgFmri objects, 258N/A looking for packages matching 'pattern', based on the function 258N/A in 'matcher' and the versioning constraint described by 258N/A 'constraint'. If 'matcher' is None, uses fmri subset matching 258N/A as the default. Returns a sorted list of PkgFmri objects, 258N/A newest versions first. If 'counthash' is a dictionary, instead 258N/A store the number of matched fmris for each package name which 258N/A # 'pattern' may be a partially or fully decorated fmri; we want 258N/A # to extract its name and version to match separately against 258N/A # XXX "5.11" here needs to be saner 157N/A """An in-memory representation of a rename object. This object records 157N/A information about a package that has had its name changed. 157N/A Renaming a package presents a number of challenges. The packaging 157N/A system must still be able to recognize and decode dependencies on 157N/A packages with the old name. In order for this to work correctly, the 157N/A rename record must contain both the old and new name of the package. It 157N/A is also undesireable to have a renamed package receive subsequent 157N/A versions. However, it still should be possible to publish bugfixes to 157N/A the old package lineage. This means that we must also record 174N/A versioning information at the time a package is renamed. 174N/A This versioning information allows us to determine which portions 174N/A of the version and namespace are allowed to add new versions. 174N/A If a package is re-named to the NULL package at a specific version, 174N/A this is equivalent to freezing the package. No further updates to 174N/A the version history may be made under that name. (NULL is never open) 174N/A The rename catalog format is as follows: 157N/A R <srcname> <srcversion> <destname> <destversion> 157N/A """Create a RenamedPackage object. Srcname is the original 215N/A name of the package, destname is the name this package 215N/A will take after the operation is successful. 215N/A Versionstr is the version at which this change takes place. No 215N/A versions >= version of srcname will be permitted.""" 258N/A "Must supply a source or destination version" 220N/A """Implementing our own == function allows us to properly 258N/A check whether a rename object is in a list of renamed 215N/A """Return a FMRI that represents the destination name and 215N/A version of the renamed package.""" 221N/A """Return a FMRI that represents the most recent version 221N/A of the package had it not been renamed."""