manifest.py revision 3402
50N/A# The contents of this file are subject to the terms of the 50N/A# Common Development and Distribution License (the "License"). 50N/A# You may not use this file except in compliance with the License. 50N/A# See the License for the specific language governing permissions 50N/A# and limitations under the License. 50N/A# When distributing Covered Code, include this CDDL HEADER in each 50N/A# If applicable, add the following below this CDDL HEADER, with the 50N/A# fields enclosed by brackets "[]" replaced with your own identifying 50N/A# information: Portions Copyright [yyyy] [name of copyright owner] 2608N/A# Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved. 1846N/A """Private helper function that returns a compiled version of a 580N/A dictionary of fnmatch patterns.""" 2293N/A """Private helper function: given an action, return True if any of its 1618N/A attributes' values matches the pattern for the same attribute in the 1846N/A attr_match dictionary, and False otherwise. Note that the patterns must 2453N/A be pre-comiled using re.compile() or _compile_fnpats.""" 2339N/A """Returns the serialized state of this object in a format 2339N/A that that can be easily stored using JSON, pickle, etc.""" 2339N/A """Allocate a new object using previously serialized state 2453N/A obtained via getstate().""" 2339N/A # decode serialized state into python objects 2339N/A """A Manifest is the representation of the actions composing a specific 2339N/A package version on both the client and the repository. Both purposes 2339N/A utilize the same storage format. 2339N/A The serialized structure of a manifest is an unordered list of actions. 2339N/A The special action, "set", represents a package attribute. 2339N/A The reserved attribute, "fmri", represents the package and version 2339N/A described by this manifest. It is available as a string via the 2339N/A attributes dictionary, and as an FMRI object from the fmri member. 2339N/A The list of manifest-wide reserved attributes is 2339N/A base_directory Default base directory, for non-user images. 2339N/A isa Package is intended for a list of ISAs. 2339N/A platform Package is intended for a list of platforms. 2339N/A relocatable Suitable for User Image. 2339N/A All non-prefixed attributes are reserved to the framework. Third 2339N/A parties may prefix their attributes with a reversed domain name, domain 2339N/A name, or stock symbol. An example might be 2339N/A as an indicator that a specific package version is supported by the 2339N/A manifest.null is provided as the null manifest. Differences against the 2339N/A null manifest result in the complete set of attributes and actions of 2339N/A the non-null manifest, meaning that all operations can be viewed as 2339N/A tranitions between the manifest being installed and the manifest already 2339N/A present in the image (which may be the null manifest). 583N/A """A generator function that returns the unsorted manifest 2205N/A contents as lines of text.""" 307N/A """Return three lists of action pairs representing origin and 307N/A destination actions. The first list contains the pairs 307N/A representing additions, the second list contains the pairs 307N/A representing updates, and the third list contains the pairs 307N/A representing removals. All three lists are in the order in 307N/A which they should be executed.""" 1271N/A # XXX Do we need to find some way to assert that the keys are 50N/A # No origin was provided, so nothing has been changed or 2026N/A # removed; only added. In addition, this doesn't need 2026N/A # to be sorted since the caller likely already does 2261N/A """handle key values that may be lists""" 2261N/A # Transform list of actions into a dictionary keyed by 2261N/A # action key attribute, key attribute and mediator, or 2261N/A # id if there is no key attribute. 307N/A # XXX Do changed actions need to be sorted at all? This is 307N/A # likely to be the largest list, so we might save significant 307N/A # time by not sorting. Should we sort above? Insert into a 1045N/A # singlesort = lambda x: x[0] or x[1] 1045N/A """Like the unix utility comm, except that this function 1045N/A takes an arbitrary number of manifests and compares them, 1045N/A returning a tuple consisting of each manifest's actions 1713N/A that are not the same for all manifests, followed by a 1045N/A list of actions that are the same in each manifest. 400N/A Content hashes for action payloads may be present in 1846N/A both signature-included and signature-excluded 1174N/A variants. In most cases (cmp_unsigned=False), we only 1174N/A want to compare the signature-included variants. When 400N/A no-signature-included comparison is requested, simply 2453N/A # Must specify at least one manifest. 2453N/A # construct list of dictionaries of actions in each 583N/A # manifest, indexed by unique key and variant combination 2453N/A # The unique key for each action is based on its 2453N/A # type, key attribute, and unique variants set 2117N/A for v
in (
"mediator-version",
2117N/A "mediator-implementation"):
1618N/A # If there is no key attribute for the 1618N/A # action, then fallback to the object 1618N/A # id for the action as its identifier. 1618N/A # catch duplicate actions here... 1618N/A # construct list of key sets in each dict 1618N/A # determine which common_keys have common actions 2286N/A """Where difference() returns three lists, combined_difference() 2286N/A returns a single list of the concatenation of the three.""" 2407N/A """Output expects that self is newer than other. Use of sets 2407N/A requires that we convert the action objects into some marshalled 2407N/A form, otherwise set member identities are derived from the 2407N/A object pointers, rather than the contents.""" 235N/A """Generate contents of dircache file containing all dirctories 289N/A referenced explicitly or implicitly from self.actions. Include 50N/A variants as values; collapse variants where possible.""" 1618N/A """Generate contents of mediatorcache file containing all 1618N/A mediators referenced explicitly or implicitly from self.actions. 1618N/A Include variants as values; collapse variants where possible.""" 1191N/A a =
"set name=pkg.mediator " \
289N/A """Generate set action supplemental data containing all facets 237N/A and variants from self.actions and size information. Each 1713N/A returned line must be newline-terminated.""" 237N/A # Package already has these attributes. 237N/A # List of possible variants and possible values for them. 1755N/A # Seed with declared set of variants as actions may be common to 1755N/A # both and so will not be tagged with variant. 237N/A # List of possible facets and under what variant combinations 237N/A # Unique (facet, value) (variant, value) combinations. 1755N/A # Facet applicable to this particular variant 1431N/A # This *must* be sorted to ensure reproducible set 237N/A # action generation for sizes and to ensure each 384N/A # combination is actually unique. 66N/A # Prevent scope leak. 72N/A # Unnecessary if we can guarantee all variants will be 136N/A # declared at package level. Omit the "variant." prefix 66N/A # from attribute values since that's implicit and can be 136N/A # added back when the action is parsed. 2293N/A # Emit a set action for every variant used with possible values 2293N/A # if one does not already exist. 2470N/A # merge_facets needs the variant values sorted and this 2470N/A # is desirable when generating the variant attr anyway. 2470N/A # Get unvarianted facet set. 2470N/A # For each variant combination, remove unvarianted 2470N/A # facets since they are common to all variants. 2470N/A # No facets unique to this combo; 2608N/A # If all possible variant combinations supported by the 2608N/A # package have at least one facet, then the intersection 2608N/A # of facets for all variants can be merged with the 2608N/A # Determine unique set of variant combinations 2608N/A # seen for faceted actions. # For each unique variant combination, determine # if the cartesian product of all variant values # supported by the package for the combination # has been seen. In other words, if the # combination is ((variant.arch,)) and the # package supports (i386, sparc), then both # (variant.arch, i386) and (variant.arch, sparc) # must exist. This code assumes variant values # for each variant are already sorted. # If any combination the package # supports has not been seen for # one or more facets, then some # facets are unique to one or # Merge the facets common to all variants if safe; # if we always merged them, then facets only # used by a single variant (think i386-only or # sparc-only content) would be seen unvarianted # At least one facet is shared between # all variant combinations; move the # common ones to the unvarianted set. # Remove facets common to all combos. # Omit the "facet." prefix from attribute values since # that's implicit and can be added back when the action # If we don't do this, action stringify will # emit this as "set name=pkg.facet" which is # then transformed to "set name=name # value=pkg.facet". Not what we wanted, but is # expected for historical reasons. # Always emit an action enumerating the list of facets # common to all variants, even if there aren't any. # That way if there are also no variant-specific facets, # package operations will know that no facets are used # by the package instead of having to scan the whole # Now emit a pkg.facet action for each variant # combination containing the list of facets unique to # A unique key for each combination is needed, # and using a hash obfuscates that interface # while giving us a reliable way to generate # a reproducible, unique identifier. The key # string below looks like this before hashing: # variant.archi386variant.debug.osnetTrue... # Omit the "facet." prefix from attribute values # since that's implicit and can be added back # when the action is parsed. # Tag action with variants. # Emit pkg.[c]size attribute for [compressed] size of package # running total so a single [c]size action can # Only emit if > 0; actions may be # A unique key for each combination is needed, # and using a hash obfuscates that interface # while giving us a reliable way to generate # a reproducible, unique identifier. The key # string below looks like this before hashing: # facet.docTruevariant.archi386... # The sizes are abbreviated in the name of byte """create dictionary of all actions referenced explicitly or implicitly from self.actions... include variants as values; collapse variants where possible""" # build a dictionary containing all actions tagged w/ # remove any tags if any entries are always delivered (NULL) # could collapse refs where all variants are present # (the current logic only collapses them if at least # one reference is delivered without a facet or """ return a list of directories implicitly or explicitly referenced by this object""" # generate actions that contain directories """A generator function that returns the supported facet attributes (strings) for this package based on the specified (or current) excludes that also match at least one of the patterns provided. Facets must be true or false so a list of possible facet values is not returned.""" # No excludes? Then use the pre-determined set of # If different excludes were specified, then look for pkg.facet # actions containing the list of facets. if a.
attrs[
"name"][:
10] ==
"pkg.facet.":
# Either a pkg.facet.common action or a # pkg.facet.X variant-specific action. # Prevent duplicates; it's # possible a given facet may be # valid for more than one unique # variant combination that's # allowed by current excludes. # Fallback to sifting actions to yield possible. """A generator function that yields a list of tuples of the form (variant, [values]). Where 'variant' is the variant attribute name (e.g. 'variant.arch') and '[values]' is a list of the variant values supported by this package. Variants returned are those allowed by the specified (or current) excludes that also match at least one of the patterns provided.""" # No excludes? Then use the pre-determined set of # If different excludes were specified, then look for # pkg.variant action containing the list of variants. if aname ==
"pkg.variant":
# Ensure variant entries exist (debug # variants may not) via defaultdict. elif aname[:
8] ==
"variant.":
# Fallback to sifting actions to get possible. """A generator function that yields tuples of the form (mediator, mediations) expressing the set of possible mediations for this package, where 'mediations' is a set() of possible mediations for the mediator. Each mediation is a tuple of the form (priority, version, implementation). # generate actions that contain mediators # Consider this mediation unavailable # if it can't be parsed for whatever """Generate actions in manifest through ordered callable list""" # These conditions are split by performance. """Generate actions in the manifest of type "type" through ordered callable list""" # These conditions are split by performance. """Generate actions in the manifest of types "atypes" through ordered callable list.""" """Generate the value of the key attribute for each action of type "type" in the manifest.""" """Find actions in the manifest which are duplicates (i.e., represent the same object) but which are not identical (i.e., have all the same attributes).""" """Return a key on which actions can be sorted.""" """Parse manifest content, stripping line-continuation characters from the input as it is read; this results in actions with values across multiple lines being passed to the action parsing code whitespace-separated instead. set name=pkg.description value="foo " \ ...will each be passed to action parsing as: set name=pkg.summary value="foo" set name=pkg.description value="foo " "bar baz" # Get an iterable for the string. if l.
endswith(
"\\"):
# allow continuation chars if not l
or l[
0] ==
"#":
# ignore blank lines & comments # Accumulate errors and continue so that as # much of the action data as possible can be """Populate the manifest with actions. 'content' is an optional value containing either the text representation of the manifest or an iterable of 'excludes' is optional. If provided it must be a length two list with the variants to be excluded as the first element and the facets to be excluded as the second element. 'pathname' is an optional filename containing the location of 'signatures' is an optional boolean value that indicates whether a manifest signature should be generated. This is only possible when 'content' is a string or 'pathname' is provided. # So we could build up here the type/key_attr dictionaries like # sdict and odict in difference() above, and have that be our # main datastore, rather than the simple list we have now. If # we do that here, we can even assert that the "same" action # can't be in a manifest twice. (The problem of having the same # action more than once in packages that can be installed # together has to be solved somewhere else, though.) raise TypeError(
"content must be str, not bytes")
# Generate manifest signature based upon # input content, but only if signatures # were requested. In order to interoperate with # older clients, we must use sha-1 here. # Make sure that either no excludes were provided or that both # variants and facet excludes were or that variant, facet and """Remove any actions from the manifest which should be """Performs any needed transformations on the action then adds The "action" parameter is the action object that should be The "excludes" parameter is the variants to exclude from the # XXX handle legacy transition issues; not needed once support # for upgrading images from older releases (< build 151) has if "opensolaris.zone" in attrs and \
"variant.opensolaris.zone" not in attrs:
attrs[
"variant.opensolaris.zone"] = \
attrs[
"opensolaris.zone"]
if aname ==
"set" and attrs[
"name"] ==
"authority":
# Translate old action to new. attrs[
"name"] =
"publisher" # add any set actions to attributes """Fill attribute array w/ set action contents.""" # To reduce manifest bloat, size and csize # are set on a single action so need splitting # into separate attributes. # ignore broken set actions # Ensure facet and variant attributes are always lists. # Possible facets list is spread over multiple actions. # Prevent duplicates; it's possible a # given facet may be valid for more than # one unique variant combination that's # allowed by current excludes. # Ancient manifest compatibility. """Produces the search dictionary for a specific manifest. A dictionary is constructed which maps a tuple of token, action type, key, and the value that matched the token to the byte offset into the manifest file. file_path is the path to the manifest file. excludes is the variants which should be allowed in this image. return_line is a debugging flag which makes the function map the information to the string of the line, rather than the byte offset to allow """Translates what actions.generate_indices produces into a dictionary mapping token, action_name, key, and the value that should be displayed for matching that token to byte offsets into the manifest file. The "lst" parameter is the data to be converted. The "cp" parameter is the byte offset into the file for the action which produced lst.""" """This method takes a string representing the on-disk manifest content, and returns a hash value.""" # This must be an SHA-1 hash in order to interoperate with # Byte stream expected, so pass encoded. """Verifies whether the signatures for the contents of the manifest match the specified signature data. Raises the 'BadManifestSignatures' exception on failure.""" """Store the manifest contents to disk.""" # We specifically avoid sorting manifests before writing # them to disk-- there's really no point in doing so, since # we'll sort actions globally during packaging operations. """Return a dictionary mapping variant tags to their values.""" """Returns the boolean of the value of the attribute 'key'.""" raise ValueError(_(
"Attribute value '{0}' not 'true' or " """Returns an integer tuple of the form (size, csize), where 'size' represents the total uncompressed size, in bytes, of the Manifest's data payload, and 'csize' represents the compressed 'excludes' is a list of a list of variants and facets which should be allowed when calculating the total.""" if (
"pkg.size" in attrs and "pkg.csize" in attrs)
and \
# If specified excludes match loaded excludes, then use # cached attributes; this is safe as manifest attributes # are reset or updated every time exclude_content, # set_content, or add_action is called. # Cache for future calls. # Facet filtering should never be applied when excluding # actions; only variant filtering. This is ugly, but # allow you to be selective and various bits in # pkg.manifest assume you always filter on both so we # have to fake up a filter for facets. # Excludes list must always have zero or 2+ items; so # append any variants and facets to manifest dict # While variants are package level (you # can't install a package without # setting the variant first), facets # from the current action should only be # included if the action is not # Lists can't be set elements. _(
"{forv} '{v}' specified multiple times").
format(
"""Return the value for the package attribute 'key'.""" """Set the value for the package attribute 'key' to 'value'.""" """This class serves as a wrapper for the Manifest class for callers that need efficient access to package data on a per-action type basis. It achieves this by partitioning the manifest into multiple files (one per action type) and then storing an on-disk cache of the directories explictly and implicitly referenced by the manifest each tagged with """Raises KeyError exception if factored manifest is not present and contents are None; delays reading of manifest until required if cache file is present. 'fmri' is a PkgFmri object representing the identity of the 'cache_root' is the pathname of the directory where the manifest and cache files should be stored or loaded from. 'contents' is an optional string to use as the contents of the manifest if a cached copy does not already exist. 'excludes' is optional. If provided it must be a length two list with the variants to be excluded as the first element and the facets to be exclduded as the second element. 'pathname' is an optional string containing the pathname of a manifest. If not provided, it is assumed that the manifest is stored in a file named 'manifest' in the directory indicated by 'cache_root'. If provided, and contents is also provided, then 'contents' will be stored in 'pathname' if it does not already # Make sure that either no excludes were provided or 2+ excludes # Do we have a cached copy? # we have no cached copy; save one # don't specify excludes so on-disk copy has # we have a cached copy of the manifest # have we computed the dircache? """Load all manifest contents from on-disk copy of manifest""" """Unload manifest; used to reduce peak memory comsumption when downloading new manifests""" """Finish loading.... this part of initialization is common to multiple code paths""" """ store the current action set; also create per-type caches. Return True if data was saved, False if not""" # this allows us to try to cache new manifests # when non-root w/o failures. """ create manifest.<typename> files to accelerate partial parsing of manifests. Separate from __storeback code to allow upgrade to reuse existing on disk manifests""" # Ensure target cache directory and intermediates exist. # create per-action type cache; use rename to avoid corrupt # files if ^C'd in the middle. All action types are considered # so that empty cache files are created if no action of that # type exists for the package (avoids full manifest loads # Add supplemental action data; yes this # does mean the cache is not the same as # retrieved manifest, but that's ok. # Signature verification is done using """Remove all manifest cache files found in the given directory (excluding the manifest itself) and the cache_root if it is # Ensure cache dir is removed if the last cache file is # removed; we don't care if it fails. # Only raise error if failure wasn't due to # cache directory not existing. """Private helper function for loading arbitrary cached manifest # we have cached copy on disk; use it # Cache file is malformed; hopefully due to bugs # that have been resolved (as opposed to actual # corruption). Assume we should just ignore the # cache and load action data. # Ignore errors encountered during cache # dump for this specific case. """ return a list of directories implicitly or explicitly referenced by this object """ generate actions of the specified type; use already in-memory stuff if already loaded, otherwise use per-action types files""" if self.
loaded:
#if already loaded, use in-memory cached version # invoke subclass method to generate action by action # This checks if we've already written out the factored # manifest files. If so, we'll use it, and if not, then # we'll load the full manifest. # invoke subclass method to generate action by action # No such action in the manifest; must be done *after* # asserting excludes are correct to avoid hiding # Assume a cached copy exists; if not, tag the action type to # avoid pointless I/O later. # These conditions are split by return # no such action in this manifest """A generator function that returns the supported facet attributes (strings) for this package based on the specified (or current) excludes that also match at least one of the patterns provided. Facets must be true or false so a list of possible facet values is not returned.""" """A generator function that yields a list of tuples of the form (variant, [values]). Where 'variant' is the variant attribute name (e.g. 'variant.arch') and '[values]' is a list of the variant values supported by this package. Variants returned are those allowed by the specified (or current) excludes that also match at least one of the patterns provided.""" """A generator function that yields set actions expressing the set of possible mediations for this package. """Load attributes dictionary from cached set actions; this speeds up pkg info a lot""" """Returns an integer tuple of the form (size, csize), where 'size' represents the total uncompressed size, in bytes, of the Manifest's data payload, and 'csize' represents the compressed 'excludes' is a list of a list of variants and facets which should be allowed when calculating the total.""" """No assignments to factored manifests allowed.""" assert "FactoredManifests are not dicts" """Store the manifest contents to disk.""" """The absolute pathname of the file containing the manifest.""" """Special class for pkgplan's need for a empty manifest; the regular null manifest doesn't support get_directories and making the factored manifest code handle this case is """Return three lists of action pairs representing origin and destination actions. The first list contains the pairs representing additions, the second list contains the pairs representing updates, and the third list contains the pairs representing removals. All three lists are in the order in which they should be executed.""" # The difference for this case is simply everything in the # origin has been removed. This is an optimization for # This method is overridden so that self.excludes is never set # on the singleton NullFactoredManifest. """Simple Exception class to handle manifest specific errors"""