Userland Consolidation Packaging Guidelines. Each component that integrates into the Userland consolidation must have at least one package manifest that describes the content to be delivered. In some cases components *may* deliver through multiple packages. Canonical component package manifests must be placed in the component's build directory. They also must be named *.p5m. In order to understand what must go in the content of a package manifest, it's useful to have an understanding of how a canonical manifest is transformed into a final manifest used for package publication. Manifest transformation takes the following basic path: canonical manifest (.../{component}/{component}.p5m) | v mogrified manifest (.../{component}/{build-dir}/manifest-$(MACH)-{component}.mogrified) | v mangled manifest file contents (.../{component}/{build-dir}/manifest-$(ARCH)-{component}.mangled) | v dependencies generated (.../{component}/{build-dir}/manifest-$(MACH)-{component}.depend) | v dependencies resolved (.../{component}/{build-dir}/manifest-$(MACH)-{component}.depend.res) | v manifest validation (.../{component}/{build-dir}/.linted-$(MACH)) | v publication manifest (.../{component}/{build-dir}/manifest-$(MACH)-{component}.published) | v publication Canonical Manifest The canonical manifest contains actions that can't otherwise be generated automatically from the data encapsulated in the component Makefile, gate transformations, build tree, and packaging tools. This includes actions for license information, some path related attributes, legacy actions, non-discoverable dependencies, users, groups, drivers, and others. Actions that are associated with objects that are specific to a single architecture should be tagged with a 'variant.arch' attribute specific to the architecture that applied to the action. Ex: file path=/usr/lib/$(MACH64)/libx86onlybits.so variant.arch=i386 Actions for editable files must include an appropriate 'preserve' attribute: file path=etc/gnu/a2ps.cfg preserve=true mode=0644 license actions should be placed in the canonical manifest. Manually generated actions * com.oracle.info.description is a terse description of what utilities, libraries and/or services the package provides. This should be short, specific, concise text, identifying the technology covered by the associated license(s). It should fit naturally in the sentence "This package may contain XXX." For example, "XXX" might be "the tar command" or "bzip2 compression software." When appropriate, this may begin with "portions of" or another, more specific qualifying clause. * com.oracle.info.tpno is the Oracle 3rd party license number. * info.classification is "org.opensolaris.category.2008:FOO" where FOO varies according to the sorts of utilities, libraries and/or services that the package provides. Existing packages contain most useful values; check them out to find the closest match. For a complete list of allowed values, refer to the Solaris system file /usr/share/lib/pkg/opensolaris.org.sections . * org.opensolaris.arc-caseid is typically "PSARC/YYYY/###" and multiple different values are allowed. * pkg.summary is a short synopsis of what the package provides. * org.opensolaris.consolidation is the name of the consolidation delivering the package. In Userland, this is $(CONSOLIDATION) (which expands to "Userland" during the build). Manifests in the Userland gate can also decorate this package attribute with an 'incorporate={incorporation-name}' decoration to specify where the package should be incorporated at the end of the userland build. A special value of 'none' will cause the package to be unincorporated and float freely from the rest of the rest of the packages. Note that unincorporated packages don't automatically get updated with the rest of the system when 'pkg update' is run unless the unincorporated package(s) are specified on the command line. Mogrified Manifest The canonical manifest is combined with a set of the transforms in $(WS_TOP)/transforms, and a set of macros to more complete package manifest using pkgmogrify(1). The transforms apply default attributes to the various actions in the canonical manifest(s). More detail about the attributes can be found in the transform file themselves. The macros applied at the time of mogrification are as follows: $(MACH) $(MACH32) $(MACH64) $(PUBLISHER) $(CONSOLIDATION) $(BUILD_VERSION) $(SOLARIS_VERSION) $(OS_VERSION) $(IPS_COMPONENT_VERSION) $(COMPONENT_VERSION) $(COMPONENT_PROJECT_URL) $(COMPONENT_ARCHIVE_URL) Dependencies Generated The mogrified manifest and the prototype install tree are passed through pkgdepend(1) to generate a set of dependencies for the package content. These dependencies are only those that "pkgdepend generate" can determine on its own. Additional dependencies that cannot be automatically determined by pkgdepend(1) should be placed in the canonical manifest. Statically defined dependencies should be described in a canonical manifest in an unresolved form (ie. the form generated by "pkgdepend generate"). Ex: depend fmri=__TBD pkg.debug.depend.file=etc/passwd \ pkg.debug.reason=usr/bin/vipw type=require depend fmri=__TBD pkg.debug.depend.file=sh \ pkg.debug.depend.path=usr/bin \ pkg.debug.depend.reason=usr/bin/psmandup \ pkg.debug.depend.type=script type=require This will allow the next step to resolve all dependencies to their proper package(s). Dependencies Resolved The manifest with unresolved dependencies is passed through pkgdepend(1) again to resolve dependencies against the package repositories. The result is a manifest that is suitable for publication. All these manifests are processed together in a single step, which is more efficient than resolving dependencies in each manifest separately. While each manifest ends up with a .depend.res copy in the build directory, the umbrella dependency resolution target is {build-dir}/.resolved-$(MACH). The resolution step is also set up to use the -e flag to pkgdepend resolve, which limits the set of packages it looks at to resolve the dependencies it generated in the previous step. This makes the resolution step a great deal faster, but requires that you keep a static list of these packages checked into the workspace, and update it when packages are added to it. Having extra packages in there is safe. In order to create this list, build and publish your component (or at least through the resolution stage) without a file "resolve.deps" in the component directory, and run "gmake sample-resolve.deps". If the file is empty (that is, no computed dependencies were found), a warning will be emitted and the file will be removed, as pkgdepend currently errors out in that case. To test, run "gmake clean" and re-publish. Don't forget to "hg add resolve.deps"! Note that there is a possibility the list of dependencies will be different on different architectures, so you should run this on both sparc and x86, and combine the two lists. Please keep the files sorted. Manifest Validation The resolved manifest(s) and prototype install tree are passed through a set of validations. This includes running pkglint(1), comparing the manifest content to the prototype install tree, and validation of the file content of the prototype install tree. Any anomalies are reported. Content validation is performed by extension to pkglint(1) in $(WS_TOP)/tools/python/userland-lint Note that when integrating new packages, and one or more of them depends on or more of the others, then this may result: WARNING pkglint.action005.1 obsolete dependency check skipped: unable to find dependency (target pkg) for (source pkg) This means that the target package was not found in the reference repo, which was the source of the pkglint cache that was created when 'gmake setup' was run after the workspace was created; thus the warning is harmless and can be ignored in this circumstance. Publication. Once manifest validation has occurred, the package(s) is/are finally published to the workspace package repository. Renames Renames in IPS are tricky. We will use a case study to illustrate how this needs to work. All of the library/python-2/FOO modules that we had in Userland at the time were renamed to library/python/FOO in s12_41. This involved: * changing the FMRI to drop the "-2" * adding an optional dependency on the old name at the version and build in which the rename occurred; this forces the rename * depend type=optional fmri=library/python-2/FOO-$(PYV)@VERSION,BUILD where FOO is the component name (e.g., "alembic", "amqp", etc.), "$(PYV)" is that literal string, VERSION was the expanded value of COMPONENT_VERSION from each component's Makefile (e.g., "0.6.0" for alembic, "1.0.12" for amqp, etc.), and BUILD was set to "5.12-5.12.0.0.0.41.0". Note that BUILD needs to be set to the value of the build you are integrating into. * creation of the file "history" (or adding to it if it already exists) in each affected component directory; one or more new lines should be added (typically one for the versionless package, plus one for each versioned instance of the package) with syntax: OLD-NAME@VERSION,BUILD NEW-NAME for renames and: OLD-NAME@VERSION,BUILD for obsoletions, where FOO, VERSION and BUILD are all as above. Most Python modules, including the two mentioned above, have such files which can serve as examples. By default, packages created from the history files will be part of the userland-incorporation. If a package needs to be included in a different incorporation, then incorporate=INCORPORATION may be placed on the end of either the rename or obsoletion line. Examples with incorporate=consolidation/desktop/gnome-incorporation may be found in components/meta-packages/history/history. It's possible to specify whether the record applies to just i386 or sparc by providing arch=i386 or arch=sparc. So far, not too bad. This gets slightly more complicated when back- porting in that BUILD needs to be set to the back-port build (e.g., "5.11-0.175.3.0.0.8.0", as was the case for tkinter-27) rather than the S12 build. If the component is at the same version as when the rename was done, then that's it. But if the component has since been upgraded, then care must be taken to set VERSION in both places above to the value that it was in s12_41 when the rename was originally done. In particular, the latest version of a package name in an earlier release branch must not be greater than the latest version of that same package name in the newer release branch. Otherwise the audits from Release Engineering will complain and our gatekeeper will make you do a follow-up push to fix this. ============================= Component EOF (End Of Feature) ============================= This is a brief description on how to remove a component from Userland for an EOF. Deleting the Component ---------------------- At some point, you will need to do a "hg remove" of the component's directory. This can be done before or after you make the necessary entry in the components/meta-packages/history/history file. In this file you will see plenty of examples like this: backup/areca@7.1-5.12.0.0.0.85.0 You will need to make sure you have the correct package name, version, and build you are targeting for the EOF for your new entry. Also check for an old SUNW entry at the top of the file. If there is one you will need to change this to match the targeted build. So something like this: SUNWareca@7.1,5.11-0.133 backup/areca@7.1-0.133 Should be changed to: SUNWareca@7.1-5.12.0.0.0.85.0 Remember that the package may have had numerous names in its history. Generally the way to find this is by using: pkg search -o search.match_type,pkg.name depend:: First with the package to be EOF'ed, and then examining each of the dependent packages. If there are require dependencies on any of the dependents, then the latter should be examined to see if it need to be either EOF'ed at the same time, or at least updated to remove the dependency you're trying to EOF. Check to see if the package you are removing is referenced by a group package (pkg list -n 'group/*'). See the 'EOFs and Removals' and 'Group Packages' sections in /usr/src/pkg/README.pkg from the on-gate for further details. Testing ------- To test your changes the first thing you will need to do is to invoke 'gmake publish' from the components/meta-packages/history directory or at the top level (a full Userland build) if you are removing multiple components. Once this is complete you will need to setup a client with the EOF package(s) installed. Then perform the following steps: 1. # pkg set-publisher \ -P -p file:///net/strax/builds/your_name/your_dir/i386/repo If you are removing just a single component: 2. pkg change-facet 'facet.version-lock.=false' 3. pkg update pkg://nightly/ After the update the component package should be removed from the test system. If you are removing multiple components: 2. pkg change-facet \ version-lock.consolidation/userland/userland-incorporation=false 3. pkg update pkg://nightly/consolidation/userland/userland-incorporation After the update the EOF component packages should be removed from the test system. For languages and anything else that has modules or is otherwise part of the Userland build environment you will need to do a full build, of both ISAs, on the test machines you just updated. # vi:set fdm=marker expandtab ts=4: