IPS for Consolidations

Requirements and Advice for Consolidations Delivering IPS repositories

This page is a work in progress.  Advice and requirements here are not yet final.

Consolidations which deliver to the OpenSolaris/Solaris Release Engineering (RE) docks must transition from delivering SVR4 packages to delivering IPS repositories.  This page captures the technical requirements for that transition, and expects basic familiarity with existing consolidation delivery practices.

• Requirements
  • General Requirements
  • Repository Format Requirements
  • Package Requirements
  • Obsoletes, Renames, and File Move Requirements
• Transition Process
• Ongoing Requirements

Requirements

General Requirements

• Consolidations must be prepared to re-deliver throughout this release with some frequency.  Consolidation assembly is a work in progress, and the distribution will need a few updates from all consolidations as new IPS features become available to make the distribution better for customers overall.

• IPS repositories should be built from source manifest files.  A consolidation is expected to create these manifests, and use them to populate a repository during the consolidation packaging build.  RE will *NOT* accept repositories created by a build-time conversion from SVR4 packages using pkgsend(1) or the IPS importer.py.

• Consolidations are expected to deliver all packages that are currently identified as from OpenSolaris/Solaris in their consolidation, including those that were previously renamed or obsoleted.  If only a subset of packages will be delivered initially, please coordinate with the release lead to confirm that expectations are being met.  This list can be retrieved from the current development package repository by reviewing the contents of consolidation/<consolidation name>/<consolidation name>-incorporation.

• Consolidations must use pkgdepend(1) to generate dependencies.  Additional manually-added dependencies are permitted, but the bulk of dependencies are expected to be generated automatically.  In general, dependencies between files should first be generated using "pkgdepend generate".  After this has completed, "pkgdepend resolve" can be run against all of the updated package manifests to build inter-package dependencies.

• "pkgdepend resolve" must run cleanly for your consolidation.

• pkglint(1) is a command that can be used to check for common packaging errors.  Consolidations should be delivering repositories that are "pkglint clean" to RE.  Any errors should be discussed with the release lead and the IPS distribution team.

• pkgfmt(1) is another useful tool to make sure that all manifests have a common and generally readable format.  We recommend, though do not require, consolidation makefiles run "pkgfmt -c" to ensure manifests stay formatted.

• The extension .p5m is recommended, though not required, for your manifest files.

Repository Format Requirements

• In order to make testing for your gatelings easy, set your publisher name to a unique name for your consolidation.  For example, ON uses on-nightly.  RE will take care of translating this to the official publisher name during their build assembly.

• Consolidations will no longer integrate individual packages for each architecture.  Instead, they'll deliver an IPS repository to RE which contains the complete set of consolidation packages for that build.

• Consolidations should deliver merged repositories and not separate ones for each architecture variant. The pkgmerge(1) command (available from build 160 as /usr/bin/pkgmerge) can be used to merge an i386 and sparc repository together. All packages, even those available on only one architecture, should be merged so their contents is correctly variant tagged.

• Consolidations should deliver a package named consolidation/<consolidation name>/<consolidation name>-all which has require dependencies on all packages delivered by that consolidation that are neither obsolete nor renamed.  This group package should contain a valid pkg.description and pkg.summary as well as a info.classification value of the form "org.opensolaris.category.2008:Meta Packages/Group Packages" (a more specific classification is preferred if available.)

Package Requirements

• pkgmogrify(1) is a very useful tool to set consolidation-level defaults and transformations, such as the ones required in this section.  It allows addition of package-level or even file-level defaults without editing each manifest.

• Every package delivered by a consolidation must specify the consolidation it was delivered from in the org.opensolaris.consolidation key.  If in doubt about the appropriate consolidation name, check the existing OpenSolaris/Solaris development builds for the correct setting.  For example, set name=org.opensolaris.consolidation value=osnet.

• Generally, each package delivered by a consolidation must depend on its consolidation incorporation.  If this is not planned for your consolidation, discuss the details with the release lead.

• Take care to set your variants correctly.  Make sure that previously "hollow" packages deliver with the global zones variant.

• Package names must currently be vetted by the OpenSolaris/Solaris release lead (ARC is expected to take over management of this namespace in the near future.)

• pkg.description and pkg.summary are expected to be useful for end-users.  The package summary is a short description of the package, suitable for display on one line.  Figure on 40 characters as a rough upper bound.  The description should be roughly a short paragraph in length, perhaps including a couple of sentences.  If there's no reason to have a description of that length, then there may be no need for one at all.  Certainly having a description identical to the summary is not useful (in that case, drop the description.)

• Packages must be classified with info.classification, and classifications must conform to one of the existing GUI classifications.  The current list of GUI classifications can be found in /usr/share/package-manager/data/opensolaris.org.sections.  New classifications are not permitted without coordination with the OpenSolaris/Solaris release lead.

• The value of the info.classification attribute should be of the form "org.opensolaris.category.2008:<category/subcategory>" (the only valid year presently is 2008.)

• Each package must have one or more license actions.

• We recommend default owners, groups, and modes for common consolidation directory trees are set using pkgmogrify rather than repeated explicitly for each file.

• Each package should continue to deliver the legacy action as delivered by OpenSolaris/Solaris today.  This defines compatibility for SVR4 package dependencies, and does not need to be changed unless if the package boundaries are changing.

Obsoletes, Renames, and File Move Requirements

• Renames must be done with care.  Specifically, any files with the preserve attribute set which belong to a renamed package must be explicitly mentioned in the new package.  See the next point about preserved files which move between packages.

• The first time a preserve=true file moves between packages, set "original_name" for that file to "<original package>:<file>".

• Package obsoletion must be done with great care.  Please make sure before doing any obsoletes that you've coordinated with any other consolidations which have dependencies on your packages.  Package obsoletions are done by delivering an updated package within your repository.  *ALL* packages which are renamed ancestors of an obsolete package must also be obsoleted in the same build.  If this isn't done properly, your obsolete package will cause failures to upgrade for our customers.  For example, if package A had been renamed to package B, and later package B and been renamed to package C, and then eventually package C is to be obsoleted, package A, B and C should be obsoleted in the same build.

• When obsoleting a package, the branch id of the version should be set to the build the obsoletion took place (for example, 1.2.3,5.11-0.158 for build 158.)  In addition to removing the obvious actions in the manifest, the pkg.description, pkg.summary and info.classification set actions should also be removed from all obsolete and renamed packages.

Transition Process

Generally speaking, the transition from delivering SVR4 to delivering IPS repositories involves a few steps.

1. Convert your consolidation per the steps above using a known baseline.
2. Compare using pkgdiff(1) that your deliveries do not deviate from the current OpenSolaris/Solaris baseline.  Any differences should be scrutinized and potentially discussed with the release lead and the IPS distribution team.
3. Work with the IPS distribution team to do a test run and get a review.  You can usually do a test run yourself by modifying src/util/distro-import/Makefile in the IPS gate to add new CONSOLIDATION_OPTIONS values for your consolidation.  Contact the IPS team if you need help.
4. Coordinate final delivery with RE and the IPS distribution team so that your packages are no longer expected to be built by the IPS importer.py command.
5. Obsolete the old SVR4 packages by using RE's pkgobs command.

Ongoing Requirements

Once the initial transition is complete, all of the transition requirements remain.  There are a few additional requirements to consider for some less-common consolidation operations.

• Package Refactor/Split
  • Package boundaries should represent minimization boundaries.  They represent a feature which is complete when combined with its dependencies.  Optional components (e.g. graphical libraries, etc.) should be split into separate packages.
  • Many packages in the WOS today do not reflect these boundaries, so we expect some splitting and refactoring over time.
  • When splitting a package with existing legacy actions, care must be taken with those actions.  If the legacy action reflects a common SVR4 package dependency, and that dependency is not satisfied by any one of the new packages, a legacy package might need to be created which specifies dependencies on each of the required new packages.  Check with the release lead when doing significant refactoring work.
  • Work is currently in progress to create a new dependency type which will mitigate some of the problems with splitting packages.

• Move Between Consolidations
  • Moving a package between consolidation needs to be done with some care.  The same files cannot be delivered by 2 consolidations at the same time.  These instructions provide a way to keep the WOS working, as well as not disrupt developers of either consolidation.
  • Put the package into the new consolidation.  When doing this, don't change the package contents, the package name, or the package version.  Explicitly put the old version number from the old consolidation into the package in the new consolidation.
  • *IN THE SAME BUILD* delete the package from the old consolidation.
  • After that WOS build has been available to developers for a bit (expect 2-3 builds will pass), do a putback to the new consolidation which changes the package version back to the default version, rather than the old one.  At this point, you may make any changes necessary to the package contents.
  • At the time of this putback, send a FLAG DAY to the developers of the new consolidation telling them they must upgrade to at least the WOS build chosen above to be able to upgrade to this putback. 

• Cross-Consolidation Dependency Introduction/Update
  • Make sure cross-consolidation dependencies are available in a published WOS build before introducing the dependent code into a consolidation.
  • Integrating a dependency and a dependent in the same build causes problems for developers in the dependent consolidation.  Only do this if ýou have told developers in the dependent consolidation how to navigate the potentially difficult transition.