Developer information » Integration Notes
en

Integration Notes

Below you will find a list of repository revisions that have "Heads Up" notices or other upgrade considerations users and/or developers should be aware of.

Notable Revisions

Revision 2268 / 1f313c3e7cdf

With the putback of changeset 2268:1f313c3e7cdf to the pkg(5) gate, publication tools such as pkgsend, pkgrecv, and pkgsign now always require that a destination repository be specified for operations that require it using environment variables or options to the program.  pkgsign also accepts filesystem-based repository locations in path form.

What changed?

For historical reasons that are no longer understood, publication tools such as pkgrecv, pkgsend, and pkgsign previously would automatically attempt to connect to a pkg.depotd(1M) process on port 10000 if a target repository was not specified either through environment or program options.

This is no longer the case.  (This behaviour was never intended for long-term use.)

Operations that publication tools perform that require a destination repository will now always exit with an error if the user has not specified a destination repository.

Additionally, pkgsign now accepts relative and absolute paths to filesystem-based repositories instead of requiring that their location is in URI form.

How could this change impact me?

If a consumer has relied on the publication tool's historical (and sometimes undocumented) behaviour of defaulting to port 10000 for a destination repository, whatever publication operations it was performing will now fail with a usage error.

The consumers in question must be updated to either set the relevant environment variables to indicate the destination repository, or must specify the destination repository using the relevant options to the publication tool in question.

See pkgsend(1), pkgrecv(1), or pkgsign(1) for more information on how to specify a destination repository for each tool. 

What are the related bug entries?

The following RFEs/Bugs were resolved by changeset 2268:

  • 17728 publication tools should require a repository to be specified
  • 18041 pkgsign should accept paths for filesystem-based repositories

Revision 1895 / 0a260cc2a689

With the putback of changeset 1895 to the pkg(5) gate, pkg(1) now supports filesystem-based repository access.  Please note that this functionality is not expected to be available in an OS build until after b139.

What changed?

There are no user-visible changes to command syntax or client operation.  The only noticeable change is that any command that accepted a repository URI such as 'http://localhost:10000' now also accepts URIs for filesystem locations such as 'file:/path/to/my/repository'.

For example, if you were creating a new image and had a repository located at '/export/myrepo', you could do this:

pkg image-create -p file:/export/myrepo /path/to/image

See the pkg(1) man page for more examples.

What does this mean?

This change means that pkg.depotd(1m) is no longer required for serving package data if a client can access the package repository via the filesystem (e.g. NFS mounts, etc.). 

What are the related bug entries?

The following RFEs/Bugs were resolved by changeset 1895:

  • 15762 client support for filesystem-based repository access
  • 10244 caching dictionaries as a class variable prevents multi-image and repo search
  • 14802 ability to have separate read / write download caches
  • 15763 test suite should not start depot server unless necessary
  • 15764 file_manager insert needs to stat() less and return more

What are the potential issues?

Currently, the client will not cache data from a repository that is accessed via the filesystem in the image's download cache (e.g. /var/pkg/download), with the assumption that access to the repository via filesystem is reliable and/or reasonably fast.

In some use case scenarios (such as NFS, SMB, etc.) filesystem-based access may be slow, unreliable, or otherwise unsuitable for client transport.  In those cases, it is advised that http-based access to the repository via pkg.depotd(1m) is used instead.  It is hoped that the client can account for the performance of available filesystem-based resources automatically in the future when retrieving package content.

Search operations performed using a file-based repository can use substantially more memory when a repository contains a large number of packages (such as /dev).  This is because the client is essentially acting as the search server in addition to the client.

To avoid this overhead, the related publisher can be configured with both an http(s) origin and the file-based origin.  When performing search operations, the client will use the http(s) based origin(s) first to perform searches, but check the file-based origin first for all package file retrieval operations.  For example:

pkg set-publisher -g file:/export/myrepo \
  -g http://localhost:10000 example.com

Thanks

Many thanks to the individuals who collaborated on, reviewed, or contributed to this project.

Feedback is welcomed on the pkg-discuss mailing list.

Revision 1736 / 727dda1e7968

Changeset 1736 in the pkg(5) gate made the following changes:

  • 11522 pkg should require publisher prefix to match repository information
  • 7156 client image api needs image creation interface
  • 12744 update_publisher over-zealously testing publisher validity
  • 14203 image-create usage doesn't mention mirror / origin options

What changed?

The pkg(1) client and the pkg.client.api will now attempt to retrieve publisher configuration information from package repositories.  This information is used to auto-configure new publishers, validate configuration requests, and to update the configuration of existing publishers.

These changes were made with a focus on simplifying publisher addition and updates of existing publisher configuration.

Further improvements and additions to this functionality are planned for future releases.

Are there any pkg.depotd(1m) changes?

While the pkg.depotd(1m) program did not change, clients are now more reliant on its correct configuration.

As already mentioned in the past, please be certain that you set the publisher.prefix property found in the repository's cfg_cache file correctly.  The pkg.depotd man page contains instructions on how to set this property.

Please also note that the repository.origins and/or repository.mirrors properties should be set as clients will use these to automatically configure new publishers and to update existing publishers.  You might also want to consider providing a description so that users have an idea of what sort of packages the repository contains.

Part of an example cfg_cache might look like this:

[publisher]
prefix = example.com
alias = None
...
[repository]
origins = http://pkg-us.example.com/,http://pkg-ca.example.com/
mirrors = http://pkg-us-mirror1.example.com/,http://pkg-ca-mirror1.example.com/
...

How has pkg(1) image-create changed?

The client remains completely compatible syntax-wise, so no changes to existing scripts are required, and behaviour when using the older syntax (other than validating publisher configuration) remains unchanged.

However, image-create now also accepts -p <uri> where <uri> is the URI of a package repository and example usage is as follows:

pkg image-create -p <uri> /target

When using this syntax, image-create will retrieve all of the publisher's configuration information from the target package repository and add all of the publishers it finds to the image.

How has pkg(1) set-publisher changed?

The syntax of the set-publisher command remains backwards compatible, so no changes to existing scripts, etc. are required.  However, set-publisher now accepts the -p option which accepts the URI of a package repository from which all publisher configuration information will be retrieved.

Any new publishers found in the retrieved configuration information will be added, while existing ones will be updated if the provided URI is already in the list of configured origins for the publisher being updated.

If a publisher name is provided, then -p will only use publisher configuration information that matches the provided name.

As an example, the old syntax to add the contrib repository might have been:

pkg set-publisher -g http://pkg.opensolaris.org/contrib \
  contrib.opensolaris.org

The new syntax is:

pkg set-publisher -p http://pkg.opensolaris.org/contrib

How has pkg.client.api changed?

Incompatible with clients using API versions 0-31.

The ImageInterface class has changed as follows:

  • The add_publisher and update_publisher methods now validate the image's publisher configuration against the origins of the publisher.  If any of the origins are found to not match, an UnknownRepositoryPublishers exception will be raised.  If one of the new repository origins does not provide publisher configuration information or it is incomplete, a RepoPubConfigUnavailable exception will be raised.

The pkg.client.api module has changed as follows:

  • A new method named image_create has been added.  See 'pydoc pkg.client.api' for details.

The pkg.client.api_errors modules has changed as follows:

  • UnknownRepositoryPublishers, RepoPubConfigUnavailable, and UnknownErrors expections have been added for use by the pkg.client.api.  API consumers are reminded that they should catch all ApiException class exceptions, although catching specific exception subclasses for case-by-case handling in addition to that is acceptable.

Feedback is welcomed on the pkg-discuss mailing list.

Revision 1618 / 04f7471c9e72

The following fixes and enhancements were integrated:

  • 5943 add 'must-accept' attribute to license actions
  • 5586 licenseinfo api needs to expose license action attributes
  • 13155 add 'must-display' attribute to license actions
  • 13158 change pkg.client.api to understand and require license acceptance
  • 13160 pkg(1) needs update for client api license acceptance changes 

These changes were made to support packages that require display or acceptance of license related data during package install and update operations.  As a result of these changes, the pkg(1) client and client API require explicit acceptance and/or indication of license display during install and update operations.

Please note that (currently) no packages exist that use this functionality, but they are expected in the near future.

Client Considerations

If any of the license actions contained in a package being installed or updated have must-accept=true, the following pkg(1) subcommands require that the new --accept option be provided before the operation will proceed:

  • install
  • fix
  • image-update
  • change-variant
  • change-facet

In addition, all of the above subcommands also now have a --licenses option to display the payload of all the licenses for packages part of the operation.  For example:

pkg install -n --licenses foo

The above command would display all of the licenses for the packages that would be installed or updated if the package 'foo' were installed or updated.  If the --accept option is not provided, and a license requires acceptance, the pkg(1) client will now exit with exit code 6, indicating license acceptance failure.  If a license requires display, the pkg(1) client will display it during install/update operations; this cannot be suppressed.

Client API Changes

Version 29:
Incompatible with clients using versions 0-28:
    The ImageInterface class has changed as follows:

  • set_plan_license_status() was added.  This is used to indicate
              whether licenses for the packages being operated on have been
              accepted or displayed.  Clients must do this if the related
              license requires acceptance or display.

    The LicenseInfo class has changed as follows:

  • get_text() may now trigger a remote retrieval of the license
              payload if needed to return the text.
  • The related package FMRI and license attributes are now
              properties: fmri, license, must_accept, and must_display.

    The PlanDescription class has changed as follows:

  • get_changes() is now a generator function.
  • get_licenses() was added to allow clients to retrieve the
              list of licenses related to the plan's operations as well
              as the current accepted and displayed status of each.  Please
              note that this function returns _all_ licenses related to the
              operation; not just those that require acceptance or display.

Publication Considerations

To use this new license acceptance functionality, simply add must-accept=true or must-display=true (as appropriate) to license actions in your package manifest.  An example pkgsend sequence might look like this:

open licensed@1.3,5.11-0
add depend type=require fmri=baz@1.0
add file /tmp/libc.so.1 mode=0555 owner=root group=bin path=/lib/libc.so.1
add license /tmp/libc.copyright license=libc.copyright must-display=True
add license /tmp/libc.license license=libc.license must-accept=True
close

Please note that this functionality is not supported before build 131, and that you should use this functionality sparingly.  must-accept=true should not be placed on the majority of open source licenses (BSD, GPL, etc.) and must-display should only be set if absolutely necessary. 

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Revision 1504 / 265a1d6b86bd

With the putback for the following issues in revision 1504:

  • 9969 client support for multiple origins desired
  • 11715 ImageConfig does not handle None for publisher correctly
  • 11793 image-create example partially disagrees with the usage

...the client now supports multiple origins for publisher repositories.  The purpose of this change was to enable greater pkg (5) server availability and redundancy.

What is an origin?

An origin is simply a location of a package repository that contains package file, manifest, and catalog data; such as: http://pkg.opensolaris.org/dev.  The default mode for the pkg.depotd process is to run as an 'origin'.

A mirror, in contrast, only contains package file data and is currently provided by a pkg.depotd server running in mirror mode.

What does this change mean?

This change means that redundancy can now more easily be built into package server infrastructure.  For example, you could have http://pkg1.example.com/, http://pkg2.example.com/, and http://pkg3.example.com/.

A client, once configured, would have pkg publisher output similar to the following:

PUBLISHER                TYPE   STATUS URI
example.com  (preferred) origin online http://pkg1.example.com/
example.com  (preferred) origin online http://pkg2.example.com/
example.com  (preferred) origin online http://pkg3.example.com/

During package operations, the client will automatically attempt to select and use the best repository origin if any data needs to be retrieved for that publisher.

In addition, ipkg branded zone creation and attach has been updated to use this additional origin and mirror information when configuring zones.

How do I use this?

For consistency, and to support this new client functionality, the image-create and set-publisher subcommands for pkg(1) have changed as follows:

image-create
 New Options:
  [-g|--origin ...] [-m|--mirror ...]

set-publisher
 New Options:
  [-g origin_to_add | --add-origin=origin_to_add ...]
  [-G origin_to_remove | --remove-origin=origin_to_remove ...]

 Deprecated Options:
  [-O uri]

So, as an example, to create a new, full image, with publisher example.com, that also has an additional mirror, two additional origins and that is stored at /aux0/example_root:

$ pkg image-create -F -p example.com=http://pkg.example.com:10000 \
  -g http://alternate1.example.com:10000/ \
  -g http://alternate2.example.com:10000/ \
  -m http://mirror.example.com:10000/ \
  /aux0/example_root

Alternatively, to add a new publisher or update an existing publisher with the above information:

$ pkg set-publisher \
  -g http://pkg.example.com:10000 \
  -g http://alternate1.example.com:10000/ \
  -g http://alternate2.example.com:10000/ \
  -m http://mirror.example.com:10000/ \
  example.com

Are there are any client compatibility concerns?

No.  Clients upgrading from older versions of the pkg client software to this version of the client software will not experience any issues.  In addition, some care has been taken to ensure older clients that understand the current image format are compatible with this new configuration, although they will be limited to using the first origin for a repository.

Do I need to update my scripts or client usage?

While the '-O' option has been deprecated for the set-publisher command and is no longer documented, it will continue to work exactly as it has in the past so no change is strictly necessary.  The command-line option compatibility for -O is currently planned to be maintained through the next major OpenSolaris release (2010.x), but will be dropped after that.

However, if you are using any scripts or programs that parse the output of the 'pkg publisher' command, and only expect a single origin to be listed for each publisher, they will need to be updated to account for this.

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Revision 1431 / 62b6033670e4

Please READ THIS  if you publish packages or run a pkg.depotd server!

Also be aware that if you have multiple instances of the opensolaris.org publisher repositories configured (i.e. ipkg.sfbay/dev, pkg.opensolaris.org, pkg.opensolaris.org/release); that will no longer work.  Please read this flag day message if have configured a publisher with any name except 'opensolaris.org' for pkg.opensolaris.org/dev, pkg.opensolaris.org/relesae or ipkg.sfbay/dev. 

Revision 1431:62b6033670e4 integrated the following fixes and enhancements:

  • 10416 server catalog v1 support desired
  • 243 need localized descriptions, other metadata at catalog level
  • 2424 Need to use UTC consistently everywhere
  • 3092 messaging api/framework needed for pkg clients (cli, gui, etc.)
  • 7063 "pkg list -a -s" needs performance improvement
  • 7163 manifests are modified by client before being written to disk
  • 8217 package fmri should be added to manifest during publishing
  • 9061 importer should not refresh indexes
  • 9446 traceback for cfg_cache operations if read-only filesystem
  • 10415 client catalog v1 support desired
  • 11094 Client transport for catalog v1
  • 11523 only permit FMRIs from same publisher for network repositories
  • 11831 server api version incompatible templates can cause traceback
  • 11832 depot needs ability to seed / alter repository configuration
  • 11954 importer shows zero packages processed unless debug enabled
  • 12006 merge utility should have a test suite

Client Software Considerations

The pkg(1) client now requires that the name (prefix) that you have configured your publisher with matches that reported by its repository (origin) when the package repository provides a version 1 catalog.

For example, if you have your client configured this way:

    $ pkg publisher
    PUBLISHER   TYPE     STATUS   URI
    dev         (preferred)  origin   online http://pkg.opensolaris.org/dev

Then the next time a pkg refresh is performed, you will see the following warning:

    $ pkg refresh
    Refreshing catalog 1/1 test
    The catalog retrieved for publisher 'dev' only contains package data for these publisher(s): opensolaris.org.  To resolve this issue, update this publisher to use the correct repository origin, or add one of the listed publishers using this publisher's repository origin.
    
    To correct the repository origin, execute the following command as a privileged user:

    pkg set-publisher -O <url> dev

    To add a new publisher using this publisher's repository origin, execute the following command as a privileged user:

    pkg set-publisher -O http://pkg.opensolaris.org/dev <publisher>

    After the new publisher has been added, this one should be removed by executing the following command as a privileged user:

    pkg unset-publisher test

Please note that this means that available packages from that publisher (excluding already installed packages) will NOT be shown in pkg list, pkg info, etc.

This change is the first step in a set of changes to not require that a user specify the publisher's name (prefix) when using image-create, set-publisher, etc. and to further simplify configuration.

Disk space usage will also increase for the client when it is configured to use a publisher repository that contains catalog v1 data.  As an example, when pkg.opensolaris.org/dev's depot software is upgraded, it now requires roughly 36MB of disk space (uncompressed) for local metadata storage (51K+ packages).  This information does compress extremely well (around 7MB gzip'd).

Disk space usage will continue to slightly increase for each new build until historical and obsolete packages are implemented, at which point it will significantly decrease again.

Depot Software Considerations

As a result of these changes, if you use the new depot server with an existing repository without using the --readonly option it will be automatically upgraded to a new repository format.

If you use an older repository with the newer depot software, but want to use it with the newer depot software, you must specify --readonly and --writable-root.  If you do that, a new catalog will be written to the --writable-root directory based on the contents of the repository.

In addition, there is now a hard requirement that you specify a default publisher prefix for the repository.  Carefully consider what you specify here as it will be used during publication if the FMRI specified at publication time does not include a publisher prefix.  The publisher prefix of any published packages will be placed into package manifests, the catalog, etc. and so changing it once set will not update the existing package data inside the repository as it is only a default.

Once a repository has been upgraded to the new format, you will be unable to use older pkg(5) depot server software, pksend, or pkgrecv with the new repository.  If you attempt to use an older client with the upgraded repository, it will fail.

As a reminder, parallel publication is not supported at this time via file://, only with http://.  Various import tests have been performed to verify that publication performance is roughly the same as it was before.

Finally, please note that this upgrade behaviour also applies if you are using 'file://' locations for destination repositories with pkgrecv or pkgsend.

Thanks

Enough thanks cannot be given to team members that helped complete this phase of the catalog v1 project over the past month.

Transport work provided by our resident guru: johansen. 

Revision 1117 / 667aecade0ab

This revision integrated enhancements and fixes for the following issues:

Changset r1117 in the pkg(5) gate implemented the following enhancements:

  • 7862 install links using p5i for search results desired
  • 8723 repository should contain publisher and repository client metadata
  • 8724 publisher metadata retrieval operation desired

The purpose of this new functionality is to allow "one-click" web-based installation of packaging software and to enable clients to retrieve metadata about the publisher and repository that can be used for policy determination and auto-configuration.

As a result of this change, repository administrators will need to add additional information to the cfg_cache file located in the repository directory (e.g. /var/pkg/repo/cfg_cache) to enable this new
functionality and to allow correct identification of depot RSS/Atom feed entries.

If you do not provide this new information, the new functionality will be disabled (return 404 errors to clients) but the depot server will otherwise function correctly.

To do this, simply start and stop the pkg.depotd server while the repository is not in use.  Once you've done that, you will see one new configuration section entitled 'publisher' and several new configuration attributes under the 'repository' section.

The easiest way to do this is to stop and start the depot server (not in readonly mode) so that the new sections and attributes will be written to the configuration file.  An example of what information you might provide is below:

    [publisher]
    alias =
    prefix = opensolaris.org
    
    [repository]
    collection~_type = core
    description = This repository serves the currently in-development release of OpenSolaris.
    detailed~_url = http://www.opensolaris.com
    legal~_uris = http://www.opensolaris.org/os/copyrights/,http://www.opensolaris.org/os/tou/,http://www.opensolaris.org/os/trademark/
    maintainer~_url = http://www.opensolaris.org/os/project/indiana/
    maintainer = Project Indiana <indi...@opensolaris.org>
    name = OpenSolaris Packaging Repository
    origins = http://pkg.opensolaris.org/dev
    refresh~_seconds = 14400
    registration~_uri =
    related_uris = http://pkg.opensolaris.org/contrib,http://pkg.opensolaris.org/pending

A detailed explanation of these attributes, their purpose, and allowed  values can be found in "pydoc pkg.client.publisher" under the "Publisher" and "Repository" class headings.

Revision 926 / 6ee411c9026a

This revision integrated enhancements and fixes for the following issues:

  • 5871 publisher apis desired
  • 3682 pkg publisher should show mirrors in default listing
  • 5071 provide direct way to change publisher to not use certificate
  • 5373 pkg commands shouldn't traceback when client can't read a key/cert file
  • 6843 problems with empty string for ssl_cert, ssl_key
  • 6897 "authority" should be "publisher"
  • 7046 client intent is not sent if same api object used for multiple operations / reasons

The pkg(5) team is always looking for ways to improve the user experience for those using the pkg(5) system and to promote a better understanding of it.  One of the ways to do that is to ensure that clear, consistent terminology is used to describe ideas and concepts.  That is why it was recently decided that a name change was in order for one of the core concepts in the pkg(5) system: authorities.

An authority is a forward or reverse domain name that identifies an organization (or group of individuals) responsible for providing one or more packages.  However, the name that was chosen for that concept caused some confusion among our users.  This was also further exacerbated by the interchangeable usage of the words "Authority" and "Repository" which are decidedly different in meaning.  As a result, starting with revision 926 of the pkg gate code,  an "Authority" is now called a "Publisher".

Before moving on to the details of the changes involved, it is important to summarise what the current overall vision is for the structure of publishers and repositories and how the client is intended to use them:

  • Publisher - a forward or reverse domain name (e.g. "opensolaris.org" or "org.opensolaris") that can be used to identify an organization that is responsible for one or more packaging repositories.  While a publisher may have (publish to) one or more repositories, pkg(5) clients are only intended to access one of them at a time (such as the "development" or "release" repositories for OpenSolaris).  Please note that for compatibility reasons, the client is currently restricted to a single repository per publisher; this will change after the 2009.06 release of OpenSolaris.
  • Repository - a location where clients can publish and retrieve package content (files contained within the package such as programs, documents, etc.) and / or metadata (information about the package such as its name, description, etc.).  It has the following characteristics:
    • may have one or more origins (URIs) used for publication and retrieval of package metadata and content (the client currently only supports one of these)
    • may have zero or more mirrors (URIs) for retrieval of package content
    • can only contain packages from a single publisher

Now that you know what a publisher and a repository is, here is a high-level summary of the changes that were needed to make this happen:

  • All references to "authority" were replaced with "publisher".
  • A publisher management and client history logging interface was added to pkg.client.api so that all pkg(5) clients can consistently and easily manage publisher and repository information.
  • The output of the "pkg publisher" subcommand (aka "pkg authority") was changed to include mirrors in the default listing.
  • The cli and gui clients included with the pkg(5) system were updated to use the new publisher management interfaces.
  • Support for generating and parsing .p5i data (pkg(5) publisher and package information files) was added to the api and to the gui client.  This functionality will eventually allow users to trigger installation of new software just by clicking a link in their web browser.
  • Fixed SSL Certificate handling to be more robust so that the client won't fail with a Python traceback when performing network operations (such as downloading package data from a repository) and so that the user will know why the failure occurred (expired certificate, etc.).
  • Fixed "pkg publisher" command to return a partial failure code of 3 if the information for one of the publishers or repositories couldn't be read (such as the SSL Certificate).

As a result of these changes the following commands have been renamed and altered as follows:

  • pkg authority -> pkg publisher
    • The output of this command also changed.  Before, executing "pkg authority" would have given you a list of authorities and the origin URIs for each repository.  Now it shows repository mirrors as well and indicates their status and type and ensures that the preferred publisher is always listed first and the rest in alphabetical order by name.  Here is a before and after comparison from my own system:
    $ pkg authority
    AUTHORITY                           URL
    opensolaris.org                     http://pkg.opensolaris.org/dev/
    ipkg (preferred)                    http://ipkg.sfbay/dev/
    extra                               https://pkg.sun.com/opensolaris/extra/

    $ pkg publisher
    PUBLISHER                             TYPE     STATUS   URI
    ipkg                     (preferred)  origin   online   http://ipkg.sfbay/dev/
    extra                                 origin   online   https://pkg.sun.com/opensolaris/extra/
    opensolaris.org                       origin   online   http://pkg.opensolaris.org/dev/
  • pkg set-authority -> pkg set-publisher
  • pkg unset-authority -> pkg unset-publisher

All of the man pages and other technical documentation provided with the packaging system has been updated, and the IPS User Guide is in the process of being updated.  Please note that the old commands are still recognised for the purposes of backwards compatibility, but they will be removed after the 2009.06 release of OpenSolaris.

Revision 873 / b95d76c53b5e

This revision integrated enhancements and fixes for the following issues:

  • 2691 ability to publish packages to local disk repository
  • 850 pkgsend traceback if invalid or unavailable repository url specified
  • 894 pkgsend should do more action verification
  • 1236 pkgsend open should not allow both -e and -n
  • 2430 pkgsend add of non-existent file causes traceback
  • 5088 server can still create a repo when in --readonly mode
  • 5294 pkgsend doesn't provide server response when operations fail
  • 5944 pkgsend traceback if invalid or no action type specified
  • 6566 catalog is always rebuilt when server config is not readonly

As a result of these changes, the following changes in behaviour should be noted:

  • pkg.depotd will no longer automatically create or populate the inst_root specified using pkg/inst_root or the -d option if started with --readonly.  Instead, it will exit with an error message.  Starting the server in a mode other than readonly will create the specified directory and any necessary subdirectories or files.
  • pkg.depotd will no longer automatically rebuild the repository catalog if not started with --readonly.  However, search indices will continue to be refreshed automatically if needed.  If, for some reason, the package catalog does need to be rebuilt on pkg.depotd startup or the old behaviour is desired, pkg.depotd can be started with the --rebuild option.  This change should result in significantly faster startup (and less disk I/O) for large repositories served by pkg.depotd in publishing mode.
  • pkgsend and the importer now support file:// URLs for publishing.  In the case of pkgsend:
    PKG_REPO=file:///path/to/repo pkgsend open foo@1.0
    pkgsend -s file:///path/to/repo open foo@1.0

In the case of solaris.py (the importer), you will need to set the REPO variable in the makefile or override it another way.  Be aware that if you specify a directory that is not a valid pkg(5) repository, publishing will fail with an error indicating that.  You must create a repository first by starting the pkg.depotd server with the desired path and then exiting.  A subcommand to create repositories will be added to pkgsend in the near future (bug 6574).

  • It is important to note that only one user at a time should publish directly to a file:// repository URL; the result of more than one user publishing to a file:// reopsitory URL at a time is completely untested and is likely to fail.  Bug 2696 will help enforce this by ensuring that a repository lock is obtained during publishing mode.  If you need the ability to allow multiple users to simultaneously publish to a repository, you must use pkg.depotd and publish to a http:// repository URL.  The tradeoff is that file:// repository URL publishing can be significantly faster and will result in significantly reduced memory usage and disk I/O.

 Significantly more verification is now done during the publishing process.  Expect things that may have appeared to work* before to no longer be accepted by the server or by the pkgsend client. Additionally, pkgsend once more produces useful error information when operations fail. 

Revision 852 / 96c4a4dace99

The following notice only applies to users running pkg gate development bits on systems running a OpenSolaris build 106 version of pkg.depotd (revision 848 / 048096a95369) or to users that have written their own depot templates for older versions of the depot software.

The changes for revision 852 / 96c4a4dace99 fixed these issues:

  • 5999 catalog.extract_matching_fmris - version filtering not correct
  • 6010 packages page should show only latest version of packages by default

As a result of these changes, the server/api used for depot templates changed incompatibly with previous versions.  As such, if you attempt to run a version of pkg.depotd after revision 848 on a system that has an older version installed, you may see this error in the logs when attempting to access depot pages:

VersionException: Incompatible API version '1' specified; expected: '2'.

The cause is that the newer development server is attempting to load the older depot templates from the system's installed /usr/share/lib/pkg directory.

To correct this, you must run the pkg.depotd process with --content-root=/path/to/new/usr/share/lib/pkg; if you are running the depot.py program from an the pkg gate src/ directory, you can do it this way:

./depot.py --content-root=../proto/root_uname -p/usr/share/lib/pkg

Revision 838 / aafbe4737188

This revision integrated enhancements and fixes for the following issues:

  • 3245 need to support creation of multi-architecture (fat) packages
  • 5862 gen_actions_by_type isn't filtered, caching breaks filter changes

Packages produced using solaris.py may contain set actions with more than two tags; this will cause exceptions in either client or depot software.

Upgrade to new versions of SUNWipkg before attempting to install such packages.

Read more about fattening packages here.

Revision 742 / 909b912fa942

This revision integrated enhancements for the following bugs:

  • * 3011 new depot index page desired
  • * 3014 revamped depot bui (web ui) desired

As part of the changes for the above bugs, Mako Templates version 0.2.2 or newer is now required to launch the pkg.depotd process.  The depot process will fail to start if you do not have it installed.

Also note that SUNWpython-setuptools is now required to build SUNWpython-mako in the pkg gate.  setuptools version 0.6c9 or later is known to work on OpenSolaris and other platforms.

Be aware that this drastically changes the presentation of the depot facing web pages.  As a result, the following repository configuration attributes are now ignored:

  • repository
  • icon
  • logo
  • style

In its place, are a set of Mako templates that can be found in the /usr/share/lib/pkg/web, that are comprised of a set of base pages and three themes.  The three themes included are: default, opensolaris.org, and sun.com.  The last two themes can only be used with appropriate authorisation by Sun Microsystems, Inc.

At the moment, to customise the included pages or themes, the /usr/share/lib/pkg directory will need to be copied to a new location (such as /var/my-new-directory), altered, and then the pkg.depotd process should be started with --content-root=/var/my-new-directory.

The repository cfg_cache located in /path/to/repo/cfg_cache also needs to be updated.  Specifically, the icon and logo paths for the repository feed have changed from this:

    [repository]
    logo = pkg-block-logo.png
    icon = pkg-block-icon.png

...to this:

    [repository]
    logo = ../~_themes/pkg-block-logo.png
    icon = ../~_themes/pkg-block-icon.png

Since the cfg_cache is written out upon repository initialisation, and these values are intended to be customised by the repository maintainer, they are never updated automatically.

Revision 452 / 01f53c6381af

This revision integrated fixes or enhancements for the following bugs:

  • * 2714 image-update failure during update phase
  • * 2751 test suite fails to detect tracebacks in log
  • * 2752 repository - rename function fails during exception handling

As part of the changes for bug 2714, CherryPy 3.1.0 is now required to launch the pkg.depotd process.  The depot process will fail to start with a message indicating that 3.1.0 is required if you do not have it installed.

Please also note that you should run "make clobber" in any hg workspaces for pkg and delete src/pkgdefs/SUNWpython-cherrypy/prototype, the contents of the packages/ directory, and finally the contents of the proto/ directory.

Revision 451 / 74f1fe85fe2d

This revision integrated fixes or enhancements for the following bugs:

  • 1324 RSS / Atom feeds of repository updates

As part of the changes for bug 1324, a new parameter to pkg.depotd was added, "--proxy-base" and a new SMF property pkg/proxy_base:

      pkg/proxy_base             (uri) This changes the base URL for the
                                 depot server and is most useful when
                                 running behind Apache or some other
                                 webserver in a reverse proxy
                                 configuration.

So, if your depot server is running as:
http://zoltar:10000

...yet, it is referenced as:

http://pkg.foo.com

...then you need to start the depot with:
\--proxy-base=http://pkg.foo.com

This is especially important for feed functionality to operate correctly.

Please also note that if you change this value, you need to restart the server in non-readonly mode at least once, and then visit http://repo_url/feed so that the feed.xml cache file is created.

Revision 428 / e011e45fe2cd

This revision integrated fixes or enhancements for the following bugs:

  • 2239 repository attribute system needed
  • 2603 hgignore needs to be updated for package gui
  • 2622 testutils directory structure problematic

As part of the changes for bug 2239, various meta-data and configuration attributes were added for repositories.  Among the attributes added was an RSS/Atom feed identifier.

This identifier is used by RSS Readers to identify the feed content generated by the pkg.depotd(1M) process.  The Universally Unique Identifier (UUID) for the feed will be generated for the repository upon initialization (usually at depot startup) if it is not already present.

If multiple copies of the same repository are being served by multiple depot processes, the feed id must be the same for each copy of the repository; otherwise RSS Readers serviced by the depot process will not be able to correctly identify new RSS entries or process the feed correctly when they connect to different depot processes.

The need for this identifier to be the same is a requirement of the RSS / Atom feed format.  It is recommended that you simply copy the attributes in the [feed] section of the _cfg\cache file (or the entire file) to the _inst\dir of any duplicate repositories.

Finally, be aware that this work is preparation for integration of bug 1324 which actually provides the RSS / Atom feed functionality that will use this new attribute.

Tags:
Created by admin on 2009/10/26 12:16
Last modified by Shawn Walker on 2011/03/21 20:50
XWiki Enterprise 2.7.1.34853 - Documentation