pkg
DEPOT
1. Summary
This document describes the architecture of a pkg(5) depot server as
implemented in pkg.depotd(1m). This includes: an overview of the depot
server's filesystem layout, operations provided by the depot server,
the interfaces provided by the depot server, and a description of how
clients communicate with the depot server for each operation.
NOTE: The image packaging system is under development. Changes to
interfaces may occur as part of the architectural review process, as
shortcomings are identified, and as new features are introduced.
Questions about planned or possible change should be asked on
pkg-discuss.
2. Discussion
A pkg(5) depot server provides a way for clients to interact with package
content and metadata contained within a pkg(5) repository, and with the
depot server itself. It accomplishes this by providing an HTTP-based
interface suitable for usage by pkg(1) clients and web user agents.
2.1. Filesystem Layout
The types of information that the depot server stores and/or retrieves can
be categorized as follows:
- depot data
This includes: configuration data, presentation content (such as
web page templates), publishing data (e.g. in-flight transactions),
and temporary data (e.g. the feed cache).
- repository data
This includes: catalog information, package content (files), package
metadata (manifests), and search data.
2.1.1. Layout
The depot server uses the following 'root' directory structures for the
storage and retrieval of depot and repository data:
- repo_dir (depot and repository data)
cfg_cache (depot data)
A file containing the cached configuration information for the
depot server.
catalog/ (repository data)
This directory contains the repository catalog and its related
metadata.
file/ (repository data)
This directory contains the file content of packages in the
repository.
Files are stored using a two-level path fragment, derived from the
SHA1-hash of a file's content, assumed to have at least 8 distinct
characters.
Example:
00/
0023bb/
000023bb53fdc7bcf35e62b7b0b353a56d36a504
index/ (repository data)
This directory contains the search indices for the repository.
pkg/ (repository data)
This directory contains the metadata (manifests) for the
repository's packages.
The manifests for each package are stored in a directory with the
same name as the package stem using a URL-encoded filename.
Example:
entire/
0.5.11%2C5.11-0.86%3A20080422T234219Z
trans/ (depot data)
This directory contains in-flight transactions for packages that
are waiting for the publication process to complete so that they
can be added to the repository's catalog.
Each transaction is stored in a directory named after the pending
transaction id and contains the manifest waiting for publication
to finish stored with the filename of 'manifest'.
Example:
1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/
manifest
updatelog/ (repository data)
This directory contains metadata detailing changes to the repository
by publishing operations.
- content_root (depot data)
web/
This directory contains all of the web presentation content for the
depot.
2.2. Operations
When communicating with the depot server via HTTP, operations are presented
via a URL-based mechanism that allows each to be versioned so that changes
in protocol can be hidden from older clients as the interfaces to operations
evolve.
Operations made available by a pkg.depotd(5) server can be accessed via GET
or POST, as appropriate for each operation, via a URL such as the following:
The above example can be broken down into four basic components:
publisher_origin_url - http://pkg.opensolaris.org/release/
operation_name - manifest
protocol_version - 0
operation_arguments - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z
Each of these components can be described as follows:
publisher_origin_url - A URL that can be used to access a depot
server's repository.
operation_name - The name of the operation that the client is
wanting to initiate.
protocol_version - An integer value representing the version of
the operation's protocol spoken by the client.
operation_arguments - String data (such as a package FMRI) that is
parsed and then used to determine what
resource(s) will be used to perform an
operation. Some operations expect arguments
or data to be passed via POST-based form data,
headers, or the request body instead.
2.2.1. Operation Types
Each operation that the depot server provides is either designed to interact
with a pkg(5) repository, or with the depot server itself. These operations
can be categorized as follows:
- content
These operations are read-only, and retrieve file data that comprises
the content of a package in a repository.
- depot
These operations are read-only, and permit retrieval of: the list of
operations that the depot server currently provides (including protocol
version and pkg(5) software version), statistics information, and other
depot information.
- metadata
These operations are read-only, and retrieve metadata related to a
package FMRI, such as its name, version, etc. stored in a repository's
catalog.
- publishing
These operations alter a repository's catalog, package metadata, and
allow storage of package content.
2.2.2. Modes
Which types of operations are available is dependent on which mode the depot
server is currently operating in:
- default
In default mode, the depot server allows content, depot, metadata,
and publishing operations.
- readonly
In readonly mode, the depot server allows content, depot, and
metadata operations.
- mirror
In mirror mode, the depot server allows content and depot
operations.
2.2.3. Content Operations
The pkg.depotd(5) server provides the following operations for retrieving
package content:
- file
Version 0:
A GET operation that retrieves the contents of a file, belonging to a
package, using a SHA-1 hash of the file's content.
Example:
URL:
a00030db8b91f85d0b7144d0d4ef241a3f1ae28f
Expects:
A SHA-1 hash of the file's content belonging to a package in the
request path.
Returns:
The contents of the file, compressed using the gzip compression
algorithm.
2.2.2. Depot Operations
- versions
Version 0:
A GET operation that retrieves text data representing what operations
are supported and version information about the depot server.
Example:
URL:
Expects:
Nothing
Returns:
text/plain data containing the version of the pkg(5) software that
the depot is based upon, a list of the operations currently
supported, and the protocol version supported for each
operation.
Sample Output:
pkg-server bfc04991436e
info 0
search 0
versions 0
catalog 0
manifest 0
add 0
file 0
abandon 0
close 0
open 0
2.2.2. Meta-data Operations
- catalog
Version 0:
A GET operation that retrieves a text/plain datastream
representing a complete catalog or an incremental update to an
existing one as requested.
Example:
URL:
Expects:
Nothing or the following headers:
If-Modified-Since: {ISO 8601 formatted date and time in UTC}
Returns:
Either the contents of a pkg(5) catalog file, or the entries
that were added since the specified date as they are found
in the catalog file, separated by newlines.
- info
Version 0:
A GET operation that retrieves a text/plain description of a
package and its licensing information specified by the provided
FMRI.
Example:
URL:
Expects:
A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix
and publisher information, and including the full version
information.
Returns:
A text/plain representation of the specified package and its
licensing information.
Sample Output:
Name: entire
Summary: entire incorporation
Publisher: Unknown
Version: 0.5.11
Build Release: 5.11
Branch: 0.101
Packaging Date: Wed Nov 19 23:57:06 2008
Size: 0.00 B
FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z
License:
- manifest
Version 0:
A GET operation that retrieves the contents of the manifest file for
a package specified by the provided FMRI.
Example:
URL:
Expects:
A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix
and publisher information and including the full version
information.
Returns:
The contents of the package's manifest file.
- p5i
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher and package information.
This is intended for consumption by clients for the purposes
of auto-configuration, metadata management policy determination,
and triggering packaging operations such as installation.
Example:
URL:
Expects:
A full or partial URL-encoded pkg(5) FMRI, excluding the
publisher prefix. If the partial or full FMRI is valid, it will
be added to the datastream as is. If it includes the wildcard
character '*', a search of the repository's catalog for matching
entries will be performed and the unique set of resulting
package stems will be added to the datastream. If no match is
found, a 404 error will be raised.
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information and the provided full or
partial FMRI or matching entries. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- publisher
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher information. This is intended
for consumption by clients for auto-configuration and metadata
management policy determination.
Example:
URL:
Expects:
Nothing
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- search
Version 0:
A GET operation that retrieves a text/plain list of packages with
metadata that matches the specified criteria.
Example:
URL:
Expects:
A URL-encoded token representing the search criteria.
Returns:
A text/plain list of matching entries, separated by newlines.
Each entry consists of a set of four space-separated values:
index - what search index the entry was found in
action - what package action the entry is related to
value - the value that the matched the search criteria
package - the fmri of the package that contains the match
Results are streamed to the client as they are found.
Sample Output:
basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim
basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim
2.2.3. Publishing Operations
- add
Version 0:
A POST operation that adds content to an in-flight transaction for
the Transaction ID specified. This could either be file content
for the package or metadata about the package.
This data is not added to the repository for retrieval until a close
operation for the specified Transaction ID is executed.
Example:
URL:
HEADERS:
X-IPkg-SetAttr1: description=Package Name
REQUEST BODY:
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
The file content (if applicable), to be added, in the request
body. Any attributes to be set in the headers in the pattern
of:
X-IPkg-SetAttr{integer}: attr=value
Returns:
Response status of 200 on success; any other status indicates
failure.
- abandon
Version 0:
A GET operation that aborts an in-flight transaction for the
Transaction ID specified. This will discard any data related to
the transaction.
Example:
URL:
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- close
Version 0:
A GET operation that ends an in-flight transaction for the
Transaction ID specified. If successful, the corresponding package
is added to the repository catalog and is immediately available to
repository users.
Example:
URL:
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- open
Version 0:
A GET operation that starts an in-flight transaction for the
package FMRI specified.
Example:
URL:
Expects:
A URL-encoded pkg(5) FMRI (excluding timestamp).
Returns:
Response status of 200 on success and an identifier for the new
transaction in the 'Transaction-ID' response header; any other
status indicates failure.
3. References