pkg DEPOT 1. Summary This document describes the architecture of a pkg(5) depot server as implemented in pkg.depotd(1m). This includes: an overview of the depot server's filesystem layout, operations provided by the depot server, the interfaces provided by the depot server, and a description of how clients communicate with the depot server for each operation. NOTE: The image packaging system is under development. Changes to interfaces may occur as part of the architectural review process, as shortcomings are identified, and as new features are introduced. Questions about planned or possible change should be asked on pkg-discuss. 2. Discussion A pkg(5) depot server provides a way for clients to interact with package content and metadata contained within a pkg(5) repository, and with the depot server itself. It accomplishes this by providing an HTTP-based interface suitable for usage by pkg(1) clients and web user agents. 2.1. Filesystem Layout The types of information that the depot server stores and/or retrieves can be categorized as follows: - depot data This includes: configuration data, presentation content (such as web page templates), publishing data (e.g. in-flight transactions), and temporary data (e.g. the feed cache). - repository data This includes: catalog information, package content (files), package metadata (manifests), and search data. 2.1.1. Layout The depot server uses the following 'root' directory structures for the storage and retrieval of depot and repository data: - repo_dir (depot and repository data) cfg_cache (depot data) A file containing the cached configuration information for the depot server. catalog/ (repository data) This directory contains the repository catalog and its related metadata. file/ (repository data) This directory contains the file content of packages in the repository. Files are stored using a two-level path fragment, derived from the SHA1-hash of a file's content, assumed to have at least 8 distinct characters. Example: 00/ 0023bb/ 000023bb53fdc7bcf35e62b7b0b353a56d36a504 index/ (repository data) This directory contains the search indices for the repository. pkg/ (repository data) This directory contains the metadata (manifests) for the repository's packages. The manifests for each package are stored in a directory with the same name as the package stem using a URL-encoded filename. Example: entire/ 0.5.11%2C5.11-0.86%3A20080422T234219Z trans/ (depot data) This directory contains in-flight transactions for packages that are waiting for the publication process to complete so that they can be added to the repository's catalog. Each transaction is stored in a directory named after the pending transaction id and contains the manifest waiting for publication to finish stored with the filename of 'manifest'. Example: 1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/ manifest updatelog/ (repository data) This directory contains metadata detailing changes to the repository by publishing operations. - content_root (depot data) web/ This directory contains all of the web presentation content for the depot. 2.2. Operations When communicating with the depot server via HTTP, operations are presented via a URL-based mechanism that allows each to be versioned so that changes in protocol can be hidden from older clients as the interfaces to operations evolve. Operations made available by a pkg.depotd(5) server can be accessed via GET or POST, as appropriate for each operation, via a URL such as the following: http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z The above example can be broken down into four basic components: publisher_origin_url - http://pkg.opensolaris.org/release/ operation_name - manifest protocol_version - 0 operation_arguments - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z Each of these components can be described as follows: publisher_origin_url - A URL that can be used to access a depot server's repository. operation_name - The name of the operation that the client is wanting to initiate. protocol_version - An integer value representing the version of the operation's protocol spoken by the client. operation_arguments - String data (such as a package FMRI) that is parsed and then used to determine what resource(s) will be used to perform an operation. Some operations expect arguments or data to be passed via POST-based form data, headers, or the request body instead. 2.2.1. Operation Types Each operation that the depot server provides is either designed to interact with a pkg(5) repository, or with the depot server itself. These operations can be categorized as follows: - content These operations are read-only, and retrieve file data that comprises the content of a package in a repository. - depot These operations are read-only, and permit retrieval of: the list of operations that the depot server currently provides (including protocol version and pkg(5) software version), statistics information, and other depot information. - metadata These operations are read-only, and retrieve metadata related to a package FMRI, such as its name, version, etc. stored in a repository's catalog. - publishing These operations alter a repository's catalog, package metadata, and allow storage of package content. 2.2.2. Modes Which types of operations are available is dependent on which mode the depot server is currently operating in: - default In default mode, the depot server allows content, depot, metadata, and publishing operations. - readonly In readonly mode, the depot server allows content, depot, and metadata operations. - mirror In mirror mode, the depot server allows content and depot operations. 2.2.3. Content Operations The pkg.depotd(5) server provides the following operations for retrieving package content: - file Version 0: A GET operation that retrieves the contents of a file, belonging to a package, using a SHA-1 hash of the file's content. Example: URL: http://pkg.opensolaris.org/release/file/0/ a00030db8b91f85d0b7144d0d4ef241a3f1ae28f Expects: A SHA-1 hash of the file's content belonging to a package in the request path. Returns: The contents of the file, compressed using the gzip compression algorithm. 2.2.2. Depot Operations - versions Version 0: A GET operation that retrieves text data representing what operations are supported and version information about the depot server. Example: URL: http://pkg.opensolaris.org/versions/0/ Expects: Nothing Returns: text/plain data containing the version of the pkg(5) software that the depot is based upon, a list of the operations currently supported, and the protocol version supported for each operation. Sample Output: pkg-server bfc04991436e info 0 search 0 versions 0 catalog 0 manifest 0 add 0 file 0 abandon 0 close 0 open 0 2.2.2. Meta-data Operations - catalog Version 0: A GET operation that retrieves a text/plain datastream representing a complete catalog or an incremental update to an existing one as requested. Example: URL: http://pkg.opensolaris.org/catalog/0/ Expects: Nothing or the following headers: If-Modified-Since: {ISO 8601 formatted date and time in UTC} Returns: Either the contents of a pkg(5) catalog file, or the entries that were added since the specified date as they are found in the catalog file, separated by newlines. - info Version 0: A GET operation that retrieves a text/plain description of a package and its licensing information specified by the provided FMRI. Example: URL: http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z Expects: A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix and publisher information, and including the full version information. Returns: A text/plain representation of the specified package and its licensing information. Sample Output: Name: entire Summary: entire incorporation Publisher: Unknown Version: 0.5.11 Build Release: 5.11 Branch: 0.101 Packaging Date: Wed Nov 19 23:57:06 2008 Size: 0.00 B FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z License: - manifest Version 0: A GET operation that retrieves the contents of the manifest file for a package specified by the provided FMRI. Example: URL: http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z Expects: A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix and publisher information and including the full version information. Returns: The contents of the package's manifest file. - p5i Version 0: A GET operation that retrieves an application/vnd.pkg5.info datastream representing publisher and package information. This is intended for consumption by clients for the purposes of auto-configuration, metadata management policy determination, and triggering packaging operations such as installation. Example: URL: http://pkg.opensolaris.org/release/p5i/0/SUNWcs Expects: A full or partial URL-encoded pkg(5) FMRI, excluding the publisher prefix. If the partial or full FMRI is valid, it will be added to the datastream as is. If it includes the wildcard character '*', a search of the repository's catalog for matching entries will be performed and the unique set of resulting package stems will be added to the datastream. If no match is found, a 404 error will be raised. Returns: Returns a pkg(5) information datastream based on the repository configuration's publisher information and the provided full or partial FMRI or matching entries. The Content-Type of the response is 'application/vnd.pkg5.info'. - publisher Version 0: A GET operation that retrieves an application/vnd.pkg5.info datastream representing publisher information. This is intended for consumption by clients for auto-configuration and metadata management policy determination. Example: URL: http://pkg.opensolaris.org/release/publisher/0 Expects: Nothing Returns: Returns a pkg(5) information datastream based on the repository configuration's publisher information. The Content-Type of the response is 'application/vnd.pkg5.info'. - search Version 0: A GET operation that retrieves a text/plain list of packages with metadata that matches the specified criteria. Example: URL: http://pkg.opensolaris.org/release/search/0/vim Expects: A URL-encoded token representing the search criteria. Returns: A text/plain list of matching entries, separated by newlines. Each entry consists of a set of four space-separated values: index - what search index the entry was found in action - what package action the entry is related to value - the value that the matched the search criteria package - the fmri of the package that contains the match Results are streamed to the client as they are found. Sample Output: basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim 2.2.3. Publishing Operations - add Version 0: A POST operation that adds content to an in-flight transaction for the Transaction ID specified. This could either be file content for the package or metadata about the package. This data is not added to the repository for retrieval until a close operation for the specified Transaction ID is executed. Example: URL: http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z HEADERS: X-IPkg-SetAttr1: description=Package Name REQUEST BODY: Expects: A Transaction ID as output by pkgsend(1) in the request path. The file content (if applicable), to be added, in the request body. Any attributes to be set in the headers in the pattern of: X-IPkg-SetAttr{integer}: attr=value Returns: Response status of 200 on success; any other status indicates failure. - abandon Version 0: A GET operation that aborts an in-flight transaction for the Transaction ID specified. This will discard any data related to the transaction. Example: URL: http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z Expects: A Transaction ID as output by pkgsend(1) in the request path. Returns: Response status of 200 on success; any other status indicates failure. - close Version 0: A GET operation that ends an in-flight transaction for the Transaction ID specified. If successful, the corresponding package is added to the repository catalog and is immediately available to repository users. Example: URL: http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z Expects: A Transaction ID as output by pkgsend(1) in the request path. Returns: Response status of 200 on success; any other status indicates failure. - open Version 0: A GET operation that starts an in-flight transaction for the package FMRI specified. Example: URL: http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98 Expects: A URL-encoded pkg(5) FMRI (excluding timestamp). Returns: Response status of 200 on success and an identifier for the new transaction in the 'Transaction-ID' response header; any other status indicates failure. 3. References