depot.py revision 3194
20N/A# The contents of this file are subject to the terms of the 20N/A# Common Development and Distribution License (the "License"). 20N/A# You may not use this file except in compliance with the License. 20N/A# See the License for the specific language governing permissions 20N/A# and limitations under the License. 20N/A# When distributing Covered Code, include this CDDL HEADER in each 20N/A# If applicable, add the following below this CDDL HEADER, with the 20N/A# fields enclosed by brackets "[]" replaced with your own identifying 20N/A# information: Portions Copyright [yyyy] [name of copyright owner] 20N/A# Copyright (c) 2007, 2015, Oracle and/or its affiliates. All rights reserved. 0N/A# pkg.depotd - package repository daemon 50N/A# XXX The prototype pkg.depotd combines both the version management server that 50N/A# answers to pkgsend(1) sessions and the HTTP file server that answers to the 50N/A# various GET operations that a pkg(1) client makes. This split is expected to 50N/A# be made more explicit, by constraining the pkg(1) operations such that they 50N/A# can be served as a typical HTTP/HTTPS session. Thus, pkg.depotd will reduce 50N/A# to a special purpose HTTP/HTTPS server explicitly for the version management 50N/A# operations, and must manipulate the various state files--catalogs, in 50N/A# particular--such that the pkg(1) pull client can operately accurately with 50N/A# XXX Although we pushed the evaluation of next-version, etc. to the pull 50N/A# client, we should probably provide a query API to do same on the server, for 50N/A# dumb clients (like a notification service). 50N/A# The default path for static and other web content. 54N/A# cherrypy has a max_request_body_size parameter that determines whether the 26N/A# server should abort requests with REQUEST_ENTITY_TOO_LARGE when the request 0N/A# body is larger than the specified size (in bytes). The maximum size supported 0N/A# by cherrypy is 2048 * 1024 * 1024 - 1 (just short of 2048MB), but the default 0N/A# here is purposefully conservative. 34N/A# The minimum number of threads allowed. 22N/A# The default number of threads to start. 46N/A# The maximum number of threads that can be started. 22N/A# The default server socket timeout in seconds. We want this to be longer than 22N/A# the normal default of 10 seconds to accommodate clients with poor quality 30N/A print(
"""cherrypy 3.1.0 or greater (but less than """ 54N/A """This is a dummy object that we can use to discard log entries 54N/A without relying on non-portable interfaces such as /dev/null.""" 54N/A """Discard the bits.""" 54N/A """Discard the bits.""" 30N/A """Optionally emit a usage message and then exit using the specified 22N/A # The full usage message isn't desired. 22N/A emsg(_(
"Try `pkg.depotd --help or -?' for more " 22N/A [-t socket_timeout] [--cfg] [--content-root] 22N/A [--disable-ops op[/1][,...]] [--debug feature_list] 22N/A [--image-root dir] [--log-access dest] [--log-errors dest] 22N/A [--mirror] [--nasty] [--nasty-sleep] [--proxy-base url] 22N/A [--readonly] [--ssl-cert-file] [--ssl-dialog] [--ssl-key-file] 22N/A [--sort-file-max-size size] [--writable-root dir] 0N/A -a address The IP address on which to listen for connections. The 0N/A default value is 0.0.0.0 (INADDR_ANY) which will listen 0N/A on all active interfaces. To listen on all active IPv6 0N/A interfaces, use '::'. 0N/A -d inst_root The file system path at which the server should find its 0N/A repository data. Required unless PKG_REPO has been set 22N/A in the environment. 26N/A -p port The port number on which the instance should listen for 22N/A incoming package requests. The default value is 80 if 26N/A ssl certificate and key information has not been 22N/A provided; otherwise, the default value is 443. 22N/A -s threads The number of threads that will be started to serve 22N/A requests. The default value is 10. 22N/A -t timeout The maximum number of seconds the server should wait for 22N/A a response from a client before closing a connection. 22N/A The default value is 60. 26N/A --cfg The pathname of the file to use when reading and writing 22N/A depot configuration data, or a fully qualified service 26N/A fault management resource identifier (FMRI) of the SMF 0N/A service or instance to read configuration data from. 0N/A --content-root The file system path to the directory containing the 0N/A the static and other web content used by the depot's 0N/A browser user interface. The default value is 0N/A --disable-ops A comma separated list of operations that the depot 26N/A should not configure. If, for example, you wanted 22N/A to omit loading search v1, 'search/1' should be 0N/A provided as an argument, or to disable all search 20N/A operations, simply 'search'. 34N/A --debug The name of a debug feature to enable; or a whitespace 21N/A or comma separated list of features to enable. 34N/A Possible values are: headers, hash=sha1+sha256, 0N/A hash=sha256, hash=sha1+sha512_256, hash=sha512_256 0N/A --image-root The path to the image whose file information will be 0N/A used as a cache for file data. 0N/A --log-access The destination for any access related information 26N/A logged by the depot process. Possible values are: 0N/A stderr, stdout, none, or an absolute pathname. The 50N/A default value is stdout if stdout is a tty; otherwise 26N/A the default value is none. 50N/A --log-errors The destination for any errors or other information 30N/A logged by the depot process. Possible values are: 50N/A stderr, stdout, none, or an absolute pathname. The 26N/A default value is stderr. 26N/A --mirror Package mirror mode; publishing and metadata operations 0N/A disallowed. Cannot be used with --readonly or 0N/A --nasty Instruct the server to misbehave. At random intervals 0N/A it will time-out, send bad responses, hang up on 22N/A clients, and generally be hostile. The option 22N/A takes a value (1 to 100) for how nasty the server 0N/A --nasty-sleep In nasty mode (see --nasty), how many seconds to 26N/A randomly sleep when a random sleep occurs. 26N/A --proxy-base The url to use as the base for generating internal 26N/A redirects and content. 26N/A --readonly Read-only operation; modifying operations disallowed. 22N/A Cannot be used with --mirror or --rebuild. 22N/A --ssl-cert-file The absolute pathname to a PEM-encoded Certificate file. 22N/A This option must be used with --ssl-key-file. Usage of 26N/A this option will cause the depot to only respond to SSL 26N/A requests on the provided port. 26N/A --ssl-dialog Specifies what method should be used to obtain the 26N/A passphrase needed to decrypt the file specified by 37N/A --ssl-key-file. Supported values are: builtin, 26N/A default value is builtin. If smf is specified, an 26N/A SMF FMRI must be provided using the --cfg option. 26N/A --ssl-key-file The absolute pathname to a PEM-encoded Private Key file. 26N/A This option must be used with --ssl-cert-file. Usage of 37N/A this option will cause the depot to only respond to SSL 37N/A requests on the provided port. 37N/A --sort-file-max-size 37N/A The maximum size of the indexer sort file. Used to 37N/A limit the amount of RAM the depot uses for indexing, 37N/A or increase it for speed. 26N/A --writable-root The path to a directory to which the program has write 26N/A access. Used with --readonly to allow server to 0N/A create needed files, such as search indices, without 0N/A needing write access to the package information. 0N/A PKG_REPO Used as default inst_root if -d not provided. 0N/A PKG_DEPOT_CONTENT Used as default content_root if --content-root 30N/A """Option exception. """ 22N/A # Track initial configuration values. long_opts = [
"add-content",
"cfg=",
"cfg-file=",
"content-root=",
"debug=",
"disable-ops=",
"exit-ready",
"help",
"image-root=",
"log-access=",
"log-errors=",
"llmirror",
"mirror",
"nasty=",
"nasty-sleep=",
"proxy-base=",
"readonly",
"rebuild",
"refresh-index",
"set-property=",
"ssl-cert-file=",
"ssl-dialog=",
"ssl-key-file=",
"sort-file-max-size=",
"writable-root="]
"minimum value is {0:d}".
format(
"maximum value is {0:d}".
format(
elif opt ==
"--add-content":
elif opt ==
"--cfg-file":
elif opt ==
"--content-root":
# A list of features can be specified using a # "," or any whitespace character as separators. # We also allow key=value debug flags, which # get set in pkg.client.debugvalues elif opt ==
"--disable-ops":
"An argument must be specified.")
elif opt ==
"--exit-ready":
elif opt ==
"--image-root":
elif opt in (
"--help",
"-?"):
elif opt ==
"--llmirror":
# ValueError is caught by caller. "for nasty option.\n Please " "choose a value between 1 and 100.")
elif opt ==
"--nasty-sleep":
# ValueError is caught by caller. elif opt ==
"--proxy-base":
# Attempt to decompose the url provided into # its base parts. This is done so we can # remove any scheme information since we "determine the hostname from " "the provided URL; please use a " if scheme not in (
"http",
"https"):
"and https are the only supported " # Rebuild the url with the sanitized components. elif opt ==
"--readonly":
elif opt ==
"--refresh-index":
# Note: This argument is for internal use # This flag is purposefully omitted in usage. # The supported way to forcefully reindex is to # kill any pkg.depot using that directory, # remove the index directory, and restart the # pkg.depot process. The index will be rebuilt # automatically on startup. elif opt ==
"--set-property":
usage(_(
"property arguments must be of " "the form '<section.property>=" elif opt ==
"--ssl-cert-file":
if arg ==
"none" or arg ==
"":
# Assume this is an override to clear "the Certificate file must be " "pathname is not a file.")
elif opt ==
"--ssl-key-file":
if arg ==
"none" or arg ==
"":
# Assume this is an override to clear "the Private Key file must be " "pathname is not a file.")
elif opt ==
"--ssl-dialog":
if arg !=
"builtin" and \
"specified. Expected: builtin, " # insecure authentication method "not a supported dialog " "type for this operating " "file path specified for " elif opt ==
"--sort-file-max-size":
elif opt ==
"--writable-root":
# Set accumulated values. # Build configuration object. usage(
"pkg.depotd: illegal option value: {0} specified " \
# If stdout is a tty, then send access output there by default instead # Check for invalid option combinations. usage(
"--add-content cannot be used with --rebuild")
usage(
"--refresh-index cannot be used with --rebuild")
usage(
"--readonly and --mirror cannot be used with --rebuild " usage(
"--mirror cannot be used with --refresh-index")
usage(
"--readonly can only be used with --refresh-index if " "--writable-root is used")
usage(
"--image-root can only be used with --llmirror.")
usage(
"--image_root and --writable-root cannot be used " usage(
"--image-root and -d cannot be used together.")
# If the image format changes this may need to be reexamined. # Set any values using defaults if they weren't provided. # Only use the first value for now; multiple bind addresses may be usage(
"Either PKG_REPO or -d must be provided")
usage(
"The --ssl-cert-file and --ssl-key-file options must " "must both be provided when using either option.")
# If the program is going to reindex, the port is irrelevant since # the program will not bind to a port. emsg(
"pkg.depotd: unable to bind to the specified " # Not applicable if we're not going to serve content # Any relative paths should be made absolute using pkg_root. 'pkg_root' # is a special property that was added to enable internal deployment of # multiple disparate versions of the pkg.depotd software. # Setup SSL if requested. emsg(
"pkg.depotd: an error occurred while " "executing [{0}]; unable to obtain the " "passphrase needed to decrypt the SSL " # Assume the configuration target was an SMF # FMRI and let svcprop fail with an error if # The key file requires decryption, but the user has requested # exec-based authentication, so it will have to be decoded first # to an un-named temporary file. emsg(
"pkg.depotd: unable to read the SSL private key " emsg(
"pkg.depotd: authentication or cryptography " "failure while attempting to decode\nthe SSL " # Redirect the server to the decrypted key file. # Setup our global configuration. "environment":
"production",
"server.shutdown_timeout":
0,
"server.socket_port":
port,
"tools.log_headers.on":
True,
# Despite its name, this only logs headers when there is an # error; it's redundant with the debug feature enabled. # Causes the headers of every request to be logged to the error # log; even if an exception occurs. # If stdin is not a tty and the pkgdepot controller isn't being used, # then assume process will be daemonized and redirect output. # Ensure log handlers are setup to use the file descriptors for # stdout and stderr as the Daemonizer (used for test suite and # SMF service) requires this. elif log_cfg[
"access"] ==
"stderr":
elif log_cfg[
"errors"] ==
"stdout":
"param":
"log.error_file",
"param":
"log.access_file",
if dest in (
"stdout",
"stderr",
"none"):
# Since we've replaced cherrypy's log handler with our # own, we don't want the output directed to a file. # Now that our logging, etc. has been setup, it's safe to perform any # Initialize repository state. # Not readonly, so assume a new repository should be created. # Already exists, nothing to do. # Automatically update search indexes on startup if not already # Only execute a index refresh here if --exit-ready was # requested; it will be handled later in the setup # process for other cases. # Ready to start depot; exit now if requested. # Next, initialize depot. # Now build our site configuration. # We have to override cherrypy's default response_class so that # we have access to the write() callable to stream data # directly to the client. "tools.staticfile.on":
True,
# This changes the base URL for our server, and is primarily # intended to allow our depot process to operate behind Apache # or some other webserver process. # Visit the following URL for more information: # Now merge or add our proxy configuration information into the # existing configuration. # Tell depot to update search indexes when possible; # this is done as a background task so that packages # can be served immediately while search indexes are # If stdin is not a tty and the pkgdepot controller isn't being used, # then assume process should be daemonized. # Translate the values in log_cfg into paths. emsg(
"pkg.depotd: unknown error starting depot server, " \
"illegal option value specified?")