depot.py revision 1633
290N/A# The contents of this file are subject to the terms of the 290N/A# Common Development and Distribution License (the "License"). 290N/A# You may not use this file except in compliance with the License. 290N/A# See the License for the specific language governing permissions 290N/A# and limitations under the License. 290N/A# When distributing Covered Code, include this CDDL HEADER in each 290N/A# If applicable, add the following below this CDDL HEADER, with the 290N/A# fields enclosed by brackets "[]" replaced with your own identifying 290N/A# information: Portions Copyright [yyyy] [name of copyright owner] 1660N/A# Copyright 2009 Sun Microsystems, Inc. All rights reserved. 395N/A# Use is subject to license terms. 883N/A# pkg.depotd - package repository daemon 290N/A# XXX The prototype pkg.depotd combines both the version management server that 448N/A# answers to pkgsend(1) sessions and the HTTP file server that answers to the 290N/A# various GET operations that a pkg(1) client makes. This split is expected to 290N/A# be made more explicit, by constraining the pkg(1) operations such that they 290N/A# can be served as a typical HTTP/HTTPS session. Thus, pkg.depotd will reduce 383N/A# to a special purpose HTTP/HTTPS server explicitly for the version management 290N/A# operations, and must manipulate the various state files--catalogs, in 395N/A# particular--such that the pkg(1) pull client can operately accurately with 849N/A# XXX Although we pushed the evaluation of next-version, etc. to the pull 1516N/A# client, we should probably provide a query API to do same on the server, for 290N/A# dumb clients (like a notification service). 290N/A# The default repository path. 290N/A# The default path for static and other web content. 383N/A# cherrypy has a max_request_body_size parameter that determines whether the 290N/A# server should abort requests with REQUEST_ENTITY_TOO_LARGE when the request 290N/A# body is larger than the specified size (in bytes). The maximum size supported 290N/A# by cherrypy is 2048 * 1024 * 1024 - 1 (just short of 2048MB), but the default 290N/A# here is purposefully conservative. 290N/A# The default port(s) to serve data from. 1660N/A# The minimum number of threads allowed. 1660N/A# The default number of threads to start. 1660N/A# The maximum number of threads that can be started. 1660N/A# The default server socket timeout in seconds. We want this to be longer than 1660N/A# the normal default of 10 seconds to accommodate clients with poor quality 1660N/A# Whether modify operations should be allowed. 1660N/A# Whether the repository catalog should be rebuilt on startup. 1660N/A# Whether the indexes should be rebuilt 1660N/A# Not in mirror mode by default 1513N/A """3.2.0) is required to use this program.""" 742N/A """This is a dummy object that we can use to discard log entries 742N/A without relying on non-portable interfaces such as /dev/null.""" 742N/A """Discard the bits.""" 941N/A """Discard the bits.""" 941N/A """Optionally emit a usage message and then exit using the specified 1191N/A # The full usage message isn't desired. 1191N/A emsg(_(
"Try `pkg.depotd --help or -?' for more " 448N/A [-t socket_timeout] [--cfg-file] [--content-root] 448N/A [--disable-ops op[/1][,...]] [--debug feature_list] 534N/A [--log-access dest] [--log-errors dest] [--mirror] [--nasty] 534N/A [--set-property <section.property>=<value>] 534N/A [--proxy-base url] [--readonly] [--rebuild] [--ssl-cert-file] 534N/A [--ssl-dialog] [--ssl-key-file] [--sort-file-max-size size] 534N/A --add-content Check the repository on startup and add any new 290N/A packages found. Cannot be used with --mirror or 954N/A --cfg-file The pathname of the file from which to read and to 954N/A write configuration information. 954N/A --content-root The file system path to the directory containing the 954N/A the static and other web content used by the depot's 534N/A browser user interface. The default value is 290N/A --disable-ops A comma separated list of operations that the depot 1191N/A should not configure. If, for example, you wanted 1191N/A to omit loading search v1, 'search/1' should be 1191N/A provided as an argument, or to disable all search 1516N/A operations, simply 'search'. 290N/A --debug The name of a debug feature to enable; or a whitespace 290N/A or comma separated list of features to enable. 290N/A Possible values are: headers. 661N/A --exit-ready Perform startup processing (including rebuilding 290N/A catalog or indices, if requested) and exit when 290N/A ready to start serving packages. 290N/A --log-access The destination for any access related information 395N/A logged by the depot process. Possible values are: 290N/A stderr, stdout, none, or an absolute pathname. The 290N/A default value is stdout if stdout is a tty; otherwise 290N/A the default value is none. 1483N/A --log-errors The destination for any errors or other information 290N/A logged by the depot process. Possible values are: 1498N/A stderr, stdout, none, or an absolute pathname. The 290N/A --mirror Package mirror mode; publishing and metadata operations 1674N/A disallowed. Cannot be used with --readonly or 1674N/A --nasty Instruct the server to misbehave. At random intervals 1674N/A it will time-out, send bad responses, hang up on 1674N/A clients, and generally be hostile. The option 1674N/A takes a value (1 to 100) for how nasty the server 395N/A --proxy-base The url to use as the base for generating internal 395N/A --readonly Read-only operation; modifying operations disallowed. 1544N/A Cannot be used with --mirror or --rebuild. 1557N/A --rebuild Re-build the catalog from pkgs in depot. Cannot be 1506N/A used with --mirror or --readonly. 395N/A --set-property Used to specify initial repository configuration 395N/A property values or to update existing ones; can 424N/A be specified multiple times. If used with --readonly 1024N/A this acts as a temporary override. 395N/A --ssl-cert-file The absolute pathname to a PEM-encoded Certificate file. 395N/A This option must be used with --ssl-key-file. Usage of 395N/A this option will cause the depot to only respond to SSL 578N/A requests on the provided port. 1228N/A --ssl-dialog Specifies what method should be used to obtain the 1172N/A passphrase needed to decrypt the file specified by 395N/A --ssl-key-file. Supported values are: builtin, 661N/A --ssl-key-file The absolute pathname to a PEM-encoded Private Key file. 395N/A This option must be used with --ssl-cert-file. Usage of 849N/A this option will cause the depot to only respond to SSL 290N/A requests on the provided port. 395N/A The maximum size of the indexer sort file. Used to 395N/A limit the amount of RAM the depot uses for indexing, 395N/A or increase it for speed. 395N/A --writable-root The path to a directory to which the program has write 395N/A access. Used with --readonly to allow server to 395N/A create needed files, such as search indices, without 395N/A needing write access to the package information. 290N/A PKG_REPO Used as default repo_dir if -d not provided. 290N/A PKG_DEPOT_CONTENT Used as default content_root if --content-root 395N/A # By default, if the destination for a particular log type is not 424N/A # specified, this is where we will send the output. 742N/A # If stdout is a tty, then send access output there by default instead 395N/A "debug=",
"disable-ops=",
"exit-ready",
"help",
"mirror",
395N/A "nasty=",
"set-property=",
"proxy-base=",
"readonly",
954N/A "rebuild",
"refresh-index",
"ssl-cert-file=",
"ssl-dialog=",
954N/A "ssl-key-file=",
"sort-file-max-size=",
"writable-root="]
1632N/A "A debug feature must be specified." 395N/A # A list of features can be specified using a 290N/A # "," or any whitespace character as separators. 290N/A "Invalid debug feature: " \
395N/A "An argument must be specified." 290N/A "You must specify a log " \
290N/A "for nasty option.\n Please " \
290N/A "choose a value between 1 and 100." 290N/A "the form '<section.property>=" 290N/A # Attempt to decompose the url provided into 290N/A # its base parts. This is done so we can 395N/A # remove any scheme information since we 506N/A "determine the hostname from " \
506N/A "the provided URL; please use a " \
513N/A "and https are the only supported " \
506N/A # Rebuild the url with the sanitized components. 883N/A # Note: This argument is for internal use 883N/A # only. It's used when pkg.depotd is reexecing 395N/A # itself and needs to know that's the case. 413N/A # This flag is purposefully omitted in usage. 413N/A # The supported way to forcefully reindex is to 413N/A # kill any pkg.depot using that directory, 395N/A # remove the index directory, and restart the 290N/A # pkg.depot process. The index will be rebuilt 1674N/A # automatically on startup. 1674N/A "the Certificate file must be " \
395N/A "pathname is not a file." 849N/A "the Private Key file must be " \
1391N/A "specified. Expected: builtin, " \
849N/A # Don't allow a somewhat 849N/A # insecure authentication method 849N/A "not a supported dialog " \
849N/A "type for this operating " \
849N/A "file path specified for " \
290N/A "a maximum sort file size." 883N/A usage(
"pkg.depotd: illegal option value: %s specified " \
1099N/A usage(
"--add-content cannot be used with --rebuild")
1099N/A usage(
"--refresh-index cannot be used with --rebuild")
1265N/A usage(
"--readonly and --mirror cannot be used with --rebuild " 1099N/A usage(
"--mirror cannot be used with --refresh-index")
1099N/A usage(
"--readonly can only be used with --refresh-index if " 1099N/A usage(
"The --ssl-cert-file and --ssl-key-file options must " 1208N/A "must both be provided when using either option.")
1099N/A # If they didn't already specify a particular port, use the 1099N/A # default SSL port instead. 1191N/A # If the program is going to reindex, the port is irrelevant since 1191N/A # the program will not bind to a port. 1191N/A print "pkg.depotd: unable to bind to the specified " \
1191N/A # Not applicable if we're not going to serve content 1191N/A print "pkg.depotd: an error occurred while " \
1265N/A "executing [%s]; unable to obtain the " \
1265N/A "passphrase needed to decrypt the SSL" \
465N/A # The key file requires decryption, but the user has requested 395N/A # exec-based authentication, so it will have to be decoded first 395N/A # to an un-named temporary file. 1099N/A print "pkg.depotd: unable to read the SSL private " \
465N/A print "pkg.depotd: authentication or cryptography " \
395N/A "failure while attempting to decode\nthe SSL " \
1191N/A # Redirect the server to the decrypted key file. 1191N/A # Setup our global configuration. 1191N/A "environment":
"production",
1191N/A "server.shutdown_timeout":
0,
1265N/A "server.socket_host":
"0.0.0.0",
1191N/A # Despite its name, this only logs headers when there is an 1391N/A # error; it's redundant with the debug feature enabled. 1391N/A # Causes the headers of every request to be logged to the error 1391N/A # log; even if an exception occurs. 1660N/A "param":
"log.access_file",
849N/A # Since we've replaced cherrypy's log handler with our 1208N/A # own, we don't want the output directed to a file. 465N/A # Now that our logging, etc. has been setup, it's safe to perform any 465N/A # remaining preparation. 465N/A # Initialize repository state. 383N/A emsg(
"pkg.depotd: repository configuration error: %s" %
_e)
395N/A emsg(
"Please use the --set-property option to provide a value, " 383N/A "or update the cfg_cache file for the repository to " 849N/A emsg(
"pkg.depotd: repository configuration error: %s" %
_e)
849N/A # Initializing the repository above updated search indices 849N/A # as needed; nothing left to do, so exit. 383N/A # ready to start depot; exit now if requested 383N/A # Next, initialize depot. 383N/A # Now build our site configuration. 395N/A # We have to override cherrypy's default response_class so that 395N/A # we have access to the write() callable to stream data 290N/A # directly to the client. 395N/A # This changes the base URL for our server, and is primarily 290N/A # intended to allow our depot process to operate behind Apache 395N/A # or some other webserver process. 395N/A # Visit the following URL for more information: 1101N/A # Now merge or add our proxy configuration information into the 448N/A emsg(
"pkg.depotd: unknown error starting depot server, " \
1101N/A "illegal option value specified?")