svc-pkg-mirror revision 2872
75c0816e8295e180f4bc7f10db3d0d880383bc1cMark Andrews# CDDL HEADER START
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# The contents of this file are subject to the terms of the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Common Development and Distribution License (the "License").
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# You may not use this file except in compliance with the License.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# See the License for the specific language governing permissions
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# and limitations under the License.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# When distributing Covered Code, include this CDDL HEADER in each
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# If applicable, add the following below this CDDL HEADER, with the
cedb0bd0c1e3c461b7e479a16d3adfd5b150f1f4Mark Andrews# fields enclosed by brackets "[]" replaced with your own identifying
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# information: Portions Copyright [yyyy] [name of copyright owner]
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# CDDL HEADER END
cedb0bd0c1e3c461b7e479a16d3adfd5b150f1f4Mark Andrews# Copyright (c) 2013 Oracle and/or its affiliates. All rights reserved.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# This is the method script for the svc:/application/pkg/mirror service
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# When called using the 'start' or 'stop' SMF method script, it adds
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# or removes a crontab entry for the user running the service, pkg5srv
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# by default.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# When called using the 'refresh' method, it runs pkgrecv(1) to update a
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# pkg(5) repository using configuration stored in the SMF instance.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# The following SMF properties are used to configure the service:
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# config/repository the local pkg5 repository we update.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# config/ref_image the reference pkg5 image that contains
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# origin information that we should update
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# config/publishers a comma-separated list of the publishers
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# from ref_image that we pkgrecv from.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# config/crontab_period the first five fields of a crontab(4)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# entry, with the 3rd field allowing the
cedb0bd0c1e3c461b7e479a16d3adfd5b150f1f4Mark Andrews# special value 'random'.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# config/debug a boolean, 'true' or 'false'; whether
cedb0bd0c1e3c461b7e479a16d3adfd5b150f1f4Mark Andrews# to log more output when debugging.
cedb0bd0c1e3c461b7e479a16d3adfd5b150f1f4Mark Andrews# Load SMF constants and functions
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Since we deal with '*' values in crontab fields, we never want
727f5b8846457a33d06f515a10a7e1aa849ddf18Andreas Gustafsson# Multiple instances of this service should not point at the
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# same local repository, since they could step on each other's toes
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# during updates, so we check for this before enabling the service.
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# check_duplicate_repos
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein ALL_REPOS=$($SVCPROP -p config/repository "$SVCNAME:*" \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein REPOS=$($SVCPROP -p config/repository "$SVCNAME:*" \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # if the unique list of repositories is not the same as the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # list of repositories, then we have duplicates.
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# In order that all instances don't hit the remote origins on the same
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# day, when configured with a 'config/crontab_period' containing a
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# special value 'random' in the 'day of the month' field of the crontab
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# schedule, we randomize the day, choosing a value from 1-28, storing
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# that to the service config instead. We then print the crontab period.
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce# add_date_jitter
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce schedule=$($SVCPROP -p config/crontab_period $SMF_FMRI \
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce # Validate the cron_period property value, checking that we have
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce # exactly 5 fields, and that 'random' only appears in the 3rd
f293a69bcd1c1dd7bdac8f4102fc2398b9e475c8Eric Luce print "config/crontab_period property must contain 5 " \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein $1 == "random" || $2 == "random" || $4 == "random" || \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein $5 == "random" {
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein print "only field 3 can have the value random";
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein check_failure $? "invalid value for config/crontab_period." \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein new_schedule=$(echo "$schedule" | $SED -e "s/random/$RAND/1")
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # Save the schedule in the instance. Note that this
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # will not appear in the running instance until the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein new_schedule=$(echo $new_schedule| $SED -e 's/ /\\ /g')
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein config/crontab_period = astring: \"$new_schedule\"
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Add a crontab entry that does periodic pkgrecvs from a series of
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# remote pkg5 origins to a local repository. This is run as part of the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# SMF start method for this service. If the repository doesn't exist,
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# we create it. We also attempt to create a zfs dataset if the parent
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# directory for the repository is the leaf of a zfs dataset.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein check_failure $? "Two or more instances of $SVCNAME contain the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austeinsame 'config/repository' value, which is not supported." $SMF_FMRI exit
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset -f schedule=$(add_date_jitter | $SED -e 's/\\//g')
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset repo=$($SVCPROP -p config/repository $SMF_FMRI)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein set -A publishers $($SVCPROP -p config/publishers $SMF_FMRI)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein if [ ! -f $repo/pkg5.repository ]; then
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein check_failure $? "unable to create repository" \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Remove the crontab entry that was added by 'schedule_updates'. This is
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# run as part of the SMF stop method for this service.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Checks whether the given repository has a publisher/prefix set,
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# and if not, sets it to the given publisher.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# set_default_publisher <path to repo> <publisher>
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein DEFAULT=$($PKGREPO -s "$repo" get -H publisher/prefix | \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Intended to be called as part of a cron job firing, this calls
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# 'pkgrecv_from_origin' for each publisher configured in the SMF
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# update_repository <smf fmri>
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset instance=$(echo $SMF_FMRI | $AWK -F: '{print $NF}')
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset lockfile=/var/log/pkg/mirror/mirror.$instance.lock
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein if [ -f $lockfile ]; then
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein check_failure 1 "A mirror operation was already running
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein under process $pid when the cron job fired. Remove $lockfile to
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein override, or check the SMF property 'config/crontab_period' to ensure
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein cron schedules don't overlap." $SMF_FMRI degrade
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein check_failure $? "unable to create lockfile" $SMF_FMRI degrade
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset repo=$($SVCPROP -p config/repository $SMF_FMRI \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset cachedir=$($SVCPROP -p config/cache_dir $SMF_FMRI \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset ref_image=$($SVCPROP -p config/ref_image $SMF_FMRI\
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein set -A publishers $($SVCPROP -p config/publishers $SMF_FMRI)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo "ERROR: no publishers found in 'config/publishers'"
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # Gather the details we need to connect to the origins
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein $PKG -R $ref_image publisher -F tsv > /tmp/pkg.mirror.$$
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # 'pkg publisher -F tsv'. It really ought to use
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # 'pkg publisher -o' option when that's available.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein while read publisher sticky syspub enabled ptype status \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austeinconfigured for publisher $pub"
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein if [ $? -eq 0 ]; then
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein if [ $? -ne 0 ]; then
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# When retrieving values from SMF, we can get the string '""'
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# (two quotes) returned. For our purposes, this is equivalent to the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# null string, so we normalize it to ''. This function reads from stdin.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein while read value; do
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Perform a pkgrecv from the given origin to the given repository.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# We assume that the repository exists.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# pkgrecv_from_origin <repo> <origin> <key path> <cert path> <FMRI>
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# <cache dir> <http_proxy> <https_proxy>
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset https_proxy=$(echo $8 | reduce_null_str)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset instance=$(echo $SMF_FMRI | $AWK -F: '{print $NF}')
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset debug_flag=$($SVCPROP -p config/debug $SMF_FMRI)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset LOG=/var/log/pkg/mirror/mirror.$instance.log
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo "$TSTAMP: $SMF_FMRI updates to $repo from $origin :" \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo $PKGRECV -s $origin -c "$cachedir"/$instance \
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein $PKGRECV -s $origin -c "$cachedir"/$instance -d "$repo" \
7208386cd37a2092c70eddf80cf29519b16c4c80Mark Andrews -m all-timestamps $key $cert '*' > $LOG.tmp 2>&1
5a4557e8de2951a2796676b5ec4b6a90caa5be14Mark Andrews elif [ $EXIT -ne 0 ]; then
7208386cd37a2092c70eddf80cf29519b16c4c80Mark Andrews # in the case of errors, getting the full pkgrecv output
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # otherwise, we only log messages containing pkg5 FMRIs
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # we only destroy the cache if a pkgrecv was successful
5a4557e8de2951a2796676b5ec4b6a90caa5be14Mark Andrews# $1 start | stop | an FMRI containing configuration
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo "Problem mirroring repository for $SMF_FMRI"
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo "Problem mirroring repository for $SMF_FMRI"
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# A note on logging.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# The following log files are created while this service is running:
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# /var/log/pkg/mirror/mirror.<instance>.log
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# This is the top-level log file for the service. This log
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# shows a summary of each pkgrecv, listing a timestamp and the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# packages that were received during that run of the cron job.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# /var/log/pkg/mirror/mirror.<instance>.run.<pid>
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# This is a temporary log file, which should contain very little
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# output - it exists to capture all other output from the service
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# If 'config/debug' is set, then this file will also include the
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# full pkgrecv(1) command that is executed.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# Another temporary log file, which captures the complete output
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# of each pkgrecv command as it runs. At the end of the pkgrecv
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# process, we extract a summary and append it to
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# mirror.<instance>.log. If 'config/debug' is set, the contents
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# of this log are appended to mirror.<instance>.log. If any errors
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# were encountered while running pkgrecv, the contents of this log
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein# are appended to mirror.<instance>.log.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset instance=$(echo $SMF_FMRI | $AWK -F: '{print $NF}')
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset LOG=/var/log/pkg/mirror/mirror.$instance.log
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein typeset debug_flag=$($SVCPROP -p config/debug $SMF_FMRI)
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # Most output should get captured by update_repository, but we
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # capture any remaining output.
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein echo "Mirror refresh failed: see $LOG for more detail."
60e5e10f8d2e2b0c41e8abad38cacd867caa6ab2Rob Austein # try to remove the cron job so we don't keep failing