0c27b3fe77ac1d5094ba3521e8142d9e7973133fMark AndrewsCopyright (C) 2000, 2001, 2004, 2016 Internet Systems Consortium, Inc. ("ISC")
0c27b3fe77ac1d5094ba3521e8142d9e7973133fMark AndrewsThis Source Code Form is subject to the terms of the Mozilla Public
0c27b3fe77ac1d5094ba3521e8142d9e7973133fMark AndrewsLicense, v. 2.0. If a copy of the MPL was not distributed with this
0c27b3fe77ac1d5094ba3521e8142d9e7973133fMark Andrewsfile, You can obtain one at http://mozilla.org/MPL/2.0/.
dafcb997e390efa4423883dafd100c975c4095d6Mark Andrews$Id: tests,v 1.15 2004/03/05 05:04:50 marka Exp $
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonWe do hourly test builds of the bind9 tree. This is an attempt to
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssondocument how they work.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson* How things work
bbad77bd5716239e70d684838f4b9a31d116f502Andreas GustafssonThe scripts driving the build system are in ~wpk/b9t. They are now
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonunder CVS control; the repository is in rc:/proj/cvs/isc/b9t (note
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonthat this is a separate repository from the bind9 one). The builds
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonare driven by cron jobs separately installed on each build system,
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonrunning as user wpk.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonThe sources are checked out, and the web reports are generated,
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssonon bb, as driven by the following cron jobs:
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafsson # Check out the current bind 9 version and make the source tarball.
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafsson # Argument to maketar.sh should be v9_0 for 9.0 release branch,
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafsson # HEAD for mainline.
bf5fa2c191361b0b06ab7c13cd31c0eb0a0ef824Andreas Gustafsson 35 2-22 * * * PLATFORM=BSD-3.1 && . $HOME/b9t/hosts/$PLATFORM/env && \
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafsson >/proj/build-reports/bind9/tarsrc.txt 2>&1
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson # run the bind 9 build status report generator
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson 30 3-22 * * * perl $HOME/b9t/bin/b9status.pl \
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson > /proj/build-reports/bind9/bind9.html 2> /dev/null
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonEach host has a separate crontab entry for building the server and
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssonrunning tests. Here are examples from bb and sol:
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson # build the BSD-3.1 version of bind 9
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson 0 3-22 * * * $HOME/b9t/bin/b9t.cron BSD-3.1
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson # bind 9 build for Solaris 5.6
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson 0 3-22 * * * $HOME/b9t/bin/b9t.cron SunOS-5.6
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonDo not confuse the shell script ~wpk/b9t/bin/b9t.cron with the crontab
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssontemplate (?) ~wpk/b9t/b9t.cron. Although they have the same name,
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonthey are not related.
bbad77bd5716239e70d684838f4b9a31d116f502Andreas GustafssonThe shell script b9t.cron then calls make, using the makefile b9t.mk
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonin the same location. This makefile moves the old status files out of
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafssonthe way and runs through the tests.
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas GustafssonThe current test schedule is as follows:
bf5fa2c191361b0b06ab7c13cd31c0eb0a0ef824Andreas Gustafsson :35 CVS tree extracted, tarball built and distributed
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson :00 Most tests begin
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson :45 Status report generator runs (was :30)
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson aix: I can't seem to access that machine; it appears to be down.
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson bb: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson durango: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson trantor: Build starts at top of hour, 0300 to 2100, odd-numbered hours
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson hp: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson irix: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson netbsd: Build starts at top of hour, 0300 to 2200 (was :45)
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson aa: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson rc: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson mirepoix: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson sol: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson truffle: Build starts at top of hour, 0300 to 2200
2188cdbef85c0be714e633fd1a5dfc571bf085adAndreas Gustafsson anthrax: Build starts at top of hour, 0300 to 2200
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonThe actual builds take place in a directory whose location differs
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssonamong systems. On most of them, it's on a local disk, under /build.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonOn some, it's on NFS; in this case the location is defined in
34274c3cd60edd969a34004012aed0b2cc2ab441Andreas GustafssonThe output from the make process is in
34274c3cd60edd969a34004012aed0b2cc2ab441Andreas Gustafsson~wpk/b9t/hosts/$PLATFORM/b9t-status, and the output from
34274c3cd60edd969a34004012aed0b2cc2ab441Andreas GustafssonThe output from the later stages of the process is under
40f53fa8d9c6a4fc38c0014495e7a42b08f52481David Lawrence/proj/build-reports/bind9/hosts/$PLATFORM. To make the files
34274c3cd60edd969a34004012aed0b2cc2ab441Andreas Gustafssonharder to find (?), they have names starting with a period:
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafsson* Common problems
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonSometime named processes fail to die when the tests are done,
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssoninterfering with the next test. Just kill them.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonOn hp.rc.vix.com, the tests often fail because of NFS I/O errors.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonWhen this happens, the machine needs to be rebooted. It will not
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssoncome up again without manually entering commands on the console.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonOn bb, the tests sometimes fail because .nfs* files stuck in the build
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssontree keep it from being completely deleted when the next test runs.
a47331123450129c1ad7d5326eecb7484b3e7348Andreas GustafssonThe .nfs* files cannot be deleted, but they can be moved, so one way
a47331123450129c1ad7d5326eecb7484b3e7348Andreas Gustafssonof fixing this is to move them to ~wpk.
bc13425bc389e526329bff6164952c4e3841cb70Andreas GustafssonOn aix, the tests routinely fail with an assertion failure related
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonto omapi socket handling - see RT #507.
bbad77bd5716239e70d684838f4b9a31d116f502Andreas Gustafsson* Failure locking
bbad77bd5716239e70d684838f4b9a31d116f502Andreas GustafssonWhen a test fails, further testing on that host is disabled in order
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonto preserve evidence. Also, tests don't start if they are already
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonrunning. Both of these rules are enforce through "lockout files"
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssoncraeted in /proj/build-reports/bind9/hosts/*/.
bc13425bc389e526329bff6164952c4e3841cb70Andreas GustafssonTo remove the lockout and allow more tests to be run, log in to bb,
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonsu, su wpk, and remove any "failed" and "running" files:
aba53c938bd29a268272540c6558efd968652ad4Andreas Gustafsson rm /proj/build-reports/bind9/hosts/*/failed
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafsson rm /proj/build-reports/bind9/hosts/*/running
bc13425bc389e526329bff6164952c4e3841cb70Andreas GustafssonThe "failed" file contains the time of failure, which is not
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonparticularly useful. The more useful information is in the various
bc13425bc389e526329bff6164952c4e3841cb70Andreas Gustafssonlog files under the build report.