cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncRevamp of Automatic VirtualBox Testing
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync======================================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncIntroduction
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis is the design document for a revamped automatic testing framework.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe revamp aims at replacing the current tinderbox based testing by a new
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsystem that is written from scratch.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe old system is not easy to work with and was never meant to be used for
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmanaging tests, after all it just a simple a build manager tailored for
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccontiguous building. Modifying the existing tinderbox system to do what
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwe want would require fundamental changes that would render it useless as
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynca build manager, it would therefore end up as a fork. The amount of work
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncrequired would probably be about the same as writing a new system from
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncscratch. Other considerations, such as the license of the tinderbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsystem (MPL) and language it is realized in (Perl), are also in favor of
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdoing it from scratch.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe language envisioned for the new automatic testing framework is Python. This
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncis for several reasons:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - The VirtualBox API has Python bindings.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Python is used quite a bit inside Sun (dunno about Oracle).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Works relatively well with Apache for the server side bits.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - It is more difficult to produce write-only code in Python (alias the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync we-don't-like-perl argument).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - You don't need to compile stuff.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncNote that the author of this document has no special training as a test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncengineer and may therefore be using the wrong terms here and there. The
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncprimary focus is to express what we need to do in order to improve
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis document is written in reStructuredText (rst) which just happens to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncbe used by Python, the primary language for this revamp. For more
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncinformation on reStructuredText: http://docutils.sourceforge.net/rst.html
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncDefinitions / Glossary
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync======================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsub-test driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A set of test cases that can be used by more than one test driver. Could
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync also be called a test unit, in the pascal sense of unit, if it wasn't so
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync easily confused with 'unit test'.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This is somewhat ambiguous and this document try avoid using it where
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync possible. When used it normally refers to doing testing by executing one or
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync more testcases.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A set of inputs, test programs and expected results. It validates system
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync requirements and generates a pass or failed status. A basic unit of testing.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Note that we use the term in a rather broad sense.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A program/script used to execute a test. Also known as a test harness.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Generally abbreviated 'td'. It can have sub-test drivers.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest manager
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Software managing the automatic testing. This is a web application that runs
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync on a dedicated server (tindertux).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The output of testing activity. Logs, results, ++. Our usage of this should
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync probably be renamed to 'test run'.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A collection of related test cases.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A computer that does testing.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctestbox script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Script executing orders from the test manager on a testbox. Started
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync automatically upon bootup.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTODO: Check that we've got all this right and make them more exact
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync where possible.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncSee also http://encyclopedia2.thefreedictionary.com/testing%20types
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - A scalable test manager (>200 testboxes).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Optimize the web user interface (WUI) for typical workflows and analysis.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Efficient and flexibile test configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Import test result from other test systems (logo testing, VDI, ++).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Easy to add lots of new testscripts.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Run tests locally without a manager.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Revamp a bit at the time.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe Testbox Side
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncEach testbox has a unique name corresponding to its DNS zone entry. When booted
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynca testbox script is started automatically. This script will query the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmanager for orders and execute them. The core order downloads and executes a
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest driver with parameters (configuration) from the server. The test driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdoes all the necessary work for executing the test. In a typical VirtualBox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest this means picking a build, installing it, configuring VMs, running the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest VMs, collecting the results, submitting them to the server, and finally
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccleaning up afterwards.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe testbox environment which the test drivers are executed in will have a
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncnumber of environment variables for determining location of the source images
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncand other test data, scratch space, test set id, server URL, and so on and so
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOn startup, the testbox script will look for crash dumps and similar on
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsystems where this is possible. If any sign of a crash is found, it will
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncput any dumps and reports in the upload directory and inform the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmanager before reporting for duty. In order to generate the proper file
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncnames and report the crash in the right test set as well as prevent
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncreporting crashes unrelated to automatic testing, the testbox script will
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynckeep information (test set id, ++) in a separate scratch directory
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync(${TESTBOX_PATH_SCRATCH}/../testbox) and make sure it is synced to the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdisk (both files and directories).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAfter checking for crashes, the testbox script will clean up any previous test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwhich might be around. This involves first invoking the test script in cleanup
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmode and the wiping the scratch space.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWhen reporting for duty the script will submit information about the host: OS
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncname, OS version, OS bitness, CPU vendor, total number of cores, VT-x support,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAMD-V support, amount of memory, amount of scratch space, and anything else that
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccan be found useful for scheduling tests or filtering test configurations.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTestbox Script Orders
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync---------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe orders are kept in a queue on the server and the testbox script will fetch
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthem one by one. Orders that cannot be executed at the moment will be masked in
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthe query from the testbox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncExecute Test Driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Downloads and executes the a specified test driver with the given
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync configuration (arguments). Only one test driver can be executed at a time.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The server can specify more than one ZIP file to be downloaded and unpacked
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync before executing the test driver. The testbox script may cache these zip
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync files using http time stamping.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAbort Test Driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Aborts the current test driver. This will drop a hint to the driver and give
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync it 60 seconds to shut down the normal way. If that fails, the testbox script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync will kill the driver processes (SIGKILL or equivalent), invoke the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testdriver in cleanup mode, and finally wipe the scratch area. Should either
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync of the last two steps fail in some way, the testbox will be rebooted.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Ask again in X seconds, where X is specified by the server.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Reboot the testbox. If a test driver is current running, an attempt at
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync aborting it (Abort Test Driver) will be made first.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Updates the testbox script. The order includes a server relative path to the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync new testbox script. This can only be executed when no test driver is
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync currently being executed.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTestbox Environment: Variables
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This will be set to C:\Windows\System32\cmd.exe on Windows.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This will contain the kBuild binary directory for the host platform.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This will be set to point to kmk_ash(.exe) on all platforms.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_NAME
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The testbox name.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This is not required by the local reporter.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_PATH_BUILDS
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The absolute path to where the build repository can be found. This should be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync a read only mount when possible.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_PATH_RESOURCES
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The absolute path to where static test resources like ISOs and VDIs can be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync found. The test drivers knows the layout of this. This should be a read only
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync mount when possible.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_PATH_SCRATCH
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The absolute path to the scratch space. This is the current directory when
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync starting the test driver. It will be wiped automatically after executing the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync (Envisioned as ${TESTBOX_PATH_SCRIPTS}/../scratch and that
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync ${TESTBOX_PATH_SCRATCH}/ will be automatically wiped by the testbox script.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_PATH_SCRIPTS
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The absolute path to the test driver and the other files that was unzipped
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync together with it. This is also where the test-driver-abort file will be put.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync (Envisioned as ${TESTBOX_PATH_SCRATCH}/../driver, see above.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_PATH_UPLOAD
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The absolute path to the upload directory for the testbox. This is for
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync putting VOBs, PNGs, core dumps, crash dumps, and such on. The files should be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync bzipped or zipped if they aren't compress already. The names should contain
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync the testbox and test set ID.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_REPORTER
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The name of the test reporter back end. If not present, it will default to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync the local reporter.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_TEST_SET_ID
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The test set ID if we're running.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This is not required by the local reporter.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTESTBOX_MANAGER_URL
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The URL to the test manager.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync This is not required by the local reporter.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync There will probably be some more of these.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTestbox Environment: Core Utilities
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-----------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe testbox will not provide the typical unix /bin and /usr/bin utilities. In
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncother words, cygwin will not be used on Windows!
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe testbox will provide the unixy utilties that ships with kBuild and possibly
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsome additional ones from tools/*.*/bin in the VirtualBox tree (wget, unzip,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynczip, and so on). The test drivers will avoid invoking any of these utilites
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdirectly and instead rely on generic utility methods in the test driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncframework. That way we can more easily reimplement the functionality of the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccore utilites and drop the dependency on them. It also allows us to quickly
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwork around platform specific oddities and bugs.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Drivers
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe test drivers are programs that will do the actual testing. In addition to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncrun under the testbox script, they can be executed in the VirtualBox development
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncenvironment. This is important for bug analysis and for simplifying local
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctesting by the developers before commiting changes. It also means the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdrivers can be developed locally in the VirtualBox development environment.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe main difference between executing a driver under the testbox script and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncrunning it manually is that there is no test manager in the latter case. The
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest result reporter will not talk to the server, but report things to a local
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynclog file and/or standard out/err. When invoked manually, all the necessary
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncarguments will need to be specified by hand of course - it should be possible
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncto extract them from a test set as well.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncFor the early implementation stages, an implementation of the reporter interface
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthat talks to the tinderbox base test manager will be needed. This will be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdropped later on when a new test manager is ready.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAs hinted at in other sections, there will be a common framework
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync(libraries/packages/classes) for taking care of the tedious bits that every
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctest driver needs to do. Sharing code is essential to easing test driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdevelopment as well as reducing their complexity. The framework will contain:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - A generic way of submitting output. This will be a generic interface with
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync multiple implementation, the TESTBOX_REPORTER environment variable
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync will decide which of them to use. The interface will have very specific
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync methods to allow the reporter to do a best possible job in reporting the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync results to the test manager.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Helpers for typical tasks, like:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Copying files.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Deleting files, directory trees and scratch space.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Unzipping files.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Creating ISOs
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - And such things.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Helpers for installing and uninstalling VirtualBox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Helpers for defining VMs. (The VBox API where available.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Helpers for controlling VMs. (The VBox API where available.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe VirtualBox bits will be separate from the more generic ones, simply because
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthis is cleaner it will allow us to reuse the system for testing other products.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe framework will be packaged in a zip file other than the test driver so we
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdon't waste time and space downloading the same common code.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe test driver will poll for the file
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync${TESTBOX_PATH_SCRIPTS}/test-driver-abort and abort all testing when it sees it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe test driver can be invoked in three modes: execute, help and cleanup. The
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdefault is execute mode, the help shows an configuration summary and the cleanup
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncis for cleaning up after a reboot or aborted run. The latter is done by the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctestbox script on startup and after abort - the driver is expected to clean up
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncby itself after a normal run.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe Server Side
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync===============
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe server side will be implemented using a webserver (apache), a database
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync(postgres) and cgi scripts (Python). In addition a cron job (Python) running
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynconce a minute will generate static html for frequently used pages and maybe
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncexecute some other tasks for driving the testing forwards. The order queries
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncfrom the testbox script is the primary driving force in the system. The total
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmakes up the test manager.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe test manager can be split up into three rough parts:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Configuration (of tests, testgroups and testboxes).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Execution (of tests, collecting and organizing the output).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Analysis (of test output, mostly about presentation).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager: Requirements
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync==========================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncList of requirements:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Two level testing - L1 quick smoke tests and L2 longer tests performed on
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync builds passing L1. (Klaus (IIRC) ment this could be realized using
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync test dependency.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Black listing builds (by revision or similar) known to be bad.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Distinguish between build types so we can do a portion of the testing with
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync strict builds.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Easy to re-configure build source for testing different branch or for
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testing a release candidate. (Directory based is fine.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Useful to be able to partition testboxes (run specific builds on some
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync boxes, let an engineer have a few boxes for a while).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Interation with ILOM/...: reset systems.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Be able to suspend testing on selected testboxes when doing maintance
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync (where automatically resuming testing on reboot is undesired) or similar
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Abort testing on seleced testboxes.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Scheduling of tests requiring more than one testbox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Scheduling of tests that cannot be executing concurrently on several
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync machines because of some global resource like an iSCSI target.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Jump the scheduling queue. Scheduling of specified test the next time a
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testbox is available (optionally specifying which testbox to schedule it
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Configure tests with variable configuration to get better coverage. Two modes:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - TM generates the permutations based on one or more sets of test script arguments.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Each configuration permuation is specified manually.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Test specification needs to be flexible (select tests, disable test, test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync scheduling (run certain tests nightly), ... ).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Test scheduling by hour+weekday and by priority.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Test dependencies (test A depends on test B being successful).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Historize all configuration data, in particular test configs (permutations
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync included) and testboxes.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Test sets has at a minimum a build reference, a testbox reference and a
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync primary log associated with it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Test sets stores further result as a recursive collection of:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - hierachical subtest name (slash sep)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - test parameters / config
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - attributes (typed?)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - test time
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - e.g. throughput
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - subresults
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - screenshots, video,...
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - The test sets database structure needs to designed such that data mining
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync can be done in an efficient manner.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Presentation/analysis: graphs!, categorize bugs, columns reorganizing
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync grouped by test (hierarchical), overviews, result for last day.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager: Configuration
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync===========================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncConfiguration of testboxes doesn't involve much work normally. A testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncis added manually to the test manager by entering the DNS entry and/or IP
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncaddress (the test manager resolves the missing one when necessary) as well as
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthe system UUID (when obtainable - should be displayed by the testbox script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncinstaller). Queries from unregistered testboxes will be declined as a kind of
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsecurity measure, the incident should be logged in the webserver log if
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncpossible. In later dealings with the client the System UUID will be the key
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncidentifier. It's permittable for the IP address to change when the testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncisn't online, but not while testing (just imagine live migration tests and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncnetwork tests). Ideally, the testboxes should not change IP address.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe testbox edit function must allow changing the name and system UUID.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOne further idea for the testbox configuration is indicating what they are
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccapable of to filter out tests and test configurations that won't work on that
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctestbox. To examplify this take the ACP2 installation test. If the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmanager does not make sure the testbox have VT-x or AMD-v capabilities, the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncis surely going to fail. Other testbox capabilities would be total number of
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncCPU cores, memory size, scratch space. These testbox capabilities should be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccollected automatically on bootup by the testbox script together with OS name,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOS version and OS bitness.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncA final thought, instead of outright declining all requests from new testboxes,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwe could record the unregistered testboxes with ip, UUID, name, os info and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccapabilities but mark them as inactive. The test operator can then activate
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthem on an activation page or edit the testbox or something.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWe use the term testcase for a test.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTestcases are organized into groups. A testcase can be member of more than one
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncgroup. The testcase gets a priority assigned to it in connection with the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncgroup membership.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTestgroups are picked up by a testbox partition (aka scheduling group) and a
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncprioirty, scheduling time restriction and dependencies on other test groups are
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncassociated with the assignment. A testgroup can be used by several testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync(This used to be called 'testsuites' but was renamed to avoid confusion with
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthe VBox Test Suite.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe initial scheduler will be modelled after what we're doing already on in the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctinderbox driven testing. It's best described as a best effort continuous
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncintegration scheduler. Meaning, it will always use the latest build suitable
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncfor a testcase. It will schedule on a testcase level, using the combined
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncpriority of the testcase in the test group and the test group with the testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncpartition, trying to spread the test case argument varation out accordingly
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncover the whole scheduilng queue. Which argument variation to start with, is
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncnot undefined (random would be best).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncLater, we may add other schedulers as needed.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe Test Manager Database
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync=========================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncFirst a general warning:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The guys working on this design are not database experts, web
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync programming experts or similar, rather we are low level guys
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync who's main job is x86 & AMD64 virtualization. So, please don't
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync be too hard on us. :-)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncA logical table layout can be found in TestManagerDatabaseMap.png (created by
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOracle SQL Data Modeler, stored in TestManagerDatabase.dmd). The physical
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncdatabase layout can be found in TestManagerDatabaseInit.pgsql postgreSQL
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncscript. The script is commented.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncData History
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWe need to somehow track configuration changes over time. We also need to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncbe able to query the exact configuration a test set was run with so we can
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncunderstand and make better use of the results.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThere are different techniques for archiving this, one is tuple-versioning
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync( http://en.wikipedia.org/wiki/Tuple-versioning ), another is log trigger
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync( http://en.wikipedia.org/wiki/Log_trigger ). We use tuple-versioning in
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthis database, with 'effective' as start date field name and 'expire' as
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthe end (exclusive).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTuple-versioning has a shortcomming wrt to keys, both primary and foreign.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe primary key of a table employing tuple-versioning is really
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync'id' + 'valid_period', where the latter is expressed using two fields
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync([effective...expire-1]). Only, how do you tell the database engine that
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncit should not allow overlapping valid_periods? Useful suggestions are
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwelcomed. :-)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncForeign key references to a table using tuple-versioning is running into
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctrouble because of the time axsis and that to our knowledge foreign keys
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncmust reference exactly one row in the other table. When time is involved
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwhat we wish to tell the database is that at any given time, there actually
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncis exactly one row we want to match in the other table, only we've no idea
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynchow to express this. So, many foreign keys are not expressed in SQL of this
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncIn some cases, we extend the tuple-versioning with a generation ID so that
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncnormal foreign key referencing can be used. We only use this for recording
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync(references in testset) and scheduling (schedqueue), as using it more widely
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwould force updates (gen_id changes) to propagate into all related tables.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - http://en.wikipedia.org/wiki/Slowly_changing_dimension
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager: Execution
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync=======================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager: Scenarios
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync=======================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#1 - Testbox Signs On (At Bootup)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync---------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe testbox supplies a number of inputs when reporting for duty:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - IP address.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - System UUID.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - OS version.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - CPU architecture.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - CPU count (= threads).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - CPU VT-x/AMD-V capability.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - CPU nested paging capability.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Chipset I/O MMU capability.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Memory size.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Scratch size space (for testing).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Testbox Script revision.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - ACK or NACK.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Testbox ID and name on ACK.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAfter receiving a ACK the testbox will ask for work to do, i.e. continue with
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncscenario #2. In the NACK case, it will sleep for 60 seconds and try again.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync1. Validate the testbox by looking the UUID up in the TestBoxes table.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If not found, NACK the request. SQL::
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync SELECT idTestBox, sName
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync FROM TestBoxes
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync WHERE uuidSystem = :sUuid
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND tsExpire = 'infinity'::timestamp;
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2. Check if any of the information by testbox script has changed. The two
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync sizes are normalized first, memory size rounded to nearest 4 MB and scratch
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync space is rounded down to nearest 64 MB. If anything changed, insert a new
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync row in the testbox table and historize the current one, i.e. set
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync OLD.tsExpire to NEW.tsEffective and get a new value for NEW.idGenTestBox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync3. Check with TestBoxStatuses:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync a) If there is an row for the testbox in it already clean up change it
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync to 'idle' state and deal with any open testset like described in
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync scenario #9.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync b) If there is no row, add one with 'idle' state.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync4. ACK the request and pass back the idTestBox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncNote! Testbox.enabled is not checked here, that is only relevant when it asks
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync for a new task (scenario #2 and #5).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncNote! Should the testbox script detect changes in any of the inputs, it should
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync redo the sign in.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncNote! In scenario #8, the box will not sign on until it has done the reboot and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync cleanup reporting!
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#2 - Testbox Asks For Work To Do
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync---------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - The testbox is supplying its IP indirectly.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - The testbox should supply its UUID and ID directly.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - IDLE, WAIT, EXEC, REBOOT, UPGRADE, UPGRADE-AND-REBOOT, SPECIAL or DEAD.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync1. Validate the ID and IP by selecting the currently valid testbox row::
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync SELECT idGenTestBox, fEnabled, idSchedGroup, enmPendingCmd
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync FROM TestBoxes
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync WHERE id = :id
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND uuidSystem = :sUuid
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND ip = :ip
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND tsExpire = 'infinity'::timestamp;
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If NOT found return DEAD to the testbox client (it will go back to sign on
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync mode and retry every 60 seconds or so - see scenario #1).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Note! The WUI will do all necessary clean-ups when deleting a testbox, so
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync contrary to the initial plans, we don't need to do anything more for
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync the DEAD status.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2. Check with TestBoxStatuses (maybe joined with query from 1).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If enmState is 'gang-gathering': Goto scenario #6 on timeout or pending
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 'abort' or 'reboot' command. Otherwise, tell the testbox to WAIT [done].
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If enmState is 'gang-testing': The gang has been gathered and execution
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync has been triggered. Goto 5.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If enmState is not 'idle', change it to 'idle'.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If idTestSet is not NULL, CALL scenario #9 to it up.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If there is a pending abort command, remove it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If there is a pending command and the old state doesn't indicate that it was
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync being executed, GOTO scenario #3.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Note! There should be a TestBoxStatuses row after executing scenario #1,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync however should none be found for some funky reason, returning DEAD
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync will fix the problem (see above)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync3. If the testbox was marked as disabled, respond with an IDLE command to the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testbox [done]. (Note! Must do this after TestBoxStatuses maintainance from
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync point 2, or abandond tests won't be cleaned up after a testbox is disabled.)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync4. Consider testcases in the scheduling queue, pick the first one which the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testbox can execute. There is a concurrency issue here, so we put and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync exclusive lock on the SchedQueues table while considering its content.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The cursor we open looks something like this::
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync SELECT idItem, idGenTestCaseArgs,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync idTestSetGangLeader, cMissingGangMembers
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync FROM SchedQueues
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync WHERE idSchedGroup = :idSchedGroup
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND ( bmHourlySchedule is NULL
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync OR get_bit(bmHourlySchedule, :iHourOfWeek) = 1 ) --< does this work?
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync ORDER BY ASC idItem;
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If there no rows are returned (this can happen because no testgroups are
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync associated with this scheduling group, the scheduling group is disabled,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync or because the queue is being regenerated), we will tell the testbox to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync IDLE [done].
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync For each returned row we will:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync a) Check testcase/group dependencies.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync b) Select a build (and default testsuite) satisfying the dependencies.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync c) Check the testcase requirements with that build in mind.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync d) If idTestSetGangLeader is NULL, try allocate the necessary resources.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync e) If it didn't check out, fetch the next row and redo from (a).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync f) Tentatively create a new test set row.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync g) If not gang scheduling:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Next state: 'testing'
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync ElIf we're the last gang participant:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Set idTestSetGangLeader to NULL.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Set cMissingGangMembers to 0.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Next state: 'gang-testing'
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync ElIf we're the first gang member:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Set cMissingGangMembers to TestCaseArgs.cGangMembers - 1.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Set idTestSetGangLeader to our idTestSet.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Next state: 'gang-gathering'
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Decrement cMissingGangMembers.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Next state: 'gang-gathering'
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync If we're not gang scheduling OR cMissingGangMembers is 0:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Move the scheduler queue entry to the end of the queue.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Update our TestBoxStatuses row with the new state and test set.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync5. If state is 'testing' or 'gang-testing':
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync EXEC reponse.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync The EXEC response for a gang scheduled testcase includes a number of
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync extra arguments so that the script knows the position of the testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync it is running on and of the other members. This means the that the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync TestSet.iGangMemberNo is passed using --gang-member-no and the IP
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync addresses of the all gang members using --gang-ipv4-<memb-no> <ip>.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync Else (state is 'gang-gathering'):
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#3 - Pending Command When Testbox Asks For Work
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-----------------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis is a subfunction of scenario #2 and #5.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAs seen in scenario #2, the testbox will send 'abort' commands to /dev/null
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwhen it finds one when not executing a test. This includes when it reports
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthat the test has completed (no need to abort a completed test, wasting lot
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncof effort when standing at the finish line).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe other commands, though, are passed back to the testbox. The testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncscript will respond with an ACK or NACK as it sees fit. If NACKed, the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncpending command will be removed (pending_cmd set to none) and that's it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncIf ACKed, the state of the testbox will change to that appropriate for the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccommand and the pending_cmd set to none. Should the testbox script fail to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncrespond, the command will be repeated the next time it asks for work.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#4 - Testbox Uploads Results During Test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync----------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#5 - Testbox Completes Test and Asks For Work
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync---------------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis is very similar to scenario #2
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#6 - Gang Gathering Timeout
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync---------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis is a subfunction of scenario #2.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWhen gathering a gang of testboxes for a testcase, we do not want to wait
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncforever and have testboxes doing nothing for hours while waiting for partners.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncSo, the gathering has a reasonable timeout (imagine something like 20-30 mins).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAlso, we need some way of dealing with 'abort' and 'reboot' commands being
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncissued while waiting. The easy way out is pretent it's a time out.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWhen changing the status to 'gang-timeout' we have to be careful. First of all,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwe need to exclusively lock the SchedQueues and TestBoxStatuses (in that order)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncand re-query our status. If it changed redo the checks in scenario #2 point 2.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncIf we still want to timeout/abort, change the state from 'gang-gathering' to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync'gang-gathering-timedout' on all the gang members that has gathered so far.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThen reset the scheduling queue record and move it to the end of the queue.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWhen acting on 'gang-timeout' the TM will fail the testset in a manner similar
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncto scenario #9. No need to repeat that.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#7 - Gang Cleanup
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-----------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWhen a testbox completes a gang scheduled test, we will have to serialize
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncresource cleanup (both globally and on testboxes) as they stop. More details
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccan be found in the documentation of 'gang-cleanup'.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncSo, the transition from 'gang-testing' is always to 'gang-cleanup'. When we
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccan safely leave 'gang-cleanup' is decided by the query::
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync SELECT COUNT(*)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync FROM TestBoxStatuses,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync WHERE TestSets.idTestSetGangLeader = :idTestSetGangLeader
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync AND TestBoxStatuses.enmState = 'gang-running'::TestBoxState_T;
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAs long as there are testboxes still running, we stay in the 'gang-cleanup'
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncstate. Once there are none, we continue closing the testset and such.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#8 - Testbox Reports A Crash During Test Execution
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync--------------------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#9 - Cleaning Up Abandond Testcase
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync----------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis is a subfunction of scenario #1 and #2. The actions taken are the same in
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncboth situations. The precondition for taking this path is that the row in the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynctestboxstatus table is refering to a testset (i.e. testset_id is not NULL).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync1. If the testset is incomplete, we need to completed:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync a) Add a message to the root TestResults row, creating one if necesary,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync that explains that the test was abandond. This is done
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync by inserting/finding the string into/in TestResultStrTab and adding
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync a row to TestResultMsgs with idStrMsg set to that string id and
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync enmLevel set to 'failure'.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync b) Mark the testset as failed.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2. Free any global resources referenced by the test set. This is done by
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync deleting all rows in GlobalResourceStatuses matching the testbox id.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync3. Set the idTestSet to NULL in the TestBoxStatuses row.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync#10 - Cleaning Up a Disabled/Dead TestBox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-----------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe UI needs to be able to clean up the remains of a testbox which for some
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncreason is out of action. Normal cleaning up of abandond testcases requires
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthat the testbox signs on or asks for work, but if the testbox is dead or
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncin some way indisposed, it won't be doing any of that. So, the testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncsheriff needs to have a way of cleaning up after it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncIt's basically a manual scenario #9 but with some safe guards, like checking
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthat the box hasn't been active for the last 1-2 mins (max idle/wait time * 2).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncNote! When disabling a box that still executing the testbox script, this
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync cleanup isn't necessary as it will happen automatically. Also, it's
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync probably desirable that the testbox finishes what ever it is doing first
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync before going dormant.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager: Analysis
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync=======================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOne of the testbox sheriff's tasks is to try figure out the reason why something
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncfailed. The test manager will provide facilities for doing so from very early
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncin it's implementation.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWe need to work out some useful status reports for the early implementation.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncLater there will be more advanced analysis tools, where for instance we can
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccreate graphs from selected test result values or test execution times.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncImplementation Plan
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync===================
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThis has changed for various reasons. The current plan is to implement the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncinfrastructure (TM & testbox script) first and do a small deployment with the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2-5 test drivers in the Testsuite as basis. Once the bugs are worked out, we
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncwill convert the rest of the tests and start adding new ones.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncWe just need to finally get this done, no point in doing it piecemeal by now!
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTest Manager Implementation Sub-Tasks
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe implementation of the test manager and adjusting/completing of the testbox
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncscript and the test drivers are tasks which can be done by more than one
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncperson. Splitting up the TM implementation into smaller tasks should allow
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncparallel development of different tasks and get us working code sooner.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncMilestone #1
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe goal is to getting the fundamental testmanager engine implemented, debugged
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncand working. With the exception of testboxes, the configuration will be done
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncvia SQL inserts.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTasks in somewhat prioritized order:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Kick off test manager. It will live in testmanager/. Salvage as much as
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync possible from att/testserv. Create basic source and file layout.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Adjust the testbox script, part one. There currently is a testbox script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync in att/testbox, this shall be moved up into testboxscript/. The script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync needs to be adjusted according to the specification layed down earlier
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync in this document. Installers or installation scripts for all relevant
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync host OSes are required. Left for part two is result reporting beyond the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync primary log. This task must be 100% feature complete, on all host OSes,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync there is no room for FIXME, XXX or @todo here.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement the schedule queue generator.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement the testbox dispatcher in TM. Support all the testbox script
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync responses implemented above, including upgrading the testbox script.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement simple testbox management page.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement some basic activity and result reports so that we can see
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync what's going on.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Create a testmanager / testbox test setup. This lives in selftest/.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 1. Set up something that runs, no fiddly bits. Debug till it works.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 2. Create a setup that tests testgroup dependencies, i.e. real tests
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync depending on smoke tests.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 3. Create a setup that exercises testcase dependency.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 4. Create a setup that exercises global resource allocation.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync 5. Create a setup that exercises gang scheduling.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Check that all features work.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncMilestone #2
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe goal is getting to VBox testing.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTasks in somewhat prioritized order:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement full result reporting in the testbox script and testbox driver.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync A testbox script specific reporter needs to be implemented for the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync testdriver framework. The testbox script needs to forward the results to
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync the test manager, or alternatively the testdriver report can talk
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync directly to the TM.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement the test manager side of the test result reporting.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Extend the selftest with some setup that report all kinds of test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement script/whatever feeding builds to the test manager from the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync tinderboxes.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - The toplevel test driver is a VBox thing that must be derived from the
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync base TestDriver class or maybe the VBox one. It should move from
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync toptestdriver to testdriver and be renamed to vboxtltd or smth.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Create a vbox testdriver that boots the t-xppro VM once and that's it.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Create a selftest setup which tests booting t-xppro taking builds from
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync the tinderbox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncMilestone #3
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncThe goal for this milestone is configuration and converting current testscases,
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncthe result will be the a minimal test deployment (4-5 new testboxes).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncTasks in somewhat prioritized order:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement testcase configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement testgroup configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement build source configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement scheduling group configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement global resource configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Re-visit the testbox configuration.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Black listing of builds.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement simple failure analysis and reporting.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement the initial smoke tests modelled on the current smoke tests.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement installation tests for Windows guests.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement installation tests for Linux guests.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement installation tests for Solaris guest.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Implement installation tests for OS/2 guest.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - Set up a small test deployment.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncFurther work
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncAfter milestone #3 has been reached and issues found by the other team members
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynchave been addressed, we will probably go for full deployment.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncBeyond this point we will need to improve reporting and analysis. There may be
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncconfiguration aspects needing reporting as well.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncOnce deployed, a golden rule will be that all new features shall have test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsynccoverage. Preferrably, implemented by someone else and prior to the feature
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncimplementation.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsyncDiscussion Logs
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync===============
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2009-07-21,22,23 Various Discussions with Michal and/or Klaus
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync-------------------------------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Scheduling of tests requiring more than one testbox.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Scheduling of tests that cannot be executing concurrently on several machines
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync because of some global resource like an iSCSI target.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Manually create the test config permutations instead of having the test
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync manager create all possible ones and wasting time.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Distinguish between built types so we can run smoke tests on strick builds as
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync well as release ones.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2009-07-20 Brief Discussion with Michal
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync----------------------------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Installer for the testbox script to make bringing up a new testbox even
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2009-07-16 Raw Input
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync--------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- test set. recursive collection of:
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - hierachical subtest name (slash sep)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - test parameters / config
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - attributes (typed?)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - test time
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - e.g. throughput
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - subresults
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - screenshots,....
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- client package (zip) dl from server (maybe client caching)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- thoughts on bits to do at once.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - We *really* need the basic bits ASAP.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - client -> support for test driver
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - server -> controls configs
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync - cleanup on both sides
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync2009-07-15 Raw Input
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync--------------------
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- testing should start automatically
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- switching to branch too tedious
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- useful to be able to partition testboxes (run specific builds on some boxes, let an engineer have a few boxes for a while).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- test specification needs to be more flexible (select tests, disable test, test scheduling (run certain tests nightly), ... )
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- testcase dependencies (blacklisting builds, run smoketests on box A before long tests on box B, ...)
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- more testing flexibility, more test than just install/moke. For instance unit tests, benchmarks, ...
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- presentation/analysis: graphs!, categorize bugs, columns reorganizing grouped by test (hierarchical), overviews, result for last day.
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- testcase specificion, variables (e.g. I/O-APIC, SMP, HWVIRT, SATA...) as sub-tests
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- interation with ILOM/...: reset systems
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- Changes needs LDAP authentication
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- historize all configuration w/ name
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync- ability to run testcase locally (provided the VDI/ISO/whatever extra requirements can be met).
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync.. [1] no such footnote
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync:Status: $Id$
cf22150eaeeb72431bf1cf65c309a431454fb22bvboxsync:Copyright: Copyright (C) 2010-2014 Oracle Corporation.