TODO revision a747113422afaa29ce72d2c5ba7f0b7ea9ec2054
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo WiedemeyerThings to do Automated Testing Framework
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo Wiedemeyer===========================================================================
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo WiedemeyerLast revised: November 30th, 2010
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo WiedemeyerThis document includes the list of things that need to be done in ATF that
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo Wiedemeyerare most requested by the users. This information used to be available in
0616b03c091e5dc3fe98e268ae0d27856cf7bd9bChristian Maederan ad-hoc bug tracker but that proved to be a bad idea. I have collected
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo Wiedemeyerall worthy comments in here.
0616b03c091e5dc3fe98e268ae0d27856cf7bd9bChristian MaederPlease note that most work these days is going into Kyua (see
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian Maederhttp://code.google.com/p/kyua/). The ideas listed here apply to the
0616b03c091e5dc3fe98e268ae0d27856cf7bd9bChristian Maedercomponents of ATF that have *not* been migrated to the new codebase yet.
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo WiedemeyerFor bug reports or ideas that apply to the components that already have
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo Wiedemeyerbeen migrated, please use the bug tracker in the URL above. Similarly,
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo Wiedemeyerwhenever a component is migrated, the ideas in this file should be revised
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo Wiedemeyerand migrated to the new bug tracker where appropriate.
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian Maeder---------------------------------------------------------------------------
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian MaederAdd build-time checks to atf-sh
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian MaederThe 0.7 release introduced build-time tests to atf-c and atf-c++, but not
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian Maederto atf-sh. Expose the functionality to the shell interface.
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian MaederThis will probably require writing an atf-build utility that exposes the C
d5f9a0b274192a496eb8d2fb8ce81c33ac2f1717Thiemo Wiedemeyercode and can be called from the shell.
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian Maeder---------------------------------------------------------------------------
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerRevisit what to do when an Atffile lists a non-existent file
d6638fccd3d43fe4499efca11cdc35a2825bcc9bThiemo Wiedemeyer---------------------------------------------------------------------------
66fd8f017efdb8a6c862c3f1856dfaef90865dd5Thiemo WiedemeyerAdd ATF_CHECK* versions to atf-c++ to support non-fatal tests
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo Wiedemeyer---------------------------------------------------------------------------
e5f5b1612cf7a4a1958c6bfd229f5bb8172879f6Christian MaederImplement race-condition tests
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo WiedemeyerI would think that stress/negative tests would be of more value than race
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo Wiedemeyercondition tests (they're similar, but not exactly the same in my mind).
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer1. Feed through as much data as possible to determine where reporting
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyer2. Feed through data quickly and terminate ASAP. The data should be
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyer captured. Terminate child applications with unexpected exit codes and
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate,
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer3. Open up a file descriptor in the test application, don't close the file
1842453990fed8a1bd7a5ac792d7982c1d2bfcd5Christian Maeder4. fork(2) a process; don't wait(2) for the application to complete.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerThere are other scenarios that could be exercised, but these are the ones
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerI could think of off the topic of my head.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer1. The thing is: how do you express any of this in a portable/abstract
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer interface? How do you express that a test case "receives data"? What
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer does that exactly mean? I don't think the framework should care about
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer this: each test should be free to decide where its data is and how to
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer3. Not sure I understand your request, but testing for "unexpected exit
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer codes" is already supported. See wiki:DesignXFail for the feature
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer design details.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer4. What's the problem with this case? The test case exits right away after
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer terminating the execution of its body; any open file descriptors,
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer leaked memory, etc. die with it.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer5. forking and not waiting for a subprocess was a problem already
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerI kinda have an idea of what Antti means with "race condition tests", but
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerevery time I have tried to describe my understanding of matters I seem to
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerbe wrong. Would be nice to have a clear description of what this involves;
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerin particular, what are the expectations from the framework and how should
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerthe feature be exposed.
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo WiedemeyerAs of now, what I understand by "race condition test" is: a test case that
654b14739364b0b04ecc6fa12e6ecca122eb6838Thiemo Wiedemeyerexercises a race condition. The test case may finish without triggering
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerthe race, in which case it just exists with a successful status.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerOtherwise, if the race triggers, the test case gets stuck and times out.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerThe result should be reported as an "expected failure" different from
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerYup. Plus some atf-wide mechanism for the operator to supply some kind of
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerguideline if the test should try to trigger the race for a second or for
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerAlright. While mocking up some code for this, I think that your two
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerrequests are complementary.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerOn the one hand, when you are talking about a "race condition" test you
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerreally mean an "expected race condition" test. Correct? If so, we need to
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerextend the xfail mechanism to add one more case, which is to report any
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerfailures as a race condition error and, if there is no failure, report the
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyertest as successful.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerOn the other hand, the atf-wide mechanism to support how long the test
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyershould run for can be thought as a "stress test" mechanism. I.e. run this
1842453990fed8a1bd7a5ac792d7982c1d2bfcd5Christian Maedertest for X time / iterations and report its results regularly without
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyerinvolving xfail at all.
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo WiedemeyerSo, with this in mind:
8836fa284a241af325aa6f41234b5130b26ec4f9Thiemo Wiedemeyer* For a test that triggers an unfixed race condition, you set xfail to
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyer race mode and define the test as a stress test. Any failures are
109b67ffce2bad83667e2f4a319d2d7f380f91afThiemo Wiedemeyer reported as expected failures.
0dba5bbaaef2f620f3b83a16ab6b229c3dd50c98Christian Maeder* For a test that verifies a supposedly-fixed race condition, you do *not*
66fd8f017efdb8a6c862c3f1856dfaef90865dd5Thiemo Wiedemeyer set xfail to race mode, and only set the test to stress test. Any
66fd8f017efdb8a6c862c3f1856dfaef90865dd5Thiemo Wiedemeyer failures are reported as real failures.
545d0cd78159cade346b579d06052638b19b0f72Thiemo WiedemeyerThese stress test cases implement a single iteration of the test and
fe1f344573c570949fa4f255c68bb9026c609832Thiemo Wiedemeyeratf-run is in charge of running the test several times, stopping on the
66fd8f017efdb8a6c862c3f1856dfaef90865dd5Thiemo WiedemeyerDoes that make sense?
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo Wiedemeyer---------------------------------------------------------------------------
a668bef7a7852dc5cf6208d7318bd5c949cc69bfThiemo WiedemeyerImplement ATF_REQUIRE_ERRNO
5107ba7da675778f2fded68493512b60eff8a455Christian MaederMost of the lines in tests against system functionality are:
5107ba7da675778f2fded68493512b60eff8a455Christian Maederif (syscall(args) == -1)
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyer atf_tc_fail_errno("flop")
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo WiedemeyerSome shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args))
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo WiedemeyerAlso, a variant which allows arbitrary return value checks (e.g. "!= 0" or
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyer"< 124" or "!= size") would be nice.
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo WiedemeyerThere's a problem with this request; not all functions fail in the same
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyerway ... in particular compare the pthread family of functions (which
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyerreturn errno) vs many native syscalls. Furthermore, compare some
4d3b50fc78a0c99ff2914fb23f3c7fba6e38d790Thiemo Wiedemeyerfcntl-like syscalls vs other syscalls. One size fits all solutions may not
macro/function name should be different because _ERRNO can be confusing.