A result is an outcome together with some annotations. The outcome indicates whether the test passed or failed. The annotations give additional information about the result, such as the manner in which the test failed, the output the test produced, or the amount of time it took to run the test.
The outcome of a test indicates whether it passed or failed, or whether some exceptional event occurred. There are four test outcomes:
PASS: The test succeeded.
FAIL: The test failed.
ERROR: A problem occurred in the test execution environment, rather than in the tested system. For example, this outcome is used when the test class attempted to run an executable in order to test it, but could not because the system call to create a new process failed.
This outcome may also indicate a defect in QMTest or in the test class.
UNTESTED: QMTest did not attempt to execute the test. For example, this outcome is used when QMTest determines that a prerequisite test failed.
Thus, running QMTest with the two previously defined tests will result in the following output:
>qmtest run... --- TEST RESULTS ------------------------------------------------------------- python_fail : FAIL Expression evaluates to false. python_pass : PASS ... --- STATISTICS --------------------------------------------------------------- 2 tests total 1 ( 50%) tests FAIL 1 ( 50%) tests PASS
In addition, the results are stored in a results file
(results.qmr by default).
An annotation is a key/value pair. Both the keys and values are strings. The value is HTML. When a test (or resource) runs it may add annotations to the result. These annotations are displayed by QMTest and preserved in the results file. If you write your own test class, you can use annotations to store information that will make your test class more informative.
The easiest way to create expectations is to tell QMTest that you expect future results to be the same as the results you just obtained. Thus, QMTest accepts result files obtained from prior test runs as expectations.
Thus, rerunning QMTest, but using
results.qmr as expectations, the test results are
displayed differently:
>qmtest run -O results.qmr... --- TEST RESULTS ------------------------------------------------------------- python_fail : XFAIL Expression evaluates to false. python_pass : PASS ... --- TESTS WITH UNEXPECTED OUTCOMES ------------------------------------------- None. --- STATISTICS --------------------------------------------------------------- 2 (100%) tests as expected