The PreviousTestRun queries expectations from a results file. Thus, running QMTest twice, the second time using the results of the first test run as expectations, will result in no unexpected results:
>qmtest run -o my_results.qmr--- TEST RESULTS ------------------------------------------------------------- exec0 : PASS exec1 : FAIL ... exec2 : PASS --- TESTS THAT DID NOT PASS -------------------------------------------------- exec1 : FAIL Expression evaluates to false. --- STATISTICS --------------------------------------------------------------- 3 tests total 1 ( 33%) tests FAIL 2 ( 67%) tests PASS>qmtest run -e previous_testrun.PreviousTestRun(file_name="my_results.qmr")--- TEST RESULTS ------------------------------------------------------------- exec0 : PASS exec1 : XFAIL ... exec2 : PASS --- TESTS WITH UNEXPECTED OUTCOMES ------------------------------------------- None. --- STATISTICS --------------------------------------------------------------- 3 (100%) tests as expected
Since taking previous test runs as expectations is a common use case, the second command above may be expressed in a more compact form as:
qmtest run -O my_results.qmrThe PreviousTestRun takes the following parameters.
The filename of the results file.
The XMLExpectationDatabase stores expectations in an XML file. Instead of containing expectations for all tests, individual expectations are computed from rules by matching test-ids as well as test-run annotations against specific rules.
<expectations>
<expectation outcome="fail" test_id=".*">
<annotation name="a" value="b"/>
</expectation>
<expectation outcome="fail" test_id="first*" />
</expectations>
The above little expectation database thus contains the following rules (subsequent matching rules override previous matching rules):
All tests are expected to fail if the annotation a has value
b.
All tests whose test-ids start with first are expected to fail.
The XMLExpectationDatabase takes the following parameters.
The filename of the xml expectations file.