If the test classes that come with QMTest do not serve your needs, you can write a new test class. A test class is a Python class derived from Test, which itself is derived from Extension. It may define parameters as discussed in Section 1, “Extension Classes”. The test class must provide a Run method that implements the way the test is performed and results are validated.

For example, if you want to test that a compiler correctly compiled a particular source file, the source file would be an argument to the test while the Run method would be responsible for running the compiler and the program generated by the compiler. The path to the compiler itself would be provided via the context ( Section 6, “Context”); that is an input to the testing system that varies depending on the user's environment.

The Run method takes two arguments: the context and the result. The context object is an instance of Context. The result object is an instance of Result. The result is initialized with the PASS outcome. Therefore, if the Run method does not modify the result, the test will pass. If the test fails, the Result.Fail method should be called to indicate failure.

The Result.Annotate method can be used to add information to the Result, whether or not the test passes. For example, annotations can be used to record the time a test took to execute, or to log the output from a command run as part of the test. Every annotation is a key/value pair. Both keys and values are strings. The key created by a test class C should have the form C.key_name. The value must be valid HTML. When results are displayed in the GUI, the HTML is presented directly to the user. When results are displayed as text, the HTML is converted to plain text. That conversion uses textual devices (such as single quotes around verbatim text) to emulate the HTML markup where possible.

As a convenience, you can use Python's dictionary notation to access annotations. For example:

    result["C.key1"] = "value"
    result["C.key2"] = result["C.key1"].upper()

  

is equivalent to:

    result.Annotate({ "C.key1" : "value"
                      "C.key2" : "VALUE" })

  

The context (like the result) is a set of key/value pairs. The keys used by a test class C should have the form C.key_name. The values are generally strings, but if a test depends on a resource, the resource can provide context values that are not strings.

If the Run raises an unhandled exception, QMTest creates a result for the test with the outcome ERROR. Therefore, test classes should be designed so that they do not raise unhandled exceptions when a test fails. However, QMTest handles the exception generated by the use of non-existant context variables specially. Because this situation generally indicates incorrect usage of the test suite, QMTest uses a special error message that instructs the user to supply a value for the context variable.