Chapter 3. Customizing QMTest

Table of Contents

1. Extensions
2. Tests
3. Test Suites
4. Test Resources
5. Test Databases
6. Expectation Databases
7. Test Targets
8. Hosts
9. Result Streams and Result Readers
10. The QMTest Configuration File

The previous chapter introduced the concepts underlying QMTest's design. These were demonstrated with particular types of tests, resources, databases, etc. . These types are designed to be extensible, i.e. QMTest can operate with arbitrary instances of these types. In fact, QMTest provides a large number of such extensions.

This chapter discusses how to use them, i.e. how to find out what extensions are available, and how to use them to construct custom test databases.

1. Extensions

In the previous chapter we have created a test database containing a number of tests. These tests were instances of test types, such as python.ExecTest. When creating a test, a user chooses a test type, and assigns test-specific parameters to it.

The same extension mechanism is used for other aspects of QMTest's functionality. All these types are extensions, which provide a generic interface for QMTest to query.

To find out what extensions are available, use qmtest extensions. To list all available test types, use:

> qmtest extensions -k test
** Available test classes **

- command.ExecTest

  Check a program's output and exit code.

- command.ShellCommandTest

  Check a shell command's output and exit code.

- python.ExecTest

  Check that a Python expression evaluates to true.

...
   

To find out more about a particular extension type, use the describe command:

> qmtest describe test command.ExecTest

class name: python.ExecTest
   Check that a Python expression evaluates to true.

class attributes:
   prerequisites     The tests on which this test depends.
   source            The source code.
   target_group      The targets on which this test can run.
   expression        The expression to evaluate.
   resources         Resources on which this test or resource depends.
   

To create a test of that type, you can call:

> qmtest create --id=my_test -a expression='True' test python.ExecTest
   

where the extension parameters are specified with the -a option.

QMTest searches for extensions in particular places, in the following order:

  1. in the paths contained in the QMTEST_CLASS_PATH environment variable.

  2. in the class-path associated with a particular test database (typically the QMTest/ subdirectory).

  3. in the site-extension path associated with a particular QMTest installation.

  4. in the set of built-in extensions that come with a particular QMTest installation.

This search allows users to provide their own extension types. See Chapter 4, Extending QMTest for a detailed discussion of how to create new extensions.

The rest of this chapter will discuss the various extension kinds that can be used to customize QMTest.