========================
pyutilib.autotest README
========================

.. _YAML: http://yaml.org/

--------
Overview
--------

The ``pyutilib.autotest`` package provides a facility for automatically
configuring tests that are executed with Python's unittest package.  This
capability is tailored for tests where one or more *solvers* are 
applied to one or more *problems*.  This testing structure is particularly
useful for evaluating the execution of external executables on datasets.

There are three main steps for configuring and applying pyutilib.autotest:

1. Create a configuration file
2. Create a test driver plugin
3. Setup the Python module that will be used to apply the tests

These steps are described in the following subsections.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Creating a Configuration File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Currently, ``pyutilib.autotest`` only supports a YAML_ configuration file.  The
top-level collection in this configuration is a mapping with the following
keys:

python
  This section contains a list of Python expressions that are executed while
  setting up the test suites.

driver
  This section specifies the name of the test driver that is used to 
  execute tests.

solvers
  This section contains a mapping of solver names to solver options.

problems
  This section contains a mapping of problem names to problem options.

suites
  This section contains a mapping of test suite names to suite configurations.

The following example illustrates the structure of a YAML configuration file.
(This example is the file ``pyutilib.autotest/examples/autotest/example1.yml`` in
the ``pyutilib.autotest`` distribution.)

::

    driver: example

    python:
        - import example

    solvers:
        cat:
        cat2:
            name: cat
            cat_options: -n
        echo:

    problems:
        p1:
            file: README.txt
        p2:
            file: example.py
        p3:
            file: LoremIpsum1.txt

    suites:

        suite1:
            categories:
                - smoke
                - suite1
            solvers:
                cat:
                cat2:
                echo:
            problems:
                p1:

        suite2:
            categories:
                - nightly
                - suite2
            solvers:
                cat:
                cat2:
                echo:
            problems:
                p1:
                p2:
                p3:
            tests:
                - solver: cat
                  problem: p1
                - solver: cat2
                  problem: p2
                - solver: echo
                  problem: p3

        suite3:
            categories:
                - suite3
            solvers:
                cat:
                catx:
                    solver: cat
                    cat_options: -n
            problems:
                p1:

The test driver ``example`` is defined in the ``example.py`` file, which is
imported with the directives in the ``python`` section.

Within the ``solvers`` and ``problems`` sections, each solver and problem can
specify additional options that are passed into the test.  Note that these
options are not distinguished for the test;  option name conflicts will
results in unpredictable behavior.

Three solvers are defined in this example, which apply the unix ``cat`` and
``echo`` commands.  By default, the name of the solver is assumed to be the
name of the key for the solver map.  Solver ``cat2`` illustrates how a solver
name can be specified separately from the solver key.

The ``suites`` section defines one or more test suites.  Each test suite
consists of a mapping with the following sections:

categories
  This optional section contains a list of strings that are solver categories.
  These categories can be used to select characteristics of the test suites
  that are executed.

solvers
  This section contains a mapping of solvers.  Note that additional options can
  be provided here to further customize the solver behavior.  By default, the
  solver name is assumed to be the key value, but the ``solver`` key can be
  used to explicitly define the solver that is associated with a customized
  solver mapping.

problems
  This section contains a list of problems.  Note that additional options can
  be provided here to further customize the test behavior. By default, the
  problem name is assumed to be the key value, but the ``problem`` key can be
  used to explicitly define the problem that is associated with a customized
  problem mapping.

tests
  This optional section contains a list of ``solver``-``problem`` pairs, which
  defines the actual tests that are defined.  Note that the ``solver`` and 
  ``problem`` values map to the keys defined in the ``solvers`` and
  ``problems`` sections of a test suite;  thus, this section can use the names of customized
  solvers and problems that are not defined in the top-level ``solvers`` and
  ``problems`` sections.  Finally, if this section is omitted, then
  all combinations of solvers and problems are used to create tests.
  
In this example, three suites are defined to illustrate different features of
the test driver:

suite1
  A simple test suite in which all solvers are applied to a single problem.

suite2
  The ``tests`` section is specified to select a subset of all combinations of
  solvers and problems to test.

suite3
  The ``catx`` solver is customized from ``cat`` to operate like the ``cat2`` solver. 


~~~~~~~~~~~~~~~~~~~~~~
Creating a Test Driver
~~~~~~~~~~~~~~~~~~~~~~

The test configuration used by ``pyutilib.autotest`` is quite generic.  It
specifies what combinations of solvers and problems are to be tested, along
with their corresponding options.  However, it does not specify how the test
is performed.  This is done by a test driver class.

In general, test driver classes are required to be plugins that can be
dynamically created by ``pyutilib.autotest`` to execute tests.  The easiest
way to define a test driver plugin is to inherit from the ``TestDriverBase``
class.  For example, the following test driver is used in by the earlier 
test configuration;  this plugin is defined in
``pyutilib.autotest/examples/autotest/example.py``.

::

    import pyutilib.autotest
    from pyutilib.component.core import *
    import pyutilib.subprocess


    class ExampleTestDriver(pyutilib.autotest.TestDriverBase):
        """
        This test driver executes a unix command and compares its output
        with a baseline value.
        """

        alias('example')

        def run_test(self, testcase, name, options):
            """Execute a single test in the suite"""
            name = options.suite+'_'+name
            cmd = options.solver+' '
            if not options.cat_options is None:
                cmd += options.cat_options+' '
            cmd += options.file
            print "Running test suite '%s'  test '%s'  command '%s'" % \
                    (options.suite, name, cmd)
            pyutilib.subprocess.run(cmd, outfile=options.currdir+'test_'+name+".out")
            testcase.failUnlessFileEqualsBaseline(
                    options.currdir+'test_'+name+".out", 
                    options.currdir+'test_'+name+".txt")

The ``alias`` function is used to specify the name of this plugin;  this is
the name of the test driver used in the test configuration file.

The ``run_test`` method executes a single test in the test suite.  Note that
this method is passed in ``testcase``, which is the test suite class.  Thus,
this method can directly apply the ``unittest`` methods that are defined in
this class (e.g. ``assertEquals``).

In this example, a unix command-line is create from the solver name, the
solver options, and the problem filename.  This is executed with the
``pyutilib.subprocess.run`` function, which redirects output to a log file.
This log file is then compared with a baseline file using the
``failUnlessFileEqualsBaseline``, which is defined in the ``unittest``
extensions in ``pyutilib.th``.

Note that a variety of other standard unit test methods can also be defined by
this test driver.  This driver is a ``ITestDriver`` plugin, and the API for
this plugin is:

::

    class ITestDriver(Interface):

        def setUpClass(self, cls, options):
            """Set-up the class that defines the suite of tests"""

        def tearDownClass(self, cls, options):
            """Tear-down the class that defines the suite of tests"""

        def setUp(self, testcase, options):
            """Set-up a single test in the suite"""

        def tearDown(self, testcase, options):
            """Tear-down a single test in the suite"""

        def run_test(self, testcase, name, options):
            """Execute a single test in the suite"""


~~~~~~~~~~~~~~~~~~~~~~
Creating a Test Module
~~~~~~~~~~~~~~~~~~~~~~

Virtually all of the work needed to create test suites is automated by
``pyutilib.autotest``.  The following test module is used in this example;
(see ``pyutilib.autotest/examples/autotest/autotest.py``):

::

    import os
    import sys
    from os.path import abspath, dirname
    currdir = dirname(abspath(__file__))+os.sep

    import pyutilib.th as unittest
    import pyutilib.autotest

    if __name__ == "__main__":
        pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals())
        unittest.main()

The first four lines are needed to identify the current directory, where
the test configuration file resides.

Note that ``pyutilib.th`` is imported as ``unittest``, which reminds the user
that this is a ``unittest`` extension package.  (Specifically, this package
contains hooks needed to dynamically add functions as test methods in test
suites.)

The ``pyutilib.autotest`` packages is imported so the ``create_test_suites``
function can be executed.  The arguments to this function are the test
configuration file, and the global dictionary.

Finally, ``unittest.main()`` is executed, as in any ``unittest`` module.
Tests can be executed using standard unittest command-line options.  One
extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES
or PYUTILIB_UNITTEST_CATEGORIES environmental variables;  if one of these
is specified, then ``pyutilib.autotest`` assumes that this data contains
a comma-separated list of categories that are used to select the test
suites that are constructed.  Specifically, if a test suite contains 
one of the specified test categories, then it will be executed.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``pyutilib_test_driver`` Command
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``pyutilib_test_driver`` command can be used to execute tests defined in a
configuration file without creating a test module.  In practice, test modules
are typically needed to support test discovery with tools like `nose`.
However, this command provides several features that are useful when
diagnosing tests.

The command-line behavior of ``pyutilib_test_driver`` extends the API of
``unittest.main()``.  The following additional options are provided to allow
the user to interrogate the tests that are defined by the test configuration
file:

--help-suites
    Print the test suites that can be executed
--help-tests=HELP_TESTS
    Print the tests in the specified test suite
--help-categories
    Print the test suite categories that can be specified



-------
License
-------

BSD.  See the LICENSE.txt file.


------------
Organization
------------

+ Directories

  * pyutilib - The root directory for PyUtilib source code

+ Documentation and Bug Tracking

  * Trac wiki: https://software.sandia.gov/trac/pyutilib

+ Authors

  * See the AUTHORS.txt file.

+ Project Managers

  * William Hart, wehart@sandia.gov

+ Mailing List

  * pyutilib-forum@googlegroups.com
    - The main list for help and announcements
  * pyutilib-developers@googlegroups.com
    - Where developers of PyUtilib discuss new features

--------------------
Third Party Software
--------------------

None.

