Metadata-Version: 1.0
Name: pyutilib.autotest
Version: 2.0
Summary: A framework for generating test suites from external configuration files.
Home-page: https://software.sandia.gov/svn/public/pyutilib/pyutilib.autotest
Author: William Hart
Author-email: wehart@sandia.gov
License: BSD
Description: ========================
        pyutilib.autotest README
        ========================
        
        .. _YAML: http://yaml.org/
        
        --------
        Overview
        --------
        
        The ``pyutilib.autotest`` package provides a facility for automatically
        configuring tests that are executed with Python's unittest package.  This
        capability is tailored for tests where one or more *solvers* are
        applied to one or more *problems*.  This testing structure is particularly
        useful for evaluating the execution of external executables on datasets.
        
        There are three main steps for configuring and applying pyutilib.autotest:
        
        1. Create a configuration file
        2. Create a test driver plugin
        3. Setup the Python module that will be used to apply the tests
        
        These steps are described in the following subsections.
        
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Creating a Configuration File
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        
        Currently, ``pyutilib.autotest`` only supports a YAML_ configuration file.  The
        top-level collection in this configuration is a mapping with the following
        keys:
        
        python
        This section contains a list of Python expressions that are executed while
        setting up the test suites.
        
        driver
        This section specifies the name of the test driver that is used to
        execute tests.
        
        solvers
        This section contains a mapping of solver names to solver options.
        
        problems
        This section contains a mapping of problem names to problem options.
        
        suites
        This section contains a mapping of test suite names to suite configurations.
        
        The following example illustrates the structure of a YAML configuration file.
        (This example is the file ``pyutilib.autotest/examples/autotest/example1.yml`` in
        the ``pyutilib.autotest`` distribution.)
        
        ::
        
        driver: example
        
        python:
        - import example
        
        solvers:
        cat:
        cat2:
        name: cat
        cat_options: -n
        echo:
        
        problems:
        p1:
        file: README.txt
        p2:
        file: example.py
        p3:
        file: LoremIpsum1.txt
        
        suites:
        
        suite1:
        categories:
        - smoke
        - suite1
        solvers:
        cat:
        cat2:
        echo:
        problems:
        p1:
        
        suite2:
        categories:
        - nightly
        - suite2
        solvers:
        cat:
        cat2:
        echo:
        problems:
        p1:
        p2:
        p3:
        tests:
        - solver: cat
        problem: p1
        - solver: cat2
        problem: p2
        - solver: echo
        problem: p3
        
        suite3:
        categories:
        - suite3
        solvers:
        cat:
        catx:
        solver: cat
        cat_options: -n
        problems:
        p1:
        
        The test driver ``example`` is defined in the ``example.py`` file, which is
        imported with the directives in the ``python`` section.
        
        Within the ``solvers`` and ``problems`` sections, each solver and problem can
        specify additional options that are passed into the test.  Note that these
        options are not distinguished for the test;  option name conflicts will
        results in unpredictable behavior.
        
        Three solvers are defined in this example, which apply the unix ``cat`` and
        ``echo`` commands.  By default, the name of the solver is assumed to be the
        name of the key for the solver map.  Solver ``cat2`` illustrates how a solver
        name can be specified separately from the solver key.
        
        The ``suites`` section defines one or more test suites.  Each test suite
        consists of a mapping with the following sections:
        
        categories
        This optional section contains a list of strings that are solver categories.
        These categories can be used to select characteristics of the test suites
        that are executed.
        
        solvers
        This section contains a mapping of solvers.  Note that additional options can
        be provided here to further customize the solver behavior.  By default, the
        solver name is assumed to be the key value, but the ``solver`` key can be
        used to explicitly define the solver that is associated with a customized
        solver mapping.
        
        problems
        This section contains a list of problems.  Note that additional options can
        be provided here to further customize the test behavior. By default, the
        problem name is assumed to be the key value, but the ``problem`` key can be
        used to explicitly define the problem that is associated with a customized
        problem mapping.
        
        tests
        This optional section contains a list of ``solver``-``problem`` pairs, which
        defines the actual tests that are defined.  Note that the ``solver`` and
        ``problem`` values map to the keys defined in the ``solvers`` and
        ``problems`` sections of a test suite;  thus, this section can use the names of customized
        solvers and problems that are not defined in the top-level ``solvers`` and
        ``problems`` sections.  Finally, if this section is omitted, then
        all combinations of solvers and problems are used to create tests.
        
        In this example, three suites are defined to illustrate different features of
        the test driver:
        
        suite1
        A simple test suite in which all solvers are applied to a single problem.
        
        suite2
        The ``tests`` section is specified to select a subset of all combinations of
        solvers and problems to test.
        
        suite3
        The ``catx`` solver is customized from ``cat`` to operate like the ``cat2`` solver.
        
        
        ~~~~~~~~~~~~~~~~~~~~~~
        Creating a Test Driver
        ~~~~~~~~~~~~~~~~~~~~~~
        
        The test configuration used by ``pyutilib.autotest`` is quite generic.  It
        specifies what combinations of solvers and problems are to be tested, along
        with their corresponding options.  However, it does not specify how the test
        is performed.  This is done by a test driver class.
        
        In general, test driver classes are required to be plugins that can be
        dynamically created by ``pyutilib.autotest`` to execute tests.  The easiest
        way to define a test driver plugin is to inherit from the ``TestDriverBase``
        class.  For example, the following test driver is used in by the earlier
        test configuration;  this plugin is defined in
        ``pyutilib.autotest/examples/autotest/example.py``.
        
        ::
        
        import pyutilib.autotest
        from pyutilib.component.core import *
        import pyutilib.subprocess
        
        
        class ExampleTestDriver(pyutilib.autotest.TestDriverBase):
        """
        This test driver executes a unix command and compares its output
        with a baseline value.
        """
        
        alias('example')
        
        def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""
        name = options.suite+'_'+name
        cmd = options.solver+' '
        if not options.cat_options is None:
        cmd += options.cat_options+' '
        cmd += options.file
        print "Running test suite '%s'  test '%s'  command '%s'" % \
        (options.suite, name, cmd)
        pyutilib.subprocess.run(cmd, outfile=options.currdir+'test_'+name+".out")
        testcase.failUnlessFileEqualsBaseline(
        options.currdir+'test_'+name+".out",
        options.currdir+'test_'+name+".txt")
        
        The ``alias`` function is used to specify the name of this plugin;  this is
        the name of the test driver used in the test configuration file.
        
        The ``run_test`` method executes a single test in the test suite.  Note that
        this method is passed in ``testcase``, which is the test suite class.  Thus,
        this method can directly apply the ``unittest`` methods that are defined in
        this class (e.g. ``assertEquals``).
        
        In this example, a unix command-line is create from the solver name, the
        solver options, and the problem filename.  This is executed with the
        ``pyutilib.subprocess.run`` function, which redirects output to a log file.
        This log file is then compared with a baseline file using the
        ``failUnlessFileEqualsBaseline``, which is defined in the ``unittest``
        extensions in ``pyutilib.th``.
        
        Note that a variety of other standard unit test methods can also be defined by
        this test driver.  This driver is a ``ITestDriver`` plugin, and the API for
        this plugin is:
        
        ::
        
        class ITestDriver(Interface):
        
        def setUpClass(self, cls, options):
        """Set-up the class that defines the suite of tests"""
        
        def tearDownClass(self, cls, options):
        """Tear-down the class that defines the suite of tests"""
        
        def setUp(self, testcase, options):
        """Set-up a single test in the suite"""
        
        def tearDown(self, testcase, options):
        """Tear-down a single test in the suite"""
        
        def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""
        
        
        ~~~~~~~~~~~~~~~~~~~~~~
        Creating a Test Module
        ~~~~~~~~~~~~~~~~~~~~~~
        
        Virtually all of the work needed to create test suites is automated by
        ``pyutilib.autotest``.  The following test module is used in this example;
        (see ``pyutilib.autotest/examples/autotest/autotest.py``):
        
        ::
        
        import os
        import sys
        from os.path import abspath, dirname
        currdir = dirname(abspath(__file__))+os.sep
        
        import pyutilib.th as unittest
        import pyutilib.autotest
        
        if __name__ == "__main__":
        pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals())
        unittest.main()
        
        The first four lines are needed to identify the current directory, where
        the test configuration file resides.
        
        Note that ``pyutilib.th`` is imported as ``unittest``, which reminds the user
        that this is a ``unittest`` extension package.  (Specifically, this package
        contains hooks needed to dynamically add functions as test methods in test
        suites.)
        
        The ``pyutilib.autotest`` packages is imported so the ``create_test_suites``
        function can be executed.  The arguments to this function are the test
        configuration file, and the global dictionary.
        
        Finally, ``unittest.main()`` is executed, as in any ``unittest`` module.
        Tests can be executed using standard unittest command-line options.  One
        extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES
        or PYUTILIB_UNITTEST_CATEGORIES environmental variables;  if one of these
        is specified, then ``pyutilib.autotest`` assumes that this data contains
        a comma-separated list of categories that are used to select the test
        suites that are constructed.  Specifically, if a test suite contains
        one of the specified test categories, then it will be executed.
        
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        The ``pyutilib_test_driver`` Command
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        
        The ``pyutilib_test_driver`` command can be used to execute tests defined in a
        configuration file without creating a test module.  In practice, test modules
        are typically needed to support test discovery with tools like `nose`.
        However, this command provides several features that are useful when
        diagnosing tests.
        
        The command-line behavior of ``pyutilib_test_driver`` extends the API of
        ``unittest.main()``.  The following additional options are provided to allow
        the user to interrogate the tests that are defined by the test configuration
        file:
        
        --help-suites
        Print the test suites that can be executed
        --help-tests=HELP_TESTS
        Print the tests in the specified test suite
        --help-categories
        Print the test suite categories that can be specified
        
        
        
        -------
        License
        -------
        
        BSD.  See the LICENSE.txt file.
        
        
        ------------
        Organization
        ------------
        
        + Directories
        
        * pyutilib - The root directory for PyUtilib source code
        
        + Documentation and Bug Tracking
        
        * Trac wiki: https://software.sandia.gov/trac/pyutilib
        
        + Authors
        
        * See the AUTHORS.txt file.
        
        + Project Managers
        
        * William Hart, wehart@sandia.gov
        
        + Mailing List
        
        * pyutilib-forum@googlegroups.com
        - The main list for help and announcements
        * pyutilib-developers@googlegroups.com
        - Where developers of PyUtilib discuss new features
        
        --------------------
        Third Party Software
        --------------------
        
        None.
        
        
Keywords: utility
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Natural Language :: English
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Unix Shell
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering
