z3c.testsetup
*************

Easy testsetups for Zope 3 and Python projects.

Setting up tests for Zope 3 projects sometimes tends to be
cumbersome. ``z3c.testsetup`` jumps in here, to support much flatter
test setups. The package supports normal Python `unit tests
<http://docs.python.org/library/unittest.html>`_ and
`doctests <http://docs.python.org/library/doctest.html>`_.

Note, that if you want integration or functional tests, that you have
to make sure, that the ``zope.app.testing`` package is available
during test runs. ``z3c.testsetup`` does **not** depend on it.

The package works in two steps:

1) It looks for testfiles in a given package.

2) It registers the tests according to your specifications.

.. note: Important note for users of ':Test-Layer:':

  The marker strings of `z3c.testsetup` changed!

  Please switch to the new syntax described below, if you are still
  using the old ':Test-Layer:' marker. It is more powerful and less
  magic.

This is a general introduction to ``z3c.testsetup``. For setup
examples you might see the ``othercave`` package contained in the
`tests/` directory. More details on special topics can be found in the
appropriate .txt files in this directory.


Basic Example
=============

Before we can find, register and execute tests, we first have to write
them down. We already have some ready to use tests available, which
can be found in a subpackage::

  >>> import os
  >>> cavepath = os.path.dirname(__file__)
  >>> cavepath = os.path.join(cavepath, 'tests', 'othercave')

In this subpackage there is a simple doctest `doctest01.txt` (please
ignore the pipes on the left)::

  >>> print_file(os.path.join(cavepath, 'doctest01.txt'))
  |  A doctest
  |  =========
  |  
  |  :doctest:
  |  
  |  This is a simple doctest.
  |  
  |    >>> 1+1
  |    2
  |  


As we can see, the doctest is marked by a special marker

   `:doctest:`. 

This marker tells the testsetup machinery, that this file contains
doctest examples that should be registered during test runs. Without
this marker, a testfile won't be registered during tests!

This is the only difference to 'normal' doctests here.

Other markers detected by ``z3c.testsetup`` are:

 - ``:unittest:``

   A replacement for ``:doctest:``, marking a Python module as
   containing unittests to run. Replaces old ``Test-Layer: python``
   marker.

 - ``:setup: <dotted.name.of.function>``

   Execute the given setup function before running doctests in this
   file.

 - ``:teardown: <dotted.name.of.function>``

   Execute the given teardown function after running doctests in this
   file.

 - ``:layer: <dotted.name.of.layer.def>``

   Use the given layer definition for tests in this file.

 - ``:zcml-layer: <ZCML_filename>``

   Use the given ZCML file and run tests in this file on a ZCML
   layer. Tests are registered using
   `zope.testing.doctest.DocFileSuite`.

 - ``:functional-zcml-layer: <ZCML_filename>``

   Use the given ZCML file and run tests in this file registered with
   `zope.app.testing.functional.DocFileSuite`.

See below for explanations of the respective markers.

.. note:: How to disable markers or make them invisible

   All markers can be written as restructured text comment (two
   leading dots followed by whitespace) like this::

     .. :doctest:

  and will still work. This way you can make the markers disappear
  from autogenerated docs etc. Markers are case-insensitive. If you
  want to disable a test, just turn ``:doctest:`` into ``:nodoctest:``
  and the file will be ignored.

Now, that we have a doctest available, we can write a testsetup
routine, that collects all tests, registers them and passes them to
the testrunner.

We have such a simple testsetup already available::

  >>> print open(os.path.join(cavepath, 'simplesetup01.py')).read()
  import z3c.testsetup
  test_suite = z3c.testsetup.register_all_tests(
      'z3c.testsetup.tests.othercave')

This is all we need in simple cases. We use

   `register_all_tests(<dotted_pkg_name>)` 

to tell the setup machinery, where to look for test files. Note, that
also files in subpackages will be found, registered and executed, when
they are marked approriately.

Let's start the testrunner and see what it gives::

  >>> import sys
  >>> sys.argv = [sys.argv[0],]
  >>> defaults = [
  ...     '--path', cavepath,
  ...     '--tests-pattern', '^simplesetup01$',
  ...     ]
  >>> from zope.testing import testrunner
  >>> testrunner.run(defaults)
    Running z3c...functional.layer.DefaultZCMLLayer [ftesting.zcml] tests:
      Set up z3c....layer.DefaultZCMLLayer [ftesting.zcml] in N.NNN seconds.
      Ran 3 tests with 0 failures and 0 errors in N.NNN seconds.
    Running z3c...functional.layer.DefaultZCMLLayer [ftesting2.zcml] tests:
      Tear down z3c...layer.DefaultZCMLLayer [ftesting.zcml] ... not supported
      Running in a subprocess.
      Set up z3c...layer.DefaultZCMLLayer [ftesting2.zcml] in N.NNN seconds.
      Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
      Tear down z3c...layer.DefaultZCMLLayer [ftesting2.zcml] ... not supported
    Running z3c.testsetup.tests.othercave.testing.UnitLayer2 tests:
      Running in a subprocess.
      Set up z3c.testsetup.tests.othercave.testing.UnitLayer1 in N.NNN seconds.
      Set up z3c.testsetup.tests.othercave.testing.UnitLayer2 in N.NNN seconds.
        Running testSetUp of UnitLayer1
        Running testSetUp of UnitLayer2
        Running testTearDown of UnitLayer2
        Running testTearDown of UnitLayer1
      Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
      Tear down z3c...tests.othercave.testing.UnitLayer2 in N.NNN seconds.
      Tear down z3c...tests.othercave.testing.UnitLayer1 in N.NNN seconds.
    Running zope.testing.testrunner.layer.UnitTests tests:
      Running in a subprocess.
      Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
        Custom setUp for  <DocTest doctest05.txt from ... (2 examples)>
        Custom tearDown for  <DocTest doctest05.txt from ... (2 examples)>
      Ran 7 tests with 0 failures and 0 errors in N.NNN seconds.
      Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
    Total: 12 tests, 0 failures, 0 errors in N.NNN seconds.
    False

As we can see, there were regular unittests as well as functional
tests run. Some of the unittests used their own layer (``UnitLayer1``)
whose location were printed and the functional tests used different
ZCML-files for configuration.

Of course, there were more tests than only the ones defined in
``doctest01.txt``. Let's have a look at the other stuff.


Defining doctests in Python modules
-----------------------------------

The doctest file described above was a pure .txt file. By default
``z3c.testsetup`` looks for doctests in files with filename extension
``.txt``, ``.rst`` and ``.py``. This means, that also doctests in
Python modules are found by default as in the following example::

  >>> print_file(os.path.join(cavepath, 'doctest08.py'))
  |  """
  |  Doctests in a Python module
  |  ===========================
  |  
  |  We can place doctests also in Python modules.
  |  
  |  :doctest:
  |  
  |  Here the Cave class is defined::
  |  
  |    >>> from z3c.testsetup.tests.othercave.doctest08 import Cave
  |    >>> Cave
  |    <class 'z3c.testsetup...doctest08.Cave'>
  |  
  |  """
  |  class Cave(object):
  |      """A Cave.
  |  
  |      A cave has a number::
  |  
  |        >>> hasattr(Cave, 'number')
  |        True
  |      
  |      """
  |      number = None
  |  
  |      def __init__(self, number):
  |          """Create a Cave.
  |  
  |          We have to give a number if we create a cave::
  |  
  |            >>> c = Cave(12)
  |            >>> c.number
  |            12
  |            
  |          """
  |          self.number = number
  |  

Here we placed the marker string ``:doctest:`` into the docstring of
the module. Without it, the module would not have been considered a
testfile.

Note that you have to import the entities (classes, functions, etc.)
from the very same file if you want to use them.


Registering regular unittests from Python modules
-------------------------------------------------

``z3c.testsetup`` provides also (limited) support for regular
`unittest` deployments as usually written in Python. An example file
could look like this::

  >>> print_file(os.path.join(cavepath, 'pythontest1.py'))
  |  """
  |  Tests with real TestCase objects.
  |  
  |  :unittest:
  |  
  |  """
  |  
  |  import unittest
  |  
  |  class TestTest(unittest.TestCase):
  |  
  |      def setUp(self):
  |          pass
  |  
  |      def testFoo(self):
  |          self.assertEqual(2, 1+1)
  |
  |

The module contains a marker ``:unittest:`` in its module docstring
instead of the ``:doctest:`` marker used in the other examples
above. It is also the replacement for the formely used ``:Test-Layer:
python`` marker.

This means, that this file is registered as a regular unittest.

If you use unittests instead of doctests, then you are mainly on your
own with setting up and tearing down tests. All this should be done by
the test cases themselves.

The only advantage of using ``z3c.testsetup`` here is, that those
tests are found and run automatically when they provide the marker.


``register_all_tests()``
========================

The `register_all_tests` function mentioned above accepts a bunch of
keyword parameters::

   register_all_tests(pkg_or_dotted_name [, extensions] [, encoding]
                      [, checker] [, globs] [, optionflags] 
                      [, setup] [, teardown]
                      [, zcml_config] [, layer_name] [, layer])

where all but the first parameter are keyword paramters and all but
the package parameter are optional.

While the `extensions` parameter determines the set of testfiles to be
found, the other paramters tell how to setup single tests.

The last five parameters are only fallbacks, that should better be
configured in doctest files themselves via marker strings.

- **extensions**:

    a list of filename extensions to be considered during doctest
    search. Default value for doctests is `['.txt', '.rst',
    '.py']`. Python tests are not touched by this (they have to be
    regular Python modules with '.py' extension).

    If we want to register .foo files, we can do so::

      >>> import z3c.testsetup
      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     extensions=['.foo'])
      >>> suite = test_suite()
      >>> get_basenames_from_suite(suite)
      ['file1.py', 'notatest1.foo', 'notatest1.foo']

    Note, that only files that contain an appropriate marker are
    found, regardless of the filename extension.


- **encoding**:

    the encoding of testfiles. 'utf-8' by default. Setting this to `None`
    means using the default value. We've hidden one doctest file, that
    contains umlauts. If we set the encoding to `ascii`, we get an
    error::

      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     encoding='ascii')
      >>> suite = test_suite()
      Traceback (most recent call last):
      ...
      UnicodeDecodeError: 'ascii' codec can't decode ...: ordinal 
      not in range(128)

    While using 'latin-1' will work::

      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     encoding='latin-1')
      >>> suite = test_suite()
      
    No traceback here.

    You can always overwrite an encoding setting for a certain file by
    following PEP 0263 ( http://www.python.org/dev/peps/pep-0263/ ).


- **checker**:

    An output checker for doctests. `None` by default. A typical
    output checker can be created like this::

      >>> import re
      >>> from zope.testing import renormalizing
      >>> mychecker = renormalizing.RENormalizing([
      ...    (re.compile('[0-9]*[.][0-9]* seconds'), 
      ...     '<SOME NUMBER OF> seconds'),
      ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
      ... ])

    This would match for example output like `0.123 seconds` if you
    write in your doctest::

      <SOME NUBMER OF> seconds

    Please see ``testrunner.txt`` for examples of usage.

    Checkers are applied to functional doctests only!


- **globs**:

    A dictionary of things that should be available immediately
    (without imports) during tests. Default is an empty dict, which
    might be populated by appropriate layers (see below). ZCML layers
    for example get you the ``getRootFolder`` method automatically.

    This parameter is a fallback which can be overriden by testfile
    markers specifying a certain layer (see below).

    The `globs` parameter applies only to doctests.


- **optionflags**:

    Optionflags influence the behaviour of the testrunner. They are
    logically or'd so that you can add them arithmetically. See

      http://svn.zope.org/zope.testing/trunk/src/zope/testing/doctest.py

    for details.


- **setup**:

    A callable that takes a `test` argument and is executed before
    every single doctest.

    The default function does nothing.

    This parameter is a fallback which can be overriden by testfile
    markers specifying a certain layer (see below).

    Specifying setup functions in a layer is also the recommended way.


- **teardown**:   

    The equivalent to `setup`.

    The default function runs

      zope.testing.cleanup.cleanUp()

    unless overriden by a layer.

    Specifying teardown functions in a layer is also the recommended
    way.


- **zcml_config**:

    A filepath of a ZCML file which is registered with functional
    doctests. In the ZCML file you can for example register principals
    (users) usable by functional doctests.

    By default any `ftesting.zcml` file from the root of the given
    package is taken. If this does not exist, an empty ZCML file of
    the z3c.testsetup package is used (``ftesting.zcml``).

    This parameter has no effect, if also a ``layer`` parameter is
    given or a docfile specifies its own layer/ZCML config (see below).

    This is a fallback parameter. Use of docfile specific layer markers
    is recommended.


- **layer_name**:

    You can name your layer, to distinguish different setups of
    functional doctests. The layer name can be an arbitrary string.

    This parameter has no effect, if also a ``layer`` parameter is
    given or a docfile specifies its own layer/ZCML config (see
    below).

    This is a fallback parameter. Use of docfile specific layer
    markers is recommended.

- **layer**:

    You can register a ZCML layer yourself and pass it as the
    ``layer`` parameter. If you only have a filepath to the according
    ZCML file, use the ``zcml_config`` paramter instead.

    This parameter overrides any ``zcml_config`` and ``layer_name``
    parameter.

    This is a fallback parameter and has no effect for docfiles
    specifying their own layer or ZCML config.


Deprectated/unsupported parameters
----------------------------------

The following ``register_all_tests``-parameters are deprecated,
starting with ``z3c.testsetup`` 0.3:

- **filter_func**

   and related (``ufilter_func``, ``pfilter_func``, etc.)

- All testtype specific parameters

  Support for testfile specific parameters (``uextensions``,
  ``fextensions``, etc.) is running out and its use deprecated.


Layers and setup/teardown functions
===================================

Starting with ``z3c.testsetup`` 0.3 there is first reasonable support
for setting up layers per testfile. This way you can easily create
setup-functions that are only run before/after certain tests.

Overall, use of layers is the recommended way from now on.


Setting up a unittest layer
---------------------------

We can tell ``z3c.testsetup`` to use a certain unittest layer using
the ``:layer:`` marker as in the following example (see
``tests/othercave/doctest02.txt``)::

    A doctests with layer
    =====================
    <BLANKLINE>
    :doctest:
    :layer: z3c.testsetup.tests.othercave.testing.UnitLayer2
    <BLANKLINE>
      >>> 1+1
      2


The ``:doctest:`` marker was used here as well, because without it the
file would not have been detected as a registerable doctest file (we
want developers to be explicit about that).

The 

 `:layer: <DOTTED_NAME_OF_LAYER_DEF>`

marker then tells, where the testsetup machinery can
find the layer definition. It is given in dotted name notation.

How does the layer definition look like? It is defined as regualr
Python code::

  >>> print open(os.path.join(cavepath, 'testing.py')).read()
  import os
  ...
  class UnitLayer1(object):
      """This represents a layer.
      A layer is a way to have common setup and teardown that happens
      once for a whole group of tests.
  <BLANKLINE>
      It must be an object with a `setUp` and a `tearDown` method, which
      are run once before or after all the tests applied to a layer
      respectively.
  <BLANKLINE>
      Optionally you can additionally define `testSetUp` and
      `testTearDown` methods, which are run before and after each single
      test.
  <BLANKLINE>
      This class is not instantiated. Therefore we use classmethods.
      """
  <BLANKLINE>
      @classmethod
      def setUp(self):
          """This gets run once for the whole test run, or at most once per
          TestSuite that depends on the layer.
          (The latter can happen if multiple suites depend on the layer
          and the testrunner decides to tear down the layer after first
          suite finishes.)
          """
  <BLANKLINE>
      @classmethod
      def tearDown(self):
          """This gets run once for the whole test run, or at most
          once per TestSuite that depends on the layer,
          after all tests in the suite have finished.
          """
  <BLANKLINE>
      @classmethod
      def testSetUp(self):
          """This method is run before each single test in the current
          layer. It is optional.
          """
          print "    Running testSetUp of UnitLayer1"
  <BLANKLINE>
      @classmethod
      def testTearDown(self):
          """This method is run before each single test in the current
          layer. It is optional.
          """
          print "    Running testTearDown of UnitLayer1"
  <BLANKLINE>
  class UnitLayer2(UnitLayer1):
      """This Layer inherits ``UnitLayer1``.
  <BLANKLINE>
      This way we define nested setups. During test runs the testrunner
      will first call the setup methods of ``UnitTest1`` and then those
      of this class. Handling of teardown-methods will happen the other
      way round.
      """
  <BLANKLINE>
      @classmethod
      def setUp(self):
          pass
  <BLANKLINE>
      @classmethod
      def testSetUp(self):
          print "    Running testSetUp of UnitLayer2"
  <BLANKLINE>
      @classmethod
      def testTearDown(self):
          print "    Running testTearDown of UnitLayer2"

In a layer you can do all the special stuff that is needed to run a
certain group of tests properly. Our setup here is special in that we
defined a nested one: ``UnitLayer2`` inherits ``UnitLayer1`` so that
during test runs the appropriate setup and teardown methods are called
(see testrunner output above).

More about test layers can be found at the documentation of
`testrunner layers API
<http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html>`_.

Specifying a ZCML file
----------------------

When it comes to integration or functional tests, we need to specify a
ZCML file to which configures the test environment for us. We can do
that using the

  `:zcml-layer: <ZCML-file-name>`

marker. It expects a ZCML filename as argument and sets up a
ZCML-layered testsuite for us. An example setup might look like so (see
``tests/othercave/doctest03.txt``)::

  A doctest with a ZCML-layer
  ===========================

  :doctest:
  :zcml-layer: ftesting.zcml

    >>> 1+1
    2

.. note:: Requires ``zope.app.testing``

   If you use ``:zcml-layer``, the ``zope.app.testing`` package must
   be available when running the tests and during test setup. This
   package is not fetched by default by ``z3c.testsetup``.

Here we say, that the the local file ``ftesting.zcml`` should be used
as ZCML configuration. As we can see in the above output of testruner,
this file is indeed read during test runs and used by a ZCML layer
called ``DefaultZCMLLayer``. This layer is in fact only a
``zope.app.testing.functional.ZCMLLayer``.

The ZCML file is looked up in the same directory as the doctest file.

When using the ``:zcml-layer:`` marker, the concerned tests are set up
via special methods and functions from `zope.app.testing`. This way
you get 'functional' or 'integration' tests out of the box: in the
beginning an empty ZODB db is setup, ``getRootFolder``, ``sync`` and
other functions are pulled into the test namespace and several things
more.

If you want a plain setup instead then use your own layer definition
using ``:layer:`` and remove the ``:zcml-layer:`` marker.


Setting up a functional ZCML layer
----------------------------------

Sometimes we want tests to be registered using the
``FunctionalDocFileSuite`` function from
``zope.app.testing.functional`` (other tests are set up using
``zope.testing.doctest.DocFileSuite``). This function pulls in even
more functions into ``globs``, like ``http`` (a ``HTTPCaller``
instance), wraps your ``setUp`` and ``tearDown`` methods into
ZODB-setups and several things more. See the definition in
http://svn.zope.org/zope.app.testing/trunk/src/zope/app/testing/functional.py?view=auto.

This setup needs also a ZCML configuration file, which can be
specified via::

  :functional-zcml-layer: <ZCML-file-name>

If a functional ZCML layer is specified in a testfile this way, it
will override any simple ``:zcml-layer:`` or ``:layer:`` definition.

An example setup might look like this (see
``tests/othercave/doctest04.txt``)::

  >>> print_file(os.path.join(cavepath, 'doctest04.txt'))
  |  A functional doctest with ZCML-layer
  |  ====================================
  |
  |  :doctest:
  |  :functional-zcml-layer: ftesting.zcml
  |
  |  We didn't define a real environment in ftesting.zcml, but in
  |  functional tests certain often needed functions should be available
  |  automatically::
  |
  |    >>> getRootFolder()
  |    <zope...folder.Folder object at 0x...>
  |

The placeholder between ``zope`` and ``folder`` was used because the
location of the Folder class changed recently. This way we cover
setups with old packages as well as recent ones.

.. note:: Requires ``zope.app.testing``

   If you use ``:zcml-layer``, the ``zope.app.testing`` package must
   be available when running the tests and during test setup. This
   package is not fetched by default by ``z3c.testsetup``.

Specifying ``setUp`` and ``tearDown`` methods
---------------------------------------------

We can specify a ``setUp(test)`` and ``tearDown(test)`` method for the
examples in a doctest file, which will be executed once for the whole
doctest file. This can be done using::

  :setup: <dotted.name.of.callable>
  :teardown: <dotted.name.of.callable>

The callables denoted by the dotted names must accept a ``test``
parameter which will be the whole test suite of examples in the
current doctest file.

An example can be found in ``doctest05.txt``::

  >>> print_file(os.path.join(cavepath, 'doctest05.txt'))
  |  A doctest with custom setup/teardown functions
  |  ==============================================
  |  
  |  :doctest:
  |  :setup: z3c.testsetup.tests.othercave.testing.setUp
  |  :teardown: z3c.testsetup.tests.othercave.testing.tearDown
  |  
  |    >>> 1+1
  |    2
  |  
  |  We make use of a function registered during custom setup::
  |  
  |    >>> myfunc(2)
  |    4
  |

The setup/teardown functions denoted in the example look like this::

  >>> print open(os.path.join(cavepath, 'testing.py'), 'rb').read()
  import os
  ...
  def setUp(test):
      print "    Custom setUp for ", test
      # We register a function that will be available during tests.
       test.globs['myfunc'] = lambda x: 2*x
  <BLANKLINE>
  def tearDown(test):
      print "    Custom tearDown for ", test
      del test.globs['myfunc'] # unregister function
  ...

As we can see, there is a function ``myfunc`` pulled into the
namespace of the doctest. We could, however, do arbitrary other things
here, set up a relational test database or whatever.
