Detailed documentation
**********************

The package works in two steps:

1) It looks for testfiles in a given package.

2) It registers the tests according to your specifications.


Initial notes
=============

* Between version 0.2 and 0.3 of ``z3c.testsetup`` a new set of testfile
  markers was introduced. If you are still using ``Test-Layer: unit`` or
  similar, please read the README.txt in the source directory carefully to
  learn how to switch to the new names. (Or see further down this page when
  reading it on pypi).

* Zope integration note: if you want zope integration or functional tests, you
  have to make sure, that the ``zope.app.testing`` and ``zope.component``
  packages are available during test runs. ``z3c.testsetup`` does **not**
  depend on it to make it usable for plain python packages.  If you want zope
  integration/functional tests, this is almost always already the case, so you
  don't need to care about this.

* If you download the source code, you can look at the examples used for
  testing and at text files that test technical aspects of z3c.testsetup.
  This can be handy when you want detailed knowledge about specific features.


Basic Example
=============

Before we can find, register and execute tests, we first have to write them
down.  z3c.testsetup includes examples used for testing (you can find them all
in the ``tests/`` subdirectory if you've downloaded the source code):

  >>> import os
  >>> cavepath = os.path.dirname(__file__)
  >>> cavepath = os.path.join(cavepath, 'tests', 'othercave')


Registering doctests
--------------------

In this example directory, there is a simple doctest ``doctest01.txt`` (please
ignore the pipes on the left):

  >>> print_file(os.path.join(cavepath, 'doctest01.txt'))
  |  A doctest
  |  =========
  |
  |  :doctest:
  |
  |  This is a simple doctest.
  |
  |    >>> 1+1
  |    2
  |

Important to note: the doctest is marked by a special marker that tells the
testsetup machinery that the file contains doctest examples that should be
registered during test runs::

   :doctest:

Without this marker, a testfile won't be registered during tests!  This is the
only difference compared to 'normal' doctests when you use z3c.testsetup.  If
you want to disable a test, just turn ``:doctest:`` into ``:nodoctest:`` (or
something else) and the file will be ignored.

.. note:: How to disable markers or make them invisible

   All markers can be written as restructured text comments (two leading dots
   followed by whitespace) like this::

     .. :doctest:

  and will still work. This way you can make the markers disappear
  from autogenerated docs etc.


Running the tests
-----------------

Now that we have a doctest available, we can write a testsetup routine that
collects all tests, registers them and passes them to the testrunner:

    >>> print open(os.path.join(cavepath, 'simplesetup01.py')).read()
    import z3c.testsetup
    test_suite = z3c.testsetup.register_all_tests(
        'z3c.testsetup.tests.othercave')

This is all we need in simple cases. We use
``register_all_tests(<dotted_pkg_name>)`` to tell the setup machinery, where
to look for test files. Note that also files in subpackages will be found,
registered and executed when they are marked approriately.

Let's start the testrunner and see what it gives:

    >>> import sys
    >>> sys.argv = [sys.argv[0],]
    >>> defaults = [
    ...     '--path', cavepath,
    ...     '--tests-pattern', '^simplesetup01$',
    ...     ]
    >>> from z3c.testsetup import testrunner
    >>> testrunner.run(defaults)
    Running z3c...layer.DefaultZCMLLayer [...ftesting.zcml] tests:
      Set up z3c...layer.DefaultZCMLLayer [...ftesting.zcml] in 2.146 seconds.
      Ran 3 tests with 0 failures and 0 errors in 0.009 seconds.
    Running z3c...layer.DefaultZCMLLayer [...ftesting2.zcml] tests:
      Tear down z3c...layer.DefaultZCMLLayer [...ftesting.zcml] ... not supported
      Running in a subprocess.
      Set up z3c...layer.DefaultZCMLLayer [...ftesting2.zcml] in 0.040 seconds.
      Ran 1 tests with 0 failures and 0 errors in 0.011 seconds.
      Tear down z3c...layer.DefaultZCMLLayer [...ftesting2.zcml] ... not supported
    Running z3c.testsetup.tests.othercave.testing.FunctionalLayer1 tests:
      Running in a subprocess.
      Set up z3c.testsetup.tests.othercave.testing.FunctionalLayer1 in 1.797 seconds.
      Ran 1 tests with 0 failures and 0 errors in 0.013 seconds.
      Tear down z3c.testsetup.tests.othercave.testing.FunctionalLayer1 in 0.001 seconds.
    Running z3c.testsetup.tests.othercave.testing.UnitLayer2 tests:
      Running in a subprocess.
      Set up z3c.testsetup.tests.othercave.testing.UnitLayer1 in 0.000 seconds.
      Set up z3c.testsetup.tests.othercave.testing.UnitLayer2 in 0.000 seconds.
        Running testSetUp of UnitLayer1
        Running testSetUp of UnitLayer2
        Running testTearDown of UnitLayer2
        Running testTearDown of UnitLayer1
      Ran 1 tests with 0 failures and 0 errors in 0.009 seconds.
      Tear down z3c.testsetup.tests.othercave.testing.UnitLayer2 in 0.000 seconds.
      Tear down z3c.testsetup.tests.othercave.testing.UnitLayer1 in 0.000 seconds.
    Running zope.testing.testrunner.layer.UnitTests tests:
      Running in a subprocess.
      Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
        Custom setUp for  <DocTest doctest05.txt from ...doctest05.txt:0 (2 examples)>
        Custom tearDown for  <DocTest doctest05.txt from ...doctest05.txt:0 (2 examples)>
      Ran 7 tests with 0 failures and 0 errors in 0.011 seconds.
      Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
    Total: 13 tests, 0 failures, 0 errors in 11.798 seconds.
    False

As we can see, there were regular unittests as well as functional tests
run. Some of the unittests used their own layer (``UnitLayer1``).  Layers are
shown in the output.  In this example, the functional tests use different
ZCML-files for configuration which results in separate test layers.


Finding doctests in Python modules
----------------------------------

The doctest file described above was a pure .txt file. By default
``z3c.testsetup`` looks for doctests in files with filename extension
``.txt``, ``.rst`` and ``.py``. This means, that also doctests in
Python modules are found just fine as in the following example:

  >>> print_file(os.path.join(cavepath, 'doctest08.py'))
  |  """
  |  Doctests in a Python module
  |  ===========================
  |
  |  We can place doctests also in Python modules.
  |
  |  :doctest:
  |
  |  Here the Cave class is defined::
  |
  |    >>> from z3c.testsetup.tests.othercave.doctest08 import Cave
  |    >>> Cave
  |    <class 'z3c.testsetup...doctest08.Cave'>
  |
  |  """
  |  class Cave(object):
  |      """A Cave.
  |
  |      A cave has a number::
  |
  |        >>> hasattr(Cave, 'number')
  |        True
  |
  |      """
  |      number = None
  |
  |      def __init__(self, number):
  |          """Create a Cave.
  |
  |          We have to give a number if we create a cave::
  |
  |            >>> c = Cave(12)
  |            >>> c.number
  |            12
  |
  |          """
  |          self.number = number
  |

Here we placed the marker string ``:doctest:`` into the docstring of
the module. Without it, the module would not have been considered a
testfile.

Note that you have to import the entities (classes, functions, etc.)
from the very same file if you want to use them.


Registering regular python unittests
------------------------------------

``z3c.testsetup`` provides also (limited) support for regular
``unittest`` deployments as usually written in Python. An example file
looks like this:

  >>> print_file(os.path.join(cavepath, 'pythontest1.py'))
  |  """
  |  Tests with real TestCase objects.
  |
  |  :unittest:
  |
  |  """
  |
  |  import unittest
  |
  |  class TestTest(unittest.TestCase):
  |
  |      def setUp(self):
  |          pass
  |
  |      def testFoo(self):
  |          self.assertEqual(2, 1+1)
  |
  |

The module contains a marker ``:unittest:`` in its module docstring instead of
the ``:doctest:`` marker used in the other examples above. This means that
the file is registered as a regular unittest. (It is also the replacement for
the formely used ``:Test-Layer: python`` marker.)

If you use unittests instead of doctests, then you are mainly on your
own with setting up and tearing down tests. All this should be done by
the test cases themselves.

The only advantage of using ``z3c.testsetup`` here is, that those
tests are found and run automatically when they provide the marker.


Registering the tests: ``register_all_tests()``
===============================================

The ``register_all_tests`` function mentioned above accepts a bunch of
keyword parameters::

   register_all_tests(pkg_or_dotted_name [, extensions] [, encoding]
                      [, checker] [, globs] [, optionflags]
                      [, setup] [, teardown]
                      [, zcml_config] [, layer_name] [, layer])

where all but the first parameter are keyword paramters and all but
the package parameter are optional.

While the ``extensions`` parameter determines the set of testfiles to be
found, the other paramters tell how to setup single tests.

The last five parameters are only fallbacks, that should better be
configured in doctest files themselves via marker strings.

- **extensions**:

    a list of filename extensions to be considered during doctest
    search. Default value for doctests is ``['.txt', '.rst',
    '.py']``. Python tests are not touched by this (they have to be
    regular Python modules with '.py' extension).

    If we want to register .foo files, we can do so:

      >>> import z3c.testsetup
      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     extensions=['.foo'])
      >>> suite = test_suite()
      >>> get_basenames_from_suite(suite)
      ['file1.py', 'notatest1.foo', 'notatest1.foo']

    Note, that only files that contain an appropriate marker are
    found, regardless of the filename extension.


- **encoding**:

    the encoding of testfiles. 'utf-8' by default. Setting this to ``None``
    means using the default value. We've hidden one doctest file, that
    contains umlauts. If we set the encoding to ``ascii``, we get an
    error:

      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     encoding='ascii')
      >>> suite = test_suite()
      Traceback (most recent call last):
      ...
      UnicodeDecodeError: 'ascii' codec can't decode ...: ordinal
      not in range(128)

    While using 'latin-1' will work:

      >>> test_suite = z3c.testsetup.register_all_tests(
      ...     'z3c.testsetup.tests.cave',
      ...     encoding='latin-1')
      >>> suite = test_suite()

    No traceback here.

    You can always overwrite an encoding setting for a certain file by
    following PEP 0263 ( http://www.python.org/dev/peps/pep-0263/ ).


- **checker**:

    An output normalizer for doctests. ``None`` by default. A typical output
    checker can be created like this:

      >>> import re
      >>> from zope.testing import renormalizing
      >>> mychecker = renormalizing.RENormalizing([
      ...    (re.compile('[0-9]*[.][0-9]* seconds'),
      ...     '<SOME NUMBER OF> seconds'),
      ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
      ... ])

    This would match for example output like ``0.123 seconds`` if you
    write in your doctest::

      <SOME NUBMER OF> seconds

    Define this checker in your testrunner .py file and add a
    ``checker=mychecker`` option to the ``register_all_tests()`` call.  Since
    version 0.5 checkers are applied to both regular and functional doctests
    (pre-0.5: only functional ones).


- **globs**:

    A dictionary of things that should be available immediately
    (without imports) during tests. Default is an empty dict, which
    might be populated by appropriate layers (see below). ZCML layers
    for example get you the ``getRootFolder`` method automatically.

    This parameter is a fallback which can be overriden by testfile
    markers specifying a certain layer (see below).

    The ``globs`` parameter applies only to doctests, not to plain python
    unittests.


- **optionflags**:

    Optionflags influence the behaviour of the testrunner. They are
    logically or'd so that you can add them arithmetically. See

      http://svn.zope.org/zope.testing/trunk/src/zope/testing/doctest.py

    for details.


- **setup**:

    A callable that takes a ``test`` argument and is executed before
    every single doctest.

    The default function does nothing.

    This parameter is a fallback which can be overriden by testfile
    markers specifying a certain layer (see below).

    Specifying setup functions in a layer is also the recommended way.


- **teardown**:

    The equivalent to ``setup``.

    The default function runs

      zope.testing.cleanup.cleanUp()

    unless overriden by a layer.

    Specifying teardown functions in a layer is also the recommended
    way.


- **allow_teardown**:

    A boolean whether teardown of zcml layers is allowed (*added in 0.5*).  In
    rare corner cases (like programmatically slapping an interface on a
    class), a full safe teardown of the zope component architecture is
    impossible.  Zope.testing has a default setting for allow_teardown of
    False, which means it uses the safe default of running every zcml layer in
    a separate process, which ensures full teardown.

    The drawback is that profiling and coverage tools cannot combine the
    profile/coverage data from separate processes.  z3c.testsetup has a
    default of False (since 0.5.1, True in 0.5).  If you want correct
    coverage/profiling output and your tests allow it, you'll have to set
    ``allow_teardown=False`` in your ``register_all_tests()`` call.


- **zcml_config**:

    A filepath of a ZCML file which is registered with functional
    doctests. In the ZCML file you can for example register principals
    (users) usable by functional doctests.

    By default any ``ftesting.zcml`` file from the root of the given
    package is taken. If this does not exist, an empty ZCML file of
    the z3c.testsetup package is used (``ftesting.zcml``).

    This parameter has no effect, if also a ``layer`` parameter is
    given or a docfile specifies its own layer/ZCML config (see below).

    This is a fallback parameter. Use of docfile specific layer markers
    is recommended.


- **layer_name**:

    You can name your layer, to distinguish different setups of
    functional doctests. The layer name can be an arbitrary string.

    This parameter has no effect, if also a ``layer`` parameter is
    given or a docfile specifies its own layer/ZCML config (see
    below).

    This is a fallback parameter. Use of per-doctest-file specific layer
    markers is recommended.

- **layer**:

    You can register a ZCML layer yourself and pass it as the
    ``layer`` parameter. If you only have a filepath to the according
    ZCML file, use the ``zcml_config`` paramter instead.

    This parameter overrides any ``zcml_config`` and ``layer_name``
    parameter.

    This is a fallback parameter and has no effect for docfiles
    specifying their own layer or ZCML config.


.. note::

  For more elaborate setups, you can use so-called TestGetters and
  TestCollectors.  They are explained in ``testgetter.txt`` in the source code
  (so you'll need to look there if you want to use it).



Available markers for configuring the tests
===========================================

We already saw the ``:doctest:`` marker above.  Other markers detected by
``z3c.testsetup`` are:

- ``:unittest:``

  A replacement for ``:doctest:``, marking a Python module as containing
  unittests to run. (Replaces old ``Test-Layer: python`` marker.)

- ``:setup: <dotted.name.of.function>``

  Execute the given setup function before running doctests in this
  file.

- ``:teardown: <dotted.name.of.function>``

  Execute the given teardown function after running doctests in this
  file.

- ``:layer: <dotted.name.of.layer.def>``

  Use the given layer definition for tests in this file. If the layer
  given is derived from ``zope.testing.functional.ZCMLLayer``, the test
  is registered using ``zope.app.testing.functional.FunctionalDocFileSuite``.

- ``:zcml-layer: <ZCML_filename>``

  Use the given ZCML file and run tests in this file on a ZCML
  layer. Tests are registered using ``zope.testing.doctest.DocFileSuite``.

- ``:functional-zcml-layer: <ZCML_filename>``

  Use the given ZCML file and run tests in this file registered with
  ``zope.app.testing.functional.FunctionalDocFileSuite``.

Markers are case-insensitive.

See further below for explanations of the respective markers.


Setting up a self-defined layer: ``:layer:``
--------------------------------------------

Starting with ``z3c.testsetup`` 0.3 there is reasonable support for setting up
layers per testfile. This way you can easily create setup-functions that are
only run before/after certain sets tests.

Overall, use of layers is the recommended way from now on.

We can tell ``z3c.testsetup`` to use a certain layer using the ``:layer:``
marker as in the following example (see ``tests/othercave/doctest02.txt``)::

    A doctests with layer
    =====================
    <BLANKLINE>
    :doctest:
    :layer: z3c.testsetup.tests.othercave.testing.UnitLayer2
    <BLANKLINE>
      >>> 1+1
      2

The ``:doctest:`` marker was used here as well, because without it the
file would not have been detected as a registerable doctest file (we
want developers to be explicit about that).

The ``:layer: <DOTTED_NAME_OF_LAYER_DEF>`` marker then tells, where the
testsetup machinery can find the layer definition. It is given in dotted name
notation and points to a layer you defined yourself.

How does the layer definition look like? It is defined as regualr
Python code:

  >>> print open(os.path.join(cavepath, 'testing.py')).read()
  import os
  ...
  class UnitLayer1(object):
      """This represents a layer.
      A layer is a way to have common setup and teardown that happens
      once for a whole group of tests.
  <BLANKLINE>
      It must be an object with a `setUp` and a `tearDown` method, which
      are run once before or after all the tests applied to a layer
      respectively.
  <BLANKLINE>
      Optionally you can additionally define `testSetUp` and
      `testTearDown` methods, which are run before and after each single
      test.
  <BLANKLINE>
      This class is not instantiated. Therefore we use classmethods.
      """
  <BLANKLINE>
      @classmethod
      def setUp(self):
          """This gets run once for the whole test run, or at most once per
          TestSuite that depends on the layer.
          (The latter can happen if multiple suites depend on the layer
          and the testrunner decides to tear down the layer after first
          suite finishes.)
          """
  <BLANKLINE>
      @classmethod
      def tearDown(self):
          """This gets run once for the whole test run, or at most
          once per TestSuite that depends on the layer,
          after all tests in the suite have finished.
          """
  <BLANKLINE>
      @classmethod
      def testSetUp(self):
          """This method is run before each single test in the current
          layer. It is optional.
          """
          print "    Running testSetUp of UnitLayer1"
  <BLANKLINE>
      @classmethod
      def testTearDown(self):
          """This method is run before each single test in the current
          layer. It is optional.
          """
          print "    Running testTearDown of UnitLayer1"
  <BLANKLINE>
  class UnitLayer2(UnitLayer1):
      """This Layer inherits ``UnitLayer1``.
  <BLANKLINE>
      This way we define nested setups. During test runs the testrunner
      will first call the setup methods of ``UnitTest1`` and then those
      of this class. Handling of teardown-methods will happen the other
      way round.
      """
  <BLANKLINE>
      @classmethod
      def setUp(self):
          pass
  <BLANKLINE>
      @classmethod
      def testSetUp(self):
          print "    Running testSetUp of UnitLayer2"
  <BLANKLINE>
      @classmethod
      def testTearDown(self):
          print "    Running testTearDown of UnitLayer2"

In a layer you can do all the special stuff that is needed to run a
certain group of tests properly. Our setup here is special in that we
defined a nested one: ``UnitLayer2`` inherits ``UnitLayer1`` so that
during test runs the appropriate setup and teardown methods are called
(see testrunner output above).

More about test layers can be found at the documentation of
`testrunner layers API
<http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html>`_.

Specifying a ZCML file: ``:zcml-layer:``
----------------------------------------

When it comes to integration tests which use zope's component architecture, we
need to specify a ZCML file which configures the test environment for us. We
can do that using the

  ``:zcml-layer: <ZCML-file-name>``

marker. It expects a ZCML filename as argument and sets up a
ZCML-layered testsuite for us. An example setup might look like so (see
``tests/othercave/doctest03.txt``)::

  A doctest with a ZCML-layer
  ===========================

  :doctest:
  :zcml-layer: ftesting.zcml

    >>> 1+1
    2

.. note:: Requires ``zope.app.testing``

   If you use ``:zcml-layer``, the ``zope.app.testing`` package must
   be available when running the tests and during test setup. This
   package is not fetched by default by ``z3c.testsetup``.

Here we say that the the local file ``ftesting.zcml`` should be used as ZCML
configuration. As we can see in the above output of testruner, this file is
indeed read during test runs and used by a ZCML layer called
``DefaultZCMLLayer``. This layer is in fact only a
``zope.app.testing.functional.ZCMLLayer``.

The ZCML file is looked up in the same directory as the doctest file. (You can
use relative paths like ``../ftesting.zcml`` just fine, btw).

When using the ``:zcml-layer:`` marker, the concerned tests are set up
via special methods and functions from ``zope.app.testing``. This way
you get 'functional' or 'integration' tests out of the box: in the
beginning an empty ZODB db is setup, ``getRootFolder``, ``sync`` and
other functions are pulled into the test namespace and several things
more.

If you want a plain setup instead then use your own layer definition
using ``:layer:`` and remove the ``:zcml-layer:`` marker.


Setting up a functional ZCML layer: ``:functional-zcml-layer:``
---------------------------------------------------------------

Sometimes we want tests to be registered using the
``FunctionalDocFileSuite`` function from
``zope.app.testing.functional`` (other tests are set up using
``zope.testing.doctest.DocFileSuite``). This function pulls in even
more functions into ``globs``, like ``http`` (a ``HTTPCaller``
instance), wraps your ``setUp`` and ``tearDown`` methods into
ZODB-setups and several things more. See the definition in
http://svn.zope.org/zope.app.testing/trunk/src/zope/app/testing/functional.py?view=auto.

This setup needs also a ZCML configuration file, which can be
specified via::

  :functional-zcml-layer: <ZCML-file-name>

If a functional ZCML layer is specified in a testfile this way, it
will override any simple ``:zcml-layer:`` or ``:layer:`` definition.

An example setup might look like this (see
``tests/othercave/doctest04.txt``):

  >>> print_file(os.path.join(cavepath, 'doctest04.txt'))
  |  A functional doctest with ZCML-layer
  |  ====================================
  |
  |  :doctest:
  |  :functional-zcml-layer: ftesting.zcml
  |
  |  We didn't define a real environment in ftesting.zcml, but in
  |  functional tests certain often needed functions should be available
  |  automatically::
  |
  |    >>> getRootFolder()
  |    <zope...folder.Folder object at 0x...>
  |

The placeholder between ``zope`` and ``folder`` was used because the
location of the Folder class changed recently. This way we cover
setups with old packages as well as recent ones.

.. note:: Requires ``zope.app.testing``

   If you use ``:zcml-layer``, the ``zope.app.testing`` package must
   be available when running the tests and during test setup. This
   package is not fetched by default by ``z3c.testsetup``.


Specifying test setup/teardown methods: ``:setup:`` and ``:teardown:``
----------------------------------------------------------------------

We can specify a ``setUp(test)`` and ``tearDown(test)`` method for the
examples in a doctest file, which will be executed once for the whole
doctest file. This can be done using::

  :setup: <dotted.name.of.callable>
  :teardown: <dotted.name.of.callable>

The callables denoted by the dotted names must accept a ``test``
parameter which will be the whole test suite of examples in the
current doctest file.  setup/teardown can be used to set up (and remove) a
temporary directory, to monkeypatch a mailer, etcetera.

An example can be found in ``doctest05.txt``:

  >>> print_file(os.path.join(cavepath, 'doctest05.txt'))
  |  A doctest with custom setup/teardown functions
  |  ==============================================
  |
  |  :doctest:
  |  :setup: z3c.testsetup.tests.othercave.testing.setUp
  |  :teardown: z3c.testsetup.tests.othercave.testing.tearDown
  |
  |    >>> 1+1
  |    2
  |
  |  We make use of a function registered during custom setup::
  |
  |    >>> myfunc(2)
  |    4
  |

The setup/teardown functions denoted in the example look like this:

  >>> print open(os.path.join(cavepath, 'testing.py'), 'rb').read()
  import os
  ...
  def setUp(test):
      print "    Custom setUp for ", test
      # We register a function that will be available during tests.
       test.globs['myfunc'] = lambda x: 2*x
  <BLANKLINE>
  def tearDown(test):
      print "    Custom tearDown for ", test
      del test.globs['myfunc'] # unregister function
  ...

As we can see, there is a function ``myfunc`` pulled into the
namespace of the doctest. We could, however, do arbitrary other things
here, set up a relational test database or whatever.


