##############################################################################
#
# Copyright (c) 2008 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""
=========================
Functional Doc Test Setup
=========================

``FunctionalDocTestSetup`` helps to find and setup functional doctests
contained in a package. The most important method therefore might be
``getTestSuite()``, which searches a given package for doctest files
and returns all tests found as a suite of functional doctests.

Functional doctest setups find and register only doctests. Those tests
can also be Python modules but if you defined real `unttest.TestCase`
classes in your tests, then you most likely have 'normal' Python unit
tests. Please see 'pythontestsetup.txt' in this case. For simple unit
doctests, that do not require a more or less complex framework setup
to be done for each test, the setups described in
'unitdoctestsetup.txt' might suit your needs better.

There are also real 'oneliners' possible, that wrap around the classes
described here. See 'README.txt' to learn more about that.

The work is done mainly in two stages:

1) The package is searched for appropriate docfiles, based on the
   settings of instance attributes.

2) The tests contained in the found docfiles are setup as functional
   tests and added to a ``unittest.TestSuite`` instance.

There are plenty of default values active, if you use instances of
this class without further modifications. Therefore we will first
discuss the default behaviour and afterwards show, how you can modify
this behaviour to suit your special expectations on the tests.


Setting up a simple test suite
------------------------------

We want to register the doctests contained in the local ``cave``
package. This can be simply archieved by doing::

   >>> from z3c.testsetup import FunctionalDocTestSetup
   >>> setup = FunctionalDocTestSetup('z3c.testsetup.tests.cave')
   >>> setup
   <z3c.testsetup.doctesting.FunctionalDocTestSetup object at 0x...>

Apparently the package to handle was passed as a string in 'dotted
name' notation. We could also pass the package itself, if it was
loaded before::

   >>> from z3c.testsetup.tests import cave
   >>> setup = FunctionalDocTestSetup(cave)
   >>> setup
   <z3c.testsetup.doctesting.FunctionalDocTestSetup object at 0x...>   

This setup is ready for use::

   >>> suite = setup.getTestSuite()
   >>> suite
   <unittest.TestSuite tests=[...]>

To sum it up, writing a test setup for a project now can be that
short::

   import z3c.testsetup
   def test_suite():
       setup = z3c.testsetup.FunctionalDocTestSetup('z3c.testsetup.tests.cave')
       return setup.getTestSuite()

This will find all .rst and .txt files in the package that provide a
certain signature (see below), register the contained tests as
functional tests and run them as part of a `unittest.TestSuite`.

Note: in many test setups you will find a code fragment like the
      following at the end of file::

        if __name__ == '__main__':
            unittest.main(default='test_suite')

      This is not neccessary for usual testrunner setups. A testrunner
      will look for appropriate filenames (modules) and if those
      modules provide a callable ``test_suite`` (usually a function)
      this callable will be called to deliver a test suite.

FunctionalDocTestSetup default values
-------------------------------------

Understanding the defaults is important, because the default values
are driving the whole process of finding and registering the test.


Which files are found by default?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Basically, all files are accepted that

1) reside inside the package passed to the constructor. This includes
   subdirectories.

2) have a filename extension `.txt` or `.rst` (uppercase, lowercase
   etc. does not matter).

3) are *not* located inside a 'hidden' directory (i.e. a directory
   starting with a dot ('.'). Also subdirectories of 'hidden'
   directories are skipped.

4) contain a ReStructured Text meta-marker somewhere, that defines the
   file as a functional doctest explicitly::

       :Test-Layer: functional

   This means: there *must* be a line like the above one in the
   doctest file. The term might be preceeded or followed by whitspace
   characters (spaces, tabs).

Only files, that meet all four conditions are searched for functional
doctests. You can modify this behaviour of course, which will be
explained below in detail.


What options are set by default?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Many options can be set, when registering functional doctests. When
using the default set of options, the following values are set::

* The setup-instance's ``setUp``-method is set as the ``setUp``
  function.

* The setup-instance's ``tearDown``-method is set as the ``tearDown``
  function.
  
     >>> setup.setUp
     <bound method FunctionalDocTestSetup.setUp of
      <z3c.testsetup.doctesting.FunctionalDocTestSetup object at 0x...>>

* The setup-instance's `globs` attribute is passed as the `globs`
  parameter. By default `globs` is a dictionary of functions, that
  should be'globally' available during testruns and it contains::

     >>> setup.globs
     {'http': <zope.app.testing.functional.HTTPCaller object at 0x...>,
      'sync': <function sync at 0x...>,
      'getRootFolder': <function getRootFolder at 0x...>}

  The functions `sync` and `getRootFolder` are provided by
  `zope.app.testing.functional`.

* The setup-instance's `optionsflags` attribute is passed. It
  includes by default the following doctest constants:

     >>> from zope.testing import doctest
     >>> setup.optionflags == (doctest.ELLIPSIS+
     ...                       doctest.NORMALIZE_WHITESPACE |
     ...                       doctest.REPORT_NDIFF)
     True

* The setup-instances `encoding` attribute is passed. Setting it in
  the constructor will expect doctest files to provide the appropriate
  encoding. By default it is set to 'utf-8':

     >>> setup.encoding
     'utf-8'

  You can set it to a another value for differently encoded doctests.
  If no encoding is set (`encoding` is None), 7-bit ASCII will be
  assumed.

* The `checker` attribute helps to renormalize expected
  output. A typical output checker can be created like this::

     >>> import re
     >>> from zope.testing import renormalizing
     >>> mychecker = renormalizing.RENormalizing([
     ...    (re.compile('[0-9]*[.][0-9]* seconds'), 
     ...     '<SOME NUMBER OF> seconds'),
     ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
     ... ])

  By default a `FunctionalDocTest` instance has no checker::

     >>> setup.checker is None
     True

Because functional tests require a ZCML layer, that defines a ZCML
setup for the tests, we provide a layer, that is driven by the file
`ftesting.zcml`, which comes with `z3c.testsetup`. The layer is
accessible as the setup instance attribute `layer`::

   >>> setup.layer
   <zope.app.testing.functional.ZCMLLayer instance at 0x...>

   >>> setup.layer.config_file
   '...ftesting.zcml'

You can define a custom layer. This is described below.


No other options/parameters are set by default.


Customizing functional test setup:
----------------------------------

You can modify the behaviour of z3c.testsetup.FunctionalTestSetup such,
that a different set of files is registered and/or the found tests are
registered with a different set of parameters/options. We will first
discuss modifying the set of files to be searched.


Customizing the doctest file search:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The searching of appropriate doctest files is basically done by the
base class `BasicTestSetup`. Its purpose is to determine the set of
files in a package, that contain functional tests. See the testfile
`basicsetup.txt` to learn more about the procedure.

The functional test setup, however, provides a special
`isDocTestFile()` method, which does additional checking. Namely it
checks for the existance of the above mentioned ReStructured Text
meta-marker::

    `:Test-Layer: functional`

This is determined by a list of regular expressions, which is also
available as an object attribute::

    >>> setup.regexp_list
    ['^\\s*:(T|t)est-(L|l)ayer:\\s*(functional)\\s*']

This is the default value of functional doctest setups.

There are two files in the `cave` subpackage, which include that
marker. We can get the list of test files using
`getDocTestFiles()``::

    >>> testfile_list = setup.getDocTestFiles()
    >>> testfile_list.sort()
    >>> testfile_list
    ['...file1.txt', '...subdirfile.txt']

    >>> len(testfile_list)
    2

The ``isTestFile()`` method of our setup object did the filtering
here::

    >>> setup.isTestFile(testfile_list[0])
    True

The file file1.rst does not contain a functional test marker::

    >>> import os.path
    >>> path = os.path.join(os.path.dirname(cave.__file__),
    ...                     'test1.rst')
    >>> setup.isTestFile(path)
    False

The `regexp_list` attribute of a ``FunctionalTestSetup`` contains a
list of regular expressions, of which each one must at least match one
line of a searched file to be accepted. If you want to include files
with different marker-strings, just change this attribute. The value
will influence behaviour of the `isTestFile()``, ``getDocTestFiles()``
and ``getTestSuite()`` methods.

If you need more complex checks here, you can derive your customized
test setup class and overwrite ``isTestFile()``.

See `basicsetup.py` for further methods how to modify test file
search, for example by choosing another set of accepted filename
extensions.


Customizing the functional doctest setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To customize the setup of your tests, you have three options:

- Pass appropriate parameters to the constructor.

- Set attributes of an existing `FunctionalDocTestSetup` instance.

- Create your own class derived from `FunctionalDocTestSetup`.

The first two ways should suit most testing environments. All the
attributes mentioned above are settable at creation time, namely:

- `setup`::

       >>> def myfunc(test):
       ...     """A useless function."""
       ...     print "Hello!"
       >>> mysetup = FunctionalDocTestSetup(cave, setup=myfunc)
       >>> mysetup.setUp(None)
       Hello!

- `teardown`::

       >>> mysetup = FunctionalDocTestSetup(cave, teardown=myfunc)
       >>> mysetup.tearDown(None)
       Hello!

- `globs`:

   `globs` is a dictionary of things (objects, functions, vars) that
   are available for every test immediately (without import or
   similar)::

       >>> mysetup = FunctionalDocTestSetup(
       ...                cave, globs={'myfunc': myfunc})
       >>> mysetup.globs
       {'myfunc': <function myfunc at 0x...>}

- `optionflags`

    See the `zope.testing.doctest` module for all optionflags::

       >>> from zope.testing import doctest
       >>> mysetup = FunctionalDocTestSetup(
       ...     cave, optionflags=(doctest.ELLIPSIS +
       ...                        doctest.REPORT_UDIFF))
       >>> mysetup.optionflags & doctest.REPORT_NDIFF == 0
       True

       >>> mysetup.optionflags & doctest.REPORT_UDIFF == doctest.REPORT_UDIFF
       True

- `checker`

    An output checker for functional doctests. `None` by default. A
    typical output checker can be created like this::

       >>> import re
       >>> from zope.testing import renormalizing
       >>> mychecker = renormalizing.RENormalizing([
       ...    (re.compile('[0-9]*[.][0-9]* seconds'), 
       ...     '<SOME NUMBER OF> seconds'),
       ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>') ])

    Then, a setup with this checker can be created::

       >>> mysetup = FunctionalDocTestSetup(cave, checker = mychecker) 
       >>> mysetup.checker
       <zope.testing.renormalizing.RENormalizing instance at 0x...>

    Let's see, whether we got the wanted checker, by passing an
    example string, which should match the first of the terms defined
    in the checker::

       >>> mysetup.checker.check_output(
       ... '''\
       ...    Test took 0.012 seconds
       ... ''', '''\
       ...    Test took <SOME NUMBER OF> seconds
       ... ''', 0)
       True

     See the `zope.testing.renormalizing` module for more things, you
     can do with checkers.

- `encoding`

    If your doctests contain non-ASCII characters, this might lead to
    problems. You can circumvent this by setting an appropriate
    encoding string in the header of your doctest files. Another
    possibility is to pass the encoding keyword. By default
    z3c.testsetup uses 'utf-8' as default encoding::

       >>> setup.encoding
       'utf-8'

    But you can set it as you like::

       >>> mysetup = FunctionalDocTestSetup(cave, encoding = 'ascii')
       >>> mysetup.encoding
       'ascii'

To setup layers, there are the following constructor options
available:

- `zcml_config`

    The path to a ZCML file, often named `ftesting.zcml`. If a
    package, provides a file `ftesting.zcml` in its root, then this is
    taken as default. The ``cave_to_let`` package in the tests/
    directory provides such an `ftesting.zcml`::

       >>> from z3c.testsetup.tests import cave_to_let
       >>> setup = FunctionalDocTestSetup(cave_to_let)
       >>> pnorm(setup.layer.config_file)
       '.../tests/cave_to_let/ftesting.zcml'

    The fallback-solution is to take the layer from the
    `z3c.testsetup` package::

       >>> setup = FunctionalDocTestSetup(cave)
       >>> pnorm(setup.layer.config_file)
       '...z3c/testsetup/ftesting.zcml'

    Now the fallback `ftesting.zcml` was taken, because the cave got
    no own ftesting.zcml.

- `layer_name`

    A string. The default is ``FunctionalLayer``.

- `layer`

    A ``zope.app.testing.functional.ZCMLLayer`` object. Setting a
    layer overrides `zcml_config` and `layer_name`.

Their usage is explained in the next section.


Cutomizing the ZCML layer
+++++++++++++++++++++++++

The ZCML layer of a FunctionalDocTestSetup is searched in four steps:

1) If a setup is called with `layer` parameter, take this.

2) If a setup is called with `zcml_config` paramter, take this.

3) If an `ftesting.zcml` can be found in the root of the package to
   search, take this (default).

4) Take the (very poor) ftesting.zcml of the z3c.testsetup package
   (fallback).

The ZCML layer registered as fallback by ``z3c.testsetup`` is very
poor. In fact it only exists, to satisfy dependencies. In most cases,
you would like to write your own ZCML configuration to register
principals etc. during functional doctests.

For this purpose, ``FunctionalDocTestSetup`` supports the constructor
parameters `zcml_config` and `layer_name`. If the first is set, then
the ZCML file denoted by the path in this variable will be used
instead of the dummy ZCML contained in ``z3c.testsetup``.

   >>> setup_w_custom_layer = FunctionalDocTestSetup(
   ...     cave,
   ...     zcml_config = 'sampleftesting.zcml')
   >>> pnorm(setup_w_custom_layer.layer.config_file)
   '.../tests/cave/sampleftesting.zcml'

You can also pass a keyword parameter `layer`, which should provide a
value with a ready-to-use ZCML layer. If this happens, the
`zcml_config` and `layer_name` parameter will have no effect. 

To show this, we first create a custom layer::

   >>> from zope.app.testing.functional import ZCMLLayer
   >>> mylayer = ZCMLLayer(
   ...     os.path.join(os.path.dirname(__file__), 'ftesting.zcml'),
   ...     __name__,
   ...     'MyFunctionalLayer')

and create a functional doctest setup with it::

   >>> setup_w_custom_layer = FunctionalDocTestSetup(
   ...     cave,
   ...     zcml_config = 'sampleftesting.zcml',
   ...     layer = mylayer)
   >>> pnorm(setup_w_custom_layer.layer.config_file)
   '.../testsetup/ftesting.zcml'

As we can see, the `mylayer` config file is registered and the
`zcml_config` parameter was skipped.


"""
