===================================================
Doc-testing the graphical content of cairo surfaces
===================================================

While it is straight-forward to compare the content of two `cairo`_ surfaces
in Python code, handling graphics is beyond doc tests. However, the `manuel`_
package can be used to extract more general test cases from a text document
while allowing to mix them with doc tests in a natural way.

The ``tl.testing.cairo`` module provides a test suite factory that uses manuel
to execute graphical tests formulated as restructured-text figures. The
caption of such a figure is supposed to be a literal Python expression whose
value is a cairo surface, and its image is used as the test expectation.
Python expressions are run in the same context as the doc-test examples.
Images need to be stored in PNG format. Image paths are relative to the doc
test file's directory and must use the forward slash, "/", as the path
separator.


Writing a graphical test
========================

Let's walk through the process of creating a test. We'll test a function that
produces a cairo image surface with a black line drawn on it inside a thin
frame. As a first step, we implement the function and write a doc-test snippet
that includes a figure to be interpreted as a graphical test. The function is
intended to be passed in via the globs:

>>> import cairo
>>> def create_surface(x1, y1, x2, y2):
...     surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 100, 100)
...     ctx = cairo.Context(surface)
...     ctx.rectangle(0, 0, 100, 100)
...     ctx.move_to(x1, y1)
...     ctx.line_to(x2, y2)
...     ctx.stroke()
...     return surface

>>> sample_txt = write('sample.txt', """\
...
... ---------------------------------------------------
... A test for the graphical content of a cairo surface
... ---------------------------------------------------
...
... >>> type(create_surface)
... <type 'function'>
...
... The ``create_surface`` function creates and draws to a cairo surface:
...
... .. figure:: foo.png
...
...     ``create_surface(25, 50, 75, 50)``
...
... """)

The test suite has 2 tests, the doc test example and the graphical test.
Running it will yield an error as the expectation image is not available yet:

>>> from tl.testing.cairo import DocFileSuite
>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Error in test /test_dir/sample.txt
Traceback (most recent call last):
  ...
Exception: Could not load expectation: foo.png
  Ran 2 tests with 0 failures and 1 errors in 0.011 seconds.
<TEAR DOWN>

The test runner can help us with creating the missing image: we tell it to
save the image our function has drawn, examine the result and use it as the
expectation if we are satisfied with it. First we create a directory for
saving test results, store it in the ``CAIRO_TEST_RESULTS`` environment
variable and run the test suite again:

>>> import os, os.path
>>> os.mkdir('results')
>>> os.environ['CAIRO_TEST_RESULTS'] = 'results'
>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Error in test /test_dir/sample.txt
Traceback (most recent call last):
  ...
Exception: Could not load expectation: foo.png
(see results/foo.png)
  Ran 2 tests with 0 failures and 1 errors in 0.011 seconds.
<TEAR DOWN>

The test run has left a PNG file in the results directory [#no-results-dir]_.
The name of the file derives from the file name of the expected image of the
example in question. Let's make sure the file is actually there and has the
correct content, i.e. a black line from left to right inside a quadratic
frame:

>>> os.listdir('results')
['foo.png']

.. figure:: testimages/correct.png

    ``cairo.ImageSurface.create_from_png(os.path.join('results', 'foo.png'))``

Now we move the image file beside our doc test and run the test suite yet
again. This time, it will pass:

>>> import shutil
>>> shutil.move(os.path.join('results', 'foo.png'), 'foo.png')
>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
  Ran 2 tests with 0 failures and 0 errors in 0.003 seconds.
<TEAR DOWN>


Detecting bugs with a graphical test
====================================

As we hack on the ``create_surface`` function, we might introduce different
kinds of bugs which we expect to be reported as failures by our test suite.

First of all, our function might draw the wrong stuff to the surface, for
example by confusing the coordinate values we pass into it. Our test will tell
us that the image created has the wrong content and the saved result of the
example shows us a vertical line instead of a horizontal one:

>>> def create_surface(x1, y1, x2, y2):
...     surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 100, 100)
...     ctx = cairo.Context(surface)
...     ctx.rectangle(0, 0, 100, 100)
...     ctx.move_to(y1, x1)
...     ctx.line_to(y2, x2)
...     ctx.stroke()
...     return surface

>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Failure in test /test_dir/sample.txt
----------------------------------------------------------------------
File "/test_dir/sample.txt", line 10, in sample.txt:
Failed example:
    create_surface(25, 50, 75, 50)
Image differs from expectation: foo.png
(see results/foo.png)
  Ran 2 tests with 1 failures and 0 errors in 0.008 seconds.
<TEAR DOWN>

.. figure:: testimages/vertical.png

    ``cairo.ImageSurface.create_from_png(os.path.join('results', 'foo.png'))``

A mismatching image is also produced by choosing the wrong pixel format for
the ImageSurface. Our expectation is an image with an alpha channel; producing
a surface without one results in a format mismatch:

>>> def create_surface(x1, y1, x2, y2):
...     return cairo.ImageSurface(cairo.FORMAT_RGB24, 100, 100)

>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Failure in test /test_dir/sample.txt
----------------------------------------------------------------------
File "/test_dir/sample.txt", line 10, in sample.txt:
Failed example:
    create_surface(25, 50, 75, 50)
ImageSurface format differs from expectation:
Expected: cairo.FORMAT_ARGB32
Got:      cairo.FORMAT_RGB24
  Ran 2 tests with 1 failures and 0 errors in 0.008 seconds.
<TEAR DOWN>

Another mistake we might make is to return something else than a cairo
ImageSurface from our function under test:

>>> def create_surface(x1, y1, x2, y2):
...     return cairo.PDFSurface('out.pdf', 100, 100)

>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Failure in test /test_dir/sample.txt
----------------------------------------------------------------------
File "/test_dir/sample.txt", line 10, in sample.txt:
Failed example:
    create_surface(25, 50, 75, 50)
Expected a cairo.ImageSurface
Got:
    <cairo.PDFSurface object at 0x...>
Ran 2 tests with 1 failures and 0 errors in 0.008 seconds.
<TEAR DOWN>

Other bugs in our function might give rise to an exception. Exceptions raised
by a test example's expression are reported as failures:

>>> def create_surface(x1, y1, x2, y2):
...     return cairo.ImageSurface(cairo.FORMAT_NONSENSE, 100, 100)

>>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
<SET UP>
Failure in test /test_dir/sample.txt
----------------------------------------------------------------------
File "/test_dir/sample.txt", line 10, in sample.txt:
Failed example:
    create_surface(25, 50, 75, 50)
Exception raised:
    Traceback (most recent call last):
      File ".../cairo.py", line ..., in evaluate
        result = eval(self.expression, globs)
      ...
    AttributeError: 'module' object has no attribute 'FORMAT_NONSENSE'
Ran 2 tests with 1 failures and 0 errors in 0.008 seconds.
<TEAR DOWN>

As no images were computed by these last two ``create_surface``
implementations, none could be saved in either case.


Test suite options
==================

The test suite factory has a signature similar to that of
``doctest.DocFileSuite``, the only incompatibility being that the cairo
doc-test suite doesn't support file encodings. We've already seen globs being
passed to the test suite in the sections above.

Let's now demonstrate all features of the test suite (with the exception of
module-relative paths) at once - multiple test files, set-up and tear-down
handlers, globs, option flags and checkers for doc tests as well as specifying
an additional Manuel object:

>>> sample_txt = write('sample.txt', """\
... >>> surface = cairo.ImageSurface.create_from_png('rgb24.png')
... >>> surface
... <cairo.ImageSurface object at <MEM ADDRESS>>
...
... .. figure:: rgb24.png
...
...     ``surface``
... """)

>>> sumple_txt = write('sumple.txt', """\
... >>> dir(cairo)
... [...ImageSurface...]
...
... .. code-block:: python
...     surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 100, 100)
...     ctx = cairo.Context(surface)
...     ctx.rectangle(0, 0, 100, 100)
...     ctx.move_to(50, 25)
...     ctx.line_to(50, 75)
...     ctx.stroke()
...
... .. figure:: foo.png
...
...     ``surface``
... """)

>>> def set_up(test):
...     print '\nSETTING UP ONE TEST\n'

>>> def tear_down(test):
...     print '\nTEARING DOWN ONE TEST\n'

>>> import doctest
>>> import manuel.codeblock
>>> import re
>>> import zope.testing.renormalizing
>>> suite = DocFileSuite(
...     sample_txt, sumple_txt,
...     setUp=set_up, tearDown=tear_down,
...     globs={'cairo': cairo},
...     optionflags=doctest.ELLIPSIS,
...     checker=zope.testing.renormalizing.RENormalizing([
...         (re.compile('0x[0-9a-f]+'), '<MEM ADDRESS>')]),
...     manuel=manuel.codeblock.Manuel())
>>> run(suite)
<SET UP>
SETTING UP ONE TEST
TEARING DOWN ONE TEST
SETTING UP ONE TEST
Failure in test /test_dir/sumple.txt
----------------------------------------------------------------------
File "/test_dir/sumple.txt", line 11, in sumple.txt:
Failed example:
    surface
Image differs from expectation: foo.png
(see results/foo.png)
TEARING DOWN ONE TEST
  Ran 6 tests with 1 failures and 0 errors in 0.011 seconds.
<TEAR DOWN>

.. figure:: testimages/vertical.png

    ``cairo.ImageSurface.create_from_png(os.path.join('results', 'foo.png'))``


.. rubric:: Footnotes

.. _cairo: http://cairographics.org/pycairo/

.. _manuel: http://pypi.python.org/pypi/manuel

.. [#no-results-dir] **Non-existent test results directory**

    In the case that the result couldn't be written to the results directory,
    this is indicated by the failure message. To demonstrate this, we make the
    testrunner try to save the image to a non-existent directory temporarily:

    >>> os.environ['CAIRO_TEST_RESULTS'] = 'non-existent'
    >>> run(DocFileSuite(sample_txt, globs={'create_surface': create_surface}))
    <SET UP>
    Error in test /test_dir/sample.txt
    Traceback (most recent call last):
      ...
    Exception: Could not load expectation: foo.png
    (could not write result to non-existent/foo.png)
      Ran 2 tests with 0 failures and 1 errors in 0.011 seconds.
    <TEAR DOWN>
    >>> os.environ['CAIRO_TEST_RESULTS'] = 'results'


.. Local Variables:
.. mode: rst
.. End:
