.. -*-doctest-*-

===================
collective.funkload
===================

Complex functional load testing and benchmarking

``collective.funkload`` provides some extensions of `Funkload
<http://funkload.nuxeo.org/>`_, a web performance testing and
reporting tool.  These extensions provide flexible yet simple ways to:

  - run benchmarks of multiple test scenarios
  - run these benchmarks against multiple setups
  - generate comparisons between those setups

All of the console scripts provided by collective.funkload provide a
"--help" option which documents the command and it's options.

.. contents:: Table of Contents

The collective.funkload Workflow
================================

1. Develop test scenarios in Python eggs as with unittest
---------------------------------------------------------

Funkload test cases can be generated using the `funkload test recorder
<http://funkload.nuxeo.org/#test-recorder>`_ and then placed in a
Python egg's "tests" package as with normal `unittest
<http://docs.python.org/dev/library/unittest.html>`_ test cases.

These test cases should be developed which reflect all of the
application's supported usage patterns.  Take care to balance
between separating tests by usage scenario (anonymous visitors,
read access, write access, etc.) and keeping the number of tests
low enough to scan results.

2. Benchmark the baseline setup
-------------------------------

If there is no single baseline, such as when comparing multiple
setups to each other, then the term baseline here might be
slightly inaccurate.  The important part, however, is to
establish that the test cases successfully cover the usage
scenarios.

Use the fl-run-bench provided by collective.funkload using
`zope.testing.testrunner
<http://pypi.python.org/pypi/zope.testing#test-runner>`_ semantics to
specify the tests to run and the "--label" option to specify a label
indicating the benchmark it the baseline (or some other label if
baseline isn't appropriate)::

  $ fl-run-bench -s foo.loadtests --label=baseline
  Running zope.testing.testrunner.layer.UnitTests tests:
    Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
  ========================================================================
  Benching FooTestCase.test_FooTest
  ========================================================================
  ...
  Bench status: **SUCCESS**
    Ran # tests with 0 failures and 0 errors in # minutes #.### seconds.
  Tearing down left over layers:
    Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.

Use "fl-build-report --html" to build an HTML report from the XML
generated by running the benchmark above::

  $ fl-build-report --html FooTest-bench-YYYYMMDDThhmmss.xml
  Creating html report ...done:
  file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html

Examine the report details.  If the test cases don't sufficiently
cover the application's supported useage patterns, repeat steps 1
and 2 until the test cases provide sufficient coverage.

3. Benchmark the other setups
-----------------------------

In turn, deploy each of the setups.  This procedure will be
dictated by the application.  If using different buildout
configurations, for example, deploy each configuration::

  $ ...

Then use the same fl-run-bench command as before (or adusted as
needed for the setup) giving a differen "--label" option
dsignating the setup::

  $ fl-run-bench -s foo.loadtests --label=foo-setup
  Running zope.testing.testrunner.layer.UnitTests tests:
    Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
  ========================================================================
  Benching FooTestCase.test_FooTest
  ========================================================================
  ...
  Bench status: **SUCCESS**
    Ran # tests with 0 failures and 0 errors in # minutes #.### seconds.
  Tearing down left over layers:
    Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.

Repeat this step for each setup.

4. Build the HTML and differential reports and the matrix index
---------------------------------------------------------------

Use the "fl-build-label-reports" command with the "--x-label" and
"--y-label" options to automatically build all the HTML reports, the
differential reports based on the labels, and an index matrix to the
HTML and differential reports.  The "fl-build-label-reports" script
will use a default title and sub-title based on the labels but may
specified using the "--title" and "--sub-title" options.  Arbitrary
text or HTML may also be included on stdin or using the "--input"
option::

  $ echo "When deciding which setup to use..." | \
  fl-build-label-reports --x-label=baseline --y-label=foo-setup \ 
  --y-label=bar-setup --title="Setup Comparison"
  --sub-title="Compare setups foo and bar against baseline"
  Creating html report ...done:
  file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html
  Creating html report ...done:
  file:///.../test_ReadOnly-YYYYMMDDThhmmss-foo-label/index.html
  Creating html report ...done:
  file:///.../test_ReadOnly-YYYYMMDDThhmmss-bar-label/index.html
  Creating diff report ...done:
  /.../diff_ReadOnly-YYYYMMDDT_hhmmss-foo-label_vs_hhmmss-baseline/index.html
  Creating diff report ...done:
  /.../diff_ReadOnly-YYYYMMDDT_hhmmss-bar-label_vs_hhmmss-baseline/index.html
  Creating report index ...done:
  file:///.../index.html

Both the "--x-label" and "--y-label" options may be given multiple
times or may use `Python regular expressions
<http://docs.python.org/dev/library/re.html>`_ to create an MxN matrix
of differential reports.  See the "fl-build-label-reports --help"
documentation for more details.

5. Examine the results using the matrix index
---------------------------------------------

Open the index.html generated by the last command to survey the
HTML reports and differential reports.

6. Repeat as changes are made
-----------------------------

As changes are made in your application or setups or to test new
setups, repeat steps 3 and 4.  When step 4 is repeated by running
"fl-build-label-reports" adjusting the  "--x-label" and
"--y-label" options as appropriate, new HTML and differential
reports will be generated as appropriate for the new load test
benchmark results and the matrix index will be updated.

fl-run-bench
============

The scripts that Funkload installs generally require that they be
executed from the directory where the test modules live.  While this
is appropriate for generating test cases with the Funkload recorder,
it's often not the desirable behavior when running load test
benchmarks.  Additionally, the argument handling for the benchmark
runner doesn't allow for specifying which tests to benchmark using the
zope.testing.testrunner semantics, such as specifying modules and
packages with dotted paths, as one is often wont to do when working
with setuptools and eggs.

To accommodate this usage pattern, the collective.funkload package
provides a wrapper around the Funkload benchmark runner that handles
dotted path arguments gracefully.  Specifically, rather than pass
``*.py`` file and TestCase.test_method arguments, the
"fl-bench-runner" provided by collective.funkload supports
zope.testing.testrunner semantics for finding tests with "-s", "-m"
and "-t".

    >>> from collective.funkload import bench
    >>> bench.run(defaults, (
    ...     'test.py -s foo -t test_foo '
    ...     '--cycles 1 --url http://bar.com').split())
    t...
    Benching FooTestCase.test_foo...
    * Server: http://bar.com...
    * Cycles: [1]...
