Benchmarks and Speed
====================

As an XML library, lxml.etree is very fast.  It is also slow.  As with all
software, it depends on what you do with it.  Rest assured that lxml is fast
enough for most applications, so lxml is probably somewhere between 'fast
enough' and 'the best choice' for yours.

This text describes where lxml.etree (lxe) excels, gives hints on some
performance traps and compares the overall performance to the original
ElementTree_ (ET) and cElementTree_ (cET) libraries by Fredrik Lundh.  The
cElementTree library is a fast C-implementation of the original ElementTree.

.. _ElementTree:  http://effbot.org/zone/element-index.htm
.. _cElementTree: http://effbot.org/zone/celementtree.htm

The statements made here are backed by the benchmark script `bench.py`_ that
comes with the lxml source distribution.  The timings cited below compare lxml
1.0 (with libxml2 2.6.24), ElementTree 1.2.6 and cElementTree 1.0.5 under
CPython 2.4.2 on a 1.6GHz AMD64 machine.

.. _`bench.py`:   http://codespeak.net/svn/lxml/branch/lxml-1.0/bench.py

The ``bench.py`` script runs a number of simple tests on the different
libraries, using different XML tree configurations: different tree sizes, with
or without attributes (-/A) and with or without ASCII or unicode text (-/S/U).
In the result extracts cited below, T1 refers to a 3-level tree with many
children at the third level, T2 is swapped around to have many children at the
root element, T3 is a deep tree with few children at each level and T4 is a
small tree, slightly broader than deep.  Most benchmarks run in a loop over
all children of the tree root.

.. contents::
.. 
   1  Bad things first
   2  Parsing and Serialising
   3  The ElementTree API
   4  Tree traversal
   5  XPath


Bad things first
----------------

First thing to say: there *is* an overhead involved in having a C library
mimic the ElementTree API.  As opposed to ElementTree, lxml has to generate
Python objects on the fly when asked for them.  What this means is: the more
of your code runs in Python, the slower your application gets.  Note, however,
that this is true for most performance critical Python applications.


Parsing and Serialising
-----------------------

These are areas where lxml excels.  The reason is that both parts are executed
entirely at the C level, without major interaction with Python code.  The
results are rather impressive.  Compared to cElementTree, lxml is about 20 to
40 times faster on serialisation::

  lxe: tostring_utf16  (SA T2)   30.9846 msec/pass
  cET: tostring_utf16  (SA T2)  715.5002 msec/pass
  ET : tostring_utf16  (SA T2)  758.5271 msec/pass

  lxe: tostring_utf16  (U- T3)    3.0509 msec/pass
  cET: tostring_utf16  (U- T3)   72.4721 msec/pass
  ET : tostring_utf16  (U- T3)   87.0735 msec/pass

  lxe: tostring_utf8   (UA T2)   26.8996 msec/pass
  cET: tostring_utf8   (UA T2)  700.4889 msec/pass
  ET : tostring_utf8   (UA T2)  745.3317 msec/pass

  lxe: tostring_utf8   (S- T3)    2.1876 msec/pass
  cET: tostring_utf8   (S- T3)   71.1290 msec/pass
  ET : tostring_utf8   (S- T3)   87.1525 msec/pass

For parsing, the difference between the libraries is smaller.  The (c)ET
libraries use the expat parser, which is known to be extremely fast::

  lxe: parse_stringIO  (SA T2)  197.7678 msec/pass
  cET: parse_stringIO  (SA T2)   38.9390 msec/pass
  ET : parse_stringIO  (SA T2)  364.3468 msec/pass

  lxe: parse_stringIO  (UA T3)   48.6735 msec/pass
  cET: parse_stringIO  (UA T3)   39.7455 msec/pass
  ET : parse_stringIO  (UA T3)  237.9971 msec/pass

The expat parser allows cET to be up to 80% faster than lxml on plain parser
performance.  The same applies to the ``iterparse()`` function.  However, if
you take a complete serialize-parse cycle, the numbers will look similar to
these::

  lxe: write_utf8_parse_stringIO  (S- T1)  187.0444 msec/pass
  cET: write_utf8_parse_stringIO  (S- T1)  828.4068 msec/pass
  ET : write_utf8_parse_stringIO  (S- T1) 1181.0658 msec/pass

  lxe: write_utf8_parse_stringIO  (UA T2)  213.6599 msec/pass
  cET: write_utf8_parse_stringIO  (UA T2)  927.2374 msec/pass
  ET : write_utf8_parse_stringIO  (UA T2) 1297.9678 msec/pass

For applications that require a high parser throughput and do little
serialization, cET is the best choice.  Also for iterparse applications that
extract small amounts of data from large XML data sets.  If it comes to
round-trip performance, however, lxml tends to be 3-4 times faster in total.


The ElementTree API
-------------------

Since all three libraries implement the same API, their performance is easy to
compare in this area.  A major disadvantage for lxml's performance is the
different tree model that underlies libxml2.  It allows lxml to provide parent
pointers for elements, but also increases the overhead of tree building and
restructuring.  This can be seen from the tree setup times of the benchmark
(given in seconds)::

  lxe:       --     S-     U-     -A     SA     UA
       T1: 0.1360 0.1214 0.1214 0.1217 0.1232 0.1226
       T2: 0.1258 0.1257 0.1250 0.1348 0.1359 0.1358
       T3: 0.0354 0.0282 0.0288 0.0850 0.0860 0.0862
       T4: 0.0006 0.0006 0.0006 0.0019 0.0018 0.0019

  cET:       --     S-     U-     -A     SA     UA
       T1: 0.0417 0.0409 0.0403 0.0410 0.0410 0.0415
       T2: 0.0413 0.0414 0.0413 0.0417 0.0411 0.0417
       T3: 0.0097 0.0100 0.0099 0.0187 0.0142 0.0146
       T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001

  ET :       --     S-     U-     -A     SA     UA
       T1: 0.2189 0.2832 0.2210 0.2646 0.2905 0.2214
       T2: 0.3022 0.2322 0.2868 0.3192 0.2290 0.3075
       T3: 0.0519 0.0553 0.0527 0.0601 0.0572 0.0911
       T4: 0.0009 0.0008 0.0008 0.0008 0.0009 0.0009

While lxml is still faster than ET in most cases (30-60%), cET can be up to
three times faster than lxml here.  One of the reasons is that lxml must
additionally discard the created Python elements after their use, when they
are no longer referenced.  ET and cET represent the tree itself through these
objects, which reduces the overhead in creating them.

The same reason makes operations like ``getchildren()`` more costly in lxml.
Where ET and cET can quickly create a shallow copy of their list of children,
lxml has to create a Python object for each child and collect them in a list::

  lxe: root_getchildren      (-- T2   )    6.3981 msec/pass
  cET: root_getchildren      (-- T2   )    0.0651 msec/pass
  ET : root_getchildren      (-- T2   )    0.0224 msec/pass

As opposed to ET, libxml2 has a notion of documents that each element must be
in.  This results in a major performance difference for creating independent
Elements that end up in independently created documents::

  lxe: create_elements       (-- T2   )   22.0083 msec/pass
  cET: create_elements       (-- T2   )    0.3920 msec/pass
  ET : create_elements       (-- T2   )    3.0865 msec/pass

Therefore, it is always preferable to create Elements for the document they
are supposed to end up in, either as SubElements of an Element or using the
explicit ``Element.makeelement()`` call::

  lxe: makeelement           (-- T2   )    4.2658 msec/pass
  cET: makeelement           (-- T2   )    0.5658 msec/pass
  ET : makeelement           (-- T2   )    3.7136 msec/pass

  lxe: create_subelements    (-- T2   )    3.7640 msec/pass
  cET: create_subelements    (-- T2   )    0.5332 msec/pass
  ET : create_subelements    (-- T2   )    6.5937 msec/pass

So, if the main performance bottleneck of an application is creating large XML
trees in memory through calls to Element and SubElement, cET is the best
choice.  Note, however, that the serialisation performance may even out this
advantage, especially for smaller trees and trees with many attributes.

A critical action for lxml is moving elements between document contexts.  It
requires lxml to do recursive adaptations throughout the moved tree structure.

The following benchmark appends all root children of the second tree to the
root of the first tree::

  lxe: append_from_document  (-- T1,T2)   11.7905 msec/pass
  cET: append_from_document  (-- T1,T2)    0.4673 msec/pass
  ET : append_from_document  (-- T1,T2)    2.0460 msec/pass

  lxe: append_from_document  (-- T3,T4)    0.1582 msec/pass
  cET: append_from_document  (-- T3,T4)    0.0224 msec/pass
  ET : append_from_document  (-- T3,T4)    0.1618 msec/pass

Although these are fairly small numbers compared to parsing, this easily shows
the different performance classes for lxml and (c)ET.  Where the latter do not
have to care about parent pointers and tree structures, lxml has to deep
traverse the appended tree.  The performance difference therefore increases
with the size of the tree that is moved.

This difference is not always as visible, but applies to most parts of the
API, like inserting newly created elements::

  lxe: insert_from_document      (-- T1,T2)   16.2342 msec/pass
  cET: insert_from_document      (-- T1,T2)    1.1786 msec/pass
  ET : insert_from_document      (-- T1,T2)    3.6107 msec/pass

Or replacing the child slice by a new element::

  lxe: replace_children_element  (-- T1   )    9.1834 msec/pass
  cET: replace_children_element  (-- T1   )    0.9731 msec/pass
  ET : replace_children_element  (-- T1   )   14.8213 msec/pass

You should keep this difference in mind when you merge very large trees.  On
the other hand, deep copying a tree is fast in lxml::

  lxe: deepcopy                  (-- T1   )   24.7359 msec/pass
  cET: deepcopy                  (-- T1   )  450.5479 msec/pass
  ET : deepcopy                  (-- T1   )  717.8308 msec/pass

  lxe: deepcopy                  (-- T3   )    2.1182 msec/pass
  cET: deepcopy                  (-- T3   )  107.2124 msec/pass
  ET : deepcopy                  (-- T3   )  173.9782 msec/pass

So, for example, if you often need to create independent subtrees from a large
tree that you have parsed in, lxml is by far the best choice here.


Tree traversal
--------------

Another area where lxml is very fast is iteration for tree traversal.  If your
algorithms can benefit from step-by-step traversal of the XML tree and
especially if few elements are of interest or the element tag name is known,
lxml is a good choice::

  lxe: getiterator_all      (-- T2   )   22.5847 msec/pass
  cET: getiterator_all      (-- T2   )   36.8212 msec/pass
  ET : getiterator_all      (-- T2   )   46.2846 msec/pass

  lxe: getiterator_islice   (-- T2   )    2.0421 msec/pass
  cET: getiterator_islice   (-- T2   )    0.3343 msec/pass
  ET : getiterator_islice   (-- T2   )   44.5898 msec/pass

  lxe: getiterator_tag      (-- T2   )    1.9593 msec/pass
  cET: getiterator_tag      (-- T2   )   11.7767 msec/pass
  ET : getiterator_tag      (-- T2   )   37.5661 msec/pass

  lxe: getiterator_tag_all  (-- T2   )    4.5667 msec/pass
  cET: getiterator_tag_all  (-- T2   )   33.5681 msec/pass
  ET : getiterator_tag_all  (-- T2   )   37.6200 msec/pass

This similarly shows in ``Element.findall()``::

  lxe: findall              (-- T2   )   26.9907 msec/pass
  cET: findall              (-- T2   )   39.1728 msec/pass
  ET : findall              (-- T2   )   50.9692 msec/pass

  lxe: findall              (-- T3   )    3.6452 msec/pass
  cET: findall              (-- T3   )   12.0210 msec/pass
  ET : findall              (-- T3   )   11.2570 msec/pass

  lxe: findall_tag          (-- T2   )    4.6065 msec/pass
  cET: findall_tag          (-- T2   )   34.0267 msec/pass
  ET : findall_tag          (-- T2   )   36.7813 msec/pass

  lxe: findall_tag          (-- T3   )    0.5884 msec/pass
  cET: findall_tag          (-- T3   )    7.6307 msec/pass
  ET : findall_tag          (-- T3   )    9.2943 msec/pass

Note that all three libraries currently use the same Python implementation for
``findall()``, except for their native tree iterator.


XPath
-----

This part of lxml does not have an equivalent in ElementTree.  However, lxml
provides more than one way of accessing it and you should take care which part
of the lxml API you use.  The most straight forward way is to call the
``xpath()`` method on an Element or ElementTree::

  lxe: xpath_method         (-- T1)    9.9304 msec/pass
  lxe: xpath_method         (-- T2)   29.3595 msec/pass
  lxe: xpath_method         (-- T3)    0.2791 msec/pass
  lxe: xpath_method         (-- T4)    0.9906 msec/pass

This is well suited for testing and when the XPath expressions are as diverse
as the trees they are called on.  However, if you have a single XPath
expression that you want to apply to a larger number of different elements,
the ``XPath`` class is the most efficient way to do it::

  lxe: xpath_class          (-- T1)    4.7921 msec/pass
  lxe: xpath_class          (-- T2)    9.6187 msec/pass
  lxe: xpath_class          (-- T3)    0.2215 msec/pass
  lxe: xpath_class          (-- T4)    0.2697 msec/pass

Note that this still allows you to use variables in the expression, so you can
parse it once and then adapt it through variables at call time.  In other
cases, where you have a fixed Element or ElementTree and want to run different
expressions on it, you should consider the ``XPathEvaluator``::

  lxe: xpath_element        (-- T1)    5.3826 msec/pass
  lxe: xpath_element        (-- T2)   11.3929 msec/pass
  lxe: xpath_element        (-- T3)    0.2514 msec/pass
  lxe: xpath_element        (-- T4)    0.3038 msec/pass

While it looks slightly slower, creating an XPath object for each of the
expressions generates a much higher overhead here::

  lxe: xpath_class_repeat   (-- T1)    6.8099 msec/pass
  lxe: xpath_class_repeat   (-- T2)   26.7462 msec/pass
  lxe: xpath_class_repeat   (-- T3)    0.3126 msec/pass
  lxe: xpath_class_repeat   (-- T4)    1.1111 msec/pass


lxml.objectify
--------------

Objectify is a data-binding API for XML based on lxml.etree, that was added in
version 1.1.  It uses standard Python attribute access to traverse the XML
tree.  It also features ObjectPath, a fast path language based on the same
meme.

Just like lxml.etree, lxml.objectify creates Python representations of
elements on the fly.  To save memory, the normal Python garbage collection
mechanisms will discard them when their last reference is gone.  In cases
where deeply nested elements are frequently accessed through the objectify
API, the create-discard cycles can become a bottleneck, as elements have to be
instantiated over and over again.

ObjectPath can be used to speed up the access to elements that are deep in the
tree.  It avoids step-by-step Python element instantiations along the path,
which can substantially improve the access time::

  lxe: attribute                  (--T T1)   14.8621 msec/pass
  lxe: attribute                  (--T T2)   61.8820 msec/pass
  lxe: attribute                  (--T T4)   14.9317 msec/pass

  lxe: objectpath                 (--T T1)   13.7311 msec/pass
  lxe: objectpath                 (--T T2)   58.5930 msec/pass
  lxe: objectpath                 (--T T4)    8.0961 msec/pass

  lxe: attributes_deep            (--T T1)   81.4488 msec/pass
  lxe: attributes_deep            (--T T2)   77.0266 msec/pass
  lxe: attributes_deep            (--T T4)   27.1226 msec/pass

  lxe: objectpath_deep            (--T T1)   63.1915 msec/pass
  lxe: objectpath_deep            (--T T2)   65.2469 msec/pass
  lxe: objectpath_deep            (--T T4)   11.0138 msec/pass

Note, however, that parsing ObjectPath expressions is not for free either, so
this is most effective for frequently accessing the same element.

A way to improve the normal attribute access time is static instantiation of
the Python objects, thus trading memory for speed.  Just create a cache
dictionary and run::

    cache[root] = list(root.getiterator())

after parsing and::

    del cache[root]

when you are done with the tree.  This will keep the Python element
representations of all elements alive and thus avoid the overhead of repeated
Python object creation.  You can also consider using filters or generator
expressions to be more selective.  By choosing the right trees (or even
subtrees and elements) to cache, you can trade memory usage against access
speed::

  lxe: attribute_cached           (--T T1)   10.8343 msec/pass
  lxe: attribute_cached           (--T T2)   55.5890 msec/pass
  lxe: attribute_cached           (--T T4)   10.9514 msec/pass

  lxe: attributes_deep_cached     (--T T1)   63.7080 msec/pass
  lxe: attributes_deep_cached     (--T T2)   65.6838 msec/pass
  lxe: attributes_deep_cached     (--T T4)   15.4514 msec/pass

Things to note: you cannot currently use ``weakref.WeakKeyDictionary`` objects
for this as lxml's element objects do not support weak references (which are
costly in terms of memory).  Also note that new element objects that you add
to these trees will not turn up in the cache automatically and will therefore
still be garbage collected when all their Python references are gone, so this
is most effective for largely immutable trees.  You should consider using a
set instead of a list in this case and add new elements by hand.

Here are some more things to try if optimisation is required:

* A lot of time is usually spent in tree traversal to find the addressed
  elements in the tree.  If you often work in subtrees, assign the parent of
  the subtree to a variable or pass it into functions instead of starting at
  the root.  This allows accessing its descendents more directly.

* Try assigning data values directly to attributes instead of passing them
  through DataElement.

* If you use custom data types that are costly to parse, try running
  ``objectify.annotate()`` over read-only trees to speed up the attribute type
  inference on read access.

Note that none of these measures is guaranteed to speed up your application.
As usual, you should prefer readable code over premature optimisations and
profile your expected use cases before bothering to apply optimisations at
random.
