extcmd - subprocess with advanced output processing
===================================================

Unlike subprocess, which just gives you a lump of output at the end, extcmd
allows you to get callbacks (via a delegate class) on all IO.

Delegates
=========

Each delegate has four methods (on_begin, on_line, on_end, on_interrupt) but it
is possible to simply specify the ones you are interested in and extcmd will do
the right thing automatically. There is an associated interface
extcmd.IDelegate, if you subclass that class in your delegates then extcmd will
trust you do everything properly. If you pass any other object as delegate then
extcmd will wrap your object in extcmd.SafeDelegate that which provides default
implementations of all the required methods.

To make some common cases easier to work with, extcmd comes with a number of
utility callback delegates: decoding and encoding from bytes to Unicode,
transforming the data, redirecting the output to other streams and even forking
the output so that you can, for example, log and display data at the same time.


Everything is encapsulated in a single module, extcmd::

    >>> import extcmd

Basic IO
========

Since IO is oriented around bytes, the first example will actually focus on
getting the basic IO work-flow right: convert bytes and Unicode at application
boundaries. Sadly Popen does not support that, let's build one that does::

    >>> unicode_popen = extcmd.ExternalCommandWithDelegate(
    ...     extcmd.Decode(
    ...         extcmd.EncodeInPython2(
    ...             extcmd.Redirect())))

So this may be somewhat hard to read but the basic looks like this:

We instantiate the ExtrnalCommandWithDelegate, this is like subprocess.Popen
metaclass as the return value is something we can use to actually run call() or
check_call(). The only argument to that is a single delegate object. We'll use
three of the delegates provided by extcmd here.

The Decode delegate simply decodes all IO from a specified encoding (uses UTF-8
by default). The Encode delegate does the reverse (which is a real no-op but
we'll grow this example in a second and it will be useful to have Unicode
strings then). Lastly the Redirect delegate sends all of stdout/stderr back to
real stdout/stderr (it is also flexible so you can redirect to file or any
other stream but we're using the defaults again).

All those delegates are connected so one delegate gives the output to another
delegate. In practice it looks like this::

    (real data from the process) -> Decode -> Encode -> Redirect

Let's see how that works now:

    >>> returncode = unicode_popen.call(["echo", "zażółć gęsią jaźń"])
    zażółć gęsią jaźń

Well that was boring, but the point here is that id did _not_ crash on any
UnicodeDecodeErrors and I actually used some non-ASCII characters.

One thing worth pointing out is that unlike in subprocess, each call() returns
the process exit code::

    >>> returncode
    0

Using Transform delegate
========================

So now we have the basics. Let's explore further. The Transform delegate
allows one to call a user specified function on each line of the output.

As before we'll build a list of participating delegate objects, we'll start
with the Decode delegate, then the Transform delegate, the Encode and lastly,
Redirect. This will look like this:

    (output from process) -> Decode -> Transform -> Encode -> Redirect

For clarity we'll define the transformation first::

    >>> def transform_fn(stream_name, line):
    ...     return "{0}: {1}".format(stream_name, line)

Then build the actual stack of delegates::

    >>> delegate = extcmd.Decode(
    ...     extcmd.Transform(transform_fn,
    ...         extcmd.EncodeInPython2(
    ...             extcmd.Redirect())))
    >>> transform_popen = extcmd.ExternalCommandWithDelegate(delegate)
    >>> returncode = transform_popen.call(["echo", "hello"])
    stdout: hello

Simple Unicode-aware sed(1)
===========================

Let's build a simple in sed(1) like program. We'll use the 're' module to
actually transform text. Let's import it now::

    >>> import re

Let's define another transformation function:

    >>> def transform_fn(stream_name, line):
    ...     return re.sub("Hello", "Goodbye", line)

And plug it into the stack we've used before:

    >>> delegate = extcmd.Decode(
    ...     extcmd.Transform(transform_fn,
    ...         extcmd.EncodeInPython2(
    ...             extcmd.Redirect())))
    >>> sed_popen = extcmd.ExternalCommandWithDelegate(delegate)
    >>> returncode = sed_popen.call(["echo", "Hello World"])
    Goodbye World

Simple tee(1)
=============

Ok, so one more example, this time tee(1)-like program. This pattern can be
used to build various kinds of programs where many consumers get to see the
data that was read.

We'll use one more delegate this time, the extcmd.Chain (which is, from
retrospective, rather unfortunately named, as it's really a "fork" while
regular delegates build a chain themselves).

So this example will save everything written to stdout to a log file, while
still displaying it back to the user::

    >>> delegate = extcmd.Chain([
    ...     extcmd.Decode(
    ...         extcmd.EncodeInPython2(
    ...             extcmd.Redirect())),
    ...     extcmd.Redirect(
    ...         stdout=open("stdout.log", "wb"),
    ...         close_stdout_on_end=True)
    ... ])
    >>> tee_popen = extcmd.ExternalCommandWithDelegate(delegate)
    >>> returncode = tee_popen.call(['echo', "Hello Tee!"])
    Hello Tee!

So this example is actually more interesting, unlike before we don't decode
_all_ data, only the data that is displayed, the stdout.log file will contain a
verbatim copy of all the bytes that were produced by the called process::

    >>> import os
    >>> assert os.path.exists("stdout.log")
    >>> with open("stdout.log", "rt") as stream:
    ...     stream.read()
    'Hello Tee!\n'
    >>> os.remove("stdout.log")

Misc stuff
==========

Apart from ExtrnalCommandWithDelegate there is a base class called
ExternalCommand that simply helps if you want to subclass and override the
call() method.

There is also the check_call() method that behaves exactly as in the subprocess
module, by raising subprocess.CalledProcessError exception on a non-zero return
code

    >>> extcmd.ExternalCommand().check_call(['false'])
    ... # doctest: +IGNORE_EXCEPTION_DETAIL
    Traceback (most recent call last):
    ...
    CalledProcessError: Command '['false']' returned non-zero exit status 1

If you don't use check_call You can also look at the return code that is
returned from each call(). The returncode is also passed to each delegate that
supports the on_end() method::


    >>> import sys
    >>> class ReturnCode(extcmd.DelegateBase):
    ...     def on_end(self, returncode):
    ...         sys.stdout.write("Return code is %s\n" % returncode)
    >>> returncode = extcmd.ExternalCommandWithDelegate(ReturnCode()).call(['false'])
    Return code is 1
    >>> returncode
    1

Each started program is also passed to the on_start() method::

    >>> import sys
    >>> class VerboseStart(extcmd.DelegateBase):
    ...     def on_begin(self, args, kwargs):
    ...         sys.stdout.write("Starting %r %r\n" % (args, kwargs))
    >>> returncode = extcmd.ExternalCommandWithDelegate(VerboseStart()).call(['true'])
    Starting (['true'],) {}
