Simple 'Hello, World" example of the NL Pipeline: nl_pipeline
configuration and use.

Quick Start:
- Edit ./etc/nl_shared.conf and set config_base to the dir you've checked
  this out into.
- Run: './clean' to set-up the sqlite DB.
- Run: './genlog & ./pipeline &' to start generating sample logs and the pipeline
- Run: './watch' repeatedly to watch events flow through the pipeline
Then:
- Foreground and kill the genlog a/o pipeline processes, then restart
  them, watching in between monitoring the flow of data through the pipeline.
- With genlog and pipeline killed, run './clean' again to remove all
  intermediary files to enable starting a completely new pipeline of
  data.


Details:
Before starting the pipeline review the contents of ./etc/nl_*.conf
specifically setting the config_base.  Feel free to change the settings
of other values there to suit your situation.  Assumed is that you
have sqlite3 installed.

To start from a completely clean starting point, create an empty DB
Note that this also removes any NL state and pid files so this starts
you off from a completely clean state - which may not be what you want
if your testing re-starting, etc.
$ ./clean

Start generating logs
$ ./genlog &

Start the pipeline.  (Note that this starts nl_pipeline in the
foreground but nl_parser and nl_loader are started as backgrounded
child processes.)
$ ./pipeline &

Watch the files and db grow
$ ./watch

Stop genlog a/o pipeline by foregrounding them and ^C.  Confirm what's
running using ps or your shell's jobs command.  Possible examples:
$ ps augxw | egrep 'genlog|pipline|parser|loader'
$ ps -p `cat ./var/run/nl_*.pid`
$ jobs -l

Restarting either should resume the flow of events.

Troubleshooting:
- Scour the .conf files for typos as errors in them are not reported.
  (Is config_base set properly in nl_shared.conf, pointing to where you
  checked all this out?)

- Use the watch script and/or ps (ps -p `cat ./var/run/nl_*.pid`) to
  see if each component is working a/o running.

- Look at the logs and/or state files for each component:
  ./var/log/*.log and ./var/run/*.state files.

- Look at the pipeline.log or pass the ./pipeline script a '-n' when
  starting, to see how it is starting the parser and loader, then
  start them individually without a '-d' arg and with a '-v' (most
  likely i.e. for the loader: 'nl_loader -c ./etc/nl_loader.conf -v').

- Increase the logging level by changing the 'level=INFO' lines in
  ./etc/nl_logging.conf then restart and look in the logs under
  ./var/log/
