# Example of nl_pipeline use

Before starting the pipeline review the contents of ./etc/nl_*.conf
setting the config_base and selecting (via commenting in/out) the db
config to use.

Assumed is that you have sqlite3 installed and optionally mysql
installed and running on localhost.  See ./my.cnf for details of
mysql db/user/password set up.

# Clean up all files and create empty DBs (both sqlite and mysql).
# Note that this also removes any NL state and pid files so this
# starts you off from a completely clean state - which may not be what
# you want if your testing re-starting, etc.
$ ./clean

# Start generating logs (this currently must be done first or the
# parser won't see the input file - see issue #301)
$ ./genlog &

# Start the pipeline
$ ./pipeline &

# Watch the files and db grow
$ ./watch

# Stop genlog a/o pipeline by foregrounding them and ^C.  Confirm
# what's running using ps or your shell's jobs command.  Possible
# examples:
$ ps augxw | egrep 'genlog|pipline|parser|loader'
$ jobs -l

Restarting either should resume the flow of events.

## Troubleshooting ##
- Scour the .conf files for typos as errors in them are not reported.
  (Is config_base set properly in nl_shared.conf, pointing to where you
  checked all this out?)

- Use the watch script and/or ps (ps -p `cat ./var/run/nl_*.pid`) to
  see if each component is working a/o running.

- Look at the logs for each component *.log files.

- Look at the pipeline.log or pass the ./pipeline script a '-n' when
  starting, to see how it is starting the parser and loader, then
  start them individually without a '-d' arg and with a '-v' (most
  likely i.e. for the loader: 'nl_loader -c ./etc/nl_loader.conf -v').

- Increase the logging level by changing the 'level=INFO' lines in
  ./etc/nl_logging.conf then restart and look in the logs under
  ./var/log/
