.. _examples:

========
Examples
========

You will find here a set of examples. The objective is to illustrate how to 
build simple to more complex workflows and submit them to soma-workflow. 

The content of the jobs is not relevant here: 
  * The "sleep" command was use as an example to stand for any job.

  * The "cp" command was use as an example of a job reading a file and writing another file.  


.. contents:: Examples
   :local:

  
Running example in light mode
=============================

In the light mode the disconnection feature is not available.
You must save your workflow to a file at the end of the script:

::

  from soma.workflow.client import Helper

  # workflow creation as described in the examples

  Helper.serialize("/tmp/workflow_example", workflow)
  

Then use the GUI to open the file */tmp/workflow_example* and use the submit 
button to submit the workflow.
In the light mode you must wait the end of the workflow to close the
application.


Simple workflows
================


Basic example
-------------

.. figure:: images/simple_example.png
  :scale: 60

  Simple example: 4 jobs with dependencies


::

  from soma.workflow.client import Job, Workflow, WorkflowController


  # create the workflow:

  job_1 = Job(command=["sleep", "60"], name="job 1")
  job_2 = Job(command=["sleep", "60"], name="job 2")
  job_3 = Job(command=["sleep", "60"], name="job 3")
  job_4 = Job(command=["sleep", "60"], name="job 4")

  jobs = [job_1, job_2, job_3, job_4]
  dependencies = [(job_1, job_2), 
                  (job_1, job_3), 
                  (job_2, job_4), 
                  (job_3, job_4)]

  workflow = Workflow(jobs=jobs, 
                      dependencies=dependencies)


  # submit the workflow:

  controller = WorkflowController("Titan", login, password)

  controller.submit_workflow(workflow=workflow,
                             name="simple example")

Run the Python script and monitor the execution of the workflow using 
the GUI (soma_workflow_gui command):

.. image:: images/simple_example_gui.png
  :scale: 60

You can close and open the GUI at any time. It will not interrupt the workflow
execution.


Worklfow with Group
-------------------

Creating groups can be useful to organise jobs within a workflow for displaying 
purpose. Groups do not have any impact on the execution of the workflow.  

.. figure:: images/simple_example_group.png
  :scale: 60

  Simple example with a group: The 30 jobs are in the group "my 30 jobs".

::

  from soma.workflow.client import Job, Workflow, Group, WorkflowController


  # create the workflow:
  jobs = []
  dependencies = []
  group_elements = []

  first_job = Job(command=["sleep", "10"], name="first job")
  last_job = Job(command=["sleep", "10"], name="last job")

  jobs.append(first_job)
  jobs.append(last_job)

  for i in range(0, 30):
    job = Job(command=["sleep", "60"], name="job " + repr(i))

    jobs.append(job)

    dependencies.append((first_job, job))
    dependencies.append((job, last_job))

    group_elements.append(job)
    
  
  thirty_jobs_group = Group(elements=group_elements,
                            name="my 30 jobs")

  workflow = Workflow(jobs=jobs,
                      dependencies= dependencies,
                      root_group=[first_job, thirty_jobs_group, last_job])


  # submit the workflow: 
  controller = WorkflowController("Titan", login, password)

  controller.submit_workflow(workflow=workflow,
                             name="Simple workflow with group")


Run the Python script and monitor the execution of the workflow using 
the GUI (soma_workflow_gui command):

.. image:: images/simple_example_group_gui.png
  :scale: 60
  
The 30 jobs within the group can now be extanded (visible) or collapsed (hided).


.. _file_transfer_examples:

Workflows with FileTransfer
===========================

These examples show example of workflows with file transfers (see :ref:`file-transfers-concept`). 
An easy way to deal with file transfer is to transfer a working directory and
work with relative path (see :ref:`working_directory_transfer_example`). 


Basic file transfer
-------------------

To run this example, create first the file "/tmp/soma_workflow_examples/myfile" 
with any content. It will stand for an input file to transfer. 

::

  from soma.workflow.client import Job, Workflow, WorkflowController, FileTransfer

  # FileTransfer creation for input files
  myfile = FileTransfer(is_input=True,
                      client_path="/tmp/soma_workflow_examples/myfile",
                      name="myfile")

  # FileTransfer creation for output files
  copy_of_myfile = FileTransfer(is_input=False,
                                client_path="/tmp/soma_workflow_examples/copy_of_myfile",
                                name="copy of my file")

  # Job and Workflow 
  copy_job = Job(command=["cp", myfile, copy_of_myfile], 
                name="copy",
                referenced_input_files=[myfile],
                referenced_output_files=[copy_of_myfile])

  workflow = Workflow(jobs=[copy_job], 
                      dependencies=[])



  # submit the workflow
  controller = WorkflowController("Titan", login, password)

  controller.submit_workflow(workflow=workflow,
                            name="simple transfer")


Run the Python script and open the GUI:

.. image:: images/file_transfer_example_gui.png
  :scale: 60

Transfer the input files, clicking the "Transfer Input Files" button:

.. image:: images/file_transfer_example_gui_1.png
  :scale: 80

The job will start as soon as the transfer ended:

.. image:: images/file_transfer_example_gui_2.png
  :scale: 80

When the workflow is done, transfer the output files clicking the 
"Transfer Output Files" button:

.. image:: images/file_transfer_example_gui_3.png
  :scale: 80

The progression and the end of the transfer are displayed in the GUI.

.. image:: images/file_transfer_example_gui_4.png
  :scale: 80

.. seealso:: :ref:`file-transfer-status`


.. _working_directory_transfer_example:

Working directory transfer
--------------------------

An easy way to work with file transfers is to transfer a working directory 
containing all the files you need and then use only relative file paths in your 
jobs.

In this example, the working directory contains 2 files: myfile1 and myfile2.
The workflow contains 2 jobs which will modify the working directory.

::

  from soma.workflow.client import Job, Workflow, WorkflowController, FileTransfer

  # Creation of the FileTransfer object to transfer the working directory
  my_working_directory = FileTransfer(is_input=True,
                                      client_path="/tmp/my_working_directory",
                                      name="working directory")

  # Jobs and Workflow
  # Do not forget to set the working_directory argument when creating the jobs
  job1 = Job(command=["cp", "myfile1" , "copy_of_myfile1"], 
             name="job1",
             referenced_input_files=[my_working_directory],
             referenced_output_files=[my_working_directory],
             working_directory=my_working_directory)

  job2 = Job(command=["cp", "myfile2" , "copy_of_myfile2"], 
             name="job2",
             referenced_input_files=[my_working_directory],
             referenced_output_files=[my_working_directory],
             working_directory=my_working_directory)

  workflow = Workflow(jobs=[job1, job2], 
                      dependencies=[])


  # submit the workflow
  controller = WorkflowController("Titan", login, password)

  controller.submit_workflow(workflow=workflow,
                             name="working directory transfer example")

Run the Python script and monitor the execution of the workflow using 
the GUI (soma_workflow_gui command):

.. image:: images/wd_transfer_example_gui.png
  :scale: 80

Transfer the working directory to the computing resource clicking the "Transfer 
Input Files" button and wait for the jobs to execute: 

.. image:: images/wd_transfer_example_gui_1.png
  :scale: 80

Transfer the working directory back the "Transfer Output Files" button: 

.. image:: images/wd_transfer_example_gui_2.png
  :scale: 80

The working directory on your machine should contain the files myfile1, myfile2,
copy_of_myfile1 and copy_of_myfile2 now.


Workflows with SharedResourcePath
=================================

A copy of the data you want to process can already exist on the computing resource
file system. In this case, you can use SharedResourcePath objects to build your 
workflows (see :ref:`shared-resource-path-concept`).

To run this example:

1. Create the directory "/tmp/soma_workflow_examples" and the file "/tmp/soma_workflow_examples/myfile" with any content you like. This directory will stand for the data you want to process.

2. Create the same data at any location on the computing resource file system, for example "$HOME/soma_workflow_examples/myfile". It will stand for the copy of the data on the computing resource.

3. On the computing resource file system, create a translation file at the location "$HOME/translation_example" for example. It will contain, the correspondences between identifiers and path: 

  ::

    my_example_dir /home/toto/soma_workflow_examples
  
4. Still on the computing resource side, register the translation path in the configuration file under the namespace "MyApp" (see :ref:`conf_server_option`):

  ::

    PATH_TRANSLATION_FILES = MyApp{/home/toto/translation_example}

Run the script to create and submit the workflow:

:: 

  from soma.workflow.client import Job, Workflow, WorkflowController, SharedResourcePath, FileTransfer

  # SharedResourcePath creation for the input file.
  # The input file is read direclty in the data directory located on the
  # the computing resource side. 
  myfile = SharedResourcePath(relative_path="myfile",
                              namespace="MyApp",
                              uuid="my_example_dir")

  # FileTransfer creation for the output file.
  # That way the output file will not be written in the data directory 
  # located on the computing resource file system.
  copy_of_myfile =  FileTransfer(is_input=False,
                                 client_path="/tmp/soma_workflow_examples/copy_of_myfile",
                                 name="copy of my file")
  
  # Job and Workflow creation 
  copy_job = Job(command=["cp", myfile, copy_of_myfile], 
                name="copy",
                referenced_input_files=[],
                referenced_output_files=[copy_of_myfile])

  workflow = Workflow(jobs=[copy_job], 
                      dependencies=[])

  # workflow submission
  controller = WorkflowController("DSV_cluster", login, password)

  controller.submit_workflow(workflow=workflow,
                             name="shared resource path example")



Open the GUI to monitor the workflow. Once the execution is done, transfer back 
the output files.

The data directory directory on your machine "/tmp/soma_workflow_examples/" should contain the 
file copy_of_myfile now.




