Adding and Removing Nodes from StarCluster
==========================================
StarCluster has support for manually shrinking and expanding the size of your
cluster based on your resource needs. For example, you might start out with
10-nodes and realize that you only really need 5 or the reverse case where you
start 5 nodes and find out you need 10. In these cases you can use
StarCluster's *addnode* and *removenode* commands to scale the size of your
cluster to your needs.

.. note::
    The examples below assume we have a 1-node cluster running called *mycluster*.

Adding Nodes
------------
Adding nodes is done using the *addnode* command. This command takes a *cluster
tag* as an argument and will automatically add a new node to the cluster:

.. code-block:: none

        $ starcluster addnode mycluster
        StarCluster - (http://web.mit.edu/starcluster) (v. 0.9999)
        Software Tools for Academics and Researchers (STAR)
        Please submit bug reports to starcluster@mit.edu

        >>> Launching node(s): node001
        >>> Waiting for node(s) to come up... (updating every 30s)
        >>> Waiting for open spot requests to become active...
        2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Waiting for all nodes to be in a 'running' state...
        2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Waiting for SSH to come up on all nodes...
        2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Configuring hostnames...
        1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Configuring /etc/hosts on each node
        2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Configuring NFS...
        >>> Mounting shares for node node001
        1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Configuring scratch space for user: myuser
        1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
        >>> Configuring passwordless ssh for root
        >>> Using existing key: /root/.ssh/id_rsa
        >>> Configuring passwordless ssh for myuser
        >>> Using existing key: /home/myuser/.ssh/id_rsa
        >>> Adding node001 to SGE

If the --alias option is not specified StarCluster will apply the next
available *nodeXXX* alias based on the current size of the cluster. In the
above example only a *master* node existed so StarCluster automatically added
the second node as *node001*. Similarly we had a *master* and *node001* then
running addnode would name the new node *node002*.

You can also manually specify an alias for the new node using the --alias (-a)
option:

.. code-block:: none

        $ starcluster addnode -a mynewnode mycluster

Running listclusters will then show your alias for the node in its output:

.. code-block:: none

        $ starcluster listclusters mycluster

And you can login directly to the new node by alias:

.. code-block:: none

        $ starcluster sshnode mycluster mynewnode

Removing Nodes
--------------
StarCluster also has the ability to remove an existing cluster node's via the
*removenode* command. This command takes at least two arguments: the *cluster
tag* representing the cluster you want to remove nodes from and a node *alias*:

.. code-block:: none

        % starcluster removenode mycluster node001
        StarCluster - (http://web.mit.edu/starcluster) (v. 0.9999)
        Software Tools for Academics and Researchers (STAR)
        Please submit bug reports to starcluster@mit.edu

        >>> Removing node node001 (i-8bec7ce5)...
        >>> Removing node001 from SGE
        >>> Removing node001 from known_hosts files
        >>> Removing node001 from /etc/hosts
        >>> Removing node001 from NFS
        >>> Cancelling spot request sir-3567ba14
        >>> Terminating node: node001 (i-8bec7ce5)

The above command takes care to properly remove the node from the cluster by
removing the node from the Sun Grid Engine queuing system, each node's ssh
known_hosts file, /etc/hosts file, and from all NFS shares. Afterwards the node
is terminated. If the node is a spot instance, as it is in the above example,
the spot instance request will also be cancelled.
