Introduction
============

This recipe adds additional filestorage and zodb_db stanzas to the zope.conf
and zeo.conf files generated by the plone.recipe.zope2instance and
plone.recipe.[zope2]zeoserver recipes.  It also creates the directories in which
the extra filestorage data files will be created.  This makes it easy to add
additional filestorages and mountpoints to a Zope 2 instance.


Supported options
=================

parts
    A list of filestorage sub-parts to be generated, one per line.  (This is
    different from buildout parts.)
zeo
    The name of a plone.recipe.zope2zeoserver or plone.recipe.zeoserver part to
    which we want to add the extra filestorage.  Defaults to the first such part
    in the buildout, if any.
zopes
    A list of names of plone.recipe.zope2instance parts to which we want to add
    the extra filestorage.  Defaults to all plone.recipe.zope2instance parts
    connected to the associated zeoserver part, if any, or all
    plone.recipe.zope2instance parts, if no ZEO is found.
    
The following options affect the generated zope.conf and zeo.conf.  Each may be
specified for all filestorage subparts in the collective.recipe.filestorage
buildout part, or for one particular filestorage subpart by placing the option
in a new buildout part called 'filestorage_subpart', where subpart is the name
of the subpart listed in the 'parts' option of this recipe.  The name of the
subpart may be interpolated by using '%(fs_part_name)s' in the option.

location
    The location of the Data.fs file, relative to the buildout root directory.
    Defaults to var/filestorage/%(fs_part_name)s/Data.fs
zodb-name
    The name of the ZODB.  Defaults to '%(fs_part_name)s'.
zodb-cache-size
    Set the ZODB cache size, i.e. the number of objects which the ZODB cache
    will try to hold.  Inherits from the associated zope part.  Defaults to
    5000.
zodb-mountpoint
    Set the path to the mountpoint.  Defaults to '/%(fs_part_name)s'.
zodb-container-class
    Set the class of the object being mounted.  Defaults to not being set.
zeo-address
    Set the port of the associated ZEO server.  Inherits from the associated
    Zope and ZEO parts.  Defaults to 8100.
zeo-client-cache-size
    Set the size of the ZEO client cache. Inherits from the associated Zope
    part.  Defaults to '30MB'.
zeo-storage
    Set the id of the ZEO storage. Defaults to '%(fs_part_name)s'.
zeo-client-name
    Set the name of the ZEO client. Defaults to '%(fs_part_name)s_zeostorage'.
zeo-client-client
    Set the persistent cache name that is used to construct the cache
    filenames. Persistent cache files are disabled by default.
blob-storage
    Set the directory to be used to store blobs for a standalone Zope instance
    or a ZEO server. Optional. Required if you're going to store blobs, though.
    Recommended value: var/blobstorage-%(fs_part_name)s
zeo-blob-storage
    Set the directory to be used to store blobs for a ZEO client. Defaults to
    using the same value as `blob-storage`.
zeo-shared-blob-dir
    Boolean that should be 'on' if the blob dir is being shared by the ZEO
    server and client. Defaults to 'on'.


Example usage
=============

Let's create and run a minimal buildout that adds an extra filestorage::

   >>> write('buildout.cfg',
   ... '''
   ... [buildout]
   ... extends = base.cfg
   ... parts =
   ...     filestorage
   ...     instance
   ...
   ... [instance]
   ... recipe = plone.recipe.zope2instance
   ... zope2-location = %(zope2_location)s
   ... user = me
   ...
   ... [filestorage]
   ... recipe = collective.recipe.filestorage
   ... parts =
   ...     my-fs
   ... ''' % globals())
   >>> print system(join('bin', 'buildout') + ' -q')

Our zope.conf should get the extra filestorage stanza automatically injected into it::

   >>> instance = os.path.join(sample_buildout, 'parts', 'instance')
   >>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
   %define INSTANCEHOME...instance
   ...
   <BLANKLINE>
   <zodb_db my-fs>
       cache-size 5000
       <filestorage >
         path .../var/filestorage/my-fs/my-fs.fs
       </filestorage>
       mount-point /my-fs
   </zodb_db>
   <BLANKLINE>
   
The recipe will also create a directory for the new filestorage::

    >>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
    True
    
Let's make sure that the conf files will be regenerated whenever we make a change to a filestorage part,
even if the direct configuration for the zope/zeo parts hasn't changed::

    >>> open('buildout.cfg', 'a').write("    my-fs-2\n")
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> 'my-fs-2' in open('parts/instance/etc/zope.conf').read()
    True

Let's make sure that the filestorage directory is not clobbered even if the filestorage part is removed
from the buildout::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     instance
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')    
    >>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
    True
    
We can override the defaults for a number of settings::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     instance
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... location = var/filestorage/%(fs_part_name)s/Data.fs
    ... blob-storage = var/blobstorage-%(fs_part_name)s
    ... zodb-name = %(fs_part_name)s_db
    ... zodb-cache-size = 1000
    ... zodb-mountpoint = /%(fs_part_name)s_mountpoint
    ... zodb-container-class = Products.ATContentTypes.content.folder.ATFolder
    ... parts =
    ...     my-fs
    ... '''.replace('%(zope2_location)s', zope2_location))
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> instance = os.path.join(sample_buildout, 'parts', 'instance')
    >>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME...instance
    ...
    <BLANKLINE>
    <zodb_db my-fs_db>
        cache-size 1000
        <blobstorage >
          blob-dir .../var/blobstorage-my-fs
          <filestorage >
            path .../var/filestorage/my-fs/Data.fs
          </filestorage>
        </blobstorage>
        mount-point /my-fs_mountpoint
        container-class Products.ATContentTypes.content.folder.ATFolder
    </zodb_db>
    <BLANKLINE>

A setting can also be modified just for one particular filestorage, by creating a new part with
the ``filestorage_`` prefix, like so::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     instance
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     my-fs
    ...
    ... [filestorage_my-fs]
    ... zodb-cache-size = 1000
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> instance = os.path.join(sample_buildout, 'parts', 'instance')
    >>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME...instance
    ...
    <BLANKLINE>
    <zodb_db my-fs>
        cache-size 1000
        <filestorage >
          path .../var/filestorage/my-fs/my-fs.fs
        </filestorage>
        mount-point /my-fs
    </zodb_db>
    <BLANKLINE>


By default, the recipe adds the extra filestorages to each plone.recipe.zope2instance part in the buildout,
but you can tell it to only add it to certain parts::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     instance1
    ...     instance2
    ...
    ... [instance1]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [instance2]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... zopes = instance1
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> 'my-fs' in open('parts/instance1/etc/zope.conf').read()
    True
    >>> 'my-fs' in open('parts/instance2/etc/zope.conf').read()
    False

Example Usage with ZEO
======================

Here is a minimal buildout including a ZEO server and two ZODB clients::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver
    ...     primary
    ...     secondary
    ...
    ... [zeoserver]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [secondary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')

This should result in the appropriate additions to zeo.conf and both zope.conf's::

    >>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
    >>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
    %define INSTANCE /sample-buildout/parts/zeoserver
    ...
    <BLANKLINE>
        <filestorage my-fs>
          path /sample-buildout/var/filestorage/my-fs/my-fs.fs
        </filestorage>
    <BLANKLINE>
    
    >>> primary = os.path.join(sample_buildout, 'parts', 'primary')
    >>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME /sample-buildout/parts/primary
    ...
    <BLANKLINE>
    <zodb_db my-fs>
     cache-size 5000
     <zeoclient>
       server 8100
       storage my-fs
       name my-fs_zeostorage
       var /sample-buildout/parts/primary/var
       cache-size 30MB
    <BLANKLINE>
     </zeoclient> 
     mount-point /my-fs
    </zodb_db>
    <BLANKLINE>

    >>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
    >>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME /sample-buildout/parts/secondary
    ...
    <BLANKLINE>
    <zodb_db my-fs>
     cache-size 5000
     <zeoclient>
       server 8100
       storage my-fs
       name my-fs_zeostorage
       var /sample-buildout/parts/secondary/var
       cache-size 30MB
    <BLANKLINE>
     </zeoclient> 
     mount-point /my-fs
    </zodb_db>
    <BLANKLINE>

As above, we can override a number of the default parameters::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver
    ...     primary
    ...     secondary
    ...
    ... [zeoserver]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [secondary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... location = var/filestorage/%(fs_part_name)s/Data.fs
    ... blob-storage = var/blobstorage-%(fs_part_name)s
    ... zodb-cache-size = 1000
    ... zodb-name = %(fs_part_name)s_db
    ... zodb-mountpoint = /%(fs_part_name)s_mountpoint
    ... zeo-address = 8101
    ... zeo-client-cache-size = 50MB
    ... zeo-storage = %(fs_part_name)s_storage
    ... zeo-client-name = %(fs_part_name)s_zeostorage_name
    ... parts =
    ...     my-fs
    ... '''.replace('%(zope2_location)s', zope2_location))
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
    >>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
    %define INSTANCE /sample-buildout/parts/zeoserver
    ...
    <BLANKLINE>
        <blobstorage my-fs_storage>
          blob-dir /sample-buildout/var/blobstorage-my-fs
          <filestorage my-fs_storage>
            path /sample-buildout/var/filestorage/my-fs/Data.fs
          </filestorage>
        </blobstorage>
    <BLANKLINE>
    >>> primary = os.path.join(sample_buildout, 'parts', 'primary')
    >>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME /sample-buildout/parts/primary
    ...
    <BLANKLINE>
    <zodb_db my-fs_db>
     cache-size 1000
     <zeoclient>
       blob-dir /sample-buildout/var/blobstorage-my-fs
       shared-blob-dir on
       server 8101
       storage my-fs_storage
       name my-fs_zeostorage_name
       var /sample-buildout/parts/primary/var
       cache-size 50MB
    <BLANKLINE>
     </zeoclient> 
     mount-point /my-fs_mountpoint
    </zodb_db>
    <BLANKLINE>
    >>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
    >>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
    %define INSTANCEHOME /sample-buildout/parts/secondary
    ...
    <BLANKLINE>
    <zodb_db my-fs_db>
     cache-size 1000
     <zeoclient>
       blob-dir /sample-buildout/var/blobstorage-my-fs
       shared-blob-dir on
       server 8101
       storage my-fs_storage
       name my-fs_zeostorage_name
       var /sample-buildout/parts/secondary/var
       cache-size 50MB
    <BLANKLINE>
     </zeoclient> 
     mount-point /my-fs_mountpoint
    </zodb_db>
    <BLANKLINE>

By default, the recipe adds the extra filestorages to the first
plone.recipe.zope2zeoserver part in the buildout, and will throw an error if
there is more than one part using this recipe.  However, you can override this
behavior by specifying a particular ZEO part.  In this case, the filestorages
will only be added to the Zopes using that ZEO, by default::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver1
    ...     zeoserver2
    ...     primary
    ...     secondary
    ...     other-zope
    ...
    ... [zeoserver1]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ... zeo-address = 8100
    ...
    ... [zeoserver2]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ... zeo-address = 8101
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ... zeo-address = 8101
    ...
    ... [secondary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ... zeo-address = 8101
    ...
    ... [other-zope]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ... zeo-address = 8100
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... zeo = zeoserver2
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> 'my-fs' in open('parts/zeoserver2/etc/zeo.conf').read()
    True
    >>> 'my-fs' in open('parts/zeoserver1/etc/zeo.conf').read()
    False
    >>> 'my-fs' in open('parts/primary/etc/zope.conf').read()
    True
    >>> 'my-fs' in open('parts/other-zope/etc/zope.conf').read()
    False

    
Error conditions
================
    
Important note: You must place all parts using the
collective.recipe.filestorage recipe before the part for the instances and
zeoservers that you are adding the filestorage to.  Otherwise you'll get an
error::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     instance
    ...     filestorage
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    While:
    ...
    Error: [collective.recipe.filestorage] The "filestorage" part must be listed before the following parts in ${buildout:parts}: instance
    <BLANKLINE>


Buildouts with multiple zeoserver parts will result in an
error if the desired ZEO to associate with is not explicitly specified::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver1
    ...     zeoserver2
    ...     primary
    ...     secondary
    ...
    ... [zeoserver1]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [zeoserver2]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [secondary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    While:
    ...
    Error: [collective.recipe.filestorage] "filestorage" part found multiple zeoserver parts; please specify which one to use with the "zeo" option.

Specifying a nonexistent zeo should result in an error::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver
    ...     primary
    ...
    ... [zeoserver]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... zeo = foobar
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    While:
    ...
    Error: [collective.recipe.filestorage] "filestorage" part specifies nonexistant zeo part "foobar".

So should specifying a nonexistent zope part::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     zeoserver
    ...     primary
    ...
    ... [zeoserver]
    ... recipe = plone.recipe.zope2zeoserver
    ... zope2-location = %(zope2_location)s
    ...
    ... [primary]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ... zeo-client = 1
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... zopes = foobar
    ... parts =
    ...     my-fs
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    While:
    ...
    Error: [collective.recipe.filestorage] The "filestorage" part expected but failed to find the following parts in ${buildout:parts}: foobar

If the Zope/ZEO parts are being automatically identified, let's make sure
that we don't accidentally "wake up" parts that would not otherwise be
included in the buildout::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     instance
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     my-fs
    ...
    ... [foobar]
    ... recipe = plone.recipe.distros
    ... urls =
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q')
    >>> 'foobar' in os.listdir(os.path.join(sample_buildout, 'parts'))
    False

Make sure that instance parts are found correctly in buildouts using 'extends'
and the += or -= options::

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... extends = base.cfg
    ... parts =
    ...     filestorage
    ...     instance
    ...
    ... [instance]
    ... recipe = plone.recipe.zope2instance
    ... zope2-location = %(zope2_location)s
    ... user = me
    ...
    ... [filestorage]
    ... recipe = collective.recipe.filestorage
    ... parts =
    ...     extendstest
    ... ''' % globals())
    >>> write('prod.cfg',
    ... '''
    ... [buildout]
    ... extends = buildout.cfg
    ... parts +=
    ...     foobar
    ...
    ... [foobar]
    ... recipe = plone.recipe.distros
    ... urls =
    ... ''' % globals())
    >>> print system(join('bin', 'buildout') + ' -q -c prod.cfg')
    >>> 'extendstest' in open(os.path.join(instance, 'etc', 'zope.conf')).read()
    True

Running the tests
=================

The subversion checkout of collective.recipe.filestorage includes a buildout
which installs a script for running the tests.

Just run::

    python2.4 bootstrap.py
    bin/buildout
    bin/test

Known issue: The tests run buildout in a separate process, so it's currently
impossible to put a pdb breakpoint in the recipe and debug during the test.
If you need to do this, set up another buildout which uses collective.recipe.filestorage
as a development egg.


Reporting bugs or asking questions
==================================

There is a shared bugtracker and help desk on Launchpad:
https://bugs.launchpad.net/collective.buildout/
