Metadata-Version: 1.0
Name: lovely.gae
Version: 1.0.0a1
Summary: Appengine related Python Packages from Lovely Systems
Home-page: http://code.google.com/p/lovely-gae/
Author: Lovely Systems
Author-email: office@lovelysystems.com
License: Apache License 2.0
Description: Lovely Appengine Packages
        *************************
        
        ================
        lovely.gae.async
        ================
        
        This package executes jobs asynchronously, it uses the appengine
        taskqueue to exectue the jobs.
        
        >>> from lovely.gae.async import defer, get_tasks
        
        The defer function executes a handler asynchronously as a job. We
        create 3 jobs that have the same signature.
        
        >>> import time
        >>> for i in range(3):
        ...     print defer(time.sleep, [0.3])
        <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
        None
        None
        
        Let us have a look on what jobs are there. Note that there is only one
        because all the 3 jobs we added were the same.
        
        >>> len(get_tasks())
        1
        
        If we change the signature of the job, a new one will be added.
        
        >>> added = defer(time.sleep, [0.4])
        >>> len(get_tasks())
        2
        
        Normally jobs are automatically executed by the taskqueueapi, we have
        a test method which executes the jobs and returns the number of jobs
        done.
        
        >>> run_tasks()
        2
        
        Now we can add the same signature again.
        
        >>> added = defer(time.sleep, [0.4])
        >>> run_tasks()
        1
        
        We can also set only_once to false to execute a worker many times with
        the same signature.
        
        >>> from pprint import pprint
        >>> defer(pprint, ['hello'], once_only=False)
        <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
        >>> defer(pprint, ['hello'], once_only=False)
        <google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
        >>> run_tasks()
        'hello'
        'hello'
        2
        
        
        
        
        
        ==========================
        DB Custom Property Classes
        ==========================
        
        Typed Lists
        ===========
        
        This property converts model instances to keys and ensures a length.
        
        >>> from lovely.gae.db.property import TypedListProperty
        >>> from google.appengine.ext import db
        
        Let us create a model
        
        >>> class Yummy(db.Model): pass
        >>> class Bummy(db.Model): pass
        
        We can now reference Yummy instances with our property. Note that we
        can also use the kind name as an argument of the kind.
        
        >>> class Dummy(db.Model):
        ...     yummies = TypedListProperty(Yummy)
        ...     bummies = TypedListProperty('Bummy', length=3)
        
        The kind arguement needs to be a subclass kind name or a db.Model.
        
        >>> TypedListProperty(object)
        Traceback (most recent call last):
        ...
        ValueError: Kind needs to be a subclass of db.Model
        
        >>> dummy = Dummy()
        >>> dummy_key = dummy.put()
        >>> yummy1 = Yummy()
        >>> yummy1_key = yummy1.put()
        >>> dummy.yummies = [yummy1]
        
        We cannot set any other type.
        
        >>> bummy1 = Bummy()
        >>> bummy1_key = bummy1.put()
        >>> dummy.yummies = [bummy1]
        Traceback (most recent call last):
        ...
        BadValueError: Wrong kind u'Bummy'
        
        The length needs to match if defined (see above).
        
        >>> dummy.bummies = [bummy1]
        Traceback (most recent call last):
        ...
        BadValueError: Wrong length need 3 got 1
        
        >>> dummy.bummies = [bummy1, bummy1, bummy1]
        >>> dummy_key == dummy.put()
        True
        
        Case-Insensitive String Property
        ================================
        
        This property allows searching for the lowercase prefix in a
        case-insensitive manner. This is usefull for autocomplete
        implementations where we do not want to have a seperate property just
        for searching.
        
        >>> from lovely.gae.db.property import IStringProperty
        >>> class Foo(db.Model):
        ...     title = IStringProperty()
        
        >>> f1 = Foo(title='Foo 1')
        >>> kf1 = f1.put()
        >>> f2 = Foo(title='Foo 2')
        >>> kf2 = f2.put()
        >>> f3 = Foo(title='foo 3')
        >>> kf3 = f3.put()
        >>> f4 = Foo(title=None)
        >>> kf4 = f4.put()
        
        The property does not allow the special seperator character which is
        just one below the  highest unicode character,
        
        >>> f3 = Foo(title='Foo 3' + IStringProperty.SEPERATOR)
        Traceback (most recent call last):
        ...
        BadValueError: Not all characters in property title
        
        Note that if we want to do an exact search, we have to use a special
        filter that can be created by the property instance.
        
        >>> [f.title for f in Foo.all().filter('title =', 'foo 1')]
        []
        
        The "equal" filter arguments can be computed with a special method on
        the property.
        
        >>> ef = Foo.title.equals_filter('Foo 1')
        >>> ef
        ('title =', u'foo 1\xef\xbf\xbcFoo 1')
        
        >>> [f.title for f in Foo.all().filter(*ef)]
        [u'Foo 1']
        
        Let us try with inequallity, e.g. prefix search. Prefix search is
        normally done with two filters using the highest unicode character.
        
        Search for all that starts with "fo" case-insensitive.
        
        >>> q = Foo.all()
        >>> q = q.filter('title >=', 'fo')
        >>> q = q.filter('title <', 'fo' + u'\xEF\xBF\xBD')
        >>> [f.title for f in q]
        [u'Foo 1', u'Foo 2', u'foo 3']
        
        Search for all that starts with 'foo 1'
        
        >>> q = Foo.all()
        >>> q = q.filter('title >=', 'foo 1')
        >>> q = q.filter('title <', 'foo 1' + u'\xEF\xBF\xBD')
        >>> [f.title for f in q]
        [u'Foo 1']
        
        >>> q = Foo.all()
        >>> q = q.filter('title >=', 'foo 2')
        >>> q = q.filter('title <=', 'foo 2' + u'\xEF\xBF\xBD')
        >>> [f.title for f in q]
        [u'Foo 2']
        
        >>> q = Foo.all()
        >>> q = q.filter('title >=', 'foo 3')
        >>> q = q.filter('title <=', 'foo 3' + u'\xEF\xBF\xBD')
        >>> [f.title for f in q]
        [u'foo 3']
        
        
        Pickle Property
        ===============
        
        A pickle property can hold any pickleable python object.
        
        >>> from lovely.gae.db.property import PickleProperty
        
        >>> class Pickie(db.Model):
        ...     data = PickleProperty()
        
        >>> pickie = Pickie(data={})
        >>> pickie.data
        {}
        >>> kp = pickie.put()
        >>> pickie.data
        {}
        >>> pickie = db.get(kp)
        >>> pickie.data
        {}
        >>> pickie.data = {'key':501*"x"}
        >>> kp = pickie.put()
        >>> pickie.data
        {'key': 'xx...xx'}
        
        If the value is not pickleable we get a validation error.
        
        >>> pickie = Pickie(data=dict(y=lambda x:x))
        Traceback (most recent call last):
        BadValueError: Property 'data' must be pickleable:
        (Can't pickle <function <lambda> at ...>:
        it's not found as __main__.<lambda>)
        
        
        Safe ReferenceProperty
        ======================
        
        >>> from lovely.gae.db.property import SafeReferenceProperty
        
        We use a new model with a gae reference and our safe reference.
        
        >>> class Refie(db.Model):
        ...     ref   = db.ReferenceProperty(Yummy, collection_name='ref_ref')
        ...     sfref = SafeReferenceProperty(Yummy, collection_name='sfref_ref')
        
        >>> refie = Refie()
        >>> refie.sfref is None
        True
        >>> refie.ref is None
        True
        
        An object to be referenced.
        
        >>> refyummy1 = Yummy()
        >>> ignore = refyummy1.put()
        
        Set the references to our yummy object.
        
        >>> refie.sfref = refyummy1
        >>> refie.sfref
        <Yummy object at ...>
        >>> refie.ref = refyummy1
        >>> refie.ref
        <Yummy object at ...>
        
        >>> refieKey = refie.put()
        
        Now we delete the referenced object.
        
        >>> refyummy1.delete()
        
        And reload our referencing object.
        
        >>> refie = db.get(refieKey)
        
        The gae reference raises an exception.
        
        >>> refie.ref
        Traceback (most recent call last):
        Error: ReferenceProperty failed to be resolved
        
        We catch the logs here.
        
        >>> import logging
        >>> from StringIO import StringIO
        >>> log = StringIO()
        >>> handler = logging.StreamHandler(log)
        >>> logger = logging.getLogger('lovely.gae.db')
        >>> logger.setLevel(logging.INFO)
        >>> logger.addHandler(handler)
        
        Our safe reference returns None.
        
        >>> pos = log.tell()
        >>> refie.sfref is None
        True
        
        Let's see what the log contains.
        
        >>> log.seek(pos)
        >>> print log.read()
        Unresolved Reference for "Refie._sfref" set to None
        
        Accessing the stale property once again we will see it was reset to None::
        
        >>> pos = log.tell()
        >>> refie.sfref is None
        True
        
        >>> log.seek(pos)
        >>> print log.read() == ''
        True
        
        The property get set to None if the reference points to a dead object but only
        if the property is not required::
        
        >>> class Requy(db.Model):
        ...     sfref = SafeReferenceProperty(Yummy, collection_name='req_sfref_ref',
        ...                                   required=True)
        
        >>> refyummy1 = Yummy()
        >>> ignore = refyummy1.put()
        
        >>> requy = Requy(sfref = refyummy1)
        >>> requyKey = requy.put()
        
        >>> requy.sfref
        <Yummy object at ...>
        
        >>> refyummy1.delete()
        
        >>> requy = db.get(requyKey)
        
        >>> pos = log.tell()
        >>> requy.sfref is None
        True
        
        >>> log.seek(pos)
        >>> print log.read()
        Unresolved Reference for "Requy._sfref" will remain because it is required
        
        =====================
        Batch marker creation
        =====================
        
        This packages provides the possibility to create markers for every N
        objects of a given query. This is useful to create batched html pages
        or to generate jobs for every N objects.
        
        A list of attribute values are created that represent the end of a
        batch at any given position in a given query. The result is stored in
        memcache and the key is provided to a callback function.
        
        >>> from lovely.gae import batch
        
        Let us create some test objects.
        
        >>> from google.appengine.ext import db
        >>> class Stub(db.Model):
        ...     c_time = db.DateTimeProperty(auto_now_add=True, required=True)
        ...     name = db.StringProperty()
        ...     age = db.IntegerProperty()
        ...     state = db.IntegerProperty()
        ...     def __repr__(self):
        ...         return '<Stub %s>' % self.name
        
        >>> for i in range(1,13):
        ...     s = Stub(name=str(i), age=i, state=i%2)
        ...     sk = s.put()
        
        >>> Stub.all().count()
        12
        
        First we make sure that we have no tasks in the queue for testing.
        
        >>> from lovely.gae.async import get_tasks
        >>> len(get_tasks())
        0
        
        So for example if we want to know any 100th key of a given kind we
        could calculate it like shown below. Note that we provide the pprint function
        as a callback, so we get the memcache key in the output.
        
        The calculate_markers function returns the memcache key that will be
        used to store the result when the calculation is completed.
        
        >>> from pprint import pprint
        >>> mc_key = batch.create_markers('Stub', callback=pprint)
        >>> mc_key
        'create_markers:...-...-...'
        
        A task gets created.
        
        >>> tasks = get_tasks()
        >>> len(tasks)
        1
        
        Let us run the task.
        
        >>> run_tasks(1)
        1
        
        We now have another task left for the callback, which is actually the
        pprint function.
        
        >>> run_tasks()
        'create_markers:...-...-...'
        1
        
        We should now have a result. The result shows that we need no batches
        for the given batch size (because we only have 12 objects).
        
        >>> from google.appengine.api import memcache
        >>> memcache.get(mc_key)
        []
        
        Let us use another batch size. This time without callback.
        
        >>> mc_key = batch.create_markers('Stub', batchsize=1)
        >>> run_tasks()
        1
        
        We now have exatly 12 keys, because the batch size was 1.
        
        >>> len(memcache.get(mc_key))
        12
        
        The default attributes returned are the keys.
        
        >>> memcache.get(mc_key)
        [datastore_types.Key.fro...
        
        We can also use other attributes. Let us get items batched by c_time
        descending. Note that it is not checked if values are not unique, so
        if a non-unique attribute is used it might be the case that batch
        ranges contains objects twice.
        
        >>> mc_key = batch.create_markers('Stub',
        ...                               attribute='c_time',
        ...                               order='desc',
        ...                               batchsize=3)
        >>> run_tasks()
        1
        >>> markers = memcache.get(mc_key)
        >>> markers
        [datetime.datetime(...
        >>> len(markers)
        4
        >>> sorted(markers, reverse=True) == markers
        True
        
        >>> mc_key = batch.create_markers('Stub',
        ...                               attribute='c_time',
        ...                               order='asc',
        ...                               batchsize=3)
        >>> run_tasks()
        1
        >>> markers = memcache.get(mc_key)
        >>> markers
        [datetime.datetime(...
        >>> len(markers)
        4
        >>> sorted(markers) == markers
        True
        
        
        We can also pass filters to be applied to the query for the batch like this:
        
        >>> mc_key = batch.create_markers('Stub',
        ...                               filters=[('state', 0)],
        ...                               attribute='c_time',
        ...                               order='asc',
        ...                               batchsize=3)
        >>> run_tasks()
        1
        >>> markers = memcache.get(mc_key)
        >>> len(markers)
        2
        
        
        
        
Keywords: appengine datastore backup batch utilities
Platform: UNKNOWN
