[[TOC]]
== Storable API ==

The Storable API is a lightweight database access layer that is used in the modu toolkit. Although it was designed with SQL-based relational databases in mind, the mapping process is sufficiently flexible to allow use with any sort of data store.

Although it shares some features with some of the other myriad object-relational database mapping systems, there is very little "magic". Some conventions are encouraged, but few are required to use Storables, encouraging adaptation to existing schemas. For more complex data systems, custom Factory classes can be implemented to customize the manner in which new objects are instantiated.

=== Compatibility Note ===
At the time of this writing, modu's official database support is limited to MySQL. This document will unfortunately contain much MySQL-specific information, but in most cases attempts are made to use SQL standard syntax.

PostgreSQL has been tested with some success (there are existing Storable implementations for a number of popular Postgres DB-API 2.0 drivers), but there's still a existing MySQLdb dependency in [source:/trunk/modu/persist/sql.py].

=== IDs ===

Storable was based around a belief in using database-unique identifiers (the term "GUID" will be used even though it's not entirely accurate). Original reasons for use of GUIDs included support of multiple-master replication, but also to avoid reliance on database-specific sequences. Also, there is a default assumption that 'id' will be the name of the identity column, but both this and the use of GUIDs can be overridden at the factory level.

Note that at this time aggregate keys are not supported by DefaultFactory, but custom factory implementations should be able to fill the need until this capability can be integrated into the Storable API.

=== Getting Started ===

To begin, we'll define our schema. This will include a `guid` table, which for now will manage a single sequence.

{{{
#!sql
CREATE TABLE `guid` (
  `guid` bigint(20)
) ENGINE=MyISAM DEFAULT CHARACTER SET utf8;

CREATE TABLE `page` (
  `id` bigint(20),
  `url_code` varchar(255),
  `title` varchar(255),
  `data` LONGBLOB,
  `active` tinyint(1),
  PRIMARY KEY (`id`),
  UNIQUE KEY url_code_idx(`url_code`),
  KEY active_idx(`active`)
) ENGINE=MyISAM DEFAULT CHARACTER SET utf8;
}}}

Basic access to data like this is fairly straightforward. Here's a trivial example:

{{{
#!python
from modu.persist import dbapi, Store

db_url = 'MySQLdb://project:fh7fkW9s@localhost/project'
db_pool = dbapi.connect(db_url)
store = Store(db_pool)

store.ensure_factory('page')
page = store.load_one('page', url_code='home-page', active=1)

print 'ID#:   %s' % page.get_id()
print 'Title: %s' % page.title)
print page.data
}}}

=== Factories and You ===

The Storable API is an easily misunderstood framework. Because the default configuration of a freshly created Store object is geared only towards creating the simplest SQL queries, new users can read too far into the provided interface and assume that it's simply a code generation tool. The reality is that Storable provides a rich mechanism for loading database objects, but based around the idea that the developer knows best for a given project.

To better understand the API, it's important to understand the function and role of Storable Factories, and how they relate to the process of loading new content. First, let's look at a snippet of code from the above example:

{{{
#!python
store = Store(db_pool)
store.ensure_factory('page')
page = store.load_one('page', url_code='home-page', active=1)
}}}

The first line is straightforward, and is instantiating the ORM interface to the provided database. Until a factory is registered, the store has no ability to load anything from the database.

The second line is really the important one, although much of its meaning is abstracted away. It is identical in function to the following code:

{{{
#!python
from modu.persist import storable
page_factory = storable.DefaultFactory('page', model_class=storable.Storable, id_col='id', guid_table='guid', use_cache=None)
store.register_factory('page', page_factory)
}}}

The plain-English explanation is "when provided with the factory ID 'page', use the following Factory to create and update Storable objects that contain their primary key in a field called 'id', which pulls keys from the GUID table 'guid', and do not cache the results". The illustrated keyword arguments are defaults, so they could be safely omitted.

Using those defaults, the DefaultFactory class will create generic Storable objects from the rows of your result set. You can override the class used for the resulting objects by passing it as the `model_class` keyword argument. The provided class must be a subclass of Storable, however.

This brings us finally to the last important line:

{{{
#!python
page = store.load_one('page', url_code='home-page', active=1)
}}}

This call to load_one() will fetch the Factory associated with the factory ID 'page'. The resulting query that the factory builds will simply take the provided keyword arguments (or an optional positional argument containing a dictionary of column-value pairs), and build the simple query:

{{{
#!sql
SELECT * FROM `page` WHERE active = 1 AND url_code = 'home-page';
}}}

=== SQL Generation ===

Use of the store to load objects is encouraged (as opposed to using straight queries) because the persistence layer will take care of all query generation, ensuring protection from SQL injection attacks, and generally keeping the code more succinct and readable. Attempts are only made to deal with the simplest (and most common) queries, and the assumption is that advanced developers will want to write their own, but there's a good deal of power still available without writing queries.

modu uses an approach found in the MySQLdb driver (and probably others), which allows for customization of a list of types that can be interpolated into queries. The following examples illustrate the new interpolation options:

==== List Types ====
{{{
#!python
obj = store.load_one('table', id=(1,2,3,4))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE id IN (1,2,3,4);
}}}

==== Inversion ====
{{{
#!python
from modu.persist import sql
obj = store.load_one('table', name=sql.NOT('phil'))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE name <> 'phil';
}}}

{{{
#!python
from modu.persist import sql
obj = store.load_one('table', id=sql.NOT((1,2,3,4)))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE id NOT IN (1,2,3,4);
}}}

==== Comparison ====
{{{
#!python
from modu.persist import sql
obj = store.load_one('table', id=sql.GT(1))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE id > 1;
}}}

{{{
#!python
from modu.persist import sql
obj = store.load_one('table', id=sql.LT(1))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE id < 1;
}}}

==== NULLs ====
{{{
#!python
obj = store.load_one('table', name=None)
}}}

{{{
#!sql
SELECT * FROM `table` WHERE ISNULL(name);
}}}

==== RAW SQL ====
{{{
#!python
from modu.persist import sql
obj = store.load_one('table', date=sql.RAW('NOW()'))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE date = NOW();
}}}

{{{
#!python
from modu.persist import sql
obj = store.load_one('table', name=sql.RAW('INSTR(%s, "substring")'))
}}}

{{{
#!sql
SELECT * FROM `table` WHERE INSTR(name, "substring");
}}}

=== Custom Models ===

Storable diverts from the usual ORM requirement of creating custom model declarations for any table you wish to use. If you're just looking to load a record from the database, there's no need to configure a custom model. However, for any significant project, you'll want to create classes that adhere to the IStorable interface.

Unlike other ORMs, there's no requirement that you specify your schema inside the model class. Things by default work in a very database-centric fashion; if a column exists in the query, it will be available on an object loaded via that query.

You must also remember the following rules when creating a Storable model:

 1. All model classes must implement the IStorable interface. Most models can simply inherit from the base Storable class.
 2. The model constructor must not take any arguments, and it must call the superclass constructor with the table name being used.
 3. Any member variables that are not table column names should be prefixed with at least one underscore.

You have some control over how the object is instantiated by the Factory by way of the load_data() method. load_data() will be called with a single result row from the database; the default behavior is to set each column name as an instance attribute on the model class. Likewise, when saving the object back to the database, all instance attributes that do not begin with an underscore are saved to their corresponding columns. Instantiation control can be achieved through customization of these instance methods, but often times a better solution is creation of a new Factory class.

=== Paginator ===

The modu persistence layer comes with a utility class for dealing with paginating result sets. It's particularly useful when paging through a large number of simple queries, but also supports some advanced usage as well.

Basic usage is as follows:

{{{
#!python
from modu.persist import page, sql
from modu.util import theme

def prepare_content(self, req):
    pager = page.Paginator()
    
    # which page to fetch
    pager.page = int(req.data.get('page', '1'))
    # the default per_page is 10
    pager.per_page = 25
    
    # we still need to set up the factory
    req.store.ensure_factory('page')
    
    # get_results() takes care of adding the limit, and
    # getting the total (pre-limit) rowcount
    results = pager.get_results(req.store, 'page', date=sql.LT(sql.RAW('NOW()')))
    
    self.set_slot('results', results)
    
    # the default theme class can render a pager control
    base_path = req.get_path(req.path)
    thm = theme.Theme(req)
    page_guide = thm.page_guide(pager, base_path)
    self.set_slot('page_guide', page_guide)

}}}
