Metadata-Version: 1.0
Name: transmogrify.htmlcontentextractor
Version: 1.0b5
Summary: This blueprint extracts out title, description and body from html either via xpath or by automatic cluster analysis
Home-page: http://github.com/djay/transmogrify.htmlcontentextractor
Author: Dylan Jay
Author-email: software@pretaweb.com
License: GPL
Description: Introduction
        ============
        
        transmogrify.htmlcontentextractor
        This blueprint extracts out title, description and body from html
        either via xpath, TAL or by automatic cluster analysis
        
        HTMLContentExtractor
        ====================
        
        This blueprint extracts out fields from html either via xpath rules or by automatic cluster
        analysis
        
        transmogrify.htmlcontentextractor
        ---------------------------------
        
        You can define a series of rules which will get applied to the to the '_text'
        of the input item. The rules use a XPATH expression or a TAL expression to
        extract html or text out of the html and adds it as key to the outputted item.
        
        Each option of the blueprint is a rule of the following form ::
        
        (N-)field = (optional)(text|html|delete|optional) xpath
        
        OR
        
        (N-)field = (optional)tal tal-expression
        
        
        "field" is the attribute that will be set with the results of the xpath
        
        "format" is what to do with the results of the xpath. "optional" means the same
        as "delete" but won't cause the group to not match. if the format is delete or optional
        then the field name doesn't matter but will still need to be unique
        
        "xpath' is an xpath expression
        
        If the format is 'tal' then instead of an XPath use can use a TAL expression. TAL expression
        is evaluated on the item object AFTER the XPath expressions have been applied.
        
        For example ::
        
        [template]
        blueprint = transmogrify.htmlcontentextractor
        title = text //div[@class='body']//h1[1]
        _permalink = text //div[@class='body']//a[@class='headerlink']
        _text = html //div[@class='body']
        _label = optional //p[contains(@class,'admonition-title')]
        description = optional //div[contains(@class,'admonition-description')]/p[@class='last']/text()
        _remove_useless_links = optional //div[@id = 'indices-and-tables']
        mimetype = tal string:text/html
        text = tal python:item['_text'].replace('id="blah"','')
        
        You can delete a number of parts of the html by extracting content to fields such as _permalink and _label.
        These items won't get used be set used to set any properties on the final content so are effective as a means
        of deleting parts of the html.
        TAL expressions are evaluated after XPath expressions so we can post process the _text XPath to produce a text
        stripped of a certain id.
        
        
        N is the group number. Groups are run in order of group number. If
        any rule doesn't match (unless its marked optional) then the next group
        will be tried instead. Group numbers are optional.
        
        Instead of groups you can also chain several blueprints togeather. The blueprint
        will set '_template' on the item. If another blueprint finds the '_template' key in an item
        it will ignore that item.
        
        The '_template' field is the remainder of the html once all the content selected by the
        XPATH expressions have been applied.
        
        
        
        transmogrify.htmlcontentextractor.auto
        --------------------------------------
        This blueprint will analyse the html and attempt to discover the rules to extract out the
        title, description and body of the html.
        
        If the logger output is in DEBUG mode then the XPaths used by the auto extrator will be output
        to the logger.
        
        
        Changelog
        =========
        
        1.0b4 (2011-02-06)
        ------------------
        
        - handle '/text()' in  xpaths
        - new 'optionaltext' rule format
        
        1.0b3 (2010-12-13)
        ------------------
        
        - simpler autogenerated xpath
        - better logging
        
        1.0b2 (2010-11-09)
        ------------------
        
        - Put condition on autofinder so can be turned off
        
        1.0b1 (2010-11-03)
        ------------------
        
        - ignore already found items. better debug
        ["Dylan Jay"]
        
        - skip templates if item already parsed
        ["Dylan Jay"]
        
        - print automaticly found XPaths
        ["Dylan Jay"]
        
        - make text fields strip tail text
        ["Vitaliy Podoba"]
        
        1.0dev (2010-03-22)
        -------------------
        
        - split the auto templatefinder out to it's own blueprint
        ["Dylan Jay"]
        
        
        
        
Keywords: transmogrifier blueprint funnelweb source plone import conversion microsoft office
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Libraries :: Python Modules
