Metadata-Version: 1.0
Name: reppy
Version: 0.2.2
Summary: Replacement robots.txt Parser
Home-page: http://github.com/seomoz/reppy
Author: Dan Lecocq
Author-email: dan@seomoz.org
License: MIT
Description: Replaces the built-in robotsparser with a
        RFC-conformant implementation that supports modern robots.txt constructs like
        Sitemaps, Allow, and Crawl-delay. Main features:
        
        - Memoization of fetched robots.txt
        - Expiration taken from the `Expires` header
        - Batch queries
        - Configurable user agent for fetching robots.txt
        - Automatic refetching basing on expiration
        
Platform: Posix; MacOS X
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
