Metadata-Version: 1.1
Name: scrapy-redirect
Version: 0.1.0
Summary: Restrict authorized Scrapy redirections to the website start_urls
Home-page: http://github.com/mapado/scrapy-redirect
Author: Balthazar Rouberol
Author-email: balthazar@mapado.com
License: The MIT License (MIT)
Description: ``scrapy-redirect`` restricts authorized HTTP redirections to the website start_urls
        
        Why?
        ----
        
        If the `Scrapy <http://scrapy.org/>`_ ``REDIRECT_ENABLED`` config key is set to ``False`` and a request to the homepage of the crawled website returns a `3XX status code <https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection>`_, the crawl will stop immediatly, as the redirection will not be followed.
        
        ``scrapy-redirect`` will force Scrapy to tolerate redirections coming from the ``start_urls`` urls, in the case  where ``REDIRECT_ENABLED = False``, to avoid this particular problem.
        
        Installation
        ------------
        
        .. code-block:: bash
        
            $ pip install scrapy-redirect
        
        
        Configuration
        --------------
        
        Install ``scrapy-redirect`` in your Scrapy middlewares by adding the following key/value pair in the ``SPIDER_MIDDLEWARES`` settings key (in ``settings.py``):
        
        .. code-block:: python
        
            SPIDER_MIDDLEWARES = {
                ...
                'scrapyredirect.HomepageRedirectMiddleware': 575,
                ...
            }
        
        Note that it is important for the middleware order value to be inferior to 600 (the `default value <http://doc.scrapy.org/en/0.16/topics/settings.html#downloader-middlewares-base>`_  of the ``'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware'`` middleware), as it must be executed before Scrapy blocks the redirection.
        
        NB: if ``REDIRECT_ENABLED = True``, ``scrapy-redirect`` does nothing.
        
        License
        -------
        
        ``scrapy-redirect`` is published under the MIT License.
Keywords: scrapy crawl scraping
Platform: Any
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: No Input/Output (Daemon)
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
