Metadata-Version: 1.1
Name: scrapy-random-useragent
Version: 0.1
Summary: Scrapy Middleware to set a random User-Agent for every Request.
Home-page: https://github.com/cnu/scrapy-random-useragent
Author: Srinivasan Rangarajan
Author-email: srinivasanr@gmail.com
License: MIT
Description: Scrapy Random User-Agent
        ========================
        
        Does your scrapy spider get identified and blocked by servers because
        you use the default user-agent or a generic one?
        
        Use this ``random_useragent`` module and set a random user-agent for
        every request. You are limited only by the number of different
        user-agents you set in a text file.
        
        Installing
        ----------
        
        Installing it is pretty simple.
        
        .. code-block:: python
        
            pip install scrapy-random-useragent
        
        Usage
        -----
        
        In your ``settings.py`` file, update the ``DOWNLOADER_MIDDLEWARES``
        variable like this.
        
        .. code-block:: python
        
            DOWNLOADER_MIDDLEWARES = {
                'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
                'random_useragent.RandomUserAgentMiddleware': 400
            }
        
        This disables the default ``UserAgentMiddleware`` and enables the
        ``RandomUserAgentMiddleware``.
        
        Then, create a new variable ``USER_AGENT_LIST`` with the path to your
        text file which has the list of all user-agents
        (one user-agent per line).
        
        .. code-block:: python
        
            USER_AGENT_LIST = "/path/to/useragents.txt"
        
        Now all the requests from your crawler will have a random user-agent
        picked from the text file.
        
Keywords: scrapy random user-agent
Platform: Any
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Framework :: Scrapy
