Metadata-Version: 1.1
Name: ooniprobe
Version: 1.2.2
Summary: Network Interference detection tool.
Home-page: https://ooni.torproject.org/
Author: Open Observatory of Network Interference
Author-email: ooni-dev@torproject.org
License: BSD 2 clause
Description: ooniprobe: a network interference detection tool
        ================================================
        
        .. image:: https://travis-ci.org/TheTorProject/ooni-probe.png?branch=master
            :target: https://travis-ci.org/TheTorProject/ooni-probe
        
        .. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png
            :target: https://coveralls.io/r/TheTorProject/ooni-probe
        
        ___________________________________________________________________________
        
        .. image:: https://ooni.torproject.org/theme/img/ooni-logo.png
            :target: https:://ooni.torproject.org/
        
        OONI, the Open Observatory of Network Interference, is a global observation
        network which aims is to collect high quality data using open methodologies,
        using Free and Open Source Software (FL/OSS) to share observations and data
        about the various types, methods, and amounts of network tampering in the
        world.
        
        
            "The Net interprets censorship as damage and routes around it."
                        - John Gilmore; TIME magazine (6 December 1993)
        
        
        ooniprobe is the first program that users run to probe their network and to
        collect data for the OONI project. Are you interested in testing your network
        for signs of surveillance and censorship? Do you want to collect data to share
        with others, so that you and others may better understand your network? If so,
        please read this document and we hope ooniprobe will help you to gather
        network data that will assist you with your endeavors!
        
        Read this before running ooniprobe!
        -----------------------------------
        
        Running ooniprobe is a potentially risky activity. This greatly depends on the
        jurisdiction in which you are in and which test you are running. It is
        technically possible for a person observing your internet connection to be
        aware of the fact that you are running ooniprobe. This means that if running
        network measurement tests is something considered to be illegal in your country
        then you could be spotted.
        
        Futhermore, ooniprobe takes no precautions to protect the install target machine
        from forensics analysis.  If the fact that you have installed or used ooni
        probe is a liability for you, please be aware of this risk.
        
        OONI in 5 minutes
        =================
        
        On debian testing or unstable::
        
            sudo apt-get install ooniprobe
        
        If you are running debian stable you can get it from backports via::
        
            sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list'
            sudo apt-get update && sudo apt-get install ooniprobe
        
        On unix systems::
        
            sudo pip install ooniprobe
        
        To install it from the current master run::
        
            sudo pip install https://github.com/TheTorProject/ooni-probe/archive/master.zip
        
        The run::
        
            mkdir my_decks
            sudo ooniresources --update-inputs --update-geoip
            oonideckgen -o my_decks/
        
        The output from the last command will tell you how to run ooniprobe to perform
        the measurment.
        
        If you would like to contribute measurements to OONI daily you can also add
        this to your crontab::
        
            @daily ooni $THE_OONI_COMMAND
        
        Run this command to automatically update your crontab:: 
        
              (crontab -l 2>/dev/null; echo "@daily ooni $THE_OONI_COMMAND") | crontab -
        
        Installation
        ============
        
        Debian based systems
        --------------------
        
        If you are running debian testing or debian unstable you can install ooniprobe
        simply with::
            
            apt-get install ooniprobe
        
        If you are running debian stable you can get it from backports via::
        
            sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list'
            sudo apt-get update && sudo apt-get install ooniprobe
        
        
        We also have our own debian repository. To install it from there run this::
        
            sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list'
            gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4
            gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add -
            sudo apt-get update && sudo apt-get install ooniprobe
        
        Linux
        -----
        
        We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as
        well as versions of Ubuntu such as natty and later releases. Running ooniprobe
        without installing it is supported with the following commands::
        
            git clone https://git.torproject.org/ooni-probe.git
            cd ooni-probe
            ./setup-dependencies.sh
            python setup.py install
        
        
        Other platforms (with Vagrant)
        ------------------------------
        
        1. Install Vagrant (https://www.vagrantup.com/downloads.html) and Install Virtualbox (https://www.virtualbox.org/wiki/Downloads)
        
        2. On OSX:
        
        If you don't have it install homebrew http://mxcl.github.io/homebrew/::
        
            brew install git
        
        On debian/ubuntu::
        
            sudo apt-get install git
        
        3. Open a Terminal and run::
        
            git clone https://git.torproject.org/ooni-probe.git
            cd ooni-probe/
            vagrant up
        
        4. Login to the box with::
        
            vagrant ssh
        
        ooniprobe will be installed in ``/ooni``.
        
        5. You can run tests with::
        
            ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt
        
        Using ooniprobe
        ===============
        
        **Net test** is a set of measurements to assess what kind of internet censorship is occurring.
        
        **Decks** are collections of ooniprobe nettests with some associated inputs.
        
        **Collector** is a service used to report the results of measurements.
        
        **Test helper** is a service used by a probe for successfully performing its measurements.
        
        **Bouncer** is a service used to discover the addresses of test helpers and collectors.
        
        Configuring ooniprobe
        ---------------------
        
        You may edit the configuration for ooniprobe by editing the configuration file
        found inside of ``~/.ooni/ooniprobe.conf``.
        
        By default ooniprobe will not include personal identifying information in the
        test result, nor create a pcap file. This behavior can be personalized.
        
        
        Updating resources
        ------------------
        
        To generate decks you will have to update the input resources of ooniprobe.
        
        This can be done with::
        
            ooniresources --update-inputs
        
        If you get a permission error, you may have to run the command as root or
        change the ooniprobe data directory inside of `ooniprobe.conf`.
        
        On some platforms, for example debian contrib, you will not get all the geoip
        related files needed. In that case it is possible to manually download them
        with ``ooniresources``::
        
            ooniresources --update-geoip
        
        Generating decks
        ----------------
        
        You can generate decks for your country thanks to the oonideckgen command.
        
        If you wish, for example, to generate a deck to be run in the country of Italy,
        you can do so (be sure to have updated the input resources first) by running::
        
            oonideckgen --country-code IT --output ~/
        
        You will now have in your home a folder called `deck-it`, containing the ooni
        deck (ends with .deck) and the inputs.
        Note: that you should not move the `deck-*` directory once it has been
        generated as the paths to the inputs referenced by the test in the deck are
        absolute. If you want your deck to live in another directory you must
        regenerated it.
        
        
        Running decks
        -------------
        
        You will find all the installed decks inside of ``/usr/share/ooni/decks``.
        
        You may then run a deck by using the command line option ``-i``:
        
        As root::
        
            ooniprobe -i /usr/share/ooni/decks/mlab.deck
        
        
        Or as a user::
        
            ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck
        
        
        Or:
        
        As root::
        
            ooniprobe -i /usr/share/ooni/decks/complete.deck
        
        
        Or as a user::
        
            ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck
        
        
        The above tests will require around 20-30 minutes to complete depending on your network speed.
        
        If you would prefer to run some faster tests you should run:
        As root::
        
            ooniprobe -i /usr/share/ooni/decks/fast.deck
        
        
        Or as a user::
        
            ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck
        
        
        Running net tests
        -----------------
        
        You may list all the installed stable net tests with::
        
        
            ooniprobe -s
        
        
        You may then run a nettest by specifying its name for example::
        
        
            ooniprobe manipulation/http_header_field_manipulation
        
        
        It is also possible to specify inputs to tests as URLs::
        
        
            ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1
        
        
        You can find the result of the test in your current working directory.
        
        By default the report result will be collected by the default ooni collector
        and the addresses of test helpers will be obtained from the default bouncer.
        
        You may also specify your own collector or bouncer with the options ``-c`` and
        ``-b``.
        
        
        Bridges and obfsproxy bridges
        =============================
        
        ooniprobe submits reports to oonib report collectors through Tor to a hidden
        service endpoint. By default, ooniprobe uses the installed system Tor, but can
        also be configured to launch Tor (see the advanced.start_tor option in
        ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if
        obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path
        to a file that should contain a set of "bridge" lines (of the same format as
        used in torrc, and as returned by https://bridges.torproject.org). If obfsproxy
        bridges are to be used, the path to the obfsproxy binary must be configured.
        See option advanced.obfsproxy_binary, in ooniprobe.conf.
        
        (Optional) Install obfsproxy
        ----------------------------
        
        Install the latest version of obfsproxy for your platform.
        
        Download Obfsproxy: https://www.torproject.org/projects/obfsproxy.html.en
        
        Setting capabilities on your virtualenv python binary
        =====================================================
        
        If your distributation supports capabilities you can avoid needing to run OONI as root::
        
        
            setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python
        
        
        Reporting bugs
        ==============
        
        You can report bugs and issues you find with ooni-probe on The Tor Projec issue
        tracker filing them under the "Ooni" component: https://trac.torproject.org/projects/tor/newticket?component=Ooni.
        
        You can either register an account or use the group account "cypherpunks" with
        password "writecode".
        
        Contributing
        ============
        
        You can download the code for ooniprobe from the following git repository::
        
        
            git clone https://git.torproject.org/ooni-probe.git
        
        
        It is also viewable on the web via: https://gitweb.torproject.org/ooni-probe.git.
        
        You should then submit patches for review as pull requests to this github repository: 
        
        https://github.com/TheTorProject/ooni-probe
        
        Read this article to learn how to create a pull request on github (https://help.github.com/articles/creating-a-pull-request).
        
        If you prefer not to use github (or don't have an account), you may also submit
        patches as attachments to tickets.
        
        Be sure to format the patch (given that you are working on a feature branch
        that is different from master) with::
        
        
            git format-patch master --stdout > my_first_ooniprobe.patch
        
        
        Setting up development environment
        ----------------------------------
        
        On debian based systems this can be done with::
        
            sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper
            mkvirtualenv ooniprobe
            python setup.py install
            pip install -r requirements-dev.txt
        
        
        Changelog
        =========
        
        v1.2.2 (Fri, 17 Oct 2014)
        -------------------------
        
        Who said friday 17th is only bad luck?
        
        * Add two new report entry keys test_start_time and test_runtime
        
        * Fix bug that lead to ooniresources not working properly
        
        v1.2.0 (Wed, 1 Oct 2014)
        -------------------------
        
        * Introduce a new tool for generating ooniprobe test decks called oonideckgen.
        
        * Introduce a new tool for updating resources used for geoip lookup and deck
          generation.
        
        * Add support for policy aware bouncing in the client.
          https://trac.torproject.org/projects/tor/ticket/12579
        
        * Various improvements to the bridge_reachability test (enable better tor
          logging and also log obfsproxy)
        
        * Fix backward compatibility with twisted 13.1 and add regression tests for
          this.
          https://trac.torproject.org/projects/tor/ticket/13139
        
        v1.1.1 (Sun, 24 Aug 2014)
        -------------------------
        
        * Update MANIFEST.in to include the manpages for ooniprobe and oonireport.
        
        * Raise a more specific exception when multiple test cases are in a single
          nettest file and the usageOptions are incoherent.
        
        v1.1.0 (Tue, 19 Aug 2014)
        -------------------------
        
        In this new release of ooniprobe we have added a new command line tool for
        listing the reports that have not been published to a collector and that allows
        the probe operator to choose which ones they would like to upload.
        
        We have also made some privacy improvements to the reports (we will sanitize
        all things that may look like file paths) and added metadata associated with
        the maxmind database being used by the probe operator.
        
        Here is a more detailed list of what has been done:
        
        * Annotate on disk which reports we have submitted and which ones we have not:
          https://trac.torproject.org/projects/tor/ticket/11860
        
        * Add tool called oonireport for publishing unpublished ooniprobe reports to a
          collector: https://trac.torproject.org/projects/tor/ticket/11862
        
        * Probe Report does not leak filepaths anymore:
          https://trac.torproject.org/projects/tor/ticket/12706
        
        * Reports now include version information about the maxmind database being
          used: https://trac.torproject.org/projects/tor/ticket/12771
        
        * We now do integrity checks on the ooniprobe.conf file so that we don't start
          the tool if the config file is missing some settings or is not consistent:
          https://trac.torproject.org/projects/tor/ticket/11983
          (thanks to Alejandro López (kudrom))
        
        * Improvements have been made to the sniffer subsystem (thanks to Alejandro
          López (kudrom))
        
        * Fix the multi protocol traceroute test.
          https://trac.torproject.org/projects/tor/ticket/12883
        
        Minor bug fixes:
        
        * Fix dns_spoof test (by kudrom)
          https://trac.torproject.org/projects/tor/ticket/12486
        
        * ooni might not look at requiresTor:
          https://trac.torproject.org/projects/tor/ticket/11858
        
        * ooni spits out gobs of tracebacks if Tor is not running and the OONI config
          says it will be:
          https://trac.torproject.org/projects/tor/ticket/11859
        
        * The README for ooni-probe should mention the bugtracker and repository
          https://trac.torproject.org/projects/tor/ticket/11980
        
        v1.0.2 (Fri, 9 May 2014)
        ------------------------
        
        * Add ooniprobe manpage.
        
        * Fix various security issues raised by the least authority audit.
        
        * Add a test that checks for Tor bridge reachability.
        
        * Record the IP address of the exit node being used in torified requests.
        
        * Captive portal test now uses the ooni-probe test templates.
        
        * Have better test naming consistency.
        
        v1.0.1 (Fri, 14 Mar 2014)
        -------------------------
        
        * Fix bugs in the traceroute test that lead to not all packets being collected.
        
        * All values inside of http_requests test are now initialized inside of setUp.
        
        * Fix a bug that lead to the input value of the report not being set in some
          circumstances.
        
        * Add bridge_reachability test
        
        v1.0.0 (Thu, 20 Feb 2014)
        -------------------------
        
        * Add bouncer support for discovering test helpers and collectors
        
        * Fix bug that lead to HTTP tests to stall
        
        * Add support for connect_error and connection_lost_error error types
        
        * Add support for additional Tor configuration keys
        
        * Add disclaimer when running ooniprobe
        
        v0.1.0 (Mon, 17 Jun 2013)
        -------------------------
        
        Improvements to HTML/JS based user interface:
        
          * XSRF protection
        
          * user supplied input specification
        
        Bugfixing and improvements to scheduler.
        
        v0.0.12 (Sat, 8 Jun 2013)
        -------------------------
        
        Implement JS/HTML based user interface.
        
        Supports:
        
          * Starting and stopping of tests
        
          * Monitoring of test progress
        
        v0.0.11 (Thu, 11 Apr 2013)
        --------------------------
        
        * Parametrize task timeout and retry count
        
        * Set the default collector via the command line option
        
        * Add option to disable the default collector
        
        * Add continuous integration with travis
        
        v0.0.10 (Wed, 26 Dec 2012)
        --------------------------
        
        ooniprobe:
        
        * Fix bug that made HTTP based tests stall
        
        * Update DNS Test example to not import the DNS Test template If you import the
        	DNS Test template it will be considered a valid test case and command line
        	argument parsing will not work as expected. see:
        	#7795 for more details
        
        * Fix major bug in DNS test template that prevented PTR lookups from working
        	properly I was calling the queryUDP function with the arguments in the wrong
        	order. Twisted, why you API no consistent?
        
        * Add support for specifying the level of parallelism in tests (aka router
        	melt mode)
        
        * Do not swallow failures when a test instance fails to run fixes #7714
        
        scripts:
        
        * Add report archival script
        
        Fix bug in TCP connect test that made it not properly log errors
        
        * Refactor failure handling code in nettest Add function that traps all the
        	supported failure and outputs the failure string representing it.
        
        documentation:
        
        * Add birdseye view of the ooniprobe architecture
        
        * Add details on the current implementation status of ooni*
        
        * Add draft ooniprobe API specification
        
        * Add instructions for supervisord configuration and clean up README.md
        
        0.0.9 (Tue, 11 Dec 2012)
        ------------------------
        
        ooniprobe:
        
        * Set the default ASN to 0
        
        * Make Beautiful soup a soft depedency
        
        * Add support for sending the ASN number of the probe:
        	the ASN number will get sent when creating a new report
        
        * Add support for obtaining the probes IP address via getinfo address as per
        	https://trac.torproject.org/projects/tor/ticket/7447
        
        * Fix bug in ooniprobe test decks
        	https://trac.torproject.org/projects/tor/ticket/7664
        
        oonib:
        
        * Use twisted fdesc when writing to files
        
        * Add support for processing the ASN number of the probe
        
        * Test reports shall follow the specification detailed inside of docs/reports.rst
        
        * Add support for setting the tor binary path in oonib/config.py
        
        scripts:
        
        * Add a very simple example on how to securely parse the ooniprobe reports
        
        documentation:
        
        * Add documentation for the DNSSpoof test
        
        * Add documentation for HTTPHeaderFieldManipulation
        
        * Clean up writing_tests.rst
        
        * Properly use the power of sphinx!
        
        Tests:
        
        * fixup Netalyzr third party plugin
        
        v0.0.8-alpha (Sun, 2 Dec 2012)
        ------------------------------
        
        ooniprobe:
        
        * Allow test resolver file to have comments.
        
        * Autostart Tor in default configuration.
        
        * Add support for starting Tor via txtorcon.
        
        * Make the sniffer not run in a separate thread, but use a non blocking fdesc.
        	Do some refactoring of scapy testing, following Factory creational pattern
        	and a pub-sub pattern for the readers and writers.
        
        * Extend TrueHeaders to support calculation of difference between two HTTP headers respectful of
        	capitalization
        
        * Implement test deck system for automating the specification of command line
        	arguments for tests
        
        * Implement sr1 in txscapy
        
        * Include socksproxy address in HTTP based tests
        
        * Include the resolver IP:Port in the report
        
        * Changes to the report format of HTTP Test template derived tests:
        	Requests are now stored inside of an array to allow
        	the storing of multiple request/response pairs.
        
        * Fix bug that lead to httpt based reports to not have the url attribute set
        	properly.
        
        * twisted Headers() class edited to avoid header fix in reference to:
        	https://trac.torproject.org/projects/tor/ticket/7432
        
        * Parametrize tor socksport for usage with modified HTTP Agent
        
        * Update URL List test to take as input also a single URL
        
        * Clean up filenames of reports generated by ooni-probe:
        	they now follow the format $testName_report_$timestamp.yamloo
        
        * Add ooniprobe prefix to logs
        
        * Respect the includeip = false option in ooniprobe.conf for scapyt derivate
        	tests:
        	If the option to not include the IP address of the probe is set,
        	change the source and destination ip address of the sent and received
        	packets to 127.0.0.1.
        
        tests:
        
        * Implement basic keyword filtering detection test.
        
        * Add ICMP support to multi protocol traceroute test
        
        * parametrize max_ttl and timeout
        
        * make max_ttl and timeout be included in the report
        
        * Port UK Mobile Network test to new API
        
        * Port daphn3 test
        
        * Randomize source port by default in traceroute test and include source port in
        	report
        
        * Test and Implement HTTP Header Field Manipulation Test (rename it to what we
        	had originally called it since it made most sense)
        
        * Implement test that detects DNS spoofing
        
        * Implement TCP payload sending test template:
        	Example test based on this test template
        
        * Make report IDs include the timestamp of the report
        
        * Add test that detects censorship in HTTP pages based on HTTP body length
        
        * Add socks proxy support to HTTP Test
        
        * Create DNS Test template:
        	Use such template for DNS Tamper test.
        	Add example usage of DNS Test Template.
        
        * Refactor captive portal test to run tests in threads
        
        oonib:
        
        * Implement basic collector for ooniprobe reports.
        	Reports can be submitted over the network via http to a remote collector.
        	Implement the backend component of the collector that writes submitted
        	reports to flat files, following the report_id naming convention.
        
        * Implement very simple HTTP Request backend that does only the part of HTTP we
        	need for testing
        
        * Make oonib a daemon
        
        * Loosen up the oonib regexp to support the timestamp report format
        
        * Add Tor Hidden Service support
        
        * Make the reporting directory of the collector configurable
        
        * Implement TCP Echo test helper.
        
        scripts:
        
        * Add fabfile for automatic deployment of ooni-probe to remote sites
        
        documentation:
        
        * Update documentation on how to setup ooniprobe.
        
        v0.0.7.1-alpha (Sun, 11 Nov 2012)
        ---------------------------------
        
        * Add software version to the report
        
        * Implement basic oonib reporting to flat files containing the report ID.
        
        * Improve HTTP Host test to work with the HTTP Requests test backend
        
        v0.0.7-alpha (Sat, 10 Nov 2012)
        -------------------------------
        
        * Add test_name key to ooniprobe reports
        
        * Port TCP connect test to the new API
        
        v0.0.4-alpha (Sat, 10 Nov 2012)
        -------------------------------
        
        * Add multi protocol multi port traceroute for UDP and TCP
        
        * Implement basic HTTP request test that does capitalization variations on the
          HTTP method.
        
        * Bugfixing and refactoring of txscapy for sending and receiving of scapy
          packets.
        
        v0.0.3-alpha (Fri, 9 Nov 2012)
        ------------------------------
        
        * Implement logging to PCAP file support
        
        * Remove dependency on trial
        
        * Port china trigger to new API
        
        * Rename keyword filtering test to HTTP keyword filtering
        
        * Refactor install documentation.
        
        * Convert header of ooniprobe script to a non docstring
        
        * Add Makefile to fetch Maxmind geoip database files
        
        * Implement GeoIP lookup support
        
        * From configuration options it is possible to choice what level of privacy
        	the prober is willing to accept. Implement config file support You are able
        	to specify basic and advanced options in YAML format
        
        * Remove raw inputs and move them to a separate repository and add Makefile to
        	fetch such lists
        
        0.0.1-alpha (Tue, 6 Nov 2012)
        -----------------------------
        
        First release of ooni-probe. woot!
        
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Framework :: Twisted
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Telecommunications Industry
Classifier: License :: OSI Approved :: BSD LicenseProgramming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2 :: Only
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: BSD
Classifier: Operating System :: POSIX :: BSD :: BSD/OS
Classifier: Operating System :: POSIX :: BSD :: FreeBSD
Classifier: Operating System :: POSIX :: BSD :: NetBSD
Classifier: Operating System :: POSIX :: BSD :: OpenBSD
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Unix
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Security
Classifier: Topic :: Security :: Cryptography
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Traffic Generation
Classifier: Topic :: System :: Networking :: Monitoring
