Merge branch 'release-1.0.4' into 1.0

Author: dangra

.travis-workarounds.sh

@@ -1,12 +1,16 @@
 language: python
 python: 2.7
+sudo: false
+branches:
+  only:
+    - master
+    - /^\d\.\d+$/
 env:
 - TOXENV=py27
 - TOXENV=precise
 - TOXENV=py33
 - TOXENV=docs
 install:
-- "./.travis-workarounds.sh"
 - pip install -U tox twine wheel
 script: tox
 notifications:
@@ -15,6 +19,9 @@ notifications:
     skip_join: true
     channels:
     - irc.freenode.org#scrapy
+cache:
+  directories:
+    - $HOME/.cache/pip
 deploy:
   provider: pypi
   distributions: "sdist bdist_wheel"

.travis.yml

@@ -6,13 +6,22 @@ Scrapy
    :target: https://pypi.python.org/pypi/Scrapy
    :alt: PyPI Version
 
+.. image:: https://img.shields.io/pypi/dm/Scrapy.svg
+   :target: https://pypi.python.org/pypi/Scrapy
+   :alt: PyPI Monthly downloads
+
 .. image:: https://img.shields.io/travis/scrapy/scrapy/master.svg
    :target: http://travis-ci.org/scrapy/scrapy
    :alt: Build Status
 
 .. image:: https://img.shields.io/badge/wheel-yes-brightgreen.svg
    :target: https://pypi.python.org/pypi/Scrapy
    :alt: Wheel Status
+   
+.. image:: http://static.scrapy.org/py3progress/badge.svg
+   :target: https://github.com/scrapy/scrapy/wiki/Python-3-Porting
+   :alt: Python 3 Porting Status
+
 
 Overview
 ========

README.rst

@@ -1,6 +1,7 @@
 import glob
 import six
 import pytest
+from twisted import version as twisted_version
 
 
 def _py_files(folder):
@@ -21,6 +22,9 @@ def _py_files(folder):
     "scrapy/spider.py",
 ] + _py_files("scrapy/contrib") + _py_files("scrapy/contrib_exp")
 
+if (twisted_version.major, twisted_version.minor, twisted_version.micro) >= (15, 5, 0):
+    collect_ignore += _py_files("scrapy/xlib/tx")
+
 
 if six.PY3:
     for line in open('tests/py3-ignores.txt'):

conftest.py

@@ -108,7 +108,10 @@
 #html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = []
+# Add path to the RTD explicitly to robustify builds (otherwise might
+# fail in a clean Debian build env)
+import sphinx_rtd_theme
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 
 # The style sheet to use for HTML and HTML Help pages. A file of that name

docs/conf.py

@@ -146,6 +146,14 @@ tests requires `tox`_.
 Running tests
 -------------
 
+Make sure you have a recent enough `tox`_ installation:
+
+    ``tox --version``
+
+If your version is older than 1.7.0, please update it first:
+
+    ``pip install -U tox``
+
 To run all tests go to the root directory of Scrapy source code and run:
 
     ``tox``

docs/contributing.rst

@@ -144,7 +144,7 @@ I get "Filtered offsite request" messages. How can I fix them?
 Those messages (logged with ``DEBUG`` level) don't necessarily mean there is a
 problem, so you may not need to fix them.
 
-Those message are thrown by the Offsite Spider Middleware, which is a spider
+Those messages are thrown by the Offsite Spider Middleware, which is a spider
 middleware (enabled by default) whose purpose is to filter out requests to
 domains outside the ones covered by the spider.
 

docs/faq.rst

@@ -28,6 +28,7 @@ First steps
 ===========
 
 .. toctree::
+   :caption: First steps
    :hidden:
 
    intro/overview
@@ -53,6 +54,7 @@ Basic concepts
 ==============
 
 .. toctree::
+   :caption: Basic concepts
    :hidden:
 
    topics/commands
@@ -110,6 +112,7 @@ Built-in services
 =================
 
 .. toctree::
+   :caption: Built-in services
    :hidden:
 
    topics/logging
@@ -138,6 +141,7 @@ Solving specific problems
 =========================
 
 .. toctree::
+   :caption: Solving specific problems
    :hidden:
 
    faq
@@ -203,6 +207,7 @@ Extending Scrapy
 ================
 
 .. toctree::
+   :caption: Extending Scrapy
    :hidden:
 
    topics/architecture
@@ -240,6 +245,7 @@ All the rest
 ============
 
 .. toctree::
+   :caption: All the rest
    :hidden:
 
    news

docs/index.rst

@@ -14,7 +14,8 @@ The installation steps assume that you have the following things installed:
 * `Python`_ 2.7
 
 * `pip`_ and `setuptools`_ Python packages. Nowadays `pip`_ requires and
-  installs `setuptools`_ if not installed.
+  installs `setuptools`_ if not installed. Python 2.7.9 and later include
+  `pip`_ by default, so you may have it already.
 
 * `lxml`_. Most Linux distributions ships prepackaged versions of lxml.
   Otherwise refer to http://lxml.de/installation.html
@@ -23,9 +24,7 @@ The installation steps assume that you have the following things installed:
   where the Python installer ships it bundled.
 
 You can install Scrapy using pip (which is the canonical way to install Python
-packages).
-
-To install using pip::
+packages). To install using ``pip`` run::
 
    pip install Scrapy
 
@@ -34,6 +33,22 @@ To install using pip::
 Platform specific installation notes
 ====================================
 
+Anaconda
+--------
+
+.. note::
+
+  For Windows users, or if you have issues installing through `pip`, this is
+  the recommended way to install Scrapy.
+
+If you already have installed `Anaconda`_ or `Miniconda`_, the company
+`Scrapinghub`_ maintains official conda packages for Linux, Windows and OS X.
+
+To install Scrapy using ``conda``, run::
+
+  conda install -c scrapinghub scrapy 
+
+
 Windows
 -------
 
@@ -58,7 +73,8 @@ Windows
 
   Be sure you download the architecture (win32 or amd64) that matches your system
 
-* Install `pip`_ from https://pip.pypa.io/en/latest/installing.html
+* *(Only required for Python<2.7.9)* Install `pip`_ from
+  https://pip.pypa.io/en/latest/installing.html
 
   Now open a Command prompt to check ``pip`` is installed correctly:: 
 
@@ -79,13 +95,80 @@ Instead, use the official :ref:`Ubuntu Packages <topics-ubuntu>`, which already
 solve all dependencies for you and are continuously updated with the latest bug
 fixes.
 
+If you prefer to build the python dependencies locally instead of relying on
+system packages you'll need to install their required non-python dependencies
+first::
+
+    sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
+
+You can install Scrapy with ``pip`` after that::
+
+    pip install Scrapy
+
+.. note::
+
+    The same non-python dependencies can be used to install Scrapy in Debian
+    Wheezy (7.0) and above.
+
 Archlinux
 ---------
 
 You can follow the generic instructions or install Scrapy from `AUR Scrapy package`::
 
     yaourt -S scrapy
 
+Mac OS X
+--------
+
+Building Scrapy's dependencies requires the presence of a C compiler and
+development headers. On OS X this is typically provided by Apple’s Xcode
+development tools. To install the Xcode command line tools open a terminal
+window and run::
+
+    xcode-select --install
+
+There's a `known issue <https://github.com/pypa/pip/issues/2468>`_ that
+prevents ``pip`` from updating system packages. This has to be addressed to
+successfully install Scrapy and its dependencies. Here are some proposed
+solutions:
+
+* *(Recommended)* **Don't** use system python, install a new, updated version
+  that doesn't conflict with the rest of your system. Here's how to do it using
+  the `homebrew`_ package manager:
+
+  * Install `homebrew`_ following the instructions in http://brew.sh/
+
+  * Update your ``PATH`` variable to state that homebrew packages should be
+    used before system packages (Change ``.bashrc`` to ``.zshrc`` accordantly
+    if you're using `zsh`_ as default shell)::
+
+      echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
+
+  * Reload ``.bashrc`` to ensure the changes have taken place::
+
+      source ~/.bashrc
+
+  * Install python::
+
+      brew install python
+
+  * Latest versions of python have ``pip`` bundled with them so you won't need
+    to install it separately. If this is not the case, upgrade python::
+
+      brew update; brew upgrade python
+
+* *(Optional)* Install Scrapy inside an isolated python environment.
+
+  This method is a workaround for the above OS X issue, but it's an overall
+  good practice for managing dependencies and can complement the first method.
+
+  `virtualenv`_ is a tool you can use to create virtual environments in python.
+  We recommended reading a tutorial like
+  http://docs.python-guide.org/en/latest/dev/virtualenvs/ to get started.
+
+After any of these workarounds you should be able to install Scrapy::
+
+  pip install Scrapy
 
 .. _Python: https://www.python.org/
 .. _pip: https://pip.pypa.io/en/latest/installing.html
@@ -95,3 +178,9 @@ You can follow the generic instructions or install Scrapy from `AUR Scrapy packa
 .. _OpenSSL: https://pypi.python.org/pypi/pyOpenSSL
 .. _setuptools: https://pypi.python.org/pypi/setuptools
 .. _AUR Scrapy package: https://aur.archlinux.org/packages/scrapy/
+.. _homebrew: http://brew.sh/
+.. _zsh: http://www.zsh.org/
+.. _virtualenv: https://virtualenv.pypa.io/en/latest/
+.. _Scrapinghub: http://scrapinghub.com
+.. _Anaconda: http://docs.continuum.io/anaconda/index
+.. _Miniconda: http://conda.pydata.org/docs/install/quick.html

docs/intro/install.rst

@@ -34,7 +34,7 @@ These are some common properties often found in broad crawls:
 
 As said above, Scrapy default settings are optimized for focused crawls, not
 broad crawls. However, due to its asynchronous architecture, Scrapy is very
-well suited for performing fast broad crawls. This page summarize some things
+well suited for performing fast broad crawls. This page summarizes some things
 you need to keep in mind when using Scrapy for doing broad crawls, along with
 concrete suggestions of Scrapy settings to tune in order to achieve an
 efficient broad crawl.
@@ -46,7 +46,7 @@ Concurrency is the number of requests that are processed in parallel. There is
 a global limit and a per-domain limit.
 
 The default global concurrency limit in Scrapy is not suitable for crawling
-many different  domains in parallel, so you will want to increase it. How much
+many different domains in parallel, so you will want to increase it. How much
 to increase it will depend on how much CPU you crawler will have available. A
 good starting point is ``100``, but the best way to find out is by doing some
 trials and identifying at what concurrency your Scrapy process gets CPU

docs/topics/broad-crawls.rst

@@ -8,7 +8,7 @@ This section describes the different options you have for deploying your Scrapy
 spiders to run them on a regular basis. Running Scrapy spiders in your local
 machine is very convenient for the (early) development stage, but not so much
 when you need to execute long-running spiders or move spiders to run in
-production continously. This is where the solutions for deploying Scrapy
+production continuously. This is where the solutions for deploying Scrapy
 spiders come in.
 
 Popular choices for deploying Scrapy spiders are:

docs/topics/deploy.rst

@@ -736,7 +736,7 @@ RetryMiddleware
 
 .. class:: RetryMiddleware
 
-   A middlware to retry failed requests that are potentially caused by
+   A middleware to retry failed requests that are potentially caused by
    temporary problems such as a connection timeout or HTTP 500 error.
 
 Failed pages are collected on the scraping process and rescheduled at the

docs/topics/downloader-middleware.rst

@@ -57,7 +57,7 @@ remain disabled. Those components include:
 
  * Extensions
  * Item pipelines
- * Downloader middlwares
+ * Downloader middlewares
  * Spider middlewares
 
 The exception must be raised in the component constructor.

docs/topics/exceptions.rst

@@ -17,7 +17,7 @@ Extensions use the :ref:`Scrapy settings <topics-settings>` to manage their
 settings, just like any other Scrapy code.
 
 It is customary for extensions to prefix their settings with their own name, to
-avoid collision with existing (and future) extensions. For example, an
+avoid collision with existing (and future) extensions. For example, a
 hypothetic extension to handle `Google Sitemaps`_ would use settings like
 `GOOGLESITEMAP_ENABLED`, `GOOGLESITEMAP_DEPTH`, and so on.
 
@@ -145,7 +145,7 @@ Here is the code of such extension::
             self.items_scraped += 1
             if self.items_scraped % self.item_count == 0:
                 logger.info("scraped %d items", self.items_scraped)
-                
+
 
 .. _topics-extensions-ref:
 

docs/topics/extensions.rst

@@ -118,7 +118,7 @@ they work as we expect.
 
 As you can see, the page markup is not very descriptive: the elements don't
 contain ``id``, ``class`` or any attribute that clearly identifies them, so
-we''ll use the ranking bars as a reference point to select the data to extract
+we'll use the ranking bars as a reference point to select the data to extract
 when we construct our XPaths.
 
 After using FireBug, we can see that each link is inside a ``td`` tag, which is

docs/topics/firebug.rst

@@ -95,7 +95,7 @@ contain a price::
 Write items to a JSON file
 --------------------------
 
-The following pipeline stores all scraped items (from all spiders) into a a
+The following pipeline stores all scraped items (from all spiders) into a
 single ``items.jl`` file, containing one item per line serialized in JSON
 format::