@@ -96,7 +96,7 @@ single Python class that defines one or more of the following methods:
96
96
.. method :: process_response(request, response, spider)
97
97
98
98
:meth: `process_response ` should return a :class: `~scrapy.http.Response `
99
- object or raise a :exc: `~scrapy.exceptions.IgnoreRequest ` exception.
99
+ object or raise a :exc: `~scrapy.exceptions.IgnoreRequest ` exception.
100
100
101
101
If it returns a :class: `~scrapy.http.Response ` (it could be the same given
102
102
response, or a brand-new one), that response will continue to be processed
@@ -131,7 +131,7 @@ single Python class that defines one or more of the following methods:
131
131
kicks in, and won't bother calling any other exception middleware.
132
132
133
133
If it returns a :class: `~scrapy.http.Request ` object, the returned request is
134
- used to instruct an immediate redirection.
134
+ used to instruct an immediate redirection.
135
135
The original request won't finish until the redirected
136
136
request is completed. This stops the :meth: `process_exception `
137
137
middleware the same as returning Response would do.
@@ -236,7 +236,7 @@ Here's an example of a log with :setting:`COOKIES_DEBUG` enabled::
236
236
2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
237
237
[...]
238
238
239
-
239
+
240
240
DefaultHeadersMiddleware
241
241
------------------------
242
242
@@ -256,7 +256,7 @@ DownloadTimeoutMiddleware
256
256
257
257
.. class :: DownloadTimeoutMiddleware
258
258
259
- This middleware sets the download timeout for requests specified in the
259
+ This middleware sets the download timeout for requests specified in the
260
260
:setting: `DOWNLOAD_TIMEOUT ` setting.
261
261
262
262
HttpAuthMiddleware
@@ -317,8 +317,8 @@ DBM storage backend (default)
317
317
318
318
.. versionadded :: 0.13
319
319
320
- A DBM _ storage backend is also available for the HTTP cache middleware. To use
321
- it (instead of the default filesystem backend) set :setting: `HTTPCACHE_STORAGE `
320
+ A DBM _ storage backend is available for the HTTP cache middleware. To use it
321
+ (note: it is the default storage backend) set :setting: `HTTPCACHE_STORAGE `
322
322
to ``scrapy.contrib.httpcache.DbmCacheStorage ``.
323
323
324
324
By default, it uses the anydbm _ module, but you can change it with the
@@ -329,7 +329,9 @@ By default, it uses the anydbm_ module, but you can change it with the
329
329
File system backend
330
330
~~~~~~~~~~~~~~~~~~~
331
331
332
- By default, the :class: `HttpCacheMiddleware ` uses a file system storage with the following structure:
332
+ A file system storage backend is also available for the HTTP cache middleware.
333
+ To use it (instead of the default DBM _ storage backend) set :setting: `HTTPCACHE_STORAGE `
334
+ to ``scrapy.contrib.downloadermiddleware.httpcache.FilesystemCacheStorage ``.
333
335
334
336
Each request/response pair is stored in a different directory containing
335
337
the following files:
@@ -414,7 +416,7 @@ HTTPCACHE_IGNORE_MISSING
414
416
415
417
Default: ``False ``
416
418
417
- If enabled, requests not found in the cache will be ignored instead of downloaded.
419
+ If enabled, requests not found in the cache will be ignored instead of downloaded.
418
420
419
421
.. setting :: HTTPCACHE_IGNORE_SCHEMES
420
422
@@ -432,7 +434,7 @@ Don't cache responses with these URI schemes.
432
434
HTTPCACHE_STORAGE
433
435
^^^^^^^^^^^^^^^^^
434
436
435
- Default: ``'scrapy.contrib.downloadermiddleware. httpcache.FilesystemCacheStorage ' ``
437
+ Default: ``'scrapy.contrib.httpcache.DbmCacheStorage ' ``
436
438
437
439
The class which implements the cache storage backend.
438
440
@@ -455,7 +457,7 @@ HttpCompressionMiddleware
455
457
.. module :: scrapy.contrib.downloadermiddleware.httpcompression
456
458
:synopsis: Http Compression Middleware
457
459
458
- .. class :: HttpCompressionMiddleware
460
+ .. class :: HttpCompressionMiddleware
459
461
460
462
This middleware allows compressed (gzip, deflate) traffic to be
461
463
sent/received from web sites.
@@ -466,7 +468,7 @@ ChunkedTransferMiddleware
466
468
.. module :: scrapy.contrib.downloadermiddleware.chunked
467
469
:synopsis: Chunked Transfer Middleware
468
470
469
- .. class :: ChunkedTransferMiddleware
471
+ .. class :: ChunkedTransferMiddleware
470
472
471
473
This middleware adds support for `chunked transfer encoding `_
472
474
@@ -666,7 +668,7 @@ UserAgentMiddleware
666
668
.. class :: UserAgentMiddleware
667
669
668
670
Middleware that allows spiders to override the default user agent.
669
-
671
+
670
672
In order for a spider to override the default user agent, its `user_agent `
671
673
attribute must be set.
672
674
0 commit comments