安装 Scrapy for Python2.7

最新推荐文章于 2025-05-27 09:04:28 发布

原创最新推荐文章于 2025-05-27 09:04:28 发布 · 726 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

收录于

scrapy

python

本文介绍Scrapy爬虫框架的安装步骤与基本使用方法。针对Mac10.11系统的安装环境，详细说明了所需依赖及解决常见安装问题的方案。包括对six库的重新安装和特定版本twisted框架的安装。最后展示了如何通过命令行启动Scrapy并列出可用命令。

1. 简介

Scrapy是基于Python的网络爬虫框架，支持Python2.7 和 Python3.3以上。

2. 安装

安装平台： Mac 10.11

2.1 先决条件

python 2.7
pip
setuptools
lxml
openssl

2.2 安装scrapy

pip install scrapy

2.3 重新安装six

Mac自带的six库需要重新安装

sudo rm -rf /Library/Python/2.7/site-packages/six*
sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*
sudo pip install six

2.4 重新安装twisted

安装scrapy时自动安装的twisted框架版本有问题，需要重新安装

pip install twisted==13.1.0

3. 使用

现在就能使用scrapy了

$ scrapy
Scrapy 1.3.3 - project: tutorial

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  check         Check spider contracts
  commands      
  crawl         Run a spider
  edit          Edit spider
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  list          List available spiders
  parse         Parse URL (using its spider) and print the results
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command