-
Notifications
You must be signed in to change notification settings - Fork 13
Open
Description
i need scrap 100.000 url with scrapy but it take much time finish it, at least 5 hour by 9000 request, but if i use
process = CrawlerProcess(get_project_settings())
process.crawl(DataamazonSpider, skus=list_of_sku_1)
process.crawl(DataamazonSpider, skus=list_of_sku_2)
process.start()
i have the error 429 too many request
Metadata
Metadata
Assignees
Labels
No labels