scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster. http://scrapy-cluster.readthedocs.io/