scrapyjs-jsdom-server

scrapyjs-jsdom-server is a Scrapyjs-compatible jsdom-based rendering service. This Node-based server is a drop-in replacement for the default server that's used with Scrapyjs.

Purpose

This project was started because the default Splash server that ScrapingHub makes almost always crashes after a few hours of heavy scraping (memory bug perhaps?).

Try using the default Splash server and if you find that you can't use it for heavy scraping, come back and use this server instead. Happy scraping!

Usage

Standalone

git clone https://github.com/paulborza/scrapyjs-jsdom-server.git
cd scrapyjs-jsdom-server
npm install

cd src
supervisor app.js

As a Node package

npm install --save scrapyjs-jsdom-server