thresher

Headless ScraperJSON scraping for Node.js

Thresher is a library for modern web scraping in Node.js. It is unique in that:

  • it is headless: URLs are rendered in a GUI-less browser, meaning the version of the HTML you scrape is the same one visitors would see on their screen
  • it is declarative: Scrapers are defined in separate JSON files. This means no programming required! It also means any other software supporting the same format could use the same scraper definitions.

Thresher was developed as part of the ContentMine stack for mining the academic literature.

npm install --save thresher

See quickscrape for a thorough example of thresher in action.

We are not yet accepting contributions, if you'd like to help please drop me an email (richard@contentmine.org) and I'll let you know when we're ready for that.

  • 0.0.1 - fork thresher from quickscrape
  • 0.0.2 - solve the dependency problem by exposing binaries from installed packages
  • 0.0.3 - fix rendering issue on some websites
  • 0.0.4 - automatic scraper selection
  • 0.0.5 - fix bug in ScraperBox for multiple matches
  • 0.0.6 - refactor, headless rendering optional, multiple URLs, automatic scraper selection
  • 0.0.7 - support scraperJSON v0.0.2: add regex capability, remove all logging, move to event-based design
  • 0.0.8 - resolve URLs with head-follow requests; refactor to have a Scraper class; supply cookies to downloader; use jsdom-lite to forestall some of the worst installation issues (thanks to @darobin).
  • 0.0.9 - fix broken release
  • 0.1.0 - complete rewrite to support scraperJSON v0.1

Copyright (c) 2014 Shuttleworth Foundation. Licensed under the MIT license.