This package has been deprecated
with modern library, there's not need for any wrapper code
npm install scrapebp
This module can be forked or depended upon for future scraping projects.
Caller only need to specify
opts and implement the custom scraper and scrape callback function.
See bin/scrapebp for details.
var ScrapeBp = ;// DemoScraper and scrapeCallback are defined// opts for ScrapeBp is preparedvar scrapebp = ;scrapebp;scrapebp;scrapebp;scrapebp;
scrapebp uses visionmedia/debug.
DEBUG=scrapebp bin/scrapebp www.yahoo.com
Reference for dependencies
write tests that covers:
- GET with query string
- POST with payload
- use of compression (
-zand check response header and decoded body)
- error handling
- character set detection with aadsm/jschardet? (in case HTTP header and HTML meta did not signals charset)
- multi-byte cut-off (https://github.com/tomas/needle/issues/88)