bettong

WIP: Bettong is a JavaScript Node.js web crawler based in Puppeteer. Based on the provided base URL, Bettong crawls pages on the same origin saves screenshots and HTML content.

Requirements

Bettong uses async/await which is only available in Node.js 8.x.x or higher.

Options

argument	type	required	default	description
baseUrl	string	true	null	URL to start crawling from. Bettong will only crawl pages that are on the same origin.
outputPath	string	false	"dist"	Relative path screenshots and/or html content will be saved to. Currently screenshots will be saved to provided-relative-path/screenshots and HTML content will be saved to provided-relative-path/html.
options	object	false	{}	Bettong options.
options.screenshot	boolean	false	true	Whether Bettong should save screenshot for each viewport.
options.html	boolean	false	true	Whether Bettong should save HTML content for each page.
options.viewports	puppeteer.Viewport	false	[ { width: 540, height: 480, }, { width: 720, height: 480 }, { width: 960, height: 480 }, { width: 1140, height: 480 } ]	Array of viewports used to take screenshots. Only used if options.screenshot is set to true. Please puppeteer docs for more information on available properties for Viewport interface.

Usage

Node.js

Install bettong

npm install --save bettong

const Bettong = require('bettong');
 
const bettong = new Bettong('https://foo.bar');
await bettong.exec();

CLI

Usage: bettong exec [options] <base-url>
 
Execute crawling starting from the required base url <base-url>
 
Options:
  -o, --output-path <path>         relative output path (default: "dist")
  -e, --exclude-pattern <pattern>  RegExp page exclude pattern (default: "")
  -s, --screenshot <screenshot>    save screenshots (default: true)
  -h, --html <screenshot>          save html content (default: true)
  -v, --viewport <viewport>        viewport for screenshots, e.g. '{"width":128,"height":128}'
  -h, --help                       output usage information

Install bettong globally

npm install -g bettong

Samples

Start crawling at https://foo.bar and exclude crawling pages that contain 'baz' in the url.

bettong exec https://foo.bar -e '.*baz.*'

Output

Bettong will save screenshots to provided-relative-path/screenshots and HTML content to provided-relative-path/html. By default this would be dist/screenshots and dist/html. A screenshot will be saved as a full page screenshot in PNG format for each viewport provided in options.viewports.