crawler-toolbox

Intro

a common crawler for webpages

Feature

• cache support (persistent with sqlite default on /$homedir/crawler.db)
• markdown support
• dom operation support (cheerio)

Dev Setup

yarn
yarn start

Test

yarn test

Project generated by gengjiawen/ts-scaffold