krake-toolkit
A package of common methods used in Krake's Web Scraping Engine
Pre-requisities
- NodeJS — v0.8.8
- Coffee-Script
Folder structure
Lib
classes in this folder are organized into 3 types
- query
- data
- usage
query
Classes that support the manipulation and conversion of Krake definitions into various formats
data
Classes that act as adaptors for writing extracted data to various Databases. Supported databases includes
- MongoDB
- MySQL
- Postgresql - HSTORE
Usage
Classes that act as clients to hook Krake slaves to the central Krake clustered computing webscraping infrastructure
Setup
git clone git@github.com:KrakeIO/krake-toolkit.gitcd krake-toolkitnpm install