Naive Puppets Marching

    deep-scrape

    0.0.5 • Public • Published

    Deep Scrape

    npm version

    Scrape and crawl pages with io.js and get a whole lot of meta data. Shows; headers, Ajax requests/responses, rendered html, Javascript AST's, dependencies, console events, and a whole lot more. Crawl sites, or scrape a single page. Add cookies or proxy requests. Fingerprints common javascript libraries, and allows you to write your own.

    Installation

    This was tested on node 0.12.x. It can be run as a module export, or a command line script.

    npm install deep-scrape
    // or clone the repository and run it as a script.

    Use Case

    • You are scraping websites with lots of javascript (Angular, Ember, Browserfy).
    • You don't mind trading a bit of performance for more detailed scraping data.
    • You would like to find potential DOM sinks and sources on your pages (Possibly for vulnerability scanning).
    • You need the most detailed metadata, metrics, and analyitics on your scraped pages.
    • You would like to fingerprint possible technologies a certain site or page uses.

    Install

    npm i deep-scrape

    DownloadsWeekly Downloads

    18

    Version

    0.0.5

    License

    MIT

    Last publish

    Collaborators

    • zacharyiles