npm

Need private packages and team management tools?Check out npm Orgs. »

crawler-request

1.2.2 • Public • Published

crawler-request

HTTP request module customized for crawlers.

version downloads node status

Installation

npm install crawler-request

Usage

Simple Request

const crawler = require('crawler-request');
 
crawler("https://stackoverflow.com/").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

PDF Parse

const crawler = require('crawler-request');
 
crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

Extend

const crawler = require('crawler-request');
 
function response_text_size(response){
    response["size"] = response.text.length;
    return response;
}
 
crawler("https://stackoverflow.com/",response_text_size).then(function(response){
    // handle response
    console.log(response.size);
});

Test

mocha or npm test

Check test folder and quickstart.js for extra usages.

Support

I use this package actively myself, so it has my top priority. You can chat on WhatsApp about any infos, ideas and suggestions.

WhatsApp

Submitting an Issue

If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository

Creating a Merge Request

GitLab calls it merge request instead of pull request.

License

MIT licensed and all it's dependencies are MIT or BSD licensed.

install

npm i crawler-request

Downloadsweekly downloads

494

version

1.2.2

license

MIT

homepage

gitlab.com

repository

Gitgitlab

last publish

collaborators

  • avatar
Report a vulnerability