scraper-api

1.0.0 • Public • Published

scraper-api

npm npm David Travis Coveralls license Beerpay

Interface to call ScraperAPI.com easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.

Install

$ npm install --save scraper-api

Usage

const scraperAPI = require('scraper-api')({
    // ... options
});
 
scraperAPI.get('http://httpbin.org/ip')
    .then(result => {
        // result => '<!doctype html> ...'
    })
    .catch(error => {
        console.error(error);
        //=> 'Internal server error'
    });

API Documentation

scraper-api

scraperAPI(options) ⇒ ScraperAPI

Creates new instance of ScraperAPI with the provided options.

Kind: Exported function
Returns: ScraperAPI - New instance of ScraperAPI.

Param Type Default Description
options Object Optional configuration options to pass into ScraperAPI.
[options.apiKey] String process.env.SCRAPER_API_KEY API key for Scraper API. Defaults to pulling API Key from environment variable.
[options.renderJs] Boolean false Render JavaScript on the page before scraping the HTML for the page.
[options.keepHeaders] Boolean false Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in options.gotOptions.headers.
[options.geoCode] String 'us' Geo code in which to use proxies for when scraping. See documentation for more information.
[options.premium] Boolean false Whether to use premium proxies. Caution: This will cost 10-25 times more than standard proxies.
[options.sessionId] Number A numeric session id to use to maintain the same proxy. See ScraperAPI.session() for more information.
[options.gotOptions] Object {} Additional options to pass into got for requests to ScraperAPI.

Example

const scraperAPI = require('scraper-api')({
    // options...
});

scraperAPI.ScraperAPI

Access to the uninstantiated ScraperAPI class.

Kind: static property of scraperAPI
Example

const ScraperAPI = require('scraper-api').ScraperAPI;
const scraperAPI = new ScraperAPI({
    // options...
});

ScraperAPI

Kind: global class

new ScraperAPI([options])

Constructor for the ScraperAPI class.

Param Type Default Description
[options] Object {} Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information.

scraperAPI.session(id, [options]) ⇒ ScraperAPI

Creates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through the same proxy. See documentation for more information.

Kind: instance method of ScraperAPI
Returns: ScraperAPI - A new instance of ScraperAPI with sessionId set to the provided id.

Param Type Default Description
id Number Session ID in which to use for the given session. Must only contain numbers.
[options] Object {} Options to override for all subsequent requests to Scraper API. Same as the global options.

Example

const session = scraperAPI.session(1234);
 
let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'

scraperAPI.get(url, [options]) ⇒ Promise.<String>

Calls Scraper API with a GET request to the provided url.

Kind: instance method of ScraperAPI
Returns: Promise.<String> - Promise that resolves with HTML source from requested URL.

Param Type Default Description
url String The URL in which to scrape.
[options] Object {} Options to override for this specific request.

Example

let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'

scraperAPI.post(url, data, [options]) ⇒ Promise.<Object>

Calls Scraper API with a POST request to the provided url with the provided data.

Kind: instance method of ScraperAPI
Returns: Promise.<Object> - Promise that resolves with object response from Scraper API. See documentation.

Param Type Default Description
url String The URL in which to scrape.
data Object | form-data Data in which to post to the provided URL. Must be either a plain object or instance of form-data.
[options] Object {} Options to override for this specific request. May be any of the global options and any additional options below.
[options.form] Boolean false Set true if provided data is form data and should be sent as such. By default, data will be sent as JSON.

Example

let result = await scraperAPI.post('https://example.com/endpoint', {
   hello: 'world',
   some: 'data'
});

More Information

keepHeaders Option

If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in options.gotOptions.headers and enabling this option.

let result = await scraperAPI.get('https://google.com', {
    keepHeaders: true,
    gotOptions: {
        headers: {
            'My-Custom-Header': 'some value'
        }
    }
});

Tests

Tests are written and provided as part of the module. You may run the tests by calling:

$ npm run test

License

MIT License. See License in the repository.

Package Sidebar

Install

npm i scraper-api

Weekly Downloads

3

Version

1.0.0

License

MIT

Unpacked Size

29.2 kB

Total Files

9

Last publish

Collaborators

  • kyleross