node package manager

phantomjscloud

Official PhantomJs Cloud Client API Library for Node.js

PhantomJsCloud

This library provides a simple and high quality mechanism for interacting with PhantomJs via Node. It leverages PhantomJsCloud.com (A Headless Browser Service) to allow multiple calls in parallel at high performance.

Requirements

  • Internet access.

Optional: Get an ApiKey by creating an account at PhantomJsCloud.com

Platforms

  • This works in Node.js and Browsers (via Browserify or Webpack).

Basic Usage

  1. Install: npm install phantomjscloud --save
  2. Use:
        var phantomJsCloud = require("phantomjscloud");
        var browser = new phantomJsCloud.BrowserApi();
        
        browser.requestSingle({ url: "http://www.example.com", renderType: "plainText" }, (err, userResponse) => {
            //can use a callback like this example, or a Promise (see the Typescript example below) 
            if (err != null) {
                throw err;
            }
            console.log(JSON.stringify(userResponse.content));
        });

or a slightly more complex example using Typescript:

        //typings will automatically be loaded 
        import phantomJsCloud = require("phantomjscloud");
        
        let apiKey = undefined;//leave undefined to use a demo key.  get a free key at https://Dashboard.PhantomJsCloud.com 
        let browser = new phantomJsCloud.BrowserApi(apiKey);
        
        //the page you wish to render 
        let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "http://www.example.com", renderType:"plainText" };
        
        //make a request to render a single page, returning the plain-text contents of the page. 
        browser.requestSingle(pageRequest)
            .then((userResponse) => {
                console.log(JSON.stringify({
                    //the content of the page you requested 
                    content: userResponse.content,
                    //metadata about your request, such as billing 
                    meta: userResponse.meta,
                    //the status code of your request 
                    statusCode: userResponse.statusCode
                }, null, "\t")); //pretty-print 
            });

Options

var browser = new phantomJsCloud.BrowserApi(apiKeyOrOptions?) : BrowserApi;

Constructing the browserApi, and optionally for setting default configuration options.

  • apiKeyOrOptions: Optional. If set, can be either an apiKey:string or an options object with the parameters {apiKey, endpointOrigin, suppressDemoKeyWarning, requestOptions, retryOptions }.
    • apiKey:string Optional. If not set, the default "demo" key is used with the public cloud endpoint. If you use the default demo key, you get 100 Pages/Day. If you sign up for an account at PhantomJsCloud.com you will get 500 Pages/Day free.
    • endpointOrigin:string Optional. Used if you subscribe to a Private Cloud + SLA. Defaults to the PhantomJs Cloud Public Endpoint.
    • suppressDemoKeyWarning Optional. set to true to not show a warning for using demo keys.
    • requestOptions Optional. override HTTP request options, default to undefined (use defaults). Takes the following parameters:
      • timeout Optional. default timeout for the network request is 65000 (65 seconds)
    • retryOptions Optional. override network failure retry options, default to undefined (use defaults) Takes the following parameters:
      • timeout Optional. assumes the network request timed out if it takes longer than this. default is 66000 (66 seconds)
      • max_tries Optional. maximum number of attempts to try the operation. default is 3
      • interval Optional. initial wait time between retry attempts in milliseconds(default 1000)
      • backoff Optional. if specified, increase interval by this factor between attempts
      • max_interval Optional. if specified, maximum amount that interval can increase to
  • RETURNS: A BrowserApi object that is used to make the requests to the PhantomJs Cloud.

browser.requestSingle(request, customOptions?, callback?) : Promise<IUserResponse>

For making a single request.

browser.requestBatch(requests, customOptions?, callback? ) : Promise<IUserResponse>[]

Submit multiple requests at the same time, and get an array of promises back.

  • requests: An array. Each element should be either be a IPageRequest or IUserRequest object.
  • customOptions: Optional. can override the options set in the BrowserApi class constructor.
  • callback: Optional. For people who don't use promises. If you use this, the function should have the signature (err: Error, item: {request, result}) => void This will be called once for each request sent.
  • RETURNS: An array of Promises. Use a construct like bluebird.all() to wait for all to finish if you wish.

Typescript Typings

If you use Visual Studio or VSCode the IntelliSense will automatically load when you: import phantomjscloud = require("phantomjscloud");

You do not need to load anything from the DefinitelyTyped nor Typings projects.

Examples

Here are some basic examples. Look at the phantomjscloud-node-examples project on github for more.

Capture Amazon.com as a PDF

Basic Javascript

    var pageRequest = { url: "https://amazon.com", renderType: "pdf" };
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));
    
    browser.requestSingle(pageRequest, (err, userResponse) => {
        if (userResponse.statusCode != 200) {
            throw new Error("invalid status code" + userResponse.statusCode);
        }
    
        fs.writeFile(userResponse.content.name, userResponse.content.data,
            {
                encoding: userResponse.content.encoding,
            }, (err) => {
                console.log("captured page written to " + userResponse.content.name);
            });
    });

Typescript with Promises

    //the page you wish to render 
    let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "https://amazon.com", renderType: "pdf" };
    
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));
    browser.requestSingle(pageRequest)
        .then((userResponse) => {
    
            if (userResponse.statusCode != 200) {
                throw new Error("invalid status code" + userResponse.statusCode);
            }
            
            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

All Parameters

Shows using all parameters in a request, capturing the page as a .jpg image. Most the parameters used are the defaults, but you can see a list of the most up-to-date default values here.

Typescript with Promises

    //the page you wish to render 
    let userRequest: phantomJsCloud.ioDatatypes.IUserRequest = {
            pages:[
                {
                    "url": "http://example.com",
                    "content": null,
                    "urlSettings": {
                        "operation": "GET",
                        "encoding": "utf8",
                        "headers": {},
                        "data": null
                    },
                    "renderType": "jpg",
                    "outputAsJson": false,
                    "requestSettings": {
                        "ignoreImages": false,
                        "disableJavascript": false,
                        "userAgent": "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34 PhantomJS/2.0.0 (PhantomJsCloud.com/2.0.1)",
                        "authentication": {
                            "userName": "guest",
                            "password": "guest"
                        },
                        "xssAuditingEnabled": false,
                        "webSecurityEnabled": false,
                        "resourceWait": 15000,
                        "resourceTimeout": 35000,
                        "maxWait": 35000,
                        "waitInterval": 1000,
                        "stopOnError": false,
                        "resourceModifier": [],
                        "customHeaders": {},
                        "clearCache": false,
                        "clearCookies": false,
                        "cookies": [],
                        "deleteCookies": []
                    },
                    "suppressJson": [
                        "events.value.resourceRequest.headers",
                        "events.value.resourceResponse.headers",
                        "frameData.content",
                        "frameData.childFrames"
                    ],
                    "renderSettings": {
                        "quality": 70,
                        "pdfOptions": {
                            "border": null,
                            "footer": {
                                "firstPage": null,
                                "height": "1cm",
                                "lastPage": null,
                                "onePage": null,
                                "repeating": "%pageNum%/%numPages%"
                            },
                            "format": "letter",
                            "header": null,
                            "height": null,
                            "orientation": "portrait",
                            "width": null
                        },
                        "clipRectangle": null,
                        "renderIFrame": null,
                        "viewport": {
                            "height": 1280,
                            "width": 1280
                        },
                        "zoomFactor": 1,
                        "passThroughHeaders": false
                    },
                    "scripts": {
                        "domReady": [],
                        "loadFinished": []
                    }
                }
            ],
            proxy:false
        };
    
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(userRequest, null, "\t"));
    browser.requestSingle(userRequest)
        .then((userResponse) => {
    
            if (userResponse.statusCode != 200) {
                throw new Error("invalid status code" + userResponse.statusCode);
            }
            
            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

Technical details

Internally this library will pool all requests and execute in a FIFO fashion. The number of parallel requests increases automatically: We gracefully ramp-up the rate of requests to match PhantomJsCloud's autoscale capacity.

Roadmap

  1. web crawler
  2. batch api framework (store results to S3, Google Cloud Storage, etc)
  3. page automation helpers (button clicking, etc. you can do this already, but you need to know a lot to do it...)

Contributing

This is the official, reference API for PhantomJsCloud.com You can help out by writing API Client Libraries for other languages (lots of requests for PHP and Python!)