PhantomJsCloud

This library provides a simple and high quality mechanism for interacting with Chrome as a scaleable cloud service. It leverages PhantomJsCloud.com (A Headless Browser Service) to allow multiple calls in parallel at high performance.

• PhantomJsCloud
  • Chrome by default.
  • Requirements
  • Platforms
  • Basic Usage
  • Options
    • phantomJsCloud.BrowserApi()
    • browser.requestSingle()
    • browser.requestBatch()
  • Typescript Typings
  • Examples - Capture Amazon.com as a PDF - All Parameters
  • Technical details
  • Roadmap
  • Contributing
    • Building

Chrome by default.

• You may also use the obsolete PhantomJs Browser instead of the default (Chrome Latest, v74+, via Puppeteer) by setting the backend="phantomjs" parameter (see options, below).

Requirements

• Internet access.
• Tested on Nodejs 10.15.2, but should work with Nodejs >= 6.x

Optional: Get an ApiKey by creating an account at PhantomJsCloud.com

Platforms

• This works in Node.js and Browsers (via Browserify or Webpack).
• Also tested to work using node via Amazon Lambda and Google Cloud Functions.

Basic Usage

1. Install: npm install phantomjscloud --save
2. Use:

        var phantomJsCloud = require("phantomjscloud");
        var browser = new phantomJsCloud.BrowserApi();
        
        browser.requestSingle({ url: "http://www.example.com", renderType: "plainText" }, (err, userResponse) => {
            //can use a callback like this example, or a Promise (see the Typescript example below)
            if (err != null) {
                throw err;
            }
            console.log(JSON.stringify(userResponse.content));
        });

or a slightly more complex example using async/await via Typescript:

        //typings will automatically be loaded
        import phantomJsCloud = require("phantomjscloud");
        
        let apiKey = undefined;//leave undefined to use a demo key.  get a free key at https://Dashboard.PhantomJsCloud.com
        let browser = new phantomJsCloud.BrowserApi(apiKey);
        
        async function doWork(){
            //the page you wish to render
            let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "http://www.example.com", renderType:"plainText" };			
 
            //make a request to render a single page, returning the plain-text contents of the page.
            const userResponse = await browser.requestSingle(pageRequest);
            console.log(JSON.stringify({
                //the content of the page you requested
                content: userResponse.content,
                //metadata about your request, such as billing
                meta: userResponse.meta,
                //the status code of your request
                statusCode: userResponse.statusCode
            }, null, "\t")); //pretty-print
 
        }
        doWork();

Options

phantomJsCloud.BrowserApi()

var browser = new phantomJsCloud.BrowserApi(apiKeyOrOptions?) : BrowserApi;

Constructing the browserApi, and optionally for setting default configuration options.

• apiKeyOrOptions: Optional. If set, can be either an apiKey:string or an options object with the parameters {apiKey, endpointOrigin, suppressDemoKeyWarning, requestOptions, retryOptions }.
  • apiKey:string Optional. If not set, the default "demo" key is used with the public cloud endpoint. If you use the default demo key, you get 100 Pages/Day. If you sign up for an account at PhantomJsCloud.com you will get 500 Pages/Day free.
  • endpointOrigin:string Optional. Used if you subscribe to a Private Cloud + SLA. Defaults to the PhantomJs Cloud Public Endpoint.
  • suppressDemoKeyWarning Optional. set to true to not show a warning for using demo keys.
  • requestOptions Optional. override HTTP request options, default to undefined (use defaults). Takes the following parameters:
    • timeout Optional. default timeout for the network request is 65000 (65 seconds)
  • retryOptions Optional. override network failure retry options, default to undefined (use defaults) Takes the following parameters:
    • timeout Optional. assumes the network request timed out if it takes longer than this. default is 66000 (66 seconds)
    • max_tries Optional. maximum number of attempts to try the operation. default is 3
    • interval Optional. initial wait time between retry attempts in milliseconds(default 1000)
    • backoff Optional. if specified, increase interval by this factor between attempts
    • max_interval Optional. if specified, maximum amount that interval can increase to
• RETURNS: A BrowserApi object that is used to make the requests to the PhantomJs Cloud.

browser.requestSingle()

browser.requestSingle(request, customOptions?, callback?) : Promise<IUserResponse>

For making a single request.

• request: Either be a IPageRequest or IUserRequest object. See https://phantomjscloud.com/docs/http-api/ for full details. The request default values can be seen here.
• customOptions: Optional. can override the options set in the BrowserApi class constructor.
• callback: Optional. For people who don't use promises. If you use this, the function should have the signature (err: Error, result: IUserResponse) => void
• RETURNS: A Promise returning a IUserResponse.

browser.requestBatch()

browser.requestBatch(requests, customOptions?, callback? ) : Promise<IUserResponse>[]

Submit multiple requests at the same time, and get an array of promises back.

• requests: An array. Each element should be either be a IPageRequest or IUserRequest object.
• customOptions: Optional. can override the options set in the BrowserApi class constructor.
• callback: Optional. For people who don't use promises. If you use this, the function should have the signature (err: Error, item: {request, result}) => void This will be called once for each request sent.
• RETURNS: An array of Promises. Use a construct like bluebird.all() to wait for all to finish if you wish.

Typescript Typings

If you use Visual Studio or VSCode the IntelliSense will automatically load when you: import phantomjscloud = require("phantomjscloud");

You do not need to load anything from the DefinitelyTyped nor Typings projects.

Examples

Here are some basic examples. Look at the phantomjscloud-node-examples project on github for more.

Capture Amazon.com as a PDF

Basic Javascript

    var pageRequest = { url: "https://amazon.com", renderType: "pdf" };
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));
    
    browser.requestSingle(pageRequest, (err, userResponse) => {
        if (userResponse.statusCode != 200) {
            throw new Error("invalid status code" + userResponse.statusCode);
        }
    
        fs.writeFile(userResponse.content.name, userResponse.content.data,
            {
                encoding: userResponse.content.encoding,
            }, (err) => {
                console.log("captured page written to " + userResponse.content.name);
            });
    });

Typescript with Promises

    //the page you wish to render
    let pageRequest: phantomJsCloud.ioDatatypes.IPageRequest = { url: "https://amazon.com", renderType: "pdf" };
    
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(pageRequest, null, "\t"));
    browser.requestSingle(pageRequest)
        .then((userResponse) => {
    
            if (userResponse.statusCode != 200) {			
                throw new Error("invalid status code" + userResponse.statusCode);
            }
            
            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

All Parameters

Shows using all parameters in a request, capturing the page as a .jpg image. Most the parameters used are the defaults, but you can see a list of the most up-to-date default values here.

Typescript with Promises

    //the page you wish to render
    let userRequest: phantomJsCloud.ioDatatypes.IUserRequest = {
            pages:[
                {
                    "url": "http://example.com",
                    "content": null,
                    "urlSettings": {
                        "operation": "GET",
                        "encoding": "utf8",
                        "headers": {},
                        "data": null
                    },
                    renderType: 'jpg',
                    outputAsJson: false,
                    requestSettings: {
                        ignoreImages: false,
                        disableJavascript: false,
                        userAgent: 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34 PhantomJS/2.0.0 (PhantomJsCloud.com/2.0.1)',
                        xssAuditingEnabled: false,
                        webSecurityEnabled: false,
                        resourceWait: 15000,
                        resourceTimeout: 35000,
                        maxWait: 35000,
                        ioWait: 2000,
                        waitInterval: 1000,
                        stopOnError: false,
                        resourceModifier: [],
                        customHeaders: {},
                        clearCache: false,
                        clearCookies: false,
                        cookies: [],
                        deleteCookies: [],
                    },
                    suppressJson: [
                        'events.value.resourceRequest.headers',
                        'events.value.resourceResponse.headers',
                        'frameData.content',
                        'frameData.childFrames',
                    ],
                    renderSettings: {
                        quality: 70,
                        viewport: {
                        height: 1280,
                        width: 1280,
                        },
                        zoomFactor: 1,
                        passThroughHeaders: false,
                        emulateMedia: 'screen',
                        omitBackground: false,
                        passThroughStatusCode: false,
                    },
                    scripts: {
                        pageNavigated: [],
                        load: [],
                        domReady: [],
                        loadFinished: [],
                    },
                    scriptSettings: {
                        stopOnError: false,
                        async: false,
                    },
                    }
            ],
            proxy:false
        };
    
    console.log("about to request page from PhantomJs Cloud.  request =", JSON.stringify(userRequest, null, "\t"));
    browser.requestSingle(userRequest)
        .then((userResponse) => {
    
            if (userResponse.statusCode != 200) {			
                throw new Error("invalid status code" + userResponse.statusCode);
            }
            
            fs.writeFile(userResponse.content.name, userResponse.content.data,
                {
                    encoding: userResponse.content.encoding,
                }, (err) => {
                    console.log("captured page written to " + userResponse.content.name);
                });
        });

Technical details

Internally this library will pool all requests and execute in a FIFO fashion. The number of parallel requests increases automatically: We gracefully ramp-up the rate of requests to match PhantomJsCloud's autoscale capacity.

Roadmap

1. web crawler
2. batch api framework (store results to S3, Google Cloud Storage, etc)
3. page automation helpers (button clicking, etc. you can do this already, but you need to know a lot to do it...)

Contributing

This is the official, reference API for PhantomJsCloud.com You can help out by writing API Client Libraries for other languages (lots of requests for PHP and Python!)

Building

If you want to build this yourself:

• install npm install mocha -g
• install dependencies npm install
• Install vscode and run the project (see .vscode/launch.json for actual execution args)