Newman's Personal Motorcade

    puppeteer-response-waiter
    TypeScript icon, indicating that this package has built-in type declarations

    1.6.0 • Public • Published

    Puppeteer-response-waiter

    build tests

    Introduction

    this package is useful when you need to wait for all responses to be received to do something like manipulating the DOM, usually when you need to track many requests at once or some requests are lately received.

    some use cases could be scraping an infinite scroll page and mostly you do not know which requests to track or to wait for.

    it a simple but powerful package, it may be used even to wait for thousands of requests at once and it guarantees to wait for all responses.

    some requests may or may not happen, when a request does not happen Puppeteer-response-waiter just skip without throwing any error.

    NOTE: if you may want to assert that a request is triggered and wait for it you may use the puppeteer built-in page function waitForResponse.

    Installation

    using npm

    npm i puppeteer-response-waiter
    

    using yarn

    yarn add puppeteer-response-waiter
    

    Usage

    A sample example to wait for all responses to be back before doing something

    const puppeteer = require('puppeteer');
    const {ResponseWaiter} = require('puppeteer-response-waiter');
    
    let browser = await puppeteer.launch({ headless: false });
    let page = await browser.newPage();
    let responseWaiter = new ResponseWaiter(page);
    await page.goto('http://somesampleurl.com');
    // start listening
    responseWaiter.listen();
    // do something here to trigger requests
    await responseWaiter.wait();
    // all requests are finished and responses are all returned back
    
    // remove listeners
    responseWaiter.stopListening();
    await browser.close();

    Wait for all image responses to be back before doing something

    using waitFor option you can filter the requests you need to wait for,combine it with onResponse you can control flow of requests and responses as you want

    const puppeteer = require('puppeteer');
    const {ResponseWaiter} = require('puppeteer-response-waiter');
    
    let browser = await puppeteer.launch({ headless: false });
    let page = await browser.newPage();
    let responseWaiter = new ResponseWaiter(page, {
            waitFor: (req) => req.resourceType() == 'image'
            
    });
    await page.goto('http://somesampleurl.com');
    // start listening
    responseWaiter.listen();
    // do something here to trigger requests
    await responseWaiter.wait();
    // all requests are finished and responses are all returned back
    
    // remove listeners
    responseWaiter.stopListening();
    await browser.close();

    Wait and use all JSON responses

    combine waitFor with onResponse you can control flow of requests and responses as you want to check correct responses, download responses, files, images...etc.

    const puppeteer = require('puppeteer');
    const {ResponseWaiter} = require('puppeteer-response-waiter');
    
    let browser = await puppeteer.launch({ headless: false });
    let page = await browser.newPage();
    let responseWaiter = new ResponseWaiter(page, {
        waitFor: (req) => req.resourceType() == 'fetch',
        // get you response here and do something with it
        onResponse: async (response)=> console.log(await response.json())// do something with response
    });
    await page.goto('http://somesampleurl.com');
    // start listening
    responseWaiter.listen();
    // do something here to trigger requests
    await responseWaiter.wait();
    // all requests are finished and responses are all returned back
    
    // remove listeners
    responseWaiter.stopListening();
    await browser.close();

    An example using custom timeout

    const puppeteer = require('puppeteer');
    const {ResponseWaiter} = require('puppeteer-response-waiter');
    
    let browser = await puppeteer.launch({ headless: false });
    let page = await browser.newPage();
    let responseWaiter = new ResponseWaiter(page, {
            timeout: 500,
    });
    await page.goto('http://somesampleurl.com');
    // start listening
    responseWaiter.listen();
    // do something here to trigger requests
    await responseWaiter.wait();
    // all requests are finished and responses are all returned back
    
    // remove listeners
    responseWaiter.stopListening();
    await browser.close();

    NOTE: The timeout option is really mandatory for the package to work, choosing the right timeout depends on the network and resource that your script is using, for most cases 100-500ms are just fine, by default Puppeteer-response-waiter use 100ms.

    Other considerations

    Sometimes when you may navigate before the requests have finished, by default Puppeteer-response-waiter will reset request count and resolve directly, you can customize this behavior(not recommended it will hang your code infinitely but it is something you may want to know).

    Install

    npm i puppeteer-response-waiter

    DownloadsWeekly Downloads

    423

    Version

    1.6.0

    License

    MIT

    Unpacked Size

    10 kB

    Total Files

    4

    Last publish

    Collaborators

    • samyouaret