Wondering what’s next for npm?Check out our public roadmap! »

    html-get

    2.8.21 • Public • Published
    microlink cdn

    Last version Build Status Coverage Status Dependency status Dev Dependencies Status NPM Status

    Get the HTML from any website, using prerendering when is necessary.

    Features

    • Get HTML markup from any website (client side apps as well).
    • Prerendering detection based on domains whitelist.
    • Speed up process blocking ads trackers.
    • Encoding body response properly.

    Headless technology like puppeteer brings us to get the HTML markup from any website, even when the target URL is client side app and we need to wait until dom events fire for getting the real markup.

    Generally this approach better than a simple GET request from the target URL, but because you need to wait for dom events, prerendering could be slow and in some scenario unnecessary (sites that use server side rendering could be resolved with a simple GET).

    html-get bring the best of both worlds, doing the following algorithm:

    • Determinate if the target URL actually needs prerendering (internally it has a whitelisted of popular site domains that don't need it).
    • If it needs prerendering, perform the action using Headless technology, blocking ads trackers requests for speed up the process, trying to resolve the main request in the minimum amount of time.
    • If it does not need prerendering or prerendering fails for any reason (for example, timeout), the request will be resolved doing a GET request.

    Install

    $ npm install puppeteer html-get --save

    Usage

    'use strict'
    
    const getHTML = require('html-get')
    
    getHTML('https://example.com').then(
      ({ url, html, stats, headers, statusCode }) =>
        console.log(`
           url: ${url}
          html: ${Buffer.from(html).byteLength} bytes (HTTP ${statusCode})
          time: ${stats.timing} (${stats.mode})
       headers: ${Object.keys(headers).reduce(
         (acc, key) => `${acc}${key}=${headers[key]} `,
         ''
       )}
    `))

    Command Line

    $ npx html-get https://example.com
    

    API

    getHTML(url, [options])

    url

    Required
    Type: string

    The target URL for getting the HTML markup.

    options

    prerender

    Type: boolean|string
    Default: 'auto'

    Enable or disable prerendering as mechanism for getting the HTML markup explicitly.

    The value auto means that that internally use a list of whitelist website that don't need to use prerendering by default. This list is used for speedup the process, using fetch mode for these websites.

    See getMode parameter for know more.

    getBrowserless

    Type: function

    A function that should return a browserless instance to be used for interact with puppeteer.

    If you don't provide a value, then the library try to load @browserless/pool or browserless from your dependencies.

    encoding

    Type: string
    Default: 'utf-8'

    Encoding the HTML markup properly from the body response.

    It determines the encode to use A Node.js library for converting HTML documents of arbitrary encoding into a target encoding (utf8, utf16, etc).

    headers

    Type: object

    Request headers that will be passed to fetch/prerender process.

    getMode

    Type: function

    A function evaluation that will be invoked to determinate the resolutive mode for getting the HTML markup from the target URL.

    The default getMode is:

    const getMode = (url, { prerender }) => {
      if (prerender === false) return 'fetch'
      if (prerender !== 'auto') return 'prerender'
      return autoDomains.includes(getDomain(url)) ? 'fetch' : 'prerender'
    }
    gotOptions

    Type: object

    Under mode=fetch, pass configuration object to got.

    puppeteerOpts

    Type: object

    Under non mode=fetch, pass configuration object to puppeteer.

    rewriteUrls

    Type: boolean
    Default: false

    When is true, it will be rewritten CSS/HTML relatives URLs present in the HTML markup into absolutes.

    License

    html-get © Microlink, Released under the MIT License.
    Authored and maintained by Kiko Beats with help from contributors.

    microlink.io · GitHub @MicrolinkHQ · Twitter @microlinkhq

    Install

    npm i html-get

    DownloadsWeekly Downloads

    425

    Version

    2.8.21

    License

    MIT

    Unpacked Size

    59.3 kB

    Total Files

    9

    Last publish

    Collaborators

    • avatar