open-graph-scraper-bypasshtmlcheck

4.11.0-1 • Public • Published

openGraphScraper

Node.js CI Known Vulnerabilities

A simple node module for scraping Open Graph and Twitter Card info off a site. For browser usage, we recommend using ky to make the requests(or a backend service) then pass in the html into open-graph-scraper using the html option.

Installation

npm install open-graph-scraper --save

Usage

Callback Example:

const ogs = require('open-graph-scraper');
const options = { url: 'http://ogp.me/' };
ogs(options, (error, results, response) => {
  console.log('error:', error); // This returns true or false. True if there was an error. The error itself is inside the results object.
  console.log('results:', results); // This contains all of the Open Graph results
  console.log('response:', response); // This contains the HTML of page
});

Promise Example:

const ogs = require('open-graph-scraper');
const options = { url: 'http://ogp.me/' };
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('error:', error);  // This returns true or false. True if there was an error. The error itself is inside the results object.
    console.log('result:', result); // This contains all of the Open Graph results
    console.log('response:', response); // This contains the HTML of page
  })

Results JSON

Check the return for a success flag. If success is set to true, then the url input was valid. Otherwise it will be set to false. The above example will return something like...

{
  ogTitle: 'Open Graph protocol',
  ogType: 'website',
  ogUrl: 'http://ogp.me/',
  ogDescription: 'The Open Graph protocol enables any web page to become a rich object in a social graph.',
  ogImage: {
    url: 'http://ogp.me/logo.png',
    width: '300',
    height: '300',
    type: 'image/png'
  },
  requestUrl: 'http://ogp.me/',
  success: true
}

Options

Name Info Default Value Required
url URL of the site. x
timeout Timeout of the request 2000 ms
html You can pass in an HTML string to run ogs on it. (use without options.url)
blacklist Pass in an array of sites you don't want ogs to run on. []
onlyGetOpenGraphInfo Only fetch open graph info and don't fall back on anything else. false
ogImageFallback Fetch other images if no open graph ones are found. true
customMetaTags Here you can define custom meta tags you want to scrape. []
allMedia By default, OGS will only send back the first image/video it finds false
decompress Set the accept-encoding to gzip/deflate true
followRedirect Defines if redirect responses should be followed automatically. true
maxRedirects Max number of redirects ogs will follow. 10
retry Number of times ogs will retry the request. 2
headers An object containing request headers. Useful for setting the user-agent {}
peekSize Sets the peekSize for the request 1024
agent Used for Proxies, Look below for notes on how to use. null
downloadLimit Maximum size of the content downloaded from the server, in bytes 1000000 (1MB)
urlValidatorSettings Sets the options used by validator.js for testing the URL Here

Note: open-graph-scraper uses got for requests and most of got's options should work as open-graph-scraper options.

Custom Meta Tag Example

const ogs = require('open-graph-scraper');
const options = {
  url: 'https://github.com/jshemas/openGraphScraper',
  customMetaTags: [{
    multiple: false, // is there more than one of these tags on a page (normally this is false)
    property: 'hostname', // meta tag name/property attribute
    fieldName: 'hostnameMetaTag', // name of the result variable
  }],
};
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('hostnameMetaTag:', result.hostnameMetaTag); // hostnameMetaTag: github.com
  })

Proxy Example

Look here for more info on how to use proxies.

const ogs = require('open-graph-scraper');
const tunnel = require('tunnel');
const options = {
  url: 'https://whatismyipaddress.com/',
  timeout: 15000,
  agent: {
    // setting proxy agent for https requests
    https: tunnel.httpsOverHttp({
      // test proxies can be found here: https://hidemy.name/en/proxy-list/?country=US&type=h#list or http://free-proxy.cz/en/proxylist/country/US/https/ping/all
      proxy: {
        host: 'proxy_ip',
        port: proxyPort,
        rejectUnauthorized: false,
      }
    })
  }
};
ogs(options)
  .then((data) => {
    const { error, result, response } = data;
    console.log('response:', response); // you should see the proxy IP in here
  })

User Agent Example

const ogs = require("open-graph-scraper");
const options = {
  url: "https://twitter.com/elonmusk/status/1364826301027115008",
  headers: {
    "user-agent": "Googlebot/2.1 (+http://www.google.com/bot.html)",
  },
};
ogs(options, (error, results) => {
  console.log("error:", error); // This returns true or false. True if there was an error. The error itself is inside the results object.
  console.log("results:", results); // This contains all of the Open Graph results
});

Tests

Then you can run the tests by running...

npm run test

Package Sidebar

Install

npm i open-graph-scraper-bypasshtmlcheck

Weekly Downloads

0

Version

4.11.0-1

License

MIT

Unpacked Size

61.9 kB

Total Files

13

Last publish

Collaborators

  • shatindle