robots-txt-guard

1.0.2 • Public • Published

robots-txt-guard Build Status

Validate urls against robots.txt rules. Configure with output from robots-txt-parse

Usage

var guard = require('robots-txt-guard');

var robotsTxt = guard({
  groups: [{
    agents: [ '*' ],
    rules: [
      { rule: 'allow', path: '/' }
    ]
  }, {
    agents: [ 'googlebot', 'twitterbot' ],
    rules: [
      { rule: 'disallow', path: '/tmp/*' },
      { rule: 'disallow', path: '/temporary/*' },
      { rule: 'noindex', path: '/temporary/*' }
    ]
  }]
});

robotsTxt.isAllowed('Googlebot', '/tmp/abc'); // false
robotsTxt.isAllowed('mozilla', '/tmp/abc'); // true
robotsTxt.isAllowed('googlebot-news', '/home.html'); // true

robotsTxt.isIndexable('googlebot', '/tmp/*'); // true
robotsTxt.isIndexable('googlebot', '/temporary/*'); // false

Readme

Keywords

none

Package Sidebar

Install

npm i robots-txt-guard

Weekly Downloads

12,871

Version

1.0.2

License

MIT

Unpacked Size

423 kB

Total Files

14

Last publish

Collaborators

  • woorank-admin
  • janpotoms
  • ndemoor
  • woorank-ci