Nightly Patch Machine

    robots-txt-guard

    1.0.2 • Public • Published

    robots-txt-guard Build Status

    Validate urls against robots.txt rules. Configure with output from robots-txt-parse

    Usage

    var guard = require('robots-txt-guard');
    
    var robotsTxt = guard({
      groups: [{
        agents: [ '*' ],
        rules: [
          { rule: 'allow', path: '/' }
        ]
      }, {
        agents: [ 'googlebot', 'twitterbot' ],
        rules: [
          { rule: 'disallow', path: '/tmp/*' },
          { rule: 'disallow', path: '/temporary/*' },
          { rule: 'noindex', path: '/temporary/*' }
        ]
      }]
    });
    
    robotsTxt.isAllowed('Googlebot', '/tmp/abc'); // false
    robotsTxt.isAllowed('mozilla', '/tmp/abc'); // true
    robotsTxt.isAllowed('googlebot-news', '/home.html'); // true
    
    robotsTxt.isIndexable('googlebot', '/tmp/*'); // true
    robotsTxt.isIndexable('googlebot', '/temporary/*'); // false

    Keywords

    none

    Install

    npm i robots-txt-guard

    DownloadsWeekly Downloads

    9,189

    Version

    1.0.2

    License

    MIT

    Unpacked Size

    423 kB

    Total Files

    14

    Last publish

    Collaborators

    • woorank-admin
    • janpotoms
    • ndemoor
    • woorank-ci