robots-txt-parse

2.0.1 • Public • Published

robots-txt-parse Build Status

Streaming robots.txt parser

usage

const parse = require('robots-txt-parse');
const fs = require('fs');

const input = fs.createReadStream(__dirname + '/robots.txt');
const result = await parse(input);

assuming this file

user-agent: *
user-agent: googlebot
disallow: /

user-agent: twitterbot
disallow: /
allow: /twitter

user-agent: mozilla
disallow: /path
noindex: /path

Sitemap: http://www.example.com/sitemap.xml

produces following output

{
  "groups": [{
    "agents": [ "*", "googlebot" ],
    "rules": [
      { "rule": "disallow", "path": "/" }
    ]
  }, {
    "agents": [ "twitterbot" ],
    "rules": [
      { "rule": "disallow", "path": "/" },
      { "rule": "allow", "path": "/twitter" }
    ]
  }, {
    "agents": [ "mozilla" ],
    "rules": [
      { "rule": "disallow", "path": "/path" },
      { "rule": "noindex", "path": "/path" }
    ]
  }],
  "extensions": [
    { "extension": "sitemap", "value": "http://www.example.com/sitemap.xml" }
  ]
}

Readme

Keywords

none

Package Sidebar

Install

npm i robots-txt-parse

Weekly Downloads

13,026

Version

2.0.1

License

MIT

Unpacked Size

12.2 kB

Total Files

17

Last publish

Collaborators

  • woorank-admin
  • janpotoms
  • ndemoor
  • woorank-ci