robotstxt-util
TypeScript icon, indicating that this package has built-in type declarations

3.0.0 • Public • Published

robotstxt-util

RFC 5234 spec compliant robots.txt builder and parser. 🦾

NPM npm version

Install

npm i -D robotstxt-util

Use

Build robots.txt documents on fly:

import RobotsTxt from 'robotstxt-util'
// or const RobotsTXT = require('robotstxt-util')

const builder = new RobotsTXT()
const somebot = builder.newGroup('somebot')
somebot.disallow('/')
// or builder.disallow('/', 'somebot')

const otherbots = builder.newGroup('*')
otherbots.allow('/app')
otherbots.disallow(['/api', '/data'])

// additional products:
builder.newProduct('Sitemap', 'https://test.com/en/sitemap.xml')
builder.newProduct('Sitemap', 'https://test.com/tr/sitemap.xml')

Parse existing robots.txt documents:

const RobotsTXT = require('robotstxt-util')

const data = `

User-agent: Googlebot
Disallow: /nogooglebot/
Disallow: /api

User-agent: *
Allow: /
Disallow: /api

Sitemap: https://test.com/en/sitemap.xml
Sitemap: https://test.com/tr/sitemap.xml

`
const parser = new RobotsTXT(data)
parser.valid() // true|false
parser.share('json') // returns an object 👇
{
  groups: {
    Googlebot: {
      rules: [
        {disallow: '/nogooglebot/'},
        {disallow: '/api'}
      ]
    },
    '*': {
      rules: [
        {allow: '/'},
        {disallow: '/api'}
      ]
    }
  },
  additional: [
    {Sitemap: 'https://test.com/en/sitemap.xml'},
    {Sitemap: 'https://test.com/tr/sitemap.xml'}
  ]
}
parser.share('robotstxt') // returns spec compliant txt 👇
`
user-agent: Googlebot
disallow: /nogooglebot/
disallow: /api

user-agent: *
allow: /
disallow: /api

Sitemap: https://test.com/en/sitemap.xml
Sitemap: https://test.com/tr/sitemap.xml
`

Parse Previously Exported Object

const RobotsTXT = require('robotstxt-util')

const builder = new RobotsTXT()

const obj = {
  groups: {
    Googlebot: {
      rules: [
        {disallow: '/nogooglebot/'},
        {disallow: '/api'}
      ]
    },
    '*': {
      rules: [
        {allow: '/'},
        {disallow: '/api'}
      ]
    }
  },
  additional: [
    {Sitemap: 'https://test.com/en/sitemap.xml'},
    {Sitemap: 'https://test.com/tr/sitemap.xml'}
  ]
}

builder.load(obj)

Contributing

If you're interested in contributing, read the CONTRIBUTING.md first, please.


Version management of this repository done by releaser 🚀


Thanks for watching 🐬

Support me on Patreon

Package Sidebar

Install

npm i robotstxt-util

Weekly Downloads

5

Version

3.0.0

License

MIT

Unpacked Size

18.9 kB

Total Files

16

Last publish

Collaborators

  • muratgozel