@mixmaxhq/express-robots

1.0.0 • Public • Published

express-robots

Express middleware for generating a robots.txt or responding with an existing file.

Using a file

app.use(robots(__dirname + '/robots.txt'));

Using an object

Basic object

app.use(robots({UserAgent: '*', Disallow: '/'}))

Will produce:

UserAgent: *
Disallow: /

Crawl Delay

You can optionally pass a CrawlDelay in just like passing in Disallow

app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5'}))

Will produce:

UserAgent: *
Disallow: /
Crawl-delay: 5

Or an array of objects

app.use(robots([
  {
    UserAgent: 'Googlebot',
    Disallow: '/no-google'
  },
  {
    UserAgent: 'Bingbot',
    Disallow: '/no-bing'
  }
]));

Will produce:

UserAgent: Googlebot
Disallow: /no-google
UserAgent: Bingbot
Disallow: /no-bing

Or either property (UserAgent | Disallow) as an array

app.use(robots([
  {
    UserAgent: [ 'Googlebot', 'Slurp' ],
    Disallow: [ '/no-google', '/no-yahoo' ]
  },
  {
    UserAgent: '*',
    Disallow: [ '/no-bots', '/still-no-bots' ]
  }
]));

Will produce:

UserAgent: Googlebot
UserAgent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
UserAgent: *
Disallow: /no-bots
Disallow: /still-no-bots

Package Sidebar

Install

npm i @mixmaxhq/express-robots

Weekly Downloads

371

Version

1.0.0

License

MIT

Unpacked Size

7.42 kB

Total Files

7

Last publish

Collaborators

  • myrosalvdorykevych
  • asiaomelianenko
  • alan-hadyk
  • leonardo-mixmax
  • vadymtemchenko
  • dmytro.prysiazhnyi
  • laszlovaspal
  • akashbhalla
  • hishank
  • oleksii.mandrichenko
  • danyloderkach
  • sergey.polyarush
  • jordins
  • eugene.safronov.dewais
  • gusth-sa
  • joneslloyd
  • kamilgruszka
  • tharindu-wj
  • njogz
  • aminao
  • enuma
  • adriandmitroca
  • mixmaxkristina
  • corridonimatias
  • mikusd
  • mixmax_it
  • andra.lazariuc
  • danaya
  • adborroto
  • aaavakian
  • dnechay
  • raphaelbs
  • guilhermemj
  • eduard.borges
  • mixmax-codeship
  • bradvogel
  • jessem3
  • jsalvata
  • gabriel-mixmax