koa-robots.txt
koa middleware to handle different robots.txt responses on different domains. The goal with this middleware, is to avoid search engine indexing of public dev/test/demo/staging-servers etc.
Install
Install with npm:
npm install koa-robots.txt --save
Setup
A minimum of an array of domain names must be supplied. Optional the response text for allowed domains and disallowed domains can be passed.
robotsTxt(allowedDomains[, allowedText][, disallowedText]);
Simplest usage:
var robotsTxt = require('koa-robots.txt');
var robotsTxtHandler = robotsTxt(['domain.com', 'www.domain.com']);
app.use(robotsTxtHandler);
Defaults
allowedText defaults to:
User-agent: *
Disallow:
disallowedText defaults to:
User-agent: *
Disallow: /