generate-robotstxt
Awesome generator robots.txt.
Installation
npm install --save-dev generate-robotstxt
Usage
; ;
File based configuration
robots-txt.config.js
moduleexports = policy: userAgent: "Googlebot" allow: "/" disallow: "/search" crawlDelay: 2 userAgent: "OtherBot" allow: "/allow-for-all-bots" "/allow-only-for-other-bot" disallow: "/admin" "/login" crawlDelay: 2 userAgent: "*" allow: "/" disallow: "/search" crawlDelay: 10 cleanParam: "ref /articles/" sitemap: "http://example.com/sitemap.xml" host: "http://example.com";
CLI
Awesome generator robots.txt Usage generate-robotstxt [options] <dest> Options: --config Path to a specific configuration file.
Contribution
Feel free to push your code if you agree with publishing under the MIT license.