node package manager
Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »



UNMAINTAINED | This project is not maintained anymore (as of 2016 November 14).

Grunt plugin for generating robots.txt.

NPM version Dependency Status Build Status


Install npm package:

npm install grunt-robots-txt

Add this line to your project's Gruntfile.js:



robotstxt is a multitask, so you can use robotstxt:dist and register some other tasks than "dist":

        robotstxt: {
            dist: {
                dest: 'src/',
                policy: [
                        ua: 'googlebot',
                        disallow: '/private/'
                        ua: 'googlebot-news',
                        disallow: ['/other/', '/another/'],
                        allow: '/new/'
                        ua: 'googlebot-third',
                        allow: ['/new1/', 'new2']
                        sitemap: ['', '']
                        crawldelay: 100
                        host: ''


  • dest: String ('./' by default)

    Path where robots.txt will be created

  • policy: Object ('' by default)

    Hash of your policy. Note that you can give an array of values for disallow in order to create multiple Disallow lines for the concerned User-agent. Same is possible for Allow and Sitemap lines.

    There can be only 1 Crawl-delay option, and only 1 Host option. If you supply more, only th last one will be added to the robots.txt file.

    As Sitemap, Crawl-delay and Host are non-standard, if you don't supply any of these in your config, your robots.txt will not include these options by default.

Build from sources

There is a build process (which involves Grunt) for this Grunt task. Basically, the process will check the code with JSHint and JSCS, and execute some tests.

You can launch this process with the grunt command:


Or if you just want to test (and not check code):

grunt test

devDependency Status Built with Grunt


This plugin for Grunt is released under the MIT License.