grunt-robots-txt

Grunt plugin which generates a robots.txt file

grunt-robots-txt

Grunt plugin for generating robots.txt.

Install npm package:

npm install grunt-robots-txt

Add this line to your project's Gruntfile.js:

grunt.loadNpmTasks('grunt-robots-txt');

robotstxt is a multitask, so you can use robotstxt:dist and register some other tasks than "dist":

    grunt.initConfig({
    
        robotstxt: {
            dist: {
                dest: 'src/',
                policy: [
                    {
                        ua: 'googlebot',
                        disallow: '/private/'
                    },
                    {
                        ua: 'googlebot-news',
                        disallow: ['/other/', '/another/'],
                        allow: '/new/'
                    },
                    {
                        ua: 'googlebot-third',
                        allow: ['/new1/', 'new2']
                    },
                    {
                        sitemap: ['http://example.com/sitemap.xml', 'http://alernate.org/sitemap.xml']
                    },
                    {
                        crawldelay: 100
                    },
                    {
                        host: 'www.example.org'
                    }
                ]
            }
        }
    
    });
  • dest: String (./ by default)

    Path where robots.txt will be created

  • policy: Object (`` by default)

    Hash of your policy. Note that you can give an array of values for disallow in order to create multiple Disallow lines for the concerned User-agent. Same is possible for Allow and Sitemap lines.

    There can be only 1 Crawl-delay option, and only 1 Host option. If you supply more, only th last one will be added to the robots.txt file.

    As Sitemap, Crawl-delay and Host are non-standard, if you don't supply any of these in your config, your robots.txt will not include these options by default.

There is a build process (which involves Grunt) for this Grunt task. Basically, the process will check the code with JSHint and JSCS, and execute some tests.

You can launch this process with the grunt command:

grunt

Or if you just want to test (and not check code):

grunt test

This plugin for Grunt is released under the MIT License.