npm

Ready to take your JavaScript development to the next level? Meet npm Enterprise - the ultimate in enterprise JavaScript.Learn more »

kerouac-robotstxt

0.4.2 • Public • Published

kerouac-robotstxt

Kerouac middleware that gives instructions to web crawlers using the Robots Exclusion Protocol.

Status: Version Build Quality Coverage Dependencies

Sponsorship

Kerouac is open source software. Ongoing development is made possible by generous contributions from individuals and corporations. To learn more about how you can help keep this project financially sustainable, please visit Jared Hanson's page on Patreon.

Install

$ npm install kerouac-robotstxt

Usage

Simply declare a robots.txt page, using this middleware.

site.page('/robots.txt', require('kerouac-robotstxt')());

The generated output will include a /robots.txt resource. If your site contains any sitemaps, which can be generated using kerouac-sitemap, the sitemap locations will be included so that search engines will automatically discover all pages of your site.

License

The MIT License

Copyright (c) 2012-2018 Jared Hanson <http://jaredhanson.net/>

install

npm i kerouac-robotstxt

version

0.4.2

license

MIT

homepage

github.com

repository

Gitgithub

last publish

collaborators

  • avatar
Report a vulnerability