Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A webpack plugin to output a robots.txt file
Awesome generator robots.txt
robots.txt agent with cache.
A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Check user agents against robots.txt files
Middleware for Express to add a robots.txt
Robots.txt bounce middleware for koa
A proper robots.txt parser and combinator that works with eulalie
The grunt task for html-snapshots
Deney all robots express middleware
Rules related to checking for any SEO issues on the page given
robots.txt middleware for Kerouac.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Express middleware to serve and generate robots.txt
Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.
Kerouac robots exclusion components for Bixby.js.
Parse robot directives within HTML meta and/or HTTP headers.
Promise based parser for robots.txt files.
powered by npms.io 🚀