Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Awesome generator robots.txt
Express middleware for generating your robots.txt
A webpack plugin to output a robots.txt file
Express middleware to serve and generate robots.txt
highlight.js syntax definition for Google's robot.txt format
Middleware for Express to add a robots.txt
Check user agents against robots.txt files
The grunt task for html-snapshots
It uses the user-agents.org xml file for detecting bots.
Higher order Next.js configuration for generating sitemap.xml and robots.txt
A proper robots.txt parser and combinator that works with eulalie
Deney all robots express middleware
Parser for robots.txt
Rules related to checking for any SEO issues on the page given
Promise based parser for robots.txt files.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Robots.txt file generator plugin for vuepress