Parser for XML Sitemaps to be used with Robots.txt and web crawlers
robots.txt agent with cache.
Check user agents against robots.txt files
A proper robots.txt parser and combinator that works with eulalie
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Express middleware to serve and generate robots.txt
Awesome generator robots.txt
A webpack plugin to output a robots.txt file
Parse robot directives within HTML meta and/or HTTP headers.
Deney all robots express middleware
Parser for robots.txt
Robots.txt generator done right.
Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites
Middleware for Express to add a robots.txt
The grunt task for html-snapshots
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)
powered by npms.io 🚀