Search results
121 packages found
koa middleware to handle different robots.txt responses on different domains
A simple, RFC-compliant robots.txt parser
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A plugin for Robots Exclusion Standard for webcheck
This library is for - generating sitemap (with image sitemap) - generating robots.txt - easier managment for page specific meta desciption and meta keywords
Awesome generator robots.txt
Grunt plugin to generate robot.txt based on angular paths
A robots.txt reader, parser and matcher.
A Wintersmith plugin for keeping search engines at bay. (Robots.txt)
Robots.txt bounce middleware for koa
generate robots.txt based on env content for static websites
Express middleware to handle different robots.txt responses on different domains
pub-server package for generating robots.txt and sitemap.xml
Webpack plugin to manage SEO files (sitemap, robots.txt) via configuration
Gatsby plugin that merges robots.txt for your site and existing subdirectories
Parse robot directives within HTML meta and/or HTTP headers.
Simple robots.txt file generation for express
A lightweight and simple robots.txt parser in node.
Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.