Search results
60 packages found
A specification compliant robots.txt parser with wildcard (*) matching support.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Generate a robots.txt for Astro
Middleware for Express to add a robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A proper robots.txt parser and combinator that works with eulalie
Vite plugin for generating robots.txt for production and development builds
It uses the user-agents.org xml file for detecting bots.
The grunt task for html-snapshots
Parser for robots.txt
Express middleware for generating your robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Deney all robots express middleware
Check user agents against robots.txt files