Search results
68 packages found
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A specification compliant robots.txt parser with wildcard (*) matching support.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Sitemap generator for TypeScript projects
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
The grunt task for html-snapshots
A TypeScript-based project for easy serving of static files with support for live reloading, compression, and typed requests.
- TypeScript
- static file serving
- live reload
- compression
- typed requests
- HTTP server
- SSL
- cors
- express middleware
- proxy
- sitemap
- feeds
- robots.txt
- compression (gzip, deflate, brotli)
Vite plugin for generating robots.txt for production and development builds
An Eleventy plugin to generate a robots.txt file for your static site
Parser for robots.txt
A CLI tool to generate robots.txt and sitemap.xml for React projects by scanning for routes
A package to generate sitemaps for Next.js applications
Generate a robots.txt for Astro
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
## About
robots.txt middleware for Kerouac.
Awesome generator robots.txt