Search results
121 packages found
Web Builder Module - Create robots.txt file
Creates robots.txt page
A web crawler frontier implementation in TypeScript backed by Redis. MicroFrontier is a scalable and distributed frontier implemented through Redis Queues.
Generate a robots.txt file for your Astro site
A middleware for the @neobeach/core to serve a robots.txt file.
Registers /sitemap.txt, /sitemap.xml and /robots.txt, can be added to Google Search Console.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Higher order Next.js configuration for generating sitemap.xml and robots.txt
Basic robots.txt generator plugin for Hexo
Basic robots.txt generator plugin for Hexo with Host options
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
An autonomous webcrawler for indexing robots.txt files.
Generate Robots.txt file with your custom configs
robots.txt generation for webpack 3
Site mapper utilizing puppeteer to generate a xml sitemap
Ensures polite outgoing HTTP requests that respects robots.txt and aren't made too close to each other
Sitemap generator for TypeScript projects
Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites
A web crawler. Automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Kerouac robots exclusion components for Bixby.js.