Search results

robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

• robots
• txt
• robots.txt
• parser
• crawler
• spider
• bot
• robotstxt
• scraper

cakroyd

published 2.0.3 • 2 years agopublished 2.0.3 2 years ago

astro-robots-txt

Generate a robots.txt for Astro

• astro-integration
• astro
• astro-component
• seo
• robots.txt
• robotstxt
• robots-txt

alextim

published 1.0.0 • 7 months agopublished 1.0.0 7 months ago

robotstxt-util

RFC 5234 spec compliant robots.txt builder and parser. 🦾

• robots
• robotstxt
• seo
• search-engine-optimization
• robotstxt-parser
• robotstxt-builder
• rfc5234

muratgozel

published 3.0.0 • 3 months agopublished 3.0.0 3 months ago

exclusion

A simple, RFC-compliant robots.txt parser

• robotstxt
• parser
• bot
• scrape
• spider
• crawl
• robot
• robots
• exclusion

russellsteadman

published 0.2.0 • 6 months agopublished 0.2.0 6 months ago

robots

Parser for robots.txt

• robots.txt
• robotstxt
• parser

ekalinin

published 0.10.1 • 7 years agopublished 0.10.1 7 years ago

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

• robots.txt
• robots
• robotstxt
• generate
• robots-txt
• user-agent
• webpack
• webpack-plugin
• plugin

evilebottnawi

published 8.0.0 • 2 years agopublished 8.0.0 2 years ago

generate-robotstxt

Awesome generator robots.txt

• robotstxt
• robots.txt
• generate
• robots-txt
• user-agent
• police
• allow
• disallow
• crawl-delay
• robot
• robots

evilebottnawi

published 8.0.3 • 4 years agopublished 8.0.3 4 years ago

react-seo-robotstxt

React Robots.txt file Generator

• seo
• robotstxt
• react-seo
• react-robotstxt

javascriptseo

published 1.0.7 • 3 years agopublished 1.0.7 3 years ago

parse-robots

Promise based parser for robots.txt files.

• crawl
• web
• robots.txt
• seo
• robotstxt
• robots
• robots-txt
• sitemap

jaygoodfellow

published 3.2.0 • 4 years agopublished 3.2.0 4 years ago

hexo-generator-robotstxt

Basic robots.txt generator plugin for Hexo

• hexo
• robots
• robotstxt

leecrossley

published 0.2.0 • 9 years agopublished 0.2.0 9 years ago

hexo-generator-robotstxt-host

Basic robots.txt generator plugin for Hexo with Host options

• hexo
• robots
• robotstxt
• host

allimist

published 0.2.1 • 7 years agopublished 0.2.1 7 years ago