npm

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

• robots.txt
• parser
• user-agent
• scraper
• spider
• bot
• robots-exclusion-standard

samclarke

published 3.0.0 • 7 months agopublished 3.0.0 7 months ago

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

• robots.txt
• robots
• robots-parser
• robots-exclusion-standard
• javascript
• js
• node-js
• robots-txt
• crawler
• crawlers

ibragim64

published 0.0.3-dev • 2 years agopublished 0.0.3-dev 2 years ago