Exclusion

A simple and RFC 9309 compliant robots.txt parser for Node.js.

Get Started

npm install --save exclusion

yarn add exclusion

This is the robots.txt parser used by the netscrape package. You may want to check out NetScrape for most simple bot use-cases.

Usage

const { RobotsTxt } = require('exclusion');

// Fetch a robots.txt file...

// Pass a robots.txt string to initialize the parser
const robotsTxt = new RobotsTxt(myRobotsTxtString);

// Check a path
robotsTxt.isPathAllowed('/certain/path', 'MyUserAgent');
// returns boolean

// Check a delay
robotsTxt.getDelay('MyUserAgent');
// returns delay in milliseconds or undefined

License

MIT (C) 2023 Russell Steadman. See LICENSE file. Visit Google deps.dev for dependency license information.