Exclusion
A simple and RFC 9309 compliant robots.txt
parser for Node.js.
Get Started
npm install --save exclusion
yarn add exclusion
This is the robots.txt
parser used by the
netscrape
package. You may want to
check out
NetScrape
for most simple bot use-cases.
Usage
const { RobotsTxt } = require('exclusion');
// Fetch a robots.txt file...
// Pass a robots.txt string to initialize the parser
const robotsTxt = new RobotsTxt(myRobotsTxtString);
// Check a path
robotsTxt.isPathAllowed('/certain/path', 'MyUserAgent');
// returns boolean
// Check a delay
robotsTxt.getDelay('MyUserAgent');
// returns delay in milliseconds or undefined
License
MIT (C) 2023 Russell Steadman. See LICENSE file. Visit Google deps.dev for dependency license information.