npm

Join us for the upcoming webinar, "Enterprise JavaScript done right: the tools you love, the security you need."Sign up here »

@crawlify.io/robots

1.0.1 • Public • Published

Crawlify/Robots

A robots.txt parser for node.js

This package comes with two options for parsing robots.txt files, either fetch - which retrieves the file from the URL provided and parses the response, or parse - which simply parses the textual response provided.

let RobotFetch = require('robot');
RobotFetch.fetch('https://reckless.agency/robots.txt', function() {
  console.log(RobotFetch.rulesets);
  console.log(RobotFetch.sitemaps);
});
 
let RobotParse = require('robot');
RobotParse.parse(someRobotsContent, function() {
    console.log(RobotParse.rulesets);
    console.log(RobotParse.sitemaps);
});

If any lines of the robots.txt cannot be understood by the parser, they will be returned in Robot.unknown.

More features will be added as we move forward.

install

npm i @crawlify.io/robots

Downloadsweekly downloads

1

version

1.0.1

license

MIT

homepage

github.com

repository

Gitgithub

last publish

collaborators

  • avatar
  • avatar
Report a vulnerability