A robots.txt parser for node.js
This package comes with two options for parsing robots.txt files, either fetch - which retrieves the file from the URL provided and parses the response, or parse - which simply parses the textual response provided.
let RobotFetch = ;RobotFetch;let RobotParse = ;RobotParse;
If any lines of the robots.txt cannot be understood by the parser, they will be returned in
More features will be added as we move forward.