Search results

2 packages found

A simple crawler made in JavaScript for Node.

published version 1.0.1, 8 months ago1 dependents licensed under $MIT
37

A really lightweight and simple robots-txt parser.

published version 1.0.0, 7 months ago0 dependents licensed under $MIT
14