Easily build flexible, scalable, and distributed, web crawlers.
Easily build flexible, scalable, and distributed, web crawlers for Node.js.
var flexible = require'flexible';// Initiate a crawler. Chainable.var crawl = flexiblecrawl'';crawluseflexiblequerystringuseflexiblerouterroute'/users/:name'crawlnavigate'' + reqparamsname;route'/search'console.log'Search document handled for:' reqparamsq;route'*'console.log'Every document is handled by this route.';on'complete' console.log'Finished!';on'error' console.errorerror;;
npm install flexible
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.