A command line tool for managing proxies & a nodejs component to used them in your own application
a Google Keyword Suggest component that can be used on command line or integrated into your node.js app
Web site generator made for the SEO ! with nodejs, markdown and plug with optionnally cloakings & mindmaps.
A web crawler made for the SEO based on plugins. Please wait or contribute ... still in beta
Expired domains finder for crawler.ninja
A simple component to check the status of a domain (whois, availability, expired, TrustFlow, number of indexed pages, ...)
Common log method for all crawler.ninja plugins
Functions used to transform, create, extract info on URIs
This module allows to execute search on Google with or without proxies. It provides different options for scraping the google results (either the list of the referenced sites or the number of results).
This module allows to execute a search on Google and get infos on the related domains (mainly trust flow, topical trust flow & whois).
Redis store for Crawler.ninja
A set of natural functions like extracting words & n-grams, remove diacritics, get top keywords, ... (experimental project)
Generate a corpus based on a Google search or from a set of urls
A simple web scraper
This module allows to execute a search on Google and get the most important backlinks from the domains's SERP
Access middelware for cocoons.io
Utility functions & logger that can be used in the different cocoons modules
Module for finding the main content on an HTML page with the help of Cheerio