jwebquery

An jQuery style web crawler(actually extend jquery).

jwebquery

jQuery like web crawler

Note

It's my first node modules and features are not well tested now.Since it's just a crawler,use it on your own risk.

Feature

  • gzip support
  • http/https support
  • cookie support
  • can set page fetch/download interval,and max parallel download limitations.
  • http proxy support
  • GET method support ,POST is not tested and upload multipart is not supported
  • auto add http head Referer

Example

var jwebquery = require("jwebquery");
(new jwebquery.Page("http://www.stackoverflow.com/")).ready(function(page){
    //get things by css selector 
    //we get every summery text of index page of stackoverflow 
    page.find(".question-summary .summary h3").each(function(index){
    //access jquery from jwebquery  
    console.log(index,jwebquery.$(this).text());
    })
}).on("error",function(err){
    console.log(err);
})
 
 
//use localhost:8099 as http proxy 
jwebquery.setProxy("http://localhost:8099/");