Site scraper for Congregator
This is an attempt to make a website scraper/parser that uses JSON templates to parse a website into article entries. It will give you an output of formatted JSON-articles which can be processed at will. It also has the ability to visit the article links and fetch a processed version of the content (done by node-read). Now go and build your own kimono! :)
var debug = require'debug''sitescraper:testapp';var util = require'util';var events = require'events';var ipc = ;return elementactive;vardebugutilinspectitem colors: true ;callbacknull item;;varvar sites = require'./template';callbacknull sitesfilterisActive;;var SiteScraper = require'sitescraper';var siteScraper =getSources: getSiteshandleEntry: handleEntryipc: ipcsockets: 15 // experiment with this number to avoid getting blocked connections (DDOS protection)waitTime: 10000timeOut: 5000;console.log'running scraper';siteScraperrun;
"site": "Hacker News""source": """host": "news.ycombinator.com""origin": "site""category": 1"guid": """url": """title": "Last day to apply to YC Hacks""ranking": 23"content":"title": "Last day to apply to YC Hacks""body": "<div class=\"post-body\" id=\"post_body_708360\"> <p>It's the last day to apply to Y Combinator's first hackathon, <a href=\"\">YC Hacks</a>. </p><p>The hackathon will be hosted at YC's office in Mountain View, CA on August 2-3. Our goal is to give smart hackers an excuse to get together and spend time building something they find interesting. We don't have a theme—we want to leave it open to any good ideas.<br></p><p>Kickoff will be at noon on Saturday, August 2. YC companies that develop platforms, services and developer tools, will be around to act as mentors. Judging will happen the evening of August 3. YC alumni are donating prizes, and the top teams will get guaranteed YC interviews for the next batch.</p><div>You can apply individually or as a team. Please have each team member fill out an application.</div><div><br></div><div>Apply <a href=\"\">here</a> by 11:59pm PST tonight. Invitations will be sent by July 7.<br> </div><div><br></div> </div>""image": ""