Using Node.js and JQuery to Crawl Public Tweets
A talk by @benjamincoe
Birdeater
Birdeater is a command-line tool for backing up a user's public Tweets in JSON format.
Usage
First, make sure you have node package manager installed:
curl http://npmjs.org/install.sh | sh
Then run the following commands to install and run Birdeater:
npm install birdeater -gbirdeater --user=shitmydadsays
Be mindful when running it, as Twitter limits the number of requests that a single client can make per hour.
How Birdeater Works
Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML. Here's how it works:
An http connection is made to a user's public timeline using the request library:
Userprototype { ;};
This returns an HTML representation of the tweets. JQuery is used to extract structured information from this:
;
I find that Node.js, coupled with JQuery, works great for building web crawlers:
- Node.js offers a non-blocking evented paradigm. This is awesome for a web-crawler, which will tend to be I/O bound.
- JQuery is a champ at parsing HTML. I find it much more intuitive than other libraries that I've used, such as Beautiful Soup.
- First and foremost, it's easy to do (take a look at lib/user.js Birdeater clocks in at about 100 lines of code).
This approach has become my hammer when web scraping tasks come up.
- Ben (@benjamincoe)