Kayako Twitter challenge
npm install --save larry-crawler
Navigate to the
node_modules directory which contains larry-crawler.
cd larry-crawler/usagenode get-tweets.js
The application fetches tweets in batches of 100. Unless forcefully killed (CTRL+C), the app will keep running until all tweets matching the defined criteria have been fetched. See result.
NOTE: A batch might produce less than 100 tweets in output if you've applied a secondary filter (like retweetCounts).
If 100 tweets were retrieved based on specified HashTag and 30 of them haven't been retweeted, then only 70 tweets are supplied in the
To access the class larry-crawler exposes for crawling twitter:
const TwitterCrawler = ;
Get your app or user credentials from https://dev.twitter.com/, then create a new object like:
const crawler =consumerKey: processenvTWITTER_CONSUMER_KEYconsumerSecret: processenvTWITTER_CONSUMER_SECRETaccessTokenKey: processenvTWITTER_ACCESS_TOKEN_KEYaccessTokenSecret: processenvTWITTER_ACCESS_TOKEN_SECRET;
If you have a twitter app, use
bearerToken instead of
The new object exposes method
getTweets() to fetch tweets based on criteria and returns a
const criteria = hashtags: 'custserv' retweetCount: $gt: 0 ;crawler;
To set the
max_id parameter for pagination,
criteriamaxIdString = statusid_str
status is an item in the
See get-tweets.js for a full example.
The module has only 1 dependancy - twitter.
Since a maximum of 100 tweeets are sent per request, an effective pagination strategy had to be implemented using the
max_id parameter so we can retrieve ALL the tweets since the very beginning. This strategy was followed to achieve pagination.
The primary challenge was to deal with the 64-bit integer ID provided by the Twitter API. JS can only provide precision upto 53 bits. Hence, the application uses
id_str field at all times and a special decrement function has been written in
usage/utils.js to operate on the string ID.